- Home
- AI agents
- Observability and cost
- Agent observability tools compared
Operate · Observability and cost
Agent observability tools compared: what each one does, where it runs and what it costs
Agent observability tools record what an AI agent did on each run as a trace of its model calls, tool calls and hand-offs, then add dashboards, alerts and evaluation on top. This page compares the tools buyers meet most often, with the hosting and the prices each vendor published, checked on 15 November 2026.
If you want the traces, dashboards and alerts run for you after launch, see Managed AI Operations.
In this guide: 84 pages
Key takeaways
- The tools split into two groups. Each cloud traces the agents built on its own platform, Microsoft Foundry into Application Insights and Amazon Bedrock AgentCore into CloudWatch, while independent tools such as Langfuse, LangSmith, Arize and Datadog collect traces from any framework and any cloud.
- Every vendor bills a different unit: gigabytes ingested on Azure and AWS, spans at Datadog and Arize, units at Langfuse, and seats plus traces at LangSmith. Count the spans per run in a pilot before you compare quotes.
- Self-hosting decides many regulated cases. Langfuse and Arize Phoenix can run in your own environment at no licence cost, LangSmith and Arize AX offer it on enterprise plans, and Datadog runs only as a service.
- Instrument once with the OpenTelemetry GenAI conventions and the backend becomes a choice you can change later. The conventions still carried Development status on 15 November 2026, with no release yet.
What do agent observability tools do?
They collect the trace of every agent run, store it and give a team a way to use it: to debug one run, to alert when runs misbehave, to score quality with evaluators and to see where the money goes.
A trace is the run laid out as a tree of timed steps, called spans: one for the agent, one per model call and one per tool call. The agent framework emits the spans, so the products differ less in what they collect than in what they do with the data afterwards. Langfuse, LangSmith and Arize put evaluation and annotation next to the traces. Datadog places agent traces beside the application monitoring many companies already run, and the cloud platforms keep the traces in the same account as the agent. The tracing sub-page explains the span names; this page compares the products that receive them.
Two terms come up on every vendor page. OpenTelemetry is the open standard for telemetry, and its GenAI semantic conventions name the agent, model-call and tool-call spans. OpenInference is a second naming scheme built on OpenTelemetry and used by Arize. Datadog, Langfuse and LangSmith document support for both, so a team that instruments with either can usually switch backends without rewriting the agent.
How do the main agent observability tools compare?
The table sets out what each tool does, how it is hosted and how its vendor prices it, from each vendor’s own pages as they read on 15 November 2026. We compared the published facts and ran no benchmark, and list prices change, so use the table to build a shortlist and then ask each vendor for a quote.
Agent observability tools, as each vendor’s own pages showed on 15 Nov 2026. Prices are list prices in US dollars, written USD.
| Tool | What it does for agents | How it is hosted | Price on the vendor page | Fits when |
|---|---|---|---|---|
| Langfuse | Tracing of agent runs, prompt management, evaluations, annotation and dashboards; accepts OpenTelemetry over HTTP | Langfuse Cloud, or self-hosted at no licence cost under the MIT licence, with enterprise features licensed separately; part of ClickHouse since January 2026 | Hobby free with 50k units a month and 30 days of data; Core USD 29 a month with 100k units and 90 days; Pro USD 199 a month; Enterprise USD 2,499 a month; extra units from USD 8 per 100k; a unit is a trace, an observation or a score (Langfuse pricing page, checked Nov 2026) | You want one open-source tool across frameworks, or traces kept on your own infrastructure |
| LangSmith (LangChain) | Tracing, dashboards and alerts, online and offline evaluation, and annotation queues; accepts OpenTelemetry | LangChain’s cloud in the US or the EU; hybrid and self-hosted deployment on the Enterprise plan | Developer USD 0 for one seat with 5k base traces a month; Plus USD 39 per seat per month with 10k base traces; more traces pay as you go; base traces kept 14 days and extended traces 180 days; Enterprise custom (LangChain pricing page, checked Nov 2026) | Your team builds on LangChain or LangGraph and wants evaluation next to the traces |
| Arize Phoenix and Arize AX | Phoenix: tracing, evaluation, prompt management, datasets and experiments. AX: the managed platform for the same tracing and evaluation work | Phoenix self-hosted at no licence cost under the Elastic License 2.0, on a laptop, Docker or Kubernetes; AX as a service, or self-hosted on the Enterprise plan | Phoenix self-hosted free; AX Free with 25k spans, 1 GB a month and 15 days; AX Pro USD 50 a month with 50k spans, 10 GB and 30 days; AX Enterprise custom (Arize pricing page, checked Nov 2026) | Your data science team already works with OpenInference, or you want a free self-hosted start with a managed option later |
| Datadog LLM Observability, listed as Agent Observability | Traces of agent steps and tool calls; evaluations, experiments, monitors and alerts; checks for personal data and prompt injection | Datadog’s service only; not offered on its US government site | 40k LLM spans free; the first 100k LLM spans a month with 15-day retention cost USD 160 on an annual plan, USD 200 month to month or USD 240 on demand; then USD 3.50 to USD 9 per 10k spans, by plan and retention (Datadog pricing page, checked Nov 2026) | You already run Datadog and want agent incidents next to your application alerts |
| Microsoft Foundry tracing with Azure Monitor Application Insights | Server-side traces of Foundry agents with no code change once a project is connected; 90 days of traces in the portal; the Agents view (preview) lists Foundry, Copilot Studio and third-party agents | In your Azure subscription, in your own Application Insights and Log Analytics resources | No separate tracing fee from Foundry; Azure Monitor Analytics Logs ingestion USD 2.30 per GB pay as you go in US East, the first 5 GB a month per billing account free, and Application Insights data kept 90 days at no charge (Microsoft Azure Monitor pricing page, checked Nov 2026) | Your agents run in Foundry or Copilot Studio |
| Amazon Bedrock AgentCore Observability with Amazon CloudWatch | Session count, latency, duration, token usage and error rates, with spans and logs, for agents on AgentCore; OpenTelemetry compatible | In your AWS account, in CloudWatch log groups you control | No separate AgentCore fee, since spans, logs and metrics bill at CloudWatch rates: span ingestion USD 0.35 per GB for the first 10 TB and standard logs ingestion USD 0.50 per GB in US East (AWS AgentCore and CloudWatch pricing pages, checked Nov 2026) | Your agents run on AgentCore or elsewhere on AWS |
| OpenTelemetry GenAI semantic conventions | The shared names for agent, model-call and tool-call spans that most tools above read or write | An open specification in the open-telemetry/semantic-conventions-genai repository, maintained by the OpenTelemetry GenAI special interest group; Development status and no release on 15 Nov 2026 | Free; Apache 2.0 licence | Always, as the instrumentation layer, so the backend stays a choice you can change |
MLflow Tracing, open source under the Apache 2.0 licence, and W&B Weave, whose pricing page now sits with its owner CoreWeave, also accept agent traces and are worth a look if your data science team already uses them.
How do the pricing models differ, and what drives the bill?
Each vendor bills a different unit, so the same agent costs a different amount in each tool. Find your own numbers first: runs per month, spans per run, and whether prompts and tool results are captured, because content capture multiplies the data stored.
Langfuse counts units, where a unit is a trace, an observation or a score, so every span and every evaluation score counts toward the plan. LangSmith charges per seat on its Plus plan and counts traces, with longer retention priced as an extended trace. Arize AX and Datadog count spans, and Datadog prices each retention tier separately. Microsoft and AWS charge mostly by the gigabyte: Foundry adds no tracing fee of its own, so the cost is Azure Monitor ingestion and retention, and AgentCore spans, logs and metrics bill at CloudWatch rates.
A one-week pilot makes the quotes comparable. Run the pilot agent with tracing on, read the span count and the gigabytes ingested from the backend, and multiply up to the volume you expect in production. An agent that plans, calls four tools and hands off once produces several spans per run, so a tool priced per span and a tool priced per trace can differ by that multiple for the same traffic. Retention is the second lever: 14 or 15 days covers most debugging, while audit and evaluation work often needs months. The agent cost monitoring page covers the cost of the agent itself.
Which tool fits which situation?
Start with the platform you build on and add an independent tool when a concrete need appears. Five situations cover most choices.
- All agents on one cloud: switch on that platform’s tracing first. Foundry traces prompt and hosted agents once a project is connected to Application Insights, and AgentCore sends spans, logs and metrics to CloudWatch, with no separate tracing product to buy from either vendor.
- Agents across clouds or frameworks: instrument with the OpenTelemetry GenAI conventions and send everything to one independent backend, so a run that crosses Azure and AWS is still one trace in one place.
- Regulated data that must stay in your environment: self-host Langfuse or Arize Phoenix, or buy the self-hosted enterprise edition of LangSmith or Arize AX, and keep content capture off by default.
- Teams that already run Datadog: add its agent tracing so agent incidents sit beside the application and infrastructure alerts the on-call team already watches.
- Teams built on LangChain or LangGraph that want evaluation and annotation beside the traces: LangSmith comes from the same company and is designed around that stack.
Keep the instrumentation vendor-neutral whichever backend you pick. Moving to another backend is then an exporter setting and a dashboard rebuild, which costs far less than instrumenting every agent again.
What should you ask a vendor before you sign?
Ask about the data first and the features second, because traces hold prompts, tool arguments and tool results, and those carry personal data and secrets.
- Where is trace data stored, in which region, and can we host it ourselves?
- Is content capture off by default, and can personal data be redacted before it leaves our process?
- How long is each kind of data kept, and can we set retention per project and delete on request?
- Which version of the OpenTelemetry GenAI conventions does the product read, and what happens to dashboards when attribute names change?
- What is the billing unit, how many units does one of our pilot runs produce, and what does usage above the plan cost?
- Can we export raw traces in OpenTelemetry format if we leave?
Put the answers in the agent’s registry record next to the owner and the risk tier, so the next review of the tool starts from what the vendor committed to in writing.
How do Microsoft, AWS and Google implement it?
Where each cloud sends agent traces and how it bills them. The Microsoft and AWS rows follow the vendor pages as read on 15 November 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Native agent tracing | Foundry tracing to Application Insights; generally available for prompt and hosted agents, workflow and external agents in preview | AgentCore Observability: spans, logs and metrics in CloudWatch, OpenTelemetry compatible | Agent Runtime tracing to Cloud Trace; span format marked experimental |
| Separate tracing fee | None from Foundry; Azure Monitor ingestion and retention charges apply | None from AgentCore; charged as per CloudWatch pricing | Traces land in Cloud Trace; Google’s pricing page did not render for our check |
| Price basis, US East | Analytics Logs ingestion USD 2.30 per GB pay as you go; the first 5 GB a month per billing account free (Microsoft Azure Monitor pricing, Nov 2026) | Span ingestion USD 0.35 per GB for the first 10 TB; standard logs ingestion USD 0.50 per GB (AWS CloudWatch pricing, Nov 2026) | Not checked for this page |
| Retention | Application Insights data 90 days and Analytics Logs 31 days at no charge; longer retention billed per GB | Set per log group; stored logs billed per GB | Cloud Trace and Cloud Logging retention settings |
| Agents from other frameworks | The Application Insights Agents view (preview) lists agents that send OpenTelemetry GenAI data, from Foundry, Copilot Studio and third parties | CloudWatch accepts OpenTelemetry from Strands Agents, LangChain and LangGraph | OpenTelemetry export from Agent Development Kit and Agent Runtime |
The Google column repeats what Google’s tracing page showed on 2 Nov 2026. Its pricing page did not render for our check on 15 Nov 2026, so no Google figure appears on this page.
Frequently asked questions
Which agent observability tools can we self-host?
Langfuse can be self-hosted at no licence cost under the MIT licence, with enterprise features sold separately. Arize Phoenix is free to self-host under the Elastic License 2.0. LangSmith and Arize AX offer self-hosting on their enterprise plans, and Datadog runs only as a service.
Do we need an independent tool if we build on Foundry or AgentCore?
Not for agents on one platform. Each traces its own agents, and Microsoft and AWS charge only for the monitoring data. Add an independent tool when agents run on more than one cloud or framework, or when you want evaluation and annotation queues beside the traces.
What is the difference between OpenTelemetry and OpenInference?
Both name the spans an agent emits. OpenTelemetry is the general open standard, and its GenAI conventions cover agents, model calls and tools. OpenInference is a naming scheme built on OpenTelemetry and used by Arize. Datadog, Langfuse and LangSmith document support for both.
How much does agent observability cost?
It depends on the unit each vendor bills and on your volume. The free allowances differ widely: 50,000 units a month on Langfuse Hobby, 5,000 base traces on LangSmith Developer, 25,000 spans on Arize AX Free and 40,000 LLM spans at Datadog. Count a week of pilot traffic and price that volume on each vendor page.
More in Observability and cost
Sources
Each source was read on 15 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Langfuse pricing
- Self-host Langfuse
- Langfuse joins ClickHouse
- LangSmith plans and pricing
- LangSmith billing
- Arize AX pricing
- Arize Phoenix
- Phoenix licence (Elastic License 2.0)
- Datadog pricing list: Agent Observability
- OpenTelemetry instrumentation for LLM Observability
- Tracing agents: concepts
- Set up tracing for agents in Microsoft Foundry
- Azure Monitor pricing
- Amazon Bedrock AgentCore pricing
- Amazon CloudWatch pricing
- Observability for Amazon Bedrock AgentCore
- Semantic conventions for GenAI agent spans
- Semantic conventions releases (v1.42.0 moved GenAI to its own repository)
- MLflow Tracing for GenAI
- W&B pricing
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner