Skip to content
  1. Home
  2. AI agents
  3. Observability and cost
  4. Agent observability tools compared

Operate · Observability and cost

Agent observability tools compared: what each one does, where it runs and what it costs

Agent observability tools record what an AI agent did on each run as a trace of its model calls, tool calls and hand-offs, then add dashboards, alerts and evaluation on top. This page compares the tools buyers meet most often, with the hosting and the prices each vendor published, checked on 15 November 2026.

If you want the traces, dashboards and alerts run for you after launch, see Managed AI Operations.

In this guide: 84 pages

Key takeaways

  1. The tools split into two groups. Each cloud traces the agents built on its own platform, Microsoft Foundry into Application Insights and Amazon Bedrock AgentCore into CloudWatch, while independent tools such as Langfuse, LangSmith, Arize and Datadog collect traces from any framework and any cloud.
  2. Every vendor bills a different unit: gigabytes ingested on Azure and AWS, spans at Datadog and Arize, units at Langfuse, and seats plus traces at LangSmith. Count the spans per run in a pilot before you compare quotes.
  3. Self-hosting decides many regulated cases. Langfuse and Arize Phoenix can run in your own environment at no licence cost, LangSmith and Arize AX offer it on enterprise plans, and Datadog runs only as a service.
  4. Instrument once with the OpenTelemetry GenAI conventions and the backend becomes a choice you can change later. The conventions still carried Development status on 15 November 2026, with no release yet.
Where each agent observability tool sitsThree stages from left to right. Instrument: the agent framework emits spans named by the OpenTelemetry GenAI conventions, or by OpenInference for Arize. Store and show: either the platform of your cloud (Microsoft Foundry sends traces to Application Insights, Amazon Bedrock AgentCore to CloudWatch, Google Agent Runtime to Cloud Trace) or an independent tool (Langfuse and Arize Phoenix, self-hosted or in their cloud; LangSmith and Arize AX in their cloud, self-hosted on enterprise plans; Datadog as a service). Use: debug one run, alert on tool calls per run, evaluate and annotate, and see cost per use case. Underneath, the billing unit differs: gigabytes ingested on Azure and AWS, spans at Datadog and Arize, units or traces at Langfuse and LangSmith.1 InstrumentIn your codeAgent framework emits spansOpenTelemetry GenAI namesor OpenInference (Arize)2 Store and showYour cloud's platformFoundry to Application InsightsAgentCore to CloudWatchAgent Runtime to Cloud TraceIndependent toolLangfuse, Phoenix: self-host or cloudLangSmith, Arize AX: cloud firstDatadog: as a service3 UseWhat teams doDebug one runAlert on tool calls per runEvaluate and annotateCost per use caseBilled by: GB ingested (Azure, AWS), spans (Datadog, Arize), units or traces (Langfuse, LangSmith)Where each agent observability tool sitsThree stages from left to right. Instrument: the agent framework emits spans named by the OpenTelemetry GenAI conventions, or by OpenInference for Arize. Store and show: either the platform of your cloud (Microsoft Foundry sends traces to Application Insights, Amazon Bedrock AgentCore to CloudWatch, Google Agent Runtime to Cloud Trace) or an independent tool (Langfuse and Arize Phoenix, self-hosted or in their cloud; LangSmith and Arize AX in their cloud, self-hosted on enterprise plans; Datadog as a service). Use: debug one run, alert on tool calls per run, evaluate and annotate, and see cost per use case. Underneath, the billing unit differs: gigabytes ingested on Azure and AWS, spans at Datadog and Arize, units or traces at Langfuse and LangSmith.1 InstrumentIn your codeAgent framework emits spansOpenTelemetry GenAI namesor OpenInference (Arize)2 Store and showYour cloud's platformFoundry to Application InsightsAgentCore to CloudWatchAgent Runtime to Cloud TraceIndependent toolLangfuse, Phoenix: self-host or cloudLangSmith, Arize AX: cloud firstDatadog: as a service3 UseWhat teams doDebug one runAlert on tool calls per runEvaluate and annotateCost per use caseBilled byGB ingested (Azure, AWS); spans (Datadog, Arize);units or traces (Langfuse, LangSmith)
Where each tool sits: the agent framework emits spans, a cloud platform or an independent tool stores and shows them, and teams use them to debug, alert, evaluate and track cost. The billing unit differs by vendor.

What do agent observability tools do?

They collect the trace of every agent run, store it and give a team a way to use it: to debug one run, to alert when runs misbehave, to score quality with evaluators and to see where the money goes.

A trace is the run laid out as a tree of timed steps, called spans: one for the agent, one per model call and one per tool call. The agent framework emits the spans, so the products differ less in what they collect than in what they do with the data afterwards. Langfuse, LangSmith and Arize put evaluation and annotation next to the traces. Datadog places agent traces beside the application monitoring many companies already run, and the cloud platforms keep the traces in the same account as the agent. The tracing sub-page explains the span names; this page compares the products that receive them.

Two terms come up on every vendor page. OpenTelemetry is the open standard for telemetry, and its GenAI semantic conventions name the agent, model-call and tool-call spans. OpenInference is a second naming scheme built on OpenTelemetry and used by Arize. Datadog, Langfuse and LangSmith document support for both, so a team that instruments with either can usually switch backends without rewriting the agent.

How do the main agent observability tools compare?

The table sets out what each tool does, how it is hosted and how its vendor prices it, from each vendor’s own pages as they read on 15 November 2026. We compared the published facts and ran no benchmark, and list prices change, so use the table to build a shortlist and then ask each vendor for a quote.

Agent observability tools, as each vendor’s own pages showed on 15 Nov 2026. Prices are list prices in US dollars, written USD.

ToolWhat it does for agentsHow it is hostedPrice on the vendor pageFits when
LangfuseTracing of agent runs, prompt management, evaluations, annotation and dashboards; accepts OpenTelemetry over HTTPLangfuse Cloud, or self-hosted at no licence cost under the MIT licence, with enterprise features licensed separately; part of ClickHouse since January 2026Hobby free with 50k units a month and 30 days of data; Core USD 29 a month with 100k units and 90 days; Pro USD 199 a month; Enterprise USD 2,499 a month; extra units from USD 8 per 100k; a unit is a trace, an observation or a score (Langfuse pricing page, checked Nov 2026)You want one open-source tool across frameworks, or traces kept on your own infrastructure
LangSmith (LangChain)Tracing, dashboards and alerts, online and offline evaluation, and annotation queues; accepts OpenTelemetryLangChain’s cloud in the US or the EU; hybrid and self-hosted deployment on the Enterprise planDeveloper USD 0 for one seat with 5k base traces a month; Plus USD 39 per seat per month with 10k base traces; more traces pay as you go; base traces kept 14 days and extended traces 180 days; Enterprise custom (LangChain pricing page, checked Nov 2026)Your team builds on LangChain or LangGraph and wants evaluation next to the traces
Arize Phoenix and Arize AXPhoenix: tracing, evaluation, prompt management, datasets and experiments. AX: the managed platform for the same tracing and evaluation workPhoenix self-hosted at no licence cost under the Elastic License 2.0, on a laptop, Docker or Kubernetes; AX as a service, or self-hosted on the Enterprise planPhoenix self-hosted free; AX Free with 25k spans, 1 GB a month and 15 days; AX Pro USD 50 a month with 50k spans, 10 GB and 30 days; AX Enterprise custom (Arize pricing page, checked Nov 2026)Your data science team already works with OpenInference, or you want a free self-hosted start with a managed option later
Datadog LLM Observability, listed as Agent ObservabilityTraces of agent steps and tool calls; evaluations, experiments, monitors and alerts; checks for personal data and prompt injectionDatadog’s service only; not offered on its US government site40k LLM spans free; the first 100k LLM spans a month with 15-day retention cost USD 160 on an annual plan, USD 200 month to month or USD 240 on demand; then USD 3.50 to USD 9 per 10k spans, by plan and retention (Datadog pricing page, checked Nov 2026)You already run Datadog and want agent incidents next to your application alerts
Microsoft Foundry tracing with Azure Monitor Application InsightsServer-side traces of Foundry agents with no code change once a project is connected; 90 days of traces in the portal; the Agents view (preview) lists Foundry, Copilot Studio and third-party agentsIn your Azure subscription, in your own Application Insights and Log Analytics resourcesNo separate tracing fee from Foundry; Azure Monitor Analytics Logs ingestion USD 2.30 per GB pay as you go in US East, the first 5 GB a month per billing account free, and Application Insights data kept 90 days at no charge (Microsoft Azure Monitor pricing page, checked Nov 2026)Your agents run in Foundry or Copilot Studio
Amazon Bedrock AgentCore Observability with Amazon CloudWatchSession count, latency, duration, token usage and error rates, with spans and logs, for agents on AgentCore; OpenTelemetry compatibleIn your AWS account, in CloudWatch log groups you controlNo separate AgentCore fee, since spans, logs and metrics bill at CloudWatch rates: span ingestion USD 0.35 per GB for the first 10 TB and standard logs ingestion USD 0.50 per GB in US East (AWS AgentCore and CloudWatch pricing pages, checked Nov 2026)Your agents run on AgentCore or elsewhere on AWS
OpenTelemetry GenAI semantic conventionsThe shared names for agent, model-call and tool-call spans that most tools above read or writeAn open specification in the open-telemetry/semantic-conventions-genai repository, maintained by the OpenTelemetry GenAI special interest group; Development status and no release on 15 Nov 2026Free; Apache 2.0 licenceAlways, as the instrumentation layer, so the backend stays a choice you can change

MLflow Tracing, open source under the Apache 2.0 licence, and W&B Weave, whose pricing page now sits with its owner CoreWeave, also accept agent traces and are worth a look if your data science team already uses them.

How do the pricing models differ, and what drives the bill?

Each vendor bills a different unit, so the same agent costs a different amount in each tool. Find your own numbers first: runs per month, spans per run, and whether prompts and tool results are captured, because content capture multiplies the data stored.

Langfuse counts units, where a unit is a trace, an observation or a score, so every span and every evaluation score counts toward the plan. LangSmith charges per seat on its Plus plan and counts traces, with longer retention priced as an extended trace. Arize AX and Datadog count spans, and Datadog prices each retention tier separately. Microsoft and AWS charge mostly by the gigabyte: Foundry adds no tracing fee of its own, so the cost is Azure Monitor ingestion and retention, and AgentCore spans, logs and metrics bill at CloudWatch rates.

A one-week pilot makes the quotes comparable. Run the pilot agent with tracing on, read the span count and the gigabytes ingested from the backend, and multiply up to the volume you expect in production. An agent that plans, calls four tools and hands off once produces several spans per run, so a tool priced per span and a tool priced per trace can differ by that multiple for the same traffic. Retention is the second lever: 14 or 15 days covers most debugging, while audit and evaluation work often needs months. The agent cost monitoring page covers the cost of the agent itself.

Which tool fits which situation?

Start with the platform you build on and add an independent tool when a concrete need appears. Five situations cover most choices.

  • All agents on one cloud: switch on that platform’s tracing first. Foundry traces prompt and hosted agents once a project is connected to Application Insights, and AgentCore sends spans, logs and metrics to CloudWatch, with no separate tracing product to buy from either vendor.
  • Agents across clouds or frameworks: instrument with the OpenTelemetry GenAI conventions and send everything to one independent backend, so a run that crosses Azure and AWS is still one trace in one place.
  • Regulated data that must stay in your environment: self-host Langfuse or Arize Phoenix, or buy the self-hosted enterprise edition of LangSmith or Arize AX, and keep content capture off by default.
  • Teams that already run Datadog: add its agent tracing so agent incidents sit beside the application and infrastructure alerts the on-call team already watches.
  • Teams built on LangChain or LangGraph that want evaluation and annotation beside the traces: LangSmith comes from the same company and is designed around that stack.

Keep the instrumentation vendor-neutral whichever backend you pick. Moving to another backend is then an exporter setting and a dashboard rebuild, which costs far less than instrumenting every agent again.

What should you ask a vendor before you sign?

Ask about the data first and the features second, because traces hold prompts, tool arguments and tool results, and those carry personal data and secrets.

  1. Where is trace data stored, in which region, and can we host it ourselves?
  2. Is content capture off by default, and can personal data be redacted before it leaves our process?
  3. How long is each kind of data kept, and can we set retention per project and delete on request?
  4. Which version of the OpenTelemetry GenAI conventions does the product read, and what happens to dashboards when attribute names change?
  5. What is the billing unit, how many units does one of our pilot runs produce, and what does usage above the plan cost?
  6. Can we export raw traces in OpenTelemetry format if we leave?

Put the answers in the agent’s registry record next to the owner and the risk tier, so the next review of the tool starts from what the vendor committed to in writing.

How do Microsoft, AWS and Google implement it?

Where each cloud sends agent traces and how it bills them. The Microsoft and AWS rows follow the vendor pages as read on 15 November 2026.

Capability, Microsoft, AWS, Google

CapabilityMicrosoftAWSGoogle
Native agent tracingFoundry tracing to Application Insights; generally available for prompt and hosted agents, workflow and external agents in previewAgentCore Observability: spans, logs and metrics in CloudWatch, OpenTelemetry compatibleAgent Runtime tracing to Cloud Trace; span format marked experimental
Separate tracing feeNone from Foundry; Azure Monitor ingestion and retention charges applyNone from AgentCore; charged as per CloudWatch pricingTraces land in Cloud Trace; Google’s pricing page did not render for our check
Price basis, US EastAnalytics Logs ingestion USD 2.30 per GB pay as you go; the first 5 GB a month per billing account free (Microsoft Azure Monitor pricing, Nov 2026)Span ingestion USD 0.35 per GB for the first 10 TB; standard logs ingestion USD 0.50 per GB (AWS CloudWatch pricing, Nov 2026)Not checked for this page
RetentionApplication Insights data 90 days and Analytics Logs 31 days at no charge; longer retention billed per GBSet per log group; stored logs billed per GBCloud Trace and Cloud Logging retention settings
Agents from other frameworksThe Application Insights Agents view (preview) lists agents that send OpenTelemetry GenAI data, from Foundry, Copilot Studio and third partiesCloudWatch accepts OpenTelemetry from Strands Agents, LangChain and LangGraphOpenTelemetry export from Agent Development Kit and Agent Runtime

The Google column repeats what Google’s tracing page showed on 2 Nov 2026. Its pricing page did not render for our check on 15 Nov 2026, so no Google figure appears on this page.

Frequently asked questions

Which agent observability tools can we self-host?

Langfuse can be self-hosted at no licence cost under the MIT licence, with enterprise features sold separately. Arize Phoenix is free to self-host under the Elastic License 2.0. LangSmith and Arize AX offer self-hosting on their enterprise plans, and Datadog runs only as a service.

Do we need an independent tool if we build on Foundry or AgentCore?

Not for agents on one platform. Each traces its own agents, and Microsoft and AWS charge only for the monitoring data. Add an independent tool when agents run on more than one cloud or framework, or when you want evaluation and annotation queues beside the traces.

What is the difference between OpenTelemetry and OpenInference?

Both name the spans an agent emits. OpenTelemetry is the general open standard, and its GenAI conventions cover agents, model calls and tools. OpenInference is a naming scheme built on OpenTelemetry and used by Arize. Datadog, Langfuse and LangSmith document support for both.

How much does agent observability cost?

It depends on the unit each vendor bills and on your volume. The free allowances differ widely: 50,000 units a month on Langfuse Hobby, 5,000 base traces on LangSmith Developer, 25,000 spans on Arize AX Free and 40,000 LLM spans at Datadog. Count a week of pilot traffic and price that volume on each vendor page.

More in Observability and cost

Sources

Each source was read on 15 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.

  1. Langfuse pricing Langfuse, undated page, read 15 November 2026
  2. Self-host Langfuse Langfuse, undated page, read 15 November 2026
  3. Langfuse joins ClickHouse Langfuse, 16 January 2026
  4. LangSmith plans and pricing LangChain, undated page, read 15 November 2026
  5. LangSmith billing LangChain, undated page, read 15 November 2026
  6. Arize AX pricing Arize, undated page, read 15 November 2026
  7. Arize Phoenix Arize, undated page, read 15 November 2026
  8. Phoenix licence (Elastic License 2.0) Arize (GitHub), undated page, read 15 November 2026
  9. Datadog pricing list: Agent Observability Datadog, undated page, read 15 November 2026
  10. OpenTelemetry instrumentation for LLM Observability Datadog, undated page, read 15 November 2026
  11. Tracing agents: concepts Microsoft Learn, 28 August 2026
  12. Set up tracing for agents in Microsoft Foundry Microsoft Learn, 31 July 2026
  13. Azure Monitor pricing Microsoft, undated page, read 15 November 2026
  14. Amazon Bedrock AgentCore pricing AWS, undated page, read 15 November 2026
  15. Amazon CloudWatch pricing AWS, undated page, read 15 November 2026
  16. Observability for Amazon Bedrock AgentCore AWS documentation, undated page, read 15 November 2026
  17. Semantic conventions for GenAI agent spans OpenTelemetry (semantic-conventions-genai repository), undated page, read 15 November 2026
  18. Semantic conventions releases (v1.42.0 moved GenAI to its own repository) OpenTelemetry, 12 June 2026
  19. MLflow Tracing for GenAI MLflow, undated page, read 15 November 2026
  20. W&B pricing Weights & Biases (CoreWeave), undated page, read 15 November 2026. Redirected to the CoreWeave Forge pricing page when read.

Planning an AI program?

Talk to the team that builds it and runs it after launch.

  • Built to SOC 2, HIPAA and GDPR standards
  • EU AI Act-aligned delivery
  • Microsoft Solutions Partner