Operate · Topic 13 of 14
AI agent observability and cost: how to monitor what agents do and what they spend
AI agent observability is the record of what an agent did on every run: what it was asked, which model it called, which tools it used and with what arguments, what it retrieved, who approved what, how long each step took and what it cost. Cost monitoring reads the same record to show where the money goes.
In this guide: 83 pages
Key takeaways
- A trace of nested spans (invoke_agent, chat, execute_tool) is the unit of agent observability. A log of the final answer misses the failure, which is usually a tool call or a hand-off in the middle.
- Capture metadata by default and content by opt-in. Prompts, tool arguments and tool results carry personal data and secrets and need redaction, a separate role and a retention date.
- The OpenTelemetry GenAI conventions are the common language and were still Development status on 2 Nov 2026, in their own repository since June 2026. Pin the version and expect attribute names to change.
- Foundry traces its agents without code changes once connected to Application Insights; AWS and Google collect OpenTelemetry telemetry from their runtimes. Independent tools matter when agents span clouds and frameworks.
- Cost is a view over the same traces, tagged to a team and a use case. Cloud budgets lag by hours or a day, so hard limits belong at the gateway and in the agent loop.
Why does an AI agent need more than application monitoring?
Because the interesting failures leave no error. The call returns 200, the model answers politely, and the answer is wrong because the agent picked the wrong tool, passed the wrong argument or read a stale document.
An agent run is a chain of choices made by a non-deterministic model: which tool to call, with what arguments, what to do with the result, whether to ask a person, when to stop. Two runs with the same input can take different paths, so without the path you cannot explain an outcome to a user, an auditor or yourself. Two publicly reported deployments show the scale this reaches. DKB, a German bank, handles 10,000 to 20,000 conversations a day with its banking assistant and traces every interaction in a self-hosted Langfuse instance, with a second instance running compliance screening by a model as judge (Langfuse case study, 31 Aug 2026). Merck KGaA gives self-hosted tracing to about 80 of its 300-plus generative AI use cases and says it tracks every prompt, response, cost and latency in real time (Langfuse case study, 2 Sep 2025). Both are vendor-published and the figures are the companies’ own.
What should you capture for every agent run?
Capture enough to replay the run: identifiers, model and agent version, every model call, every tool call with its arguments and result, what was retrieved, every hand-off, every approval, the token counts and the errors. The list splits into what the conventions record by default and what you must switch on.
- On by default: conversation and session id (gen_ai.conversation.id), agent name and version, model and provider, latency per span, error.type, and input and output tokens per model call (gen_ai.usage.input_tokens and output_tokens). Google treats user ids as content, so they sit behind the opt-in switch there.
- Opt-in: prompts, outputs and system instructions (gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions), which hold personal data, customer content and sometimes secrets.
- Tool calls: name and call id by default; definitions, arguments and results (gen_ai.tool.call.arguments, gen_ai.tool.call.result) by opt-in. Arguments are where a prompt injection or an over-broad permission becomes visible.
- Retrieval and hand-offs: the document or data source id (gen_ai.data_source.id); nested invoke_agent spans per sub-agent; invoke_workflow, plan and memory operations such as search_memory.
- Approvals: request id, the tool and arguments shown to the approver, the approver identity, the decision and the time, copied onto the trace from wherever the platform logs it.
- User feedback and evaluation scores linked to the trace id, so the online evaluation loop can find the run.
The spec defines the content switch OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT with the values NO_CONTENT (the default), SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT. Frameworks use their own names: Agent Framework has ENABLE_SENSITIVE_DATA in Python and EnableSensitiveData in .NET, both off by default, and Google Agent Runtime uses OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. Name the SDK in the runbook.
How do traces and spans work for an agent?
A trace is one run laid out as a tree of timed steps called spans. The OpenTelemetry GenAI conventions name three span types: invoke_agent for the agent, chat for each model call and execute_tool for each tool call, with newer additions for workflows, planning, memory and MCP.
The names carry the target, so a span reads invoke_agent order-status-agent or execute_tool get_order. A hand-off to another agent is a nested invoke_agent span under the caller, which keeps a multi-agent run in one trace. Tool servers continue it: for MCP (the Model Context Protocol, the standard agents use to call tools) the client puts the trace headers inside params._meta and the server opens a child span such as tools/call get-weather. Hosted or provider-managed tools, where the provider makes the call, do not receive the context and show as a gap. Metrics sit beside the traces, and two of them, gen_ai.invoke_agent.inference_calls and tool_calls, are the ones to alert on, because a loop shows up there within minutes and on the bill a day later. The sub-page on tracing AI agents covers the detail.
Where do the OpenTelemetry GenAI conventions stand?
They are the shared language and they are not stable. On 2 Nov 2026 every GenAI span, metric, event and attribute carried Development status, which means names and shapes can still change, and the conventions had just moved house.
With core semantic-conventions v1.42.0 (12 Jun 2026) the OpenTelemetry project deprecated every gen_ai attribute, metric, event and span in the core repository and moved them to a dedicated repository, open-telemetry/semantic-conventions-genai. The old pages on opentelemetry.io are stubs marked Moved, and on 2 Nov 2026 the new repository had no releases and an empty changelog. The practical effect is a version switch: many SDKs read OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental to move from the older attribute names (gen_ai.system) to the current ones (gen_ai.provider.name), and a dashboard keyed on the old name goes blank without an error. Pin the setting and test dashboards after each SDK release. All three clouds and the main independent tools emit or accept the conventions; OpenInference (used by Arize) and OpenLLMetry remain in use as alternative schemas.
Platform observability vs independent tools: which do you need?
Start with the platform you build on, because its tracing is built in and tied to its identity, evaluation and cost views. Add an independent tool when agents span clouds or frameworks, when you want annotation queues next to the traces, or when regulated data must stay self-hosted.
Observability products for agents, as the vendor pages showed on 2 Nov 2026.
| Product | What it traces and shows | Standards and integrations | Status seen 2 Nov 2026 |
|---|---|---|---|
| Microsoft Foundry tracing and Agent Monitoring Dashboard | Server-side traces once a project is connected to Application Insights, 90 days in the portal; token usage, latency, run success rate, evaluation scores, red-team results; estimated cost per agent in Foundry Control Plane | OpenTelemetry GenAI; LangChain, LangGraph, OpenAI Agents SDK, Agent Framework; external agents through the AI Gateway | Tracing GA for prompt and hosted agents; workflow and external agents preview; the built-in dashboard, recurring evaluations, red-team scans and alerts marked preview |
| Azure Monitor Application Insights, Agents view | One view for Foundry, Copilot Studio and third-party agents; sort by most tokens used; Grafana dashboards, including coding agents (GitHub Copilot, Claude Code, Codex, Gemini CLI) | OpenTelemetry GenAI over OTLP | Menu label still Agents (Preview), page dated 16 Oct 2026 |
| Amazon Bedrock AgentCore Observability and CloudWatch generative AI observability | Session count, latency, duration, token usage, error rates; spans and logs in CloudWatch; CloudTrail for Gateway and Agent Registry | OpenTelemetry; Strands Agents, LangChain, LangGraph; Runtime, Built-in Tools, Gateways, Memory, Identity | Both GA 13 Oct 2025; no charge beyond standard CloudWatch pricing (AWS) |
| Amazon Bedrock model invocation logging | Full request, response and metadata per model call to CloudWatch Logs or S3, with identity.arn and token counts | bedrock-runtime endpoint only, not bedrock-mantle | Disabled by default |
| Google Agent Runtime tracing and Agent Observability | Traces, logs and metrics in Cloud Trace with session and span views; content capture is a separate switch | OpenTelemetry GenAI; Telemetry API | Span format described as experimental; Agent Observability preview at the platform launch (Apr 2026) |
| Datadog LLM Observability (Agent Observability) | LLM calls, workflows and agents; tool use and hand-offs; cost; scans and redacts sensitive data | OpenTelemetry GenAI 1.37+, OpenInference, OpenLLMetry, Langfuse; OpenAI Agents SDK, LangGraph, CrewAI | No GA or preview label; bills on ingested LLM spans |
| Langfuse | Open-source, self-hostable tracing, prompt management, evaluation, annotation queues | OpenTelemetry | Joined ClickHouse 16 Jan 2026; stays open source and self-hostable |
| Arize Phoenix and Arize AX | Open-source tracing and evaluation (Phoenix); managed platform (AX) | Built on OpenTelemetry, instrumented with OpenInference | Live |
| LangSmith | Tracing, dashboards, alerts, online evaluators, annotation queues; an engine that detects recurring issues; an LLM gateway that auto-traces | Cloud, hybrid or self-hosted | Live |
| W&B Weave | Sessions, turns, steps, tools and sub-agents as first-class objects; evaluations, scorers, guardrails | Python and TypeScript SDKs | Live; Weights & Biases is part of CoreWeave |
| MLflow Tracing, Dynatrace, New Relic | MLflow: OpenTelemetry-compatible tracing with one-line auto-tracing. Dynatrace: agent paths, tool calls, agent-to-agent calls, online judge evaluations. New Relic: LLM call tracing, token usage, feedback | OpenTelemetry GenAI | Live; agent coverage and GA labels not stated on the New Relic page |
How do you keep sensitive data out of traces, or locked down when it has to be there?
Three controls, in this order: capture content only where a named reason requires it, redact before the span leaves the process, and put the content that remains behind a separate role with its own retention.
Decide per environment and redact at the source. Agent Framework’s guidance is to enable sensitive data only in development or testing environments; in production, turn it on for a named agent, for a named period, when an investigation or a compliance screen needs it, and Google’s Agent Runtime page says to have end-user consents, notices and data handling policies in place first. Datadog scans and redacts sensitive data in LLM spans, and Uber’s internal GenAI gateway redacts personal data before any third-party model call while keeping audit logs (Uber Engineering, 11 Jul 2024). Any gateway can strip identifiers, card numbers and credentials before a span is exported.
Separate the store. Foundry and Application Insights route the seven sensitive GenAI attributes to a dedicated AppGenAIContent table that you can mark Protected, readable only by the Privileged Monitoring Data Reader role. From 30 Sep 2026 Application Insights stops copying GenAI content into the AppDependencies, AppTraces and AppEvents tables for new data, and a temporary opt-out ends 30 Sep 2027. On AWS, invocation logs land in your own log groups and buckets, so their policies, keys and retention are the control; on Google, content capture is a separate variable from telemetry, so traces can run with content off.
How do you alert on an agent, and what do you do when one misbehaves?
Alert on the signals a model cannot fake: tool calls, model calls and tokens per run, error rate, latency, online evaluation scores and spend per use case. Then rehearse the containment actions, because each platform’s kill switch leaves something running.
Set rate-of-change alerts on gen_ai.invoke_agent.tool_calls and inference_calls first, because a runaway loop is visible there within minutes while cloud cost tools report hours later: Azure Cost Management budgets are evaluated every 24 hours and never stop consumption, AWS Budgets can apply actions but updates up to three times a day, and AWS Cost Anomaly Detection can lag 24 hours. For the incident itself, two published frameworks apply: the OWASP GenAI Incident Response Guide 1.0 (28 Jul 2025) and the CoSAI AI Incident Response Framework V1.0 (27 Oct 2025), which builds on NIST SP 800-61r3, lists agentic containment steps such as disabling tools and tightening authorisation, and for memory, resetting it and isolating sessions, and recommends logging prompt and response pairs, tool traces, system prompts and model versions so an incident can be reconstructed.
Containment is platform-specific and partial. In Microsoft 365, blocking a Foundry or SharePoint agent only affects its availability in Copilot Chat; a Foundry Control Plane stop does not terminate existing runs; the Copilot Studio quarantine API leaves the agent usable in the maker’s test pane; disabling an Entra agent identity stops token issuance and is reversible. On AWS, repoint the AgentCore Runtime named endpoint to a previous version and deny the tool in AgentCore Policy. On Google, block the agent at Agent Gateway and remove its IAM bindings. Test each switch on a throwaway agent and write down what it leaves running.
How do Microsoft, AWS and Google implement it?
The same capability, named the way each cloud names it, with statuses as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Agent tracing | Foundry tracing to Application Insights; GA for prompt and hosted agents, workflow and external agents preview; 90 days in the portal | AgentCore Observability with CloudWatch generative AI observability, GA 13 Oct 2025 | Agent Runtime tracing to Cloud Trace with the Telemetry API; span format marked experimental |
| Content capture switch | ENABLE_SENSITIVE_DATA (Python) or EnableSensitiveData (.NET) in Agent Framework, default off | Model invocation logging, disabled by default; full bodies to CloudWatch Logs or S3 | OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, separate from the telemetry switch |
| Sensitive content access | AppGenAIContent table, Protected, Privileged Monitoring Data Reader role; old tables stop receiving content 30 Sep 2026 | Your own CloudWatch log group and S3 bucket policies, keys and retention | Cloud Trace and Cloud Logging IAM; Google asks for consents and notices before content capture |
| Dashboards and alerts | Agent Monitoring Dashboard (marked preview on the Foundry overview page); Application Insights Agents view (preview); Alerts (preview) | CloudWatch dashboards and alarms on session count, latency, token usage, error rate | Agent Observability (preview at platform launch, Apr 2026); Cloud Monitoring |
| Cost per agent | Estimated cost per agent in Foundry Control Plane; API Management token metrics and token limits per key | Application inference profiles with cost allocation tags; per-request metadata in invocation logs; AgentCore consumption billing | Context caching discounts; Provisioned Throughput; attribution not covered on the pages we read |
| Other frameworks and clouds | Register external agents through the Foundry Control Plane AI Gateway and send OpenTelemetry telemetry to the same Application Insights resource | CloudWatch accepts OpenTelemetry from Strands, LangChain, LangGraph; cross-account monitoring | OpenTelemetry export from Agent Development Kit and Agent Runtime |
Microsoft’s blog dated 16 Mar 2026 announced general availability of evaluations, monitoring and tracing in Foundry; the Learn pages on 2 Nov 2026 still carried preview labels on the dashboard features above, so this page follows the docs.
When is agent observability not worth it, and what goes wrong?
Do not build a custom tracing stack for one agent on one platform: switch on the platform’s tracing and spend the effort on the evaluation set. Do not capture full content in production because it looks useful; capture it where a named process needs it. Once tracing runs, these are the ways it fails.
Personal data and secrets in spans
Prompts, tool arguments and tool results carry customer data, health data and credentials. Every major source makes content capture opt-in. Redact before export and keep content off unless a process requires it.
Everyone can read the prompts
Before the AppGenAIContent split, anyone with Log Analytics Reader could read prompt content in Application Insights. Check who holds reader roles on your trace store, on every cloud.
Gaps in the trace
Hosted or provider-managed MCP tools do not receive your trace context, and Bedrock invocation logging misses the bedrock-mantle endpoint. A run that crosses a gap looks shorter and cheaper than it was.
Convention churn, dropped metrics, duplicate spans
A dashboard keyed on gen_ai.system breaks when the SDK moves to gen_ai.provider.name; Azure API Management custom metrics silently drop data past 100 values per dimension or 1,000 time series per namespace; instrumenting both the chat client and the agent records content twice. Pin the version and bound the dimensions, and instrument at one layer.
Mistaking a trace for a verdict
A complete trace shows what happened, not whether it was right. Quality needs evaluators with criteria.
What should you set up?
The controls that turn tracing into observability, in the order we install them.
- Platform tracing on for every agent, plus an independent tool where agents cross clouds.
- One convention version pinned per SDK and the content-capture variable named per SDK, both written in the runbook.
- Content capture off in production by default, with an exception process that names the agent, the reason and the end date, and redaction at the gateway or exporter.
- A separate role for trace content (Privileged Monitoring Data Reader on Azure; dedicated log groups and buckets on AWS; separate IAM on Google) and a retention period per table or bucket.
- Alerts on tool calls per run, model calls per run, tokens per run, error rate, latency and online evaluation scores, with thresholds from the first month of traces.
- Cost tags at the gateway (key to team, application and cost centre) and a hard token quota per key, since budgets report hours or a day late.
- Every registry record linked to its dashboard, alerts and cost view, and a rehearsed containment playbook per platform.
Enterprise example · Worked example · illustrative
A 4,000-person insurer finds a claims agent that calls the same tool 40 times
Illustrative scenario. A property and casualty insurer with 4,000 staff, a claims-triage agent on Microsoft Foundry that reads claim documents and drafts a coverage recommendation for an adjuster, and a broker-query agent on Amazon Bedrock AgentCore.
The claims agent had run for three months with Foundry tracing on and content capture off, and the adjusters were happy. Then the monthly spend doubled with no change in claim volume. The Application Insights Agents view, sorted by most tokens used, showed the pattern in ten minutes: in about 3% of runs the agent called the document-search tool 30 to 40 times for one claim, each call resending the growing context, before it answered from the first result. Every one of those runs had returned a clean answer and a 200 status, and no alert existed for tool calls per run.
The cause was a retrieval change: a reindexed SharePoint library now returned near-duplicate chunks, so the agent kept searching for the clause it had already found. The fix went in three places: a cap of eight tool calls per run in the agent loop, an alert on gen_ai.invoke_agent.tool_calls above twelve, and a deduplication step in the search tool. To confirm the duplicates the team switched on content capture for one week, for this agent only, under an approved exception, behind the privileged reader role, and deleted it afterwards.
The broker agent on AgentCore got a loop cap in its code, a CloudWatch alarm on tool calls per run and the same deduplication step in its search tool, and both agents were tagged to their cost centres. Two months later the next incident, a model upgrade that changed tool selection, was caught by the online evaluation sample on day one.
- Owner
- Claims operations lead for the triage agent; distribution lead for the broker agent; both named in the registry record
- Identity
- Entra agent identity for the Foundry agent; AgentCore Identity workload identity for the broker agent; traces carry the agent id
- Tracing
- Foundry tracing with content off; AgentCore Observability to CloudWatch; one week of content capture by exception
- Risk tier
- Tier 2 for both (read shared data, recommend to a person); tier 3 if either agent wrote to the claims system
- Alerts
- Tool calls per run above twelve; model calls and tokens per run; error rate; online evaluation score below threshold
- Cost control
- Eight-tool-call cap in the loop; token quota per key; cost tags to two cost centres
- Approval
- Content capture needs the data protection officer’s approval with an end date; alert threshold changes need the owner’s approval
- Retirement
- Content purged after seven days; traces kept 90 days; retirement revokes the identity and archives the dashboards
How do you start?
- Switch on platform tracing for every agent, with content capture off, and confirm one trace per run.
- Pin the convention version in each SDK and write the content-capture variable names into the runbook.
- Decide the content policy: where content may be captured, who approves it, who can read it and when it is deleted.
- Set the first five alerts from the first month of traces: tool calls per run, model calls per run, tokens per run, error rate, latency.
- Tag cost at the gateway to team and use case, and add a hard token quota per key.
- Link each registry record to its dashboard, alerts and cost view, and rehearse the kill switch per platform.
Frequently asked questions
What is the difference between AI agent observability and LLM observability?
LLM observability records single model calls: prompt, completion, tokens, latency. Agent observability records the whole run as a trace: every model call, every tool call with arguments and results, retrieval, hand-offs and approvals, nested under one root span. The second includes the first.
Should we log prompts and responses in production?
Not by default. The OpenTelemetry conventions, Microsoft, AWS and Google all make content capture opt-in. Capture content for a named agent and reason, redact it at export, put it behind a privileged role and give it a retention date. Where a compliance screen requires full content, treat the trace store as a system of record.
Is the OpenTelemetry GenAI standard stable enough to build on?
Build on it, and expect change. Every GenAI span, metric and attribute was Development status on 2 Nov 2026, and the conventions moved to their own repository in June 2026. All three clouds and the main independent tools emit or accept them, so the alternative is a proprietary schema with less support.
How do we monitor agents built on Copilot Studio?
Copilot Studio agents appear in the Application Insights Agents view (preview on 2 Nov 2026) alongside Foundry and third-party agents, and the Microsoft 365 admin center registry shows usage and risk signals. For cost, the Copilot Credits meter is the unit to watch.
How quickly can we catch a runaway agent?
Within minutes if you alert on tool calls and model calls per run from the traces, and in hours or a day if you rely on cloud cost tools, since Azure budgets are evaluated every 24 hours and Cost Anomaly Detection can lag 24 hours. Put a step cap in the agent loop and a token quota at the gateway as well.
Do we need Datadog or Langfuse if we use Foundry or AgentCore?
Not for one platform. Add an independent tool when agents span clouds and frameworks, when you want annotation queues beside the traces, or when a self-hosted instance is a data residency requirement. Langfuse is open source and self-hostable; LangSmith offers cloud, hybrid and self-hosted deployment.
In this topic
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Semantic conventions for GenAI agent spans
- Semantic conventions for GenAI client spans and content capture
- Semantic conventions for GenAI metrics
- Semantic conventions for Model Context Protocol
- Semantic conventions releases (v1.42.0 moved GenAI to its own repository)
- Tracing agents: concepts
- Set up tracing for agents in Microsoft Foundry
- Restrict access to sensitive GenAI content in traces
- Agent Monitoring Dashboard in Microsoft Foundry
- Agents view in Application Insights
- Observability in Microsoft Agent Framework
- Generally available: evaluations, monitoring and tracing in Microsoft Foundry
- Emit token metrics for LLM APIs (llm-emit-token-metric policy)
- Create and manage budgets in Cost Management
- Amazon Bedrock AgentCore is now generally available
- Amazon CloudWatch generative AI observability is now generally available
- Observability for Amazon Bedrock AgentCore
- Amazon Bedrock model invocation logging
- Managing your costs with AWS Budgets
- AWS Cost Anomaly Detection
- Tracing for Agent Runtime
- Gemini Enterprise Agent Platform release notes
- LLM Observability
- OpenTelemetry instrumentation for LLM Observability
- Langfuse joins ClickHouse
- How DKB runs its AI banking assistant on Langfuse
- How Merck builds its central GenAI platform with Langfuse
- Arize Phoenix documentation
- LangSmith observability
- W&B Weave
- MLflow Tracing for GenAI
- Dynatrace for AI observability
- Introduction to AI monitoring
- Scaling AI at Uber with the GenAI Gateway
- AI Incident Response Framework V1.0
- GenAI Incident Response Guide 1.0
- Manage agents in Foundry Control Plane
- Agent actions in the Microsoft 365 admin center
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner