- Home
- AI agents
- Observability and cost
- What to log
Operate · Observability and cost
What should you log for an AI agent?
Logging an AI agent means deciding, event by event, what to record, in how much detail and for how long: runs, model calls, tool calls, retrieval, memory writes, hand-offs, approvals, guardrail decisions, and identity and configuration changes. Metadata is logged for every run; prompts, outputs and tool arguments only where a named purpose needs them.
In this guide: 83 pages
Key takeaways
- Log three streams with different readers: telemetry for operators, an audit stream for auditors and a security stream for the security operations centre. One event often feeds two of them, with different fields and retention.
- Never sample the events that prove control: approvals, guardrail blocks, identity changes, configuration changes and stops. Sample model-call content and full traces where volume or cost demands it.
- Set detail by risk tier. A low-risk internal assistant needs metadata and tool names; an agent that changes records about people or money needs tool arguments and, for its decision runs, content behind a privileged role.
- Logs leak too. AWS says Bedrock Guardrails PII masking does not cover model invocation logs or tool-call arguments, and in July 2026 an AgentCore SDK release wrote raw prompts and responses into CloudWatch spans.
- Volume costs money and breaks queries: Datadog bills on ingested LLM spans, API Management custom metrics drop data beyond 100 values per dimension, and instrumenting both the chat client and the agent records content twice.
Which events should an agent log?
Log every event that changes what the agent knows, what it can do or what it did, plus the cost and timing of each step. The table separates what to record on every run from what to record only with a reason, and shows where each event usually lands.
Agent log events, what to record by default and with a reason, and their usual home. Span names follow the main branch of the OpenTelemetry GenAI conventions on 2 Nov 2026; the conventions have Development status, so pin the SDK and schema version and retest after upgrades.
| Event | Record on every run | Record with a reason | Usual home |
|---|---|---|---|
| Run start and end | Run and conversation ids, agent name and version, caller, duration, outcome | The full user request | Root span of the trace; audit record |
| Model call | Provider, model and version, input, output, cached and reasoning tokens, latency, error.type | Prompt, system instructions and output text | chat span |
| Tool call | Tool name, call id, target system, status, latency | Arguments and results; always for tools that write or send | execute_tool span; gateway log |
| Retrieval | Data source id, document ids, sensitivity labels | The retrieved passages | Trace; Purview AccessedResources |
| Memory write or read | Memory store, operation, record id | The remembered content | create_memory and search_memory spans |
| Hand-off to another agent | Called agent, task id, task state | The message passed | Nested invoke_agent or A2A spans |
| Approval | Request id, approver, decision, time, what was shown | Nothing extra: approvals are kept in full | Approval service; audit stream |
| Guardrail or security decision | Rule, action taken (block, redact, allow), score | The offending input, behind a privileged role | Security stream, such as the Defender BehaviorInfo table |
| Identity and permission change | Agent identity created, disabled, granted or revoked | Nothing extra: kept in full | Identity provider audit log |
| Configuration change | New agent version, prompt, model, tool or knowledge source | The diff | Registry and deployment pipeline |
| Feedback and evaluation score | Score, evaluator, trace id | Free-text feedback | Evaluation store, linked to the trace |
Treat cost as a derived field. Store the token counts and billing dimensions with the run, compute the estimate from the rates in force when the run happened, and keep the price list version next to it, because list prices change and a figure without its price version cannot be checked later.
How much detail does each risk tier need?
Detail should rise with the harm an agent can do. Three tiers are enough for most fleets, and they let the platform set the content switches automatically from the registry record.
- Low tier, such as an internal question-and-answer assistant over published policies: metadata on every run, tool names without arguments, and content capture off in production, switched on for a named investigation with an end date.
- Medium tier, such as an agent that drafts while a person sends: everything above, plus arguments for every tool that writes, sends or deletes, and the approval record in full.
- High tier, such as an agent that changes customer records, payments or decisions about people: everything above, plus content for the decision runs behind a privileged role, the documents retrieved, and retention matched to the longest legal duty.
Name the SDK when you write the content switch into the runbook, because the names differ: OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT in the OpenTelemetry spec, ENABLE_SENSITIVE_DATA in Agent Framework for Python and OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT on Google Agent Runtime, where Google asks for end-user consents and notices before you turn it on.
What should never reach agent logs?
Secrets, and any personal or regulated data that no named purpose needs. Agents move both through tool arguments and tool results, which is where most masking stops.
Credentials come first. Tool arguments can carry API keys, OAuth tokens and connection strings, and a log that keeps them hands an attacker the agent’s access. Scrub known secret patterns at the gateway before export, and run secret scanning over log stores the way you do over code repositories.
Masking has gaps. AWS documents that Bedrock Guardrails PII masking does not apply to model invocation logs, which keep the original input, or to tool-call arguments, tool results and tool definitions. Redact in your own pipeline before the span or log line leaves the process, and test the redaction with planted test data.
Logging code has bugs like any other code. CVE-2026-15737 (July 2026) covered AgentCore Python SDK releases that wrote raw prompts and full agent responses into OpenTelemetry spans in CloudWatch, readable by anyone with log access, and the fix needed an upgrade plus a purge of the affected logs. Keep a purge procedure for every log store before you need one.
How do you keep logging affordable and queryable?
Sample the high-volume telemetry, never the control events, and keep duplication and cardinality under watch. Ingestion and storage are billed by volume, and some metric pipelines drop data without an error when they hit their limits.
- Sample traces by rule: keep every failed, flagged, escalated, approved or unusually expensive run, and a share of the routine ones.
- Give each stream its own sampling and retention policy, and test that the policy works: send a test approval and a test guardrail block through the pipeline and confirm both arrive in full.
- Count at both ends. Alert when the number of events exported from the agent differs from the number stored in the backend, because a dropped export looks the same as a quiet day. The tracing sub-page covers duplicate instrumentation and large payloads.
- Watch cardinality. API Management custom metrics drop data beyond 100 values per dimension or 1,000 time series per namespace, so keep per-user and per-conversation ids in logs and out of metric dimensions.
- Price the backend before you switch content on. Datadog bills its LLM Observability product on ingested LLM spans, so capture and sampling decisions show up on the invoice.
How do Microsoft, AWS and Google implement it?
Where agent log events land by default on each cloud, and the known gaps, as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Traces and metrics | Application Insights through Foundry tracing; Agent Framework emits OpenTelemetry GenAI spans | CloudWatch through AgentCore Observability | Cloud Trace through Agent Runtime telemetry |
| Model call content | AppGenAIContent table with protected access in preview: Privileged Monitoring Data Reader or a matching custom role | Model invocation logging to CloudWatch Logs or S3, disabled by default; bodies over 100 KB go to S3 | A separate content switch; Google asks for consents and notices first |
| Interaction and API audit | Purview Audit for Copilot, Copilot Studio and Foundry, with XPIADetected and JailbreakDetected flags | CloudTrail for AgentCore Gateway and Agent Registry calls | Cloud Audit Logs |
| Known gaps to document | Hosted or provider-managed MCP tools do not receive your trace context | Invocation logging covers the bedrock-runtime endpoint and misses bedrock-mantle | The span format is described as experimental and subject to change |
Frequently asked questions
Should we log the model’s reasoning?
Log what the provider returns. OpenAI keeps reasoning tokens hidden and bills them as output, and Google returns thought summaries, so the reasoning itself is often unavailable. Keep the summaries where you get them, plus every tool call, since the calls are the actions. MAS asks that monitoring of agentic AI extend to reasoning processes where relevant.
Do we need to log every agent run?
Log metadata for every run, because counts, costs and error rates depend on it, and keep approvals, guardrail blocks and identity changes in full. Content and full traces can be sampled, as long as every failed, flagged or escalated run is kept.
Who should be able to read agent logs?
Metadata can sit with the usual operations roles. Content belongs behind a separate, privileged role with its own access review, the way Application Insights can protect the AppGenAIContent table (in preview), because prompts and tool results often hold personal data, customer content and sometimes secrets.
Can agent logs go into our existing SIEM?
Yes, and the security stream should. Send guardrail blocks, injection flags, identity changes and unusual tool activity to the SIEM with the trace id, so an analyst can open the full trace from the alert. Keep bulky content out of the SIEM and link to it instead.
More in Observability and cost
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Semantic conventions for GenAI client spans and content capture
- Semantic conventions for GenAI agent spans
- Observability in Microsoft Agent Framework
- Restrict access to sensitive content in traces
- Audit logs for Copilot and AI applications
- Real-time agent protection during runtime
- API Management policy reference: llm-emit-token-metric
- Amazon Bedrock model invocation logging
- Remove PII from conversations by using sensitive information filters
- CVE-2026-15737
- Tracing for Agent Runtime
- LLM Observability
- Reasoning models
- Gemini thinking
- Guidelines on Artificial Intelligence Risk Management, response to feedback
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner