- Home
- AI agents
- Observability and cost
- Agent cost monitoring
Operate · Observability and cost
How much does an AI agent cost to run, and how do you monitor it?
What an AI agent costs to run is the sum of its model tokens (input, output, cached and reasoning), its tool and gateway calls and its runtime seconds, multiplied by how many steps each request takes. Agent cost monitoring attributes that spend to a team and a use case and caps it before a budget alert arrives.
In this guide: 83 pages
Key takeaways
- Agents cost more than chatbots because one request becomes many model calls, each resending a growing context. The FinOps Foundation calls context window creep the biggest hidden cost in production AI.
- Every platform bills on consumption: tokens, per-second runtime, per-call gateways, per-evaluation judges or Copilot Credits. Learn the meter for each agent before the first invoice.
- Cloud budgets report hours or a day late, and Azure budgets never stop spend. Real-time control is a token quota per key at the gateway, a step cap in the agent loop and an alert on tool calls per run.
- The levers are caching, routing, effort settings, batch tiers and committed capacity. Each has a failure case, and the test is cost per successful outcome rather than cost per token.
Where does the cost of an AI agent come from?
The cost comes from seven meters, and the agent loop multiplies all of them: every turn resends the conversation, most tool calls add a model call to read the result, and every sub-agent repeats the pattern. The table names each meter and how the vendors billed it on 2 Nov 2026.
Cost drivers for an AI agent and the vendor pricing models behind them, as published on 2 Nov 2026. No unit prices are quoted here; re-read the pricing page before quoting one.
| Meter | Why agents multiply it | How vendors bill it |
|---|---|---|
| Input tokens | APIs are stateless, so each turn resends the history and the retrieved context; the FinOps Foundation notes that session cost grows faster than the number of turns | Per million input tokens on every model API; Copilot Studio meters tokens as Copilot Credits for reasoning models |
| Output and reasoning tokens | Reasoning models think in hidden tokens before each action; OpenAI bills them as output tokens and Google bills for the full thoughts even when the API returns only a summary | Per million output tokens; effort or thinking settings change the volume (OpenAI reasoning.effort, Anthropic effort, Google thinking_level) |
| Cache reads and writes | Agents resend the same instructions and tool definitions on every call, so caching pays when the prefix is stable and read often | Discounted cache reads (Anthropic 0.1 times base input on most models and lower on its newest; OpenAI up to 95% off; Google implicit caching 90% off); cache writes at a premium on newer families (Anthropic 1.25 times for a 5-minute cache, 2 times for an hour; OpenAI 1.25 times on newer models) |
| Tool calls and gateways | Each tool call is a round trip plus a model call to read the result; a loop can call one tool dozens of times | AgentCore Gateway per invocation and AgentCore Policy per authorisation request; Copilot Studio agent action 5 credits, tenant graph grounding 10, agent flow actions 13 per 100 |
| Runtime seconds and memory | Long-running agents hold compute while they wait on tools and models | AgentCore Runtime per second of CPU and memory, with CPU scaling to zero during I/O wait; no upfront commitment or minimum fee |
| Evaluations and judges | Online evaluation runs a judge model on a sample of every run; red-team scans run thousands of prompts | AgentCore built-in evaluators on tokens, custom evaluators per 1,000 evaluations; Foundry playground evaluations enabled by default and billed on consumption |
| Computer use | Each turn sends a screenshot as image input and gets one or more actions back, so a 40-step task can mean dozens of model calls with images unless the model batches actions | Anthropic puts a screenshot at roughly 1,000 to 1,800 input tokens; Copilot Studio charges 5 Copilot Credits per computer-use step, 15 for premium models |
Microsoft’s own example shows the multiplier: in Copilot Studio one follow-up message makes two model calls, billed at 14,100 tokens or 141 Copilot Credits with a warm cache, and 461 credits if the same question starts a new conversation (Microsoft Learn, 30 Oct 2026).
How do you attribute agent spend to a team and a use case?
Attribute at the gateway, with one key per team, application and cost centre, and tag production separately from experimentation. The cloud tools roll the tags up with a lag; the traces give the per-run detail the invoice never will.
On Azure, the API Management policy llm-emit-token-metric sends token metrics to Application Insights with dimensions such as subscription id, user id and API id, plus up to five custom ones; custom metrics silently drop data past 100 values per dimension or 1,000 time series per namespace. Foundry Control Plane shows estimated cost per agent once Application Insights is connected.
On AWS, application inference profiles carry cost allocation tags into Cost Explorer and the cost and usage report at daily grain; tags must be activated in Billing, are not retroactive, can take up to 24 hours to appear, and do not work with the Responses and Chat Completions APIs, which use Projects and Workspaces instead. Per-request requestMetadata (up to 16 entries) lands in invocation logs for per-prompt attribution but is not enforced by Bedrock, and IAM principal attribution identifies the caller.
The FinOps Foundation’s guidance (FinOps for AI Overview, updated 28 Oct 2026) is to measure the use case, not the token: map each API key to one team, application and cost centre, and report cost per inference, per user and per successful outcome as distributions, with showback before chargeback. It also admits there is no accepted allocation method for multi-agent workloads, so decide in advance whether a sub-agent’s spend belongs to the caller or to its own owner.
Uber’s GenAI Gateway is the publicly reported pattern (Uber Engineering, 11 Jul 2024): one gateway in front of every model provider that manages usage and budgeting across teams and produces audit logs for cost attribution. Merck KGaA reports the same discipline on a central self-hosted platform (Langfuse case study, 2 Sep 2025).
How do you cap spend before the budget alert arrives?
Cap it in three layers that act at different speeds: a hard token quota per key at the gateway, a step and loop cap inside the agent, and an alert on tool calls per run from the traces. Cloud budgets sit behind all three because they report hours or a day late.
- Gateway quotas. The API Management policy llm-token-limit sets tokens-per-minute and a token-quota per hour, day, week, month or year, keyed by subscription, user or any counter-key, and returns 429 or 403 when exceeded. It works with the OpenAI chat and responses APIs, the Anthropic Messages API on v2 tiers, and the Gemini Enterprise Agent Platform API (the page still uses the old name, Vertex AI API). It counts per gateway, so limits are not aggregated across regions, and concurrent requests can temporarily exceed the limit, so treat it as a ceiling rather than an exact meter.
- Loop caps. Put a maximum number of steps and tool calls per run in the agent code or the orchestrator, and a maximum conversation length. Copilot Studio’s own advice is to start a new conversation for a new topic rather than resume a long one and to keep instructions concise.
- Platform limits. Copilot Studio blocks agents at 125% of prepaid capacity unless pay-as-you-go is attached, including in a conversation already in progress. Decide before launch whether a block or an overage is the right failure for each agent.
- Trace alerts on gen_ai.invoke_agent.tool_calls, inference_calls and tokens per run, which fire within minutes of a runaway loop.
- Cloud budgets. Azure Cost Management budgets alert on actual and forecast spend but, in Microsoft’s words, resources are not affected and consumption is not stopped; data arrives in 8 to 24 hours and budgets are evaluated every 24 hours. AWS Budgets can apply actions such as IAM policies but its data updates up to three times a day. AWS Cost Anomaly Detection runs about three times a day with up to 24 hours of detection lag and covers Bedrock third-party models. Use them for the monthly picture, not the incident.
Which levers cut the cost without cutting quality?
Five levers, each with a condition: caching when the prefix is stable and read often; routing when evaluations show the cheaper model holds quality; lower effort for routine tasks; batch and flex tiers when nobody is waiting; committed capacity when volume is steady. Measure each against cost per successful outcome.
Caching. Put stable content first (instructions, tool definitions, reference documents), keep the context append-only and track the hit rate and the write volume. Bedrock prompt caching went generally available on 7 Apr 2025 with checkpoint minimums from 512 to 4,096 tokens by model and a common 5-minute lifetime; Azure OpenAI caching is on by default once 1,024 identical prefix tokens are present. The FinOps warning: if cached content is not read back enough times before it expires, the write premium costs more than caching saves. Semantic caching at the gateway (API Management llm-semantic-cache-lookup and store) answers repeated questions without a model call.
Routing. Foundry model router (version 2025-11-18; GA date not found) is a trained model that sends each prompt to an underlying model: Balanced picks the cheapest within about 1% to 2% of the best quality, Cost allows 5% to 6%, Quality ignores cost. Its effective context window is that of the smallest model in the pool, and session affinity for cache reuse was in preview. Amazon Bedrock Intelligent Prompt Routing (GA 22 Apr 2025) routes between two models of one family on predicted response quality with a fallback model, is optimised for English prompts, and AWS says it can reduce costs by up to 30 percent. Run your evaluation set through the router before switching; the Evaluation and testing topic explains how.
Effort and model choice. Reasoning dials are now explicit: OpenAI reasoning.effort, Anthropic effort, Google thinking_level. Anthropic says its maximum setting adds significant cost for relatively small quality gains, that lower effort means fewer and terser tool calls, and that levels are recalibrated between model versions, so rerun the effort sweep on your evaluations rather than carrying a setting over. OpenAI’s method is to prototype with the most capable model, set up evaluations, then swap in smaller models where they meet the accuracy target.
Tiers and commitments. Azure Batch offers a 24-hour target turnaround at 50% less than global standard; Bedrock batch inference is 50% below on-demand, the Flex tier is 50% below Standard and Priority is 75% above it. Provisioned throughput bills per unit per hour regardless of tokens, reservations discount it without guaranteeing capacity, and Google Provisioned Throughput is a fixed-cost, fixed-term subscription. Commit only for steady, latency-sensitive production traffic. No primary source we read reports audited cost savings from an enterprise agent program, so the figures above are vendor figures.
How do Microsoft, AWS and Google implement it?
The cost controls each cloud documents for agents, as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Attribution | API Management llm-emit-token-metric dimensions; estimated cost per agent in Foundry Control Plane; Copilot Credits pooled per tenant | Application inference profiles with cost allocation tags; per-request metadata; IAM principal attribution; Projects and Workspaces | Not covered on the Google pages we read; context caching and Provisioned Throughput are documented |
| Hard caps | API Management llm-token-limit per key (rate and quota, 429 or 403); Copilot Studio block at 125% of prepaid capacity | AWS Budgets actions (up to three updates a day); AgentCore Policy denies tool calls; step caps in Strands or LangGraph code | Not covered on the pages we read; step caps in Agent Development Kit code |
| Caching | Azure OpenAI prompt caching on by default from 1,024 prefix tokens; semantic caching in API Management | Bedrock prompt caching, GA 7 Apr 2025, implicit and explicit modes | Implicit caching on by default at a 90% discount; explicit caching with storage charges and a 60-minute default lifetime |
| Routing | Foundry model router with Balanced, Cost and Quality modes; model subsets governed by Azure Policy | Intelligent Prompt Routing, GA 22 Apr 2025, within one model family, English-optimised | Not documented as a managed router on the pages we read |
| Batch and committed capacity | Batch at 50% off global standard; provisioned throughput with reservations and spillover | Batch inference 50% off; Flex, Standard, Priority and Reserved tiers | Provisioned Throughput, fixed cost and fixed term |
| Agent platform billing | Copilot Studio in Copilot Credits (prepaid packs or pay-as-you-go); Foundry on model consumption | AgentCore consumption pricing: Runtime per second, Gateway per invocation, Evaluations per token or per 1,000, Policy per request | Agent Runtime on Gemini Enterprise Agent Platform; model tokens plus platform usage |
Unit prices on the AgentCore and Bedrock pricing pages were read through a summarising fetch and are deliberately not reproduced; quote a figure only after reading the page on the day.
Frequently asked questions
How much does an AI agent cost per month?
It depends on four numbers you can measure: requests per month, model calls per request, tokens per call and the share of calls that hit a cache. An agent that calls three tools and reasons at each step can cost many times a chatbot answering the same question. Run a week of traces and multiply out; our AI agent pricing guide covers vendor pricing models.
Why did our agent bill double with no change in traffic?
The usual causes are a loop (one tool called many times per run), a longer context (a new knowledge source or a prompt change that broke the cache), a model upgrade that reasons more, or a router that moved traffic to a dearer model. Sort traces by tokens per run and compare last month with this week.
Does prompt caching always save money?
No. Cache writes cost a premium on newer model families, so content that is cached but not read back enough times before it expires costs more than it saves. Caching pays when the prefix is stable, large and reused within the cache lifetime. Track the hit rate and the write volume.
Should we use a model router to cut cost?
Only after your evaluation set confirms the cheaper model holds quality on your tasks. Managed routers can cut cost, but the effective context window drops to the smallest model, cache hits fall when models alternate, and the vendor can update the pool. Pin the model subset and rerun evaluations after each change.
What does a computer-use agent cost compared with an API integration?
Far more per task, because each turn sends a screenshot as image input and returns one or more actions, so a task of 40 clicks can mean dozens of model calls with images unless the model batches actions. Anthropic puts one screenshot at roughly 1,000 to 1,800 input tokens and Copilot Studio charges per step. Reserve it for systems with no API.
More in Observability and cost
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- FinOps for AI Overview
- GenAI FinOps: How token pricing really works
- Copilot Studio billing: messages and capacity management
- Emit token metrics for LLM APIs (llm-emit-token-metric policy)
- Limit LLM API token usage (llm-token-limit policy)
- Enable semantic caching for LLM APIs in API Management
- Prompt caching in Azure OpenAI
- Model router for Microsoft Foundry
- Provisioned throughput for Foundry Models
- Batch deployments in Azure OpenAI
- Create and manage budgets in Cost Management
- Amazon Bedrock AgentCore pricing
- Amazon Bedrock pricing
- Amazon Bedrock prompt caching is now generally available
- Prompt caching for faster model inference
- Amazon Bedrock Intelligent Prompt Routing is now generally available
- Use application inference profiles to track costs
- Include metadata in requests for cost attribution
- Managing your costs with AWS Budgets
- AWS Cost Anomaly Detection
- Context caching overview
- Provisioned Throughput overview
- Prompt caching
- Effort
- Computer use tool
- Prompt caching
- Reasoning models guide
- A practical guide to building agents
- Computer use in Copilot Studio
- Scaling AI at Uber with the GenAI Gateway
- How Merck builds its central GenAI platform with Langfuse
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner