- Home
- AI agents
- What is an AI agent?
- Why AI agents fail in production
Plan · What is an AI agent?
Why do AI agents fail in production?
AI agents fail in production for reasons that rarely show up in a pilot: no written definition of done, wrong tool calls at scale, prompt injection through what they read, permissions nobody scoped, cost that grows with autonomy, and model changes nobody retested. The published figures are stark, and every cause has a known control.
In this guide: 83 pages
Key takeaways
- MIT NANDA’s preliminary 2025 study found 60% of organizations evaluated enterprise-grade AI tools, 20% reached a pilot and 5% reached production; Gartner, as reported by Reuters, expects more than 40% of agentic AI projects to be cancelled by the end of 2027.
- On benchmarks the top-scoring agent completed 30% of simulated office tasks on its own (TheAgentCompany, September 2025 version) and the top-scoring tool-using agents passed under half of tau-bench’s tasks. Your own evaluation set on your own data is the number that matters.
- The causes are specific and known: wrong tool or arguments, too many tools, injected instructions, shared credentials, rubber-stamped approvals, untested model changes, cost that scales with autonomy, silent behavioural failures, and no acceptance criteria.
- The controls are specific too: acceptance criteria written first, an evaluation set with expected tool trajectories, an identity per agent, policy at the tool boundary, traces of every run, versioning with rollback, and a registry record with an owner.
How many AI agent projects fail?
A large share, on every published measure, though the measures differ in what they count. The table gives the figures with their scope, because a survey of 153 leaders, an analyst forecast and a benchmark of simulated office tasks measure different things.
MIT NANDA’s report is the figure most people quote, and it needs its scope. It is a preliminary research note (July 2025) built on a review of more than 300 publicly disclosed initiatives, 52 interviews and a survey of 153 senior leaders, and it covers enterprise generative AI systems broadly, agents included. Its funnel is the useful part. Sixty percent of organizations evaluated enterprise-grade or custom tools, 20% reached a pilot and 5% reached production, and the authors attribute most failures to brittle workflows, a lack of contextual learning and misalignment with day-to-day operations. They also found external partnerships succeeding at about twice the rate of internal builds, which we quote with the obvious caveat that we are an external partner.
The benchmark numbers are the other half. They measure models, where the MIT study measured organizations, and they move every quarter, so treat the direction as the finding: agents do worse on open-ended, long and consequential tasks than headline benchmarks suggest, and they are inconsistent between runs of the same task. METR’s finding that at least 16% of successful long runs were illegitimate on review is the one to remember when a vendor shows a demo.
Published failure figures and what each one measures.
| Figure | What it measures | Publisher and date |
|---|---|---|
| 95% of organizations getting zero return; 60% evaluated enterprise-grade tools, 20% reached a pilot, 5% reached production | Enterprise-grade and custom AI systems, from a review of 300+ public initiatives, 52 interviews and 153 survey responses; preliminary findings | MIT NANDA, The GenAI Divide, July 2025 |
| More than 40% of agentic AI projects cancelled by the end of 2027; only about 130 of thousands of agentic AI vendors are real | An analyst forecast with no sample size given; cites escalating costs and unclear business value | Gartner, 25 Jun 2025, as reported by Reuters |
| The "most competitive agent" completed 30% of tasks autonomously | TheAgentCompany, a benchmark of tasks in a simulated software company | arXiv 2412.14161, version 3 of 10 Sep 2025 |
| Top-scoring function-calling agents succeed on under 50% of tasks; pass^8 under 25% in retail | tau-bench: tool use while talking to a simulated user in retail and airline domains | arXiv 2406.12045, 17 Jun 2024 |
| On tasks over eight hours, at least 16% of successful runs were illegitimate on review; the public frontier about 12 hours at 50% success and about 1.5 hours at 80% | Autonomous task completion by frontier models, February to March 2026 | METR Frontier Risk Report, 19 May 2026 |
| One in five organizations reported a breach due to shadow AI; 97% of those with AI breaches had no AI access controls | 600 organizations, breaches from March 2024 to February 2025, by Ponemon Institute for IBM | IBM Cost of a Data Breach Report, 30 Jul 2025 |
None of these figures is about your agent. They are the reason to insist on an evaluation set from your own cases before anyone quotes a success rate.
Why do agents fail once they leave the pilot?
Because production has what a pilot lacks: volume, adversarial input, real permissions, cost at scale and change over time. Nine causes recur in the failures the vendors document and in the ones we see, and each one maps to a control.
A pilot typically runs on friendly input, with the builder’s own permissions, on one model version. Production removes all three conditions at once.
Nine causes, the evidence for each, and the control that addresses it.
| Cause | What the evidence says | The control |
|---|---|---|
| No written definition of done | MIT’s funnel attributes most failures to brittle workflows and misalignment with daily operations; Copilot Studio’s own evaluation guidance starts with "define what success looks like" | Acceptance criteria agreed before the build, with a baseline and a pass bar |
| Wrong tool or wrong arguments | Anthropic: "the most common failures are wrong tool selection and incorrect parameters", worse with look-alike names and schemas that omit conventions | Fewer tools with clear descriptions and strict schemas; tool search above about 10 tools; trajectory evaluation |
| Too many tools in context | OpenAI: fewer than 20 functions at the start of a turn; Google: 10 to 20 active; Anthropic: accuracy degrades past 30 to 50; Microsoft 365 Copilot: quality may degrade past 10 | A tool budget per agent, deferred loading, and tool groups |
| Injected instructions in what the agent reads | OWASP ASI01, agent goal hijack; adaptive attacks bypassed 12 published defenses, with success above 90% for most of them (October 2025 paper) | Checks on every untrusted channel including tool results; deterministic policy on tool calls; approval on writes; egress limits |
| Shared or human credentials | IBM: 97% of organizations with AI breaches had no AI access controls; Microsoft: reusing user accounts for agents breaks Conditional Access and governance | An identity per agent, delegated or autonomous access decided per tool, no static secrets |
| Rubber-stamped approvals | OWASP ASI09, human-agent trust exploitation; Microsoft’s overreliance research: mitigations can backfire, and the mere presence of sources can increase trust | Approval by risk tier, the exact tool and arguments shown, safe reads auto-allowed, the edit rate monitored |
| Model change nobody retested | Foundry GA models retire on an 18-month lifecycle (12 months for Anthropic, DeepSeek, Fireworks and Mistral models) and Standard deployments auto-upgrade at retirement; Anthropic recalibrates effort levels between versions | Pinned model versions, a regression run of the evaluation set before any switch, rollback by named endpoint or alias |
| Cost that scales with autonomy | Anthropic: agents use about 4 times the tokens of a chat, multi-agent systems about 15 times; consumption billing follows the loops | A cost figure per run in the traces, rate and token limits at the gateway, a single agent before multi-agent |
| Silent behavioural failures | AWS added Failure Insights (preview, July 2026) for "silent behavioral failures that produce no error signal" | Online evaluation on sampled traffic; alerts on evaluation scores and not only on errors |
The guardrails, identity, security, observability and evaluation topics each take one of these controls in depth.
Which controls move an agent from pilot to production?
Seven, in the order we install them: acceptance criteria, an evaluation set with expected trajectories, an identity per agent, policy at the tool boundary, traces with a cost figure, versioning with rollback, and a registry record with an owner and a kill switch.
- Acceptance criteria, written before the build: the workflow, the inputs, what a correct result is, the baseline it replaces and the pass bar. QueryNow’s acceptance criteria generator page helps you draft one, every QueryNow build starts from a written definition of done, and the acceptance-criteria sub-page under Evaluation covers them in depth.
- An evaluation set from real cases, with the expected answer and the expected tool trajectory, run offline before release and online on sampled production traffic afterwards. Score the path as well as the answer: an agent can reach the right answer through a forbidden tool call. Run each case several times and track consistency, as tau-bench’s pass^k metric does.
- An identity per agent, with delegated or autonomous access decided per tool, short-lived scoped tokens and no shared secrets. All three clouds provide it: Microsoft Entra Agent ID, AgentCore Identity and Google Agent Identity.
- Policy at the tool boundary, outside the model: which tools, with which arguments, under which identity. AgentCore Policy (GA 3 Mar 2026) runs Cedar rules on every Gateway call with a log-only mode for testing; Foundry agent guardrails add tool-call and tool-response checks (preview); Google Agent Gateway is the enforcement point, with Model Armor inline.
- Traces of every model call and tool call, with token counts and latency by default and content capture off unless you have a reason and a retention rule. Derive a cost per run from the tokens and put it on the same dashboard as the evaluation scores.
- Versioning and rollback: immutable agent versions, a named production endpoint or alias that does not move on every change, pinned model versions, and a regression run before any model, prompt, tool or knowledge change reaches it.
- A registry record with an owner, a sponsor, a risk tier, an approval state and a review date, and a kill switch tested on a throwaway agent: block or quarantine, disable the identity, deny the tools.
This is also how we scope a build: one workflow and a written definition of done, with these controls as the delivery checklist. The how-we-engage page describes it.
What does a production-ready agent look like on paper?
Six documents, each with an owner. If any one of them is missing, the agent is still a pilot, however long it has been running and however many people use it.
- The acceptance criteria and the baseline, signed by the business owner.
- The evaluation report: the set, the pass rates per criterion, the trajectory scores, the date and the model version.
- The identity and access record: the agent identity, whether it acts for a user or as itself, every tool and the data it reaches, and the policy rules.
- The autonomy decision: the level, the AWS scope, the approval rules and the edit-rate threshold for the next review.
- The runbook: the kill switch per platform and what it leaves running, the rollback path, and the on-call owner.
- The registry entry: owner, sponsor, risk tier, approval state, review date and retirement plan.
Every QueryNow build starts from a written definition of done; these six documents are what it contains.
How do Microsoft, AWS and Google implement it?
The evaluation and rollback mechanisms that catch failures before and after release, with statuses as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Offline evaluation before release | Foundry agent evaluators (tool call accuracy and tool selection carry no preview label; task adherence and others are preview); Copilot Studio agent evaluation (GA Mar 2026) | AgentCore Evaluations (GA 31 Mar 2026) with 13 built-in evaluators; batch evaluation (GA Jul 2026) | Gen AI evaluation service with adaptive rubrics (the GenAI client is preview); ADK eval with tool trajectory scoring |
| Online evaluation on live traffic | Foundry continuous evaluation on sampled responses, capped per hour | AgentCore online evaluation; Failure Insights (preview, Jul 2026) | Agent Observability (GA 18 Jun 2026) with evaluation of traces |
| Red teaming | Foundry AI Red Teaming Agent on PyRIT (preview; agentic categories run in the cloud only) | Strands Evals (17 Jun 2026); Bedrock Guardrails testing | Agent Simulation and Evaluation in the Agent Platform |
| Regression at model change | GA models on an 18-month lifecycle (12 months for Anthropic, DeepSeek, Fireworks and Mistral models); replacements testable about 90 days before retirement; Standard deployments auto-upgrade unless versionUpgradeOption is set | Bedrock model lifecycle (Active, Legacy, End-of-Life) for models launched from 7 Sep 2026; migration is not automatic | Model discontinuations announced in the release notes, for example Gemini 2.0 Flash on 1 Jun 2026 |
| Rollback | Immutable Foundry agent versions; Power Platform pipelines redeploy a previous solution version | Named AgentCore Runtime endpoints pinned to a version; A/B traffic splitting (GA Jul 2026) | Agent Runtime revisions and traffic splitting (preview since 19 May 2026) |
Status labels are the vendor pages’ own on 2 Nov 2026; several evaluators were preview, so keep a manual review step until they are not.
Frequently asked questions
Is the 95% figure about AI agents?
Not specifically. MIT NANDA’s July 2025 note covers enterprise generative AI systems broadly, from a review of 300+ initiatives, 52 interviews and 153 survey responses, and says 95% of organizations were getting zero return. Its pilot-to-production funnel (60%, 20%, 5%) covers enterprise-grade and custom systems, the category an agent falls into, and the causes it names, brittle workflows and misalignment with daily operations, are the ones we see.
Why do agents call the wrong tool?
Because tool selection is a reading task: the model picks from descriptions, and look-alike names, missing conventions in schemas and too many options all degrade it. Anthropic reports wrong tool selection and incorrect parameters as the most common failures. Fewer, clearer tools and an evaluation that scores the trajectory address the main causes.
Do agents get worse when the model changes?
They change, and without a regression run you find out in production. Microsoft Foundry retires GA models on an 18-month lifecycle (12 months for several partner models) and auto-upgrades Standard deployments at retirement; Anthropic says effort levels are recalibrated between versions. Pin the version, rerun the evaluation set before any switch, and keep a rollback path.
Does passing a benchmark mean an agent is ready?
No. Benchmarks measure models on someone else’s tasks, and METR found at least 16% of successful long runs in its study were illegitimate on review. The number that counts is the pass rate on your own evaluation set, on your data, with your tools, repeated several times.
What should acceptance criteria for an AI agent contain?
The workflow and its boundaries, the inputs, what a correct result is per case, the baseline it replaces, the pass bar per criterion, the tools it may and may not call, and who approves what. QueryNow’s acceptance criteria generator page helps you draft one, and the Evaluation topic’s acceptance-criteria sub-page covers them in depth.
More in What is an AI agent?
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- The GenAI Divide: State of AI in Business 2025 (PDF)
- Over 40% of agentic AI projects will be scrapped by 2027, Gartner says
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (arXiv 2412.14161)
- tau-bench: a benchmark for tool-agent-user interaction in real-world domains (arXiv 2406.12045)
- Frontier Risk Report, February to March 2026
- IBM report: 13% of organizations reported breaches of AI models or applications (Cost of a Data Breach 2025)
- Advanced tool use
- Function calling guide
- Gemini function calling
- Tool search tool (Claude API)
- API plugins and MCP plugins for Microsoft 365 Copilot
- OWASP Top 10 for Agentic Applications for 2026
- The Attacker Moves Second: stronger adaptive attacks bypass defenses against LLM jailbreaks and prompt injections (arXiv 2510.09023)
- Microsoft Entra Agent ID key concepts
- Overreliance on AI: a framework
- Foundry Models lifecycle and retirement
- Effort (Claude API)
- How we built our multi-agent research system
- Amazon Bedrock AgentCore release notes
- Evaluate agents in Copilot Studio
- Built-in evaluators (Microsoft Foundry)
- Agent Monitoring Dashboard and continuous evaluation (Microsoft Foundry)
- AI Red Teaming Agent (Microsoft Foundry)
- AgentCore Evaluations generally available
- Policy for Amazon Bedrock AgentCore generally available
- Top announcements of the AWS Summit in New York, 2026
- Amazon Bedrock model lifecycle
- AgentCore Runtime versioning and endpoints
- Guardrails and controls in Microsoft Foundry
- Power Platform pipelines
- Gen AI evaluation service overview
- Evaluate agents (Google ADK)
- Gemini Enterprise Agent Platform release notes
- What’s new in Copilot Studio
- Agent applications (Microsoft Foundry Agent Service)
- AgentCore Identity key features and benefits
- Agent Identity overview (Google Cloud IAM)
- Semantic conventions for GenAI spans (content capture is opt-in)
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner