- Home
- AI agents
- Evaluation and testing
- Acceptance criteria for AI agents
Operate · Evaluation and testing
How do you write acceptance criteria for an AI agent?
Acceptance criteria for an AI agent are the written, testable statements of what the agent must do on real data before the people who own the work sign it off: the outcome, the sample it is measured on, the pass level, the boundaries it may never cross and who it escalates to. They become the evaluation set.
In this guide: 83 pages
Key takeaways
- A criterion is good when someone who does the work today can test it on real data and say pass or fail. "Answers accurately" is a wish; "every required field correct on 50 real invoices from last quarter, on three runs" is a criterion.
- Five shapes cover most agents: accuracy on a fixed sample, confidence and escalation, boundaries, audit, and sign-off on a real sample. Add a tool-path criterion for any agent that acts.
- The workflow owner writes them with the builder and signs the measured results. For an agent that writes to a system of record or moves money, risk or compliance signs too.
- Each criterion maps to test cases, a grader and a threshold, so the signed sheet and the evaluation set are the same document in two forms. Vendors ship the graders; you supply the cases.
What makes an acceptance criterion good or bad?
A good criterion names an outcome, a sample, a pass level and a run count, and a person who does the work could check it without the builder in the room. A bad one describes a feeling about the agent. The table shows the same six intentions written both ways.
Six criteria written badly and well. The good versions are illustrative and the numbers are examples, not recommendations.
| Intention | Bad (untestable) | Good (testable) |
|---|---|---|
| Accuracy | The agent extracts invoice data accurately | Vendor, invoice date, total and PO number are all correct on 50 real invoices from last quarter, 50 of 50, on three consecutive runs |
| Escalation | The agent handles edge cases | An invoice with no PO number, or a confidence score below 0.8, is sent to the AP lead with the reason stated; none is auto-approved; checked on 20 such cases |
| Boundaries | The agent is secure | The agent reads only records the signed-in user can open and never calls the payment tool; 40 adversarial cases from the fraud team produce zero successes |
| Grounding | The agent does not hallucinate | Every policy answer cites the policy section it came from and the section contains the claim, on 100 policy questions; the judge is calibrated on 30 hand-labelled answers |
| Tool path | The agent works with the order system | For an exchange, the tool path is look up order, check stock, create exchange, in that order, with the right order id, on 60 cases |
| Speed and cost | The agent is fast and cheap | A full answer within 8 seconds at the 95th percentile over the test set, and no run makes more than 8 tool calls |
Copilot Studio’s evaluation guidance says to define what success looks like before choosing test methods, and that order matters: a criterion written after the build describes what the agent happens to do rather than what the work requires.
Which criteria does every agent need?
Every agent needs five, and an agent that acts needs a sixth. The five are the ones we write most often, and the acceptance criteria generator and template on this site produce them in this shape.
- Accuracy on a fixed sample: the required outputs, on a named set of real items, with a pass level and a run count. The run count matters because agents are non-deterministic; the tau-bench benchmark reported that the strongest tool-calling agents of 2024 passed fewer than half its tasks once and, in its retail domain, under a quarter eight times in a row, which is why we ask for several runs rather than one.
- Confidence and escalation: what the agent does when it is unsure, who it sends the item to, and the rule that low-confidence items are never auto-approved.
- Boundaries: where the agent runs, what data it may read (within the signed-in user’s permissions), which tools it may call, and what it may never do. The never list becomes the red-team scope.
- Audit: every decision logged with its source, its tool calls and, where a person approved it, the approver and the time. The Observability and cost topic lists what to capture.
- Sign-off: the team that owns the work approves a real sample before go-live, and the sheet records who, when and on what sample.
- For an agent that acts: the expected tool path for each action type, so trajectory evaluators have a ground truth, plus a cap on tool calls per run.
Add a model-change criterion for any agent that will live more than a year: the full set reruns before the vendor retires or upgrades the model, and the owner re-signs. Microsoft Foundry models follow published 18-month or 12-month lifecycles, and Azure Standard deployments auto-upgrade at retirement unless you change the setting.
Who writes the criteria, and who signs?
The person who owns the workflow writes them with the builder, in the owner’s words, and signs the measured results. The builder may not write them alone, because a builder who sets the pass level grades their own work.
Three signatures cover most agents. The workflow owner (the AP manager, the service director, the HR lead) signs that the criteria describe the job and later that the results meet them. The agent owner from the registry record signs that the agent will be run and monitored against the same criteria. For an agent in the highest risk tier, one that writes to a system of record, moves money or touches regulated data, the risk or compliance function signs the boundaries and the audit criterion, and the data protection officer signs any criterion that involves personal data.
Sign measured results. The sheet that goes to the signer carries the dataset size, the run count, the score per criterion, the judge calibration figure and the red-team findings. A signer who sees "97 of 100 credit cases, five of five runs, three failures listed below" can decide; a signer who sees "passed" cannot.
This is how we work. Every engagement starts with one scoped workflow and a written definition of done, agreed before the build starts, and the run phase after launch is measured against those criteria. Commercial terms can be tied to the criteria you sign; the statement of work sets them. The point for your own program is the same whoever builds the agent: the criteria exist before the code does, and the people who do the work hold the pen.
How does a criterion become the evaluation set?
Each criterion maps to a set of cases, a grader and a threshold, and the run count on the sheet becomes the number of trials per case. The vendors supply graders for the common shapes; you supply the cases, the expected outputs and the expected tool paths.
Accuracy criteria become exact-match or schema checks over the sample: Foundry’s Azure OpenAI graders and custom evaluators, AgentCore’s correctness evaluator with ground truth, or an overlap score such as ADK’s response_match_score (ROUGE-1) where exact match is too strict. Grounding criteria become a judge with a rubric (Foundry Groundedness, AgentCore faithfulness, ADK rubric-based criteria), calibrated against the hand-labelled sample the criterion names. Tool-path criteria become trajectory evaluators: Foundry’s Tool Selection, Tool Input Accuracy and Tool Call Success, AgentCore’s tool selection and tool parameter accuracy, ADK’s tool_trajectory_avg_score with exact match by default. Boundary criteria become the red-team scope and the safety evaluators (Prohibited Actions, Sensitive Data Leakage, Indirect Attack in Foundry; safety and harmfulness in AgentCore; safety_v1 in ADK).
Escalation criteria need cases that trigger them: 20 invoices with no PO number, 20 with a low confidence score, each with the expected routing. Audit criteria are checked on the trace rather than the answer: for each case, the trace holds the tool calls, the sources and the approver. Speed and cost criteria become thresholds on latency and on tool calls per run, which ADK reports as efficiency metrics without a pass or fail and which you turn into one.
Keep the mapping in the dataset itself: label each case with the criterion it tests, store the dataset with the agent version, and grow it from production. DKB’s domain experts add flagged real conversations to datasets that run as regression tests before each release (Langfuse case study, 31 Aug 2026), which is a signed criterion kept alive. When a case fails after launch, the fix goes in, the case stays, and the owner re-signs the next release on the larger set.
How do Microsoft, AWS and Google implement it?
Where the criteria run on each cloud, as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Accuracy and grounding graders | Foundry Azure OpenAI graders, Groundedness, Relevance, and Rubric and custom evaluators (preview); Copilot Studio compare-meaning and text-similarity test methods | AgentCore correctness, faithfulness, context relevance; Bedrock Evaluations for RAG | ADK response_match_score, final_response_match_v2, rubric-based criteria; Gen AI evaluation service adaptive rubrics |
| Tool-path ground truth | Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success, Task Navigation Efficiency | Tool selection accuracy, tool parameter accuracy, goal success rate on Strands and LangGraph traces | tool_trajectory_avg_score, exact match by default; user simulation |
| Sign-off evidence | Evaluation results in the Foundry portal and the GitHub Action report with confidence intervals (preview); Copilot Studio evaluations run as the signed-in user | Evaluation results and batch reports in AgentCore; A/B test results | adk eval reports and pytest output from AgentEvaluator.evaluate |
Annotation queues for the human sample live in the independent tools (Langfuse, LangSmith, Arize); the cloud portals show the evaluation results.
Frequently asked questions
What is the difference between acceptance criteria and evaluation metrics?
A criterion is the business statement of what must be true, signed by the owner. A metric is the number a grader produces. One criterion usually needs one or two metrics with a threshold; a metric without a criterion has no pass level anyone agreed to.
How many acceptance criteria should an AI agent have?
Between five and ten. Fewer than five leaves out escalation, boundaries or audit; more than ten means some are really test cases. Each criterion then carries 30 to 50 cases, so a ten-criterion agent has a few hundred cases, which is what our worked examples use.
Can acceptance criteria be used in a contract?
Yes, when they are testable and the sample is named. Our own engagements tie the build and the run phase to the criteria you sign, with the commercial terms set in the statement of work. Whoever builds your agent, insist that the criteria exist before the build and that acceptance means measured results on real data.
What if the owner cannot say what the pass level should be?
Measure the people doing the work today on the same sample, and set the first pass level at that figure. An agent that matches the current error rate with a full audit trail is a release candidate; one that must be perfect before anyone will sign never leaves the pilot.
More in Evaluation and testing
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Agent evaluation in Copilot Studio
- Built-in evaluators for Microsoft Foundry
- Evaluate agents with the Foundry GitHub Action
- Foundry Models lifecycle and retirement
- Evaluations for Amazon Bedrock AgentCore
- Evaluate agents with the Agent Development Kit
- Gen AI evaluation service overview
- tau-bench: a benchmark for tool-agent-user interaction in real-world domains (Yao et al.)
- How DKB runs its AI banking assistant on Langfuse
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner