Operate · Topic 14 of 14
How do you evaluate and test AI agents?
Evaluating an AI agent means checking that it does the job well enough to release and keeps doing it after every change. You run a fixed test set before release (offline) and score samples of real traffic after it (online), grading both the final answer and the path the agent took: which tools it called and with which arguments.
In this guide: 83 pages
Key takeaways
- Write the acceptance criteria before the build and turn each one into test cases. The criteria are the evaluation set.
- Score the path as well as the answer. Tool selection, tool arguments, result use and task adherence are where agents fail, and all three clouds ship evaluators for them.
- A model as judge scales and has measured biases (position, verbosity, self-preference). Calibrate it against human labels and keep exact rules wherever a rule is possible.
- Agents are non-deterministic, so run each case several times and track consistency. tau-bench reported that the strongest tool-calling agents of 2024 passed under half its tasks once and, in its retail domain, under a quarter eight times in a row.
- Model retirements are scheduled and auto-upgrades are the default on Azure Standard deployments. A regression suite that reruns on every change keeps behaviour from shifting unseen.
Why is evaluating an agent different from testing software?
Because the same input can produce a different path and a different answer on each run, and because a run can reach the right answer through a forbidden tool call. An evaluation runs many cases many times and scores the answer, the path and the consistency.
The public benchmark that made this plain is tau-bench (Yao et al., June 2024), which tests tool-using agents against simulated customers in retail and airline domains. Its paper reports that even the strongest function-calling agents of the time succeeded on fewer than half the tasks, and that when asked to pass the same task eight times in a row (the pass^8 metric it introduced) the retail score fell below a quarter. The numbers have moved since; the method stands. Evaluation also has to look inside the run, and Foundry, AgentCore and Google’s Agent Development Kit all score the trajectory. LinkedIn described the human side in April 2024: a linguist team evaluating up to 500 conversations a day, engineers reviewing first, then annotators, then members, and the admission that automatic evaluation was still a work in progress. The automatic layer has grown since, and the human layer is still what calibrates it.
Offline vs online evaluation: what runs when?
Offline evaluation runs a fixed dataset before a release and gates it; online evaluation scores a sample of live traffic after release and watches for drift. You need both, and they share the same criteria and graders.
The evaluation stages, what each needs and where the clouds run it, as the vendor pages showed on 2 Nov 2026.
| Stage | When and why | Inputs and graders | Where it runs |
|---|---|---|---|
| Offline, dataset | Before every release and after every model, prompt, tool or knowledge change | Golden dataset with expected outputs and tool paths; exact rules, a calibrated judge, a human sample; several runs per case | Foundry evaluations; AgentCore dataset and batch evaluations (batch GA Jul 2026); Google Gen AI evaluation service and adk eval; Copilot Studio agent evaluation |
| CI gate | On every pull request or build | The same dataset with thresholds that block the merge | Foundry GitHub Action microsoft/ai-agent-evals (preview; confidence intervals and pairwise tests); deepeval test run in pytest; adk eval; promptfoo; Copilot Studio REST APIs and connectors |
| Online, continuous | On a sample of live runs as they complete | Judge and safety evaluators on real traces; flagged runs to a human queue | Foundry continuous evaluation with an hourly cap (default 100 runs); AgentCore online evaluation; Langfuse and LangSmith online evaluators |
| Scheduled | Daily or weekly, to catch drift | The golden dataset rerun on a schedule | Foundry scheduled evaluation (recurring evaluations marked preview); AgentCore batch evaluations |
| Red team | Before release and after every change to tools or permissions | Adversarial prompts, automated and human; attack success rate per category | Foundry AI Red Teaming Agent (preview); PyRIT, garak, promptfoo |
| A/B and canary | When two versions are candidates | Live traffic split between versions, scored online | AgentCore A/B testing (GA Jul 2026); Google Agent Runtime revisions with traffic splitting (preview since 19 May 2026) |
Microsoft’s blog dated 16 Mar 2026 announced general availability of Foundry evaluations; on 2 Nov 2026 the Learn pages still marked several features preview, and this page follows the docs.
How do you write acceptance criteria and build the golden dataset?
Write the criteria with the person who does the work today, each one testable on real data, then collect the cases that test it. Copilot Studio’s guidance is to define what success looks like before choosing test methods, and that order is the method.
A criterion names an outcome, a sample and a pass level: every required field extracted correctly on 50 real documents; a credit issued only when the policy conditions are met; low-confidence items sent to a named person and never auto-approved; every answer grounded in a cited policy. Those shapes (accuracy on a fixed sample, boundaries, escalation, grounding) cover most agents. Our acceptance criteria template and generator produce them in that form, and the sub-page on acceptance criteria for AI agents shows good and bad examples and who signs.
The golden dataset is the criteria made concrete: real inputs with the expected output and, for an agent, the expected tool path, since ADK’s tool_trajectory_avg_score and Foundry’s tool evaluators need that ground truth. Start with 30 to 50 cases per criterion from last quarter’s real work, add the edge cases the owner fears most, and label each case with its criterion. Then grow the set from production: DKB’s domain experts add flagged real conversations to datasets that run as regression tests before each release (Langfuse case study, 31 Aug 2026), and Klarna described test-driven development for its support assistant in LangSmith (LangChain case study, 12 Feb 2025).
Which graders should you use, and what are the limits of LLM-as-judge?
Use an exact rule wherever a rule is possible, a model as judge for qualities a rule cannot capture, and people to calibrate the judge. A judge is cheap and fast and has measured biases, so its scores count only after a check against human labels.
Deterministic graders are exact match, regular expressions, schema checks and overlap scores such as F1, BLEU and ROUGE; they spend no judge-model tokens and give the same answer on every run, so use them for extracted fields, tool names, argument values and formats. LLM-as-judge means a second model scores the output against a rubric for groundedness, coherence, task adherence or helpfulness. The MT-Bench paper (Zheng et al., 2023) found strong judges reached over 80% agreement with human raters, the level humans reach with each other, and named three biases: position (preferring the first of two answers), verbosity (preferring the longer one) and self-enhancement (preferring its own style). Microsoft adds that attack-success scoring by generative models can be non-deterministic, with a chance of false positives.
Calibrate before you trust. Take a sample the owner has labelled by hand, run the judge, measure agreement and set the threshold where the judge’s pass matches the owner’s; repeat whenever the judge model or the rubric changes, because a judge upgrade silently shifts every score. Google’s evaluation service now generates adaptive rubrics, per-prompt pass or fail tests, and Foundry added a Rubric evaluator in preview. Judges cost tokens: AgentCore bills built-in evaluators on tokens and custom evaluators per 1,000 evaluations, and Foundry playground evaluations are billed by default, so sample online traffic rather than scoring every run.
How do you evaluate the path the agent took?
Score each step against the expected tool path: did the agent pick the right tool, pass the right arguments, use the result, finish the task and take a reasonable number of steps. The clouds and the open-source libraries name these the same way, which makes the criteria portable.
- Microsoft Foundry built-in agent evaluators (page dated 9 Sep 2026): Tool Call Accuracy, Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success and Task Navigation Efficiency carry no preview label; Task Adherence, Task Completion, Intent Resolution, Output Quality, Tool Use Quality, Rubric and custom evaluators are preview. Conversation-level evaluation scores a whole conversation, with RAG and safety evaluators (Groundedness; Prohibited Actions, Sensitive Data Leakage, Indirect Attack) alongside.
- Amazon Bedrock AgentCore Evaluations (preview 2 Dec 2025, GA 31 Mar 2026): converts Strands and LangGraph traces in OpenTelemetry or OpenInference format and scores them for correctness, helpfulness, tool selection accuracy, tool parameter accuracy, safety, goal success rate, context relevance, faithfulness, harmfulness and stereotyping, plus custom and third-party evaluators. Failure Insights (preview, Jul 2026) looks for recurring failure patterns, including silent behavioural failures that produce no error.
- Google Agent Development Kit: tool_trajectory_avg_score (exact match by default), response_match_score (ROUGE-1), final_response_match_v2 (a judge), rubric-based criteria, hallucinations_v1, safety_v1, multi-turn criteria and efficiency metrics such as token_usage_v1 that are reported and never pass or fail; adk eval runs it all from the command line or pytest.
- Open-source libraries: DeepEval separates trajectory metrics (Task Completion, Step Efficiency, Plan Adherence) from component metrics (Tool Correctness, Argument Correctness); Ragas adds Topic Adherence, Tool Call Accuracy, Tool Call F1 and Agent Goal Accuracy to its RAG metrics.
Give efficiency metrics a threshold even where the library reports them without one: a right answer in 40 tool calls is a cost incident.
What do you retest when the model changes?
Everything in the golden dataset, before the change reaches production, because model changes are now scheduled, frequent and sometimes automatic. The retest is the same offline evaluation with the change swapped in, compared against a baseline on pinned versions.
The schedules are published. In Microsoft Foundry a generally available model follows an 18-month lifecycle (12 months for Anthropic, DeepSeek, Fireworks and Mistral models), with at least 60 days’ notice before a GA retirement; replacements are testable about 90 days before it, Standard deployments auto-upgrade at retirement unless versionUpgradeOption is changed, and Provisioned deployments do not. Amazon Bedrock’s policy for models launched from 7 Sep 2026 moves a model from Active to Legacy to End-of-Life, when requests fail and migration does not happen automatically. Anthropic retired two Claude 4 models on 15 Jun 2026 and another on 5 Aug 2026, with a further retirement scheduled for 30 Nov 2026.
Settings do not carry over. Anthropic says effort levels are recalibrated between model versions and that you should run a fresh effort sweep on your evaluations rather than carry the setting over; OpenAI warns that its newest model is more sensitive to instructions in skills and AGENTS.md files. A model swap is therefore a prompt change and an effort change at the same time. Harvey, a legal AI company, described the practice in Google’s launch post for Gemini 4 Argon (13 Oct 2026): it reran its legal agent evaluations on the new model and reported improvements over the previous one. The Agent lifecycle management sub-page covers version pinning and rollback.
Which evaluation tools exist beyond the three clouds?
A dozen independent tools cover datasets, judges, trajectory metrics, CI integration and red teaming, most of them open source. The table lists the ones we see in enterprise use, as their pages showed on 2 Nov 2026.
Independent evaluation and red-teaming tools, as of 2 Nov 2026.
| Tool | What it does for agents | Licence and ownership |
|---|---|---|
| LangSmith | Datasets, offline experiments, online evaluators, annotation queues, alongside tracing | LangChain; cloud, hybrid or self-hosted |
| Langfuse | Datasets, LLM-as-judge, annotation queues, prompt management, alongside tracing | Open source and self-hostable; joined ClickHouse 16 Jan 2026 |
| Arize Phoenix and AX | LLM, code or human evaluators; supports Ragas, DeepEval and Cleanlab evaluators | Phoenix open source on OpenTelemetry; AX managed |
| W&B Weave | Evaluation API, scorers, leaderboards, guardrails | Weights & Biases, part of CoreWeave |
| promptfoo | Evaluations and red teaming from one CLI; CI checks and pull-request scanning | MIT; announced it is joining OpenAI on 9 Mar 2026 and says it stays open source |
| DeepEval | G-Eval, agentic trajectory and component metrics, pytest integration | Open source |
| Ragas | RAG metrics (Faithfulness, Context Precision and Recall) and agent metrics (Topic Adherence, Tool Call Accuracy, Agent Goal Accuracy) | Open source |
| PyRIT | Red-teaming toolkit with attack strategies and scoring; the engine under Foundry’s AI Red Teaming Agent | Microsoft, MIT |
| garak | Vulnerability scanner with probes (DAN, encoding injection) and detectors | NVIDIA-hosted, Apache-2.0 |
| Inspect | Evaluation framework with custom and MCP tools, sandboxing (Docker, Kubernetes, Modal) and bridges to Claude Code, Codex CLI and Gemini CLI | UK AI Security Institute and Meridian Labs |
How do Microsoft, AWS and Google implement it?
The evaluation services of the three clouds, with statuses as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Managed agent evaluation | Foundry evaluations with built-in agent, RAG and safety evaluators; several agent evaluators preview; Copilot Studio agent evaluation for low-code agents | AgentCore Evaluations, GA 31 Mar 2026; batch evaluations and A/B testing GA Jul 2026; Bedrock Evaluations for models and RAG (RAG evaluation GA 20 Mar 2025) | Gen AI evaluation service with adaptive rubrics (GenAI Client SDK preview; legacy module no longer actively developed); ADK evaluation and adk eval |
| Trajectory scoring | Tool Call Accuracy, Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success, Task Navigation Efficiency | Tool selection accuracy, tool parameter accuracy, goal success rate; traces from Strands and LangGraph | tool_trajectory_avg_score, multi-turn task success, efficiency metrics reported without pass or fail |
| Online evaluation | Continuous evaluation on completed responses with an hourly cap (default 100); scheduled evaluation; recurring evaluations preview | Online, on-demand, batch and dataset evaluation plus simulation; Failure Insights preview | Evaluation of agent traces and response quality; Agent Observability preview |
| CI integration | GitHub Action microsoft/ai-agent-evals (preview) with confidence intervals and pairwise tests; Copilot Studio REST APIs and connectors | AgentCore Evaluations APIs and the AgentCore CLI (1.0, Oct 2026) | adk eval CLI and AgentEvaluator.evaluate for pytest |
| Red teaming | AI Red Teaming Agent on PyRIT (treat as preview); agentic risk categories in five regions | Safety, harmfulness and stereotyping evaluators; no managed agent red-teaming service found in the docs we read | safety_v1 and hallucinations_v1 criteria; no managed agent red-teaming service found in the docs we read |
| Model lifecycle | 18-month GA lifecycle (12 for some partners); Standard deployments auto-upgrade at retirement, Provisioned do not | Active, Legacy, End-of-Life for models launched from 7 Sep 2026; no automatic migration | Discontinuations announced in release notes; two Gemini 2.0 models discontinued 1 Jun 2026 |
AgentCore Evaluations and Policy GA dates come from the update notes on the AWS News Blog post; the separate What’s New posts were not opened.
When is a full evaluation program too much, and what goes wrong?
An agent that drafts text for one person to edit needs a short checklist and a weekly review. The program on this page is for agents that act, serve many users or touch regulated data. Once you run it, these are the ways it fails.
Judge bias and drift
Position, verbosity and self-preference bias are documented, and a judge model upgrade shifts every score. Calibrate against human labels and recalibrate after any judge or rubric change.
Testing the answer and not the path
An agent can reach the right answer through a forbidden tool call, or the wrong one through the right steps. Score trajectories, and treat a forbidden call as a failure whatever the answer.
Golden sets that go stale, scored once
A dataset written once never covers what users ask a year later, and one pass proves little for a non-deterministic system. Pull flagged production traces in on a schedule and run each case several times.
Red-team blind spots
Foundry’s agentic red-team tests were single-turn and English-only, with synthetic data and mock tools, and did not cover function tool calls or non-Foundry agents on 2 Nov 2026. Add human, multi-turn attacks against your real tools in a purple environment.
Unplanned model changes
Azure Standard deployments auto-upgrade at retirement, and preview models can be force-upgraded with 30 days’ notice. Without a regression suite wired to the change, behaviour shifts reach production unseen.
Benchmark overreach and unbudgeted judges
A public leaderboard does not predict performance on your tools and data; Microsoft’s advice is to compare quality, latency and cost together. Judges consume tokens, so put the evaluation cost in the agent’s budget and sample online traffic.
What should you set up?
The controls that make an evaluation program hold, in the order we install them.
- Acceptance criteria signed by the workflow owner before the build, each one testable on real data, with a pass level and a run count.
- A golden dataset of 30 to 50 cases per criterion with expected outputs and tool paths, labelled by criterion and stored with the agent version.
- Graders chosen per criterion: exact rules first, a judge calibrated on a hand-labelled sample second, a human queue for flagged runs.
- Trajectory evaluators on (tool selection, arguments, result use, task completion, step count) with thresholds.
- A CI gate that runs the dataset several times on every change and blocks the release below threshold, with baseline and candidate on pinned model versions.
- Online evaluation on a sample of live runs, flagged runs curated into the dataset monthly, and a red-team scan in the release gate.
- A model-change calendar from the vendor lifecycle pages, with a full rerun before each retirement or upgrade, and the results linked from the registry record.
Enterprise example · Worked example · illustrative
A 12,000-person retailer gates a returns agent with 320 cases
Illustrative scenario. An apparel retailer with 12,000 staff, a customer-service agent on Amazon Bedrock AgentCore that handles returns and credits through the order system, and a second agent on Microsoft Foundry that answers store staff from the policy library.
The returns agent could issue a credit, which put it in the highest risk tier, so the customer-service director signed eight acceptance criteria before the build. Two carried the weight: a credit is issued only when the return window, the item condition and the payment method all pass policy, on 100 real cases from the last quarter; and any case below the confidence threshold goes to a named team lead and is never auto-approved.
The dataset held 320 cases: 100 credit cases with the expected decision and tool path (look up order, check policy, create return, issue credit), 60 exchanges, 40 escalations, 70 policy questions and 50 adversarial cases from the fraud team. Exact rules graded decisions, tool names and amounts; a judge graded grounding and tone, calibrated on 80 hand-labelled cases, where it agreed on 74. Each case ran five times in AgentCore Evaluations and every credit case had to pass five of five. The first candidate failed three credit cases the same way each run: it credited a final-sale item because the search tool returned the general policy rather than the final-sale clause, a retrieval fault fixed in the knowledge source. The red-team scan found that a crafted order note could make the agent reveal another customer’s order; the tool now returns only the authenticated customer’s orders, and the attack case joined the dataset.
In production, online evaluation scores one run in ten, flagged runs land in a queue the team lead reviews each morning, and the dataset grows by about 20 cases a month. When AWS listed the agent’s model as Legacy, the team reran all 320 cases on the replacement and found it needed a lower effort setting to stop it checking the policy twice. The store-staff agent on Foundry runs the same program.
- Owner
- Customer-service director signs the criteria and the release; a named team lead owns the annotation queue
- Identity
- AgentCore Identity workload identity for the returns agent; Entra agent identity for the store-staff agent; credits carry the agent id and the approver
- Risk tier
- Tier 3 for the returns agent (writes to the order system and moves money); tier 2 for the policy agent
- Evaluation gate
- 320 cases, five runs each, credit cases five of five; judge calibrated on 80 hand-labelled cases; red-team scan per category
- Approval
- Release approved only with the evaluation report attached; low-confidence credits approved by the team lead in the queue
- Monitoring
- Online evaluation on one run in ten; alerts on tool calls per run and on evaluation score below threshold
- Model change
- Full rerun and effort sweep before the Legacy model reaches End-of-Life; pinned model versions on both agents
- Retirement
- Dataset, evaluation reports and red-team findings archived with the agent version; identity revoked at retirement
How do you start?
- Write the acceptance criteria with the workflow owner and get them signed; the template or the generator on this site gives you the shape.
- Collect 30 to 50 real cases per criterion with expected outputs and tool paths, and label them.
- Pick a grader per criterion, rules first, and calibrate any judge on a hand-labelled sample before you trust it.
- Run the set five times on the current build, record the baseline, and set thresholds from it.
- Wire the run into CI and the release gate, add a red-team scan for any agent that acts, and switch on online evaluation for a sample of live runs.
- Put the vendor model-retirement dates on the calendar and rerun the set before each one.
Frequently asked questions
What is the difference between evaluation and testing for an AI agent?
Testing asserts that a known input gives a known output, once. Evaluation scores many cases, run several times, against criteria, and grades the answer, the path the agent took and how consistently it passes, which is what a non-deterministic system needs.
What metrics should we use to evaluate an AI agent?
Task success against written acceptance criteria first. Then trajectory metrics (tool selection, argument accuracy, result use, task completion, step count), quality and safety metrics (groundedness, prohibited actions, data leakage), consistency across runs, and cost and latency per successful outcome.
Can we rely on LLM-as-judge?
As a scaled first pass, once it is calibrated. Strong judges reached over 80% agreement with humans in the MT-Bench study, and they carry position, verbosity and self-preference biases. Check the judge against a human-labelled sample, set thresholds from that check, and recalibrate after any judge or rubric change.
How many test cases does an agent need?
Enough to cover each criterion with real variety: we start at 30 to 50 cases per criterion, run five times each, and grow the set from flagged production runs every month. Add variety before adding volume.
What is AgentCore Evaluations, and is it generally available?
AgentCore Evaluations is the managed evaluation service in Amazon Bedrock AgentCore. It scores agent traces from Strands and LangGraph with built-in and custom evaluators, online or on datasets. It entered preview on 2 Dec 2025 and became generally available on 31 Mar 2026, with batch evaluations and A/B testing in July 2026.
How do we evaluate an agent built in Copilot Studio?
Copilot Studio has an agent evaluation feature: generate, import or write test sets, single-response or multi-turn, with test methods such as compare meaning and text similarity and four safety evaluators, automated through REST APIs and connectors. Its page carried no preview banner on 5 Oct 2026, and we found no GA date.
In this topic
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Built-in evaluators for Microsoft Foundry
- Agent Monitoring Dashboard and continuous evaluation
- Observability in Microsoft Foundry
- Evaluate agents with the Foundry GitHub Action
- AI Red Teaming Agent
- Foundry Models lifecycle and retirement
- Agent evaluation in Copilot Studio
- Generally available: evaluations, monitoring and tracing in Microsoft Foundry
- PyRIT: Python Risk Identification Toolkit
- Evaluations for Amazon Bedrock AgentCore
- Amazon Bedrock AgentCore adds quality evaluations and policy controls for deploying trusted AI agents
- Amazon Bedrock AgentCore release notes
- Amazon Bedrock AgentCore pricing
- Evaluate models and RAG in Amazon Bedrock
- Amazon Bedrock RAG evaluation is now generally available
- Amazon Bedrock model lifecycle
- Gen AI evaluation service overview
- Evaluate agents with the Agent Development Kit
- Gemini Enterprise Agent Platform release notes
- Gemini 4 Argon is available on Google Cloud
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (Zheng et al.)
- tau-bench: a benchmark for tool-agent-user interaction in real-world domains (Yao et al.)
- Effort
- Claude Platform release notes
- Latest model guide
- Musings on building a generative AI product
- How DKB runs its AI banking assistant on Langfuse
- How Klarna uses LangSmith and LangGraph
- LangSmith observability
- Langfuse joins ClickHouse
- promptfoo blog, including the announcement that promptfoo is joining OpenAI
- CoreWeave news
- Arize Phoenix documentation
- W&B Weave
- promptfoo
- DeepEval metrics introduction
- Ragas available metrics
- garak: LLM vulnerability scanner
- Inspect evaluation framework
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner