- Home
- AI agents
- Evaluation and testing
- Offline and online evaluation
Operate · Evaluation and testing
What is the difference between offline and online evaluation for AI agents?
Offline evaluation tests an AI agent against a fixed set of cases before a release, with known answers and expected tool paths. Online evaluation scores a sample of real runs after release, using judges, rules and user signals. Offline tells you whether a version may ship; online tells you whether it still works on real traffic.
In this guide: 83 pages
Key takeaways
- Offline evaluation gates a release and online evaluation watches it. Both should use the same criteria and graders, so a score means the same thing before and after launch.
- Run each offline case several times. The tau-bench paper found tool-using agents succeeded on fewer than half of its tasks, and consistency across eight tries fell below 25% in its retail domain, so a single pass hides how often an agent fails.
- Compare the candidate with the baseline on the same cases. Microsoft’s Foundry evaluation GitHub Action (preview) reports confidence intervals and pairwise significance tests between the two versions.
- Online scoring samples live runs. Foundry continuous evaluation fires as each response completes, with Random or Intelligent sampling and an hourly cap that defaults to 100 runs.
- Calibrate an LLM judge against human labels before you trust it, and again after you change the judge model or rubric. The MT-Bench paper names position, verbosity and self-enhancement biases.
How do you build an offline set that predicts production?
Build it from real requests, cover the cases that hurt, and record the expected path as well as the expected answer. A set that holds only the happy path will pass every release and predict nothing.
Slices of an offline evaluation set for an agent, and what to record as the expected result for each.
| Slice | What goes in | Expected result to record |
|---|---|---|
| Common tasks | The requests that make up most real traffic, sampled from logs or pilot users | The answer or outcome, and the tools that should be called |
| Edge cases | Missing data, ambiguous requests, long inputs, the other languages your users write in | A clarifying question, a safe default or an escalation |
| Known failures | Runs that went wrong in production or testing | The corrected behaviour, signed off by a domain expert |
| Out of scope | Requests the agent must refuse or hand to a person | The refusal or hand-off, with no tool call |
| Adversarial | Instructions injected into documents, tool results and messages | No unauthorised tool call and no data sent out |
| Permission boundaries | Requests from users who lack access to the data involved | An answer limited to what that user may see |
Score the path as well as the answer. Foundry has evaluators for tool selection, tool input accuracy and task adherence, Google’s Agent Development Kit compares the actual tool trajectory with the expected one, and DeepEval separates trajectory metrics from component metrics. An agent can reach the right answer through a forbidden tool call, and an answer-only score will never show it.
How do you decide that a new version beats the baseline?
Run the baseline and the candidate on the same cases, several times each, and compare them criterion by criterion with an interval around each score. A candidate ships when it is no worse on any must-pass criterion and better where you meant it to be.
Repeat each case. Agents are non-deterministic, so report a pass rate per case across runs, and for critical tasks a consistency measure such as the pass^k metric from tau-bench, which counts a case as passed only when all k tries succeed.
Use statistics that fit the sample size. The Foundry evaluation GitHub Action (microsoft/ai-agent-evals, preview) runs evaluators against a named agent version and reports confidence intervals and pairwise significance tests between a baseline and a candidate. With a small golden set, a two-point difference is often noise, so read the interval before you announce a gain.
Calibrate the judge first. The MT-Bench paper found that strong judges agree with human preferences more than 80% of the time, the same level at which humans agree with each other, and named position, verbosity and self-enhancement biases. Score a human-labelled sample with the judge, compare the two, and repeat whenever the judge model or rubric changes. Wherever a criterion allows it, use a deterministic check instead, such as schema validation or an exact match on an amount.
How do you sample and score live traffic?
Score a sample of completed runs with the same graders you used offline, add the signals only production has, and route the doubtful runs to people. Online scoring reads real content, so it needs the content capture and access controls set for that agent’s risk tier.
- Sampling: Foundry continuous evaluation rules fire on completed responses, with Random or Intelligent sampling and max_hourly_runs defaulting to 100; AgentCore Evaluations offers online, on-demand, batch and dataset evaluation; Langfuse and LangSmith run online evaluators on traced runs. Oversample new versions, high-risk tiers and runs that called write tools, record each run’s sampling probability, and report results per sampling group so the oversampled runs do not skew the averages.
- Production-only signals: user feedback, escalations to a person, edits a person made before sending, reopened tickets and reversed transactions. Many of these live in the downstream system, so join them to the trace id.
- Efficiency: tokens, tool calls and latency per completed task. Google’s Agent Development Kit reports efficiency metrics without making them pass or fail, which is a sensible default.
- Drift: rerun the golden dataset against the live version on a schedule. Foundry calls this scheduled evaluation, and its recurring evaluations were marked preview on 2 Nov 2026.
- Cost: judges consume tokens. Foundry playground evaluations are on by default and billed with consumption, and AgentCore bills built-in evaluators on tokens and custom ones per 1,000 evaluations, so set sample rates with the bill in view.
How do online findings flow back into the offline set?
Through a review queue. Flagged runs go to people who know the domain, they write down the correct behaviour, and the corrected case joins the golden dataset, so each production failure becomes a test the next release must pass.
LinkedIn described an early form of this on its engineering blog on 25 Apr 2024: a team of linguists with tooling to evaluate up to 500 daily conversations and a layered review from engineers to annotators to members, while calling automatic evaluation "the holy grail, but still a work in progress". DKB’s banking assistant, publicly reported by Langfuse on 31 Aug 2026, closes the loop in the same way: flagged sessions go to expert annotation queues and become regression tests that run before each release.
Set a promotion rule before a reviewed failure joins the set: strip personal data, write the expected outcome and tool path, and have the domain owner approve the case. Keep the golden set honest as it grows. Date each case, tag the slice it belongs to, retire cases that no longer match the product, and keep a frozen subset so that scores stay comparable from one quarter to the next.
How do Microsoft, AWS and Google implement it?
Offline and online evaluation on the three clouds, with statuses as the vendor pages showed them on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Offline evaluation | Foundry built-in agent evaluators; several, such as Task Adherence and Task Completion, still in preview | AgentCore Evaluations, generally available 31 Mar 2026, with dataset and batch runs | Gen AI evaluation service with adaptive rubrics; adk eval for the Agent Development Kit |
| Release gate in CI | GitHub Action microsoft/ai-agent-evals (preview) | Batch evaluations and A/B testing, generally available July 2026 | The adk eval CLI and AgentEvaluator.evaluate in pytest |
| Online scoring | Continuous evaluation with Random or Intelligent sampling, 100 runs an hour by default | AgentCore online evaluation of traces in OpenTelemetry or OpenInference format | Evaluation of agent traces and response quality; the new GenAI Client SDK is in preview |
Microsoft’s blog dated 16 Mar 2026 announced general availability of Foundry evaluations, while the Learn pages still marked several evaluators and features preview on 2 Nov 2026. This page follows the docs.
Frequently asked questions
How big should an offline evaluation set be?
Big enough that a real regression shows up above the noise, and small enough that experts can check every expected answer. Start with each known failure plus the common tasks, run every case several times, and grow the set from production cases instead of invented ones.
Can online evaluation replace offline evaluation?
No. Online scores arrive after users have seen the output, so they detect problems and cannot prevent them. Offline evaluation is the gate before release; online evaluation tells you when the offline set has stopped matching real traffic.
Should we use an LLM as the judge?
For qualities a rule cannot check, such as whether an answer follows from its sources, yes, once its scores match a human-labelled sample. Keep rules for anything a rule can decide, and recheck the judge after changing its model or rubric, because the scores can shift without warning.
How often should online evaluation run?
Continuously on a sample, with a higher rate for new versions and high-risk agents, plus a scheduled rerun of the golden dataset, daily or weekly, to catch drift. Set the rates with the cost in view, since every run scored by an LLM judge consumes model tokens.
More in Evaluation and testing
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Built-in evaluators reference
- Agent Monitoring Dashboard in Microsoft Foundry
- How to run an evaluation in GitHub Action
- Observability in Microsoft Foundry
- Evaluations for Amazon Bedrock AgentCore
- Amazon Bedrock AgentCore release notes
- Amazon Bedrock AgentCore pricing
- Gen AI evaluation service overview
- Evaluate agents with the Agent Development Kit
- DeepEval metrics introduction
- Musings on building a Generative AI product
- How DKB runs its AI banking assistant on Langfuse
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner