- Home
- AI agents
- Evaluation and testing
- Regression testing when models change
Operate · Evaluation and testing
How do you regression test an AI agent when the model changes?
Regression testing an AI agent means rerunning a fixed evaluation set whenever something underneath the agent changes, above all the model, and comparing the result with the current version before the change reaches users. Vendors now retire models on published schedules and some deployments upgrade automatically, so the test runs on the vendor’s calendar as well as yours.
In this guide: 83 pages
Key takeaways
- More than the model name counts as a model change: a new version, a retirement, a new effort or reasoning setting, a router update, a prompt, a tool schema, a knowledge source, an SDK upgrade or a new judge model can each change behaviour.
- Know the calendars. In Microsoft Foundry most generally available models live 18 months, with at least 60 days’ notice and the replacement testable about 90 days before retirement; on Amazon Bedrock, models launched from 7 Sep 2026 move from Active to Legacy to End-of-Life, and requests fail at end of life.
- Pin versions and choose the upgrade rule on purpose. Azure’s versionUpgradeOption decides whether a Standard deployment follows the vendor’s default, Provisioned deployments never auto-upgrade, and a retired model answers with errors.
- Settings do not carry across versions. Anthropic says effort levels are recalibrated between models and recommends a fresh effort sweep on your evaluations, and OpenAI warns that its newest model is more sensitive to instructions in skills and AGENTS.md files.
- Roll out through a canary. AgentCore A/B testing (generally available July 2026) and Agent Runtime traffic splitting (preview since 19 May 2026) send a share of live traffic to the candidate. In Foundry’s legacy Agent Applications publishing experience an application routes all traffic to one deployment, so check the current publishing model before you design a canary there.
What counts as a model change?
Anything that changes how the model behaves for your agent, whether or not the model name changes. The list below is the set of triggers we wire to the regression suite.
- A new model version or a different model, including one the provider updates in place; the current version of the Foundry model router, for example, is updated in place.
- A scheduled retirement, which forces a move whether you planned one or not.
- A change of effort or reasoning setting. OpenAI, Anthropic and Google each expose one, and the levels differ by model.
- A router change, such as a new model in the pool or a new routing mode. The Foundry router’s context window is limited by the smallest model in the pool, and routing can lower prompt cache hits.
- A prompt, instruction or AGENTS.md change.
- A tool change: a new tool, a changed schema or description, or a new MCP server version.
- A knowledge change: a new index, chunking rule or embedding model.
- A framework or SDK upgrade, which can change how tool calls, reasoning state and retries are handled.
- A new judge model or rubric, which moves the scores even when the agent stays the same.
When will the vendors change your model?
On published schedules that differ by vendor and are often shorter than enterprise release cycles. The table summarises the lifecycle rules and recent dates as the vendors published them on 2 Nov 2026.
Model lifecycle rules and dates by vendor, as published on 2 Nov 2026. Dates change often; check the vendor page before you plan.
| Vendor | Lifecycle rule | What happens at the end | Dates to know |
|---|---|---|---|
| Microsoft Foundry | Generally available models: 18 months, limited to existing customers from 12; 12 months for Anthropic, DeepSeek, Fireworks and Mistral models; at least 60 days’ notice for a GA retirement and 30 for preview | Requests return 410 Gone; Standard deployments auto-upgrade unless versionUpgradeOption says otherwise, Provisioned deployments do not | gpt-4o 2024-05-13 retires 14 Jan 2027, with gpt-5.1 as the replacement |
| Amazon Bedrock | For models launched from 7 Sep 2026: Active, then Legacy (6 months for most models, 45 days for some), then End-of-Life; model cards show an "EOL no sooner than" date | Requests fail and migration is not automatic; in Legacy, existing customers may lose access after 15 days of inactivity | Bedrock Agents Classic closed to new customers on 30 Jul 2026 with a frozen model catalogue |
| Per-model dates in the Gemini Enterprise Agent Platform release notes | The model stops being available | Gemini 2.0 Flash and 2.0 Flash-Lite discontinued on 1 Jun 2026 | |
| Anthropic, Claude API | Per-model retirement dates in the release notes | Requests to the retired model stop working | Claude Sonnet 4 and Opus 4 retired 15 Jun 2026 and Opus 4.1 on 5 Aug 2026; Sonnet 4.5 scheduled for 30 Nov 2026 |
| OpenAI | Deprecations announced per model and API, with shutdown dates | The API or model is shut down | Assistants API shut down 26 Aug 2026; Agent Builder shuts down 30 Nov 2026 |
Read your own estate as well as the vendor pages. Azure exposes a lifecycleStatus field through its Models API and Bedrock a modelLifecycle field, so a nightly job can list every deployed model with its retirement date and open a ticket when one enters the notice window.
How do you run the regression test?
Freeze a baseline, run the candidate on the same golden set with several tries per case, compare per criterion and per slice, then canary before you switch. The steps below are the procedure we use for a model change.
- Pin the current version and record its scores as the baseline, together with the exact model version, effort setting, prompt and tool versions.
- Deploy the candidate side by side under a new deployment, alias or named endpoint, with nothing else changed.
- Rerun the effort or reasoning sweep for the candidate first, because the levels are recalibrated between model versions.
- Run the full golden dataset on both versions, several tries per case, with the same judge model and rubric.
- Compare per criterion and per slice: task success, tool path, refusals and escalations, tokens and cost per task, and latency. An average gain can hide a loss on the adversarial or out-of-scope slice.
- Send a share of live traffic to the candidate and compare online scores with the baseline before you move the alias or endpoint.
- Switch the pin, keep the old version reachable until its end of life, and attach the comparison to the registry record as the approval evidence.
What breaks when a model changes, and how do you catch it?
The failures are usually quiet: shorter tool sequences, cut-off answers, reasoning state lost between steps or a jump in cost. Each one has a specific check in the regression run.
Symptoms after a model change, their usual causes and the check that catches each one, based on vendor guidance as of 2 Nov 2026.
| Symptom | Likely cause | Check in the regression run |
|---|---|---|
| The agent skips a search or stops early | A lower effort level, which Anthropic says makes skipped checks and fewer, terser tool calls more likely | Tool path match and task completion per slice |
| Empty or cut-off responses | Output caps that count reasoning tokens as well as visible output | The rate of incomplete responses; raise the cap if needed |
| Errors or weaker results in multi-step runs | Reasoning state not passed back between tool calls, such as OpenAI reasoning items or Google thought signatures | Multi-turn cases that span several tool calls |
| Instructions followed differently | Newer models weigh skills and AGENTS.md files differently, as OpenAI warns for its newest model | Out-of-scope and policy cases |
| Higher cost per task | Reasoning tokens billed as output, or more tool calls per task | Tokens and cost per completed task against the baseline |
| Fewer cache hits or a smaller context window | A router that spreads requests across models, limited by the smallest model in its pool | Cache hit rate and long-input cases |
| Scores move although the agent did not change | A new judge model or rubric | Rescore the baseline with the new judge before you compare |
Resistance to injected instructions also changes between versions. Google reported 0.0% attack success for Gemini 4 Carbon against 0.7% for Gemini 4 Argon on the Gray Swan indirect prompt-injection benchmark, a vendor-reported figure, so keep the adversarial slice in every run even when the new model is described as safer.
How do Microsoft, AWS and Google implement it?
The controls each cloud offers for model changes, as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Pinning and upgrade rule | versionUpgradeOption: OnceNewDefaultVersionAvailable, OnceCurrentVersionExpired or NoAutoUpgrade | Call a specific model id; AgentCore named endpoints stay on a version until you update them | Specific model versions on Gemini Enterprise Agent Platform; discontinuation dates in the release notes |
| Lifecycle data by API | lifecycleStatus in the Azure Models API | The modelLifecycle field in Amazon Bedrock | Release notes; no API field verified for this page |
| Side-by-side comparison | Evaluation GitHub Action with pairwise tests between versions (preview) | Batch evaluations and A/B testing, generally available July 2026 | Agent Runtime revisions with traffic splitting, preview since 19 May 2026 |
Microsoft makes the replacement for a retiring model testable in Global Standard about 90 days before retirement, which is the natural moment to start the regression run.
Frequently asked questions
When should we start testing a replacement model?
As soon as the vendor announces the retirement and the replacement is available. In Foundry that means at least 60 days before a GA retirement, with the replacement usually testable about 90 days ahead, and retirement dates cannot be extended. Starting in the last month leaves no time to fix prompts or tools.
Should we turn off automatic model upgrades?
For agents with signed acceptance criteria, yes: pin the version and move it yourself after the regression run. The trade-off is that a pinned model still retires on the vendor’s date, so pinning only works with a calendar and someone who owns it.
How many test runs per case are enough?
Enough to see whether the agent is consistent. The tau-bench authors found consistency across eight tries fell below 25% in one domain, so one run can pass a case the agent usually fails. Several tries per case on the critical slices, and fewer on the rest, is a workable balance.
Do we need to retest when only the prompt changes?
Yes. A prompt change can alter tool choices and refusals as much as a model change, and prompts usually change more often than models. Run the same suite, and keep prompts versioned in a registry so the comparison names the exact versions.
More in Evaluation and testing
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Foundry Models lifecycle and retirement
- Model router in Microsoft Foundry
- How to run an evaluation in GitHub Action
- Foundry agent applications
- Amazon Bedrock model lifecycle
- Amazon Bedrock Agents Classic maintenance mode
- AgentCore Runtime versioning and endpoints
- Amazon Bedrock AgentCore release notes
- Gemini Enterprise Agent Platform release notes
- Introducing Gemini 4 Carbon
- Claude Platform release notes
- Effort
- Reasoning models
- Using the latest model
- Deprecations
- Gemini thinking
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner