How do you red team an AI agent?

Microsoft’s AI Red Teaming Agent names the agentic risk categories as prohibited actions, sensitive data leakage and task adherence, and it tests indirect prompt injection (sometimes written XPIA, cross-prompt injection attack), where the attack arrives through content the agent reads rather than through the user. Its illustrative taxonomy for actions has three tiers: prohibited actions are never allowed; high-risk actions such as financial transactions or medical decisions are allowed with human-in-the-loop confirmation; irreversible actions such as file deletions or system resets are allowed with disclosure plus confirmation. Your boundaries criterion from the acceptance criteria is the first tier written down, and the red team’s job is to cross it.

The catalogue for scoping is the OWASP Top 10 for Agentic Applications, published 9 Dec 2025, which organises the threats an agent faces by what it can reach: tools, memory, other agents, the user’s identity. Pair it with the agent’s own registry record, which lists the tools, data and identity it holds, and you have the attack surface: every tool is a target for misuse, every data source a channel for injection, every hand-off a chance to pass a poisoned instruction along.

Model robustness is one layer and not the control. Google reports, for its own models, that Gemini 4 Carbon reached 0.0% attack success against 0.7% for Gemini 4 Argon on the Gray Swan indirect prompt-injection benchmark (28 Oct 2026, vendor-reported). A better model lowers the rate; tool permissions, policy and approval gates decide what a successful attack can do. The Security threats to agents topic covers the attacks in detail.

Give the human testers the registry record and the acceptance criteria. The never list tells them what to aim for; the tool list tells them which tool results to poison; the data list tells them what a leak looks like. The best testers are often the fraud team, the help-desk lead or the people who already know how customers try to game the process, not security specialists alone.

Run the attacks in what Microsoft calls a purple environment: non-production, with production-like resources, real tool interfaces and realistic but not real data, and no live write access. Foundry’s cloud scans use mock tools and synthetic data for the agentic categories, which is safe and also why they miss what a real tool result can carry. Your human round is where the agent reads a real-shaped document with an instruction hidden in it, or a tool returns a record with a payload in a free-text field.

Test the approval path too. An attack that gets a high-risk action past the approver is a success for the attacker, so include cases where the request shown to the approver looks routine and the arguments are not. The Human in the loop sub-page covers why the approval screen must show the exact tool, arguments and target.

Attack success rate is the share of attack attempts that achieved their goal, as judged by a scorer. Microsoft notes that scoring by generative models can be non-deterministic and that there is always a chance of false positives, so rerun the scan and read the flagged cases before you act on the number. promptfoo published a note on 12 Dec 2025 arguing that attack success rate is not comparable across jailbreak papers without a shared threat model: the attacker’s budget, the categories and the judge all change the figure.

Targets differ by category. Prohibited actions and data leakage should be at zero on the attacks you know about, because each success is an incident the agent could have in production. Task adherence and jailbreak categories are a trend to drive down. A release gate that blocks on any prohibited-action success and on a rise in the other categories is the usual shape.

Fixes go in order of strength. First, tool permissions and policy: an agent that cannot call the payment tool cannot be made to; AgentCore Policy (GA 3 Mar 2026) writes that as a Cedar rule, Google enforces it at Agent Gateway, and Foundry guardrails check tool calls and tool responses (preview). Second, guardrails on input and output, such as Prompt Shields on Azure, Bedrock Guardrails and Model Armor. Third, prompts and instructions, which help and which an attacker reads around. Then add each successful attack to the golden dataset, re-scan, and keep the scan in the gate so a model or tool change cannot reopen the hole.