Govern and secure · Topic 10 of 14
What are guardrails for AI agents, and how do you set them up?
Guardrails for AI agents are checks that sit outside the model and inspect what goes in, what the agent does with tools and what comes out. They block, redact, flag or hand off to a person when a rule is broken. Some are deterministic rules; others are classifiers that estimate risk. Neither kind is enough alone.
In this guide: 83 pages
Key takeaways
- Guardrails are enforced outside the model at fixed points: a filter on input, a policy on every tool call, a check on tool results and a filter on output, plus limits on rate, spend and egress, and a person before irreversible actions.
- Deterministic rules hold for the cases they cover when implemented correctly. Classifiers reduce risk and get bypassed: in October 2025 adaptive attacks beat 12 published defenses with success above 90% for most.
- All three clouds moved guardrails to the tool-call boundary in 2025 and 2026: AgentCore Policy (generally available 3 Mar 2026), Foundry tool call and tool response points (preview), Copilot Studio external threat detection (preview), Model Armor on Agent Gateway (generally available 24 Jun 2026).
- The defaults need changing. Copilot Studio external threat detection allows a tool call if the provider has not answered within one second and is set per environment. Bedrock's prompt-attack filter does not evaluate tool results, and its PII masking does not reach invocation logs. A Foundry agent guardrail fully overrides the model's.
Why do AI agents need guardrails?
Agents need guardrails because they act with real credentials on content nobody has checked. A chatbot that reads a poisoned email produces a bad answer; an agent that reads the same email can send data out, change records or call a tool, and the model cannot tell its user's instruction from one hidden in a document.
The 2025 incidents made the pattern concrete. Microsoft had fixed EchoLeak (CVE-2025-32711) by its disclosure in June 2025, a chain in which one crafted email made Microsoft 365 Copilot leak data with no click from the user; it got past the injection classifier and used auto-fetched images. Noma Labs reported ForcedLeak in Salesforce Agentforce in September 2025: instructions in a web-to-lead field ran when an employee asked about the lead, and data left through an image URL on an expired but allow-listed domain. Each time, a deterministic control on the exit path closed the hole, and Microsoft's write-up of 29 July 2025 says its defense "does not rely on our ability to block all prompt injections".
The other two reasons are the limit of the classifiers, covered below, and the loop itself. An agent in a loop can spend a budget, call a tool a thousand times or send a thousand emails before anyone looks. Rate limits, spend limits and approval thresholds are guardrails too, and need no model to enforce.
How do guardrails work, layer by layer?
Guardrails work as a series of checks at fixed points on the request path, each with a defined action: block, redact, annotate or hand to a person. We set up eight layers, in roughly this order.
- Input filters on every untrusted channel, not only the chat box: a prompt-attack classifier on the user prompt and a document-attack check on retrieved documents, emails, web pages and tool results. Azure Prompt Shields take the two in separate fields and, for Foundry agents, can scan at the tool response point (preview). Bedrock requires input tags around user content with InvokeModel; untagged input is not filtered.
- Content safety on input and output. All three clouds ship harm-category checks; denied topics, word lists, protected-material detection, thresholds and language coverage differ by service.
- PII and secrets redaction in both directions: Bedrock sensitive information filters block or mask PII types and custom regex, Model Armor uses Sensitive Data Protection templates, and Defender's real-time agent protection blocks secret exfiltration on tool calls.
- Grounding checks on answers. Bedrock contextual grounding checks and Automated Reasoning checks (generally available 6 Aug 2025, detect only, English US) test an answer against sources or a formal policy. Azure groundedness detection is preview and retires on 30 April 2027, so build that check elsewhere.
- A tool-use policy outside the model: a deterministic engine that allows or denies each tool call from the caller's identity and the arguments. AgentCore Policy (generally available 3 Mar 2026) evaluates Cedar rules on every request through an AgentCore Gateway in log-only or enforce mode; the AWS launch example lets a tool that returns money to a customer run only for a caller with a matching role claim and an amount under a set limit. Foundry's tool call point and Task Adherence (both preview) flag a planned call that does not match the request.
- Egress and rendering allow lists: network egress controls for Foundry hosted agents (preview), unsafe email domain blocking in Defender, trusted URL lists for images and links. This layer closed the 2025 image-URL leaks.
- Rate and spend limits at a gateway. Azure API Management's llm-token-limit policy sets tokens per minute and quotas per period, keyed by subscription, IP address or any expression. No per-agent spend cap was confirmed on AgentCore on 2 November 2026; AWS Budgets and CloudWatch alarms monitor spend, and a hard stop needs a request or token limit enforced elsewhere.
- Human approval for irreversible or outbound actions. Microsoft lists consent before sending a drafted email as a core mitigation, and Meta's Agents Rule of Two (31 October 2025) says an agent with untrusted input, private data and an outbound channel in one session should not run without supervision.
Deterministic rules or classifiers: which guardrail goes where?
Use a deterministic rule wherever the condition can be written down (tools, arguments, domains, amounts, rates) and a classifier wherever it cannot (is this text an attack, does it contain a name). The two fail in different ways, so pair them.
The kinds of guardrail, how each fails and where each belongs.
| Kind | Examples | How it fails | Where it belongs |
|---|---|---|---|
| Deterministic rule | Tool allow list; Cedar policy on arguments; egress allow list; token quota; approval threshold | Only on cases it was never told about; or on an implementation defect; for the cases it covers it holds | Tool calls, egress, spend, approvals |
| Classifier | Prompt Shields; Bedrock prompt attack filter; Model Armor injection detection; PII detectors; Task Adherence | Probabilistically: adaptive attacks have bypassed published defenses, accuracy drops outside English, false positives | User input, documents, tool results, output text |
| Formal check | Bedrock Automated Reasoning checks against a policy you upload | Detect only; English (US) only; adds latency; may skip encoded text | High-stakes answers against written rules |
| Human gate | Approval before send, pay or delete; Copilot Studio multistage approvals (preview) | Approval fatigue once users click Always allow | Irreversible and outbound actions |
The 2025 design research points the same way: Google DeepMind's CaMeL paper (24 March 2025) separates control flow from untrusted data and solved 77% of AgentDojo tasks with provable security, against 84% undefended, and a 14-author paper of 10 June 2025 (Beurer-Kellner, Buesser, Creţu and others) names six patterns that limit what an injected instruction can reach.
Guardrails vs identity, evaluation and governance: what does each one do?
Guardrails decide at run time whether one request, tool call or answer passes. Identity decides what the agent may reach at all. Evaluation measures behaviour before release and after changes. Governance decides who signs for which risk. A gap in any of the four shows up as a guardrail incident, so if a guardrail keeps blocking a call the identity should never have allowed, fix the identity.
Four neighbouring controls.
| Control | Question it answers | When it acts | Example |
|---|---|---|---|
| Guardrails | May this input, tool call or output pass? | Every request, in line | AgentCore Policy denies a payment above the limit |
| Identity and access | What can this agent reach, and for whom? | When tokens are issued | Microsoft Entra Agent ID with scoped permissions; AgentCore Identity |
| Evaluation | Does the agent behave as agreed, on a test set? | Before release; after model, prompt or tool changes | Foundry evaluators; AgentCore Evaluations (generally available 31 Mar 2026) |
| Governance | Who owns the risk and the approval? | At intake and review | A registry record with owner, risk tier and approval state |
How do Microsoft, AWS and Google implement guardrails?
All three ship content filters, prompt-attack detection and PII handling, and all three added a control at the tool-call boundary in 2025 or 2026. The statuses are as the vendor pages showed on 2 November 2026; the sub-page on guardrails on Azure, AWS and Google goes product by product.
Guardrail products by cloud, as seen 2 Nov 2026.
| Vendor | Text guardrails | Tool-call control | Status seen 2 Nov 2026 |
|---|---|---|---|
| Microsoft | Azure AI Content Safety: Prompt Shields (generally available since Aug 2024), harm categories, protected material; Microsoft Foundry guardrails and controls | Foundry tool call and tool response intervention points; Task Adherence; Copilot Studio external threat detection; Defender real-time protection | Foundry agent guardrails preview (20 Oct 2026); Copilot Studio detection preview (1 Jun 2026); Defender coverage of Copilot Studio and Foundry agents preview (1 Jul 2026) |
| AWS | Amazon Bedrock Guardrails: content filters including prompt attack, denied topics, word filters, sensitive information filters, contextual grounding, Automated Reasoning checks | AgentCore Policy: Cedar rules evaluated at an AgentCore Gateway, log only or enforce | Guardrails generally available; Automated Reasoning 6 Aug 2025; Policy 3 Mar 2026 in 13 Regions; organization-level enforcement documented, launch date not stated |
| Model Armor: prompt injection and jailbreak, responsible-AI filters, Sensitive Data Protection, malicious URLs; floor settings per project | Model Armor on Agent Gateway and on Google-managed MCP servers | Core service since 3 Feb 2025 with no separate GA label; MCP integration 22 Apr 2026; Agent Gateway integration 24 Jun 2026; organization policies requiring filters 21 Oct 2026 |
Independent options run on any cloud: NVIDIA NeMo Guardrails (open source), Meta's Llama Guard 4 and Prompt Guard 2 classifiers (29 April 2025), and the runtime products that consolidated in 2025 and 2026 (Lakera into Check Point, Protect AI and Koi into Palo Alto Networks, CalypsoAI into F5).
Can a classifier stop prompt injection on its own?
No. The October 2025 adaptive-attack research and the vendors' own documentation both say so, which is why a classifier belongs inside a design that assumes some injections get through.
"The Attacker Moves Second" (Nasr, Carlini, Sitawarin and co-authors, 10 October 2025) evaluated 12 recent defenses and bypassed them with attack success above 90% for most, while the majority had reported near-zero attack success. An April 2025 study by Hackett, Birch and co-authors tested six protection systems, naming Azure Prompt Shields and Meta's Prompt Guard, and reached up to 100% evasion success in some instances.
The vendors say the same. Microsoft's MSRC post of 29 July 2025 calls hardened system prompts, Spotlighting and Prompt Shields probabilistic mitigations and keeps deterministic blocks in a separate layer that works even when an injection succeeds; Meta's Llama Guard 4 model card notes the guard model may itself be vulnerable to prompt injection.
Two design rules follow. Simon Willison's lethal trifecta (16 June 2025): an agent with access to private data, exposure to untrusted content and the ability to communicate externally can be made to leak the data, so remove one of the three. Meta's Agents Rule of Two says the same as policy. In practice that means splitting agents by trust boundary, pinning tool lists and putting approval on the outbound step.
How do Microsoft, AWS and Google implement it?
The same guardrail capability, named the way each cloud names it, with statuses as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Prompt-attack and injection detection | Prompt Shields: user prompt attacks (on by default) and document attacks (off by default); an October 2026 classifier update rolling out from Spain Central | Bedrock Guardrails prompt attack filter; prompt leakage detection on the Standard tier; needs input tags with InvokeModel | Model Armor prompt injection and jailbreak detection; inputs under three words return no match |
| Tool-call policy | Foundry tool call intervention point (preview); Task Adherence (preview); Copilot Studio external threat detection (preview) | AgentCore Policy, Cedar rules at the Gateway (generally available 3 Mar 2026) | Model Armor on Agent Gateway (generally available 24 Jun 2026) |
| Tool results scanned | Foundry tool response intervention point (preview); Defender real-time protection for Foundry agents (preview) | Neither the prompt attack filter nor the PII filter evaluates tool results; call ApplyGuardrail on them yourself | Model Armor on Google-managed MCP servers (generally available 22 Apr 2026) |
| PII and secrets | PII control in Foundry (preview); Purview DLP for Microsoft 365 Copilot; Defender secret exfiltration block | Sensitive information filters block or mask on input and output; masking does not reach invocation logs or the trace match field | Sensitive Data Protection, basic or advanced templates; inspect only or inspect and block |
| Grounding and output checks | Groundedness detection (preview, retires 30 Apr 2027, not for agents); protected material for text and code | Contextual grounding checks; Automated Reasoning checks (generally available 6 Aug 2025, detect only) | Responsible-AI filters; malicious URL detection on the first 256 URLs |
| Organization-wide enforcement | Guardrail assigned per model deployment or agent; Defender custom rules; no tenant-wide switch for Copilot Studio external detection | AWS Organizations Bedrock policies and account-level enforced guardrails; the most restrictive setting wins | Floor settings per project; custom organization policies requiring filters (generally available 21 Oct 2026) |
| Rate and spend limits | API Management llm-token-limit: tokens per minute and quotas per period | Per-agent spend cap unconfirmed on the pages read; AWS Budgets and CloudWatch alarms | Unconfirmed on the pages read |
Google's release notes show no separate GA label for Model Armor's core service, so this page gives its introduction date and the GA dates of its integrations.
When should you not add a guardrail, and what goes wrong?
Do not add a classifier where a rule will do: an allow list of five tools needs no model to police it. Do not let a filter stand in for identity scoping. Once guardrails run, these are the documented ways they fail.
Fail-open defaults
Copilot Studio external threat detection allows the tool call if the provider has not answered within one second, unless the error behaviour is set to Block the query. Run a new policy log-only first, then enforce with the fail-closed setting.
Configuration that does not propagate
Copilot Studio detection is set per environment, with no tenant-wide switch, PowerShell or API to enforce it, so a new environment starts unprotected. A Foundry agent with no custom guardrail inherits its model deployment's, and an assigned agent guardrail fully overrides the model's. On AWS a guardrail attached higher in the organization tree is not added to one attached lower.
Channels the filter never sees
Bedrock's prompt attack filter does not evaluate toolResult content or tool definitions, and untagged InvokeModel input is not filtered. Foundry scans tool calls and tool responses only when those intervention points are configured; Spotlighting and groundedness do not apply to agents. Defender for Cloud's AI threat protection scans text tokens only.
Redaction that stops short of the logs
Bedrock PII masking applies to prompts and model responses. The input field in model invocation logs keeps the original request, the trace match field holds the raw value, and PII in tool call arguments, tool results and tool definitions is neither blocked nor masked. Use CloudWatch log data protection and scan tool traffic separately.
Bypass, false positives and language gaps
Adaptive attacks beat 12 defenses in the October 2025 study. Prompt Shields are trained mainly on English, and Automated Reasoning is English (US) only. Microsoft's October 2026 classifier update targets the balance between detection and unnecessary filtering.
The guard's own data path
Copilot Studio sends the prompt, chat history, prior tool outputs and the proposed tool call to the external provider, which may process data outside your region under its own data handling terms. Task Adherence may process data in US or EU regions.
What should you set up?
The guardrail set we install for an agent that reads untrusted content and calls tools, in order.
- A trust map: which inputs are untrusted (email, tickets, web pages, tool results, MCP tool descriptions), which data is private, which actions are outbound or irreversible. If one agent has all three, split it or gate it.
- Input tagging and a prompt-attack classifier on the user prompt, plus a document-attack check on every retrieved document and tool result.
- A deterministic tool policy outside the model: an allow list of tools per agent, argument limits (amounts, record counts, recipients) and caller identity checks, run log-only for a week before enforce.
- Egress and rendering allow lists: approved domains for fetches and links, no auto-fetched images from untrusted sources, no outbound email to unlisted domains.
- PII and secrets handling in both directions, with log data protection on the invocation logs and a scan on tool traffic.
- Rate, token and spend limits at the gateway, keyed per agent identity, with an alert before the cap and a kill switch after it.
- Human approval on irreversible or outbound actions, with the approval request showing the exact action and arguments, and a cap on approvals per person per hour.
- Fail-closed error behaviour, central enforcement (organization policies, floor settings, environment groups), an injection test suite run before every model or prompt change, and a review date for every preview feature.
Enterprise example · Worked example · illustrative
A 4,000-person distributor puts guardrails on a customer-credit agent
Illustrative scenario. A building-products distributor with 4,000 staff, a customer-service agent on Amazon Bedrock AgentCore that answers order questions and issues account credits, tools reached through an AgentCore Gateway, and an order system behind an API.
The agent read three untrusted inputs (customer emails, the notes field in the order system, attachments), held private data (order history, addresses, partial card numbers) and could act outward (issue a credit, email the customer). That is the full trifecta, so the team weakened one leg before writing a single filter: outbound email went behind an approval, and credits got a hard ceiling.
Bedrock Guardrails handled the text: the prompt attack filter on the Standard tier on tagged user input, sensitive information filters masking card numbers and addresses in both directions, and a denied topic that kept the agent off warranty disputes, which a person handles. Because the prompt attack filter does not evaluate tool results, every tool result passed through a separate ApplyGuardrail call before it reached the model. Because PII masking does not reach invocation logs, CloudWatch log data protection masked the same types there.
AgentCore Policy held the rules that must never bend. A Cedar policy allowed the credit tool only for a caller whose token carried the credit-agent role and only for an amount below a set ceiling; anything larger was denied to the agent and routed to a supervisor, who issued it through the finance system under their own credentials. In nine days of log-only mode the policy would have denied 14 calls, 11 of them the agent retrying a credit after a timeout. The team fixed the retry logic and switched to enforce.
The test set caught the rest. Forty injection cases drawn from the 2025 incident write-ups (instructions in an order note, a reply-to address in an email signature, a link to an image on an unlisted domain) ran before launch and after every model change. Egress was limited to the order API, the email service and two document domains. Six months in, the dashboard showed three blocked prompt attacks a week, no credit over the ceiling, and two supervisor approvals a day.
- Owner
- Customer-service operations lead; a business sponsor in finance signs the credit ceiling
- Identity
- AgentCore Identity workload identity; the credit tool needs the credit-agent role claim
- Guardrails
- Bedrock Guardrails on input and output; ApplyGuardrail on every tool result; CloudWatch log data protection on invocation logs
- Tool policy
- AgentCore Policy in enforce mode: amount ceiling and role check, after nine days in log-only mode
- Approval
- Credits above the ceiling and all outbound email go to a supervisor queue showing the exact action and arguments
- Risk tier
- Tier 3 (moves money and sends email); retested on every model or prompt change with the 40-case injection set
- Monitoring
- Interventions, policy denials and approval counts on one dashboard; alert when denials spike
- Retirement
- Workload identity disabled and gateway target removed first, policy engine removed after; guardrail versions and test results archived with the registry record
How do you start?
- Draw the trust map for one agent: untrusted inputs, private data, outbound actions. Decide which leg of the trifecta to remove or gate.
- Turn on the platform's classifiers with blocking on the user prompt and document channels, and change the fail-open and per-environment defaults listed above.
- Write the tool policy as rules a reviewer can read: which tools, which arguments, which caller. Run it log-only for a week.
- Close the exits: an egress allow list, no untrusted image fetches, outbound email behind approval.
- Build a 30 to 50 case injection test set from the public incident write-ups and run it before every model, prompt or tool change.
- Put interventions, denials and approvals on one dashboard with an alert on spikes, and name the person who reads it.
- Review the preview features you depend on after Ignite and re:Invent in late 2026, and before the April 2027 Azure retirements.
Frequently asked questions
What is the difference between guardrails and content filters?
Content filters are one kind of guardrail: classifiers that score text for harm categories. Guardrails as a whole also cover prompt-attack detection, PII handling, tool-call policies, egress limits, rate and spend caps and human approval.
How do you add guardrails to an agentic AI system?
In three places: on the text the agent reads and writes (classifiers and PII filters), on every tool call (a policy engine such as AgentCore Policy or Foundry's tool call point) and on the exits (egress allow lists, approval before outbound actions). Start with the tool policy and the exits.
Do guardrails stop prompt injection?
They reduce it and do not stop it. Adaptive attacks bypassed 12 published defenses in the October 2025 study, at above 90% success for most, and Microsoft says its own defense does not rely on blocking all injections. Design so that a successful injection cannot reach private data and an outbound channel in one session.
Which guardrail defaults should you change on Copilot Studio, Bedrock and Foundry?
Copilot Studio: set external threat detection's error behaviour to Block the query and turn it on in every environment. Bedrock: tag user input, scan tool results with ApplyGuardrail and protect invocation logs. Foundry: assign a guardrail to each agent with the tool call and tool response points configured, because the agent guardrail overrides the model's.
Where should a human approve an agent's action?
Before any action that is irreversible or leaves the company: sending email, paying, deleting, changing permissions. Show the exact action and arguments, and watch for approval fatigue; a user who clicks Always allow has removed the control.
Do guardrails add latency and cost?
Yes. Each service-side check is a call in the request path, while local rules cost little. Automated Reasoning checks add latency that grows with the policy's variables, Spotlighting raises token counts, and API Management token counters are per gateway, so concurrent requests can briefly exceed a limit. Measure the whole path.
In this topic
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Prompt Shields in Azure AI Content Safety
- Guardrails and controls overview in Microsoft Foundry
- What's new in Azure AI Content Safety
- Enable external threat detection and protection for Copilot Studio custom agents (preview)
- Real-time protection for AI agents during runtime
- Threat protection for AI services (Defender for Cloud)
- Azure API Management llm-token-limit policy
- How Microsoft defends against indirect prompt injection attacks
- Detect prompt attacks with Amazon Bedrock Guardrails
- Remove PII from conversations by using sensitive information filters
- Amazon Bedrock Guardrails overview
- Enforce guardrails across accounts (Guardrails enforcements)
- Automated Reasoning checks in Amazon Bedrock Guardrails are now generally available
- Policy in Amazon Bedrock AgentCore is now generally available
- Amazon Bedrock AgentCore adds quality evaluations and policy controls for deploying trusted AI agents
- Model Armor overview
- Model Armor release notes
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections
- Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems
- Defeating Prompt Injections by Design (CaMeL)
- Design Patterns for Securing LLM Agents against Prompt Injections
- The lethal trifecta for AI agents: private data, untrusted content, and external communication
- Agents Rule of Two: a practical approach to AI agent security
- Llama Guard 4 model card
- NVIDIA NeMo Guardrails
- CVE-2025-32711 (EchoLeak)
- ForcedLeak: agent risks exposed in Salesforce Agentforce
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner