Build · Topic 05 of 14
Knowledge and retrieval for AI agents: how agents find company data with the right permissions
Knowledge and retrieval is how an AI agent looks things up in your own content (documents, wikis, tickets, databases) before it answers or acts. The basic pattern is retrieval-augmented generation (RAG): find the relevant passages, put them in the prompt, answer with citations. In an enterprise the retrieval must also respect each user’s permissions.
In this guide: 83 pages
Key takeaways
- Retrieval quality bounds answer quality: a strong model answering from the wrong passage is still wrong. Chunking, hybrid search and reranking are where most of the gain is; Anthropic’s contextual retrieval cut top-20 retrieval failures by 49%, and 67% with reranking, in its own tests.
- Agentic retrieval (the model plans several searches, runs them and checks it has enough) ships on both big clouds with limits: Azure’s GA API covers minimal, extractive retrieval only, with LLM query planning still preview; Bedrock’s runs on managed knowledge bases only (no preview label, GA date unverified).
- Permission-aware retrieval is a filter, and AWS says so in writing: Bedrock’s ACL-aware filtering “does not constitute true authorization”. Your application authenticates the user; the source system stays the authority.
- Foundry IQ knowledge bases were generally available on 2 Jun 2026 with some sources still preview. Permissions you fix in SharePoint before indexing matter more than any product.
- Measure retrieval separately from the answer, or you cannot tell a search failure from a generation failure.
Why does retrieval matter more than the model?
Retrieval matters more because the model was not trained on your contracts, tickets or procedures, and a stronger model answering from the wrong passage is still wrong. What the index returns, in what order, with what permissions, decides whether the answer is right and safe to show, with a citation.
Two failure types come from retrieval. Wrong content: a chunk that lost its document context, a keyword that vector search missed (product codes, names and dates are where keyword search wins), or fifty chunks with no reranking, so the useful one sits in the middle of the window. Forbidden content: Microsoft 365 Copilot and every permission-aware agent return whatever the user can already open, so a SharePoint site shared with Everyone except external users becomes an answer. The fix is governance before indexing, and one Copilot licence unlocks the SharePoint Advanced Management reports and reviews that do it.
The published deployments are mostly knowledge search: Microsoft reports a Teams app giving 35,000 Wells Fargo bankers guidance on 1,700 internal procedures, with response times down from 10 minutes to 30 seconds (Microsoft, 24 Jul 2025, vendor story). We have built this layer many times; the scanner we built for a European pharmaceutical regulator retrieves the applicable regulation, approved and on-hold claims, dossiers, previously approved materials and training corrections from Azure AI Search before any model forms a verdict, and the example below shows the controls.
How does RAG work in an enterprise?
RAG has an ingestion side and a query side. Ingestion parses content, splits it into chunks, turns chunks into embeddings and stores them in an index with metadata and, in enterprise systems, the document’s access control list. The query side finds the best chunks with hybrid search, reranks them and passes the top few to the model with citations.
- Parse and chunk. Bedrock Knowledge Bases’ default chunking is about 300 tokens, keeping sentences whole, with fixed-size, hierarchical, semantic and no-chunking options; OpenAI vector stores default to 800-token chunks with a 400-token overlap. Anthropic’s contextual retrieval adds a short, chunk-specific context before embedding and BM25 indexing; it cut top-20 retrieval failures by 49%, and 67% with reranking (19 Sep 2024).
- Embed and index, with permissions. The index stores vectors, metadata and the ACL: Azure AI Search holds synchronised ACL, RBAC and Purview label metadata; Bedrock managed knowledge bases ingest allow and deny lists per document; Amazon S3 Vectors (GA 2 Dec 2025) is a low-cost vector store for Bedrock.
- Search two ways and merge. Keyword search (BM25) finds exact matches such as product codes; vector search matches meaning. Hybrid search runs both and merges with reciprocal rank fusion.
- Rerank. A second, more expensive model re-scores the top results before they reach the LLM: Azure AI Search semantic ranker, Bedrock reranker models (since 1 Dec 2024) and a managed reranker inside Bedrock managed knowledge bases at no extra cost.
- Answer with citations. The model gets the top chunks, the question and an instruction to cite. Every answer that cannot point at a passage is a failure to log, whether retrieval missed the evidence or the model failed to cite it.
- Structured data takes a different route: generate SQL. Bedrock Knowledge Bases connect to structured stores and generate SQL (since 4 Dec 2024); Microsoft’s Fabric IQ models business data as ontologies and semantic models for agents (preview as a Foundry knowledge source).
Evaluate the two sides separately. Retrieval quality with labelled relevance judgments (NDCG, fidelity and holes in Foundry’s Document Retrieval evaluator); groundedness and relevance of the answer with an LLM judge; completeness against a ground-truth answer. Amazon Bedrock evaluations run retrieve-only and retrieve-and-generate jobs and accept results from external RAG systems.
What is agentic retrieval, and when do you need it?
Agentic retrieval is retrieval where a model plans the search: it breaks the question plus chat history into subqueries, runs them in parallel, reranks and merges the results, checks whether it has enough and, if not, searches again or fetches the full document. You need it for multi-part questions over several sources; a lookup does not need it.
On Azure, agentic retrieval lives in Azure AI Search knowledge bases (renamed from knowledge agents in the 2025-11-01-preview API) and is what Foundry IQ runs. The GA REST API 2026-04-01 covers minimal, extractive retrieval over search index, blob, OneLake and web sources; LLM query planning, answer synthesis, higher reasoning effort and multi-turn messages remained preview in 2026-08-01-preview on 2 Nov 2026, and objects created in the portals may need migrating.
On AWS, Bedrock fully managed knowledge bases (2 Mar 2026) run agentic retrieval through the AgenticRetrieveStream API: planning, retrieval, full-document expansion, then an answer with citations. It works on managed knowledge bases only, supports only the BLOCK guardrail action, can restore sessions from AgentCore Memory (14 Aug 2026) and takes reviewed, versioned retrieval instructions (19 Oct 2026). On Google, Agent Search and RAG Engine (renamed from Vertex AI Search and Vertex AI RAG Engine on 22 Apr 2026) are the equivalents; we did not verify their feature-level GA status.
Cost is the trade-off: more subqueries, more reranking and more planning tokens per question. Microsoft’s own worked example puts 2,000 agentic retrievals at about USD 4.32 at illustrative rates (Microsoft Learn, agentic retrieval overview, 16 Sep 2026). The agentic RAG vs RAG sub-page sets out when the extra cost pays.
What is permission-aware retrieval, and why is ACL filtering not authorization?
Permission-aware retrieval (security trimming) means the agent returns only passages the person asking is allowed to open. It is implemented as a filter on the index, and a filter is only as good as the identity you pass it and the permissions you synced, which is why AWS writes that its ACL-aware filtering “does not constitute true authorization”.
Four ways to make retrieval permission-aware, as implemented on 2 Nov 2026.
| Pattern | How it works | Where | Watch for |
|---|---|---|---|
| String filters | Store user and group IDs on each document and filter at query time | Any search API; Azure AI Search with any API version; OpenAI vector stores through attributes | Your code builds the filter; a bug returns everything |
| Native identity enforcement | The index stores synced ACL, RBAC or label metadata; the query carries the user’s Entra token and the service trims results | Azure AI Search (ADLS Gen2 ACLs, blob RBAC scopes, SharePoint ACLs, Purview labels, preview); Foundry IQ under the caller’s Entra identity | Changes appear after the next sync; label enforcement is preview, single-tenant and must be projected onto every chunk |
| Crawl plus real-time check | Allow and deny lists ingested per document, filtered by the email in userContext; seven connectors re-check the source at query time | Bedrock managed knowledge bases; the check fails closed | Email is the only identifier and a mismatch fails silently; large groups lag after a sync; S3 and custom sources get no real-time check |
| Retrieve in place | Reuse an index that already enforces identity instead of copying content | Microsoft 365 Copilot Retrieval API: permission-trimmed chunks from SharePoint, OneDrive and Copilot connectors; 25 results per call | Included with a Copilot licence; pay-as-you-go for unlicensed users was preview |
AWS’s own responsibility list is the right design for every cloud: authenticate the user upstream, pass a verified identity, keep identities consistent across systems, and handle reassigned email addresses before they reach the index. The permission-aware retrieval sub-page (second release) will go through each pattern with code.
RAG vs knowledge graphs vs fine-tuning vs agent memory: which do you need?
RAG for most company knowledge, a knowledge graph when questions span the whole corpus or follow relationships, fine-tuning almost never for knowledge, and memory for what the agent learned from its own users. They are complements with different update cycles, and the first decision is which one your question type needs.
Four ways to give an agent knowledge.
| Approach | Best for | Update cycle | Products on 2 Nov 2026 | Risk |
|---|---|---|---|---|
| RAG (vector and hybrid) | Specific questions answered by a few passages | Re-index on change | Foundry IQ; Bedrock Knowledge Bases; Copilot connectors; Agent Search and RAG Engine | Oversharing; chunking; poisoning of the corpus |
| Knowledge graph and GraphRAG | Whole-corpus questions and relationship chains, answered from extracted entities and community summaries | Graph rebuild on change; expensive | Microsoft GraphRAG (open source); Bedrock Knowledge Bases GraphRAG with Neptune Analytics (GA 7 Mar 2025) | Extraction errors compound; cost |
| Fine-tuning | Style, format and domain language; rarely facts | Retrain | Model vendors’ fine-tuning services | Facts go stale inside the weights; no citations; no per-user permissions |
| Agent memory | What this user told the agent before | Written by the agent, per session | AgentCore Memory (GA); Memory Bank (GA); Foundry memory (preview) | Scope leaks; erasure; poisoning |
The knowledge graphs sub-page (second release) covers when a graph earns its cost. The agent memory topic covers the last row.
How do Microsoft, AWS and Google package retrieval for agents?
Each cloud now sells a managed knowledge layer that an agent connects to, with permissions synced from the sources, and each renamed or re-tiered it during 2026. The table lists the parts and their status as the vendor pages showed on 2 Nov 2026.
Retrieval products by vendor, with status on 2 Nov 2026.
| Vendor | Product | What it is | Status |
|---|---|---|---|
| Microsoft | Foundry IQ | Managed knowledge layer on Azure AI Search; one knowledge base shared by many agents; ACL sync, Purview labels, queries under the caller’s Entra identity | Knowledge bases GA 2 Jun 2026 with an SLA; Serverless and the Work IQ, Fabric IQ, File Search, Azure SQL and MCP sources preview; the portals give preview-only access |
| Microsoft | Work IQ and Fabric IQ | Work IQ: a context layer over Microsoft 365 with a CLI and MCP server; Fabric IQ: ontologies and semantic models over OneLake | Work IQ CLI and MCP server GA 16 Jun 2026; both preview as Foundry knowledge sources |
| Microsoft | Microsoft 365 Copilot connectors (until recently documented as Microsoft Graph connectors) | 100+ prebuilt synced connectors, custom connectors, and federated MCP connectors that fetch live data without indexing | Available; write actions for federated connectors noted from early October 2026, rollout not checked |
| AWS | Amazon Bedrock Knowledge Bases, managed | AWS-managed store, embedding and reranker at no extra cost, seven connectors, ACL-aware and agentic retrieval, an AgentCore Gateway (MCP) target | Added to the document history on 2 Mar 2026; no preview label on 2 Nov 2026, GA date unverified; the type AWS recommends |
| AWS | Amazon Bedrock Knowledge Bases, customer-managed | Bring your own vector store (OpenSearch, Aurora, Neptune, S3 Vectors); GraphRAG with Neptune Analytics; SQL over structured data | GA; GraphRAG GA 7 Mar 2025; S3 Vectors GA 2 Dec 2025; multimodal retrieval GA 30 Nov 2025 |
| Agent Search, RAG Engine, Agent Retrieval | Google renamed Vertex AI Search, Vertex AI RAG Engine and Vector Search 2.0 to these names on 22 Apr 2026 | RAG Engine GA in some Regions, allowlist or preview in others; Agent Search status not checked |
Two things changed in 2026 that architecture documents written in 2025 will have wrong: Amazon Q Business closed to new customers in favour of Amazon Quick (the dates were not verified), and Azure’s knowledge agents became knowledge bases. Date every diagram.
How do Microsoft, AWS and Google implement it?
The same capability on each cloud, with statuses as the vendor pages showed on 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Managed knowledge base | Foundry IQ knowledge bases (GA 2 Jun 2026) on Azure AI Search | Bedrock managed knowledge bases (in the docs since 2 Mar 2026, no preview label, GA date unverified); customer-managed with your own vector store | RAG Engine (GA in some Regions); Agent Search |
| Agentic retrieval | GA for minimal extractive retrieval (API 2026-04-01); LLM query planning, answer synthesis and multi-turn preview | AgenticRetrieveStream on managed knowledge bases: plan, retrieve, expand to full document, answer with citations; BLOCK guardrails only | Not verified at feature level in this review |
| Permission-aware retrieval | Synced ACLs, RBAC scopes and Purview labels enforced under the caller’s Entra token (label enforcement preview); Copilot Retrieval API trims by identity | ACL-aware filtering by email plus real-time checks on seven connectors; fails closed; “does not constitute true authorization” | Not verified in this review |
| Connectors | 100+ Copilot connectors, synced or federated through MCP; Foundry IQ sources: blob, SharePoint, OneLake, web | Seven managed connectors; S3 and custom sources with your own ACL metadata | Not detailed here |
| Knowledge graphs | Microsoft GraphRAG, open source (v3.3.0 on GitHub) | Bedrock Knowledge Bases GraphRAG with Neptune Analytics (GA 7 Mar 2025) | Not verified |
| Retrieval evaluation | Foundry Document Retrieval evaluator (NDCG, fidelity, holes); groundedness and relevance evaluators | Bedrock RAG evaluation: retrieve-only and retrieve-and-generate jobs; external RAG results accepted | Not verified |
Google renamed its retrieval line on 22 Apr 2026 and our research did not check its feature-level statuses page by page, so several Google cells say so rather than guess. Ignite and re:Invent in late 2026 will move several Microsoft and AWS rows.
When should you not build retrieval, and what goes wrong?
Do not build retrieval over content whose permissions you have not reviewed; the agent will publish the oversharing faster than any audit would have found it. Do not index a system of record you could query directly, and do not reach for agentic retrieval when a single hybrid search answers the question.
Oversharing exposed by AI
The agent returns what the user can already open. Run the SharePoint Advanced Management reports, Restricted Content Discovery and site access reviews first; Restricted Content Discovery on sites over 500,000 items can take more than a week to propagate.
Permission sync lag
Query-time trimming uses the ACLs in the index. Azure needs an explicit refresh for SharePoint changes inherited from a parent; Bedrock group membership is as fresh as the last sync and large groups can lag after it.
ACL-aware mistaken for authorization
Bedrock filters by the email you pass and cannot verify it; a mismatched email fails silently and returns nothing; a reassigned address returns the wrong person’s results. Authenticate upstream and manage the identity lifecycle.
Label and preview gaps
Azure’s Purview label enforcement is preview, single-tenant and needs RBAC; labels must be projected onto every chunk or chunk-level results are not filtered.
Poisoned corpus and injected documents
PoisonedRAG reports a 90% attack success rate from five crafted texts per target question in a database of millions. Retrieved documents can carry instructions: CVE-2025-32711 (Microsoft 365 Copilot, CVSS 9.3, 11 Jun 2025) disclosed information that way.
Bad chunking, no reranking, unmeasured retrieval
Chunks that lost their document context are not found; fifty unranked chunks bury the right one in the middle of the window; and if you only score final answers you cannot tell a retrieval failure from a generation failure.
Preview drift and renamed products
Portal-created Azure objects use preview APIs; Bedrock agentic retrieval is managed-only with BLOCK guardrails; in 2026 Q Business closed to new customers, Bedrock Agents became Bedrock Agents Classic, and Vertex AI was renamed Gemini Enterprise Agent Platform.
What should you set up?
The controls that make a retrieval layer safe for an agent to use, in the order we install them.
- A permissions review of every source before indexing: SharePoint Advanced Management reports, Restricted Content Discovery as a brake, site access reviews, Purview labels on the sensitive sets.
- One knowledge base per audience, shared by the agents that serve it, with the sources listed in its registry entry.
- Chunking chosen per content type and tested: sentence-preserving defaults for prose, hierarchical for long documents, contextual retrieval where failures cluster.
- Hybrid search plus a reranker, with the top-k and reasoning effort set by the evaluation score, and citations required in every answer.
- Permission-aware retrieval wired to a verified identity: Entra token on Azure, verified email in userContext on Bedrock, with leavers and reassigned addresses handled upstream.
- A labelled retrieval evaluation set (relevance judgments per query) run on every index, model and chunking change, separate from the answer evaluation.
- Guardrails on retrieved content before it reaches the model: prompt-injection detection on chunks and DLP on what the agent may surface.
- Freshness and retirement rules: indexer schedules, a date on every source, and removal of superseded documents so a withdrawn policy stops being an answer.
Enterprise example · QueryNow
A compliance scanner for a European pharmaceutical regulator, built on retrieval
QueryNow delivery. A production platform for a European pharmaceutical regulator, the national body that polices medicines advertising in its market, scanning marketing assets against national and EU health-claims rules.
The scanner we built for a European pharmaceutical regulator replaced a review that took a human 2 to 3 hours per asset: cross-referencing every claim on a package, advert or post against national and EU regulations, approved ingredient databases and previously approved materials, with different reviewers flagging different issues and no trail from a verdict to its source.
Retrieval is the core of the build. Text and visuals are extracted from each asset, claims are identified, and a hybrid semantic and vector search pipeline over Azure AI Search retrieves the applicable regulatory documents: approved health claims, on-hold claims, regulatory guidelines, training corrections from past reviews, product dossiers and previously approved materials. Only then does the model run a structured check across 11 rule categories, each with its own pass or fail logic, evidence requirement and fix suggestion, and every finding links to the regulatory document that informed it.
The result is a first-pass verdict in about 2 minutes per asset, down from 2 to 3 hours, with 620+ assets scanned in production. The reviewer stays in the loop: flagged rules need reviewer attention, and a one-click submit-for-internal-review step hands edge cases and judgment calls to the regulator’s reviewers. Every scan stores the prompts, model parameters, raw responses, retrieved documents and timestamps.
Two retrieval lessons from the build. Regulations alone are not enough; the index also holds previously approved materials and training corrections from past reviews, which is what makes the system behave like an experienced reviewer. And citations are the trust mechanism: without the link from each finding to the regulation, the output is one more opinion.
- Owner
- The regulator’s reviewers and admins own the compliance rules, the Azure AI Search indexes and user management through the admin panel; QueryNow built the platform
- Identity
- Role-based access: customers upload and see only their own scan results; reviewers and admins see and audit every scan
- Risk tier
- First-pass assistant: the scan is a first pass, flagged rules need reviewer attention, and judgment calls go to the regulator’s reviewers
- Approval
- Submit-for-internal-review workflow and a reviewer dashboard with a status per asset: compliant, review needed, rejected, pending
- Data access
- Retrieval scoped to the regulator’s own index of regulations, claims, dossiers, previously approved materials and training corrections; identical files replayed by SHA-256 fingerprint within a configurable window
- Monitoring
- Full audit trail per scan: prompts, model parameters, raw responses, retrieved documents, confidence scores and timestamps
- Retirement
- Not described in the published case study. Our recommendation for any retrieval build: a date and an owner on every indexed source, and scheduled removal of superseded rules
How do you start?
- Pick one question type and one audience, and list the sources that answer it. If the sources are a database, generate SQL instead of indexing.
- Review permissions on those sources and fix oversharing before the first index run.
- Index with the managed service of your cloud (Foundry IQ, Bedrock managed knowledge bases), hybrid search and a reranker on, citations required.
- Build a labelled retrieval evaluation set of 50 to 100 real questions with the passages that answer them, and score retrieval separately from answers.
- Wire identity through: Entra token or verified email, tested with a user who should see nothing.
- Add agentic retrieval only for the question types the evaluation shows a single search cannot answer, at the lowest reasoning effort that passes.
- Register the knowledge base with an owner, sources, refresh schedule and review date.
Frequently asked questions
What is the difference between RAG and agentic RAG?
RAG runs one search for the question and answers from the top results. Agentic RAG lets a model plan several subqueries, run them in parallel, rerank and merge, check sufficiency and search again or fetch the full document before answering. It costs more per question and helps on multi-part questions over several sources.
Is permission-aware retrieval the same as authorization?
No. It is a filter on the index driven by the identity you pass. AWS states that Bedrock’s ACL-aware filtering “does not constitute true authorization” and that your application must authenticate users; Azure trims by the user’s Entra token. The source system remains the authority, and the sync lag between them is your risk.
Is Foundry IQ generally available?
Knowledge bases were generally available with an SLA from 2 Jun 2026, with Foundry IQ Serverless and several knowledge sources (Work IQ, Fabric IQ, File Search, Azure SQL, MCP) in preview, and the portals still giving preview-only access on 2 Nov 2026. Availability depends on the Azure AI Search REST API version you call.
Do we still need Microsoft Graph connectors for Copilot?
Yes, under their current name: Microsoft 365 Copilot connectors. They either sync external content and its permissions into Microsoft Graph or federate through MCP to fetch live data without indexing. The underlying API is still called the Microsoft Graph connectors API.
When should we use a knowledge graph instead of RAG?
When questions span the whole corpus (which themes recur across all audit findings) or follow relationship chains, which vector search answers poorly. GraphRAG extracts entities and relations, summarises communities and answers by global or local search; Bedrock offers it with Neptune Analytics and Microsoft publishes it as open source. It costs more to build and maintain.
Can an agent’s knowledge base be poisoned?
Yes. PoisonedRAG reports a 90% attack success rate with five crafted texts per target question in a corpus of millions, and retrieved documents can carry prompt injections, as CVE-2025-32711 showed in Microsoft 365 Copilot. Control who can write to indexed sources and screen retrieved chunks before they reach the model.
How do we measure whether retrieval is working?
Separately from the answer. Build labelled relevance judgments for real queries and score the retrieval step with NDCG and similar metrics (Foundry’s Document Retrieval evaluator reports NDCG, fidelity and holes), then score groundedness and completeness of the answer. Bedrock evaluations run retrieve-only and retrieve-and-generate jobs.
In this topic
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Agentic retrieval in Azure AI Search
- Migrate agentic retrieval code to the latest version
- Document-level access control in Azure AI Search
- Hybrid search in Azure AI Search
- What is Foundry IQ?
- What’s new in Microsoft Foundry at Build 2026
- Work IQ CLI
- Microsoft 365 Copilot connectors overview
- Microsoft 365 Copilot Retrieval API overview
- Data, privacy and security for Microsoft 365 Copilot
- SharePoint Advanced Management features in Microsoft 365 Copilot licenses
- Restricted Content Discovery
- RAG evaluators (Foundry)
- Amazon Bedrock Knowledge Bases overview
- Managed and customer-managed knowledge bases
- Access Control Lists awareness enablement (Bedrock managed knowledge bases)
- Amazon Bedrock Knowledge Bases agentic retrieval
- Chunking in Amazon Bedrock Knowledge Bases
- Amazon Bedrock document history
- Amazon Bedrock Knowledge Bases GraphRAG now generally available
- Multimodal retrieval in Amazon Bedrock Knowledge Bases
- Amazon S3 Vectors now generally available
- Amazon Q Business availability change
- Evaluate RAG in Amazon Bedrock
- Gemini Enterprise Agent Platform release notes
- RAG Engine overview
- Retrieval and vector stores
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization, arXiv 2404.16130
- GraphRAG releases
- Introducing Contextual Retrieval
- Lost in the Middle: How Language Models Use Long Contexts, arXiv 2307.03172
- PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation, arXiv 2402.07867
- CVE-2025-32711
- AI-powered success, with more than 1,000 stories of customer transformation
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner