- Home
- AI agents
- Knowledge and retrieval
- Agentic RAG vs RAG
Build · Knowledge and retrieval
Agentic RAG vs RAG: what is the difference?
Classic RAG runs one search for the question and answers from the top results. Agentic RAG puts a model in charge of the search: it plans subqueries, runs them in parallel, reranks and merges, checks whether it has enough, and searches again or fetches the full document before answering. It answers harder questions at a higher cost per question.
In this guide: 83 pages
Key takeaways
- Agentic RAG changes the query side only: the index, chunks and permissions are the same, and a weak index stays weak.
- Availability on 2 Nov 2026 came with limits: Azure’s GA covers minimal, extractive retrieval (LLM query planning and answer synthesis still preview); Bedrock’s runs on managed knowledge bases only, with BLOCK guardrails only and no verified GA date.
- The cost is real and settable: more subqueries, reranking and planning tokens per question, with reasoning effort (minimal, low, medium on Azure) as the main dial.
- Decide with an evaluation set: switch on agentic retrieval for the question types a single hybrid search fails, and keep classic RAG for lookups.
How does classic RAG work, and where does it fall short?
Classic RAG embeds the question, runs one search (hybrid keyword and vector in good setups), optionally reranks, and passes the top chunks to the model with the question. It falls short when the question has several parts, needs several sources, depends on the chat history, or needs a whole document rather than a chunk.
The single pass is a feature for most traffic. A policy lookup, a product code, a ticket status: one well-tuned hybrid search with a reranker answers these for the cost of one search, one rerank pass and one model call. Anthropic’s contextual retrieval results (19 Sep 2024) show how much headroom the index itself has: adding chunk context cut top-20 retrieval failures by 49%, and 67% with reranking, with no change to the query side.
Where it breaks: compare our 2025 and 2026 supplier terms for payment deadlines in Germany needs two documents, a filter and a comparison, and one query embedding may land between them. A follow-up such as and for Austria? has no meaning without the previous turn. A question answered only by section 4.2 of a 90-page contract may need the whole document rather than one chunk. These are the cases that drove the 2025 and 2026 releases.
What does agentic RAG add?
Four things: a planning step that turns the question and the chat history into subqueries, parallel execution of those subqueries with reranking and merging, a sufficiency check with a loop that searches again or expands a chunk to its full document, and an activity trace returned with the citations so you can see what the planner did.
Classic RAG and agentic RAG, step by step.
| Step | Classic RAG | Agentic RAG |
|---|---|---|
| Query | The user’s text, embedded as is | A model rewrites it into several subqueries using the chat history |
| Search | One hybrid search | Subqueries run in parallel across one or more knowledge sources |
| Rank | Optional reranker on one result set | Each set reranked, then merged |
| Check | None | Sufficiency check; search again or fetch the full document if a chunk is not enough |
| Answer | Model answers from the top chunks | Extractive grounding with citations, or a synthesised answer (preview on Azure) |
| Trace | Logs if you wrote them | Activity trace with the subqueries, sources and token counts |
| Steering | Filters, ranking weights, query rewriting | Reasoning effort setting; reviewed, versioned retrieval instructions (Bedrock, 19 Oct 2026) |
On Azure this runs inside an Azure AI Search knowledge base, which Foundry IQ exposes to agents; on AWS it is the AgenticRetrieveStream API of Bedrock managed knowledge bases (planning, retrieval, full-document expansion, answer with citations). Bedrock’s version can also restore a session from AgentCore Memory and query long-term memory records (14 Aug 2026), which joins memory and retrieval in one call.
When is agentic RAG worth the cost?
When the evaluation set shows question types that a single search cannot answer and the business value of answering them covers the extra tokens. Multi-part questions, questions across sources and conversational follow-ups qualify; lookups do not.
The cost side has numbers. Each question now spends planning tokens, several searches, several rerank passes and, with answer synthesis, a second model call. Microsoft’s own worked example puts 2,000 agentic retrievals at about USD 4.32 at illustrative rates (Microsoft Learn, agentic retrieval overview, 16 Sep 2026), and lists the levers: fewer knowledge sources per base and lower reasoning effort. Latency rises with each loop, which matters in a chat and matters less in a batch.
The quality side needs your own measurement, because the vendor pages carry no accuracy figures for agentic retrieval that we could verify. Build 50 to 100 real questions with the passages that answer them, score retrieval (NDCG and similar) and the answer (groundedness, completeness) for classic and agentic retrieval, and read the difference per question type. In our builds the gain concentrates in two places: comparisons across documents and follow-up turns in a conversation.
Then route. Keep classic RAG as the default path, send the question types that failed to agentic retrieval at the lowest reasoning effort that passes, and cap the loop count. A router that classifies the question type costs one small model call.
What is generally available, and what is still preview?
On 2 Nov 2026 agentic retrieval was generally available on Azure and AWS with limits that change what you can rely on, and Google’s renamed products were not checked at feature level. The table shows the lines.
Agentic retrieval status by vendor, as the vendor pages showed on 2 Nov 2026.
| Vendor | Generally available | Preview or limited | Notes |
|---|---|---|---|
| Microsoft (Azure AI Search, Foundry IQ) | Minimal, extractive retrieval over search index, blob, OneLake and web sources (REST API 2026-04-01); Foundry IQ knowledge bases (2 Jun 2026) | LLM query planning, answer synthesis, low and medium reasoning effort, multi-turn messages (2026-08-01-preview); Work IQ, Fabric IQ, File Search, Azure SQL and MCP sources; Foundry IQ Serverless | The portals create preview objects that may need migrating; billing by tokens plus Azure OpenAI tokens for planning and synthesis |
| AWS (Bedrock Knowledge Bases) | Fully managed knowledge bases with AgenticRetrieveStream: in the document history since 2 Mar 2026, no preview label on 2 Nov 2026, GA date unverified; AgentCore Memory integration (14 Aug 2026); retrieval instructions (19 Oct 2026) | None labelled on the pages we read | Managed knowledge bases only; BLOCK guardrail action only; a secondary source dates GA to 17 Jun 2026, unverified |
| Google (Gemini Enterprise Agent Platform) | RAG Engine in some Regions | RAG Engine allowlist or preview in other Regions | Agent Search and Agent Retrieval were renamed from Vertex AI Search and Vector Search 2.0 on 22 Apr 2026; agentic features not verified |
Re-check these rows after Microsoft Ignite and AWS re:Invent in late 2026.
How do Microsoft, AWS and Google implement it?
How each cloud exposes agentic retrieval to an agent, as seen 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Entry point | Azure AI Search knowledge base retrieve API; a Foundry IQ knowledge base attached to a Foundry Agent Service, Agent Framework or Copilot Studio agent | AgenticRetrieveStream on a managed knowledge base; AgentCore Gateway (MCP) target | RAG Engine and Agent Search as tools for agents; not verified in detail |
| Planning model | Optional Azure OpenAI model in Foundry Models for query planning (preview) | Managed planning inside the API | Not verified |
| Dials | Reasoning effort minimal (GA), low and medium (preview); number of knowledge sources | Retrieval instructions, reviewed and versioned | Not verified |
| Guardrails on retrieval | Azure AI Content Safety on the agent; Purview labels honoured by the knowledge base | Bedrock Guardrails with the BLOCK action only | Model Armor on prompts and responses |
| Trace | Activity trace with subqueries and token counts in the response | Streamed response with citations; CloudWatch generative AI observability | Agent Observability (GA 18 Jun 2026) |
Google renamed its retrieval line on 22 Apr 2026 and our research did not check its feature-level statuses page by page, so the Google cells say so rather than guess.
Frequently asked questions
Is agentic RAG the same as an AI agent with a search tool?
Close, and the difference is where the loop runs. An agent with a search tool decides in its own reasoning when to search again; agentic retrieval moves planning, parallel subqueries, reranking and the sufficiency check into the retrieval service, so the agent makes one call and gets grounded results with a trace.
Does agentic RAG fix a bad index?
No. It changes the query side only. Chunking, embeddings, hybrid search, reranking and permissions are the same underneath, and a planner that writes five subqueries against a poorly chunked index returns five poor result sets. Fix the index first; Anthropic’s contextual retrieval figures show how much that alone recovers.
How much more does agentic retrieval cost than classic RAG?
It depends on subqueries, sources and reasoning effort. Microsoft’s worked example puts 2,000 agentic retrievals at about USD 4.32 at illustrative rates (Microsoft Learn, agentic retrieval overview, 16 Sep 2026), plus Azure OpenAI tokens for planning and synthesis. Measure your own cost per question type and cap the loop.
Is agentic retrieval generally available on Azure?
Partly. The 2026-04-01 REST API makes minimal, extractive retrieval generally available over search index, blob, OneLake and web sources. LLM query planning, answer synthesis, higher reasoning effort and multi-turn messages were still preview in 2026-08-01-preview on 2 Nov 2026, and the portals create preview objects.
More in Knowledge and retrieval
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Agentic retrieval in Azure AI Search
- Migrate agentic retrieval code to the latest version
- What is Foundry IQ?
- What’s new in Microsoft Foundry at Build 2026
- RAG evaluators (Foundry)
- Amazon Bedrock Knowledge Bases agentic retrieval
- Managed and customer-managed knowledge bases
- Amazon Bedrock document history
- Evaluate RAG in Amazon Bedrock
- Introducing Contextual Retrieval
- Gemini Enterprise Agent Platform release notes
- RAG Engine overview
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner