- Home
- AI agents
- Agent memory and context
- Context engineering
Build · Agent memory and context
What is context engineering?
Context engineering is the practice of deciding what goes into a model’s context window on each call: the instructions, the conversation, retrieved documents, memory records, tool definitions and tool results, in what order and within what budget. Prompt engineering wrote the prompt. Context engineering runs the whole window, because an agent fills it with things the developer never typed.
In this guide: 83 pages
Key takeaways
- The window is a budget. Every token of instructions, retrieved text, memory and tool output costs money on every call and competes for the model’s attention, which the Lost in the Middle study showed is weakest in the middle of a long input.
- Context engineering differs from prompt engineering in scope: a prompt is text a person wrote; the context is assembled per call from retrieval, memory, tools and history, mostly by code.
- The tooling matured in 2025 and 2026: context editing and compaction from Anthropic and OpenAI, deferred tool loading on every major platform, and memory services that return only the records that match the turn.
- Cost and quality trade off per task. Anthropic’s own evaluation reported a 39% gain from memory plus context editing and an 84% token reduction over a 100-turn test; compaction can also drop the one fact a later turn needs.
What goes into the context window, and why?
Seven things, on a typical agent call: the system instructions, the tool definitions, the memory records retrieved for this user, the retrieved documents, the conversation history, the results of earlier tool calls in this task, and the current user message. Each is there for a reason, and each has a cost.
The seven inputs to a context window, why each is there, and what usually goes wrong.
| Input | Why it is there | Typical failure |
|---|---|---|
| System instructions | Role, rules, output format, what to refuse | Grows with every incident until nobody reads it; conflicting rules |
| Tool definitions | So the model can choose and call tools | Hundreds of definitions loaded up front; tool search and deferred loading fix this |
| Memory records | User preferences and past-task facts for continuity | Everything retrieved, nothing filtered; stale records |
| Retrieved documents | Company knowledge the model did not train on | Too many chunks, no reranking, no citations |
| Conversation history | What was said in this session | Replayed in full; old tool results never cleared |
| Tool results | The data the agent just fetched or the action it took | Raw JSON dumps of thousands of tokens; errors pasted in full |
| Current message | The task at hand | Buried at the end of 150,000 tokens of everything else |
Order matters as much as content. The Lost in the Middle paper (2023) measured models answering best when the relevant passage sat at the start or end of a long input and noticeably worse in the middle, and Microsoft’s own Foundry guidance says to retrieve stable user profile memory early in the conversation. Put the rules and the task where the model will find them.
Context engineering vs prompt engineering: what is the difference?
Prompt engineering is writing the instruction a model gets; context engineering is managing everything the model sees across a multi-step task, most of which code assembles at run time from retrieval, memory and tools. The second is a systems design job with a token budget.
Prompt engineering and context engineering side by side.
| Aspect | Prompt engineering | Context engineering |
|---|---|---|
| Unit of work | One prompt, written by a person | One context window per call, assembled by code |
| Inputs | Instructions and examples | Instructions, tools, memory, retrieval, history, tool results |
| Where it lives | A string in the code or a prompt registry | The orchestration layer: retrieval, memory and tool-loading settings |
| Main risk | Ambiguous wording | Budget overrun, lost facts, injected content in retrieved text |
| How you test it | Review outputs against examples | Trace the window: what was in it, in what order, at what cost |
| Versioning | Prompt versions | Prompt versions plus retrieval, memory and tool configuration |
The two are complements. A clear prompt still matters; it is now one of seven inputs, and often the smallest.
How do retrieval, tool results, memory and instructions share the window?
They share it through four operations the platforms now provide: load tools on demand instead of all at once, clear or compact old history, filter memory by scope and metadata, and cap retrieval by count and reasoning effort. The 2025 and 2026 releases made each of these a setting rather than custom code.
- Deferred tool loading. Instead of sending every tool definition on every call, tools are marked for deferred loading and the model calls a search tool to load only what it needs. Anthropic’s tool search tool supports up to 10,000 deferred tools per request and no longer needs a beta header (17 Feb 2026); OpenAI offers tool search with namespaces; AgentCore Gateway has semantic tool search; Foundry Toolbox fronts many tools behind one MCP endpoint with tool search (page dated 26 Oct 2026). Anthropic’s engineering post gives the scale of the problem: five MCP servers came to 58 tools and about 55,000 tokens of definitions before any work, and its tool search raised accuracy on an MCP evaluation from 49% to 74% while cutting context by about 85% (24 Nov 2025, vendor-reported).
- Clearing and compaction. Anthropic’s context editing (beta) clears old tool results on the server, by default once input passes 100,000 tokens, keeping the last three tool uses; its compaction (beta, 14 Sep 2026) replaces older turns with a model-written summary on demand or at a threshold. OpenAI exposes the same idea as a context management setting with a compact threshold and a compact endpoint.
- Memory with filters. AgentCore long-term records carry up to 10 indexed metadata keys (6 May 2026) so a retrieval returns the records for this tenant and topic; Foundry’s memory search tool scopes to the user through {{$userId}}; Memory Bank returns memories by similarity to the current turn.
- Retrieval with a budget. Azure AI Search agentic retrieval has a reasoning effort setting (minimal, low, medium) and bills by tokens; the Microsoft 365 Copilot Retrieval API returns at most 25 results per call. Rerank, then pass the top few chunks with citations rather than the top fifty.
- Instructions kept short and placed well. A system prompt that grew by one rule per incident is a context problem. Move stable rules into tool descriptions and retrieval instructions (Bedrock added reviewed, versioned retrieval instructions on 19 Oct 2026), and keep the prompt to what the model must always know.
What are the cost and quality trade-offs?
More context raises cost with every uncached token sent and raises quality only until the useful signal is diluted; less context cuts cost and risks losing the fact a later step needs. The practical answer is to measure both per task and set a budget, since neither a vendor default nor a bigger window settles it.
The case for less. Anthropic’s internal agentic-search evaluation (29 Sep 2025, vendor-reported) found a 39% improvement with the memory tool and context editing together, 29% with context editing alone, and an 84% token reduction across a 100-turn test. Anthropic’s contextual retrieval work (19 Sep 2024) cut top-20 retrieval failures by 49%, and 67% with reranking, by adding a short chunk-specific context before embedding, which means fewer, better chunks in the window.
The case for care. Compaction summaries drop details; Anthropic documents writing your own summarisation prompt and pairing compaction with a memory store for facts that must survive. Background memory extraction lags (AgentCore says a minute or more), so a fact may be missing from the next call. Retrieved text can carry instructions, which is how CVE-2025-32711 (Microsoft 365 Copilot, CVSS 9.3, 11 Jun 2025) disclosed information: the more you put in the window, the more you must screen.
What we measure: tokens per completed task (input and output, by component), the share of the window taken by tool results, the hit rate of retrieved chunks that the answer actually cites, and task success on an evaluation set before and after each change to the context assembly. Then we set a ceiling per task class and alert when a run exceeds it.
How do you start with context engineering?
Start by looking at one real window: trace a production task and read what the model saw, in order, with a token count per component. In the windows we trace, tool results and history usually take most of the budget and the instructions a small fraction of it.
- Trace one task end to end and dump the context of each call with token counts by component (instructions, tools, memory, retrieval, history, tool results).
- Set a budget per task class and a ceiling that triggers an alert.
- Turn on deferred tool loading and remove tools the task never calls.
- Clear old tool results and compact history, then check the evaluation set still passes.
- Cap retrieval: rerank, pass the top chunks with citations, and give agentic retrieval the lowest reasoning effort that keeps the score.
- Scope memory to the user and filter by metadata; retrieve stable profile facts first.
- Re-run the evaluation set after every change to the assembly, and again when the model changes.
How do Microsoft, AWS and Google implement it?
Where each cloud puts the context-engineering controls, as seen 2 Nov 2026.
Capability, Microsoft, AWS, Google
| Capability | Microsoft | AWS | |
|---|---|---|---|
| Memory retrieval into the window | Foundry memory search tool with {{$userId}} (preview); Copilot memory applied automatically | AgentCore Memory retrieval by namespace and metadata; on the managed harness memory is on by default per actor | Memory Bank similarity retrieval; memory profiles |
| Knowledge retrieval | Foundry IQ knowledge bases (GA 2 Jun 2026); reasoning effort minimal (GA), low and medium (preview); Copilot Retrieval API, 25 results per call | Bedrock Knowledge Bases agentic retrieval on managed knowledge bases; retrieval instructions (19 Oct 2026) | Agent Search, RAG Engine and Agent Retrieval (renamed from the Vertex AI products on 22 Apr 2026) |
| Tool loading | Foundry Toolbox: one MCP endpoint with tool search | AgentCore Gateway semantic tool search | Agent Gateway and Agent Registry for tool discovery |
| History and compaction | Managed by the agent framework; Foundry chat summary memory (preview) | Short-term events with expiry; summary strategy | Agent Platform Sessions |
| Screening what enters | Azure AI Content Safety Prompt Shields (GA); Purview DLP for Copilot | Bedrock Guardrails on prompts, responses and knowledge-base retrievals (BLOCK action only on agentic retrieval) | Model Armor on prompts and responses |
Anthropic’s context editing and compaction and OpenAI’s compaction are features of the model vendors’ APIs; availability depends on the model, API and hosting platform, and both Anthropic features were beta on 2 Nov 2026.
Frequently asked questions
Is context engineering just a new name for prompt engineering?
No. Prompt engineering writes the instruction; context engineering manages the whole window the model sees on each call, including retrieved documents, memory, tool definitions and tool results that code assembles at run time. The term spread in mid-2025 as agents started filling windows with content nobody typed.
What is the difference between context engineering and RAG?
RAG is one source of context: documents retrieved from company content. Context engineering decides how much of that retrieval goes into the window, in what order, next to memory, tools and history, and within what token budget. Good RAG with no budget still overflows the window.
Does a bigger context window remove the need for context engineering?
No. A 1M-token window (Anthropic, generally available 13 Mar 2026) bills every uncached token you send, and models still use the middle of long inputs worse than the ends. Larger windows make the budget decision bigger, and compaction and deferred tool loading remain the main savings.
How much context should an enterprise agent use per call?
As little as passes the evaluation set. Trace a real task, count tokens per component, set a ceiling per task class and alert on it. Anthropic reported an 84% token reduction over a 100-turn test from its memory tool and context editing (vendor-reported, 29 Sep 2025), which shows how much a window can carry that the task never uses.
More in Agent memory and context
Sources
Each source was read on 2 November 2026. The date shown is the date on the page, or the page’s ms.date for Microsoft Learn; an undated page shows the date it was read.
- Lost in the Middle: How Language Models Use Long Contexts, arXiv 2307.03172
- Managing context on the Claude Developer Platform
- Context editing
- Compaction
- Tool search tool
- Advanced tool use
- Claude Developer Platform release notes
- Introducing Contextual Retrieval
- Conversation state
- Tool search
- Foundry Toolbox overview
- What is memory in Foundry Agent Service
- Agentic retrieval in Azure AI Search
- Microsoft 365 Copilot Retrieval API overview
- AgentCore Gateway semantic tool search
- AgentCore Memory adds metadata on long-term memory records
- Memory on the AgentCore managed harness
- Amazon Bedrock Knowledge Bases agentic retrieval
- Amazon Bedrock document history
- CVE-2025-32711
- Gemini Enterprise Agent Platform release notes
Planning an AI program?
Talk to the team that builds it and runs it after launch.
- Built to SOC 2, HIPAA and GDPR standards
- EU AI Act-aligned delivery
- Microsoft Solutions Partner