The reference architecture for a production AI agent

Enterprise leaders keep finding the same pattern in different words. According to EY, 71% of respondents said they were actively using or piloting AI across departments, and 44% pointed to process automation as the leading use case, while 42% named high language-model costs and 41% named protection of proprietary and first-party data as major scaling challenges.

The OECD says difficulty estimating AI return on investment in advance is the most frequently reported obstacle to adoption in firms. A 2025 MIT NANDA study reviewed more than 300 publicly disclosed AI initiatives, interviewed representatives from 52 organizations, and surveyed 153 senior leaders, which shows how much experimentation exists relative to the smaller set of systems that actually run work.

A production AI agent should be built as a governed control plane between users, models, enterprise data, and business systems, with delegated identity, policy-controlled tools, bounded memory, retrieval and context assembly, human approval, and end-to-end semantic telemetry.

Most pilots fail because work is already scattered

Production work rarely lives in one place. It sits across ticketing systems, documents, email, approval queues, spreadsheets, and line-of-business applications, and the rules that govern it are often implicit in human judgment rather than written as executable policy.

Teams also tend to evaluate agents on conversational quality instead of task completion, error rates, authorization correctness, cost, and downstream business outcomes. A fluent answer can still be the wrong action, and in enterprise operations the wrong action is often more expensive than no action.

The incentives push in the same direction. Demonstrations get funded first, while integration, monitoring, rollback, incident response, model version management, and ownership are treated as things to sort out later. By the time those requirements appear, the agent has already been asked to touch the systems that matter.

Identity has to be part of the action path

The first design choice is identity. A production agent needs separate, auditable identities for the human requester, the agent or workload, the model invocation, and each tool or downstream service.

That separation is how you make least privilege real at the tool boundary, distinguish read from write from execution, and answer a basic audit question about who was allowed to do what, for which task, and under which policy. The practical flow is straightforward. Authenticate the user through enterprise identity, create a correlated task and agent identity, evaluate purpose, role, resource, and risk policies, mint a short-lived credential for the specific tool call, record the authorization decision and result, then revoke or expire access when the task ends.

That same logic should hold for data. Retrieval must be identity-aware before content reaches the model, because access permissions, lineage, freshness, retention, and classification metadata are often inconsistent across source systems. Without that filter, an agent can surface stale, unauthorized, or conflicting records and still sound confident.

Tools should be mediated, not exposed

Tool access is where many designs become fragile. An agent should call enterprise APIs through a mediation layer, not receive unrestricted network access or direct database credentials, because the moment a model can roam, the boundary between suggestion and execution disappears.

Each registered tool should have a typed schema, input validation, explicit read, write, or execute classification, data classification and ownership, rate and transaction limits, idempotency behavior, approval requirements, and rollback or compensating-action support. If the tool can move money, alter records, send messages, or trigger a business process, it needs those controls before it goes live.

The same principle applies when the model output is imperfect or unsafe. Tool-level controls still have to hold, which is why restricted parameters, blocklists, mandatory confirmation, rate limits, and immediate access revocation belong in the execution path rather than in a policy document. That is the difference between advice and control.

Memory is useful only when it is bounded

Memory is where useful systems start to drift if nobody defines the boundaries. A production agent should separate working memory, conversation memory, task memory, long-term memory, and organizational knowledge, with the last one preferably kept in source systems rather than copied into the agent itself.

Every durable memory item should carry an owner, source, timestamp, purpose, sensitivity label, retention period, correction path, and deletion mechanism. Memory writes should be validated and, for sensitive domains, require approval or deterministic checks, because persistent memory creates a second path for bad information to survive and influence later tasks.

This is also where retrieval discipline matters. A context service should combine the current user request, authorized enterprise records, relevant retrieved documents, conversation state, task state, policy and business rules, and prior approved outcomes. Use structured queries for authoritative records and semantic retrieval for unstructured content, but never treat vector similarity as an authorization mechanism.

If you want a deeper operating model for that layer, our free paper on compliance-grade retrieval lays out the controls that make retrieval auditable rather than merely helpful.

Bounded workflows beat open-ended autonomy

The most reliable agent systems do not wander. They run bounded workflows with maximum steps and elapsed time, allowed tool sets, budget and token limits, retry and timeout policies, state transitions, human approval checkpoints, and safe termination or rollback.

Deterministic workflow logic should handle high-impact actions. Model-based planning is useful for task decomposition, interpretation, and selecting among preapproved capabilities, but the process itself should remain explicit enough that operators can see where the task is, what changed, and who approved the next step.

1. A user authenticates through enterprise identity and submits a request into the task layer.

2. The control plane assembles context from authorized records, approved documents, task state, and policy rules.

3. The orchestrator plans within fixed limits, chooses from registered tools, and asks for human approval when the action is irreversible or high risk.

4. Tool calls execute through a mediation layer with short-lived credentials, validation, rate limits, and rollback support.

5. The system records semantic telemetry, evaluates the result against business outcome and policy, and writes the trace to the audit log.

Observability is the only way to know whether it worked

Infrastructure metrics alone are not enough. A production agent needs a tamper-evident, privacy-controlled trace that records user, agent, model, and tool identities, agent and prompt versions, retrieved sources and document versions, policy decisions, tool sequence and arguments subject to redaction, latency, token use, cost, human approvals and overrides, errors, retries, termination reasons, and the final business outcome.

That trace should let an operator reconstruct the causal path of a task, not just its request latency. Agent observability should capture planning stages, retrieval provenance, policy outcomes, tool calls, approval latency, and evaluation results, because those are the places where compliance failures and operational failures actually happen.

Evaluation should be continuous and replayable against production-like tasks, with attention to task success and completion rate, unauthorized-action rate, hallucination or unsupported-claim rate, retrieval precision and citation correctness, policy violation rate, human override and escalation rate, cost per successful task, latency and failure recovery, and drift. If the evaluation does not test the thing the workflow was built to do, it is theater.

Some problems still do not belong to agents

The honest counterpoint is that not every job should be wrapped in this architecture. If the process is poorly understood, the data is not governed, or the action has no clear approval path, an agent adds moving parts before it adds value.

It also costs more than a prompt and an API key. Identity mediation, context assembly, approval routing, replayable traces, and tool governance add engineering and operational overhead, and the return only shows up when the task is frequent enough, controlled enough, and expensive enough to justify the control plane.

Some problems still belong to supervised workflows, supervised models, or plain software. Constraint-based optimization, explicit state machines, and narrow classifiers will often outperform a general agent when the decision is stable and the rules are known.

The question for operations leaders is whether the organization can make one repeatable decision, with the right identity, the right tool, the right approval, and the right trace, every time.

If that is the workflow you want gone, tell us the workflow you want gone, we build it in your environment in two weeks and run it from $1,200 a month, and the final setup payment is due only after it meets the acceptance criteria you signed off on. QueryNow can start there.