The pattern is consistent enough to set your watch by. A team ships an agent that handles a real task impressively. Leadership sees it work and scope expands. Six months later the agent is in production, the spend is well above projection, and no one can say with confidence whether the system is getting better or worse.
The instinct at that point is to treat it as a model problem — move to the newest frontier model, add reasoning, try a different framework. Accuracy moves by single digits. Reliability does not move at all.
That is not a failure of judgment. It is what happens when a demo-grade system is asked to carry production load. A demo-grade agent proves the model can perform a task under favorable conditions: a narrow flow, light error handling, informal evaluation, and a human in the room who forgives the misses. A production-grade agent has to survive messy inputs, tool failures, partial context, permission boundaries, retries, audits, safety constraints, cost ceilings, latency targets, versioning, regression tests, rollback paths, and a clear answer to who owns the mistake.
Those are two different systems. The first is a capability demonstration. The second is an engineering platform around the model — evaluation, monitoring, orchestration, memory and state management, tool governance, deployment, incident handling. Almost every organization I work with built the first one well, because that is what the pilot was scoped and funded to deliver. The second system rarely appears in the pilot budget at all, and it is the one that determines whether the agent can be trusted at scale.
What follows is where these systems break, and what the production version costs to build.
1. Name the system you are operating
“Agent” now describes everything from a single model call with one tool to a multi-step planner with memory, sub-agents, and an open-ended task surface. These systems do not share failure modes, do not need the same controls, and should not be held to the same evaluation bar. Before any reliability conversation is useful, the organization has to agree on which one it is running:
- Workflow: a deterministic path through a small number of tools, with the model choosing branches but not control flow. Most production “agents” today are this, and that is a reasonable place to be.
- Bounded agent: the model controls the loop, but the tool surface, recursion depth, and budget are bounded. This is where most reliable enterprise agents live.
- Open agent: the model controls both the loop and the planning. Genuinely valuable, operationally expensive, and rare in production for defensible reasons.
The category is a budget decision as much as a technical one. Governing a workflow as though it were an open agent buys controls you will never exercise. Running a workflow on open-agent assumptions produces a system that improvises its way through structured tasks. Naming the category is the cheapest decision on this list and it sets the cost of every decision after it.
2. Reliability is a long-tail problem, not a model problem
Pre-launch evaluation covers a set of representative tasks. Production traffic is a long tail of unanticipated phrasings, edge cases, partial information, and work the agent was never designed for but is now being asked to do, because it worked well enough that people started relying on it.
Three failure modes dominate real traffic:
- Tool-call drift. The agent calls the right tool with subtly wrong arguments — a misformatted parameter, a wrong identifier, a stale value carried forward from earlier context. The tool errors, the agent attempts recovery, and the recovery frequently compounds the original problem.
- Plan collapse on novel inputs. When input falls outside the patterns the system was tuned on, planning quality degrades sharply. The agent commits to an early step it cannot reverse, then spends context repairing the consequences.
- Loops and runaway tasks. Without enforced budgets, an agent that cannot reach a satisfying answer keeps working — more tool calls, more reasoning, more tokens. It presents as gradual latency creep until someone reads the invoice.
None of these are model failures in any pure sense. They emerge from the interaction between the model, the tools, and the absence of bounds. That is worth stating plainly to a board or an exec team, because it redirects the remediation budget from model licensing to the surrounding system, where it actually moves the number.
3. Observability is the cost of operating agents at all
A conventional service is debuggable because it has logs, stack traces, and deterministic inputs. An agent has none of that by default. When something goes wrong, what the team has is a final response and a general sense that the agent got confused. Without structured traces, even a strong engineering team is reduced to re-running prompts and hoping the failure reproduces. That is a tooling gap, not a skill gap, and it is fixable with a quarter of focused work.
Recording every prompt, completion, and reasoning token is the wrong reflex. It creates a high-value repository of customer data, credentials, retrieved documents, and model output, while still failing to guarantee access to a model’s hidden chain of thought. Start with structured events that reconstruct what the system did.
A minimum useful trace per invocation:
- Invocation and correlation identifiers, including parent-child relationships for sub-agents and handoffs.
- Model and version, token counts, latency, retries, and estimated cost per call.
- Tool events: tool name, approved capability, argument schema or redacted arguments, result status, and latency.
- Context provenance: source identifiers, retrieval or memory operation, and the policy that permitted the context.
- Control events: guardrail decisions, authorization checks, approvals, budget limits, timeouts, and circuit breakers.
- Outcome and failure classification, so reliability work can be prioritized by failure type instead of by whichever incident was loudest.
This shape is standardizing. The OpenTelemetry GenAI semantic conventions model agent invocations, model calls, and tool execution as related spans with common attributes for models and token usage. That gives you a portable trace instead of another framework-specific dashboard you will have to migrate off later.
Content capture is a separate policy decision with its own risk profile. Prompts, completions, tool results, and user data should be redacted by default, selectively sampled, access-controlled, audited on debug access, and deleted on an explicit retention schedule. A trace that copies a secret out of a tool response into a broadly readable observability system has enlarged the incident, not diagnosed it.
The operating discipline is to retain enough structure to isolate whether a failure came from planning, tools, context, policy, or budget, and to elevate raw content only through a controlled diagnostic path.
4. Tool design is the highest-leverage investment available
Tool design moves reliability more than model selection does, and it is the least likely item to be on a roadmap — it reads as plumbing next to a frontier model upgrade, and it is harder to put in a status update. In most engagements I run, a focused pass on tools produces a larger reliability gain than any model swap on the table.
Production-quality tools differ from prototype tools in specific, fundable ways:
- Descriptions written for the model, not the human. The description is the contract. Ambiguity in the contract produces wrong calls at scale.
- Narrow, opinionated, versioned interfaces. A tool that does one thing reliably beats a flexible tool with many valid-looking misuses. Schema changes are API changes and deserve the same process.
- Structured errors that tell the model what to do next. “Invalid input” is a dead end. “Invalid input: field ‘date’ must be ISO-8601” is a recoverable one.
- Caller-stable idempotency keys for consequential actions. Agents and runtimes retry. The same logical action must carry the same key across retries, and the service must replay the stored result rather than perform the action twice.
- Policy enforcement outside the model. Authorization, argument validation, rate limits, and approval requirements belong in the runtime or tool gateway, where they hold regardless of what the model decides.
Framed as a budget line, this is the item with the best return per engineer-week in the entire program.
5. Protocol compatibility is not trust
Protocols such as the Model Context Protocol (MCP) make tools and context easier to discover and connect. Agent-to-agent protocols such as A2A make it easier for independent agents to exchange tasks and results. Both are useful interoperability layers. Neither makes the party on the other end trustworthy.
An MCP server is a trust boundary for both tools and context. Its descriptions influence model behavior, its responses can carry hostile instructions, and its tools can request consequential actions. Each server needs an allowlisted capability surface, authenticated transport, narrowly scoped authorization, output validation, and a review and revocation lifecycle. Tokens issued for one service should not be passed through to another; the MCP security guidance treats token passthrough and confused-deputy behavior as explicit risks.
An inter-agent handoff is the same kind of boundary. The receiving agent has to validate who sent the task, on whose behalf, for what purpose, with which permissions, and under what budget. A protocol can prove two systems speak the same language. It cannot decide whether one should be allowed to instruct the other. That decision is yours, and it belongs in an architecture review before the integration ships, not in an incident review after.
6. Cost is the failure mode that never pages anyone
Agents do not fail loudly when they cost too much. They run, they return results, and the bill arrives at month end without an alert or an error rate for anyone to escalate, which is precisely why cost is usually the first agent problem to reach the CFO.
The compounding factors are well understood and rarely instrumented:
- Context growth per step. Each step appends prior reasoning, tool output, and retrieved context. By step ten the model is processing a multiple of what it processed at step one.
- Tools that return large payloads. One verbose response inflates every subsequent step in the loop.
- Recursive or sub-agent spawning without allocated budget. A single outer invocation fans out into dozens of inner calls.
- Latency-driven retries. A timeout that retries the whole invocation doubles the cost of every slow call.
The controls are ordinary: token and monetary budgets per invocation, wall-clock deadlines, recursion-depth and sub-agent limits, maximum tool calls and payload sizes, and cost telemetry per agent type so you can see which agents are economical and which are quietly expensive.
These have to be enforced by the runtime. Telling the model its remaining budget can help it choose a shorter path, but it is guidance, not control — the same model that is stuck in a loop cannot be the only thing responsible for ending the loop. The runtime rejects or terminates the work when the hard limit is reached. That is the difference between a forecastable line item and a variable one.
7. Agent identity and delegated authority are the new attack surface
Conventional applications typically run under a long-lived service credential. Carrying that pattern into agents grants every task the full authority of the application, even when the initiating user, the purpose, and the risk are all different. It is the reasonable default inherited from how the rest of the estate is built, and it is the wrong default here.
A production agent should hold a distinct workload identity — or a task-scoped identity derived from one — with short-lived credentials. Delegation should carry the initiating user, approved purpose, permitted resources and actions, expiration, and correlation identifier. Every tool authorizes that delegated request on its own, since a call arriving from the agent proves nothing about whether it was permitted.
The questions worth answering before an agent is deployed:
- What is the worst-case action this agent could take if it followed an instruction it should have refused?
- How would that instruction reach it — direct user input, retrieved content, a tool result, a sub-agent?
- Which user and which workload authorized this specific task, and is that delegation still valid?
- Which actions require human confirmation, and which do not?
- Can access be revoked immediately, and can an auditor reconstruct every use of the delegated authority?
- If the model is prompt-injected, what is the containment and recovery path?
Organizations that settle this boundary early scope permissions narrowly, use short-lived credentials instead of shared secrets, require confirmation for irreversible actions, and preserve revocation and audit trails. A successful injection may still influence the model, but it cannot manufacture authority the runtime never granted. That is a containment guarantee you can describe to a regulator or a customer.
8. Persistent memory is production data
Memory changes the operating model. A bad response is transient; a poisoned memory can shape tasks for weeks. A fact stored for one user can surface in another user’s session. An outdated preference can quietly override a correction someone made last month. Once an agent remembers, it is holding retained data with all the obligations that implies.
Treat persistent agent memory as governed application data:
- Provenance: record who or what supplied each memory, when it was derived, and the evidence behind it.
- Expiration and revalidation: give memories a lifetime appropriate to their source, since nothing stored stays true by default.
- User and tenant isolation: enforce separation in storage and retrieval, not in a prompt instruction.
- Correction and deletion: let authorized users inspect, correct, and remove retained information, including derived records where required.
- Poisoning detection: monitor unusual writes, conflicting facts, untrusted sources, and memories that attempt to change agent policy or tool behavior.
Memory writes deserve their own authorization and audit events. “The model decided this was worth remembering” is not a retention policy, and it will not survive contact with a privacy review.
9. Evaluation has to measure the path
Task completion alone is a misleading metric. Two agents can post the same completion rate while differing materially in cost, latency, tool-call accuracy, and how many completions involved a recovered error along the way. One of them is ready to scale and the other is not, and the headline number will not tell you which.
A defensible evaluation covers at least:
- Task completion — did the agent achieve the user-visible goal?
- Path quality — did it take a reasonable route, or work through several wrong branches before recovering?
- Tool-call accuracy — right tools, right arguments, or improvisation?
- Policy and identity compliance — did every action stay inside the user’s delegated authority, approval requirements, and data boundary?
- Memory quality — did it retrieve and write appropriate, correctly scoped, current information?
- Cost and latency per task — at what economic envelope did completion occur?
- Failure classification — when it failed, what type of failure was it?
That last one carries more weight than it looks like it should, because without it remediation gets prioritized by anecdote. Process-level evaluation is what converts “the agent feels better this quarter” into a claim you can defend in a review. Without it, behavior drifts unobserved as prompts and models change, and no one can tell whether a new version is genuinely better or only better on the sample someone happened to read.
10. Ownership is the precondition for the other nine
An agent’s behavior is set by the system prompt, tool surface, identity and policy layer, retrieval and memory systems, model choice, evaluation suite, and production telemetry. In most organizations each of those sits with a different team, each of which is doing its job well. What is missing is an owner of the user-visible behavior of the agent as a whole.
The result is structural, not cultural: every team optimizes the surface it controls, no one is accountable end to end, and the system settles at a reliability ceiling that everyone can describe and no one is empowered to move.
The same pattern shows up in production RAG systems and in AI security programs, for the same reason: a system spanning multiple disciplines without explicit end-to-end ownership accumulates local optimizations that never add up to a better whole.
For agents, a named owner with authority across prompts, tools, identity, memory, retrieval, and evaluation is the precondition for everything else on this list. It is also the only item here that cannot be delegated to the engineering org — it is an org design decision, and it is made above them.
What production-grade agents look like
Production-grade agents have less in common with their demo predecessors than most roadmaps assume.
They are explicitly categorized as workflow, bounded, or open, and operated accordingly. They carry structured, privacy-aware observability that lets a human reconstruct an invocation without retaining every sensitive input. Their tools are designed for the model, with idempotency and policy enforced outside it. MCP servers and agent handoffs are treated as trust boundaries. Identities and credentials are short-lived and tied to a delegated purpose. Memory is governed data. Budgets are enforced by the runtime, evaluation measures the path, and one person owns the system end to end. The same controls written out as a checkable inventory are on the AI platform capability checklist.
None of this depends on a smarter model. It depends on whether the organization is funding a system or a capability demonstration, and that distinction usually was not available to make at the time the agent was first funded, because the agent had not yet proven it would matter this much.
It is available to make now. The reliability ceiling in production agents is set by tool design, observability, identity, memory, budget enforcement, evaluation discipline, and ownership. The model is the one variable on that list you can change with a config flag, which is exactly why it absorbs attention the others deserve.
Reliable agents are built around the model, not out of it.
If your agents work in evaluation but break in production, or if cost and reliability are no longer predictable as you scale, I work with engineering leadership to diagnose where the system actually fails and build the platform that production-grade agents require. Get in touch.