Engineering Agentic Systems That Hold in Production

The defining shift of 2025–2026 was not a more capable model. It was the move from single-turn assistants, which answer, to agents, which act — decomposing a goal, calling tools, reading results, and proceeding over many steps with limited supervision. A chatbot that is wrong is an annoyance the user corrects in the next message. An agent that is wrong has already taken the action, and the next step builds on it.

This is why pilots mislead. A scripted demonstration exercises the happy path a handful of times; production exercises the full distribution of inputs thousands of times, unattended, where every percentage point of unreliability compounds into rework, oversight cost, and eroded trust. The binding constraint on enterprise agentic AI is therefore not capability but reliability under autonomy — and reliability is engineered, not summoned by scale [6].

A production-grade agent is not a prompt; it is a system with separable planes, each separately testable and governable. We find five are load-bearing.

2.1  The context plane

What the agent knows when it reasons. Retrieval, grounding, and the assembly of relevant, current, permissioned information into the working context. Most "hallucination" in production is a context-plane failure — the model reasoned correctly over the wrong or missing facts [1].

2.2  The reasoning & planning plane

Goal decomposition, step selection, and routing — the model deciding what to do next, often interleaving reasoning with action [2]. This plane benefits from explicit structure: smaller, verifiable sub-goals outperform a single open-ended trajectory [6].

2.3  The tool & action boundary

Where intent becomes effect. A typed, permissioned boundary — schemas, validation, and least-privilege scopes — converts a class of free-text errors into caught, structured failures before they reach a system of record [3,7].

2.4  The memory plane

State that survives the step: episodic traces, semantic stores, and task state. Memory is what lets an agent be consistent across a long task — and, ungoverned, what lets a stale or poisoned fact propagate [5].

2.5  The verification plane

The plane most pilots omit and every production system needs: a dedicated check on the agent's work — evaluators, policy gates, and human-in-the-loop approval at defined thresholds [4]. Verification is the lever that arrests the compounding described next.

Treat each step of a task as succeeding with probability p. If a task requires n sequential steps that each must be correct, and we take steps as approximately independent, end-to-end success is:

The consequence is unforgiving. At a genuinely strong 95% per step, a 10-step task succeeds 60% of the time and a 20-step task only 36%. Even at 99% per step, a 30-step task fails roughly one time in four. The headline capability of the model barely moves this curve; the number of unverified steps moves it decisively.

Two corrections follow directly. First, shorten the unverified chain — decompose long tasks into checkpointed sub-tasks so a failure is caught and retried locally rather than propagated. Second, break the independence assumption deliberately with verification: a check that catches and repairs an error mid-chain restores per-step reliability toward 1 at that point, flattening the curve. This is why the verification plane is not optional polish — it is the control that makes long-horizon work viable at all.

Reliability is achieved by mapping each failure mode to a specific control, then enforcing it in the architecture rather than hoping the model behaves.

A useful design heuristic underlies the table: move every responsibility the model does not need to own out of the model. Control flow, validation, permissions, and idempotency belong in deterministic code around the agent; the model is reserved for the judgment only it can supply. The most reliable agentic systems are, by design, the ones that ask the model to do the least.

An agentic system that is not measured is not operated — it is hoped. Four metrics form the minimum operating set:

Step-level reliability (p). The per-action success rate, measured against ground truth on a representative evaluation set. This is the lever Figure 1 magnifies.

Task success rate. End-to-end completion to the defined acceptance criteria — the number the business actually experiences.

Mean steps between failures. How far the agent runs unattended before intervention — the direct measure of how much autonomy is safe to grant.

Cost per successful outcome. Total token and tool cost divided by successful tasks — failures are not free, and this ties reliability to unit economics [8] (the subject of a companion paper in this series).

These are produced by a three-layer evaluation regime: offline regression suites that gate every change, online canary measurement on live traffic shares, and continuous re-evaluation triggered by model and dependency updates. Without this regime, a model provider's silent update can move p — in either direction — with no one the wiser until the quarter's numbers arrive.

The chain from engineering to margin is short and strict. Reliability earns trust; trust permits autonomy; autonomy removes the human cost that otherwise caps the return on every agent. An agent supervised on every action saves little; an agent trusted to run unattended within a bounded scope changes the operating cost of the function it runs. Everything upstream in this paper exists to make that trust defensible.

For the executive, the practical output is a procurement and operating bar. Before an agentic system is allowed to run a real process, require: (i) a measured task-success rate against a named workload and acceptance criteria — not a demo; (ii) a typed boundary documenting exactly what the agent can and cannot do; (iii) a dedicated verification plane with defined human-in-the-loop thresholds; and (iv) a single accountable owner of the number, with the metric set above reported on a standing cadence. A vendor or team that cannot produce these is selling capability, not an operation.

This is also where the build-versus-operate question resolves. The model is a commodity input; the reliability discipline around it is the durable asset, and it is owned by whoever is accountable for the outcome — ideally an operator inside the customer's perimeter, answerable to the customer's KPI rather than a platform's roadmap [8].

The pⁿ model assumes step independence; in practice errors correlate — a poisoned context degrades many downstream steps at once — so the curve is a clarifying lower-bound intuition, not a precise predictor. Evaluation coverage is itself an open problem: a task-success number is only as honest as the representativeness of its eval set, and adversarial and long-tail inputs remain hard to enumerate. Non-stationarity — provider model updates changing behavior beneath a stable interface — makes one-time certification insufficient and continuous assurance mandatory. Finally, the frontier of genuinely long-horizon autonomy, where verification is itself performed by models, is active research; we treat human-defined acceptance criteria as the ground truth of record until that matures.

Production agentic AI is won or lost in the engineering between the model and the business, not in the model itself. The compounding-error curve makes the stakes quantitative: reliability must be constructed, measured, and owned. Firms that treat it as a discipline — typed boundaries, verification planes, bounded autonomy, and a metric set with a name attached — will operate agents the business can trust unattended. Firms that treat reliability as something the next model will provide will keep shipping demonstrations. AI strategy ends where the bill begins; agentic engineering is where reliability does.