Agentic AI in production: architecture, guardrails, and evaluation
Most teams reaching for agents don't need one. A field guide to agent architecture, guardrails and evals — and the production maths that decide if it ships.
AMDIM · Engineering
June 12, 2026
Key takeaways
- Choosing an agent over a workflow is a trade-off, not an upgrade — autonomous step selection comes with cost, unpredictability, and harder debugging.
- Five workflow patterns (chaining, routing, parallelisation, orchestrator-workers, evaluator-optimiser) cover the majority of production AI use cases.
- Prompt injection is a structural risk, not a bug — knock out one leg of the lethal trifecta (private data + untrusted content + external comms) to close the attack path.
- Evaluate with pass^k, not pass@1 — a model that succeeds once in eight attempts is not production-ready, no matter how good it looks in a demo.
- Budget 15× the token cost of a single chat turn when sizing an agentic system — and account for compounding error rates across every step.
- At 95% per-step reliability, a 20-step agent completes end-to-end about a third of the time — design for short horizons, checkpoints, and human gates.
Gartner projects that more than 40% of agentic AI projects will be cancelled by end of 2027 — not because the technology fails, but because the approach does. This is a production field guide for the teams determined to be in the other 60%.
What actually counts as agentic?
The term 'agentic' has been stretched to cover almost any system that calls an LLM. The Anthropic definition is more useful: it centres on who — or what — decides what happens next. In workflows, that decision is made by predefined code. In agents, the LLM itself dynamically directs its own processes, choosing which tools to call and in what sequence.
Agentic AI System
Systems where LLMs dynamically direct their own processes and tool usage — maintaining control over how they accomplish tasks — as opposed to deterministic workflows where sequences of LLM calls and tool uses are predefined in code.
Source: Anthropic · Building Effective Agents · 2024
Agent vs Workflow: When to use each
| Feature | Autonomous Agent | Deterministic Workflow |
|---|---|---|
| Step sequence known at design time | ||
| Handles novel, unpredictable inputs | ||
| Predictable latency and cost per run | ||
| Easy to debug and test | ||
| Right for >80% of enterprise AI use cases |
When do you actually need an agent?
Workflow patterns cover the majority of production AI use cases. Before reaching for autonomous step selection, work through these five patterns — each is cheaper, more predictable, and easier to audit than the equivalent agent architecture.
Five Workflow Patterns Before Reaching for Agents
Prompt Chaining
Fixed sequential steps where each LLM call feeds the next. Cleanly decomposable tasks — draft, critique, revise — where the sequence is known in advance. Simple to debug; cost is predictable.
Routing
A classifier LLM reads an input and routes it to the most appropriate specialised prompt or sub-pipeline. Enables tight, well-tested prompts for each category rather than one prompt trying to handle everything.
Parallelisation
Run multiple LLM calls simultaneously and aggregate the results. Use for independent sub-components or ensemble scoring where multiple independent assessments reduce variance.
Orchestrator-Workers
A central orchestrator decomposes a goal and delegates to specialist workers, which execute bounded sub-tasks. The orchestrator retains control; the step decomposition is largely predetermined.
Evaluator-Optimiser
One LLM generates, another critiques against clear criteria, and the loop repeats until a quality threshold is met. Powerful for tasks with clear quality signals — compliance with a rubric, code correctness, translation quality.
/ The rule
Find the simplest solution possible — only increase complexity when the task demands it.
The most expensive architectural mistake in 2026 is reaching for an autonomous agent where a workflow would have shipped. Most production value still lives in deterministic workflows. They're just less interesting to demo.
Guardrails start from an uncomfortable truth
Prompt injection — where malicious content in an agent's environment hijacks its instructions — has held the number-one position in OWASP's Top 10 for LLM Applications for two consecutive editions. It is not a bug in your code. It is a structural property of any system that processes untrusted content and takes consequential actions.
The lethal trifecta
An agent becomes an exfiltration risk the moment it combines: access to private data, exposure to untrusted content, and the ability to communicate externally. Knock out any one of the three and the attack path closes. Meta's Rule of Two limits any single agent session to two of the three — route the rest through human approval.
ALLOWED_TABLES = {"orders", "shipments"} # explicit allow-list, never "*"
def query_db(sql: str, table: str) -> list[dict]:
if table not in ALLOWED_TABLES: # 1. allow-list, not free choice
raise PermissionError(f"table {table!r} not permitted")
return db.run(sql, read_only=True) # 2. read-only credential
def issue_refund(order_id: str, amount: float, ctx: Session) -> Result:
if amount > ctx.auto_approve_limit: # 3. human gate on irreversible action
return ctx.escalate_to_human(order_id, amount)
return payments.refund(order_id, amount) # scoped, idempotent, auditedHow do you evaluate something that won't sit still?
You can't ship on vibes, and you can't test an agentic system like deterministic software. A passing unit test on the happy path tells you almost nothing about production reliability — because production is not one run, it's eight.
On τ-bench retail tasks, GPT-4o scored ~61% on a single attempt — but its pass^8 reliability dropped below 25%. A model that looks great once can be wildly unreliable across eight runs. Production is the eighth run.
— Sierra AI · τ-bench evaluation paper · 2024
Reliability Compounding: Why Per-Step Rates Mislead
| Per-step success | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 95% | 77% | 60% | 36% |
| 99% | 95% | 90% | 82% |
Measure pass^k, not pass@1
pass^k (all k attempts succeed) is what production cares about. At 95% per-step reliability, a 20-step agent succeeds end-to-end about a third of the time. Build in checkpoints and human gates — each one cuts the exponent.
Where it goes wrong
Seven Failure Modes to Design Around
- 01
Reaching for an agent where a workflow would do
The majority of production AI value still lives in deterministic workflows. Building an agent adds cost, unpredictability and debugging complexity — most of the time for no benefit over a well-designed workflow.
- 02
Over-tooling past ten tools
Every tool added to an agent's set increases the probability of the wrong tool being called with the wrong parameters. Minimal, clearly scoped tool sets outperform large, general ones in production evaluation.
- 03
Optimising pass@1 only
Reporting pass@1 scores in demos and then deploying into a production environment where the agent runs eight times a day is a category error. Measure pass^k before shipping — especially for customer-facing or financially consequential tasks.
- 04
Accidentally assembling the lethal trifecta
Private data access plus untrusted content processing plus external communication capability — not always assembled intentionally, but the attack path exists the moment all three are present in a single agent session.
- 05
Runaway cost from recursive sub-agents
Multi-agent systems accumulate context aggressively. Without explicit management strategies, token costs compound turn by turn — the 15× multiplier becomes a 50× one before anyone notices the bills.
- 06
Logging only final outputs, not trajectories
When something goes wrong in an agentic system, the failure is almost never in the final output — it's in a tool call six steps earlier. Log every tool call, every intermediate state, every model decision. Observability is not optional.
- 07
Agent washing
Marketing a deterministic pipeline as an 'autonomous agent' to stakeholders creates a gap between expectation and reality that eventually damages trust. Name what you've built accurately — and the expectations you set will match what gets delivered.
Frequently asked questions
In a workflow, the sequence of LLM calls and tool uses is predefined in code — the code decides what happens next. In an agentic system, the LLM itself dynamically decides which steps to take, which tools to call, and in what order. The distinction determines your cost predictability, debuggability, and the risk profile of the system.
The most reliable structural defence is Meta's Rule of Two: limit any single agent session to at most two of the three ingredients that make injection dangerous — access to private data, exposure to untrusted content, and the ability to communicate externally. Beyond that: validate all inputs and outputs against schemas, scope every tool credential to least privilege, and put a human gate in front of irreversible actions.
Evaluate with pass^k (all k attempts succeed) rather than pass@1 (at least one succeeds). Run the full task multiple times and measure whether results are consistently acceptable — not just occasionally impressive. Build automated LLM-as-judge scoring for daily drift detection once you are live.
The root causes tend to cluster around three: (1) the system was designed against pass@1 metrics that don't reflect production conditions, (2) error rates compound across steps in ways not accounted for at design time, and (3) the system was built without the observability infrastructure needed to diagnose problems when they occur.
Is your agent architecture production-ready?
The Agent Readiness Scorecard identifies gaps in your architecture, guardrails, and evaluation setup before you commit to production.
Take the Agent Readiness Scorecard/ go deeper
Put this to work on your actual numbers.
A ten-minute assessment maps exactly where you are today — and what to do first.
/ keep reading
Agentic AI · 14 min
The Agentic AI Playbook: Architecture, RAG, Multi-Agent Systems, and What Actually Works in 2026
A practitioner's guide to the full agentic AI stack — from the agent reasoning loop to multi-agent orchestration, RAG pipelines, and the evaluation frameworks that keep it all honest.
Engineering · 10 min
Pilot to production: why most AI stalls — and the operating model that ships it
Only ~48% of AI projects reach production. The blocker isn't the model — it's the operating model. Here's the playbook that ships.
Article · 7 min
Cloud-ready isn't AI-native — and that gap is where pilots die
Most AI pilots stall on infrastructure, not models. Here is the real gap between cloud-capable and AI-native, and how to close it without a rebuild.