A demo agent and a production agent are not the same thing. They share a vocabulary, often the same framework, sometimes the same prompts. What they do not share is the infrastructure underneath that makes the second one a system you can put in front of customers. The gap is where most agent projects fail, quietly and expensively, somewhere between the demo that went well and the rollout that did not.
The good news is that the infrastructure required is not exotic. It is the same boring discipline that turned web apps from CGI scripts into systems that handle a million requests an hour. The bad news is that “boring” is exactly what the current AI discourse rewards skipping. Frameworks ship with the demo path lit up and the production path absent. You have to build the production path yourself.
The four things that separate the two
1. Caching on every retrieval
Demo agents call the model fresh every time. Production agents recognise that most queries are variations of queries the system has already seen, and serve a cached answer when the cached answer is the right answer. The cost difference is one to two orders of magnitude. The latency difference is similar. The quality difference, if the cache is keyed correctly, is zero.
The discipline is in the cache key. Cache by what the query means, not by the raw text of it. A cache that misses on every typo is not a cache. A cache that returns yesterday’s answer when the underlying data has changed is worse than no cache at all. Both failure modes are detectable in evals if you write the evals to look for them.
2. Deterministic fallbacks under failure
Demo agents assume the model will be available and the call will succeed. Production agents assume neither. Every call has a fallback path: a cheaper model, a deterministic rule, a routed escalation, a polite refusal. The fallback is exercised on every load test. The fallback path runs through the same logging and observability as the primary path so you can tell, at a glance, what fraction of traffic the fallbacks are catching today.
A familiar rescue story in this market: an agent whose failure mode was “show the customer an error and let them figure it out.” Customers did figure it out. They figured out the agent did not work and stopped using it. The fallback was a one-day fix that should have been built before the first user saw the system.
3. Observability that answers real questions
Demo agents have logs. Production agents have observability. The difference is being able to answer a question like “which finance queries ran slow enough yesterday for the customer to notice, and what did they have in common?” without writing new code. The schema is intentional, the dashboards are pre-built, and every decision the agent makes is reconstructable from the logged record.
The discipline that pays off here is logging the inputs the model saw, the retrieved context, the prompt that was actually rendered (templates are deceiving), the model version, the response, and the post-processing applied to that response. When a stakeholder asks why the agent made a particular decision two weeks ago, you can reconstruct the answer from the log. When you cannot, you have to apologise instead, and trust in the system erodes.
4. Eval gates on every change
Demo agents are tested when someone remembers. Production agents are tested on every change to the prompt, the model, the retrieval index, or the post-processing. The eval set runs automatically in the deployment pipeline. A change that drops the eval score is blocked until someone deliberately overrides the gate with a written justification. The override is also logged.
This is the discipline that catches the silent regressions, the model updates that shift behaviour in ways nobody noticed until production found them. It is also the discipline that makes the system upgradeable. When a new model ships, you do not deploy it because the vendor’s blog post said it was better. You deploy it because it scored higher on your golden set, the fixed collection of real cases with known-correct answers.
What this costs
Adding the production discipline to an agent system typically adds a third to a half to the initial build cost. It pays that back many times over in the first year of operating cost, because the system stops failing in ways that consume the entire team’s attention. The trade is overwhelmingly worth it. The team that has to choose between “ship faster” and “ship sustainably” is the team that ends up doing the work twice.
The order that works
Build the eval suite first. Build the observability second. Build the agent third. Wire the caching, fallbacks, and gates as you go, not as an afterthought. The agent shipped this way is not flashier than the demo. It just keeps working when the demo stops being interesting and the real users start using the system in ways nobody designed for.
That is the only definition of “production” that means anything.