Every AI system we've inherited from another team, and every one we've watched fail after launch, fails for the same structural reason: the thing that got tested and the thing that got shipped were subtly different systems. Not different models — different distributions of input. A demo runs against a curated set of examples someone chose because they worked. Production runs against whatever a real user types, on a bad day, with a typo, asking two questions at once.
01The eval set is not the traffic
This is the single most common gap we see. A classification model built for a support queue might hit 94% accuracy on a labeled set of 500 tickets. Six weeks after launch, live accuracy has settled at closer to 78%. The model didn't degrade — the eval set was built from historical tickets that had already been through human triage, so it was implicitly filtered toward clean, single-intent examples. Production traffic includes compound tickets ("also, can you cancel my subscription and refund the last two charges"), tickets in a second language mid-sentence, and tickets that are really two separate issues someone typed as one message. None of that showed up in the 500 examples, because a human had already resolved the ambiguity before the ticket was labeled.
The fix isn't a bigger eval set — it's an eval set sampled from the actual distribution the system will see, including the messy tail, refreshed on a schedule rather than built once and left static. We build eval sets from live traffic wherever we can, sampled continuously, not from a historical archive that was pre-cleaned by the process the AI system is meant to replace.
02Retrieval degrades as the corpus grows, silently
A RAG system tuned and tested against a 2,000-document knowledge base commonly holds up fine through the pilot. At 50,000 documents, recall on the same query set can drop meaningfully — more documents means more near-duplicate chunks competing for the same top-k slots, and a chunking strategy tuned for a smaller, cleaner corpus starts returning plausible-looking but wrong context more often. This doesn't show up as an error. It shows up as the model confidently answering from the wrong document, and nobody sees a stack trace. If you don't have a retrieval-quality metric — recall@k against a held-out query set, tracked over time, not eyeballed once at launch — you won't know this is happening until a user tells you the answer was wrong, and by then it's happened to an unknown number of people who didn't say anything.
03Agents fail by not stopping
A multi-step agent with no explicit termination condition doesn't crash when it gets confused — it keeps going. We've seen agent loops that called the same tool nine or ten times in a row because a downstream API returned an ambiguous response and nothing in the agent's logic distinguished "try again" from "this isn't working, stop and ask." The failure mode isn't a wrong answer; it's a five-minute delay and an unexpectedly large API bill, discovered in a monthly invoice rather than a monitoring dashboard. Every agent system needs an explicit budget — max steps, max cost, max wall-clock time — and a defined behavior for what happens when the budget runs out, decided before launch, not improvised in an incident channel afterward.
04The prototype-to-production gap is a testing gap, not a model gap
When a demo works and the production version doesn't, the instinct is to blame the model — try a newer one, add a bigger context window, throw more prompt engineering at it. That's rarely the actual cause. The gap is almost always: the demo's test cases weren't representative, the failure modes weren't instrumented, or there was no automated way to catch a regression before a user did. Swapping the model treats a symptom. The system still has no way of knowing when it's wrong.
What we build before we consider a system launch-ready:
- An evaluation harness that runs against real, continuously sampled traffic — not a fixed set built once
- A retrieval-quality metric tracked over time for anything doing RAG, not just checked once during development
- An explicit stop condition and cost budget for every agentic loop
- Monitoring that treats "system was confidently wrong" as a distinct, trackable event — not lumped in with server errors, since it's invisible to standard uptime monitoring
- A defined review cadence where a human actually reads a sample of real outputs, not just the ones a user complained about
None of this is exotic. It's the same discipline that's existed in software testing for decades, applied to a class of system where the failure mode is quieter — a wrong answer looks exactly like a right answer until someone checks. The teams that get burned aren't the ones who chose the wrong model. They're the ones who tested a system built for a demo, and shipped a system that has to survive a Tuesday afternoon.