Two of our more recent AI adoption discussions with prospective customers last month hit the same wall but from opposite directions. One was with a 1,500-person industrial manufacturer deciding how to build an estimating and knowledge base system. The other was with a startup whose consumer buying assistant was demoing well internally but then started ignoring its own tools in front of the people about to fund it.
What we saw was different industries and different stacks, but had identical failures. Their demo worked but the system underneath it did not.
Most AI pilots die in the gap between those two sentences, and almost never because the model was wrong. They die for four major reasons we have started to see as patterns. The data was never modeled as something a machine could act on. One agent was handed the whole job. The team started by using the smartest model available. And, lastly, the feedback loop ran slower than the system drifted. If your pilot is stalling somewhere between the demo and real users, one of these four is the reason and a good place to dig into.
Key Takeaways
- AI pilots almost never fail because the model was wrong.
- Unstructured retrieval is a commodity capability. Production work needs a deterministic data model underneath it.
- One agent holding too many tools writes randomness into the system through overlap, ambiguous naming, and constant tool visibility.
- The smartest available model often hides defects a smaller model would have surfaced immediately.
- A slow feedback loop lets a system drift further from the business than any single failed run reveals.
Mistake 1: You Modeled a Demo, Not Your Business
This gets missed because retrieval is easy now. Point a model at a folder of PDFs, wire up semantic search, and it answers questions. That is a real capability but it is also the commodity part. It is built for unstructured content where “I found something close” is an acceptable outcome.
Production work is different. When a system has to hold state, take an action, and be provably correct afterward, close fails and is never good enough. That requires a deterministic model of your process, which means you have actual schemas and actual constraints. Agents sit on top of it. They do not replace it.
The industrial manufacturer’s hardest problem was estimating, not retrieval. Their team was busy choosing a platform when the real work was getting a decade of pricing, part relationships, and shop-floor logic into a shape a machine could reason over. That data modeling work does not change based on which engineer, tool or cloud you pick. You do it, or the agent invents its way around the gaps.
If you are looking at a folder of tribal company knowledge hoping an agent will make sense of it, that is a knowledge and rules engine problem. It is engineering and not prompting.
Mistake 2: You Gave One Agent the Whole Job
The buying assistant showed us the second failure pattern live. A user handed the agent every piece of information it needed, and the agent still skipped the tool that renders the comparison table and answered in freeform prose instead. This doesn’t mean you have a broken tool but that you have too many jobs in one place.
That agent was collecting parameters, deciding which data source to query, filling structured fields, formatting output, and holding a conversational tone. Every exposed tool is cognitive load, because the model has to decide on every turn which one applies. Three things made it worse, and they are the three we see most often.
Two tools did overlapping work, where one was returning raw results and one was formatting them. Any time two tools can plausibly satisfy the same intent, you have written randomness into the system. The naming was ambiguous, with the formatting tool called get_quote even though the quote had already been retrieved, so semantically there was nothing left to get and the model moved on. And every tool stays visible at every stage, when a tool that cannot legitimately be called right now should not be on the menu right now. Gate the tool surface by stage and the model physically cannot drift. That does break the provider’s prompt cache and raise your per-call cost, which is usually a trade worth making.
The underlying fix is process decomposition. Break the work into stages, aim for one tool per stage, and test each agent in isolation against its own scope. You cannot verify a seventeen-step chain end to end on day one. You can verify step four.
Take a workflow with seventeen steps and give every step a 99% success rate, which sounds excellent. Multiply it out and you are shipping something that fails roughly one session in six.
The instinct is to fix that by upgrading the model. It usually makes things worse and always makes them more expensive. A frontier model’s objective is to complete the task. If your data has a hole in it, a small model reports that it does not have the information and the workflow fails, which is the correct behavior and tells you exactly where the hole is.
A larger model will likely infer around the gap, produce something plausible, and hide the defect until volume creeps in and finds it. We watched a model bypass a broken tool by chaining two others together. For sure, it’s an impressive engineering instinct. But at the same time, it also returned an answer that should never have existed and charged us for the privilege.
The buying assistant team hit this directly. They moved to a newer, smarter model expecting better results and got worse tool adherence, because the model decided it knew a shortcut.
A great rule of thumb is to start with the smallest model that could plausibly work and build your naming, schemas, and constraints around its limitations. Moving up a tier later is nearly painless. Moving down, after you have built on capabilities you did not know you were relying on, is a rewrite and costly. And once you are looking at thousands of concurrent users, model tier stops being a quality decision and becomes a cost decision so review that decision today.
Mistake 4: Your Feedback Loop Is Slower Than the Drift
Because models are not deterministic, the quality of an AI build is a direct function of how much correction it receives while being built. That makes vendor cadence a technical risk and not a scheduling preference.
If a partner tells you they will come back in two months with a prototype, you should brace yourself for trouble. Either the system drifts away from what the business needs, or you become dependent on work you have no way to assess or maintain effectively later. Six months later you have a quarter-working system sitting inside an environment you already owned, and nobody can tell you which quarter is real. Insist on weekly checkpoints against modules small enough to judge on their own, or your engineers and non-technical stakeholders will end up with a game of hot potato.
You can apply the same logic to knowledge gaps. Rather than waiting for users to find what your knowledge base is missing, you can embed what you have, cluster it, and ask a large model where the coverage holes are. This is the smart way to do it. You can run that overnight and then get a list of questions to take to your experts. Our advice is to interview them on a call and pipe the recorded transcript in, instead of asking senior people to type long explanations into a chat window.
Four Mistakes, Four Fixes
| Mistake | Where It Surfaces | What Breaks | The Fix |
| Unmodeled Data | Data layer | Agent invents around the gaps | Deterministic schema before agent design |
| Overloaded Agent | Tool-selection layer | Model picks the wrong tool, drifts into prose | Decompose into staged, single-purpose agents |
| Wrong Model Tier | Model layer | Larger models mask defects instead of surfacing them | Start smallest, upgrade only when proven necessary |
| Slow Feedback Loop | Delivery cadence | System drifts faster than corrections arrive | Weekly checkpoints against small, judgeable modules |
The Fix Is Judgment, Not a Better Prompt
If the demo works and production does not, the cause is almost always unmodeled data, an overloaded agent, a model tier picked for the demo, or a feedback loop too slow to catch drift. None of those are fixed by a better prompt alone.
Every system we build like this is architected so the client owns the data model, the agents, and the resulting output, with no dependency on us to keep it running.
We have written about why pilots and production systems fail differently and where humans belong in agent workflows.
Is Your Pilot Stuck Between Demo and Production?
If you want this applied to your own system, our AI Readiness Audit maps one workflow, the data behind it, and the decomposition that would let it survive real users. Or bring us the system that broke and we’ll walk through it together.
