A team we advise had a consumer buying assistant passing eight out of eight evaluation scenarios. It ran on Sonnet. It looked finished. We told them to move it down to Haiku. The score dropped to one out of eight. Four weeks later it was back at eight out of eight, on the smaller model, at roughly a tenth of the token cost. Then they moved it back up to Sonnet out of curiosity and it ran better than it ever had on the original build. That sequence is not a quirk of their system. It is the most reliable way we know to find out what is actually wrong with an agent, and almost nobody does it, because it feels like sabotage.
The strongest model in the room hides your bugs
Our CTO, Anton Titov, frames it as a question of what a model does when it hits a wall.
A frontier model’s objective is to complete the task. Give it a broken tool, an ambiguous parameter, or a gap in your data, and it will route around the obstacle, because routing around obstacles is exactly what it was trained to do. We watched one bypass a broken tool by chaining two others together to produce an answer. Technically impressive. It also returned a result that should never have existed, cost extra to generate, and gave us no signal that anything was wrong.
A smaller model does not have that capacity. It says it cannot complete the task and stops. That is not a worse outcome. That is a diagnostic.
The buying assistant team saw this in reverse first. They had upgraded to a newer, smarter model expecting cleaner behavior, and got worse tool adherence instead, because the model decided it knew a faster path through their workflow and skipped a step. Smarter meant more willing to improvise, and improvisation is the enemy of a system that has to produce the same output shape every time.
What the failures actually revealed
The interesting part of the drop from eight to one is not the number. It is what the number pointed at.
One tool in their stack failed roughly ten percent of the time on the larger model. On Haiku it failed eighty percent of the time. That gap was not a model limitation. It was a bad description and badly specified arguments, sitting there the whole time, invisible because the bigger model had been quietly compensating. Once they rewrote the description and constrained the schema, that same tool went to around ninety-five percent on the small model.
The rest of the recovery followed the same pattern. Getting from one back to four was tool work, mostly naming and parameter definitions. Getting from four back to eight was the system prompt, which turned out to be fourteen thousand characters of accumulated patches mixing conversational style, behavioral goals, and tool guidance in no particular order. The larger model had been absorbing all of that. The smaller one could not, so it surfaced.
None of those were model problems. Every one of them would have shipped to production undetected.
Why moving back up is easy and moving down is a rewrite
The direction matters more than most teams expect.
Build on a small model and you are forced to write clear schemas, unambiguous tool names, and a tight prompt, because nothing else works. Every one of those constraints is still valid when you move up a tier. Scaling from a smaller model to a larger one inside the same family is close to painless.
Build on a frontier model and you are unknowingly depending on its capacity to reason past your mistakes. When cost or latency eventually forces you down a tier, all of those hidden dependencies surface at once, and you are not tuning anymore, you are rebuilding.
There is a cost dimension too, and it stops being academic at volume. Input tokens on a mid-tier model run around ten times what the cheapest tier costs. During development that difference is invisible. Multiply it by real session volume and model tier stops being a quality decision and becomes a unit-economics decision.
How to run this on your own system
Pick the smallest model you have access to and point your existing agent at it. Expect the score to collapse. That collapse is the map.
Work the failures in order. Tool names and parameter descriptions first, because that is where most of the damage lives. Then the system prompt, checking specifically for accumulated patches and mixed concerns. Then composition, meaning whether one agent is being asked to hold conversation, memory, data collection, and formatting at the same time.
Run this against a fixed set of scenarios with known correct outcomes, and run each scenario several times, because the same input will not always produce the same path. A single pass tells you nothing.
When you get back to your original score on the small model, try the large one again. If the system is actually fixed rather than patched, the improvement is obvious, because the model is no longer spending capacity fighting your architecture.
This is the same sequence we use when we take over an agent automation build that has stalled, and it is usually the first week of the engagement. It is also why we architect agent systems on durable orchestration, where every tool call and retry is traceable. You cannot debug what you cannot see, and the architecture patterns that hold up under production load all assume you can.
If your agent works and you do not know why
That is the actual risk here. An agent that passes your tests on a frontier model has told you almost nothing about whether it will hold up, because you have not separated the quality of your system from the capacity of the model papering over it.
Once the small model has handed you a list of failures, the next two questions are where they live. Most of them sit in the prompt, which is covered in the four anti-patterns that rot a system prompt. The rest are composition problems that no prompt edit will fix, covered in why your agent is dropping steps.
If you want a read on where your own build sits, our AI Readiness Audit covers exactly this, and you can bring the system to the founders directly. We have also written about why pilots and production systems fail differently, which is the broader version of this problem.
