Don't Give Up on Cheap Models — Give Up on Fragile Loops
The Agent Optimization Manual · No. 3
Conventional wisdom says frontier models are worth their price for agent orchestration because cheap models are unreliable. We set out to measure exactly how unreliable, and the answer flipped the conclusion: the cheap model wasn’t unreliable. Our loop was unforgiving. Fixed properly, it did the same work at roughly a fourteenth of the cost per successful result.
The setup
Same research agent, same tasks, run on two Gemini tiers — 2.5 Pro and 2.5 Flash — with every model call and outcome metered. We split reliability from quality on purpose, because they’re different failures: an attempt is “did the run actually deliver a result,” and quality is “how good was the result when it did.” Conflating them hides exactly the thing we found.
The first cut looked like a clean win for Pro:
| Model | Attempt rate | Quality (when it delivered) | Cost/run |
|---|---|---|---|
| Pro | ~95% | 79% | 37¢ |
| Flash | ~40% | 79% | ~1–3¢ |
Look closely at the middle column. When Flash delivered, its quality equaled Pro’s — identical, 79% vs 79% on the best strategy. Flash wasn’t producing worse research. It was producing the same research and then failing to hand it over less than half the time. That’s not a capability gap. That’s a delivery gap, and delivery gaps are fixable.
The trace told us it was one bad round
We pulled every Flash run that failed to deliver and looked at the traces. The pattern was almost unanimous: the agent did 15 to 40 healthy, productive tool calls — real research, gathering the right facts — and then one round came back empty. No text, no tool call, nothing. And our loop, like most agent loops, read “the model returned no tool calls” as “the model is done,” recorded a finish, and ended the run one step before delivery.
Eleven of fifteen failures were that exact shape: a good run, one blank round, terminated. (The rest were nearly as recoverable — a couple wrote the answer as prose instead of calling the delivery tool, a couple hit the round cap while still legitimately working.) Almost none were “the model got confused and produced garbage.” The research was done. The loop just gave up at the worst possible moment.
This is the trap of treating an empty model turn as a completion signal. A model that has more work to do and momentarily returns nothing is indistinguishable, to a naive loop, from a model that’s finished — unless you define “finished” as an explicit act (see Manual No. 1). Until then, every transient blank round is a coin flip on whether your run survives.
The fix: don’t put your hands up
We added one rule to the loop: if a turn comes back with no tool call and no text, that is not a finish — nudge and continue. A single line back to the model (“you haven’t delivered yet; keep going and call the submit tool when you’re done”), bounded by a small retry cap so a genuinely stuck model can’t spin forever.
Then we reran Flash:
| Attempt (no recovery) | Attempt (+recovery) | |
|---|---|---|
| full-history | 50% | 100% |
| rolling-summary | 67% | 100% |
| prod-truncate | 17% | 67% |
| sliding-window | 17% | 33% |
On the two good context strategies, the nudge took Flash from a coin-flip to delivering every single time — and quality held (83% and 77%). The two weaker strategies improved but didn’t fully recover, for an honest reason: they throw away context the agent needs, so when the agent blanks, there’s less intact work to resume. Recovery can save a good run interrupted; it can’t save a strategy that already destroyed the evidence.
The number that should change your architecture
The right way to price an agent isn’t cost per run — it’s cost per successful outcome: what you actually pay, on average, for one good delivered result, counting the runs that failed and had to be retried. That’s the number your finance team feels.
- Pro, full-history: 37¢/run ÷ 89% success = 42¢ per successful result
- Flash + recovery, full-history: 3¢/run ÷ 100% success = 3¢ per successful result
Same quality — 83% vs 79% — for one-fourteenth the cost per successful result. The frontier premium we assumed we were paying for reliability evaporated the moment we stopped letting the loop quit on a blank round. For this workload, the correct architecture isn’t “use the expensive model because it’s reliable.” It’s “use the cheap model and make your loop resilient.”
What this generalizes to
Not every task survives this — some genuinely need a frontier model’s reasoning, and we’ll publish where the line falls. But the meta-lesson is architectural and it holds broadly:
- Measure cost per successful outcome, not per run. Per-run cost systematically flatters expensive-but-reliable models and punishes cheap-but-recoverable ones. The metric hides the arbitrage.
- Reliability is often a property of your loop, not the model. Before you pay 10–15× for a bigger model to “fix reliability,” check whether your loop is quitting on recoverable failures. Ours was.
- A blank turn is not a finish. Define done as an explicit act, and treat silence as “keep going,” not “stop.”
How we measured it
Every figure here is per-call metered cost and a boolean delivery outcome, captured at the gateway — which is what let us split attempt from quality, price by successful outcome, and pull the failure traces to see the one bad round. You cannot do this analysis on blended token counts and a “run finished” flag; the whole finding lives in the gap between “the run ended” and “the run delivered.” That gap is exactly what a control plane over your agents makes visible.
Part of The Agent Optimization Manual — measured lessons on making agents cheaper and more reliable, every number metered at the gateway, not estimated.
What is an Agentic Control Plane? → · Read Manual No. 1: end on a tool call →
- . Your Agent's Bottleneck Is Almost Never the Model
- . One Sentence in Our System Prompt Doubled the Agent's Bill
- . Your Agent's Last Move Should Be a Tool Call, Not Text
- . Don't Give Up on Cheap Models — Give Up on Fragile Loops · you are here