The Agent Optimization Manual
Measured field notes on where agent cost and reliability actually come from. The recurring finding: it’s the harness around the model, not the model itself — and it’s invisible unless you meter every call. Every number below is captured at the gateway, not estimated.
New entries when we have something measured worth publishing. Subscribe at the bottom of any post for the drops.
Your agent's bottleneck is almost never the model
The thesis, and why harness decisions dominate agent economics.
Don't give up on cheap models — give up on fragile loops
A cheap model at the same quality as one 14× more expensive per successful result, once the loop stopped quitting on a blank round.
One sentence in our system prompt doubled the agent's bill
A live budget counter was silently disabling prompt caching — and hurting the most expensive agents most.
Your agent's last move should be a tool call, not text
28% of runs delivered nothing, silently. Structured delivery made the failure rare and visible.
Methodology, in brief: real tool-calling agents run on a deterministic task with planted-fact scoring (no LLM judge), each run metered per call at the gateway — fresh vs cached tokens, delivery outcome, full trace. Findings are reported with the honest caveats (sample sizes, provider specifics, and the claims we had to retract when more data arrived). We build the control plane that produces this instrumentation; that's why we can run the experiments.