We Benchmarked 13 Models on Real Agent Runs
How a model does on one tool call tells you almost nothing about whether it can run an agent. We tested 13 models both ways in June 2026, and the rankings came out very different.
Most builders pick an agent’s model on instinct: one good model for everything, or always the best one. A single agent run is several kinds of work. Choosing a tool, pulling fields from a web page, writing a query, summarizing, and holding a multi-step plan together are different jobs. They aren’t equally hard or equally costly.
We tested the current flagships from Anthropic, OpenAI and Google, plus strong open models (Llama, DeepSeek, Qwen, GLM). Each ran as isolated tool calls across seven categories, and as a real agent doing a multi-step job end to end.
Two ways to test a model
Isolated microtests measure one decision at a time. Given a question and six tools, pick the right one. Given a schema, write correct SQL. Given messy HTML, extract clean JSON. We grade each from 0 to 1, with deterministic checks where we can and a panel of three judge models for the prose tasks. Cost is real dollars: live pricing times actual tokens.
The agent eval asks whether a model can run an agent. It’s a multi-turn loop where the model searches, reads what came back, decides what’s next and finishes the job. Every step depends on the last. We ran each scenario three times and report the spread, because one lucky run is a data point, not a ranking.
Finding 1: single-call skill doesn’t predict agent skill
On the microtests nearly everyone did fine. Across all seven categories every model landed between 0.83 and 0.95. When capable models tie, the cheapest competent one wins.
Then we ran the same models as agents:
| Model | Isolated microtest avg | Agent completion |
|---|---|---|
| Claude Opus 4.8 | 0.95 | 0.77 ±0.08 |
| Claude Sonnet 4.6 | 0.95 | 0.66 ±0.32 |
| Claude Haiku 4.5 | 0.94 | 0.65 ±0.22 |
| GPT-5.2 | 0.94 | 0.52 ±0.37 |
| GPT-4o | 0.92 | 0.50 ±0.24 |
| Qwen 2.5 72B | 0.94 | 0.48 ±0.42 |
| GPT-5 mini | 0.95 | 0.47 ±0.34 |
| Llama 3.3 70B | 0.92 | 0.43 ±0.37 |
| GLM 5.2 | 0.95 | 0.43 ±0.43 |
| GPT-4o mini | 0.92 | 0.42 ±0.36 |
| Gemini 2.5 Pro | 0.85 | 0.35 ±0.30 |
| Gemini 2.5 Flash | 0.94 | 0.22 ±0.15 |
| DeepSeek V3.2 | 0.83 | 0.06 ±0.06 |
The microtest column is nearly flat. The agent column runs from 0.77 down to 0.06.
DeepSeek scores 0.83 on isolated calls and 0.06 as an agent. It writes a fine query in a vacuum, then never drives a loop to the finish. Qwen and GLM ace the microtests (0.94 and 0.95) and finish fewer than half their agent runs.
If you pick your loop model off a function-calling leaderboard, which tests isolated calls, you’ll pick the wrong one.
Finding 2: cheap models were erratic, not worse
Look at the ± column. Opus completes at 0.77 ±0.08, so it’s steady. The cheaper models swing: GLM ±0.43, Qwen ±0.42, Llama ±0.37.
A model that finishes a task 0% of the time on one run and 90% on the next is a coin flip. A coin-flip orchestrator is worse than a reliably mediocre one, because you can’t build on it. A single run hides this. GLM looked like a perfect 1.00 until we ran it three times and got 0.43 ±0.43.
So here’s the routing rule the data supports. Keep one consistent model on the loop. Send the leaf work (extraction, summarizing, a single call) to whatever is cheapest and competent.
In our set, Claude Haiku was the best value driver, at 0.65 completion for a fraction of frontier cost. Opus was the most reliable, for hard, long-horizon agents. The priciest model is rarely the best buy for the loop, and never for the leaves.
The Berkeley Function-Calling Leaderboard also reports a drop-off in multi-turn tasks. Our addition is measuring it on full agent runs, with repeats.
Finding 3: in production, the loop is where the money goes
We checked the benchmark’s premise against a real corpus: 285,000+ tool calls metered through our gateway. One frontier model on the orchestration loop was 77% of the spend across 30% of the calls. The cheap model did most of the work for a few percent of the cost.
That’s why the loop model is the one decision worth getting right. The full breakdown is in the cost teardown and the loop tax.
What we don’t trust yet
Treat this as a first version.
- Only two agent scenarios. A real test would cover a dozen kinds of agent: a scout, a coder, a multi-agent crew. The harness is built. The scenario set needs to grow before any ranking is final.
- Simulated tools. The agent eval uses mocked tool backends so runs are repeatable. The better test scores models on real outcomes. Every real call’s cost and result is recorded, so that’s the next version.
- Judges aren’t humans. Three judge models beat one, but the completion and prose scores still need a human spot-check before you bet on a single row.
What we’d stand behind: isolated skill doesn’t predict agent skill, cheap models were erratic on the loop, and the expensive model is rarely the best buy. Put a consistent model on the loop and a cheap one on the leaves.
To check this on your own workload, you need to see what each call costs. That’s the job of agent cost monitoring. ACP logs every tool call your coding agents make, and prices the model calls it handles. One command sets it up:
curl -sf https://agenticcontrolplane.com/install.sh | bash
Running coding agents? ACP records each tool call its hook sees and makes the dangerous shell commands (a force-push, destructive SQL, curl | sh) wait for you. One command, free up to 5 agents:
curl -fsSL https://agenticcontrolplane.com/install.sh | bash
irm https://agenticcontrolplane.com/install.ps1 | iex