Skip to content
Agentic Control Plane

We Benchmarked 13 Models on Real Agent Runs

David Crowe David Crowe · · 9 min read
benchmark model-selection cost agents
Share X HN LinkedIn

How a model does on one tool call tells you almost nothing about whether it can run an agent. We tested 13 models both ways in June 2026, and the rankings came out very different.

Most builders pick an agent’s model on instinct: one good model for everything, or always the best one. A single agent run is several kinds of work. Choosing a tool, pulling fields from a web page, writing a query, summarizing, and holding a multi-step plan together are different jobs. They aren’t equally hard or equally costly.

We tested the current flagships from Anthropic, OpenAI and Google, plus strong open models (Llama, DeepSeek, Qwen, GLM). Each ran as isolated tool calls across seven categories, and as a real agent doing a multi-step job end to end.

Two ways to test a model

Isolated microtests measure one decision at a time. Given a question and six tools, pick the right one. Given a schema, write correct SQL. Given messy HTML, extract clean JSON. We grade each from 0 to 1, with deterministic checks where we can and a panel of three judge models for the prose tasks. Cost is real dollars: live pricing times actual tokens.

The agent eval asks whether a model can run an agent. It’s a multi-turn loop where the model searches, reads what came back, decides what’s next and finishes the job. Every step depends on the last. We ran each scenario three times and report the spread, because one lucky run is a data point, not a ranking.

Finding 1: single-call skill doesn’t predict agent skill

On the microtests nearly everyone did fine. Across all seven categories every model landed between 0.83 and 0.95. When capable models tie, the cheapest competent one wins.

Then we ran the same models as agents:

Model Isolated microtest avg Agent completion
Claude Opus 4.8 0.95 0.77 ±0.08
Claude Sonnet 4.6 0.95 0.66 ±0.32
Claude Haiku 4.5 0.94 0.65 ±0.22
GPT-5.2 0.94 0.52 ±0.37
GPT-4o 0.92 0.50 ±0.24
Qwen 2.5 72B 0.94 0.48 ±0.42
GPT-5 mini 0.95 0.47 ±0.34
Llama 3.3 70B 0.92 0.43 ±0.37
GLM 5.2 0.95 0.43 ±0.43
GPT-4o mini 0.92 0.42 ±0.36
Gemini 2.5 Pro 0.85 0.35 ±0.30
Gemini 2.5 Flash 0.94 0.22 ±0.15
DeepSeek V3.2 0.83 0.06 ±0.06

The microtest column is nearly flat. The agent column runs from 0.77 down to 0.06.

DeepSeek scores 0.83 on isolated calls and 0.06 as an agent. It writes a fine query in a vacuum, then never drives a loop to the finish. Qwen and GLM ace the microtests (0.94 and 0.95) and finish fewer than half their agent runs.

If you pick your loop model off a function-calling leaderboard, which tests isolated calls, you’ll pick the wrong one.

Same models, two tests: isolated call vs full agent run00.250.50.751Claude Opus 4.8Claude Opus 4.8 isolated: 0.95Claude Opus 4.8 agent: 0.77 ±0.080.77Claude Sonnet 4.6Claude Sonnet 4.6 isolated: 0.95Claude Sonnet 4.6 agent: 0.66 ±0.32Claude Haiku 4.5Claude Haiku 4.5 isolated: 0.94Claude Haiku 4.5 agent: 0.65 ±0.22GPT-5.2GPT-5.2 isolated: 0.94GPT-5.2 agent: 0.52 ±0.37GPT-4oGPT-4o isolated: 0.92GPT-4o agent: 0.5 ±0.24Qwen 2.5 72BQwen 2.5 72B isolated: 0.94Qwen 2.5 72B agent: 0.48 ±0.42GPT-5 miniGPT-5 mini isolated: 0.95GPT-5 mini agent: 0.47 ±0.34Llama 3.3 70BLlama 3.3 70B isolated: 0.92Llama 3.3 70B agent: 0.43 ±0.37GLM 5.2GLM 5.2 isolated: 0.95GLM 5.2 agent: 0.43 ±0.43GPT-4o miniGPT-4o mini isolated: 0.92GPT-4o mini agent: 0.42 ±0.36Gemini 2.5 ProGemini 2.5 Pro isolated: 0.85Gemini 2.5 Pro agent: 0.35 ±0.3Gemini 2.5 FlashGemini 2.5 Flash isolated: 0.94Gemini 2.5 Flash agent: 0.22 ±0.15DeepSeek V3.2DeepSeek V3.2 isolated: 0.83DeepSeek V3.2 agent: 0.06 ±0.060.06isolated tool-call scoreagent-run completion
Gray dot: average score on isolated single-tool-call microtests. Indigo dot: completion rate on full metered agent runs (±SD in the table above). The left column of the leaderboard barely moves. The right column is the ranking that matters.

Finding 2: cheap models were erratic, not worse

Look at the ± column. Opus completes at 0.77 ±0.08, so it’s steady. The cheaper models swing: GLM ±0.43, Qwen ±0.42, Llama ±0.37.

A model that finishes a task 0% of the time on one run and 90% on the next is a coin flip. A coin-flip orchestrator is worse than a reliably mediocre one, because you can’t build on it. A single run hides this. GLM looked like a perfect 1.00 until we ran it three times and got 0.43 ±0.43.

So here’s the routing rule the data supports. Keep one consistent model on the loop. Send the leaf work (extraction, summarizing, a single call) to whatever is cheapest and competent.

In our set, Claude Haiku was the best value driver, at 0.65 completion for a fraction of frontier cost. Opus was the most reliable, for hard, long-horizon agents. The priciest model is rarely the best buy for the loop, and never for the leaves.

The Berkeley Function-Calling Leaderboard also reports a drop-off in multi-turn tasks. Our addition is measuring it on full agent runs, with repeats.

Finding 3: in production, the loop is where the money goes

We checked the benchmark’s premise against a real corpus: 285,000+ tool calls metered through our gateway. One frontier model on the orchestration loop was 77% of the spend across 30% of the calls. The cheap model did most of the work for a few percent of the cost.

That’s why the loop model is the one decision worth getting right. The full breakdown is in the cost teardown and the loop tax.

What we don’t trust yet

Treat this as a first version.

  • Only two agent scenarios. A real test would cover a dozen kinds of agent: a scout, a coder, a multi-agent crew. The harness is built. The scenario set needs to grow before any ranking is final.
  • Simulated tools. The agent eval uses mocked tool backends so runs are repeatable. The better test scores models on real outcomes. Every real call’s cost and result is recorded, so that’s the next version.
  • Judges aren’t humans. Three judge models beat one, but the completion and prose scores still need a human spot-check before you bet on a single row.

What we’d stand behind: isolated skill doesn’t predict agent skill, cheap models were erratic on the loop, and the expensive model is rarely the best buy. Put a consistent model on the loop and a cheap one on the leaves.

To check this on your own workload, you need to see what each call costs. That’s the job of agent cost monitoring. ACP logs every tool call your coding agents make, and prices the model calls it handles. One command sets it up:

curl -sf https://agenticcontrolplane.com/install.sh | bash

Running coding agents? ACP records each tool call its hook sees and makes the dangerous shell commands (a force-push, destructive SQL, curl | sh) wait for you. One command, free up to 5 agents:

macOS · Linux · WSL
curl -fsSL https://agenticcontrolplane.com/install.sh | bash
Windows · PowerShell
irm https://agenticcontrolplane.com/install.ps1 | iex

What the installer writes  ·  Getting started

Share X HN LinkedIn
Get the next data drop
What agents actually cost, new tool-surface captures, and the occasional incident post-mortem — sent when we publish something worth your inbox, not on a schedule. Unsubscribe anytime.
Share: Twitter LinkedIn
Related posts

← back to blog