Run It Ten Times: What a Coding Harness Does That It Doesn't Tell You
Run agents? ACP lets you see, control, and price every tool call they make — allow/ask/deny policy, per-session cost, and a full audit log for Claude Code, Cursor, Codex, and OpenClaw, in one command:
curl -sf https://agenticcontrolplane.com/install.sh | bashGetting started → · see your first governed call → · free up to 5 agents · prefer fully on-device? add --local
Benchmarks run a harness once per task and take its word for what happened. That is fine for a leaderboard and useless for the question a team actually has before handing an agent real work: what does this thing do when it runs unattended, again and again, with a clock on it?
We ran five coding harnesses on the same task with the same model, ten times each, inside the Terminal-Bench runner, with a gateway recording every tool call independently of the harness. The pass rates were the least interesting output. What the repetition and the second record showed was a set of behaviors none of the harnesses report about themselves.
sanitize-git-repo, per harness, at the benchmark's 15-minute agent budget and at 30 minutes. DeepSeek V4 Flash via OpenRouter for every harness. Trials lost to upstream rate limiting are excluded and counted beside each bar. 2026-09-03 to 09-04.1. The clock decides as much as the harness
The task’s own budget is 15 minutes. At that budget Claude Code passed 2 of 10 and Qwen Code 2 of 10. With the same harnesses, same model, same task and a 30-minute budget, Claude Code passed 8 of 10 and Qwen Code 6 of 10.
Nothing about the harnesses changed. Claude Code’s successful runs take 12 to 19 minutes on this task; at 15 minutes it was finishing on the wrong side of the line half the time. A single-run benchmark score at the declared budget reads as “this harness cannot do this task.” Ten runs at two budgets read as “this harness needs seventeen minutes.” Those are different facts, and only the second one is useful to someone deciding how long to let an agent run.
2. Retry budget is a harness trait
Midway through the study the host serving the model began refusing requests with 429s. That was bad luck for the schedule and good luck for the study, because it put every harness under the same stress and they responded very differently.
| Harness | Refusals absorbed per trial, median | Outcome |
|---|---|---|
| pi | 64 | Retried with backoff and finished every run |
| Claude Code | 33 | Lost two of twenty runs to refusals; finished the rest |
| Qwen Code | 0 recorded | Finished every run |
| Codex | 1 to 2 | Quit; half its trials lost |
Codex tries a WebSocket first, gets refused, falls back to HTTPS, and after a couple of 429s reports “exceeded retry limit” and ends the turn. Seven of its fourteen trials at the 30-minute budget died that way, on a task it had barely started. pi, the harness with the smallest wrapper of the five, kept retrying for minutes and completed. Under a flaky provider, the retry policy decides who finishes before the model gets a vote.
3. One harness finished the task and never exited
OpenCode’s record reads 1 pass in 20 attempts. That number is an artifact. Every OpenCode run that reached the end of its turn, seven of seven, then sat idle until the runner killed it at the budget and logged a timeout. One of those killed runs had already written the correct files and passed the verifier anyway.
The proxy in front of the model showed every request answered, so nothing was hanging on the network. OpenCode’s own log stops after “turn complete” and stays silent for twenty minutes. On a trivial task the same invocation exits in three minutes. Something in the post-turn path does not resolve on larger sessions. A benchmark records that as a failure. A CI job records it as a hung runner. Neither tells you the work was done.
4. Cache economics belong to the harness and the host together
Qwen Code and pi cached 89 to 92 percent of their prompt tokens. Claude Code cached 29 percent on the same host, with the same model, on the same task. That is not Claude Code’s normal behavior: across 22 real Claude Code sessions on Anthropic’s API in the same week, 1.6 billion prompt tokens, the cache-read rate was 97 percent.
The difference is mechanism. Anthropic caches at breakpoints the harness marks explicitly and tolerates the prompt changing between them. A third-party host caches automatically only when the prefix is byte-identical, and Claude Code’s prefix moves between turns. Qwen Code and pi keep theirs still. So the cache column of any cross-harness comparison is a property of the pair, and anyone pointing Claude Code at a non-Anthropic backend through the base-URL override should expect their cache economics to change by a factor of three.
5. Everyone misses the same secret
When a harness finished and still failed, it was almost always the same miss. The task plants four credentials in three files; the AWS access key ID inside a large tokenized-dataset JSON was the one left behind, run after run, across harnesses. Search-by-pattern finds the obvious ones. The one that looks like data in a data file survives. Ten runs turn “it missed one” into “this is the one they miss.”
6. Headless Codex was running outside the control plane
This one came from the second record rather than the runs. Codex requires each hook to be trusted once, interactively, before it runs; under codex exec an untrusted hook is skipped without a message. Our first Codex runs through the control plane completed the task and produced no session in the gateway at all. The fix is a flag on the exec command. The lesson is that a CI runner with hooks installed for Codex runs outside the control plane until someone has opened the interactive session once, and nothing in the run will say so.
What the second record was for
Every one of these was invisible from inside the harness. The benchmark saw timeouts, not completed work. The harness saw its own retry limit, not a policy that lost half its trials. Claude Code’s built-in meter reported a session cost with an unknown basis; pi’s reported zero. The cache rate looked like a harness property until real sessions on a different host said otherwise.
The gateway sat outside all of that, recording each call as it happened, under each harness’s own client name, from inside the container. Its count of tool calls per run agreed with the harness’s own trajectory files in every case we checked, which is the boring result you want from a second set of books. Where it earned its keep was in what the harness did not write down at all.
Method
- Runner: Harbor 0.22, task
terminal-bench/sanitize-git-repo(find and replace four planted credentials in a cloned repo; tests require exactly three files changed and byte-identical results). Each harness ran as a Harbor agent subclass that installs the control plane’s hook inside the container before the trial. - Model:
deepseek/deepseek-v4-flashvia OpenRouter for every harness in its native wire format. Harness versions: Claude Code 2.1.259, Codex 0.153.1, OpenCode 1.18.27, Qwen Code 0.23.0, pi current on 2026-09-03. - Budgets: the task’s declared 900-second agent timeout, and 1,800 seconds. Both conditions were run for every harness. Trials that ended in an upstream rate-limit error before completing are reported as lost, not failed.
- Infrastructure caveat: OpenRouter routed every request for this model to a single host and did not fail over when it returned 429s, regardless of the account’s provider settings. Twenty-two trials were lost to that overnight and replaced in daytime runs. Retry behavior is reported from the harness’s own logs.
- What we did not do: ten runs per condition is enough to see the effects above and not enough to rank harnesses by a few points of pass rate. The model is a cheap one; absolute pass rates say more about it than about the harnesses. The policy layer had little to refuse on this task, and we did not test whether the planted credentials were flagged when read into context; both belong to a task chosen for them.