What 588 Model-Judged Tool Calls Taught Us About Approval Prompts
Running Claude Code? See, control, and price every tool call it makes — allow/ask/deny policy, a per-session bill at API rates, and a full audit log, in one command:
curl -sf https://agenticcontrolplane.com/install.sh | bashirm https://agenticcontrolplane.com/install.ps1 | iexFull Claude Code guide → · see your first governed call → · free up to 5 agents
Approval prompts have a quiet failure mode. When we replayed one workspace’s approval history, 42 inline prompts had gone unanswered or been declined. Twenty-five of them were abandoned: the developer had moved on, the prompt timed out, and the agent carried on with something else.
That’s the problem with asking about everything: people stop reading the questions. “Just add approvals” turns every action into a ticket, and a ticket queue nobody reads is not a control.
So for eight days we let a model answer some of them.
The setup
Two layers, in a fixed order.
Code first. Rules you write — allow, ask, deny per tool — plus floors that hold destructive shapes no matter what. These are deterministic and they run before anything else. A call your rules deny never reaches the model. A call a floor holds never reaches the model.
The model only in the gap. Some calls land on “ask”: your rules don’t say yes and don’t say no. Those, and only those, go to a decision model — Jev, from TypeSafe, which returns a typed choice (allow / ask / deny), a risk label and a calibrated confidence. We send it the redacted tool input, the harness name and the working-directory name. Never the transcript.
In decide mode, by default, the model can turn an ask into an allow. That’s the only direction it moves anything, and only when all of these hold:
- the call is read-only, and our command classifier independently agrees it’s a read
- confidence is at least 0.95
- no structural hold fires — a command hidden inside a wrapper (
bash -c,xargs,eval), project code being run (npm test,make,node scripts/…), a sensitive read (shell history, credential paths), or text that looks addressed to the judge - someone is actually at the keyboard: the harness reports an attended session
Anything else stays an ask. Writes always ask.
A session, from the audit log
The first clear on the workspace, 2026-10-16 at 21:57:55 UTC:
Bash gh pr view … --json
decision allow
reason cleared by judge: allow p=0.99 risk=readonly conf=0.98
No prompt. The agent got its answer and kept going.
Later in the same workspace, a compound command that read something and then ran a project script:
reason project code runs (node scripts/…)
Held. The model’s verdict didn’t matter; running code from the repo is a structural hold, and it asked.
Both rows are quoted from the workspace's audit log, with the command shortened (…); they are log text, not console screenshots.
That’s the whole design in two rows. Routine reads get out of the way. Anything that executes, writes or reaches somewhere sensitive still comes to you.
Eight days of numbers
One workspace, one developer, Claude Code sessions, 2026-10-16 to 2026-10-24.
judged calls 588
model said allow 467
model said ask 110
model said deny 11
cleared without a prompt 220 (37% of judged)
judge latency p50 216 ms · p95 374 ms
backend failures 0
What it cleared was unglamorous, which is the point: git fetch (29), gh pr view (26), gh pr checks (24), git grep (22), git status (20), git show (14), gh api GETs (14), gh issue view (10), git worktree (9), git log (9), git diff (8).
The judge cost about $0.00003 per call. These eight days cost under two cents; a day at the default 2,000-call cap would be about six.
The model said yes 467 times. 220 got through.
This is the number we’d look at first in anyone else’s system.
model said allow 467
cleared 220
held 247
of which, checks on the verdict itself:
allow, but labelled destructive 81
command inside a wrapper 61
classifier disagrees it's a read 21
project code runs 9
risk = exfiltration 9
the rest: held for other reasons (the confidence bar, ceiling guards and the rest), not itemized
Over half of the model’s yeses were held by the checks around it. Some of those are the model being inconsistent with itself — “allow” with a risk label of “destructive” is a contradiction, and a contradiction is a hold. Most are shapes a language model reads as routine and a parser reads as “this runs something.”
The test set shows why the confidence bar is set where it is. Commands like env, printenv, ls ~/.ssh and grep AKIA ~/.aws came back labelled read-only, at confidence between 0.38 and 0.88. They are reads. They are also reads of secrets. The 0.95 bar held every one — and those paths are now on the never-clear list regardless. Across a 240-case set built to probe this — reads, near-reads, and things dressed up as reads — none of the cases marked must-not-clear cleared.
The model is good at telling a routine git log from something odd. It is not a parser, and it is not your policy. Code-based and model-based control do different jobs, and the useful part is the order they run in.
It’s opt-in per workspace and clears on Claude Code and Pi, the harnesses that report whether a person is in the session; a tool type needs 5 of your approvals before it clears, and two denials, if they’re more than 5% of its approvals, take it back off. Setup, modes and what is sent: model-judged approvals.
What this data doesn’t tell you
One workspace. One developer’s habits, heavy on git and GitHub CLI reads. Eight days. Claude Code only. Your clear rate will depend on how much of your agent’s day is reads your rules already ask about — if your rules allow git status outright, the judge never sees it. The latency numbers are the judge call alone, from one region.
The best check is your own traffic: turn on shadow for a week, then compare the model’s verdicts with your own answers on the same calls.
I think of this as the agentic control plane doing the boring part of approvals, so the prompts that remain are the ones worth reading.