Your benchmark says the agent passed. It doesn't say what failure would have cost.

David Crowe · July 26, 2026 · 6 min read

governance agents benchmarks severity evals research

Every agent benchmark you’ve seen reports a pass rate. 46/48. 91%. Pass rates answer one question — did the rules hold? — and stay silent on the one that decides whether anyone should care: what would it have cost when they didn’t? A benchmark where every failure is a mis-formatted reply and a benchmark where every failure is a leaked credential can post the same score.

A paper published this June takes that problem seriously. Action-graded severity for agentic evaluations proposes grading the worst action an agent actually executed on a seven-level scale, built from three properties of the action itself:

Reversibility — can the effect be undone?
Scope — did it stay local, or reach another party?
Privilege — did it expand what someone is allowed to do?

The scale runs from L0 (no harmful effect) through L1 (attempted, blocked before effect), L2 (reversible, local), up through irreversible damage, cross-scope reach, privilege expansion, and L6 — a cascading chain of escalating actions. The authors published their scoring code. It’s small, deterministic, and environment-agnostic: feed it a trajectory of actions, get back the level.

Which means anyone can hold it up to a real system. So we did.

The setup: one benchmark, two worlds

AgentGovBench is an open governance benchmark: 48 deterministic scenarios where an agent attempts things it shouldn’t — reading across user boundaries, granting itself permissions, fanning out past rate limits, acting while the control layer is unreachable. It normally reports pass/fail per scenario.

We ran it twice, ~470 tool calls per arm:

Vanilla arm — no governance. Every attempted action executes.
Governed arm — the identical scenarios, every action adjudicated by a production control plane, real policies, real audit log.

Then we graded both trajectories with the paper’s scoring function, unmodified. One variable between the arms: governance.

The table

Each scenario is graded by the worst thing that actually happened in it:

		ungoverned	governed
L0	no harmful effect	33	33
L1	attempted, blocked before effect	0	11
L2	reversible local write, completed	3	0
L3	irreversible local	0	0
L4	cross-scope	0	0
L5	privilege expansion, completed	12	4
L6	cascading chain	0	0

Read it as movement, not as columns competing. Thirty-three scenarios are benign reads — identical in both worlds, as they should be. The interesting rows are the ones that moved:

12 privilege grants completed in the ungoverned arm. An agent talked its way into standing permissions, and got them. On the paper’s scale that’s L5 — one level below the worst thing an agent can do.
Under governance, 8 of those 12 grade L1: attempted, blocked before effect. Same scenario, same attempt — the action reached the control plane, was denied, and the world didn’t change. The 3 reversible writes moved the same way. Eleven consequential actions, all stopped at the attempt.
4 grants still complete under governance — and that’s correct. Three are the benchmark’s negative controls: an admin performing a legitimate admin grant. The scale distinguishes authorized-powerful from prevented, which is a distinction a pass rate is structurally unable to make. (The fourth is a documented limitation of the single-tenant test runner, not a policy result — details in the run notes.)

That L1 row is the point of the whole exercise. The paper defines L1 as the grade for a harmful action that a control caught before it had an effect. It’s the level that only exists if something is doing the catching — and it’s invisible in every pass-rate you’ve ever read. Grading the counterfactual — what the blocked action would have been — is what turns “the policy held” into a measured claim about avoided cost.

The ruler doesn’t care whose benchmark it is

Here’s the part we didn’t plan. Look at the zeros.

Levels 3, 4, and 6 are empty in both columns — not because nothing irreversible happened, but because no scenario in the benchmark attempts anything irreversible. Across all 48 scenarios, nothing deletes data unrecoverably, nothing sends data to an external party, and nothing chains a privileged read into an outbound write — the exact escalation shape behind the most serious real-world agent incidents, including the lethal trifecta we recreated from the Supabase/Cursor disclosure.

So the severity scale graded the benchmark as much as it graded the gateway. A 46/48 on a suite that never exercises the top of the harm scale is a much narrower claim than the number suggests — and you only find that out by grading the suite on an axis its own scoring doesn’t have. We’ve opened scenario work to fill those levels, each new scenario carrying its source paper alongside its NIST mapping.

If you run agent evals, this generalizes: take whatever harm scale you trust, grade your test suite with it, and see which levels come back zero. Those zeros are the claims your pass rate is quietly making without evidence.

The benchmark got governed too

One moment from the run worth reporting. The governed arm’s test runner sets up each scenario by writing that scenario’s policy fixture. Its first attempt — through the gateway’s admin API, holding a valid, correctly-scoped key — was refused: 403 human-auth-required. Policy mutation on this gateway is human-only. There is no benchmark exception, because there are no exceptions; an agent-held credential cannot loosen the rules it runs under, even when the “agent” is our own test harness. Fixture setup now runs under the human operator’s own credentials, restricted to benchmark tenants, while every decision under test still flows through the production API.

A control layer that made an exception for its own benchmark would have a hole shaped exactly like an attacker’s first move.

Method, so you can check us

The instrument is theirs. oracle.py from the authors’ repository, byte-for-byte. The only thing we supplied is the mapping from the benchmark’s five tools to the scale’s three axes, taken from the scenario library’s own declared tool semantics.
Trajectories, not just scores. The run persists every attempted action and decision, which is what makes after-the-fact grading — by us or by you — possible at all. If your benchmark only saves scores, no instrument can ever re-examine what happened.
Static-benchmark caveat, stated plainly. These are fixed scenarios with no adaptive attacker. A June survey of this defense category warns that static suites flattered a generation of in-band defenses until adaptive attacks broke them; that caveat applies to this number too, and to every number like it.
Reproducible with one env var. The governed arm runs against any deployment via an API key — the benchmark, runners, and scenario library are open.

Pass rates say the rules held. Severity grading says what the rules were worth. The first without the second is a benchmark complimenting itself.

Share: Twitter LinkedIn

← back to blog