overview / benchmarks / false rejects
Latency says the gate is cheap. The ledger says how much it did. Neither answers the question a stranger actually asks — will it get in my way — and that question decides whether a guard is still installed on day two. A tool that refuses honest work once a day is uninstalled by lunch, however correct it is the rest of the time.
BENCHMARK.md §3b · marked proven. Machine: Apple M2, arm64, macOS (Darwin 24.2.0). Toolchain: Apple clang 17.0.0, node v26.5.0, python 3.14.6.
rabadon records both halves of the answer without being asked to. A STOP is a refusal that happened. rabadon wrong <rule> "<why>" writes a WRONG_REFUSAL — the operator, after the fact, saying that refusal should not have fired. The false-reject rate is one divided by the other, and it needs no survey, no instrumentation and no memory of what happened.
Measured 2026-09-05. The ledger is live and grows under the operator's own work, so a count read tomorrow is a larger count — the date is part of the claim, not decoration.
| window | refusals | declared wrong | rate |
|---|---|---|---|
| all 28 days with traffic (2026-08-04 → 09-04) | 524 | 70 | 13.4 % |
| the same, counting only rules that still refuse | 524 | 59 | 11.2 % |
| since 2026-08-28 | 192 | 1 | 0.5 % |
The middle row exists because 11 of the 70 name a rule that never appears as a refusal: ctest-red-block, tests-red, tests-are-RED and two more spellings of what is red-base today. The rule was renamed; its refusals moved to the new id and the operator's verdicts stayed on the old ones, which puts a numerator over a denominator it does not belong to. The 13.4 % row is still the one quoted, because the correction moves the number in this project's own favour and a guard that rounds its own error rate down has stopped measuring itself.
Both rows are real and the difference between them is the whole story: 69 of the 70 wrong refusals happened on two days, 26 and 27 August. That is what tuning a rule against real work looks like — the rules that were wrong were ctest-tail-hides-verdict (37 wrong of 152), red-suite-test-write (7 of 17) and red-base (10 of 65), each argued down with a written reason on the ledger. In the 192 refusals since, one was called wrong.
What those 192 stopped, in the operator's own repos: 34 git commit -m "wip" against a project whose guard demands a real message, 34 actions on a red test base, 24 truncating redirects onto a tracked file, 22 force-pushes to a shared branch, 21 recursive deletes resolving outside the project, 6 blind in-place source rewrites, 5 attempts to take apart a project's own .rabadon/ law, and one edit that would have put red-base into disabled[].
Reproduce, on your own ledger, offline:
It reads ~/.rabadon/spool/*.jsonl and nothing else. On a machine that has never refused anything it prints that, rather than a rate computed from zero.
What this number is not. It is one operator, one machine, 148 projects on the ledger when it was first measured (2026-09-03), and the same person who wrote the rules is the one judging them — a stranger's false-reject rate on a codebase rabadon has never seen is unmeasured, and no amount of local traffic can stand in for it. The honest claim is narrow: on the work this machine actually does, since 2026-08-28 the gate refuses about one honest action in a hundred and ninety-two.