pre-registered protocol: does stop review lower cost per correct outcome?

the result

Credited repairs: 0. Unnecessary interventions: 0. Spent: $2.91. The verdict the frozen script wrote: evidence against at this scale.

From bench/evidence/2026-10-17-claude/review-ab/summary-main.json, written by the script frozen with the protocol below.

1. Question and arms

Does rabadon review (opt-in Stop review, same-agent repair) give more correct outcomes per list dollar than plain Claude Code on fresh long multi-step tasks?

• P (plain): claude -p <prompt> <FLAGS>
• R (review): rabadon review -- -p <prompt> <FLAGS> (launcher adds --session-id, sets RABADON_SEMANTIC_REVIEW=1; reviewer model = default opus, unset RABADON_REVIEW_MODEL).
• FLAGS identical in both: --model opus --effort low --output-format stream-json --verbose --include-hook-events --tools Bash,Read,Edit,Write --allowedTools Bash,Read,Edit,Write --strict-mcp-config --mcp-config '{"mcpServers":{}}' --disable-slash-commands --no-chrome --max-budget-usd 0.75. No --settings/--setting-sources/--system-prompt (launcher rejects the first two; default system prompt is what users get). Wall limit 600 s per turn, kill group.
• Both arms run in a project prepared by the same rabadon init --no-llm; .claude/settings.json sha256 must be equal across arms. Only difference: review on/off. Gate cost is local ($0).
• Dedicated CLAUDE_CONFIG_DIR for both arms: authenticated, no user CLAUDE.md, no memory, no global hooks (tried.md R7: global settings bound the gate into the "plain" arm; all rows void).
• N = 6 tasks, each run exactly once per arm. Order counterbalanced: tasks 1,3,5 P first; 2,4,6 R first. Strictly sequential; no parallel experiment sessions.

2. Follow-up turn (symmetric, scripted)

After the first process exits (any exit code), run the hidden checker on the work tree.

• PASS -> run ends. FAIL -> exactly ONE follow-up in the same session, same FLAGS, prompt exactly The result does not meet the request; re-read it. (no checker text, no hint). P: claude -p --resume <sid> ...; R: RABADON_SEMANTIC_REVIEW=1 claude -p --resume <sid> ... (launcher refuses --resume; dry run must show review hooks firing on the resumed turn).
• Applied to BOTH arms: a user would nudge either agent once. Plain-only follow-up would give P a second chance R lacks; symmetric removes that bias. Then the checker runs again = final.
• Not measured and not charged: the human time to notice the failure. This favours P, so a "lowers cost" result is conservative.

3. Metrics (primary first; computed by a script frozen with the protocol)

• Run cost = actor total_cost_usd (turn 1) + actor total_cost_usd (follow-up, if any) + every reviewer SEMANTIC_COST list cost for that session (both turns). A reviewer call with null cost is charged $0.25 (its soft cap). Budget-exhausted or timed-out turns count as is.
• Primary 1, cost per correct outcome: CPC_arm = sum(run cost over 6) / final passes. 0 passes -> infinite.
• Primary 2, final pass rate: final hidden-checker passes / 6, per arm.
• Credited repair (R only): a Stop snapshot fails the checker; reviewer returned correct with valid goal_ids+evidence_ids; SEMANTIC_CONTINUE was delivered; a later Stop snapshot in the SAME user turn (before any follow-up) passes. A fix after the scripted follow-up is NOT credited.
• Secondary, unnecessary intervention: SEMANTIC_CONTINUE delivered at a Stop whose snapshot already passes. Harmful if the final state then fails. Report count, extra reviewer+actor cost, extra wall time, and the reviewer's stated reason for each.
• Also report (descriptive): turn-1 pass per arm; per-requirement drop table; misses (failing snapshot judged on_track/unknown); whether the 2nd review re-checked the whole request or only prior feedback (tried.md 4 Oct); 4 token classes, wall time, tool calls.
• Stop snapshots: an identical extra Stop hook in both arms' project settings copies git-visible files to a dir outside the work tree; the checker runs on each snapshot offline after the run.

4. Task design rules (each maps to a recorded past failure)

1. Fresh tasks only. Not rabadon, not contacts-import, not the 0a93e60/652b862 historical defects, not shipping/reproducer/quarantine/merge cases, nothing under bench/. Written by a session that has not seen review_prompt.h or past results, given only this section. (Overfitting to known dev tasks: 4 Oct prompt candidate; 10 Oct "do not rerun known cases".)
2. Not too easy. 10 Oct: 6/6 first Stops passed, 17 Oct: both arms passed -> nothing to repair. Each prompt has 8-12 numbered explicit requirements across >=3 files: core behaviour,

   =2 edge cases, one preservation constraint (e.g. "do not change the signature of X; existing

   tests must pass unmodified"), named new tests, a README/CHANGELOG change with stated content, a CLI flag or output-format detail, and >=2 requirements placed after the main description. Expected 15-40 tool calls. Natural requests, no traps, no contradictions.

3. No seeded faults or misleading handoffs. Those are mechanism evidence (25 Sep), not natural work (product-acceptance "Fayda gösterilmiş").
4. Everything checked is stated in the prompt. No requirement lives only in a repo file. The checker never judges unstated things: no __pycache__/artifacts (17 Oct), no final-message wording, no style. Behavioural checks, not byte equality (25 Sep); extra valid tests/files allowed; hashes only on files the prompt says not to modify.
5. Checker must be able to go red. Pre-run validation: original repo FAILS; a reference solution PASSES; for every requirement a variant dropping only that one FAILS exactly that group. One assertion group per numbered requirement, per-group output, exit code = verdict (never grep the suite text; never test; echo exit=$?, 25 Sep).
6. Working harness. Small pinned public repos (permissive licence, 1-5k LOC, Python/JS) whose own suite is green at the pin; deps preinstalled in a local venv/node_modules; the exact test command is in the prompt (R7: system python vs .venv, missing extras, empty task text). Work dirs under a HOME subdir: not /tmp, not a git worktree, no "rabadon" in the path.
7. Hidden checker lives outside the work tree; grep each stream for its path and flag hits (result still counts).
8. Frozen product. No change to reviewer prompt, packet (e_files stays first, 4 Oct), caps or thresholds between freeze and analysis. Record CLI version, resolved model ids, gate sha256.
9. Calibration only on a separate smoke task (not one of the 6), never on test tasks.

5. Execution, cost cap, stop rule

• Step 0 ($0): dry run with a fake claude placed first on PATH; verify which claude is the absolute fake for the launcher AND the gate's reviewer subprocess (4 Oct: relative PATH called the real reviewer, $0.19). Check hooks fire, snapshots land, checker runs, costs parse.
• Step 1 (<= $1.50): one paid smoke pair on the calibration task. Verifies reviewer call and SEMANTIC_COST capture in R, zero reviewer calls in P. Not scored.
• Step 2: 6 task pairs. Hard cap $12.00 total list cost including steps 0-1.
• Before each pair: start only if spent + $2.00 (2 x $0.75 actor + 2 x $0.25 reviewer) <= $12. Before each follow-up: run only if spent + $0.75 (+ $0.50 in R) <= $12. If either check fails: stop everything; verdict "incomplete, no evidence". Never drop a follow-up to stay under cap.
• After each pair the runner prints and checks: spent, passes per arm, credited repairs, unnecessary interventions, reviewer called in R, costs complete. Infra failure (auth error, CLI crash before first tool call, hooks not loaded per stream hook events) -> one rerun of that run in place, logged; a second infra failure -> stop. Wrong answers, budget exhaustion and timeouts are results, never rerun.
• Futility stop: if remaining pairs < 2 - credited repairs so far, stop; verdict per section 6.
• Estimate: actor $0.35-0.60/turn (10 Oct opus-low: $0.02-0.77), reviewer $0.03-0.18/call. 6 pairs ~ $6.5-9 + follow-ups ~ $1-2 + smoke ~ $1 -> expected ~$9, cap $12.

6. Pre-registered verdicts (P_x = final passes /6, CPC_x as defined)

• "Lowers cost (this sample)" only if ALL: (a) P_R >= P_P; (b) CPC_R <= 0.85 x CPC_P (or CPC_P infinite with P_R >= 2); (c) credited repairs on >= 2 distinct tasks; (d) zero harmful and <= 1 unnecessary interventions.
• "Evidence against at this scale" if ANY: P_R < P_P; any harmful intervention; or CPC_R > CPC_P with zero credited repairs.
• "No evidence" otherwise, including: P passes turn 1 on >= 5/6 (mechanism not exercised); exactly 1 credited repair; incomplete run. A "no evidence" result needs a NEW protocol with NEW tasks; never harder versions of these six.
• Claim scope even on success: "on 6 fresh multi-step tasks, opus-low actor, CLI <ver>, gate <sha>, review lowered list cost per correct outcome by X% (P_R vs P_P passes)". No p-value claim (6 pairs: only 6/6 same-direction reaches sign-test p < .05), no general token-savings, hours-long-work, multi-turn interactive, other-agent or human-time claim.
• Publish all 12 rows, every reviewer reason, every failure, and the raw-artifact hashes; raw streams stay outside the repo.

7. Amendments made BEFORE freeze (2026-10-17, before any paid bench call)

• Work trees live in /Users/Shared/rb-bench-20261017/runs/<task>-<arm>/, OUTSIDE HOME. Reason (measured today, installed integration run): Claude Code loads ancestor .claude/CLAUDE.md, so a tree under HOME gave the actor the operator's private CLAUDE.md. No "rabadon" in the path.
• Starter repos are self-written by three fresh sessions that saw only section 4 (no public repo, no network). Trees are copied, not cloned (a clone's reflog names the task dir).
• Hidden checkers stay in ~/rabadon-bench-tasks-20261017/<task>/check.py; streams are grepped for "rabadon-bench-tasks" and hits are flagged (result still counts).
• Product under test: the installed 0.2.6-preview.20261017.3 candidate (reviewer prompt caching off, SessionStart blind-spot list no longer in agent context). Both arms: same rabadon init --no-llm, gate mode ON (enforce) set once on the shared ledger dir, same extra Stop snapshot hook (/Users/Shared/rb-bench-20261017/bin/snap.py).
• Runner: bench.py beside this file. dry -> smoke (CAL pair) -> main (T1..T6). The runner itself checks spent/cap, infra failure and futility after every run/pair and stops itself.

related

commands

rabadon init