live assist verdict — 2026-09-25

What actually ran

Claude Code (exact version in the manifest), model claude-fable-5-1, low effort. Six paired tasks, 12 real model runs, plus one prior transport probe. Each arm starts from the same source/prompt/checker, in a fresh local fixture. Arm order alternates by task. Model budget $1 list-price estimate per arm, deadline 180 seconds; none hit either limit. The user required the executive to validate the work before handoff; the executive selected this model, not an explicit user model choice.

Four repairs: seeded import mismatch; vendored historical MarkupSafe multiline striptags regression; deliberate successful-check/edit/failure reproduction; and a shipping boundary error with a misleading old test suggestion. Two negative controls: healthy implementation and expected grep-no-match. Historical provenance: MarkupSafe e49d257. The source half of that fix was reverted, not a new invented bug. The other repair cases are explicitly seeded, and the regression prelude is controlled.

The manifest was written before model runs. It holds full initial sources, prompts, acceptance, fixed model, binary hash, order and limits. Global settings were excluded using --setting-sources ""; assist uses only local explicit PostToolUse/Failure hooks pointing at the lab binary. Off has no Rabadon hook. No global hook was installed, user mode was not enabled, no second repair model was used. The Python hook recorder is measurement instrumentation and adds launch overhead; its cost is included in the task times.

Results

Token volume = input + cache creation + cache read + output, as reported by the CLI's final result. Categories and per-model usage remain separate in raw data. This is not an invoice: the account uses a subscription, and cache has different prices. The CLI's list-price estimate rose 4.97%; token volume rose 10.24%. Wall-time differences are confounded by cache, cold starts and server latency. Neither a speed advantage nor population-wide harm is established by one pair per task. A successful outcome improvement was not observed, so expansion to a large fleet was rejected.

The sole pilot note occurred in the historical case after two deliberately requested repetitions. Assist used 7 tools; its trajectory included a broader search before narrowing to the source. A 205-byte note did not itself encode 12882 tokens; the observed difference includes the different tool trajectory and sampling. Causal attribution needs more than this pair. What is already clear is that the pilot supplies no basis for claiming lower total token use.

The regression arm ran python3 -B check.py; echo "exit=$?" and variants with cat/git status. The trailing command exited zero, so Claude emitted PostToolUse even though the checker had failed. Exact command identity also changed. The candidate missed the signal and supplied no advice; the model repaired the source by itself. Unit JSON replay did not expose this behavior.

All recorded native hook exits were zero; no agent was restarted or blocked. Both controls generated no extra context. These are useful transport properties, but they do not establish helpful repair or token savings.

What counts as repair here

A separate evaluator ran the original checker plus held-out assertions against the final source. It also checked that check.py stayed byte-identical; controls had to leave all original files unchanged. All 12 final outputs passed. The executor, not the test script, discovered/applied each final repair.

Negative control: replacing the shipping boundary assertion made the local checker exit zero while the original external acceptance exited one. This validates the evaluator's rejection of a fake green. It does not prove that Rabadon prevents an adversarial builder from tampering: same UID is not an isolation boundary, and neither arm attempted to attack the external evaluator.

The separate live transport probe forced two failures. The CLI stream shows Rabadon's hook response; the model's subsequent final answer names Rabadon and correctly paraphrases the delivered advice, although that name/advice were not in the input prompt. The model then repaired arithmetic and passed the unchanged checker. This demonstrates delivery and continued work, not causal superiority.

Evidence and continuation

Pre-run manifest, per-run results, decision summary. Every run under bench/evidence/2026-09-25-live/runs/ has its model stream, acceptance output, final source and result; assist runs also have raw hook events. Transport and false-fix controls are in the same evidence directory. No credentials or auth-status output are included.

Archived candidate contains the patch and exact experiment tools; applying it is an explicit reproduction step, not a product installation. The active deliverable retains only streaming ledger reads and the missing-evidence counter correction, which have independent tests. Worker handoff now specifies that verified package. The general in-run repair design remains an executive responsibility, not an unproven plan handed to the worker as if it already worked.

Retained package verification

After removing the candidate from runtime, make -j2 all, LC_ALL=C make test, npm test (15/15) and docs truth (42/42) passed in the separate normal lab clone. The full suite's paired machine-integrity check passed 9/9: 20 artifacts and shared settings unchanged, watch mode preserved. The 26 pilot-only tests left the active suite with their removed implementation and remain in candidate.patch.

The retained source was remeasured in watch mode, five samples per arm, 32 MiB ordinary-event ledger: median peak RSS 101.359 MiB before, 2.625 MiB after; median wall time 1036.571 ms before, 1034.585 ms after. The scan time remains. This is one synthetic Stop workload, not whole-agent memory. Raw data and binary hashes: watch-memory.json. The active benchmark is bench/ledger_memory.py (default mode watch).