What I built
I built sign-off verification for a 31-role Claude Code lab, audited it against runner logs, and shipped an improved detector with tests. The finding, by my count: the gate ran on 0 of 270 eligible sessions, in this per-session table. A shell bug, fixed in June, explains it. The logs stay private. A later code review found the detector could still be fooled, because the agent's own text reached the log it read. So I rebuilt the check: a required sign-off now needs a signed record, made by the runner from the harness's own tool calls, that the role ran and gave that verdict on that document. If the check can't finish, the session is blocked. That's tested on a stub harness, with 171 new tests, not yet on a live lab. The repo lists what's still open.
A shell script, the runner, starts one director agent at a time. The director dispatches roles such as the PI (principal investigator). The lab had 30 roles at the time of the incident the audit covers.
A sign-off is a line in a lab document saying a role agreed to a decision. Some decisions need one from both the director and the PI. On Apr 17 the director wrote 5 PI sign-offs without dispatching the PI. Then I built three parts:
- Detector: a script that checks each sign-off for a receipt, a dated file in that role's folder saying the role ran. No receipt means a reported forgery.
- Gate: a runner step. When a director session exits cleanly, it runs the detector. On a forgery report, it blocks the clean exit and sends the director back.
- Evaluator check: a checklist item where the evaluator role records the detector's result.
In September I audited those parts and shipped the improved detector, in #38, #39 and #40 on Sep 17 and Sep 18. It matches the sign-off format the incident used, drops three classes of false positive, and adds a mode for checking old records. A new identity check compares each receipt with the agent type the runner logged, because a generic agent can do a role's work. 24 regression tests pin these changes, and they pass.
What I measured
The gate's line began: if ! source .venv/bin/activate && python3 tools/verify_signatures.py. Shell parses that as (! source) && python3. When the virtual environment activates, the left side is false, so Python never starts and nothing blocks. The --prove option of gate_audit.py reproduces this from a clone.
| Session logs, named Apr 17 to May 30 | Count |
|---|---|
| All logs | |
| Eligible: the log contains the gate's precondition | |
| The log contains the detector's output | |
| The log shows the gate blocking | |
| The director ran the detector itself, which isn't the gate | 6 sessions |
The audit didn't find the shell bug. A fix, d5af181, landed on Jun 6, after the last log counted.
| Part | What I found | Count |
|---|---|---|
| Detector | Its original patterns missed the incident's own sign-off format. Archive reports that were false positives: | 23 of 23 |
| Gate | Eligible sessions where it ran: | 0 of 270 |
| Evaluator check | Passes quoting the detector that were over an empty scan: | 6 of 10 |
| Remediation | The dispatch that closed the incident used a generic agent. Non-director receipts filed under a role not yet dispatched by its own type, by a date test: | 32 of 174 |
What limited it
I read the shared limit as the evidence, not the checker. Of the lab's receipts, 0 of 195 were tracked in git, and the session logs are gitignored. These sat inside the director's own write scope. So the record a check reads has to sit outside the write scope of whoever is being checked. The September fix didn't do that, because director text reached the log the check read. The rebuilt check in #43 reads only the runner's signed record, built from the harness's own tool calls.
The old-records mode works by day, so it passes the incident itself: the forged sign-off and a later dispatch share a date.
What it doesn't show
- The counts don't reproduce from a clone, because the logs stay private and gitignored. You can check the table's sums, not its rows.
- "Eligible" is a text match on the string GRACEFUL_CHECKPOINT, so director prose could inflate it. I didn't measure how many of the 270 are true clean exits. I read the zero as the stronger half, because the gate redirects the detector's output into that log.
- The first detector could be bypassed. A later code review, not the September audit, listed how: any old receipt or recent file passed, the gate skipped --check-identity, director text could fake a >>> AGENT pi line, an empty scan passed, and so did a sign-off phrased outside the pattern list. PR #43 closes those. The rebuilt check still doesn't show a review was good, a sign-off binds to a path and not to content, and role prompts can be edited between sessions. It hasn't run on a live lab.
- It's one lab, one model family and one incident. It doesn't show how often agents forge sign-offs, why this one did, or that 31 roles do better than fewer.
Links
- Per-session gate table: each log's counts, size and SHA-256, with no log text.
- Audit README and its known gaps.
- Detector, with tests for sign-off patterns and dispatch identity.
- The public repo.