Repository navigation
The proof runner: bun run proof <slice> proves a slice end to end, and Slice 0 is proof ready - #139
Merged
Merged
Conversation
…nd source-reading checks leave test files Work in progress before the verifier proof run and records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd pass on the change Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s let it read the instructions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er's proof lines, and rules-from-main checks a slice set to done bun run proof <slice> (scripts/proof/) runs a slice's steps in order, each passed, failed or waiting on a named STATE.md → Waiting on Devesh item, and writes a short report. .github/workflows/proof.yml runs it after each deploy to main, daily and by hand, keeps the report as the run's artifact and comments it on the standing "Proof reports" issue. bun run cycle --banner shows each proof-ready or done slice's latest run and what to do. rules-from-main refuses a pull request that sets a slice to done unless its link is a passing proof run of that slice on main, read from GitHub's API, and its body carries the run's report. The field "Seen by Devesh:" is renamed "Proof passed:". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le changes, and AGENTS.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng its files, so no test reads a source path Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted, the verifier stops only its own worker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e Supabase file cite the files that hold their facts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elper or fixture outside a test folder never counts A test that read a new fixture from disk outside a test folder threw on main's code for want of it and was counted as proven. Main's copy now holds every file of the change that isn't product code, so only product code differs between the two runs. lcov.ts is now the line-coverage item's file byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le's decision; the database and migration gap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n scratch copies it removes Every step runs against the local stack and scratch clones, branches and repositories of the proved commit: the refusals are handed to the gate's hook or judged by main's copy (pr --head) and never run; GitHub's run records are read with gh reads only; the scripted agent session and the verifier's run use claude -p with only ANTHROPIC_EVALS_KEY and wait without it. Six steps whose safeguards are still on other branches fail on today's main and pass on theirs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he Waiting key's spending per run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a code file with options, a read whose comment claims a fixture, createReadStream
A typed field ({ type: "text" }) no longer trips them. The check lives in scripts/done-gate/source-reading.ts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n it changes a .sql file there, not the folder's README Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n the console, so require() and a spread prop can't get past Biome's import rule never sees a required subpath, and its raw-HTML rule sees only the JSX attribute. The guards read text, as the tests they replaced did, and also catch innerHTML, outerHTML, insertAdjacentHTML and document.write. Lint keeps the Supabase rule, now tested against require() too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…settings test reads no source guardrails-save.test.tsx drives the Settings page's real hooks with the network faked: a PUT checked against the shared schema, signed in, to the company's guardrails, then the shown policies refreshed; a bad policy never leaves the browser; a refused save refreshes nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he configuration screen So the console's entry can't stop mounting Boot unnoticed: with main.tsx rendering the app directly, the check fails (no configuration screen). The build goes to dist-e2e-no-settings, ignored like dist-e2e; .agents/ is ignored too (fix-when-touched entry 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on the local stack only Step 5 finds and reads each test; the gate already ran it under the shared lock with a fresh console build, and a single test run by hand could collide with another checkout or reuse its console on 4173. Step 6 never calls a real provider. The API tests call app.fetch. The wording test says it is a wording guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er check, the save test and the done rules now cover, and what still gets past The done rules' text-import hole is narrowed, not closed, and STATE.md names the ways still open. The verifier wording pin names no source path, so the done rules let it read the instructions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s own, marks read from the file The outside review found the check counted tests it should not. Now: - both sides run in scratch clones of main's commit (with git history), the change's copy holding every changed file and main's holding all but product code, so product code is all that differs; - in a test file that can't load on main, each test counts only when it, run alone, runs an added line of the missing file beyond what loading the file runs; - a new file under a product folder that only tests name is a helper or fixture and comes along to main; - a test edited only by deleting a line from it is judged; - marks are read from the test file for every test the change touches, the gate's own tests included, so the stop and the PR body show the same marks the check accepts; - a stop counts the tests proven, not the marked ones, and a stop without Docker names this check; - a link in the change stays a link; a test MAY_SKIP lets skip is noted, not judged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d only by another helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…product code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tes it, and the gate's own files never name one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the second on the final instructions showing the schema check and the duplicate, failure and retry cases Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in; STATE, decisions, rule changes and AGENTS.md say what the check now does The helper rule follows the gate's own product definition (migrations and seeds aside), and the two comments that had become untrue (the check list, the moved-lines note in cli.ts) are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The interrupted-work step now runs in a linked worktree with the first session an hour in the past, so it passes on the checkpoint branch's current head; each step's time limit stops the commands it starts; the banner judges a run as rules-from-main does and gives up within 8 s; rules-from-main refuses a done slice changed while it keeps its link and a done slice with an unticked item; the gate and ci.yml steps run main's rules-from-main workflow as GitHub runs it; the verifier helper hands the hook only PATH and home; the verifier step's sample is a reachable route missing only its cross-company test, under the shared-stack lock; the agent session's checks judge meaning; a refused evals key waits on Devesh; the report fits one comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into feat/verifier-checks-test-layers # Conflicts: # ROADMAP.md # STATE.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… gitleaks' results.sarif Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into feat/verifier-checks-test-layers
…for source code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…test-layers # Conflicts: # ROADMAP.md # STATE.md
…' into feat/proof-runner # Conflicts: # STATE.md # scripts/done-gate/pr.ts
# Conflicts: # ROADMAP.md # STATE.md # apps/console/test/app-error-boundary.test.tsx # apps/console/test/boot.test.tsx # apps/console/test/console-boot-honesty.test.tsx # apps/console/test/conversation-link.test.tsx # apps/console/test/guardrails-save.test.tsx # apps/console/test/pages-adoption-tasks-agents-settings.test.tsx # apps/console/test/transcript-xss.test.tsx # apps/console/test/vite-api-url-honesty.test.tsx # lessons.md # scripts/done-gate-source-reading.test.ts # scripts/lint-structure.test.ts # scripts/verifier-instructions.test.ts
Deploying revenue-os-console with
|
| Latest commit: |
be47321
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://df892120.revenue-os-console.pages.dev |
| Branch Preview URL: | https://feat-proof-runner.revenue-os-console.pages.dev |
…anged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ronous Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…steps' clones no longer trip over git's clean-up Bun's synchronous spawn can miss a finished command's exit and spin forever (oven-sh/bun#34069, fix unreleased): it froze about one whole-suite run in five when four ran at once, and CI's test step for over two hours. The test preload runs a full memory clean-up before each synchronous spawn, the workaround the fix's author gives. On GitHub's runner a proof step's clone failed copying the lock file git's automatic background clean-up held after a fetch; the steps' git commands now run with that clean-up off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an-up switched off in the proof steps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t by the code they test, and true marks where main's copy holds it Main's tests-proven check (#136) refused 66 of this branch's tests. scripts/proof.test.ts and the Slice 0 proof tests are split so each file's first missing import is the new file its tests run; the Slice 0 steps now prove the checkout's files as they are on disk, committed into a scratch repository, since the check's copy leaves the change uncommitted on main's commit. Tests of the gate's own code, the tracker's parser and proof.yml, which main's copy carries over (none is product code), carry the "behaviour already on main" mark. The run's 100 minutes are now proven by a run whose clock is moved past them. The step for a new test that already passes on main must now pass here, as its safeguard is on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…check counts it as product code the workflow runs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # STATE.md # lessons.md
…ps passed, 2 wait on the evals key, none failed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the change, as GitHub's checkout makes it The check's copy on GitHub's runner failed 17 of them though every local copy passes; this test shows the copy's own output when it fails, in the test step that prints it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unner's real report checked by rules-from-main, the banner command run with a proof-ready slice - ROADMAP.md: the #137 item's clause "the gate's tests-proven check passes on it …", lost while taking main earlier, is back as main has it. - scripts/proof-report-link.test.ts: the report the runner writes with GitHub's run settings, pasted into a pull request body, passes rules-from-main's link check; the body without it is refused. - scripts/cycle.test.ts: the banner command in a scratch repo whose origin/main has a proof-ready slice shows its latest run through a stand-in for GitHub's command line tool, and says it could not read them when that fails. It fails with the banner's lookup removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r, coverage on, as the tests-proven check does Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ository at load, so the tests-proven check names why it fails there; the copy test works in a checkout without origin/main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng FETCH_HEAD, which the tests-proven check's copy on GitHub's runner found not valid The temporary load-time line stays for this one run on GitHub, so a remaining failure there names its cause. The copy test claims only what it checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tory at load on GitHub is gone; it named the cause Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ository at load and, when the fetch leaves no ref, say on one line what the fetch did Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etching nothing from a checkout that may be shallow
GitHub's runner left the tests-proven check's copy shallow, and git, fetching from a shallow repository, exits 0 but
writes no ref ("rejected … because shallow roots are not allowed to be updated"). The helper now commits the files
on disk as one fresh commit, also its origin/main. The copy test makes its copy shallow, so it fails with the earlier
helper as the check failed on GitHub. Both temporary diagnosis lines are gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap: Slice 0 — The proof runner
Rule change: scripts/done-gate/ (proof-link.ts, proof-run.ts, pr.ts, cli.ts) and scripts/done-gate-proof.test.ts · tighter ·
rules-from-mainnow refuses a pull request that sets a slice to done, or changes a done slice's "Proof passed:", unless its link is a run of the proof workflow that succeeded as a whole and passed every step of that slice, on a commit already on main, every item of the slice is ticked, and its body shows that run's report heading andRun:line; it also refuses a change to a done slice's title, goal, Proof line, blocker or items while the slice keeps its linkRule change: scripts/done-gate-verifier-run.ts · neutral · the recorded verifier run the gate's tests use can also hand the gate's hook an environment and a time limit of the caller's choosing; Slice 0's proof uses that to give the hook only the PATH and home folder, and the gate's tests are unchanged
Rule change: .github/workflows/proof.yml · tighter · a new workflow runs the proof of every proof-ready or done slice after each deploy to main, daily and by hand, and posts each report to the "Proof reports" issue (every slice's report is posted before the step fails for one it could not post); it reads code and run records, writes only issues (its posting job holds that alone), never runs on a pull request, gives its steps 100 of the job's 120 minutes, and installs Claude Code 2.1.288 (pinned) and the browser checks' browser for the two Slice 0 steps a model takes part in, whose commands alone get ANTHROPIC_EVALS_KEY
Rule change: scripts/cycle.ts · tighter · the banner shows each proof-ready or done slice's latest proof run and tells the session what to do about it, judging the run as
rules-from-maindoes, and names the field "Proof passed:"Rule change: AGENTS.md · tighter · Definition of done: the passing run's date and link go after "Proof passed:" in a pull request that carries the run's report, and it says what
rules-from-mainchecks of it; Commands listsbun run proof <slice>Rule change: .claude/skills/task-loop/SKILL.md · neutral · step 12 follows the rename and says the proof workflow runs a proof-ready slice's proof
Rule change: package.json · neutral · adds the
bun run proof <slice>scriptRule change: scripts/done-gate-verifier.test.ts and scripts/done-gate-restart.test.ts · neutral · their sample roadmaps use the renamed field "Proof passed:" (the restart test came to main with #133 and still wrote "Seen by Devesh:", which the tracker's parser now reports as renamed)
Rule change: scripts/done-gate-tests-proven.test.ts · neutral · its two tests that call main's pull-request check now wait for it, since this PR makes that check asynchronous (it asks GitHub about proof runs); what they check is unchanged
Rule change: tests/setup.local.ts · neutral · before each synchronous spawn in a test run, a full memory clean-up, so Bun's lost-exit bug (oven-sh/bun#34069) can't freeze the run; what the tests check is unchanged
Fix-when-touched: 1 · not applicable · the files this PR touches under scripts/ (cycle.ts, cycle.test.ts, done-gate/pr.ts, done-gate/cli.ts, done-gate/proof-link.ts, done-gate/proof-run.ts, done-gate-verifier.test.ts, done-gate-verifier-run.ts, done-gate-restart.test.ts) carry no old task, milestone or section codes and no untrue comments, and the new files (scripts/proof/, scripts/proof-slice-0.test.ts and the other new tests) cite none
Fix-when-touched: 4 · not applicable · this PR changes one step of the task-loop skill to follow the field's rename and restates no drifting fact; turning the lessons its last section greps for into checks is a change of its own
Exception: scripts/proof/slice-0-rules.ts · the sample written into a scratch branch, for main's copy of the rules to refuse
Exception: scripts/proof-slice-0.test.ts · read only to write a changed copy into a scratch clone, never to check what it says
Behaviour already on main: scripts/done-gate-proof.test.ts · main's copy carries the gate's own code and the tracker's parser over (scripts/done-gate/proof-run.ts, proof-link.ts, pr.ts and docs/tracker/parse.js are not product code), so it passes there
Behaviour already on main: scripts/done-gate-tests-proven.test.ts · this change only awaits the pull-request check, now asynchronous; what it checks is main's, and main's copy carries the gate's own code over, so it passes there
Behaviour already on main: scripts/proof-slice-0-fingerprint.test.ts · it reads ROADMAP.md with docs/tracker/parse.js, and neither is product code, so main's copy holds this change's Proof line and parser too
Behaviour already on main: scripts/proof-workflow.test.ts · main's copy carries .github/workflows/proof.yml and the gate's own code over (neither is product code), so it passes there
Behaviour already on main: scripts/proof-workflow.test.ts · main's copy carries .github/workflows/proof.yml over (it is not product code), so it passes there
Behaviour already on main: scripts/tracker-proof-passed.test.ts · main's copy carries the tracker's parser, docs/tracker/parse.js, over (it is not product code), so it passes there
New dependency: Claude Code 2.1.288 (npm @anthropic-ai/claude-code), installed in the proof job only, because Slice 0's Proof line needs a real scripted agent session and a real verifier run; recorded in STATE.md → Decisions in force
What
A slice of the roadmap can now be closed by a proof that runs by itself, and Slice 0 has its whole proof: one step for each check its Proof line names, 28 in all.
The runner.
bun run proof <slice>runs that slice's steps one after another and writes a short report: for each step, what it did, what it saw, and any file it saved (a transcript, for example). Each step ends one of three ways:ANTHROPIC_EVALS_KEY). The report then names that STATE.md → Waiting on Devesh item, and the slice stays as it is.Each step has a time limit that the commands it starts obey, and the run as a whole stops at 100 minutes. So the report is always written before GitHub's 120-minute limit on the job.
GitHub runs it on its own (.github/workflows/proof.yml) for every slice marked
proof readyordone: after each deploy to main, once a day (08:53 India time), and by hand from the Actions tab. Each run keeps the report for 90 days and posts its text as a comment on one issue, "Proof reports". The workflow opens that issue the first time, mentioning Devesh, so GitHub notifies him of every report.Every session start shows the latest proof run of each
proof readyordoneslice, and says what to do: set a passing one to done (it prints the exact line to write), add an item for a failure, or leave a waiting one alone. It judges a run exactly asrules-from-maindoes, so it never asks for a link that check would refuse. It gives up after 8 seconds, so a slow GitHub can't hold up a session's start.rules-from-mainrefuses a pull request that marks a slice done unless:Proof passed:link is a run of the proof workflow that succeeded as a whole and passed every step of that slice, on a commit already on main;Run:line.Once a slice is done, its title, goal, Proof line and items can't change while it keeps its link; to change them, set it back to
proof ready. The field "Seen by Devesh:" is renamed "Proof passed:" everywhere.Slice 0's 28 steps (scripts/proof/slice-0*.ts). Most work in a throwaway copy of the commit being proved: a clone whose main is that commit, or a small repository holding only its gate. They delete it afterwards. Four work in the checkout itself:
bun run cycle --banner, which fetches origin/main, so in CI it judges main's newest commit; it then runs, by name, the two tests in scripts/cycle.test.ts that the Proof line cites for the hooks behind the banner;ghand compares commits there;Nothing is pushed, no pull request is opened and the cloud is never touched. A command that must be refused is only handed to the gate's check, the way Claude Code hands it a tool call; it never runs. Some examples:
A branch can't escape main's rules. One step makes a scratch branch that loosens its own copy of the gate (its scripts/done-gate.ts passes anything) and adds a line that switches the type checker off. The branch's own copy passes it. Main's rules-from-main workflow refuses it, run the way GitHub runs it: main's copy of the workflow file (which only
pull_request_targetguarantees), its checkout of main, and its two steps, with the branch fetched as data. Another step does the same with a branch that rewrites ci.yml so itschecksjob checks nothing. It also checks that main's copy refuses a ci.yml rewrite on its own when the body doesn't explain it.Merges. One step commits an unverified change and checks the gate refuses its merge, then runs, by name, the two merge tests in scripts/done-gate.test.ts the Proof line cites. Another proves a pull request's head (its checks recorded as passed, the verifier's PASS): its plain
gh pr merge 7 --squash --match-head-commit <sha>goes through, while the same merge wrapped insh -c,bash -c,evalorxargsis refused, and so is--admin. A stand-inghanswers for GitHub.Interrupted work. One step plays a session that wrote a checkpoint in a worktree an hour ago and never stopped, then starts a new session there. The new session must be shown the checkpoint and held at the done gate until the change passes.
The scripted agent session. One step starts a real Claude Code (
claude -p) in a fresh clone of main. Its only credential is ANTHROPIC_EVALS_KEY: no GitHub token, a home folder of its own, and git only for looking around and making a branch (never a push). It holds four exchanges:It checks each answer, and that no file or branch changed, and saves the transcript in the report. An answer names its item when it uses the words only that item has among its slice's items (in its title or its whole text), or quotes a short title whole. The /ready answer is judged against what the worker's code checks: today, only its token.
The verifier's run. Another step runs the verifier agent the same way on a small finished change: a database function, the API route that calls it, and an API test for one company only. It expects a FAIL whose ruling line names the missing other-company test.
Without the key, both model steps wait. If Anthropic refuses the key (its spending limit used up), they wait too, naming the key's Waiting item.
Fitted onto main after #132-#135
Main gained four Slice 0 safeguards while this was built: every stop says what it did not run (#132), the checkpoint and interrupted-work safeguards (#133), line coverage (#134), and done rules that see removed and moved code (#135). This branch merges main (a normal merge commit) and:
packages/shared/src/proof-sample.ts:2 exports proofSample …); before, a refusal of the barrel's own line was enough;Then #136 (tests proven) and #137 (the verifier checks test layers) merged, and this branch took main again:
scripts/done-gate/pr.tsruns A change's tests are proven to test it: each new or edited test must fail on main's code #136's check of "behaviour already on main" marks and then this branch's check of a slice set to done;bun run gate proven.Slice 0 is set to
proof ready. This is its last item, so AGENTS.md → Definition of done says the pull request that lands it marks the slice proof ready. The proof workflow then runsbun run proof 0on GitHub after the next deploy to main and daily. The slice becomesdoneonly through a pull request that links a passing run, andrules-from-mainchecks that link. UntilANTHROPIC_EVALS_KEYis saved in GitHub, the two model steps wait, so the slice stays proof ready. With Slice 0 proof ready, the current slice becomes Slice 1, which nothing blocks.Three problems CI found
This PR's first CI run sat in the test step for over two hours. Its log, readable once a push cancelled it, showed the first two problems; the next run showed the third:
bun run proofargument check moved intotoProvein scripts/proof/cli.ts, unchanged, so its test runs it in-process, not only through a subprocess, which coverage can't see.Each group was fixed by one agent and checked by another told to refute it. A reviewer caught one mark the check could not see, because bun reported its test at the wrong line, and the in-test import of the 100-minute limit. Both are fixed. No test was lost or renamed, and none lost an assertion. The two tests-proven tests that failed in the first run were already fixed on this branch: they now wait for main's pull-request check.
On GitHub only, the check still failed 17 Slice 0 proof tests in its copy, though every local copy passed, a CI-shaped one included. Two temporary runs found why:
FETCH_HEAD: not a valid SHA1, right after a fetch that had succeeded.warning: rejected refs/proof/base because shallow roots are not allowed to be updated. The check's copy on GitHub's runner is shallow (why is not known; local copies never are). Fetching from a shallow repository, git exits 0 but writes nothing unless given--update-shallow.The fix: the tests' scratch repository is now one fresh commit of the checkout's files, also its origin/main, and nothing is fetched from the checkout. The steps' own clones (scripts/proof/scratch.ts) fetch into a named ref rather than FETCH_HEAD. The proof workflow checks out full history (
fetch-depth: 0), so a real proof run never starts from a shallow checkout. scripts/proof-slice-0-copy.test.ts makes its copy shallow, as GitHub left the check's: with the earlier helper it fails with GitHub's exact error, and with this one it passes. Both temporary lines are gone.What the verifier found
The verifier ruled FAIL twice:
rules-from-mainaccepts the runner's real report. scripts/proof-report-link.test.ts now pastes the reportprovewrites with GitHub's run settings into a body: the check accepts it and refuses the body without it.scripts/cycle-hook.sh --bannerwith a stand-in gh, seeing the run line and the "could not read them" line; it fails with the banner's lookup removed.checksfailed on the FETCH_HEAD problem above, and the copy test claimed more than it checked. Both are fixed.Why
Until now nothing could close a slice: "done" needed a proof run and a place to see it, and nothing checked the link an agent would write. Now a slice closes only on a run GitHub itself vouches for, and Slice 0's run exercises every safeguard the slice built.
For example, the step that hands the gate
gh release create v1.0.0passes only because the gate refuses it. On a copy of the repo whose gate, hooks and guards refuse nothing, every one of the 22 refusal steps fails, and a test checks exactly that.The workflow follows the deploy rather than the tests, because from Slice 2 on the deploy is what puts the new code on the test server.
Outside review
An outside reviewer went through this pull request at commit 40da44b. They found one blocker, nine problems worth fixing and nine smaller points. Every blocker and should-fix is now fixed, each with a test that failed before the fix.
To check that, I ran this branch's tests against the reviewed code (40da44b's sources with the new tests, plus the new function names so the files load). 26 of the 79 tests the first round of fixes came with fail there. The 6 tests added in the last round fail on the code just before their fix.
The blocker: the interrupted-work step could never pass.
Problems worth fixing, all fixed:
rules-from-mainrefuses. One run proves every due slice. If another slice failed, the run as a whole failed, yet the banner said "set it to done" for the slice that passed. Now both read the run through one function, and the banner says to wait for a later run instead. A test feeds the banner's own line intorules-from-mainboth ways.sleep 6now fails in about 1.2 seconds.pull_request, judges with the branch's own gate, judges nothing, or doesn't count ci.yml as a rule file.bunsees no token in any hook call.rules-from-mainrefuses any change to such a slice's title, goal, Proof line or items, and a slice set to done with an unticked item; the tracker's parser flags that too.Not changed, and why:
rules-from-mainstill asks GitHub's API without a login. A refusal for GitHub's rate limit says so, and re-running the check fixes it. Passing a token into thatpull_request_targetjob would change the check every pull request runs, so it belongs in its own pull request.proof readyordoneslices, and Slice 0 becomesproof readyonly when its last item lands, so its first real run comes after every safeguard is in place.proof ready, its proof fails with "no proof steps yet", and the banner has the session add an item for that.Evidence
bun run proof 0on this branch with main merged through A test's child processes are stopped, with everything they started, however the test ends #140 (226aa57, 2026-10-04) ran all 28 steps and saved its report. None failed:sh -cor run with--admin;checkspassed on their commit and skipped the apply step, and 1 skipped its job;bun run gate proven(""one is still one" passes on main's code too, so it doesn't test this change");bun run proof 0now waits only on the key.The real Claude Code, without a real key.
claude auth status, with only a stand-in key and a scratch home, reports"authMethod": "api_key", "apiKeySource": "ANTHROPIC_API_KEY", so the session never uses Devesh's login. With that stand-in key, the agent-session step startedclaude -pin the clone, and the clone's SessionStart hooks ran (the banner reached the session).The tests: 88 tests across scripts/proof-run, proof-report, proof-cli, proof-due, proof-scratch, proof-workflow, proof-slice-0, proof-slice-0-runs, proof-slice-0-named-tests, proof-slice-0-fingerprint, proof-agent-checks, cycle-proof, tracker-proof-passed and done-gate-proof (.test.ts each). They check:
claude: they wait without the key, get only that key and a home of their own, hold one resumed conversation in a fresh clone and save the transcript. They fail when the session makes a branch or the verifier passes the change;Main's copy versus this branch's on a pull request that sets Slice 0 to done, in a scratch repo, using a real staging-migrations run (37107299842): main's copy passes it; this branch's copy asks GitHub's API without a login, then refuses it as "a run of .github/workflows/staging-migrations.yml, not of the proof workflow".
bun run gateon the final commit (be47321), run bare, exit 0: "Done gate ✓ typecheck, lint, guards, 1314 tests, 67 tests proven, database policies, 29 browser checks". It shows the two exceptions this body names: the sample line that switches the type checker off, which the proof writes into a scratch branch for main's copy to refuse; and the test that reads a workflow or gate file only to write a changed copy into a scratch clone. No line this pull request adds is under apps/, services/ or packages/, so main's line-coverage rule has nothing to judge and no coverage mark is needed.Verifier, on be47321: PASS. "on be47321 GitHub's checks and the local gate pass;
bun run proof 0passed 26 steps and 2 wait on Devesh's evals key; the shallow-copy failure is fixed and its test fails with the old code". It ran the gate andbun run proof 0itself, rebuilt the shallow-fetch failure with plain git, and checked its earlier findings stay fixed (it ruled FAIL on 530f849 and d0aef97; see What the verifier found).GitHub's
checkson be47321: every step passed, "Tests proven against main's code" among them: "67 passed on this change and failed on main's code, of 93 new or edited test(s) in 18 file(s)".This body, with 9999 in place of PR-THIS (and STATE.md's and ROADMAP.md's PR-THIS too, in a scratch copy), passes
bun run gate prwith main's copy.Not yet seen:
🤖 Generated with Claude Code