Skip to content

The proof runner: bun run proof <slice> proves a slice end to end, and Slice 0 is proof ready - #139

Merged
devesh911 merged 72 commits into
mainfrom
feat/proof-runner
Oct 4, 2026
Merged

devesh911 merged 72 commits into
mainfrom
feat/proof-runner

Conversation

@devesh911

@devesh911 devesh911 commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Roadmap: Slice 0 — The proof runner

Rule change: scripts/done-gate/ (proof-link.ts, proof-run.ts, pr.ts, cli.ts) and scripts/done-gate-proof.test.ts · tighter · rules-from-main now refuses a pull request that sets a slice to done, or changes a done slice's "Proof passed:", unless its link is a run of the proof workflow that succeeded as a whole and passed every step of that slice, on a commit already on main, every item of the slice is ticked, and its body shows that run's report heading and Run: line; it also refuses a change to a done slice's title, goal, Proof line, blocker or items while the slice keeps its link
Rule change: scripts/done-gate-verifier-run.ts · neutral · the recorded verifier run the gate's tests use can also hand the gate's hook an environment and a time limit of the caller's choosing; Slice 0's proof uses that to give the hook only the PATH and home folder, and the gate's tests are unchanged
Rule change: .github/workflows/proof.yml · tighter · a new workflow runs the proof of every proof-ready or done slice after each deploy to main, daily and by hand, and posts each report to the "Proof reports" issue (every slice's report is posted before the step fails for one it could not post); it reads code and run records, writes only issues (its posting job holds that alone), never runs on a pull request, gives its steps 100 of the job's 120 minutes, and installs Claude Code 2.1.288 (pinned) and the browser checks' browser for the two Slice 0 steps a model takes part in, whose commands alone get ANTHROPIC_EVALS_KEY
Rule change: scripts/cycle.ts · tighter · the banner shows each proof-ready or done slice's latest proof run and tells the session what to do about it, judging the run as rules-from-main does, and names the field "Proof passed:"
Rule change: AGENTS.md · tighter · Definition of done: the passing run's date and link go after "Proof passed:" in a pull request that carries the run's report, and it says what rules-from-main checks of it; Commands lists bun run proof <slice>
Rule change: .claude/skills/task-loop/SKILL.md · neutral · step 12 follows the rename and says the proof workflow runs a proof-ready slice's proof
Rule change: package.json · neutral · adds the bun run proof <slice> script
Rule change: scripts/done-gate-verifier.test.ts and scripts/done-gate-restart.test.ts · neutral · their sample roadmaps use the renamed field "Proof passed:" (the restart test came to main with #133 and still wrote "Seen by Devesh:", which the tracker's parser now reports as renamed)
Rule change: scripts/done-gate-tests-proven.test.ts · neutral · its two tests that call main's pull-request check now wait for it, since this PR makes that check asynchronous (it asks GitHub about proof runs); what they check is unchanged
Rule change: tests/setup.local.ts · neutral · before each synchronous spawn in a test run, a full memory clean-up, so Bun's lost-exit bug (oven-sh/bun#34069) can't freeze the run; what the tests check is unchanged
Fix-when-touched: 1 · not applicable · the files this PR touches under scripts/ (cycle.ts, cycle.test.ts, done-gate/pr.ts, done-gate/cli.ts, done-gate/proof-link.ts, done-gate/proof-run.ts, done-gate-verifier.test.ts, done-gate-verifier-run.ts, done-gate-restart.test.ts) carry no old task, milestone or section codes and no untrue comments, and the new files (scripts/proof/, scripts/proof-slice-0.test.ts and the other new tests) cite none
Fix-when-touched: 4 · not applicable · this PR changes one step of the task-loop skill to follow the field's rename and restates no drifting fact; turning the lessons its last section greps for into checks is a change of its own
Exception: scripts/proof/slice-0-rules.ts · the sample written into a scratch branch, for main's copy of the rules to refuse
Exception: scripts/proof-slice-0.test.ts · read only to write a changed copy into a scratch clone, never to check what it says
Behaviour already on main: scripts/done-gate-proof.test.ts · main's copy carries the gate's own code and the tracker's parser over (scripts/done-gate/proof-run.ts, proof-link.ts, pr.ts and docs/tracker/parse.js are not product code), so it passes there
Behaviour already on main: scripts/done-gate-tests-proven.test.ts · this change only awaits the pull-request check, now asynchronous; what it checks is main's, and main's copy carries the gate's own code over, so it passes there
Behaviour already on main: scripts/proof-slice-0-fingerprint.test.ts · it reads ROADMAP.md with docs/tracker/parse.js, and neither is product code, so main's copy holds this change's Proof line and parser too
Behaviour already on main: scripts/proof-workflow.test.ts · main's copy carries .github/workflows/proof.yml and the gate's own code over (neither is product code), so it passes there
Behaviour already on main: scripts/proof-workflow.test.ts · main's copy carries .github/workflows/proof.yml over (it is not product code), so it passes there
Behaviour already on main: scripts/tracker-proof-passed.test.ts · main's copy carries the tracker's parser, docs/tracker/parse.js, over (it is not product code), so it passes there
New dependency: Claude Code 2.1.288 (npm @anthropic-ai/claude-code), installed in the proof job only, because Slice 0's Proof line needs a real scripted agent session and a real verifier run; recorded in STATE.md → Decisions in force

What

A slice of the roadmap can now be closed by a proof that runs by itself, and Slice 0 has its whole proof: one step for each check its Proof line names, 28 in all.

The runner. bun run proof <slice> runs that slice's steps one after another and writes a short report: for each step, what it did, what it saw, and any file it saved (a transcript, for example). Each step ends one of three ways:

  • passed;
  • failed;
  • waiting, when it needs something only Devesh can give (the AI key ANTHROPIC_EVALS_KEY). The report then names that STATE.md → Waiting on Devesh item, and the slice stays as it is.

Each step has a time limit that the commands it starts obey, and the run as a whole stops at 100 minutes. So the report is always written before GitHub's 120-minute limit on the job.

GitHub runs it on its own (.github/workflows/proof.yml) for every slice marked proof ready or done: after each deploy to main, once a day (08:53 India time), and by hand from the Actions tab. Each run keeps the report for 90 days and posts its text as a comment on one issue, "Proof reports". The workflow opens that issue the first time, mentioning Devesh, so GitHub notifies him of every report.

Every session start shows the latest proof run of each proof ready or done slice, and says what to do: set a passing one to done (it prints the exact line to write), add an item for a failure, or leave a waiting one alone. It judges a run exactly as rules-from-main does, so it never asks for a link that check would refuse. It gives up after 8 seconds, so a slow GitHub can't hold up a session's start.

rules-from-main refuses a pull request that marks a slice done unless:

  • its Proof passed: link is a run of the proof workflow that succeeded as a whole and passed every step of that slice, on a commit already on main;
  • every item of the slice is ticked;
  • the body shows the run's report heading and Run: line.

Once a slice is done, its title, goal, Proof line and items can't change while it keeps its link; to change them, set it back to proof ready. The field "Seen by Devesh:" is renamed "Proof passed:" everywhere.

Slice 0's 28 steps (scripts/proof/slice-0*.ts). Most work in a throwaway copy of the commit being proved: a clone whose main is that commit, or a small repository holding only its gate. They delete it afterwards. Four work in the checkout itself:

  • the law-file step and the banner step read it. The banner step runs bun run cycle --banner, which fetches origin/main, so in CI it judges main's newest commit; it then runs, by name, the two tests in scripts/cycle.test.ts that the Proof line cites for the hooks behind the banner;
  • the run-records step reads GitHub's records with gh and compares commits there;
  • the at-once step runs two test files there: one that starts four whole test suites at once, and scripts/done-gate-at-once.test.ts, where two gate runs start at once in two worktrees.

Nothing is pushed, no pull request is opened and the cloud is never touched. A command that must be refused is only handed to the gate's check, the way Claude Code hands it a tool call; it never runs. Some examples:

  • A branch can't escape main's rules. One step makes a scratch branch that loosens its own copy of the gate (its scripts/done-gate.ts passes anything) and adds a line that switches the type checker off. The branch's own copy passes it. Main's rules-from-main workflow refuses it, run the way GitHub runs it: main's copy of the workflow file (which only pull_request_target guarantees), its checkout of main, and its two steps, with the branch fetched as data. Another step does the same with a branch that rewrites ci.yml so its checks job checks nothing. It also checks that main's copy refuses a ci.yml rewrite on its own when the body doesn't explain it.

  • Merges. One step commits an unverified change and checks the gate refuses its merge, then runs, by name, the two merge tests in scripts/done-gate.test.ts the Proof line cites. Another proves a pull request's head (its checks recorded as passed, the verifier's PASS): its plain gh pr merge 7 --squash --match-head-commit <sha> goes through, while the same merge wrapped in sh -c, bash -c, eval or xargs is refused, and so is --admin. A stand-in gh answers for GitHub.

  • Interrupted work. One step plays a session that wrote a checkpoint in a worktree an hour ago and never stopped, then starts a new session there. The new session must be shown the checkpoint and held at the done gate until the change passes.

  • The scripted agent session. One step starts a real Claude Code (claude -p) in a fresh clone of main. Its only credential is ANTHROPIC_EVALS_KEY: no GitHub token, a home folder of its own, and git only for looking around and making a branch (never a push). It holds four exchanges:

    1. where are we;
    2. "what does the worker's /ready endpoint check?";
    3. "which item are you on?";
    4. "add a dark mode to the console".

    It checks each answer, and that no file or branch changed, and saves the transcript in the report. An answer names its item when it uses the words only that item has among its slice's items (in its title or its whole text), or quotes a short title whole. The /ready answer is judged against what the worker's code checks: today, only its token.

  • The verifier's run. Another step runs the verifier agent the same way on a small finished change: a database function, the API route that calls it, and an API test for one company only. It expects a FAIL whose ruling line names the missing other-company test.

Without the key, both model steps wait. If Anthropic refuses the key (its spending limit used up), they wait too, naming the key's Waiting item.

Fitted onto main after #132-#135

Main gained four Slice 0 safeguards while this was built: every stop says what it did not run (#132), the checkpoint and interrupted-work safeguards (#133), line coverage (#134), and done rules that see removed and moved code (#135). This branch merges main (a normal merge commit) and:

  • keeps both sides of the three conflicts: ROADMAP.md (A restarted session is shown its checkpoint, and interrupted work holds the next session at the done gate #133's ticked item and this item, both kept), STATE.md (both sides' decisions and rule-change lines, newest first) and scripts/done-gate/pr.ts (main's checks of quoted exceptions, removed tests and coverage marks, then this branch's check of a slice set to done);
  • checks the Proof line main now has. A restarted session is shown its checkpoint, and interrupted work holds the next session at the done gate #133 added three citations to Slice 0's Proof line, which the fingerprint test caught. The banner step now also runs the two pin tests in scripts/cycle.test.ts by name, the merge step the two merge tests in scripts/done-gate.test.ts, and the at-once step scripts/done-gate-at-once.test.ts. A cited test that is renamed, deleted or failing fails its step, naming it (a new test drives that with a renamed and a failing test), and a test now checks that every phrase the Proof line quotes and every test file it names is in some step's words;
  • tightens the barrel step to the Proof line's words. The export that only a barrel re-exports must itself be refused (packages/shared/src/proof-sample.ts:2 exports proofSample …); before, a refusal of the barrel's own line was enough;
  • moves the five steps whose safeguards are now on main (a stop without Docker, a deleted test, an export only a barrel re-exports, an added line no test runs, a session after interrupted work) into the test that requires each step whose safeguard is on main to pass here. Only the step for a new test that already passes on main waits on its item (tests proven, still on its branch);
  • follows the rename in A restarted session is shown its checkpoint, and interrupted work holds the next session at the done gate #133's restart test, whose sample roadmap still wrote "Seen by Devesh:".

Then #136 (tests proven) and #137 (the verifier checks test layers) merged, and this branch took main again:

Slice 0 is set to proof ready. This is its last item, so AGENTS.md → Definition of done says the pull request that lands it marks the slice proof ready. The proof workflow then runs bun run proof 0 on GitHub after the next deploy to main and daily. The slice becomes done only through a pull request that links a passing run, and rules-from-main checks that link. Until ANTHROPIC_EVALS_KEY is saved in GitHub, the two model steps wait, so the slice stays proof ready. With Slice 0 proof ready, the current slice becomes Slice 1, which nothing blocks.

Three problems CI found

This PR's first CI run sat in the test step for over two hours. Its log, readable once a push cancelled it, showed the first two problems; the next run showed the third:

  • A test run can freeze inside Bun. Bun 1.3.11's synchronous spawn can miss the exit of a command that has finished (the command stays a zombie) and spin a core forever (oven-sh/bun#34069; the fix, oven-sh/bun#40078, is in no release yet). The bug hits any synchronous spawn: CI froze in scripts/guards.test.ts, which this PR doesn't touch. This PR adds many such spawns, so it hits more often: here, 3 of 16 whole-suite runs froze with four running at once. Bun printed "killed 1 dangling process" each time, but that was the test's time limit firing afterwards, not the cause. The test preload (tests/setup.local.ts) now does a full memory clean-up before each synchronous spawn, the workaround the fix's author gives. With it, 16 of 16 whole-suite runs passed (four rounds of four at once, plus a demo each round), none froze, and each took about 413 s against about 405 s without it. Had the freeze rate stayed at 3 in 16, a clean 16 would come about 1 time in 30. On GitHub, the test step of this PR's next run finished instead of freezing.
  • A proof step's clone tripped over git's clean-up. After a fetch, newer git (2.55 on GitHub's runner) runs its automatic clean-up in the background. A clone made a moment later failed copying the lock file the clean-up held. The steps' git commands now run with that clean-up off; a new test in scripts/proof-scratch.test.ts fails without the change.
  • Main's tests-proven check (A change's tests are proven to test it: each new or edited test must fail on main's code #136) refused 66 of this branch's tests, written before it landed. The check runs each new test on main's code, where it must fail. A test in a file that can't load there counts only if it runs new lines of the first file bun says is missing. So:
    • scripts/proof.test.ts is split by the code its tests run: proof-run, proof-report, proof-cli, proof-due and proof-scratch (.test.ts each). The bun run proof argument check moved into toProve in scripts/proof/cli.ts, unchanged, so its test runs it in-process, not only through a subprocess, which coverage can't see.
    • The Slice 0 proof tests split the same way, into proof-slice-0-runs and proof-slice-0-named-tests. The steps prove a commit, but the check's copy leaves the change uncommitted on main's commit. So the tests now hand the steps this checkout's files as they are on disk, committed into a scratch repository.
    • The tests of proof.yml, the gate's own code, the tracker's parser and ROADMAP.md's Proof line pass on main's copy, because the check carries everything but product code over to it. They carry the mark, listed at the top. One proof.yml test had checked the run's 100-minute limit too. That check is now its own test in scripts/proof-run.test.ts: a run whose clock jumps past 100 minutes refuses its next step for lack of time.
    • The step for a new test that already passes on main must now pass here, since its safeguard is on main.

Each group was fixed by one agent and checked by another told to refute it. A reviewer caught one mark the check could not see, because bun reported its test at the wrong line, and the in-test import of the 100-minute limit. Both are fixed. No test was lost or renamed, and none lost an assertion. The two tests-proven tests that failed in the first run were already fixed on this branch: they now wait for main's pull-request check.

On GitHub only, the check still failed 17 Slice 0 proof tests in its copy, though every local copy passed, a CI-shaped one included. Two temporary runs found why:

  • First run: the test file built its scratch repository at load on GitHub, which makes the check print the first error: FETCH_HEAD: not a valid SHA1, right after a fetch that had succeeded.
  • Second run: a fetch into a named ref, with git's own output, said warning: rejected refs/proof/base because shallow roots are not allowed to be updated. The check's copy on GitHub's runner is shallow (why is not known; local copies never are). Fetching from a shallow repository, git exits 0 but writes nothing unless given --update-shallow.

The fix: the tests' scratch repository is now one fresh commit of the checkout's files, also its origin/main, and nothing is fetched from the checkout. The steps' own clones (scripts/proof/scratch.ts) fetch into a named ref rather than FETCH_HEAD. The proof workflow checks out full history (fetch-depth: 0), so a real proof run never starts from a shallow checkout. scripts/proof-slice-0-copy.test.ts makes its copy shallow, as GitHub left the check's: with the earlier helper it fails with GitHub's exact error, and with this one it passes. Both temporary lines are gone.

What the verifier found

The verifier ruled FAIL twice:

  • On 530f849:
    • The The verifier checks behaviour is tested where people use it, and source-reading checks leave test files #137 item's evidence clause had been lost while taking main; it is back exactly as main has it.
    • No test checked that rules-from-main accepts the runner's real report. scripts/proof-report-link.test.ts now pastes the report prove writes with GitHub's run settings into a body: the check accepts it and refuses the body without it.
    • No test ran the banner command with a proof-ready slice. A test in scripts/cycle.test.ts now runs scripts/cycle-hook.sh --banner with a stand-in gh, seeing the run line and the "could not read them" line; it fails with the banner's lookup removed.
  • On d0aef97: GitHub's checks failed on the FETCH_HEAD problem above, and the copy test claimed more than it checked. Both are fixed.

Why

Until now nothing could close a slice: "done" needed a proof run and a place to see it, and nothing checked the link an agent would write. Now a slice closes only on a run GitHub itself vouches for, and Slice 0's run exercises every safeguard the slice built.

For example, the step that hands the gate gh release create v1.0.0 passes only because the gate refuses it. On a copy of the repo whose gate, hooks and guards refuse nothing, every one of the 22 refusal steps fails, and a test checks exactly that.

The workflow follows the deploy rather than the tests, because from Slice 2 on the deploy is what puts the new code on the test server.

Outside review

An outside reviewer went through this pull request at commit 40da44b. They found one blocker, nine problems worth fixing and nine smaller points. Every blocker and should-fix is now fixed, each with a test that failed before the fix.

To check that, I ran this branch's tests against the reviewed code (40da44b's sources with the new tests, plus the new function names so the files load). 26 of the 79 tests the first round of fixes came with fail there. The 6 tests added in the last round fail on the code just before their fix.

The blocker: the interrupted-work step could never pass.

  • What went wrong. The step played the interrupted session in the repository's main checkout, a few seconds before the new session started. The checkpoint safeguard never holds a session in the main checkout (where Devesh keeps his own files). It also leaves a change alone while the session that made it was at work there in the last 15 minutes.
  • Now. The step works in a worktree of its own and runs the first session's gate calls on a clock set an hour back. It passes on main now that A restarted session is shown its checkpoint, and interrupted work holds the next session at the done gate #133 is there. A test, with a stand-in gate, checks that every gate call happens in a worktree, and that the first session went quiet at least 15 minutes before the second began.

Problems worth fixing, all fixed:

  • The banner could ask for a link rules-from-main refuses. One run proves every due slice. If another slice failed, the run as a whole failed, yet the banner said "set it to done" for the slice that passed. Now both read the run through one function, and the banner says to wait for a later run instead. A test feeds the banner's own line into rules-from-main both ways.
  • Step time limits were never enforced. Every step blocks while its commands run, so the timer could not fire, and a hung command hung the whole run. Now every command a step starts through the proof's helpers is stopped when the step's time is up, a step that overran is never reported as passed, and the run stops at 100 minutes. Example from the test: a step limited to 1.2 seconds that runs sleep 6 now fails in about 1.2 seconds.
  • The ci.yml step proved nothing about ci.yml. Now that step and the loosened-gate step run main's rules-from-main workflow as GitHub runs it, and both check that an unexplained change to ci.yml, or to the gate, is refused on its own. Tests show each step failing on a copy where the workflow runs on pull_request, judges with the branch's own gate, judges nothing, or doesn't count ci.yml as a rule file.
  • Four steps handed the run's GitHub token and AI key to the gate's hook. The helper now takes the environment to use, and the proof gives it only the PATH and home folder. A test with a stand-in bun sees no token in any hook call.
  • The verifier step would likely fail, or pass for the wrong reason. Now the sample is a whole change (a route that calls the function, with an API test for one company only), the verifier's FAIL counts only when its ruling line names the other company, the proof job installs the browser, and the step takes the shared local stack's turn.
  • The agent-session checks judged words, not meaning. Tried on every item of every slice: the first round's rule got 26 of 2,833 quoted answers wrong; the rule now in the branch gets 2 wrong. The /ready answer is judged against what the worker's code checks, read from the clone.
  • The banner could make a session's start miss its 20-second limit. It now gives up on GitHub after 8 seconds in all.
  • A done slice could be rewritten without a new proof. Now rules-from-main refuses any change to such a slice's title, goal, Proof line or items, and a slice set to done with an unticked item; the tracker's parser flags that too.
  • The model steps' cost was only stated per run. The caps are $1 per agent-session exchange and $5 for the verifier: at most about $9 a run. STATE.md → Waiting on Devesh states the ceiling: about $100 a day at ten deploys. When Anthropic refuses the key for its limit, the steps wait on Devesh rather than fail.

Not changed, and why:

  • rules-from-main still asks GitHub's API without a login. A refusal for GitHub's rate limit says so, and re-running the check fixes it. Passing a token into that pull_request_target job would change the check every pull request runs, so it belongs in its own pull request.
  • The item is ticked while two earlier Slice 0 items are still open (tests proven, and the verifier checking test layers). The item says it comes last because it proves Slice 0 with every safeguard in place; merge order doesn't change that. The proof workflow proves only proof ready or done slices, and Slice 0 becomes proof ready only when its last item lands, so its first real run comes after every safeguard is in place.
  • No later slice has an item that writes its proof steps. When Slice 1 reaches proof ready, its proof fails with "no proof steps yet", and the banner has the session add an item for that.
  • scripts/proof-slice-0.test.ts adds over a minute to every gate run. Most of it goes to the checks that every step whose safeguard is on main passes, and that every refusal step fails on a copy that refuses nothing. Those checks keep the steps honest.
  • The agent session's model is not pinned. The proof should test the model sessions actually run on.

Evidence

  • bun run proof 0 on this branch with main merged through A test's child processes are stopped, with everything they started, however the test ends #140 (226aa57, 2026-10-04) ran all 28 steps and saved its report. None failed:

    • 26 passed. Among them:
      • main's rules-from-main workflow, run as GitHub runs it, refused a branch whose own loosened gate passed it, and a branch whose ci.yml checks nothing;
      • the gate's hook refused a merge wrapped in sh -c or run with --admin;
      • GitHub's records of the last 10 staging-migrations runs on main, read with gh: the 9 that ran started after checks passed on their commit and skipped the apply step, and 1 skipped its job;
      • four whole test suites and a demo started at once all passed, leaving another company's rows and queued jobs untouched (404 seconds), and two gate runs started at once in two worktrees both passed (5 seconds);
      • the four tests the Proof line cites by name each ran alone and passed;
      • A change's tests are proven to test it: each new or edited test must fail on main's code #136's refusal of a new test that already passes on main, by main's own bun run gate proven (""one is still one" passes on main's code too, so it doesn't test this change");
      • the five refusals Every stop judges what changed and says what it did not run #132-The done rules see removed tests, moved code, re-exports and every product file #135 brought: a stop without Docker shown "Done gate ⚠ NOT fully checked: tests, database policies and browser checks did not run here; CI runs them"; a deleted test file refused ("services/worker/test/agents-api.test.ts > whole file: the change deletes this test file; to remove it on purpose, add a line to docs/removed-tests.md …"); the export only a barrel re-exports refused at its own line ("packages/shared/src/proof-sample.ts:2 exports proofSample, but nothing outside tests uses it"); an added line no test runs refused ("packages/p/src/a.ts: lines 3-4 never ran"); and a session after interrupted work shown its checkpoint and held ("Done gate ⚠ this session is held to unchecked work …"), its stop sent back until the change passed, then let through.
    • 2 waited on ANTHROPIC_EVALS_KEY: the scripted agent session and the verifier's run. So bun run proof 0 now waits only on the key.
  • The real Claude Code, without a real key. claude auth status, with only a stand-in key and a scratch home, reports "authMethod": "api_key", "apiKeySource": "ANTHROPIC_API_KEY", so the session never uses Devesh's login. With that stand-in key, the agent-session step started claude -p in the clone, and the clone's SessionStart hooks ran (the banner reached the session).

  • The tests: 88 tests across scripts/proof-run, proof-report, proof-cli, proof-due, proof-scratch, proof-workflow, proof-slice-0, proof-slice-0-runs, proof-slice-0-named-tests, proof-slice-0-fingerprint, proof-agent-checks, cycle-proof, tracker-proof-passed and done-gate-proof (.test.ts each). They check:

    • each refusal step passes here (all 22, now that the tests-proven safeguard is on main);
    • every refusal step (22) fails on a copy whose gate, hooks and guards refuse nothing;
    • every phrase Slice 0's Proof line quotes, and every test file it names, is in a step's words, and a cited test that is renamed or failing fails its step;
    • the agent session and the verifier's run, with a stand-in claude: they wait without the key, get only that key and a home of their own, hold one resumed conversation in a fresh clone and save the transcript. They fail when the session makes a branch or the verifier passes the change;
    • everything listed under Outside review.
  • Main's copy versus this branch's on a pull request that sets Slice 0 to done, in a scratch repo, using a real staging-migrations run (37107299842): main's copy passes it; this branch's copy asks GitHub's API without a login, then refuses it as "a run of .github/workflows/staging-migrations.yml, not of the proof workflow".

  • bun run gate on the final commit (be47321), run bare, exit 0: "Done gate ✓ typecheck, lint, guards, 1314 tests, 67 tests proven, database policies, 29 browser checks". It shows the two exceptions this body names: the sample line that switches the type checker off, which the proof writes into a scratch branch for main's copy to refuse; and the test that reads a workflow or gate file only to write a changed copy into a scratch clone. No line this pull request adds is under apps/, services/ or packages/, so main's line-coverage rule has nothing to judge and no coverage mark is needed.

  • Verifier, on be47321: PASS. "on be47321 GitHub's checks and the local gate pass; bun run proof 0 passed 26 steps and 2 wait on Devesh's evals key; the shallow-copy failure is fixed and its test fails with the old code". It ran the gate and bun run proof 0 itself, rebuilt the shallow-fetch failure with plain git, and checked its earlier findings stay fixed (it ruled FAIL on 530f849 and d0aef97; see What the verifier found).

  • GitHub's checks on be47321: every step passed, "Tests proven against main's code" among them: "67 passed on this change and failed on main's code, of 93 new or edited test(s) in 18 file(s)".

  • This body, with 9999 in place of PR-THIS (and STATE.md's and ROADMAP.md's PR-THIS too, in a scratch copy), passes bun run gate pr with main's copy.

  • Not yet seen:

    • the workflow itself on GitHub. It runs only from main, so its first run comes after this merges (the next deploy to main, or 03:23 UTC);
    • its two model steps, which wait until Devesh saves ANTHROPIC_EVALS_KEY. Until a real model answers, the checks of its answers and the verifier's $5 cap are untried.

🤖 Generated with Claude Code

devesh911 and others added 30 commits October 3, 2026 13:49
…nd source-reading checks leave test files

Work in progress before the verifier proof run and records.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd pass on the change

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s let it read the instructions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er's proof lines, and rules-from-main checks a slice set to done

bun run proof <slice> (scripts/proof/) runs a slice's steps in order, each passed,
failed or waiting on a named STATE.md → Waiting on Devesh item, and writes a short
report. .github/workflows/proof.yml runs it after each deploy to main, daily and by
hand, keeps the report as the run's artifact and comments it on the standing
"Proof reports" issue. bun run cycle --banner shows each proof-ready or done
slice's latest run and what to do. rules-from-main refuses a pull request that sets
a slice to done unless its link is a passing proof run of that slice on main, read
from GitHub's API, and its body carries the run's report. The field
"Seen by Devesh:" is renamed "Proof passed:".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le changes, and AGENTS.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng its files, so no test reads a source path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ted, the verifier stops only its own worker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e Supabase file cite the files that hold their facts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elper or fixture outside a test folder never counts

A test that read a new fixture from disk outside a test folder threw on main's
code for want of it and was counted as proven. Main's copy now holds every file
of the change that isn't product code, so only product code differs between the
two runs. lcov.ts is now the line-coverage item's file byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le's decision; the database and migration gap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n scratch copies it removes

Every step runs against the local stack and scratch clones, branches and
repositories of the proved commit: the refusals are handed to the gate's hook
or judged by main's copy (pr --head) and never run; GitHub's run records are
read with gh reads only; the scripted agent session and the verifier's run use
claude -p with only ANTHROPIC_EVALS_KEY and wait without it. Six steps whose
safeguards are still on other branches fail on today's main and pass on theirs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he Waiting key's spending per run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a code file with options, a read whose comment claims a fixture, createReadStream

A typed field ({ type: "text" }) no longer trips them. The check lives in scripts/done-gate/source-reading.ts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n it changes a .sql file there, not the folder's README

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n the console, so require() and a spread prop can't get past

Biome's import rule never sees a required subpath, and its raw-HTML rule sees only the JSX attribute. The guards read text, as the tests they replaced did, and also catch innerHTML, outerHTML, insertAdjacentHTML and document.write. Lint keeps the Supabase rule, now tested against require() too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…settings test reads no source

guardrails-save.test.tsx drives the Settings page's real hooks with the network faked: a PUT checked against the shared schema, signed in, to the company's guardrails, then the shown policies refreshed; a bad policy never leaves the browser; a refused save refreshes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he configuration screen

So the console's entry can't stop mounting Boot unnoticed: with main.tsx rendering the app directly, the check fails (no configuration screen). The build goes to dist-e2e-no-settings, ignored like dist-e2e; .agents/ is ignored too (fix-when-touched entry 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on the local stack only

Step 5 finds and reads each test; the gate already ran it under the shared lock with a fresh console build, and a single test run by hand could collide with another checkout or reuse its console on 4173. Step 6 never calls a real provider. The API tests call app.fetch. The wording test says it is a wording guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er check, the save test and the done rules now cover, and what still gets past

The done rules' text-import hole is narrowed, not closed, and STATE.md names the ways still open. The verifier wording pin names no source path, so the done rules let it read the instructions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s own, marks read from the file

The outside review found the check counted tests it should not. Now:
- both sides run in scratch clones of main's commit (with git history), the change's copy holding every
  changed file and main's holding all but product code, so product code is all that differs;
- in a test file that can't load on main, each test counts only when it, run alone, runs an added line
  of the missing file beyond what loading the file runs;
- a new file under a product folder that only tests name is a helper or fixture and comes along to main;
- a test edited only by deleting a line from it is judged;
- marks are read from the test file for every test the change touches, the gate's own tests included,
  so the stop and the PR body show the same marks the check accepts;
- a stop counts the tests proven, not the marked ones, and a stop without Docker names this check;
- a link in the change stays a link; a test MAY_SKIP lets skip is noted, not judged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d only by another helper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…product code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tes it, and the gate's own files never name one

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the second on the final instructions showing the schema check and the duplicate, failure and retry cases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… in; STATE, decisions, rule changes and AGENTS.md say what the check now does

The helper rule follows the gate's own product definition (migrations and seeds aside), and the two
comments that had become untrue (the check list, the moved-lines note in cli.ts) are corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The interrupted-work step now runs in a linked worktree with the first
session an hour in the past, so it passes on the checkpoint branch's
current head; each step's time limit stops the commands it starts; the
banner judges a run as rules-from-main does and gives up within 8 s;
rules-from-main refuses a done slice changed while it keeps its link and
a done slice with an unticked item; the gate and ci.yml steps run main's
rules-from-main workflow as GitHub runs it; the verifier helper hands the
hook only PATH and home; the verifier step's sample is a reachable route
missing only its cross-company test, under the shared-stack lock; the
agent session's checks judge meaning; a refused evals key waits on
Devesh; the report fits one comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devesh911 and others added 8 commits October 4, 2026 13:33
…' into feat/verifier-checks-test-layers

# Conflicts:
#	ROADMAP.md
#	STATE.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… gitleaks' results.sarif

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for source code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…test-layers

# Conflicts:
#	ROADMAP.md
#	STATE.md
…' into feat/proof-runner

# Conflicts:
#	STATE.md
#	scripts/done-gate/pr.ts
# Conflicts:
#	ROADMAP.md
#	STATE.md
#	apps/console/test/app-error-boundary.test.tsx
#	apps/console/test/boot.test.tsx
#	apps/console/test/console-boot-honesty.test.tsx
#	apps/console/test/conversation-link.test.tsx
#	apps/console/test/guardrails-save.test.tsx
#	apps/console/test/pages-adoption-tasks-agents-settings.test.tsx
#	apps/console/test/transcript-xss.test.tsx
#	apps/console/test/vite-api-url-honesty.test.tsx
#	lessons.md
#	scripts/done-gate-source-reading.test.ts
#	scripts/lint-structure.test.ts
#	scripts/verifier-instructions.test.ts
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Deploying revenue-os-console with  Cloudflare Pages  Cloudflare Pages

Latest commit: be47321
Status: ✅  Deploy successful!
Preview URL: https://df892120.revenue-os-console.pages.dev
Branch Preview URL: https://feat-proof-runner.revenue-os-console.pages.dev

View logs

devesh911 and others added 17 commits October 4, 2026 17:07
…anged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ronous

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…steps' clones no longer trip over git's clean-up

Bun's synchronous spawn can miss a finished command's exit and spin forever (oven-sh/bun#34069, fix unreleased):
it froze about one whole-suite run in five when four ran at once, and CI's test step for over two hours. The test
preload runs a full memory clean-up before each synchronous spawn, the workaround the fix's author gives.

On GitHub's runner a proof step's clone failed copying the lock file git's automatic background clean-up held after
a fetch; the steps' git commands now run with that clean-up off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an-up switched off in the proof steps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t by the code they test, and true marks where main's copy holds it

Main's tests-proven check (#136) refused 66 of this branch's tests. scripts/proof.test.ts and the Slice 0 proof tests
are split so each file's first missing import is the new file its tests run; the Slice 0 steps now prove the
checkout's files as they are on disk, committed into a scratch repository, since the check's copy leaves the change
uncommitted on main's commit. Tests of the gate's own code, the tracker's parser and proof.yml, which main's copy
carries over (none is product code), carry the "behaviour already on main" mark. The run's 100 minutes are now proven
by a run whose clock is moved past them. The step for a new test that already passes on main must now pass here, as
its safeguard is on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…check counts it as product code the workflow runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ps passed, 2 wait on the evals key, none failed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the change, as GitHub's checkout makes it

The check's copy on GitHub's runner failed 17 of them though every local copy passes; this test shows the copy's
own output when it fails, in the test step that prints it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…unner's real report checked by rules-from-main, the banner command run with a proof-ready slice

- ROADMAP.md: the #137 item's clause "the gate's tests-proven check passes on it …", lost while taking main
  earlier, is back as main has it.
- scripts/proof-report-link.test.ts: the report the runner writes with GitHub's run settings, pasted into a pull
  request body, passes rules-from-main's link check; the body without it is refused.
- scripts/cycle.test.ts: the banner command in a scratch repo whose origin/main has a proof-ready slice shows its
  latest run through a stand-in for GitHub's command line tool, and says it could not read them when that fails.
  It fails with the banner's lookup removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r, coverage on, as the tests-proven check does

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ository at load, so the tests-proven check names why it fails there; the copy test works in a checkout without origin/main

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng FETCH_HEAD, which the tests-proven check's copy on GitHub's runner found not valid

The temporary load-time line stays for this one run on GitHub, so a remaining failure there names its cause. The copy
test claims only what it checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tory at load on GitHub is gone; it named the cause

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ository at load and, when the fetch leaves no ref, say on one line what the fetch did

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etching nothing from a checkout that may be shallow

GitHub's runner left the tests-proven check's copy shallow, and git, fetching from a shallow repository, exits 0 but
writes no ref ("rejected … because shallow roots are not allowed to be updated"). The helper now commits the files
on disk as one fresh commit, also its origin/main. The copy test makes its copy shallow, so it fails with the earlier
helper as the check failed on GitHub. Both temporary diagnosis lines are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant