The lens ledger records the SCOPE the audit skill ran at, and a whole-repo or diff-only review is refused

[thedavidmeister]

Closes #155

The gap, restated from the code

#153 made the lens ledger record that the audit skill ran: run-timings sits in the runner's live pipe, writes one row per Skill invocation the instant the harness announces it, and record_gate refuses a verdict on a PR with no row. What the row did not carry is the scope, and a lens pointed at the wrong scope is not a weaker review — it is a review of different code. Both wrong scopes are real invocations, so both pass NoLens:

• diff-only structurally cannot see ramifications (raindex#2778: signer<256> silently resolves to row 0, falsified only by reading the callee, which reverts);
• whole-repo on a PR buries the merge-relevant finding (rainlanguage/rain.deploy#21: twelve findings, five bearing on the PR, seven pre-existing — and the one that mattered, a new public API shipped without a [package] version bump against an already-published version in a repo that autopublishes on push to main, was one line in a list of twelve).

The one real invocation in existence is in this state. The single audit-skill call in review-runs/20260729T171735Z.jsonl (line 57) describes its scope in prose — "Scope: the changed lines plus callers of blockNumberStore.refresh/reset" — and declares none. the_real_runs_invocation_declared_its_scope_in_prose_and_therefore_declared_none asserts exactly that against the verbatim args, so the measurement this rests on is a test rather than a claim in an issue.

Where the scope comes from

The runner writes it, from the invocation. lens_invocation already reads the announced Skill event out of the harness's live stream; it now also reads the declared scope off the same input.args and lens_record puts it on the row. Nothing is parsed out of verdict text and nothing is asked of the model at verdict time — a model-written scope claim is a claim about itself, which is the reasoning that made #146 choose the linter shape over a findings record and #153 choose the stream-json observable.

Why args is still the carrier and why that is not a self-report: the model authors the string, but it does not author the fact that the harness announced it, it cannot amend the ledger (its tool array is [Glob, Grep, Read, Skill, mcp__fsm__*] — an absent write tool, not a deny rule), and the row is on disk before the announced tool has run. It buys the same thing #153's invocation half buys: the credit cannot be had without paying for a real invocation at a declared scope, and the record says which. It does not become proof, and the README says so.

The vocabulary is pinned to the audit skill's own (rainlanguage/claude-audit-skills#66), three values and no fourth: whole-repo, pr:<number>, paths:<comma-separated globs>.

The ruling: for a PR verdict, only pr:<this PR's number> is legal

• whole-repo reviewed the repository.
• paths:<globs> reads the files the list names and nothing that decides whether they are right. On a PR the only file list to hand is the diff's, so a path-scoped invocation is the diff-only lens under another name — and even where the globs reach wider they are a list the caller assembled, not the ramification set the "would understanding it change the ruling on THIS diff?" test derives.
• pr:<other> declares it reviewed a different PR than the one it was credited to.
• a value that does not parse is wrong, not absent: this filter only ever writes the three canonical values, so an unrecognised one came from somewhere else and cannot be honoured.

RecordGate::WrongLensScope, exit 6, sits immediately after NoLens and before SolConvention. After, because "at what scope" is not a question about a PR whose source was never checked out or whose skill was never invoked — asking it there reports a scope defect to a vetter whose next move is pr_checkout. Before the two below, for #151's own reason one level in: there is no point reporting a pragma or an anchor range to a vetter whose lens read the wrong code, and both are answered by re-reading at pr:<n> anyway.

Exit 6 and not a second spelling of 5, by the same test #153 used for 5 vs 4: five says the code was not read, six says the wrong code was read, and the repairs differ (a checkout plus an invocation, versus re-invoking a skill that already ran). The codes are now one tested table (record_gate_exit_code) that the write's arms read, rather than eight literals scattered through them — that is what makes six's distinctness a mutation-killable property instead of a comment.

Absent, wrong, unobserved — three states, three messages

#153 established that a record must distinguish absent from unobserved. One level in, that becomes three:

state	evidence	outcome
no ledger at all	SourceOnly	records, stamped invocation UNOBSERVED (…)
a row, no scope key	ScopeUndeclared	exit 6 — "…declared NO SCOPE at all"
a row whose scope is not this PR's	WrongScope	exit 6 — "…at the WRONG SCOPE — whole-repo reviewed the REPOSITORY…"

The scope key is omitted, never null, when nothing was declared — a null would read as a value. An undeclared scope is still refused, and the message says why: the skill's own standing rule is a whole-repo snapshot, so declaring nothing is declaring whole-repo.

A scope named more than once with different values declares none, which is the ruling #153 already makes for an invocation naming two PRs and rests on the same fact — args is prose, and "scope pr:21, not whole-repo" states two values a reader would have to rank. review-prompt.txt therefore tells the vetter to write pr:<number> and not to write the other two even to say they are not being used.

The stamp names the scope off the row, so the record distinguishes a PR-scoped review from a whole-repo sweep without inferring it from the findings:

🤖 ai:vetter
vet-protocol 4
lens source@6a370a5d… + audit skill invoked at pr:386
Reviewed 6a370a5d…: ready — closes #386
cost 412 — concurrency guard in a store poll loop

VET_PROTOCOL 3 → 4, and what it actually retires

Confirmed from the code before relying on it: vetted_at_head requires both halves — body.contains("Reviewed {head}:") && verdict_protocol(&body).is_current() — and is_current() is self == VetProtocol::Stamped(VET_PROTOCOL), so an unstamped comment is VetProtocol::Unknown and never current.

Then measured live rather than assumed. 330 open PRs across rainlanguage + cyclofinance + S01-Issuer, each PR's latest trusted 🤖 ai:vetter comment read via GraphQL:

latest vetter verdict	count
no vetter comment at all	201
unstamped (→ VetProtocol::Unknown)	98
vet-protocol 1	31
vet-protocol 2	0
vet-protocol 3	0

So the honest statement of the consequence: the bump retires 0 additional stored verdicts, because all 129 stored verdicts are already non-current under protocol 3 (98 unidentifiable, 31 stamped 1). Its function here is forward-looking and it is still required — it makes the ledger contract identifiable, so any verdict a rebuilt cron records under the scope-blind protocol 3 (i.e. in the window between #153 landing and this landing) is retired rather than read as current.

That measurement surfaces something worth flagging on its own: the installed cron binary predates #145. It is still stamping vet-protocol 1 (e.g. rainlanguage/rain.deploy#21), so neither #131's coverage gate nor #151's lens gate is in the binary the cron is running — the same "a gate only bites once the cron rebuilds the flake" observation #153's QA made, still true. Not this PR's to fix; raising it because it decides when any of these three gates first bites.

What this deliberately does not do

• No verification of the scope against the checkout or the changed-file set. A paths: list is refused for a PR outright rather than compared glob-by-glob against the diff, and a pr:<n> scope is checked for the NUMBER only. Extracting and matching path sets out of prose is the fragile half; the ruling above makes it unnecessary, because on a PR there is exactly one legal answer.
• No check on what the skill concluded. #146 is right that correctness, security and design are not decidable from source text; they stay the vetter's.
• nr.md is untouched. #154 is the caller-side half in flight against this same repo and owns that file; this PR builds the observable and the gate it needs, and review-prompt.txt is the vetter-side declaration.
• claude-audit-skills#66 is not a prerequisite for the gate to work. The prompt declares the scope, the runner observes it, and the gate checks it — all inside this repo. #66 is what makes the skill HONOUR the declared scope; until it lands, a declared pr:<n> buys a correctly-scoped ledger row and a skill still following its own whole-repo rule.

QA

• Discriminating tests: lens_gate_tests::{the_real_runs_invocation_declared_its_scope_in_prose_and_therefore_declared_none, only_the_three_scope_values_parse_and_only_in_strict_form, a_scope_is_read_out_of_args_only_when_the_string_declares_exactly_one, a_lens_row_carries_the_declared_scope_and_omits_the_key_when_there_is_none, only_this_prs_scope_is_correct_and_an_absent_one_is_not_a_wrong_one, an_invocation_at_the_wrong_scope_is_not_a_missing_invocation, the_lens_gate_and_the_scope_gate_partition_the_evidence, a_ref_that_names_no_pr_is_no_source_even_with_a_tree_and_a_ledger, the_full_stamp_names_the_scope_the_row_recorded, a_refusing_state_has_no_stamp_to_write, the_scope_refusals_say_whether_a_scope_was_declared_and_name_the_legal_one, a_verdict_whose_lens_ran_at_the_wrong_scope_is_refused_and_a_pr_scoped_one_records, the_scope_gate_refuses_every_verdict_not_only_ready, the_scope_refusal_outranks_the_coverage_refusal, the_human_no_sha_and_file_list_refusals_all_outrank_the_scope_gate, each_refusal_carries_its_own_exit_code_and_the_two_lens_codes_differ, the_review_prompt_makes_the_vetter_declare_the_scope_as_a_token}, scope_coverage_tests::the_scope_refusal_outranks_the_convention_gate, and tests/run_timings.rs::the_declared_scope_is_written_onto_the_row_by_the_runner — each fails on base because LensScope, the scope field, LensScopeState, RecordGate::WrongLensScope and the prompt clause do not exist there (the type-level ones cannot compile on base, which is the strongest form of that). the_declared_scope_is_written_onto_the_row_by_the_runner and an_audit_invocation_is_on_disk_before_the_tool_it_announces_could_have_run drive the REAL binary as a child process through the real pipe, and the latter now also asserts the real run's row carries no scope key — the whole point of the field is that the process the model does not control writes it, so a unit test over lens_record alone would assert our own belief about that rather than observe it. Both repo-root conformance tests resolve through repo_root_text, whose lookup is pinned by repo_root_tests, so neither can pass by not running. Suite: 632 unit + 7 run_usage + 47 require_qa_block + 8 run_timings + 7 refresh_human_queue green (base: 615 + 7 run_timings — 17 unit and 1 integration test are new).

• Mutations applied: 34 mutants, 34 killed, 0 survivors, 0 that failed to compile. Baseline committed before mutating; the tree restored between each and re-verified green after the last (mutate-155.py, gitignored by the repo's /* default). Direction, grouped by what each breaks:

  • the gate fails open — lens_scope_refusal ScopeUndeclared→None, WrongScope→None → an_invocation_at_the_wrong_scope_is_not_a_missing_invocation; the gate deleted from record_gate → a_verdict_whose_lens_ran_at_the_wrong_scope_is_refused_and_a_pr_scoped_one_records.
  • the two gates stop partitioning the evidence — lens_scope_refusal claims NoSource, and lens_refusal claims WrongScope as NotInvoked → the_lens_gate_and_the_scope_gate_partition_the_evidence / an_invocation_at_the_wrong_scope_is_not_a_missing_invocation. This pair is the one that needed a purpose-built test: the two gates are ADJACENT in the chain and their states are disjoint, so swapping the two if let blocks is inert and no ordering test can see it — what is load-bearing is that neither gate answers for the other's states, which is what the partition test asserts over all six.
  • absent collapses into wrong, or into correct — LensScopeState::Undeclared→Correct, →Wrong(vec![]) → only_this_prs_scope_is_correct_and_an_absent_one_is_not_a_wrong_one; the message swapping "NO SCOPE" for "WRONG SCOPE" → the_scope_refusals_say_whether_a_scope_was_declared_and_name_the_legal_one.
  • the required scope widens — any pr:<n> accepted, whole-repo accepted, paths: accepted, and a later wide scope retracting a correct one → only_this_prs_scope_is_correct_and_an_absent_one_is_not_a_wrong_one.
  • the vocabulary loosens — pr:0 accepted, paths: with no globs accepted, whole-repo prefix-matched → only_the_three_scope_values_parse_and_only_in_strict_form.
  • the args scan loosens — two declared scopes credited to the first, and splitting on , (which shreds a paths: glob list) → a_scope_is_read_out_of_args_only_when_the_string_declares_exactly_one.
  • the row stops being honest — the scope never written, written as null instead of omitted, and an empty scope string read as a declaration → a_lens_row_carries_the_declared_scope_and_omits_the_key_when_there_is_none / the_ledger_reader_keeps_lens_rows_and_survives_a_half_written_line.
  • the invocation predicate absorbs the scope (so a scopeless invocation reads as never having happened) → only_a_skill_tool_use_naming_the_audit_skill_and_exactly_one_pr_is_an_invocation.
  • the evidence resolver fails open — an unparseable PR ref defaulting to 0 instead of NoSource → a_ref_that_names_no_pr_is_no_source_even_with_a_tree_and_a_ledger.
  • the stamp lies — the scope dropped from the full stamp, and a wrongly-scoped verdict given a stamp anyway → the_full_stamp_names_the_scope_the_row_recorded / the_lens_gate_and_the_scope_gate_partition_the_evidence.
  • position, from both sides — below SolConvention → scope_coverage_tests::the_scope_refusal_outranks_the_convention_gate; below Uncovered → the_scope_refusal_outranks_the_coverage_refusal; above NoDiff → the_human_no_sha_and_file_list_refusals_all_outrank_the_scope_gate; narrowed to ready → the_scope_gate_refuses_every_verdict_not_only_ready.
  • the exit code — 6 → 5 → each_refusal_carries_its_own_exit_code_and_the_two_lens_codes_differ. This mutant is the reason the codes became a table: with the literals in the write's arms it was unkillable, because nothing in the repo drives record_verdict_apply (which needs gh), and codes 3/4/5 were in the same position.
  • the message stops naming what was declared — the diff-only fault reworded as whole-repo, and only the first of two wrong scopes reported → the_scope_refusals_say_whether_a_scope_was_declared_and_name_the_legal_one.
  • the deployment — the runner writing the row with the scope stripped → run_timings::the_declared_scope_is_written_onto_the_row_by_the_runner; the prompt's declaration clause removed → the_review_prompt_makes_the_vetter_declare_the_scope_as_a_token; the README's stamp example losing at pr:386 → the_readme_documents_the_lens_stamp_and_the_ledger.

• Oracle: the vetter's own trace, review-runs/20260729T171735Z.jsonl. The scope extractor is exercised against the verbatim args of the only real audit-skill invocation in existence (line 57), which is what makes "the real run declared no scope" a measured fact rather than a reading of the issue. The three scope literals come from rainlanguage/claude-audit-skills#66's pinned contract, not invented here; the two failure modes and their costs (twelve-findings-on-rain.deploy#21, raindex#2778's reverting callee) come from the issue's own measurements, and raindex#2778 was independently re-derived from LibSubParse.subParserContext, which reverts. The VET_PROTOCOL consequence is measured live off GitHub (330 open PRs, GraphQL) rather than estimated from the queue snapshot. parse_pr_ref and pr_refs_in (pre-existing, separately tested) stay the only PR-ref parsers — this change owns one new parser, LensScope::parse, for one closed three-value vocabulary.

• Category check: The lens ledger records that the skill ran but not at what scope, so a whole-repo or diff-only review passes NoLens #155 asks for (a) the scope on the ledger row, written by the runner from the invocation and never from verdict text; (b) a record_gate refusal for whole-repo-on-a-PR and for diff-only, positioned immediately after NoLens and before SolConvention, with a distinct exit code; (c) absent kept distinct from unobserved, with the refusal saying which; (d) VET_PROTOCOL 3 → 4, having confirmed the vetted_at_head and VetProtocol::Unknown semantics from the code; (e) review-prompt.txt declaring the scope without becoming an evidence-of-scope preamble. Covered: (a) lens_invocation + lens_record, read off the announced Skill event in the runner's pipe; (b) RecordGate::WrongLensScope, exit 6, position driven from both sides by four ordering tests; (c) three states, three messages, scope key omitted rather than null; (d) bumped, semantics confirmed and quoted above, consequence measured; (e) the prompt declares a token and the gate reads it off the invocation, with the "nothing to narrate" clause extended to cover a scope claim specifically, asserted by the_review_prompt_makes_the_vetter_declare_the_scope_as_a_token. Nothing is Refs rather than Closes.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@thedavidmeister, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 38 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c8d7aa67-c6bc-4a79-8dea-1d7be0a27af6

📥 Commits

Reviewing files that changed from the base of the PR and between d06bc93 and aec57c5.

📒 Files selected for processing (4)

• README.md
• pr-review-report-rs/src/main.rs
• pr-review-report-rs/tests/run_timings.rs
• review-prompt.txt

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[thedavidmeister]

QA evidence — run locally on aec57c5

Mutation results for the new gate

34 mutants, 34 killed by the test named for each, 0 survivors, 0 that failed to compile. Driver: mutate-155.py (gitignored by this repo's /* default), one targeted edit at a time, tree restored between each and re-verified green after the last. Baseline was committed before mutating.

#	mutation	killed by
1	lens_scope_refusal: ScopeUndeclared → None	an_invocation_at_the_wrong_scope_is_not_a_missing_invocation
2	lens_scope_refusal: WrongScope → None	same
3	lens_scope_refusal claims NoSource (the gate above's state)	the_lens_gate_and_the_scope_gate_partition_the_evidence
4	lens_refusal claims WrongScope as NotInvoked	an_invocation_at_the_wrong_scope_is_not_a_missing_invocation
5	LensScopeState::Undeclared → Correct	only_this_prs_scope_is_correct_and_an_absent_one_is_not_a_wrong_one
6	LensScopeState::Undeclared → Wrong(vec![])	same
7	any pr:<n> counts as correct	same
8	whole-repo counts as correct	same
9	paths: counts as correct	same
10	a later wide scope retracts a correct one	same
11	parse: pr:0 accepted	only_the_three_scope_values_parse_and_only_in_strict_form
12	parse: paths: with no globs accepted	same
13	parse: whole-repo prefix-matched	same
14	scan: two declared scopes → the first wins	a_scope_is_read_out_of_args_only_when_the_string_declares_exactly_one
15	scan: split on , too (shreds a paths: glob list)	same
16	row: the scope never written	a_lens_row_carries_the_declared_scope_and_omits_the_key_when_there_is_none
17	row: written as null instead of omitted	same
18	reader: an empty scope string is a declaration	the_ledger_reader_keeps_lens_rows_and_survives_a_half_written_line
19	lens_invocation requires a scope (a scopeless call reads as never having happened)	only_a_skill_tool_use_naming_the_audit_skill_and_exactly_one_pr_is_an_invocation
20	evidence: an unparseable PR ref defaults to 0 instead of NoSource	a_ref_that_names_no_pr_is_no_source_even_with_a_tree_and_a_ledger
21	stamp: the scope dropped from the full stamp	the_full_stamp_names_the_scope_the_row_recorded
22	stamp: a wrongly-scoped verdict gets one anyway	the_lens_gate_and_the_scope_gate_partition_the_evidence
23	gate deleted from record_gate	a_verdict_whose_lens_ran_at_the_wrong_scope_is_refused_and_a_pr_scoped_one_records
24	gate below SolConvention	scope_coverage_tests::the_scope_refusal_outranks_the_convention_gate
25	gate below Uncovered	the_scope_refusal_outranks_the_coverage_refusal
26	gate above the NoDiff guard	the_human_no_sha_and_file_list_refusals_all_outrank_the_scope_gate
27	gate narrowed to ready	the_scope_gate_refuses_every_verdict_not_only_ready
28	exit code 6 → 5	each_refusal_carries_its_own_exit_code_and_the_two_lens_codes_differ
29	message: "NO SCOPE" reworded to "WRONG SCOPE"	the_scope_refusals_say_whether_a_scope_was_declared_and_name_the_legal_one
30	message: the diff-only fault reworded as whole-repo	same
31	message: only the first of two wrong scopes reported	same
32	runner writes the row with the scope stripped	run_timings::the_declared_scope_is_written_onto_the_row_by_the_runner
33	prompt: the scope-declaration clause removed	the_review_prompt_makes_the_vetter_declare_the_scope_as_a_token
34	README: the stamp example loses at pr:386	the_readme_documents_the_lens_stamp_and_the_ledger

Two of these are worth calling out because they are the ones that needed a test built for them rather than an existing one:

• Queue vs retired python: ledger lines with missing 'source' are included as ai-campaign #3/classify_ci treats a STALE check conclusion as green-contributing #4, the partition. The lens gate and the scope gate are ADJACENT in the chain and their LensEvidence states are disjoint, so swapping the two if let blocks is inert and no ordering test can see it. What is actually load-bearing is that neither gate answers for the other's states — so the_lens_gate_and_the_scope_gate_partition_the_evidence walks all six states and asserts each is refused by exactly one gate (or neither, for the two recordable ones) and has a stamp iff it is recordable. Without it, Queue vs retired python: ledger lines with missing 'source' are included as ai-campaign #3 and classify_ci treats a STALE check conclusion as green-contributing #4 both change which refusal a vetter reads and nothing notices.
• record-verdict: add relink verdict + --cost sidecar (3b-i, unblocks vetter wiring) #28, the exit code. With the codes as literals in record_verdict_apply's arms this mutant was unkillable — nothing in the repo drives that function, which needs gh, and codes 3/4/5 were in the same position. That is why the codes became one tested table (record_gate_exit_code) that the arms read; each arm still owns only its message.

VET_PROTOCOL consequence, measured

vetted_at_head requires both halves — body.contains("Reviewed {head}:") && verdict_protocol(&body).is_current(), with is_current() being self == VetProtocol::Stamped(VET_PROTOCOL) — and an unstamped comment is VetProtocol::Unknown, so never current. Confirmed in the code before relying on it.

Measured live over 330 open PRs in rainlanguage + cyclofinance + S01-Issuer (each PR's latest trusted 🤖 ai:vetter comment, via GraphQL): 201 have no vetter comment, 98 are unstamped, 31 are vet-protocol 1, and 0 are at 2 or 3.

So: the 3 → 4 bump retires 0 additional stored verdicts — all 129 stored verdicts are already non-current under protocol 3. The bump is still required, forward-looking: it makes the recorded contract identifiable, so any verdict a rebuilt cron writes under the scope-blind protocol 3 (the window between #153 landing and this landing) is retired rather than read as current.

Side finding, not this PR's to fix: the installed cron binary is still stamping vet-protocol 1 (e.g. rainlanguage/rain.deploy#21), so it predates #145 — neither #131's coverage gate nor #151's lens gate is in the binary the cron runs. Same "a gate only bites once the cron rebuilds the flake" observation #153's QA made. It decides when any of these three gates first bites.

Suites and CI-equivalent checks

• cargo test: 632 unit + 7 run_usage + 47 require_qa_block + 8 run_timings + 7 refresh_human_queue, all green. Base was 615 unit + 7 run_timings; 17 unit and 1 integration test are new.
• rainix-rs-static (rustfmt + clippy) green, run in nix develop github:rainlanguage/rainix/53e96a7d…#rust-shell with src/main.rs and tests/run_timings.rs touched first, so a warm target/ could not mask a lint.
• pre-commit run --all-files green in the same shell. denofmt reflowed the new README prose on the first pass — that reflow is committed as aec57c5, and the second pass is clean with nothing left modified.

[thedavidmeister]

Reviewed aec57c5: ready — scope read off the announced Skill event in the live pipe, WrongLensScope exit 6 after NoLens, absent/undeclared/wrong distinguished, VET_PROTOCOL 4 (retires nothing retroactively — 0 stored verdicts at protocol 2+, measured), 34/34 mutants killed including the two-gate partition test. Merged after #156; CLEAN against it, constant namespaces disjoint.