Check the Solidity toolchain rule against the run's own trace

[thedavidmeister]

sol-toolchain (#195) gave the producer a way to ASK which toolchain a
checkout's own CI judges it with. Nothing read the answer back. A rule with no
check cannot report its own violation, so the next wrong rule would be found the
way this one was — by a parked PR whose one permitted back-off attempt had gone
on a diff that could never pass.

Worth being precise, because the framing is easy to get wrong: the prompt was
not violated in #116. It said github:rainlanguage/rainix#sol-shell and the
producer ran exactly that. The rule was wrong, not the compliance. So this is not
"prose does not bind"; it is that nothing could tell us the rule was wrong until
a PR parked.

The measurement, which is what chose the option

The issue asks for the exposure before the choice. Over the 21 retained
traces (4,574 Bash calls, 508 nix develop/run/shell invocations):

Measured	Count
Solidity CHECKS (is_sol_check, so no forge soldeer install)	197, across 23 work clones, in 5 runs
…that named no checkout at all	8 — every one a forge --version health check
…run in a shell that checkout's own CI does not run	170 of the 189 resolvable
Busiest single run	73 checks

Reproduce the first three rows with the subcommand this PR adds:

for f in runs/*.jsonl; do pr-review-report sol-toolchain-audit "$f"; done

Every retained trace predates #195 (it merged today), so none of them ever
called sol-toolchain and all 189 resolvable checks report unasked — the
honest verdict for a run that had no way to ask. The 170 figure is the same
corpus re-audited with each clone's live sol-toolchain answer injected ahead of
it, which is what turns unasked into a comparison.

The skew is not notional. Every rainix-pin repo's CI runs RAINIX_SHA
53e96a7d (2026-07-10); the github:rainlanguage/rainix#sol-shell those runs
used floats on a rainix HEAD 37 commits and 20 days ahead of it. 9 of the
mismatches are the mirrored mistake — the checkout's OWN flake used against a
repo whose CI reads the reusable's pin and never its flake.lock. Broken down by
the audit itself: 159 unpinned-rainix-HEAD against a reusable pin, 9 mirrored,
2 other.

So "nothing, with a number" is out: 170 checks is not a small exposure. But
the shape — tens per run, concentrated in a handful of clones — is what argues
against the blocking hook.

Why the audit and not the hook

• A hook pays per Bash call what the trace already knows. 73 resolutions on
  the busiest run, each a RAINIX_SHA read out of the reusable at a floating
  ref (two GitHub reads apiece), on the critical path of every nix develop.
  The audit resolves nothing: the answer and the nix develop that followed it
  are both already recorded in the trace.
• A hook must fail OPEN on an unresolvable answer, or a network blip stops
  the run — so the guard is absent exactly when the network is flaky. An audit
  computed from a file after the fact cannot be lost that way.
• A hook has to infer which clone a command line means. The audit has the
  same problem and reports it as a blind spot instead of guessing: 8 of 197
  named no checkout, and every one of those was a forge --version health check
  rather than a check on a diff.
• It fits the repo's existing shape — distill-trace, run-timings,
  run-metrics are all post-run trace readers, and CLAUDE.md's north star puts
  parsing in the binary.

What it gives up is real and is stated in the code: it does not prevent the
wasted attempt inside the run it audits. It moves discovery of the next wrong
rule from a parked PR weeks later to the end of the run that obeyed it.

foreign is deliberately outside the judgment. A foreign checkout's toolchain is
one no nix develop enters, so there was never a shell to match; it reports as
unmatchable and changes no exit code, which is what #195's own text already
tells the producer to expect.

What landed

• pr-review-report sol-toolchain-audit <trace> — per Solidity check:
  matched / skew / unasked / unmatchable, plus the audit's own two blind
  spots (names no checkout; enters the working directory's flake without naming
  that directory). Exit 3 on skew or unasked only — failing a run on what the
  checker cannot see is the fail-closed direction, and a checker that stops a run
  on its own ignorance is worse than the skew it hunts.
• campaign-run.sh runs it after run-metrics and appends the report to the run
  log. Best-effort, like every line around it: a skew is something to READ at the
  end of a run, not a reason to fail a run whose PRs are already open.
• README section with the table above and the recipe; CLAUDE.md transition-table
  row.

The classification reuses is_sol_check — #195's own definition of a check whose
verdict depends on the shell — so forge soldeer install is excluded in both
places by the same predicate. What is new is finding the commands to ask it
about in a line a RUN writes rather than a line a workflow writes: && chains,
pipelines, bash -c '…', env VAR=V …, and several nix develops per line. 9
of the 241 Solidity invocations in the corpus are bash -c wrapped and 12 carry
more than one shell.

QA

• Discriminating tests: a_check_run_in_the_named_shell_matches,
  a_check_run_in_a_different_shell_is_skew,
  an_answer_does_not_reach_backwards_to_a_check_that_preceded_it,
  a_check_in_a_checkout_never_asked_about_is_unasked,
  a_foreign_checkout_is_recorded_not_judged,
  a_dependency_fetch_is_not_a_check,
  a_check_wrapped_in_bash_c_is_still_a_check,
  each_invocation_on_one_line_is_classified_separately,
  root_names_the_checkout_over_the_working_directory,
  the_repo_own_flake_entered_by_cd_matches_its_repo_flake_answer,
  the_audits_blind_spots_are_named_and_do_not_fail_the_run,
  skew_and_unasked_both_fail_the_audit,
  the_audits_own_invocation_is_not_a_toolchain_question,
  an_answer_with_no_verify_line_carries_its_mode,
  the_outer_command_flag_wins_over_a_wrapped_one,
  only_an_absolute_working_directory_names_a_checkout,
  a_repo_flake_answer_matches_the_checkout_however_the_question_spelled_it,
  a_trace_yields_its_bash_calls_in_issue_order_with_their_output — each fails
  on base (every one names a function this PR adds, so on 9c07414 the module
  does not compile; each was then verified against a real mutation of the line it
  is about, below)
• Mutations applied: (17 applied to src/main.rs, one per behaviour a new test claims; the two that survived the
  first pass are recorded as survivors and the tests they should have killed were
  strengthened/added before re-running — both then killed)
  • classify_sol_verify matched arm *named == used → != → a_check_run_in_the_named_shell_matches
  • classify_sol_verify skew arm → returns Matched → a_check_run_in_a_different_shell_is_skew
  • sol_audit answer collection → hoisted to a pre-pass, so answers reach backwards → an_answer_does_not_reach_backwards_to_a_check_that_preceded_it
  • classify_sol_verify None => Unasked → None => Matched → a_check_in_a_checkout_never_asked_about_is_unasked
  • classify_sol_verify Some(Err(mode)) => Unmatchable → Skew → a_foreign_checkout_is_recorded_not_judged
  • is_sol_check forge soldeer early-return deleted → a_dependency_fetch_is_not_a_check
  • runs_sol_check bash -c / sh -c unwrap deleted → SURVIVED the first pass (the test's only case was a chained command, and &&/| splitting already reaches inside the quotes); test strengthened in place with an unchained bash -c 'forge fmt --check' → now a_check_wrapped_in_bash_c_is_still_a_check
  • sol_audit invocation loop → .take(1), only the first shell on a line → each_invocation_on_one_line_is_classified_separately
  • sol_verify_target → shell_cwd checked before --root → root_names_the_checkout_over_the_working_directory
  • classify_sol_verify canonical-path equivalence arm deleted → SURVIVED the first pass (the existing repo-flake test matches on the first arm, because answer and question spelled the checkout identically); new test added → now a_repo_flake_answer_matches_the_checkout_however_the_question_spelled_it
  • sol_audit_lines exit code → counts Untargeted too → the_audits_blind_spots_are_named_and_do_not_fail_the_run
  • sol_audit_lines exit code → skew > 0 only, dropping unasked → skew_and_unasked_both_fail_the_audit
  • sol_toolchain_question → starts_with("sol-toolchain") instead of == → the_audits_own_invocation_is_not_a_toolchain_question
  • sol_toolchain_answer → Err(String::new()), mode dropped → an_answer_with_no_verify_line_carries_its_mode
  • nix_develop_inner → .next() instead of .min_by_key(offset), so an inner -c beats an outer --command → the_outer_command_flag_wins_over_a_wrapped_one
  • shell_cwd → d.starts_with('/') guard deleted → only_an_absolute_working_directory_names_a_checkout
  • sol_audit_calls → recorded output not paired onto its call → a_trace_yields_its_bash_calls_in_issue_order_with_their_output
• Oracle: the 21 retained runs/*.jsonl producer traces, read as data. Expected
  toolchains come from each repo's OWN .github/workflows as sol-toolchain
  reads them, and the pin from RAINIX_SHA in the rainix reusable — never from
  this code. The command shapes in the tests (--root, cd … &&, env -C,
  bash -c '…', two shells on one line) are transcribed from commands runs
  actually issued, not invented.
• Category check: issue asks (a) measure the exposure first — how many Solidity
  verification commands per run, across how many clones, and how many used a
  toolchain sol-toolchain would not have named; (b) choose among blocking hook
  / after-the-fact detection / removing the second place to get it wrong /
  nothing-with-a-number, on that evidence; (c) if a guard is built, say what it
  does when the answer is unresolvable. Covered a, b, c.

Closes #203

[coderabbitai]

Warning

Review limit reached

@thedavidmeister, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c61382f8-0084-48f3-9b9c-a56b20866ad4

📥 Commits

Reviewing files that changed from the base of the PR and between 9c07414 and e5bc8f4.

📒 Files selected for processing (4)

• CLAUDE.md
• README.md
• campaign-run.sh
• pr-review-report-rs/src/main.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[thedavidmeister]

Reviewed e5bc8f4: ready

Verified before merge: 20 SUCCESS / 1 skipped / 0 failures, MERGEABLE/CLEAN, 0 unresolved threads. 999 tests from a 981 baseline; 17 mutations. CodeRabbit reviewed and passed.

The measurement is what chose the option. 197 Solidity checks across 23 work clones in 5 runs — tens per run, not per-file — and of the 189 naming a resolvable checkout, 170 ran in a shell that checkout's own CI does not run. Live, not notional: rainix HEAD is 37 commits and 20 days ahead of the RAINIX_SHA every rainix-pin repo's CI actually uses.

The caveat it raised is the part I would keep. #195 merged at 08:03Z today, so every retained trace predates it and all 189 report unasked from the shipped tool — none of those runs COULD have asked. The 170 is that corpus re-audited with each clone's live sol-toolchain answer injected. So it measures the blast radius of one wrong rule rather than post-#195 compliance, which is currently unobservable — and being unobservable is the argument for building the check rather than against it.

Detection over a blocking hook, decided on shape rather than size: a hook would pay 73 toolchain resolutions on the busiest run, each two GitHub reads out of a reusable at a floating ref, on the critical path of every Bash call — to compute something ALREADY RECORDED IN THE TRACE. The sol-toolchain output sits beside the nix develop that followed it, so the audit needs no network, no clone still on disk, and cannot stop a run. It fits the repo's existing shape too: distill-trace, run-timings and run-metrics are all post-run trace readers. What it gives up is stated in the code — it does not prevent the wasted attempt inside the run it audits, it moves discovery of the NEXT wrong rule from a parked PR weeks later to the end of the run that obeyed it.

I checked the two things that would have made this wrong. The exit code cannot contaminate a run: it is || true, log-only, with the reasoning in the runner — a skew is something to read at the end of a run, not a reason to fail a run whose PRs are already open. And it refuses to judge what it cannot see, labelling Untargeted and UnknownShell as blind spots of the AUDIT rather than faults of the run, which matters because counting them as violations would inflate the very number the audit exists to report. Unmatchable covers every exit-3 mode and exits 0, with foreign explicitly outside the judgment since no shell can match it. The mode: is carried rather than collapsed to "unresolved", because the four modes want different things from a reader — collapsing them would undo the discrimination #195 exists for.

Two of its 17 mutations survived first pass and both were REAL gaps rather than equivalents: a bash -c unwrap whose only test case passed without it, because splitting on && already reached inside a chained command's quotes; and a canonical-path arm the existing test matched around because answer and question spelled the checkout identically. Both closed with new cases.

Worth knowing on the first runs after this: exit 3 also fires on unasked, and the entire retained corpus predates #195, so early reports will judge against a rule that started existing this morning. Correct going forward — after #195, not asking IS the violation.