audit: testsAfter is 84, and the record now names the tree it holds at

[thedavidmeister]

Closes #86

audit/mutation-test-scans.json claimed summary.testsAfter: 102. No tree in
the range the record covers has ever had 102 tests, and the record carried no
field naming the tree the count was supposed to hold at — so the number could
not be checked against anything.

Two changes, both to the one record:

• summary.testsAfter: 102 → 84.
• new top-level "testsAfterCommit": "ae21818a51d1b8c7df14d8c32eb39521e9c1a146",
  placed immediately after commit.

ae21818 is the merge of #39, the last of the three coveragePrs the record
itself lists (#36, #37, #39). It is the tree on which the campaign's whole
output had landed, so it is the tree every after number in summary is
measured at. commit (c72eb89) stays what it is: the tree the scan ran
against, and the tree testsBefore is measured at.

Re-derived from scratch, not taken from the issue

The issue's numbers were re-derived independently before being used, by
counting function test… declarations across every test/**/*.t.sol at every
commit from c72eb89 to HEAD:

9, 21, 22, 27, 29, 42, 51, 84, 94, 95, 104, 106, 109, 110, 121, 123, 132, 134

102 occurs nowhere in the range.

Declaration counting is only a proxy for what forge test runs, so the three
loadbearing values were then re-measured with forge test --list --json, which
is the actual oracle — a worktree per SHA, deps installed, the run's own count:

tree	what it is	forge test --list	declarations
c72eb89	the record's commit	9	9
ae21818	merge of #39, last coverage PR	84	84
935c725	HEAD at time of writing	134	134

forge test --list agrees with the declaration count at all three, and every
test function at both ae21818 and HEAD is external, so there is no
internal/private declaration inflating either figure. 9 matches the
record's existing testsBefore, which independently confirms the record's
commit field is the before-tree it claims to be.

Where the issue's proposed fix was corrected

The issue proposed "testsAfterCommit": "ae21818" — a 7-character prefix. This
lands the full 40-character SHA instead, matching the existing commit
field in the same record and the "exact SHA" wording in the tool's README. A
short prefix is a weaker anchor than a full SHA and grows ambiguous as history
grows, which is the same class of defect the field exists to close.

Field name and position are deliberately identical to what
rainlanguage/adversarial-mutation-test#23 proposes for the shared schema
(top-level, immediately after commit), so this record already conforms when
that lands. That issue is still open with no PR against it as of this writing.

Decision: no CI validator in this repo, and its ledger

The obvious next step is a CI check that re-counts the two trees and fails when
the ledger disagrees. This PR deliberately does not add one.

• Build cost: low — the check is ~50 lines of shell, and it is written and
  run below as this PR's own harness.
• Carrying cost: high, and it grows. It needs fetch-depth: 0 plus two
  extra full forge builds of historical trees on every CI run. Worse, those
  trees are frozen while solc/foundry/deps move underneath them, so the check
  reds main the first time an old tree stops building — for a reason that has
  nothing to do with whether the ledger is right.
• Removal cost later: a workflow, its pin, and whatever else grew onto it.
• Where it belongs: the check is cheap and exact at write time, when the
  tool already knows the SHA it measured at, and expensive and rotting at read
  time. That is the tool's job, which is what
  Scan record schema has no field naming the tree its after-campaign counts describe, so testsAfter is unfalsifiable in place adversarial-mutation-test#23 is for. Vendoring a per-repo
  validator into one consumer would leave the next repo's ledger just as
  unfalsifiable.

The correction landed here is a data fix; the enforcement that stops it
recurring is upstream, and this record is now shaped so that upstream check can
run against it.

QA

• Discriminating tests: ledger-falsifiability harness over
  audit/mutation-test-scans.json (reads the record's own commit /
  testsAfterCommit, counts forge test --list --json at each named tree,
  compares to testsBefore / testsAfter) — fails on base with
  FAIL testsAfter: ledger claims 102 but names no commit -- unfalsifiable,
  exit 1, verified by running it against git show origin/main:audit/mutation-test-scans.json
  before the fix and against the fixed file after (exit 0). Not committed to the
  repo; see the decision and its ledger above.
• Mutations applied: "testsAfter": 84 → 102 → harness FAILs
  (at ae21818… = 84, ledger says 102, exit 1); "testsAfterCommit" →
  a003022 → harness FAILs (at a003022 = 42, ledger says 84, exit 1);
  "testsAfterCommit" line deleted → harness FAILs (claims 84 but names no commit -- unfalsifiable, exit 1). All three killed; each run printed real
  per-tree counts first, proving it reached forge rather than short-circuiting.
• Oracle: forge test --list --json at each SHA in a fresh worktree — the
  runner's own enumeration, independent of the ledger and of any grep over
  sources. Cross-checked against function test… declaration counts at every
  commit from c72eb89 to HEAD (agreement at all three loadbearing trees:
  9 / 84 / 134), and against the fact that every test function at ae21818 and
  HEAD is external.
• Category check: issue audit/mutation-test-scans.json testsAfter reads 102, a count that occurs at no commit in the range, and no field names the tree it describes #86 asks for two things in this repo — correct
  testsAfter, and add a field naming the tree it holds at. Both covered.
  The issue's second half (the shared schema in
  Scan record schema has no field naming the tree its after-campaign counts describe, so testsAfter is unfalsifiable in place adversarial-mutation-test#23) is explicitly not this repo's to
  land; the field name and position here are chosen to match it.

The record is data, not Solidity, so the reproducing "test" is a harness that
reads the ledger's own commit fields, counts what forge test --list reports at
each named tree, and compares. It was run against the pre-fix ledger (red)
before the fix, then against the fixed ledger (green), then against three
mutants.

RED — harness against origin/main's ledger, before the fix:

ledger: commit=c72eb89b029c9c7733d4c6deff849cc5da93dd5b testsBefore=9
ledger: testsAfterCommit=<absent> testsAfter=102
OK   testsBefore: forge test --list at c72eb89b029c9c7733d4c6deff849cc5da93dd5b = 9, ledger says 9
FAIL testsAfter: ledger claims 102 but names no commit -- unfalsifiable
BEFORE exit=1

GREEN — same harness against the ledger as this PR lands it:

ledger: commit=c72eb89b029c9c7733d4c6deff849cc5da93dd5b testsBefore=9
ledger: testsAfterCommit=ae21818a51d1b8c7df14d8c32eb39521e9c1a146 testsAfter=84
OK   testsBefore: forge test --list at c72eb89b029c9c7733d4c6deff849cc5da93dd5b = 9, ledger says 9
OK   testsAfter: forge test --list at ae21818a51d1b8c7df14d8c32eb39521e9c1a146 = 84, ledger says 84
AFTER exit=0

Mutation matrix — each mutant is a single-value break of exactly what this
PR changed. Every run printed a real forge test --list count for both trees
first, which is the proof the harness actually ran rather than short-circuiting;
a harness that never reached a tree could not have printed = 9 and = 84.

#	mutation	expected	observed	verdict
M1	testsAfter 84 → 102 (the original wrong value)	fail	FAIL testsAfter: forge test --list at ae21818… = 84, ledger says 102, exit 1	KILLED
M2	testsAfterCommit → a003022 (merge of #37, a real commit in the same range)	fail	FAIL testsAfter: forge test --list at a003022 = 42, ledger says 84, exit 1	KILLED
M3	testsAfterCommit line deleted	fail	FAIL testsAfter: ledger claims 84 but names no commit -- unfalsifiable, exit 1	KILLED

M2 is the mutant that proves the new field is load-bearing rather than
decorative: point it at a different real tree in the same range and the count
stops matching.

Full suite and formatter, in the flake shell, run on the merge commit
e768b2c (origin/main merged in, not rebased) so semantic conflicts outside
the diff markers are covered:

$ nix develop -c forge fmt --check
FMT exit=0

$ nix develop -c forge test
Ran 16 test suites in 2.30s (17.13s CPU time): 133 tests passed, 0 failed, 0 skipped (133 total tests)

(133, not the 134 measured for the table above: #97 landed on main between
the two runs and deleted a longhand duplicate of assertRejected. The 134 in
the table is 935c725, the tree the counts were taken at.)

No Solidity changed in this PR, so the suite is a regression guard rather than
the fix's own evidence; the fix's evidence is the RED/GREEN pair and the
mutation matrix above.

[coderabbitai]

Warning

Review limit reached

@thedavidmeister, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 42 minutes

Limit details: You’ve used all 1 included review currently available under your plan.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6f5fc8ed-6ded-4f3b-b610-aa5ffbbc0725

📥 Commits

Reviewing files that changed from the base of the PR and between c8dff5c and e768b2c.

📒 Files selected for processing (1)

• audit/mutation-test-scans.json

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}