fail-pattern: abort when it matches the baseline, and ship known-good patterns per harness

README.md

@@ -82,17 +82,16 @@ nix run github:rainlanguage/adversarial-mutation-test#mutation-probe -- mutants.
 
 `mutation-probe --help` is the complete manual. The short form: the mutants file
 names the suite command as argv (artifact regeneration included — the probe runs
-exactly that per verdict), a proof-of-run regex reading the suite's own
-pass/fail tally, and the mutants as exact-string `(file, target, replacement)`
-triples that must match exactly once.
+exactly that per verdict), the harness whose output it should read, and the
+mutants as exact-string `(file, target, replacement)` triples that must match
+exactly once.
 
 ```toml
 [suite]
 root = "."
 command = ["nix", "develop", "-c", "cargo", "test"]
-proof = '(\d+) passed; (\d+) failed'
-fail-pattern = 'test (\S+) \.\.\. FAILED'   # optional: names the killer
-timeout-secs = 1800                          # optional
+harness = "cargo"   # or "forge" — supplies proof + fail-pattern
+timeout-secs = 1800 # optional
 
 [[mutants]]
 name = "M01 guard inverted"
@@ -101,15 +100,28 @@ target = "if !ok {"
 replacement = "if ok {"
 ```
 
+`harness` exists because the two regexes it stands in for — a proof-of-run over
+the suite's own pass/fail tally, and a `fail-pattern` naming the test that
+killed a mutant — describe a harness's OUTPUT FORMAT, not a repo. Written per
+campaign they get written wrong the same two ways: too wide, matching passing
+result lines so that mutants are credited to tests which cannot kill them; or
+too narrow, matching nothing and emptying the killer column without ever saying
+so. Both are still available (`proof`, `fail-pattern`) and override the
+harness's, and the probe now aborts on the first class — a fail-pattern that
+captures anything from the GREEN baseline is matching passing lines by
+construction — and prints `killer NOT NAMED` per kill for the second.
+
 Verdicts: `KILLED` (failing tally, or non-zero exit with proof of a run) /
 `SURVIVED` (ran green: a real gap) / `NO-RUN` (no proof the suite ran — crash,
 compile error, timeout — never scored as survived) / `HARNESS-ERROR` (target not
-matched exactly once). A red, silent, or zero-test baseline aborts before any
-probe; writes are atomic and every restore is verified byte-exact; a hung
-suite's whole process group is killed at `timeout-secs`. Exit 0 only when every
-probed mutant is killed; 1 on any non-kill; 2 when the pass cannot be trusted.
-`--only <substring>` re-runs a subset while strengthening a killer;
-`--json <path>` writes the machine-readable report.
+matched exactly once). None of them is read from `fail-pattern`: a broken
+pattern degrades attribution and never the verdict. A red, silent, or zero-test
+baseline aborts before any probe; writes are atomic and every restore is
+verified byte-exact; a hung suite's whole process group is killed at
+`timeout-secs`. Exit 0 only when every probed mutant is killed; 1 on any
+non-kill; 2 when the pass cannot be trusted. `--only <substring>` re-runs a
+subset while strengthening a killer; `--json <path>` writes the machine-readable
+report.
 
 ## Scan record template
 

mutation-probe-rs/fixtures/cargo-1.95.0-green.txt

@@ -0,0 +1,8 @@
+    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.00s
+     Running unittests src/lib.rs (target/debug/deps/cargo_toy-9851a04534560ea2)
+
+running 1 test
+test tests::unit_passes ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s
+

mutation-probe-rs/fixtures/cargo-1.95.0-red.txt

@@ -0,0 +1,81 @@
+   Compiling cargo-toy v0.0.0 (/home/gildlab/code/amt-issue-13-forge-evidence/cargo-toy)
+    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.18s
+     Running unittests src/lib.rs (target/debug/deps/cargo_toy-9851a04534560ea2)
+
+running 2 tests
+test tests::unit_passes ... ok
+test tests::unit_fails_multiline ... FAILED
+
+failures:
+
+---- tests::unit_fails_multiline stdout ----
+
+thread 'tests::unit_fails_multiline' (284639) panicked at src/lib.rs:27:9:
+assertion `left == right` failed: generated source differs:
+line one
+line two
+  left: "a\nb\nc"
+ right: "a\nb\nd"
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+
+
+failures:
+    tests::unit_fails_multiline
+
+test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
+
+error: test failed, to rerun pass `--lib`
+     Running tests/integration.rs (target/debug/deps/integration-4de883e90258a294)
+
+running 2 tests
+test integration_fails ... FAILED
+test integration_passes ... ok
+
+failures:
+
+---- integration_fails stdout ----
+
+thread 'integration_fails' (284642) panicked at tests/integration.rs:8:5:
+assertion `left == right` failed
+  left: 1
+ right: 2
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+
+
+failures:
+    integration_fails
+
+test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
+
+error: test failed, to rerun pass `--test integration`
+   Doc-tests cargo_toy
+
+running 2 tests
+test src/lib.rs - two (line 2) ... ok
+test src/lib.rs - three (line 11) ... FAILED
+
+failures:
+
+---- src/lib.rs - three (line 11) stdout ----
+Test executable failed (exit status: 101).
+
+stderr:
+
+thread 'main' (284702) panicked at src/lib.rs:5:1:
+assertion `left == right` failed
+  left: 3
+ right: 4
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+
+
+
+failures:
+    src/lib.rs - three (line 11)
+
+test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.12s
+
+error: doctest failed, to rerun pass `--doc`
+error: 3 targets failed:
+    `--lib`
+    `--test integration`
+    `--doc`

mutation-probe-rs/fixtures/forge-1.0.0-nightly-red-traces.txt

@@ -0,0 +1,146 @@
+Warning: This is a nightly build of Foundry. It is recommended to use the latest stable version. Visit https://book.getfoundry.sh/announcements for more information. 
+To mute this warning set `FOUNDRY_DISABLE_NIGHTLY_WARNING` in your environment. 
+
+Compiling 22 files with Solc 0.8.25
+Solc 0.8.25 finished in 653.36ms
+Compiler run successful!
+proptest: Saving this and future failures in cache/fuzz/failures
+proptest: If this test was run on a CI system, you may wish to add the following line to your copy of the file. (You may need to create it.)
+cc 8b5dede949389349e6efb74f7812838e6847e5ecccff479e0cb60fa9bbb572bc
+
+Ran 4 tests for test/Evidence.t.sol:EvidenceTest
+[PASS] testAppliedIsIdempotent() (gas: 266)
+[FAIL: assertion failed: // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(1));
+}
+ != // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(2));
+}
+] testGeneratedSourceMatchesSnapshot() (gas: 5529)
+Traces:
+  [5529] EvidenceTest::testGeneratedSourceMatchesSnapshot()
+    ├─ [0] VM::assertEq("// SPDX-License-Identifier: MIT\npragma solidity =0.8.25;\n\nlibrary LibGenerated {\n    bytes32 constant HEAD = bytes32(uint256(1));\n}\n", "// SPDX-License-Identifier: MIT\npragma solidity =0.8.25;\n\nlibrary LibGenerated {\n    bytes32 constant HEAD = bytes32(uint256(2));\n}\n") [staticcall]
+    │   └─ ← [Revert] assertion failed: // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(1));
+}
+ != // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(2));
+}
+
+    └─ ← [Revert] assertion failed: // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(1));
+}
+ != // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(2));
+}
+
+
+[PASS] testHeadGenesisIsNotZero() (gas: 212)
+[FAIL: assertion failed: 1 != 2] testSingleLineFailure() (gas: 3333)
+Traces:
+  [3333] EvidenceTest::testSingleLineFailure()
+    ├─ [0] VM::assertEq(1, 2) [staticcall]
+    │   └─ ← [Revert] assertion failed: 1 != 2
+    └─ ← [Revert] assertion failed: 1 != 2
+
+Suite result: FAILED. 2 passed; 2 failed; 0 skipped; finished in 1.77ms (1.39ms CPU time)
+
+Ran 5 tests for test/Shapes.t.sol:ShapesTest
+[FAIL: Custom(1, 2)] testCustomErrorRevertIsUncaught() (gas: 5877)
+Traces:
+  [5877] ShapesTest::testCustomErrorRevertIsUncaught()
+    ├─ [663] Reverter::boom() [staticcall]
+    │   └─ ← [Revert] Custom(1, 2)
+    └─ ← [Revert] Custom(1, 2)
+
+[FAIL: assertion failed: 958 >= 10; counterexample: calldata=0x8054777000000000000000000000000000000000000000000000000000000000000007a6 args=[1958]] testFuzz_BoundedIsAlwaysSmall(uint256) (runs: 1, μ: 747, ~: 747)
+Traces:
+  [3833] ShapesTest::testFuzz_BoundedIsAlwaysSmall(1958)
+    ├─ [0] VM::assertLt(958, 10) [staticcall]
+    │   └─ ← [Revert] assertion failed: 958 >= 10
+    └─ ← [Revert] assertion failed: 958 >= 10
+
+[FAIL: revert: plain string reason] testPlainRevert() (gas: 495)
+Traces:
+  [495] ShapesTest::testPlainRevert()
+    └─ ← [Revert] revert: plain string reason
+
+[PASS] testThisOnePasses() (gas: 255)
+[FAIL: assertion failed: 3 != 4] test_snake_case_name_fails() (gas: 3355)
+Traces:
+  [3355] ShapesTest::test_snake_case_name_fails()
+    ├─ [0] VM::assertEq(3, 4) [staticcall]
+    │   └─ ← [Revert] assertion failed: 3 != 4
+    └─ ← [Revert] assertion failed: 3 != 4
+
+Suite result: FAILED. 1 passed; 4 failed; 0 skipped; finished in 1.84ms (2.76ms CPU time)
+
+Ran 2 tests for test/Invariant.t.sol:InvariantTest
+[FAIL: assertion failed: 1 != 0]
+	[Sequence] (original: 1, shrunk: 1)
+		sender=0x0000000000000000000000000000000000000020 addr=[test/Invariant.t.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=inc() args=[]
+ invariant_NeverIncrements() (runs: 0, calls: 0, reverts: 0)
+Traces:
+  [22454] Counter::inc()
+    └─ ← [Stop]
+
+  [10932] InvariantTest::invariant_NeverIncrements()
+    ├─ [2402] Counter::n() [staticcall]
+    │   └─ ← [Return] 1
+    ├─ [0] VM::assertEq(1, 0) [staticcall]
+    │   └─ ← [Revert] assertion failed: 1 != 0
+    └─ ← [Revert] assertion failed: 1 != 0
+
+[PASS] testCounterStartsAtZero() (gas: 7803)
+Suite result: FAILED. 1 passed; 1 failed; 0 skipped; finished in 4.54ms (2.54ms CPU time)
+
+Ran 3 test suites in 13.14ms (8.15ms CPU time): 4 tests passed, 7 failed, 0 skipped (11 total tests)
+
+Failing tests:
+Encountered 2 failing tests in test/Evidence.t.sol:EvidenceTest
+[FAIL: assertion failed: // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(1));
+}
+ != // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(2));
+}
+] testGeneratedSourceMatchesSnapshot() (gas: 5529)
+[FAIL: assertion failed: 1 != 2] testSingleLineFailure() (gas: 3333)
+
+Encountered 1 failing test in test/Invariant.t.sol:InvariantTest
+[FAIL: assertion failed: 1 != 0]
+	[Sequence] (original: 1, shrunk: 1)
+		sender=0x0000000000000000000000000000000000000020 addr=[test/Invariant.t.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=inc() args=[]
+ invariant_NeverIncrements() (runs: 0, calls: 0, reverts: 0)
+
+Encountered 4 failing tests in test/Shapes.t.sol:ShapesTest
+[FAIL: Custom(1, 2)] testCustomErrorRevertIsUncaught() (gas: 5877)
+[FAIL: assertion failed: 958 >= 10; counterexample: calldata=0x8054777000000000000000000000000000000000000000000000000000000000000007a6 args=[1958]] testFuzz_BoundedIsAlwaysSmall(uint256) (runs: 1, μ: 747, ~: 747)
+[FAIL: revert: plain string reason] testPlainRevert() (gas: 495)
+[FAIL: assertion failed: 3 != 4] test_snake_case_name_fails() (gas: 3355)
+
+Encountered a total of 7 failing tests, 4 tests succeeded

mutation-probe-rs/fixtures/forge-1.0.0-nightly-red.txt

@@ -0,0 +1,74 @@
+Compiling 22 files with Solc 0.8.25
+Solc 0.8.25 finished in 638.67ms
+Compiler run successful!
+proptest: Saving this and future failures in cache/fuzz/failures
+proptest: If this test was run on a CI system, you may wish to add the following line to your copy of the file. (You may need to create it.)
+cc 8b5dede949389349e6efb74f7812838e6847e5ecccff479e0cb60fa9bbb572bc
+
+Ran 4 tests for test/Evidence.t.sol:EvidenceTest
+[PASS] testAppliedIsIdempotent() (gas: 266)
+[FAIL: assertion failed: // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(1));
+}
+ != // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(2));
+}
+] testGeneratedSourceMatchesSnapshot() (gas: 5529)
+[PASS] testHeadGenesisIsNotZero() (gas: 212)
+[FAIL: assertion failed: 1 != 2] testSingleLineFailure() (gas: 3333)
+Suite result: FAILED. 2 passed; 2 failed; 0 skipped; finished in 404.84µs (269.75µs CPU time)
+
+Ran 5 tests for test/Shapes.t.sol:ShapesTest
+[FAIL: Custom(1, 2)] testCustomErrorRevertIsUncaught() (gas: 5877)
+[FAIL: assertion failed: 958 >= 10; counterexample: calldata=0x8054777000000000000000000000000000000000000000000000000000000000000007a6 args=[1958]] testFuzz_BoundedIsAlwaysSmall(uint256) (runs: 1, μ: 747, ~: 747)
+[FAIL: revert: plain string reason] testPlainRevert() (gas: 495)
+[PASS] testThisOnePasses() (gas: 255)
+[FAIL: assertion failed: 3 != 4] test_snake_case_name_fails() (gas: 3355)
+Suite result: FAILED. 1 passed; 4 failed; 0 skipped; finished in 817.47µs (966.39µs CPU time)
+
+Ran 2 tests for test/Invariant.t.sol:InvariantTest
+[FAIL: assertion failed: 1 != 0]
+	[Sequence] (original: 1, shrunk: 1)
+		sender=0x0000000000000000000000000000000000000020 addr=[test/Invariant.t.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=inc() args=[]
+ invariant_NeverIncrements() (runs: 0, calls: 0, reverts: 0)
+[PASS] testCounterStartsAtZero() (gas: 7803)
+Suite result: FAILED. 1 passed; 1 failed; 0 skipped; finished in 1.47ms (786.05µs CPU time)
+
+Ran 3 test suites in 4.24ms (2.69ms CPU time): 4 tests passed, 7 failed, 0 skipped (11 total tests)
+
+Failing tests:
+Encountered 2 failing tests in test/Evidence.t.sol:EvidenceTest
+[FAIL: assertion failed: // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(1));
+}
+ != // SPDX-License-Identifier: MIT
+pragma solidity =0.8.25;
+
+library LibGenerated {
+    bytes32 constant HEAD = bytes32(uint256(2));
+}
+] testGeneratedSourceMatchesSnapshot() (gas: 5529)
+[FAIL: assertion failed: 1 != 2] testSingleLineFailure() (gas: 3333)
+
+Encountered 1 failing test in test/Invariant.t.sol:InvariantTest
+[FAIL: assertion failed: 1 != 0]
+	[Sequence] (original: 1, shrunk: 1)
+		sender=0x0000000000000000000000000000000000000020 addr=[test/Invariant.t.sol:Counter]0x5615dEB798BB3E4dFa0139dFa1b3D433Cc23b72f calldata=inc() args=[]
+ invariant_NeverIncrements() (runs: 0, calls: 0, reverts: 0)
+
+Encountered 4 failing tests in test/Shapes.t.sol:ShapesTest
+[FAIL: Custom(1, 2)] testCustomErrorRevertIsUncaught() (gas: 5877)
+[FAIL: assertion failed: 958 >= 10; counterexample: calldata=0x8054777000000000000000000000000000000000000000000000000000000000000007a6 args=[1958]] testFuzz_BoundedIsAlwaysSmall(uint256) (runs: 1, μ: 747, ~: 747)
+[FAIL: revert: plain string reason] testPlainRevert() (gas: 495)
+[FAIL: assertion failed: 3 != 4] test_snake_case_name_fails() (gas: 3355)
+
+Encountered a total of 7 failing tests, 4 tests succeeded

mutation-probe-rs/fixtures/forge-1.7.1-green.txt

@@ -0,0 +1,8 @@
+No files changed, compilation skipped
+
+Ran 2 tests for test/Evidence.t.sol:EvidenceTest
+[PASS] testAppliedIsIdempotent() (gas: 266)
+[PASS] testHeadGenesisIsNotZero() (gas: 212)
+Suite result: ok. 2 passed; 0 failed; 0 skipped; finished in 325.01µs (152.22µs CPU time)
+
+Ran 1 test suite in 4.38ms (325.01µs CPU time): 2 tests passed, 0 failed, 0 skipped (2 total tests)