mutation-probe: the skill's probe harness as a tested, flake-shipped Rust bin - #10
Conversation
The skill's mutation passes have hand-rolled this machinery per campaign, and the hand-rolls kept re-introducing the same integrity bugs: zero-match mutants scored as survived, unproven suite runs scored at all, red baselines probed, imperfect restores poisoning later probes. The bin owns apply/score/restore once, tested; the mutants file keeps the adversarial half with the agent.
The self-pass (10 mutants over verdict/integrity logic) left one survivor: dropping the failed>0 clause from the kill condition — every failing fixture also carried a non-zero exit, so the tally clause was never discriminated. The new test pins the scenario the clause exists for: a wrapper that swallows the suite's exit code must not launder a failure its own tally reports.
|
Warning Review limit reached
Next review available in: 28 minutes Limit details: You’ve used all 1 included review currently available under your plan. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (9)
WalkthroughThis change adds a Rust ChangesMutation Probe Harness
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant MutationProbe
participant TestSuite
participant SourceFiles
participant JSONReport
CLI->>MutationProbe: load configuration and select mutants
MutationProbe->>TestSuite: run green baseline
TestSuite-->>MutationProbe: return proof output and suite result
MutationProbe->>SourceFiles: apply one exact-match mutant
MutationProbe->>TestSuite: run mutated suite
TestSuite-->>MutationProbe: return proof output and suite result
MutationProbe->>SourceFiles: restore and verify original bytes
MutationProbe->>JSONReport: write campaign results
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 12
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/rust.yml:
- Around line 7-8: Update .github/workflows/rust.yml lines 7-8 and 10-11 to pin
both reusable workflow references to immutable commit SHAs and replace secrets:
inherit with an explicit CACHIX_AUTH_TOKEN mapping. In each called workflow’s
workflow_call declaration, declare the CACHIX_AUTH_TOKEN secret so only that
secret is delegated.
In @.gitignore:
- Line 1: Update the .gitignore target-directory pattern so it ignores target
directories at any repository depth, including mutation-probe-rs/target, rather
than only the root-level /target path.
In `@flake.nix`:
- Around line 39-44: Update the mutation-probe package definition to derive its
version from the package manifest via lib.importTOML instead of hardcoding it.
Add metadata for description, license, and the explicit main program so nix run
exposes the intended package information.
In `@mutation-probe-rs/Cargo.toml`:
- Around line 1-6: Add a rust-version field to the [package] manifest for the
mutation-probe crate, setting the MSRV to Rust 1.82 to match the
Option::is_none_or usage in main.rs. Keep the existing package metadata
unchanged.
In `@mutation-probe-rs/src/main.rs`:
- Around line 261-270: Update the stdout and stderr reader threads around
out_thread and err_thread to read each stream with a bounded buffer of a few
MiB, retaining the most recent bytes rather than using unbounded read_to_end.
Preserve enough tail output for the proof regex and fail-pattern result lines,
and ensure excessive mutant logging cannot grow memory without limit.
- Around line 347-361: Update the argument matching loop so the positional
config-path arm only accepts arguments that do not begin with a dash; reject any
leading-dash argument through the existing unexpected-argument failure path,
regardless of whether config_path has already been set. Preserve normal
positional config-path handling for non-flag arguments and the existing
--json/--only behavior.
- Around line 446-456: Update the mutant write error handling around
std::fs::write so a failed write first restores the original contents to path
before calling fail. Report both the write failure and whether restoration
succeeded, while preserving the existing abort behavior and clean-tree
guarantee.
- Around line 248-301: Update run_suite to accept a compiled Regex reference and
use it directly when calling classify_suite, removing the per-invocation
Regex::new and expect. Update both run_suite call sites in main to pass the
existing &proof value.
- Around line 212-215: Update the baseline diagnostic around the failed-count
and exit-status check to split the two cases: report the failed count when
failures are present, and report the non-zero exit status when the suite exits
unsuccessfully without failures. Extend the relevant unit tests to assert the
diagnostic text for both scenarios.
- Around line 272-294: Update the timeout handling around the child process wait
loop to terminate the entire process group, not just the direct child: use the
process-group identifier established for the spawned suite and an explicit
group-kill mechanism such as Unix killpg, then wait for termination before
joining out_thread and err_thread. Preserve the existing timeout outcome while
ensuring descendant processes cannot keep the output pipes open and block the
joins.
In `@mutation-probe-rs/tests/toy.rs`:
- Around line 9-17: Extend the integration-test script in CHECK_SH with a case
that emits a nonzero failed count while exiting successfully, then assert the
mutation probe preserves and reports that zero-exit failure tally. Keep the
existing BOOM, GUARD, and CAP checks unchanged.
In `@skills/adversarial-mutation-test/SKILL.md`:
- Around line 139-143: Update the mutation-probe CLI documentation around the
--only option to state that it performs substring matching and may select
multiple mutants; instruct users to provide an unambiguous substring, including
for the M07 example.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 32b5fe8f-2765-44df-bdb7-bcf2c054ee11
⛔ Files ignored due to path filters (2)
Cargo.lockis excluded by!**/*.lockflake.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
.claude-plugin/marketplace.json.claude-plugin/plugin.json.github/workflows/rust.yml.gitignoreCargo.tomlREADME.mdflake.nixmutation-probe-rs/Cargo.tomlmutation-probe-rs/src/main.rsmutation-probe-rs/tests/toy.rsskills/adversarial-mutation-test/SKILL.md
| uses: rainlanguage/rainix/.github/workflows/rainix-rs-test.yaml@main | ||
| secrets: inherit |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sha="$(git ls-remote https://github.com/rainlanguage/rainix.git refs/heads/main | awk 'NR == 1 { print $1 }')"
test -n "$sha"
printf 'Current rainix main SHA: %s\n' "$sha"
for workflow in rainix-rs-test.yaml rainix-rs-static.yaml; do
echo "== $workflow workflow_call secrets =="
gh api "repos/rainlanguage/rainix/contents/.github/workflows/${workflow}?ref=${sha}" \
--jq '.content' | base64 --decode \
| sed -n '/workflow_call:/,/^[^ ]/p'
doneRepository: rainlanguage/adversarial-mutation-test
Length of output: 384
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '== caller workflow =='
cat -n .github/workflows/rust.yml
sha="$(git ls-remote https://github.com/rainlanguage/rainix.git refs/heads/main | awk 'NR == 1 { print $1 }')"
printf '\nCurrent rainlanguage/rainix main SHA: %s\n' "$sha"
for workflow in rainix-rs-test.yaml rainix-rs-static.yaml; do
printf '\n== %s at %s ==\n' "$workflow" "$sha"
gh api "repos/rainlanguage/rainix/contents/.github/workflows/${workflow}?ref=${sha}" \
--jq '.content' | base64 --decode | cat -n
doneRepository: rainlanguage/adversarial-mutation-test
Length of output: 4992
🌐 Web query:
GitHub Actions reusable workflows secrets inherit workflow_call secret declaration explicit secrets behavior
💡 Result:
In GitHub Actions, reusable workflows do not automatically receive access to secrets from the caller workflow [1][2]. To provide secrets to a reusable workflow, you must explicitly pass them or use the inherit keyword [3][4]. Explicit Secret Passing To pass secrets explicitly, you must define the expected secrets in the reusable workflow's on.workflow_call.secrets configuration [5][1]. In the caller workflow, you then map the secrets using the secrets keyword within the job call [5][6]. Example of explicit declaration in the reusable workflow: on: workflow_call: secrets: MY_SECRET: required: true Example of explicit mapping in the caller workflow: jobs: call-workflow: uses: octo-org/example-repo/.github/workflows/reusable.yml@main secrets: MY_SECRET: ${{ secrets.MY_SECRET }} Using 'secrets: inherit' The secrets: inherit keyword allows a calling workflow to pass all of its available secrets to the directly called reusable workflow [3][4]. This approach is simpler but has broader security implications, as it grants the reusable workflow access to all secrets available to the caller [7][6]. Key Behavioral Notes: 1. No Automatic Recursive Forwarding: Inheritance and explicit passing are only one-hop processes [8]. If workflow A calls workflow B, and B calls workflow C, workflow C will only receive secrets from workflow B if B explicitly passes or inherits them [8][7]. 2. Declaration Requirements: When using secrets: inherit, you are not required to explicitly define the secrets in the reusable workflow's on.workflow_call block to reference them [3][8]. However, when passing secrets explicitly, you must declare them [1]. 3. Environment Secrets: The on.workflow_call trigger does not support the environment keyword [3]. If a reusable workflow includes an environment at the job level, environment secrets will take precedence, and secrets passed from the caller may not be used in the same way [3][9]. 4. Security Best Practices: Because secrets: inherit exposes all secrets to the called workflow, it is generally recommended to use explicit mapping to follow the principle of least privilege, unless you are using a trusted, internally governed platform workflow [7][6].
Citations:
- 1: https://latchkey.dev/learn/github-actions/reusable-workflow-secrets-inherit-not-passed-in-ci
- 2: https://latchkey.dev/learn/github-actions/github-actions-reusable-workflow-secret-not-inherited
- 3: https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows
- 4: https://github.blog/changelog/2022-05-03-github-actions-simplify-using-secrets-with-reusable-workflows/
- 5: https://stackoverflow.com/questions/74263511/unable-to-access-repository-secrets-in-reusable-workflows
- 6: https://adaptive-enforcement-lab.com/secure/github-actions-security/workflows/reusable/secret-patterns/
- 7: https://qaskills.sh/blog/ci-workflow-reusable-inputs-secrets
- 8: https://github.com/github/docs/blob/main/content/actions/how-tos/reuse-automations/reuse-workflows.md
- 9: https://docs.github.com/en/enterprise-server@3.17/actions/how-tos/reuse-automations/reuse-workflows
Pin reusable workflows and restrict secret delegation.
Both reusable workflows use the mutable main branch. Pin each reference to an immutable commit SHA.
Both workflows use secrets.CACHIX_AUTH_TOKEN. Declare this secret under workflow_call in each called workflow, then map only that secret in the caller instead of using secrets: inherit.
🧰 Tools
🪛 zizmor (1.29.0)
[error] 7-7: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[warning] 7-7: secrets unconditionally inherited by called workflow (secrets-inherit): this reusable workflow
(secrets-inherit)
📍 Affects 1 file
.github/workflows/rust.yml#L7-L8(this comment).github/workflows/rust.yml#L10-L11
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rust.yml around lines 7 - 8, Update
.github/workflows/rust.yml lines 7-8 and 10-11 to pin both reusable workflow
references to immutable commit SHAs and replace secrets: inherit with an
explicit CACHIX_AUTH_TOKEN mapping. In each called workflow’s workflow_call
declaration, declare the CACHIX_AUTH_TOKEN secret so only that secret is
delegated.
Source: Linters/SAST tools
There was a problem hiding this comment.
Declining both halves — they are rainix's calls, and the second one would break CI.
SHA-pinning the reusables. uses: …@main is the org convention for consumers, and the pinning happens one level in: rainix-rs-test.yaml and rainix-rs-static.yaml each carry env: RAINIX_SHA: 53e96a7d0a97d7c7c75c3b2412521324776fdac6 and run the toolchain as nix develop github:rainlanguage/rainix/${{ env.RAINIX_SHA }}#rust-shell -c …. So what actually executes is already pinned to an immutable commit; @main is the channel through which rainix rolls that pin out to every consumer at once. rain.orderbook and rain.math.float call the same two workflows at @main. Pinning here alone would freeze this repo on one rainix commit and silently drift from the rest of the org — a change worth making org-wide in rainix, not unilaterally in this PR.
secrets: inherit → explicit mapping. This one is not just convention, it would fail the workflow. Both reusables declare:
on:
workflow_call:with no secrets: block at all, while referencing secrets.CACHIX_AUTH_TOKEN internally (in the nix-cachix-setup composite call). GitHub rejects a caller that passes a secret the callee has not declared under on.workflow_call.secrets, so secrets: { CACHIX_AUTH_TOKEN: … } here is an invalid-workflow error until rainix declares it. The suggested fix therefore starts in rainix, and shared CI is rainix's to own — this repo is a consumer. Happy to raise it there if you want the least-privilege mapping org-wide; it is a lockstep change across every consumer, not a one-file edit.
The nix-build job below, which this PR does own, is pinned by SHA (actions/checkout@34e1148…, nix-installer-action@a7ad9c4…).
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== Caller workflow =='
cat -n .github/workflows/rust.yml
for workflow in rainix-rs-test.yaml rainix-rs-static.yaml; do
echo
echo "== rainlanguage/rainix/.github/workflows/${workflow} at main =="
gh api "repos/rainlanguage/rainix/contents/.github/workflows/${workflow}?ref=main" \
--jq '.content' | base64 --decode | cat -n
doneLength of output: 5827
@thedavidmeister, verified.
The reusable workflows set RAINIX_SHA to 53e96a7d0a97d7c7c75c3b2412521324776fdac6 for the Nix Rust toolchain. The consumer references @main intentionally so rainix can update shared CI behavior across consumers.
The reusable workflows do not declare workflow_call.secrets. An explicit CACHIX_AUTH_TOKEN mapping from this caller would therefore be invalid until rainix changes its reusable-workflow interface.
This repository cannot make the proposed changes safely in isolation. I withdraw this finding.
🐇
✏️ Learnings added
Learnt from: thedavidmeister
Repo: rainlanguage/adversarial-mutation-test PR: 10
File: .github/workflows/rust.yml:7-8
Timestamp: 2026-08-16T16:55:36.185Z
Learning: In `.github/workflows/rust.yml`, consumers intentionally invoke `rainlanguage/rainix` reusable workflows at `main` so rainix can roll out shared CI updates organization-wide. The reusable workflows pin their Nix Rust toolchain with `env.RAINIX_SHA` and are owned by the rainix repository.
Learnt from: thedavidmeister
Repo: rainlanguage/adversarial-mutation-test PR: 10
File: .github/workflows/rust.yml:7-8
Timestamp: 2026-08-16T16:55:36.185Z
Learning: The `rainlanguage/rainix` reusable workflows `rainix-rs-test.yaml` and `rainix-rs-static.yaml` reference `secrets.CACHIX_AUTH_TOKEN` internally but currently declare no `on.workflow_call.secrets` interface. Callers must use `secrets: inherit`; an explicit `CACHIX_AUTH_TOKEN` mapping requires a coordinated interface change in rainix and its consumers.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
✅ Review thread resolved.
You are interacting with an AI system.
rs-static's second step runs the rainix hook bundle over all files, and this repo had never been held to it: mdformat rewraps README + SKILL.md prose, yamlfmt normalizes the workflows, nixfmt + statix settle flake.nix (inherit over re-assignment), and the git-hooks.nix-generated .pre-commit-config.yaml is committed like its issue-pr-cron sibling. /result gitignored.
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.pre-commit-config.yaml:
- Line 1: Replace the tracked /nix/store symlink represented by
.pre-commit-config.yaml with a portable, committed pre-commit configuration, or
point it to a valid repository-relative target; ensure clean checkouts can
resolve and use the configuration without Nix-specific paths.
In `@skills/adversarial-mutation-test/SKILL.md`:
- Line 576: Update the scan-record example’s skillVersion value to 0.31.0,
matching the declaration at the top of the sample skill; alternatively,
explicitly mark it as a placeholder.
- Around line 181-182: Update the mutation workflow instructions around
“Restore” to require the probe’s built-in restore path for byte-exact
restoration and verification, rather than an unconditional VCS restore. Remove
or replace the VCS restoration guidance while retaining the requirement that
mutations are fully restored before the next probe and never committed.
- Around line 159-161: Update the testing guidance in SKILL.md to allow
strengthening existing tests when they claim to cover a surviving mutant,
including the cases referenced near lines 183-188 and 606-610. Clarify that
tests may be modified or augmented for this purpose, while preserving the
requirement to avoid unnecessary duplicate tests and to evaluate the complete
suite.
- Around line 568-584: Update the fenced JSON scan-record example to remove the
inline // comments while preserving the same fields and values, so the block is
valid strict JSON.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 9c0f1c7d-4f1b-4b0b-ab53-cf7b53f39165
📒 Files selected for processing (6)
.github/workflows/version-hygiene.yaml.gitignore.pre-commit-config.yamlREADME.mdflake.nixskills/adversarial-mutation-test/SKILL.md
💤 Files with no reviewable changes (1)
- .github/workflows/version-hygiene.yaml
…skill CodeRabbit round + user ruling folded in: - write_atomic (temp+rename) both directions: no failure mode leaves a file truncated; fs::write's truncate-then-fail violated the never-dirty contract. - Suite output capped per stream (tail kept): a mutant that makes the suite log in a loop costs O(cap) memory, not OOM. - Timeout kills the suite's PROCESS GROUP: killing only sh left descendants holding the pipes and the reader joins hung; e2e test proves both halves. - RED baseline names its actual defect (tally vs lying exit code), e2e twin for the zero-exit failing tally, --help manual, unknown flags rejected, regex compiled once, rust-version pinned, flake version from the manifest, nested target/ ignored, pre-commit config materialized (was a dangling store symlink). - SKILL.md rebuilt method-only per ruling: tooling something must SHRINK the skill — machinery prose is enforced by the bin now, the manual lives in --help and the README, and the skill text (a per-invocation context cost) drops below its pre-tool length. Frontmatter reordered so the formatter cannot fold the version key into the description scalar.
main landed 0.31.0 (behaviour-sized fan-out groups, PR #18) while this branch condensed SKILL.md to method-only prose, so both sides rewrote the same three regions. Resolution keeps the branch's condensed form and folds main's SEMANTIC delta into it rather than restoring the long text: the campaign's grouping step now sizes a group by its enumerated behaviours (a file is not indivisible, an over-sized group dies rather than degrading, an unpriceable unit slices smaller), the survey step prices units in behaviours, and the fan-out slicing bullet partitions on the behaviour axis with an estimate treated as a floor. The long-form dispatcher/scan-record sections main still carries are already condensed into this branch's own sections, so the HEAD side stands. Version goes to 0.32.0: main consumed 0.31.0, and this branch changes skills/, so the repo's own version-hygiene gate requires a bump past the PR base. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git-hooks.nix generates this file into the nix store and symlinks it in when you enter the rainix dev shell, so its contents are absolute /nix/store paths from whichever machine last generated it — a checkout on any other machine cannot resolve a single hook entry. It landed here as a tracked symlink, then as a tracked copy of the generated JSON; both are the same defect, and there is no portable version to commit because the file is a build artifact, not source. Every rainix consumer already gitignores it (rain.orderbook, rain.math.float, rainix itself), and CI regenerates it inside the shell: rainix-rs-static runs `nix develop github:rainlanguage/rainix/<sha>#rust-shell -c pre-commit run --all-files`, which brings its own config. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review asked for --only to be documented as a substring filter or changed to
exact matching. Keep the substring semantics, for three reasons.
It is fail-safe in the only direction that matters. A substring match is a
SUPERSET of the exact match, so over-selection can only add verdicts — never
hide one. Exit 0 still requires every probed mutant to be KILLED, so a broader
--only is strictly harder to pass, and the dangerous direction is already closed:
a value matching nothing is a hard error ("--only matched no mutant", exit 2),
never a silent no-op that scores an empty pass.
Exact matching would not even deliver what it promises. Nothing validates that
mutant names are unique, so `--only "M07 guard inverted"` selects every mutant
carrying that name, and singleton selection is not a property the config can
offer. Meanwhile names carry prose by design (the name is what the report and
the ledger read), so exact matching means typing "M-hang the suite sleeps
forever" — and losing the ability to re-probe a family after one fix.
It is also a live interface. Campaigns are already running this bin from this
branch, so switching to exact matching would turn their `--only M07` invocations
into exit-2 aborts mid-pass.
So: an end-to-end test pins the contested behaviour rather than leaving it to
prose — two mutants, "M07 …" and "M070 …", where `--only M07` probes BOTH and the
extra match's survival still fails the pass. It fails under exact matching
(selects nothing, aborts) and under first-match-wins (reports one mutant, exits
0, laundering the survivor). --help states the multi-match consequence and the
fail-safe reason instead of just naming the match type.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kill main's PR #16 made the pre-existing suite's probe pass a first-class part of the per-unit loop. This branch had already condensed the skill to method-only, so the resolution keeps HEAD's prose and folds in #16's semantics rather than restoring main's long form: - per-unit loop step 2 is now baseline green THEN a full probe of the pre-existing suite with none of your own tests written, each kill crediting a named pre-existing test, the survivors being step 4's worklist, and writing early forfeiting attribution that costs a second clone, a second full pass and a matrix diff to recover. - step 4 opens on that survivor set. - Principles gains the condensed rule beside "Confirm the mutation is live". Version goes to 0.33.0 in all four places: main landed #18 (0.31.0) and #16 (0.32.0) while this branch was open, and version-hygiene fails a skills/ change whose plugin.json version equals the PR base's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Reviewed cd144d4: ready — Closes #10's issue. Ships This binary has been doing real work all day before it landed. Four campaigns across two repos ran their matrices on it, and it caught four harness defects that a hand-rolled loop would have reported as results: a red baseline that would have scored 95 mutants against a suite failing on its own; a fail-pattern crediting Two threads are answered rather than patched, and both are the right call:
Three PRs are stacked on this branch and unblock on the merge: #22 (fail-pattern harnesses, already MERGEABLE), #21 (per-tree probe lock), #20 (narrow-first escalation). CodeRabbit's check reads |
1 participant
Every mutation pass has hand-rolled its apply/run/score/restore harness (~8KB of Python per campaign), and the hand-rolls kept re-introducing the exact integrity bugs the skill warns about: zero-match mutants scored as survived, unproven suite runs scored at all, red baselines probed, restores unverified. This extracts the machinery into one tested tool shipped by this repo's nix flake; the adversarial half — deriving which mutants prove the suite discriminates — stays with the agent, as a declarative mutants file.
The bin (
mutation-probe-rs/, workspace at root so the rainix rust reusables pick it up):The mutants file names the suite root, the suite command as argv (artifact regeneration included — the probe runs exactly one command per verdict), a proof-of-run regex reading the suite's own passed/failed tally (multiple matches sum, for multi-binary harnesses), an optional killer-naming fail-pattern, and mutants as
(name, file, target, replacement)— target must occur exactly once.Verdict taxonomy: KILLED (tally shows failures, or non-zero exit with proof present — the tally is trusted over a lying wrapper exit code, and vice versa), SURVIVED (ran green: a real gap), NO-RUN (no proof of running — crash / compile error / timeout can never score), HARNESS-ERROR (target not found exactly once — mutates nothing, never "survived"). A red, silent, or zero-test baseline aborts before any probe; every restore is verified byte-exact and the tree is re-checked pristine before each apply; exit 0 only when every probed mutant is KILLED, 1 on any non-kill, 2 when the pass itself cannot be trusted.
Skill text (version 0.33.0, plugin.json + marketplace.json + the README scan-record template bumped in lockstep): SKILL.md gains "The probe harness" section with the file format and moves the enforceable integrity rules from prose-to-reimplement into the tool; the two rules the bin cannot see (commit before probing, keep targets out of the oracle) stay explicitly agent-owned. README gains the consumer section. Extraction only — the method content is otherwise untouched.
CI: rainix-rs-test + rainix-rs-static reusables @main, plus a nix-build job guarding the
nix runpath consumers actually take (cargo green does not imply the package builds).nix build .#mutation-probegreen locally with in-derivationdoCheck; the store bin smoke-runs. Version hygiene (lockstep + bump-on-skills-change) is gated by the repo's existing CI.QA
Review round
process_group(0)+killpg), so a descendant holding the output pipes cannot outlive the deadline; restores are atomic writes; captured output is capped and keeps the tail; a red baseline names the actual failing test. Each is pinned by a new test above.--onlystays a substring filter: over-selection is fail-safe (a substring match is a superset of the exact match, so it can only add verdicts, never hide one, and exit 0 still demands every probed mutant be KILLED), the dangerous direction is already closed (a value matching nothing is a hard error, exit 2), exact matching cannot deliver singleton selection anyway since nothing validates mutant-name uniqueness, and it would turn in-flight--onlycalls in campaigns already running this bin into exit-2 aborts.rust.ymlpinning (both halves) belongs in rainix, which owns shared CI:@mainis the consumer convention and the pin lives one level in (both reusables carryRAINIX_SHAand runnix develop github:rainlanguage/rainix/${RAINIX_SHA}#rust-shell, asrain.orderbookandrain.math.floatalso call them), and thesecrets: inherithalf would actively break CI, since both reusables declareon: workflow_call:with nosecrets:block while referencingsecrets.CACHIX_AUTH_TOKENand GitHub rejects a caller passing an undeclared secret. That thread is deliberately left unresolved.cargo test, baseline asserted green at 23 before probing, tree verified byte-exact after):killpgdowngraded to a child-onlykill→ KILLED byhung_suite_times_out_as_no_run_and_restores;--onlyswitched to exact-name matching → KILLED byonly_is_a_substring_filter_so_a_prefix_selects_every_matchandonly_filter_narrows_the_pass;--onlytruncated to the first match → KILLED byonly_is_a_substring_filter_so_a_prefix_selects_every_matchandweak_suite_scores_all_four_verdicts_and_restores. 3/3 killed, 0 survived.maintwice while open (Size a fan-out group by its behaviours, not by "a module" (0.31.0) #18 at 0.31.0, then Per-unit loop: probe the pre-existing suite BEFORE writing any test (0.31.0) #16 at 0.32.0), so the version lands at 0.33.0 in all four places. Per-unit loop: probe the pre-existing suite BEFORE writing any test (0.31.0) #16's probe-before-writing rule is folded into this branch's condensed per-unit loop rather than restoring main's long form: step 2 is now baseline green then a full probe of the pre-existing suite with none of your own tests written (each kill crediting a named pre-existing test, the survivors being step 4's worklist, and writing early forfeiting that attribution), step 4 opens on that survivor set, and Principles gains the condensed rule beside "Confirm the mutation is live".Summary by CodeRabbit
New Features
Documentation
Tests
Chores
.gitignore.