Skip to content

mutation-probe: the skill's probe harness as a tested, flake-shipped Rust bin #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thedavidmeister merged 8 commits into from
Aug 16, 2026
Merged

mutation-probe: the skill's probe harness as a tested, flake-shipped Rust bin #10

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5412d46
mutation-probe: the probe harness as a tested flake-shipped bin
thedavidmeister Aug 12, 2026
b2e8725
probe self-mutation: kill the surviving exit-code-lies mutant
thedavidmeister Aug 12, 2026
7ded129
static: conform to the rainix pre-commit bundle
thedavidmeister Aug 12, 2026
d195fd8
review round: atomic writes, capped capture, group kill, method-only …
thedavidmeister Aug 12, 2026
2f20168
Merge main into the mutation-probe bin branch
thedavidmeister Aug 16, 2026
ef06455
Stop tracking the generated .pre-commit-config.yaml
thedavidmeister Aug 16, 2026
525aafd
Keep --only a substring filter, and pin that with a test
thedavidmeister Aug 16, 2026
cd144d4
Merge main; fold #16's probe-before-writing rule into the condensed s…
thedavidmeister Aug 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"name": "adversarial-mutation-test",
"source": "./",
"description": "Find BUGS and harden the test suite for a whole repository — adversarial (spec as oracle, code as suspect; surface candidates for triage) + mutation (break each line, prove a test catches it). Whole-repo, resumable, language-agnostic.",
"version": "0.30.0",
"version": "0.31.0",
"author": {
"name": "Rain Open Source Software Ltd"
},
Expand Down
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "adversarial-mutation-test",
"displayName": "Adversarial Mutation Testing",
"version": "0.30.0",
"version": "0.31.0",
"description": "Find BUGS and harden the test suite for a whole repository. Two co-equal halves: ADVERSARIAL — treat the spec as the oracle and the code as suspect, hunt for inputs where the code is wrong, and surface candidates for triage (never self-adjudicate); and MUTATION — break each line and prove a test catches it. Whole-repo, resumable, language-agnostic.",
"author": {
"name": "Rain Open Source Software Ltd",
Expand Down
25 changes: 25 additions & 0 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: rust
on: [push, pull_request]
permissions:
contents: read
jobs:
test:
uses: rainlanguage/rainix/.github/workflows/rainix-rs-test.yaml@main
secrets: inherit
Comment on lines +7 to +8

@coderabbitai coderabbitai Bot Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

sha="$(git ls-remote https://github.com/rainlanguage/rainix.git refs/heads/main | awk 'NR == 1 { print $1 }')"
test -n "$sha"
printf 'Current rainix main SHA: %s\n' "$sha"

for workflow in rainix-rs-test.yaml rainix-rs-static.yaml; do
  echo "== $workflow workflow_call secrets =="
  gh api "repos/rainlanguage/rainix/contents/.github/workflows/${workflow}?ref=${sha}" \
    --jq '.content' | base64 --decode \
    | sed -n '/workflow_call:/,/^[^ ]/p'
done

Repository: rainlanguage/adversarial-mutation-test

Length of output: 384

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== caller workflow =='
cat -n .github/workflows/rust.yml

sha="$(git ls-remote https://github.com/rainlanguage/rainix.git refs/heads/main | awk 'NR == 1 { print $1 }')"
printf '\nCurrent rainlanguage/rainix main SHA: %s\n' "$sha"

for workflow in rainix-rs-test.yaml rainix-rs-static.yaml; do
  printf '\n== %s at %s ==\n' "$workflow" "$sha"
  gh api "repos/rainlanguage/rainix/contents/.github/workflows/${workflow}?ref=${sha}" \
    --jq '.content' | base64 --decode | cat -n
done

Repository: rainlanguage/adversarial-mutation-test

Length of output: 4992

🌐 Web query:

GitHub Actions reusable workflows secrets inherit workflow_call secret declaration explicit secrets behavior

💡 Result:

In GitHub Actions, reusable workflows do not automatically receive access to secrets from the caller workflow [1][2]. To provide secrets to a reusable workflow, you must explicitly pass them or use the inherit keyword [3][4]. Explicit Secret Passing To pass secrets explicitly, you must define the expected secrets in the reusable workflow's on.workflow_call.secrets configuration [5][1]. In the caller workflow, you then map the secrets using the secrets keyword within the job call [5][6]. Example of explicit declaration in the reusable workflow: on: workflow_call: secrets: MY_SECRET: required: true Example of explicit mapping in the caller workflow: jobs: call-workflow: uses: octo-org/example-repo/.github/workflows/reusable.yml@main secrets: MY_SECRET: ${{ secrets.MY_SECRET }} Using 'secrets: inherit' The secrets: inherit keyword allows a calling workflow to pass all of its available secrets to the directly called reusable workflow [3][4]. This approach is simpler but has broader security implications, as it grants the reusable workflow access to all secrets available to the caller [7][6]. Key Behavioral Notes: 1. No Automatic Recursive Forwarding: Inheritance and explicit passing are only one-hop processes [8]. If workflow A calls workflow B, and B calls workflow C, workflow C will only receive secrets from workflow B if B explicitly passes or inherits them [8][7]. 2. Declaration Requirements: When using secrets: inherit, you are not required to explicitly define the secrets in the reusable workflow's on.workflow_call block to reference them [3][8]. However, when passing secrets explicitly, you must declare them [1]. 3. Environment Secrets: The on.workflow_call trigger does not support the environment keyword [3]. If a reusable workflow includes an environment at the job level, environment secrets will take precedence, and secrets passed from the caller may not be used in the same way [3][9]. 4. Security Best Practices: Because secrets: inherit exposes all secrets to the called workflow, it is generally recommended to use explicit mapping to follow the principle of least privilege, unless you are using a trusted, internally governed platform workflow [7][6].

Citations:

Pin reusable workflows and restrict secret delegation.

Both reusable workflows use the mutable main branch. Pin each reference to an immutable commit SHA.

Both workflows use secrets.CACHIX_AUTH_TOKEN. Declare this secret under workflow_call in each called workflow, then map only that secret in the caller instead of using secrets: inherit.

🧰 Tools
🪛 zizmor (1.29.0)

[error] 7-7: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

[warning] 7-7: secrets unconditionally inherited by called workflow (secrets-inherit): this reusable workflow

(secrets-inherit)

📍 Affects 1 file
  • .github/workflows/rust.yml#L7-L8 (this comment)
  • .github/workflows/rust.yml#L10-L11
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/rust.yml around lines 7 - 8, Update
.github/workflows/rust.yml lines 7-8 and 10-11 to pin both reusable workflow
references to immutable commit SHAs and replace secrets: inherit with an
explicit CACHIX_AUTH_TOKEN mapping. In each called workflow’s workflow_call
declaration, declare the CACHIX_AUTH_TOKEN secret so only that secret is
delegated.

Source: Linters/SAST tools

@thedavidmeister thedavidmeister Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining both halves — they are rainix's calls, and the second one would break CI.

SHA-pinning the reusables. uses: …@main is the org convention for consumers, and the pinning happens one level in: rainix-rs-test.yaml and rainix-rs-static.yaml each carry env: RAINIX_SHA: 53e96a7d0a97d7c7c75c3b2412521324776fdac6 and run the toolchain as nix develop github:rainlanguage/rainix/${{ env.RAINIX_SHA }}#rust-shell -c …. So what actually executes is already pinned to an immutable commit; @main is the channel through which rainix rolls that pin out to every consumer at once. rain.orderbook and rain.math.float call the same two workflows at @main. Pinning here alone would freeze this repo on one rainix commit and silently drift from the rest of the org — a change worth making org-wide in rainix, not unilaterally in this PR.

secrets: inherit → explicit mapping. This one is not just convention, it would fail the workflow. Both reusables declare:

on:
  workflow_call:

with no secrets: block at all, while referencing secrets.CACHIX_AUTH_TOKEN internally (in the nix-cachix-setup composite call). GitHub rejects a caller that passes a secret the callee has not declared under on.workflow_call.secrets, so secrets: { CACHIX_AUTH_TOKEN: … } here is an invalid-workflow error until rainix declares it. The suggested fix therefore starts in rainix, and shared CI is rainix's to own — this repo is a consumer. Happy to raise it there if you want the least-privilege mapping org-wide; it is a lockstep change across every consumer, not a one-file edit.

The nix-build job below, which this PR does own, is pinned by SHA (actions/checkout@34e1148…, nix-installer-action@a7ad9c4…).

@coderabbitai coderabbitai Bot Aug 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '== Caller workflow =='
cat -n .github/workflows/rust.yml

for workflow in rainix-rs-test.yaml rainix-rs-static.yaml; do
  echo
  echo "== rainlanguage/rainix/.github/workflows/${workflow} at main =="
  gh api "repos/rainlanguage/rainix/contents/.github/workflows/${workflow}?ref=main" \
    --jq '.content' | base64 --decode | cat -n
done

Length of output: 5827

@thedavidmeister, verified.

The reusable workflows set RAINIX_SHA to 53e96a7d0a97d7c7c75c3b2412521324776fdac6 for the Nix Rust toolchain. The consumer references @main intentionally so rainix can update shared CI behavior across consumers.

The reusable workflows do not declare workflow_call.secrets. An explicit CACHIX_AUTH_TOKEN mapping from this caller would therefore be invalid until rainix changes its reusable-workflow interface.

This repository cannot make the proposed changes safely in isolation. I withdraw this finding.

🐇

✏️ Learnings added 
Learnt from: thedavidmeister
Repo: rainlanguage/adversarial-mutation-test PR: 10
File: .github/workflows/rust.yml:7-8
Timestamp: 2026-08-16T16:55:36.185Z
Learning: In `.github/workflows/rust.yml`, consumers intentionally invoke `rainlanguage/rainix` reusable workflows at `main` so rainix can roll out shared CI updates organization-wide. The reusable workflows pin their Nix Rust toolchain with `env.RAINIX_SHA` and are owned by the rainix repository.

Learnt from: thedavidmeister
Repo: rainlanguage/adversarial-mutation-test PR: 10
File: .github/workflows/rust.yml:7-8
Timestamp: 2026-08-16T16:55:36.185Z
Learning: The `rainlanguage/rainix` reusable workflows `rainix-rs-test.yaml` and `rainix-rs-static.yaml` reference `secrets.CACHIX_AUTH_TOKEN` internally but currently declare no `on.workflow_call.secrets` interface. Callers must use `secrets: inherit`; an explicit `CACHIX_AUTH_TOKEN` mapping requires a coordinated interface change in rainix and its consumers.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

✅ Review thread resolved.

You are interacting with an AI system.

static:
uses: rainlanguage/rainix/.github/workflows/rainix-rs-static.yaml@main
secrets: inherit
# The rainix reusables above build via cargo. Consumers run the flake PACKAGE
# (`nix run …#mutation-probe`), which cargo CI does not exercise — a
# workspace/lockfile drift can leave cargo green while the package fails to
# build. This job guards the path consumers actually take.
nix-build:
name: nix-build
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
persist-credentials: false
- uses: DeterminateSystems/nix-installer-action@a7ad9c4f0c65208097f4d34f3cfa1913b80cce5c # main
- run: nix build .#mutation-probe --print-build-logs
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/target
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
228 changes: 228 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Workspace root so the rainix rust reusables (cargo test / rainix-rs-static at
# the repo root) pick up the crate under mutation-probe-rs/.
[workspace]
resolver = "2"
members = ["mutation-probe-rs"]

[profile.release]
opt-level = 2
20 changes: 20 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,26 @@ the eventualities".
See [`skills/adversarial-mutation-test/SKILL.md`](skills/adversarial-mutation-test/SKILL.md)
for the full method.

## The probe harness (`mutation-probe`)

The mutate → run → score → restore machinery is a tested Rust bin shipped by this
repo's nix flake — campaigns author mutants declaratively and never hand-roll the
harness (hand-rolls kept faking matrices: zero-match mutants scored as survived,
crashed suites scored at all, imperfect restores poisoning later probes).

```sh
nix run github:rainlanguage/adversarial-mutation-test#mutation-probe -- mutants.toml
```

The mutants file names the suite command (artifact regeneration included), a
proof-of-run regex that reads the suite's own pass/fail tally, and the mutants as
exact-string `(file, target, replacement)` triples that must match exactly once.
Verdicts: `KILLED` / `SURVIVED` / `NO-RUN` (no proof the suite ran — never scored
as survived) / `HARNESS-ERROR` (invalid mutant). A red, silent, or zero-test
baseline aborts the pass; every restore is verified byte-exact. Exit 0 only when
every probed mutant is killed. See the probe-harness section of
[`SKILL.md`](skills/adversarial-mutation-test/SKILL.md) for the file format.

## License

[DecentraLicense 1.0](LICENSE) (`LicenseRef-DCL-1.0`).
61 changes: 61 additions & 0 deletions flake.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading