ci: guard the testsuite workflow contract - #1044
Conversation
castrojo
left a comment
There was a problem hiding this comment.
Verified the guard actually enforces the contract it describes. I ran scripts/check-testsuite-workflow-ref.py from this branch against the current tree (passes) and mutation-tested three violations — all are caught with exit 1:
run-testsuite.ymlref changed@v1→@ee2a5b9(the exact #929 regression) → 'must contain exactly one direct testsuite workflow reference at @v1'test_ref: v1→test_ref: main→ 'must pass exactly one test_ref: v1'- A new workflow calling
projectbluefin/testsuite/.github/workflows/e2e.ymldirectly → rejected, told to use the local wrapper
Current state on main: run-testsuite.yml:24 is the only direct testsuite reference and passes test_ref: v1, so the check is green today. Wiring it as a step in the validate job of pr-validation.yml (after checkout) is the right place. The comment fix at pr-validation.yml:234 ('SHA' → 'workflow ref') matches the post-#1012 @v1 reality, and the docs/skills/ci/SKILL.md addition keeps the skill doc in sync per repo policy.
Minor (non-blocking): the regex requires the ref to be the last token before any # comment, which is the house style, so that's fine.
CI note: PR Validation run 31229938653 is action_required — awaiting maintainer approval to run; nothing has failed.
|
Thanks for mutation-testing it rather than just reading it — the three violations you ran are exactly the cases the guard exists for, and #929 being one of them is the one I most wanted covered. On the minor note: agreed, and leaving it as-is. To be precise about what the pattern does, since "last token before any uses: projectbluefin/testsuite/.github/workflows/e2e.yml@v1 # pinned -> captures "v1"
uses: projectbluefin/testsuite/.github/workflows/e2e.yml@v1# pinned -> captures "v1"
No changes pushed — your read matches the intent and the house style, so there is nothing here I want to loosen. On the CI note: 🐝 Hive Agent: |
Assisted-by: GPT-5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The validate job was failing: `validate-docs` caps a SKILL.md at 180 lines, ci/SKILL.md already sits at exactly 180 on testing, and this branch's four-line bullet pushed it to 184. Rework rather than trim to fit. The testsuite-contract bullet collapses to two lines, the neighbouring @v1 bullet tightens from six to four, and the detail those lose -- that run-testsuite.yml is the single permitted direct caller, the two invariants the script enforces, the projectbluefin#929 freeze it prevents, and why config:best-practices makes the Renovate exclusion necessary -- moves into references/workflow-map.md. That is the progressive-disclosure split the skill docs already use, and it is why the cap exists. ci/SKILL.md is back to 180. Verified locally: validate-docs.py reports "documentation ok: 13 skills, 41 Markdown files", and check-testsuite-workflow-ref.py still passes against the current tree. Also rebased onto testing; the branch was 16 commits behind. No conflicts, and pr-validation.yml did not change in that range.
8c8a57f to
fe9dd2b
Compare
|
Thanks again for the mutation testing — heads up that I've force-pushed twice since your approval, so it may need a re-look.
Fixed by rework, not by trimming to fit. The testsuite-contract bullet is now two lines and the neighbouring Verified locally: $ python3 .github/scripts/validate-docs.py
documentation ok: 13 skills, 41 Markdown files
$ python3 scripts/check-testsuite-workflow-ref.py
Testsuite workflow contract passed: canonical wrapper uses @v1 with test_ref: v1Also rebased onto 🐝 Hive Agent: |
What does this change?
Add a repository-level regression check for the Bluefin testsuite workflow contract. It requires the canonical wrapper to use the managed
@v1workflow andtest_ref: v1, and rejects direct testsuite E2E calls from other Bluefin workflows. PR validation runs the check.Why?
Issue #929's latest release run executed
projectbluefin/testsuite/.github/workflows/e2e.yml@ee2a5b9while checking outtest_ref: v1. The stale nested workflow caused the release-gate smoke failures; the owner-side fix is tracked in projectbluefin/actions#409. Bluefin already uses the managed wrapper after #1012, so this change prevents that contract from regressing in this repository.Validation
python3 scripts/check-testsuite-workflow-ref.pyactionlint .github/workflows/*.ymljust checkgit diff --checkpre-commitwas unavailable in the environment (command not found)Refs #929