feat(qa): TestRail QA automation — Playwright pipeline with grounded specs, spec cache, and ephemeral Docker envs

[Miraeld]

Description

TestRail QA automation pipeline for Imagify, built on Playwright + ephemeral Docker environments (Canary was retired mid-branch; no Canary code remains). No product code is changed — the diff is entirely QA infrastructure (.claude/, Tests/e2e/, bin/, .distignore, .gitignore).

Note

Parts of this pipeline were AI-generated (Claude Code) and human-reviewed. The full run flow was executed live before this PR was refreshed — see "What was tested".

Two user-facing goals:

1. /testrail-scenarios — generate TestRail test cases from merged PRs (staged as YAML for human review, published on explicit confirm, deduped via refs).
2. /testrail-run — fetch a TestRail run (run ID / milestone / active), translate each human-written case into a Playwright .spec.ts, execute sequentially with trace/video evidence, keep a live results ledger, and post results back to TestRail only after user confirmation.

Type of change

• New feature (non-breaking change which adds functionality).
• Bug fix (non-breaking change which fixes an issue).
• Enhancement (non-breaking change which improves an existing functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as before).
• Sub-task of #(issue number)
• Chore
• Release

Detailed scenario

What was tested

Automated, live end-to-end run (2026-07-03): bin/qa-env.sh up (zip build → Docker up → WP install on nginx + apache → plugin install → API key seed → DB snapshot) → TestRail run 1283 resolved via API → cases C26359 and C26362 translated per the doctrine → compile gate → executed → both PASS (~1.1s each, trace + video captured) → results ledger written → DB snapshot reset verified → post-reset replay PASS → qa-env.sh down. Posting to TestRail was intentionally skipped (both cases already PASSED in the run). The two resulting cached specs are committed under Tests/e2e/testrail-cases/ as living proof.

Execution surfaced and fixed 5 bugs review alone would not have caught, notably: .distignore was missing /.ai, /node_modules, /generatedpackages — bin/build-zip.sh packed 240 MB of local artifacts (including browser profiles with cookies) into imagify.zip; now 2.9 MB / 1,280 files, verified clean. Also: Playwright cleans outputDir at test start (reporter JSON must live outside it), trace/video nest in per-test subdirs (resolved by glob), DB dump/restore moved inside the DB container (newer mariadb client refuses non-SSL TCP), Node ≥ 18 requirement documented.

How to test

Requirements: Docker Desktop, Node ≥ 18, TESTRAIL_USERNAME / TESTRAIL_API_KEY in the environment, optionally IMAGIFY_TESTS_API_KEY.

/testrail-run --run-id 1283 --cases 26359,26362   # env auto-provisioned AND torn down by the agent
/testrail-scenarios #<PR>                          # stage cases → review → publish (user-gated)
/testrail-setup --check                            # grounding drift report, no writes
bash bin/qa-env.sh up && bash bin/qa-env.sh down   # environment lifecycle standalone

The run agent never posts to TestRail without an explicit confirmation, and never deletes cached specs without a user choice.

Affected Features & Quality Assurance Scope

No runtime plugin feature is affected — zero PHP changes vs develop (verified: the diff excludes everything outside .claude/, Tests/, bin/, .distignore, .gitignore). QA scope is the pipeline itself: environment provisioning, TestRail API interaction (read + gated write), Playwright execution, and the distribution-zip content (.distignore change — see Risks).

Technical description

Documentation

Reading order for review:

1. .claude/testrail/translation.md — translation doctrine: how prose TestRail cases become Playwright specs (oracle-first reading, step classification, POM → grounded-spec → live-observation noun resolution, plan-before-code, failure-interpretation hierarchy).
2. .claude/agents/testrail-run-agent.md — the run flow: provision → resolve run → coverage pre-check → per-case (ground → cache check → seed → translate → lint/compile gate → execute → outcome → results ledger → LIFO teardown) → disposition → gated posting → env teardown.
3. bin/qa-env.sh + Tests/e2e/qa-env/ — ephemeral environments: one nginx + one apache WordPress from bin/build-zip.sh's zip, generated .ai/settings.local.json, DB snapshots for per-case reset. Environments never outlive the run.
4. .claude/testrail/specs/ — grounded feature specs (drift-stamped via derived_sha, mapped by testrail_sections, per-case apache_cases).
5. Tests/e2e/testrail-cases/ — the spec compile-cache: one committed .spec.ts per case with a @steps-hash header; unchanged case + fresh grounding → reused with zero re-translation. End of run: user-gated promote-to-CI / keep / delete.

New dependencies

None in the plugin. Tooling-only: Docker (dev machine requirement for TestRail runs), mariadb/wordpress/nginx Docker images pulled at provision time. No composer/npm dependency changes.

Risks

• .distignore change alters distribution-zip content — mitigated: verified the built zip (2.9 MB, 1,280 files) contains no .ai/, node_modules/, .claude/, or Tests/ paths and installs+activates correctly on both qa-env sites; wordpress.org CI is unaffected (fresh checkout, no npm step). This exclusion list fix should also be backported to develop as a standalone PR (flagged).
• TestRail writes — gated behind explicit user confirmation in both agents; dedup via refs prevents duplicate cases; duplicate-section guard at publish.
• Secrets — API keys live in env vars or the gitignored generated config; agents are instructed never to print them; the cookie-leaking zip path is closed.
• Runaway automation — sequential execution only, pagination caps, bounded repair loops, mandatory env teardown at end of run.

Mandatory Checklist

Code validation

• I validated all the Acceptance Criteria. If possible, provide screenshots or videos.
• I triggered all changed lines of code at least once without new errors/warnings/notices.
• I implemented built-in tests to cover the new/changed code.

Code style

• I wrote a self-explanatory code about what it does.
• I protected entry points against unexpected inputs.
• I did not introduce unnecessary complexity.
• Output messages (errors, notices, logs) are explicit enough for users to understand the issue and are actionnable.

Unticked items justification

Built-in tests: no product PHP changed, so there is nothing for PHPUnit to cover. The executable pieces were validated by live execution instead: qa-env.sh full lifecycle (up / reset / down), the Playwright config via a real two-case run with evidence, and the two committed specs under Tests/e2e/testrail-cases/ are runnable tests (executed green twice, including after a DB reset). Agent instruction files (.claude/*.md) are not unit-testable by nature.

Additional Checks

• In the case of complex code, I wrote comments to explain it.
• When possible, I prepared ways to observe the implemented system (logs, data, etc.)
• I added error handling logic when using functions that could throw errors (HTTP/API request, filesystem, etc.)

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Coverage ∅ diff coverage

Metric	Results
Coverage variation	Report missing for 3fb41021
Diff coverage	✅ ∅ diff coverage (50.00%)

View coverage diff in Codacy

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (3fb4102)	Report Missing	Report Missing	Report Missing
Head commit (7d14a41)	19533	629	3.22%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#1160)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

¹ Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}