From d38a12caacad7700a8683dd9f73b96eb9d3f7470 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= Date: Fri, 26 Jun 2026 21:33:44 +0200 Subject: [PATCH 01/16] =?UTF-8?q?docs(qa):=20add=20Canary=20=C3=97=20TestR?= =?UTF-8?q?ail=20consolidated=20spec=20and=20Canary=20=C3=97=20WordPress?= =?UTF-8?q?=20spec?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design-only documents for the upcoming QA automation deep audit: - canary-testrail-spec.md: consolidated spec covering TestRail scenario generation, TestRail run execution via Canary, and Canary E2E integration in the pipeline - canary-wp-spec.md: Canary × WordPress foundational spec (WP fixtures, agent inheritance pattern, session lifecycle reference) No implementation — brainstorm/audit prep for week of 2026-06-30. Co-Authored-By: Claude Sonnet 4.6 --- .claude/canary-testrail-spec.md | 935 ++++++++++++++++++++++++++++++++ .claude/canary-wp-spec.md | 637 ++++++++++++++++++++++ 2 files changed, 1572 insertions(+) create mode 100644 .claude/canary-testrail-spec.md create mode 100644 .claude/canary-wp-spec.md diff --git a/.claude/canary-testrail-spec.md b/.claude/canary-testrail-spec.md new file mode 100644 index 00000000..3586066b --- /dev/null +++ b/.claude/canary-testrail-spec.md @@ -0,0 +1,935 @@ +# Spec: Canary × TestRail — Consolidated QA Automation for Imagify + +**Status:** Draft — brainstorm/design only. No implementation in this document. +**Author:** Gaël Robin +**Audience:** Future Claude sessions with zero prior context. This document is self-contained. +**Next step:** Deep audit session. + +--- + +## 0. Orientation — read this first + +This spec consolidates three QA-automation features for the **Imagify WordPress plugin** +(`wp-media/imagify-plugin`) into one design document: + +1. **Feature 1 — TestRail scenario generation:** an agent analyses one or more GitHub PRs, + drafts human-readable TestRail test scenarios, stages them for review, and (on "publish") + creates them in TestRail in the right section, deduplicating against PRs already covered. +2. **Feature 2 — TestRail run execution:** an agent fetches every test scenario in a TestRail + run (by milestone or run ID), drives a real browser via **Canary** to execute each one, + collects pass/fail/blocked outcomes with artifacts, and (on confirm) posts results back to + TestRail with evidence. +3. **Feature 3 — Canary E2E in the delivery pipeline:** a new `canary-e2e` agent, a drop-in + replacement for the existing `e2e-qa-tester`, that records rich Canary sessions (trace, + video, HAR, console) per QA flow during a normal issue-workflow run, selectable at + orchestrator startup via an `e2e_mode` flag. + +All three share one substrate: **Canary** (browser recording + QA platform) and a set of +**battle-tested WordPress fixture snippets** wrapped in a reusable **agent inheritance pattern** +(WP base → Imagify-specific). + +### The existing delivery pipeline (context) + +Imagify ships via an agentic Claude Code pipeline: + +``` +orchestrator → grooming-agent → backend-agent → lead-reviewer → qa-engineer → e2e-qa-tester → release-agent +``` + +- Driven by `.claude/skills/orchestrator/SKILL.md` and `.claude/skills/issue-workflow/SKILL.md`. +- Agents live in `.claude/agents/`. +- `qa-engineer` is the QA gate; it currently spawns `e2e-qa-tester` for UI/browser changes. +- `e2e-qa-tester` drives the browser via `mcp__playwright`, writes specs to `Tests/e2e/specs/`, + and publishes screenshots via temp-branch commit + SHA `raw.githubusercontent.com` URLs. + **It is the current stable default and is NOT modified by any feature here.** + +### What Canary is (engine + sandbox) + +Canary is a browser recording + QA platform built on two npm packages, used as-is via `npx` +(**we never fork or modify them**): + +| Package | Command | Role | +|---------|---------|------| +| `@usecanary/cli` | `npx @usecanary/cli` | Playwright daemon + QuickJS sandbox + session lifecycle | +| `@usecanary/ui` | `npx @usecanary/ui` | Local web viewer for recorded sessions | + +What we own and customise is the **skill pack** — the markdown agent/skill files that drive the CLI. + +**QuickJS sandbox constraints (the step-script runtime — critical, non-negotiable):** + +- No `require()` / `import` — ESM and CJS both fail. +- No Node.js APIs — no `fs`, `path`, `http`, `process.env`. +- No top-level `fetch()` — QuickJS has no fetch. Run fetch inside the browser via + `page.evaluate(async () => fetch(...))`. +- No helper files — every utility must be **inlined verbatim** in each step script. +- Globals available: `browser` (Canary's Playwright wrapper), `saveScreenshot`, `console`. +- One browser **persists across all steps in a session** — you log in once, in step 1. + +**Session lifecycle:** + +```bash +npx @usecanary/cli session start --name "Flow name" --capture trace,video,har,console +npx @usecanary/cli run # repeat per step; appends to session.json["steps"] +npx @usecanary/cli session end # writes results.json + report.html + artifacts +``` + +Session ID format: `--` — e.g. +`p0-a--mcp-abilities-discovery-mqubrrhk-3f7e94`. + +**Artifacts per session** at `~/.canary/sessions//`: + +``` +session.json ← session metadata + step scripts (written during run) +results.json ← step results + summary (written at session end) +report.html ← self-contained pass/fail report (written at session end) +trace.zip ← Playwright trace (npx playwright show-trace trace.zip) +video/ ← .webm recording +network.har ← all HTTP traffic +console.log ← browser console output +screenshots/ ← PNG per saveScreenshot() call +manifest.json ← artifact inventory +profile/ ← browser profile +``` + +**`results.json` schema (the contract every Canary-driven agent reads):** + +```json +{ + "summary": { + "stepsPassed": 10, "stepsFailed": 0, "stepsTotal": 10, + "commandCount": 34, "consoleErrors": 6, "networkFailures": 6 + }, + "steps": [ + { "name": "login-admin", "ok": true, "exitCode": 0, "durationMs": 3681, + "startedAt": "2026-06-26T02:43:22.479Z", "script": "..." } + ], + "artifactList": [ + { "kind": "trace", "path": "trace.zip", "bytes": 5528027 }, + { "kind": "video", "path": "video/xxx.webm", "bytes": 3955579 }, + { "kind": "har", "path": "network.har", "bytes": 29806509 }, + { "kind": "console", "path": "console.log", "bytes": 3259 }, + { "kind": "screenshot", "path": "screenshots/step.png", "label": "...", + "step": "step-name", "bytes": 405656 } + ] +} +``` + +**Canary failure-signal gotchas (discovered in live sessions — internalise these):** + +- `summary.consoleErrors > 0` is **NOT** a fail signal. QuickJS `console.log` may route through + the error channel. Use `summary.stepsFailed` instead. +- `summary.networkFailures > 0` is **NOT** a fail signal. WP admin always emits some 4xx/5xx + (favicon 404 etc.). Use `summary.stepsFailed`. +- Video requires `--capture video` at session start. +- The session **must** be explicitly ended (`session end`) or `results.json`/`report.html` are + never written. A crash mid-session leaves artifacts incomplete. +- WP REST nonces expire (~12h TTL). Re-fetch the nonce at the start of each REST step, not just + once at login. + +### The Canary replay advantage (the reason this is worth building) + +Canary step scripts are recorded into `session.json` and can be **replayed without Claude**: + +```bash +npx @usecanary/cli run ~/.canary/sessions//steps/.js +``` + +This means: **the first run is Claude-driven (inference cost); every subsequent replay is free** +(zero inference). This underpins Feature 2's path to free CI reruns (see §8). + +--- + +## 1. Architecture overview — how the three features relate + +``` + ┌─────────────────────────────────────────────┐ + │ Shared infrastructure │ + │ • Canary CLI (npx, unmodified) │ + │ • WP fixture snippets (login/nonce/REST/...) │ + │ • Agent inheritance: WP base → Imagify │ + │ ~/.claude/agents/canary-wp-session-agent │ + │ .claude/agents/canary-imagify-session-… │ + │ • TestRail REST client knowledge (or MCP) │ + └───────────────┬──────────────┬───────────────┘ + │ │ + ┌────────────────────────────────┘ └───────────────────────────┐ + │ │ +┌───────▼─────────────────┐ ┌──────────────────────────────┐ ┌──────────────────▼─────────────┐ +│ FEATURE 1 │ │ FEATURE 2 │ │ FEATURE 3 │ +│ TestRail scenario gen │ │ TestRail run execution │ │ Canary E2E in pipeline │ +│ │ │ │ │ │ +│ PR(s) → draft scenarios │ │ milestone/run → fetch cases │ │ orchestrator e2e_mode flag │ +│ → stage → review → │ │ → Canary executes each │ │ → qa-engineer routes to │ +│ publish to TestRail │ │ → outcomes + artifacts │ │ canary-e2e | e2e-qa-tester │ +│ │ │ → confirm → post to TestRail │ │ → records sessions per flow │ +│ writes the test CASES │──▶│ EXECUTES those cases via │ │ → PR comment + Tests/e2e/specs │ +│ that Feature 2 runs │ │ Canary (shares §6 fixtures) │ │ (shares §6 fixtures + agents) │ +└──────────────────────────┘ └──────────────┬───────────────┘ └─────────────────────────────────┘ + │ + recorded step scripts + │ + ▼ + §8 Replay → free CI reruns +``` + +**How they connect:** + +- **Feature 1 feeds Feature 2.** Feature 1 produces TestRail cases (human-readable steps); + Feature 2 reads those exact cases and executes them via Canary. They are two halves of the + "release QA" loop: generate coverage, then run it. +- **Feature 2 and Feature 3 share the Canary execution substrate.** Both spawn the + `canary-imagify-session-agent` to drive the browser with the §6 WP fixtures. The difference is + the *source of truth* for what to test: + - Feature 3 reads `qa-plan.md` (P0/P1/P2 flows derived from a diff) — pre-merge, per PR. + - Feature 2 reads TestRail cases (`custom_steps_separated`) — release-time, full suite. +- **Feature 3 is pipeline-embedded; Features 1 & 2 are standalone skills.** Feature 3 plugs into + the orchestrator/qa-engineer flow. Features 1 & 2 are invoked on demand (`/testrail-*`). +- **All three reuse the agent inheritance pattern (§6).** The WP base agent is the single source + of truth for login/nonce/REST/AJAX; the Imagify agent extends it with plugin specifics. + +--- + +## 2. Agent & skill inventory + +### New skills (standalone entry points) + +| Skill | Feature | Trigger | Inputs | Outputs | +|-------|---------|---------|--------|---------| +| `/testrail-scenarios` | 1 | user invokes with PR number(s)/URL(s) or `--since-tag` | PR refs, optional `--force-pr=`, optional section override | staging files under `.ai/testrail/pending/`, summary, then created case IDs on publish | +| `/testrail-run` | 2 | user invokes with milestone name/ID or `--run-id` | milestone/run identifier, optional `--cases=` filter | results table (case → outcome + Canary artifacts), then TestRail results posted on confirm | + +> Names are proposals. The earlier draft used `/testrail-release-setup` and +> `/testrail-release-process`; this spec renames to `/testrail-scenarios` and `/testrail-run` to +> decouple from the word "release" (the features are useful per-PR too). Final names are an open +> question (§9). + +### New agents + +| Agent | Scope | Feature | What it does | Inputs | Outputs | +|-------|-------|---------|--------------|--------|---------| +| `testrail-scenario-agent` | project | 1 | Analyses PR(s), drafts scenarios, writes staging files, and on confirm creates TestRail sections/cases | PR diff + description + linked issue spec; TestRail section map | staging YAML; created case IDs (JSON) | +| `testrail-run-agent` | project | 2 | Fetches run cases, orchestrates Canary execution per case, aggregates results, posts to TestRail on confirm | run/milestone ID; TestRail cases | results table; posted result IDs (JSON) | +| `canary-e2e` | project | 3 | Drop-in for `e2e-qa-tester`: records a Canary session per P0/P1 flow, posts results table to PR, writes Playwright specs | `qa-plan.md` path, `e2e_mode: "canary"`, PR number | same JSON contract as `e2e-qa-tester` | +| `canary-imagify-session-agent` | project | 2 & 3 | Drives one Canary session against Imagify using the WP base + Imagify specifics | session name, flow steps/assertions, env (URL/user/pass) | `~/.canary/sessions//` + parsed `results.json` summary (JSON) | +| `canary-wp-session-agent` | **user** | 2 & 3 | WP-generic Canary base: login, nonce, REST, AJAX, notices. Shared across all WP plugin repos | (read as base by the Imagify agent) | — (instruction base, not invoked directly) | +| `canary-wp-verify-agent` | **user** | 3 (optional) | WP-generic diff → QA-plan helper (fork of marketplace `verify-agent`) | git diff | QA-plan draft | + +### Modified agents/skills + +| File | Feature | Change | +|------|---------|--------| +| `.claude/agents/qa-engineer.md` | 3 | Becomes `e2e_mode`-aware: writes `.ai/qa-plan.md`; routes to `canary-e2e` or `e2e-qa-tester`; merges either's (identical-shape) results into the PR comment | +| `.claude/skills/orchestrator/SKILL.md` | 3 | Adds the `e2e_mode` prompt to startup calibration; passes `e2e_mode` in every downstream dispatch | + +### Explicitly untouched + +| File | Why | +|------|-----| +| `.claude/agents/e2e-qa-tester.md` | The stable fallback. Feature 3 must not modify it; a one-word startup choice rolls back to it. | +| `@usecanary/cli`, `@usecanary/ui` | Engine. Used via `npx`, never forked. | + +--- + +## 3. Feature 1 — TestRail scenario generation + +### 3.1 Goal + +Expand QA coverage beyond "what changed" by auto-drafting structured TestRail cases from PRs, +keeping a human in the loop (review → approve → publish), and never duplicating cases for a PR +already covered. + +### 3.2 Flow + +``` +/testrail-scenarios + +1. Resolve PR scope: + • explicit PR numbers/URLs, OR + • --since-tag → git log v{last_tag}..HEAD → merged PR list +2. Dedupe per PR (see 3.5): GET /get_cases/3?refs= + • cases found → skip, print "⏭ Skipped PR # — cases already exist" + • --force-pr= regenerates for that PR (does not delete old cases) +3. For each in-scope PR: testrail-scenario-agent reads PR description + diff summary + + linked issue spec → drafts scenarios (see 3.4) +4. Write one staging file per PR → .ai/testrail/pending/.yml +5. Print a human-readable summary of every drafted scenario +6. STOP. Wait for the user to review/edit the staging files and say "publish". +7. On "publish": + a. For each staging file: create new_section if specified (POST /add_section/3) + b. POST /add_case/ per scenario, refs= + c. Print created case IDs + d. Clean .ai/testrail/pending/ +``` + +The review gate is **mandatory and explicit**: the agent stages, summarises, then halts. It never +creates cases in the same turn it drafts them. The user edits YAML freely (titles, steps, section, +smoke flag) before publishing. + +### 3.3 Staging file format (`.ai/testrail/pending/.yml`) + +```yaml +pr: 1133 +pr_url: https://github.com/wp-media/imagify-plugin/pull/1133 +pr_title: "Add MCP + Abilities to Imagify" +section_id: 7685 # existing target section (API Requests) +new_section: "MCP Abilities" # subsection to CREATE under section_id (null if not needed) + +cases: + - title: "MCP - Discover abilities - happy path" + smoke_test: true + preconditions: | + - Imagify plugin active + - Valid API key configured + steps: + - action: "Navigate to Settings > Imagify. Ensure API key is saved." + expected: "Settings page loads. API key field shows a valid key." + - action: "Connect an MCP-compatible client to the Imagify MCP server." + expected: "Connection succeeds. No error." + - action: "Call the discover-abilities tool." + expected: "Returns a list of available abilities (optimize-media, bulk-optimize, …)." + + - title: "MCP - Discover abilities - invalid API key" + smoke_test: false + preconditions: | + - Imagify plugin active + - Invalid or missing API key + steps: + - action: "Connect MCP client. Call discover-abilities with no valid API key set." + expected: "Returns an error response indicating authentication failure." +``` + +Field mapping (staging YAML → TestRail `add_case` payload): + +| Staging YAML | TestRail field | +|--------------|----------------| +| `title` | `title` | +| `smoke_test` | `custom_smoketest` (bool) | +| `preconditions` | `custom_preconds` (HTML — agent wraps lines in `

`/`
`) | +| `steps[].action` | `custom_steps_separated[].content` | +| `steps[].expected` | `custom_steps_separated[].expected` | +| (constant) | `template_id: 2`, `type_id: 7`, `priority_id: 2` | +| `pr_url` | `refs` | + +### 3.4 What scenarios the agent drafts (per PR) + +The agent reads PR description, diff summary, and linked issue spec, then drafts cases covering: + +- **Happy path** — standard user, default settings, expected flow. +- **Happy-path variants** — relevant settings combinations (WebP on/off, plan tiers). +- **Missing prerequisites** — no API key, quota exceeded, wrong plan/license. +- **Network/API failures** — timeout, 5xx, malformed response. +- **Edge cases** — empty state, max limits, unexpected data. +- **Permission levels** — admin vs editor vs subscriber when relevant. +- **Plugin conflicts** — relevant 3rd-party plugins when the change touches integration points. +- **Regression guard** — at least one case confirming prior behaviour still holds (for fixes). + +Bias toward **more, more-automatable** cases (each becomes future Canary/Playwright coverage). No +artificial cap per PR. Each case is also evaluated for `smoke_test: true` (critical path that must +always pass). Pure tracking/analytics PRs with no user-visible behaviour: the agent may draft a +single minimal "event fires" case or skip — see §9. + +### 3.5 Deduplication + +**Strategy:** TestRail's `refs` field links a PR to its cases. + +- On create, `refs = `. +- On the next run, per in-scope PR: `GET /get_cases/3?refs=`. + - cases found → skip the PR. + - none found → draft and stage. +- `--force-pr=` regenerates and stages new cases for a PR even if cases exist (does **not** + delete the old ones — manual cleanup in TestRail). + +**Fallback if `refs` filtering is unsupported in this TestRail version** (open question §9): +query all cases in the relevant section and match by title prefix, OR maintain a local index at +`.ai/testrail/index.json` (persists between runs; NOT cleaned with `pending/`). + +### 3.6 Section mapping + +The agent picks the closest existing section; if none fits, it proposes a new subsection under +`Regression (33)` via the `new_section` field (created only on publish, never silently). + +| Feature area | Section | +|---|---| +| MCP / Abilities | Regression > API Requests (7685) → new: MCP Abilities | +| Settings | Regression > Settings (32) → appropriate subsection | +| General settings | Regression > Settings > general settings (7689) | +| Optimization | Regression > Settings > optimization (7690) | +| Next-gen (WebP/AVIF) | Regression > Settings > optimization > next-gen (7694) → Webp (7697), Avif (7698) | +| File optimization | Regression > Settings > optimization > file optimization (7695) | +| Bulk optimization | Regression > Bulk Optimization (3766) | +| Media library | Regression > Media library (4976) | +| 3rd-party (WooCommerce) | Regression > 3rd party compatibility > Woocommerce (7686) | +| 3rd-party (WP Rocket) | Regression > 3rd party compatibility > WP Rocket (7687) | +| Action Scheduler | Regression > Action scheduler (4975) | +| Promotions | Regression > Promotions (1082) | +| Smoke tests | flag `custom_smoketest: true` (NOT a separate section) + Smoke test section (4525) for pure smoke cases | +| New feature, no match | propose new subsection under Regression (33) | + +### 3.7 MCP vs API + +TestRail may be reachable via an MCP server (not confirmed). The agent must handle both: + +- **If a TestRail MCP is available:** prefer its tools for `get_cases` / `add_section` / + `add_case` — typed, less error-prone. +- **Otherwise (default assumption):** use the TestRail REST API (see §5) with Basic auth from + `TESTRAIL_USERNAME` / `TESTRAIL_API_KEY`. + +The agent probes for the MCP first; if absent, falls back to REST. Both paths produce identical +staging files and identical `refs`-based dedup behaviour. + +--- + +## 4. Feature 2 — TestRail run execution + +### 4.1 Goal + +Run an entire TestRail test run automatically via Canary browser automation, gather pass/fail/ +blocked outcomes with rich artifacts, let the user review, then post results back to TestRail with +evidence — turning the full regression suite into an automated pass instead of a manual one. + +### 4.2 Flow + +``` +/testrail-run > + +1. Resolve the run: + • --run-id given → use it directly + • milestone name/id → GET /get_milestones/3 → GET /get_runs/3?milestone_id= + → if multiple runs, prompt user to pick (or --run-id) [open question §9] +2. GET /get_tests/ → all cases in the run (title, custom_preconds, + custom_steps_separated, case_id) +3. For each case (sequential or bounded-parallel): + a. testrail-run-agent maps the case → a Canary session plan: + preconditions → setup steps; each custom_steps_separated entry → a Canary step + b. spawns canary-imagify-session-agent to record the session + (one Canary session per TestRail case — see 4.3) + c. reads results.json → derives outcome (see 4.4) +4. Aggregate → results table: case → pass/fail/blocked + session id + artifact paths +5. Print the table. STOP. Ask: "Update TestRail with these results? (y/n)" +6. On confirm, per case: + POST /add_result_for_case// with status + evidence comment (see 4.5) +7. Print posted result summary. +``` + +The TestRail write is gated behind explicit user confirmation. The agent never posts results in +the same turn it executes — the user sees outcomes first. + +### 4.3 How Canary sessions map to TestRail cases + +- **One Canary session per TestRail case.** Session name = `TR-: ` so the + artifact directory and report are traceable back to the case. +- **Preconditions → setup steps.** `custom_preconds` is parsed into setup steps (login is always + step 1 via the wp-login fixture; "valid API key configured" → a settings precheck step, etc.). +- **Each `custom_steps_separated` entry → one Canary step.** The step's `expected` becomes the + assertion the step's `console.log("PASS/FAIL …")` checks. +- The `canary-imagify-session-agent` translates the human-readable TestRail step into concrete + Canary script using the §6 fixtures. Where a step is too ambiguous to automate, the case is + marked **blocked** (not failed) and flagged for manual review. + +### 4.4 Outcome derivation (from `results.json`) + +| Condition | TestRail status | +|-----------|-----------------| +| `summary.stepsFailed == 0` and all assertion `console.log` lines are PASS | **Passed (1)** | +| any step `ok == false` / `exitCode != 0`, or an assertion logs FAIL | **Failed (5)** | +| env unreachable, session couldn't start, or a step was un-automatable/ambiguous | **Blocked (2)** | + +Reminder: do NOT use `consoleErrors` / `networkFailures` as fail signals (see §0). + +### 4.5 Evidence in the result comment + +The result comment for each case includes per-step outcomes plus pointers to Canary artifacts: + +```json +{ + "status_id": 5, + "comment": "Automated Canary run — 2026-06-26\nSession: TR-4521--mcp-discover-abilities-xxx\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\nStep 3: ❌ Failed — assertion 'returns ability list' did not hold\n\nTrace: npx playwright show-trace ~/.canary/sessions/TR-4521--…/trace.zip\nReport: ~/.canary/sessions/TR-4521--…/report.html\nScreenshot: ", + "elapsed": "45s" +} +``` + +Evidence to include: +- Per-step pass/fail summary derived from `results.json.steps`. +- `elapsed` from summed `durationMs`. +- Trace replay command (local path — reviewer runs `npx playwright show-trace`). +- Path to `report.html`. +- Optionally, published screenshot URLs (same temp-branch-commit → SHA-raw-URL mechanism as + `e2e-qa-tester`; see §6.5) when a hosted image is wanted in TestRail. + +### 4.6 Failure handling + +- Step fails → case marked Failed, remaining steps captured but the failure is the verdict. +- Env unreachable / session won't start → case Blocked with reason. +- Step un-interpretable → case Blocked + "manual review" flag (never silently Failed). + +--- + +## 5. TestRail API reference (shared by Features 1 & 2) + +**Base URL:** `https://wpmediaqa.testrail.io/index.php?/api/v2/` +**Auth:** Basic (`TESTRAIL_USERNAME:TESTRAIL_API_KEY`) +**Project ID:** 3 · **Suite ID:** 3 · **Step template ID:** 2 + +| Action | Endpoint | +|--------|----------| +| Find cases by PR URL (dedup) | `GET /get_cases/3?refs=` | +| List sections | `GET /get_sections/3&suite_id=3` | +| Create section | `POST /add_section/3` | +| Create case | `POST /add_case/` | +| List milestones | `GET /get_milestones/3` | +| List runs for a milestone | `GET /get_runs/3?milestone_id=` | +| Get tests in a run | `GET /get_tests/` | +| Post a result | `POST /add_result_for_case//` | + +**Case payload (Step template):** + +```json +{ + "title": "MCP - Discover abilities - happy path", + "template_id": 2, + "type_id": 7, + "priority_id": 2, + "refs": "https://github.com/wp-media/imagify-plugin/pull/1133", + "custom_smoketest": false, + "custom_preconds": "

- Imagify active
- Valid API key

", + "custom_steps_separated": [ + { "content": "Navigate to Settings > Imagify.", "expected": "Settings page loads." } + ] +} +``` + +**Result payload:** + +```json +{ + "status_id": 1, + "comment": "Automated run …", + "elapsed": "45s" +} +``` + +Status IDs: **1 = Passed · 5 = Failed · 2 = Blocked**. + +**Known section IDs:** + +``` +Regression (33) + ├── Settings (32) + │ ├── general settings (7689) + │ └── optimization (7690) + │ ├── next-gen (7694) → Webp (7697), Avif (7698) + │ └── file optimization (7695) + ├── Smoke test (4525) + ├── API Requests (7685) ← MCP / Abilities go here + ├── Media library (4976) + ├── Bulk Optimization (3766) + ├── 3rd party compatibility (4980) + │ ├── Woocommerce (7686) + │ └── WP Rocket (7687) + ├── Action scheduler (4975) + └── Promotions (1082) +``` + +--- + +## 6. Feature 3 — Canary E2E in the pipeline + +### 6.1 Problem with today's `e2e-qa-tester` + +- Improvises WP interactions (login, nonce, REST, AJAX) from scratch — wrong ~30% of the time. +- No rich evidence: no trace, no video, no HAR — just a spec file and a screenshot URL. Debugging + failures is blind. +- The QA plan lives only in the orchestrator's context — invisible to developer and reviewer. + +### 6.2 Vision + +A new `canary-e2e` agent — a **drop-in replacement** for `e2e-qa-tester` that uses Canary CLI +instead of `mcp__playwright`. Opt-in at orchestrator startup; `e2e-qa-tester` stays the untouched +default. Both share the **same JSON return contract**, so `qa-engineer` is agnostic to which ran. + +### 6.3 Mode selection at orchestrator startup + +During the existing calibration step the orchestrator asks one extra question: + +> **E2E mode** — `playwright` (default, stable) or `canary` (experimental, richer artifacts)? + +Answered once; stored as `e2e_mode`; passed in every downstream dispatch. If the issue has no +UI/browser changes, the flag is ignored. + +``` +orchestrator startup + → calibration (autonomy level) + e2e_mode ("playwright" | "canary") + → grooming → implementation → review … + → qa-engineer [receives e2e_mode] + diff has UI/browser changes? + yes + e2e_mode == "canary" → spawn canary-e2e + yes + e2e_mode == "playwright" → spawn e2e-qa-tester (current default) + no → skip E2E entirely + merge E2E results into the final GitHub PR comment +``` + +### 6.4 `canary-e2e` agent design + +- **Inputs:** `qa-plan.md` path, `e2e_mode: "canary"`, PR number, env (`E2E_URL`, `WP_USER`, + `WP_PASS`). +- **Behaviour:** for each **P0/P1** flow in `qa-plan.md`: + 1. Record a Canary session via `canary-imagify-session-agent` (session name = the flow label, + e.g. `P0-A: MCP abilities discovery`). + 2. Read `results.json` → build a markdown results row. + 3. Write a Playwright spec to `Tests/e2e/specs/` (translated from the flow — see §6.6). +- **P2 flows** are documented in the plan but Canary sessions are optional. +- **Publishes:** screenshots via temp-branch-commit → SHA raw URL (§6.5); trace replay command per + session. +- **Returns:** the **same JSON shape** as `e2e-qa-tester` so `qa-engineer` needs no branching. +- **Tools:** `Bash` (drives `npx @usecanary/cli`), `Read`, `Edit`, `Write`, `Glob`, `Grep`, + `WebFetch`. Notably **not** `mcp__playwright` — Canary is the browser driver. + +### 6.5 `qa-plan.md` format (shared by qa-engineer ↔ canary-e2e ↔ e2e-qa-tester) + +Produced by `qa-engineer`, written to `.ai/qa-plan.md`, consumed by whichever E2E agent runs. +The `P0-A`/`P0-B` labels become Canary session names. + +```markdown +## QA Plan — PR #1234 + +### P0 — Must pass before merge + +#### P0-A: +- **Entry:** +- **Steps:** + 1. + 2. +- **Assertions:** + - +- **Risk:** +- **Canary session name:** `P0-A: ` + +### P1 — Should pass +#### P1-A: +(same structure) + +### P2 — Nice to have / regression guard +#### P2-A: +- documented; Canary session optional +``` + +### 6.6 Artifact flow & PR comment + +``` +Developer machine (local only — NOT committed) + ~/.canary/sessions// → session.json, results.json, report.html, + trace.zip, video/, network.har, console.log, screenshots/ + +GitHub PR comment (posted by qa-engineer via canary-e2e's JSON): + ├─ qa-plan (P0/P1/P2 markdown table) + ├─ Canary results table (from results.json) + ├─ Screenshot images (raw.githubusercontent.com SHA URLs) + └─ "Trace: npx playwright show-trace " per session + +GitHub Actions CI (committed by canary-e2e): + Tests/e2e/specs/ ← Playwright specs derived from the flows (existing CI re-runs them) + +Optional future: upload ~/.canary/sessions/ as CI artifact "canary-qa-" +``` + +**Results table format (from `results.json`):** + +```markdown +### Canary QA Sessions + +| Flow | Steps | Result | Trace | +|------|-------|--------|-------| +| P0-A: MCP abilities discovery | 10/10 | ✅ PASS | `npx playwright show-trace ~/.canary/sessions/p0-a--…/trace.zip` | +| P0-B: Settings page save | 5/6 | ❌ FAIL — step "save-settings" exit 1 | `npx playwright show-trace ~/.canary/sessions/p0-b--…/trace.zip` | + +### Screenshots + +| Step | Screenshot | +|------|-----------| +| login-admin | ![login](https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/p0-a-login-admin.png) | +``` + +**Screenshot publishing (same mechanism as today's `e2e-qa-tester`):** + +```bash +cp ~/.canary/sessions//screenshots/*.png .e2e-screenshots/ +git add -f .e2e-screenshots/ && git commit -m "chore(qa): Canary QA screenshots" && git push +SHA=$(git rev-parse HEAD) # URL base: https://raw.githubusercontent.com/wp-media/imagify-plugin/$SHA/.e2e-screenshots/ +git rm --cached .e2e-screenshots/*.png && git commit -m "chore(qa): remove Canary QA screenshots" && git push +``` + +### 6.7 Playwright spec translation (not mechanical) + +`canary-e2e` writes fresh Playwright specs that pass the same assertions — it does **not** transpile +QuickJS to Node (different runtimes, different error semantics). + +| Canary QuickJS | Playwright TypeScript | +|---|---| +| `browser.getPage("main")` | `page` fixture | +| `page.evaluate(async () => fetch(...))` | identical | +| `console.log("PASS: …")` | `expect(...).toBe(...)` | +| step exit 1 | thrown assertion | +| no `import` | `import { test, expect } from '@playwright/test'` | +| inlined helpers | POM methods in `Tests/e2e/pages/` | + +--- + +## 7. Shared infrastructure + +### 7.1 Agent inheritance pattern (WP base → Imagify) + +WP plugins share identical WP patterns but differ in plugin specifics. Split accordingly: + +``` +~/.claude/agents/ ← USER-level, shared across ALL WP plugin repos + canary-wp-session-agent.md # login, nonce, REST, AJAX, notices + canary-wp-verify-agent.md # WP QA-plan format + verify flow (optional) + +.claude/agents/ ← PROJECT-level, Imagify only + canary-imagify-session-agent.md # extends WP base + Imagify URLs/slugs/MCP/fixtures +``` + +**`canary-wp-session-agent.md`** knows only WordPress-generic patterns: `wp-login.php` login, +REST nonce via `admin-ajax.php?action=rest-nonce` (plain text, `.trim()`), authenticated REST +calls (`X-WP-Nonce`), `admin-ajax.php` POST with `_ajax_nonce`, admin-notice assertions. + +**`canary-imagify-session-agent.md`** extends it with: Imagify admin URLs (settings, bulk +optimization, custom folders), ability slugs, the MCP/abilities endpoint +(`/wp-json/wp-abilities/v1/abilities`), the `imagify-abilities` fixture, Imagify selectors/notice +patterns. + +The "extends" instruction at the top of the plugin agent: + +```markdown +Before doing anything, read `~/.claude/agents/canary-wp-session-agent.md` in full. +Its WP fixture snippets and rules are your base. The sections below OVERRIDE or EXTEND them. +``` + +**Why this split:** the WP base stays lean (never learns about Imagify/WP Rocket/BackWPup); each +plugin repo owns a thin agent with only its specifics; fixing a WP pattern (e.g. nonce endpoint +change) happens once and every plugin inherits it; adding a new plugin = one new file, zero changes +to the base. Worth formalising across `wp-media` projects once the base is stable. + +### 7.2 WP fixture snippets (canonical — never improvised, always copied verbatim) + +Carried in `canary-wp-session-agent.md`. Because QuickJS has no `require()`, these are inlined per +step. + +```js +// FIXTURE: wp-login — always step 1; browser persists for all subsequent steps +const page = await browser.getPage("main"); +await page.goto("{E2E_URL}/wp-login.php", { waitUntil: "domcontentloaded" }); +await page.locator("#user_login").fill("{WP_USER}"); +await page.locator("#user_pass").fill("{WP_PASS}"); +await page.locator("#wp-submit").click(); +await page.waitForURL("**/wp-admin/**", { timeout: 10000 }); +console.log("Login:", page.url().includes("wp-admin") ? "PASS" : "FAIL"); +``` + +```js +// FIXTURE: wp-nonce — GET with credentials:same-origin; result needs .trim() +const nonce = await page.evaluate(async () => { + const r = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); + return (await r.text()).trim(); +}); +``` + +```js +// FIXTURE: wp-rest-get — authenticated GET to WP REST API +const result = await page.evaluate(async ([url, nonce]) => { + const r = await fetch(url, { headers: { "X-WP-Nonce": nonce }, credentials: "same-origin" }); + return { status: r.status, body: await r.text() }; +}, [url, nonce]); +``` + +```js +// FIXTURE: wp-rest-post — authenticated POST to WP REST API +const result = await page.evaluate(async ([url, nonce, body]) => { + const r = await fetch(url, { + method: "POST", + headers: { "Content-Type": "application/json", "X-WP-Nonce": nonce }, + body: JSON.stringify(body), + credentials: "same-origin" + }); + return { status: r.status, body: await r.text() }; +}, [url, nonce, body]); +``` + +```js +// FIXTURE: wp-ajax — POST to admin-ajax.php with _ajax_nonce +const result = await page.evaluate(async ([action, nonce, data]) => { + const form = new FormData(); + form.append("action", action); + form.append("_ajax_nonce", nonce); + Object.entries(data).forEach(([k, v]) => form.append(k, String(v))); + const r = await fetch("/wp-admin/admin-ajax.php", { method: "POST", body: form, credentials: "same-origin" }); + return { status: r.status, body: await r.text() }; +}, [action, nonce, data]); +``` + +```js +// FIXTURE: wp-assert-notice — assert success/error admin notice +const notice = await page.waitForSelector(".notice-success, .notice-error", { timeout: 5000 }); +const cls = await notice.getAttribute("class"); +console.log(cls.includes("notice-success") ? "PASS: success notice" : "FAIL: error notice"); +``` + +**Imagify-specific (belongs in `canary-imagify-session-agent.md`, NOT the WP base):** + +```js +// FIXTURE: imagify-abilities — assert all expected slugs in WP Abilities response +const result = await page.evaluate(async () => { + const nonceResp = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); + const nonce = (await nonceResp.text()).trim(); + const resp = await fetch("/wp-json/wp-abilities/v1/abilities", { + headers: { "X-WP-Nonce": nonce }, credentials: "same-origin" + }); + return { status: resp.status, body: await resp.text() }; +}); +const data = JSON.parse(result.body); +const slugs = Array.isArray(data) ? data.map(a => a.name) : []; +console.log("PHP Fatal:", result.body.includes("Fatal error") ? "FAIL" : "PASS"); +const expected = [ + "imagify/get-settings", "imagify/update-settings", "imagify/get-account", + "imagify/get-stats", "imagify/get-media-status", "imagify/get-nextgen-coverage", + "imagify/optimize-media", "imagify/bulk-optimize", + "imagify/generate-missing-nextgen", "imagify/restore-media" +]; +for (const slug of expected) console.log(slugs.includes(slug) ? `PASS: ${slug}` : `FAIL: ${slug} NOT FOUND`); +``` + +### 7.3 Session-agent trust guard (critical pipeline detail) + +The Canary marketplace `session-agent`/`verify-agent` refuse to act on **relayed** confirmations — +they require a direct user message. In our pipeline these agents are invoked *transitively* +(`orchestrator → qa-engineer → canary-e2e → canary-imagify-session-agent`, or +`/testrail-run → testrail-run-agent → canary-imagify-session-agent`). The trust guard **must be +removed** in our forked `canary-wp-session-agent`/`canary-imagify-session-agent`, or every nested +spawn deadlocks waiting for a direct user confirmation that never comes. + +--- + +## 8. Canary replay advantage — free CI reruns (Feature 2 & 3) + +Recorded Canary step scripts in `session.json` replay without Claude: + +```bash +npx @usecanary/cli run ~/.canary/sessions//steps/.js # zero inference +``` + +Implications: + +- **Feature 2 — TestRail run execution:** the *first* execution of a TestRail case is Claude-driven + (it interprets human-readable steps into a Canary script). Once recorded, that script is the + durable automation for the case. A nightly/release CI job can **replay** all recorded scripts for + a run — no inference, deterministic, fast — and still post fresh results + artifacts to TestRail. + A recorded script could be associated back to its case (e.g. stored under + `Tests/e2e/canary/TR-/` or referenced from the case) so the next run replays instead of + re-inferring. Only *new* or *changed* cases need Claude. +- **Feature 3 — pipeline E2E:** the per-PR Canary sessions are recorded; the committed Playwright + specs in `Tests/e2e/specs/` already give CI a free rerun path. The Canary scripts themselves can + additionally be replayed locally to reproduce a failure deterministically without re-driving + through Claude. + +**Phasing:** +- Phase A (initial): Claude executes every run (inference cost per run). Simplest; ship this first. +- Phase B (scale): persist recorded scripts; CI replays them for unchanged cases → zero inference + at scale; Claude only handles new/changed cases. + +Open: exact storage location and the "case → recorded script" linkage (§9). + +--- + +## 9. Files to create / modify + +**User-level** (`~/.claude/agents/` — shared across all WP plugin repos): + +| File | Action | Feature | Notes | +|------|--------|---------|-------| +| `~/.claude/agents/canary-wp-session-agent.md` | Create | 2, 3 | Fork of marketplace `session-agent` + WP-generic fixtures only; **trust guard removed** for pipeline use | +| `~/.claude/agents/canary-wp-verify-agent.md` | Create (optional) | 3 | Fork of marketplace `verify-agent` + WP QA-plan format | + +**Project-level** (`.claude/` — Imagify only): + +| File | Action | Feature | Notes | +|------|--------|---------|-------| +| `.claude/agents/canary-imagify-session-agent.md` | Create | 2, 3 | Extends WP base; adds Imagify URLs, ability slugs, MCP endpoint, `imagify-abilities` fixture | +| `.claude/agents/canary-e2e.md` | Create | 3 | New E2E agent — same JSON contract as `e2e-qa-tester`, Canary CLI internally | +| `.claude/agents/testrail-scenario-agent.md` | Create | 1 | Analyse PR(s) → stage → publish cases; MCP-or-REST | +| `.claude/agents/testrail-run-agent.md` | Create | 2 | Fetch run → orchestrate Canary execution → post results | +| `.claude/skills/testrail-scenarios/SKILL.md` | Create | 1 | Standalone entry point for scenario generation | +| `.claude/skills/testrail-run/SKILL.md` | Create | 2 | Standalone entry point for run execution | +| `.claude/agents/qa-engineer.md` | Modify | 3 | Write `.ai/qa-plan.md`; receive `e2e_mode`; route to `canary-e2e`/`e2e-qa-tester` | +| `.claude/skills/orchestrator/SKILL.md` | Modify | 3 | Add `e2e_mode` startup prompt; pass it in every dispatch | +| `.claude/agents/e2e-qa-tester.md` | **Untouched** | 3 | Stable fallback — no changes | +| `.github/workflows/e2e.yml` | Modify (optional) | 2, 3 | Canary artifact upload / replay job if CI runs Canary | +| `Tests/e2e/` | Existing | 3 | Playwright specs still committed here — no structural change | +| `.ai/testrail/pending/` | Runtime dir | 1 | Staging YAML (cleaned on publish) | +| `.ai/testrail/index.json` | Runtime file | 1 | Dedup fallback index (only if `refs` filtering unavailable) | +| `.ai/qa-plan.md` | Runtime file | 3 | Shared QA plan | + +**Environment / secrets required:** + +| Var | Used by | Notes | +|-----|---------|-------| +| `TESTRAIL_USERNAME`, `TESTRAIL_API_KEY` | 1, 2 | TestRail Basic auth | +| `E2E_URL`, `WP_USER`, `WP_PASS` | 2, 3 | WP fixture login target | +| `CANARY_HOME` (maybe) | 2, 3 | Override `~/.canary` artifact root in CI (§ open question) | + +--- + +## 10. Open questions for the audit + +**Cross-cutting** +1. **TestRail MCP availability** — is a TestRail MCP server actually connected? If yes, prefer it + over REST in both Features 1 & 2. Confirm and document its tool names. +2. ~~**Skill naming**~~ — **resolved:** `/testrail-scenarios` + `/testrail-run` (decoupled from + "release", more general). Original names `/testrail-release-setup` / `/testrail-release-process` + are retired. + +**Feature 1** +3. **`refs` API filter** — confirm `GET /get_cases/3?refs=` works in this TestRail version. + If not, adopt the `.ai/testrail/index.json` fallback. +4. **Section auto-creation** — always propose in YAML and create only on publish (safer), vs allow + auto-create? Recommended: propose-then-create-on-confirm (already the design). +5. **Tracking/analytics-only PRs** — skip case generation, or generate a minimal "event fires" + case? (Relevant: the current branch wires wp-mixpanel and fires a "Media Optimized" event.) + +**Feature 2** +6. **Multiple runs per milestone** — if a milestone has several runs (smoke + full), which does + `/testrail-run` target? Prompt to choose, default to latest, or require `--run-id`? +7. **Parallelism** — execute cases sequentially or bounded-parallel? Canary uses one persistent + browser per session; parallel cases = parallel sessions = resource/port contention to validate. +8. **Recorded-script persistence & linkage** — where do recorded Canary scripts live for replay + (`Tests/e2e/canary/TR-/`?), and how is a case linked to its script so CI replays + instead of re-inferring? (Phase B of §8.) + +**Feature 3** +9. **Artifact location in CI** — `~/.canary/sessions/` is per-machine. If CI records (not just + replays Playwright specs), override via `CANARY_HOME`? Where do artifacts land? +10. **Spec-generation strategy** — `canary-e2e` writes fresh Playwright specs passing the same + assertions (recommended; QuickJS ≠ Node), not a mechanical transpile. Confirm. +11. **CI vs local Canary** — should CI boot the WP env and record fresh Canary sessions, or only + run committed Playwright specs (fast, no Canary dep)? Recommendation: local-only Canary + initially; CI runs Playwright specs only. +12. **WP fixture coverage** — the §7.2 snippets cover patterns seen in live Imagify sessions. Audit + `Tests/e2e/specs/` for uncovered patterns (`wp-block-editor`, `wp-media-library-upload`, + `wp-multisite`). +13. **Return-contract parity** — verify `canary-e2e`'s JSON exactly matches `e2e-qa-tester`'s so + `qa-engineer` needs zero branching. Run `/contract-check` once both exist. + +--- + +## 11. Implementation order (suggested) + +1. **Shared base first:** `canary-wp-session-agent` (user) + `canary-imagify-session-agent` + (project) with the §7.2 fixtures and the trust guard removed. Everything else depends on these. +2. **Feature 3 (pipeline E2E):** `canary-e2e` + `qa-engineer`/`orchestrator` changes. Highest + day-to-day value; exercises the shared agents on real PRs. +3. **Feature 1 (scenario generation):** `/testrail-scenarios` + `testrail-scenario-agent` + + dedup. Produces the cases Feature 2 consumes. +4. **Feature 2 Phase A (Claude-driven run):** `/testrail-run` + `testrail-run-agent` reusing the + Canary session agents; post results on confirm. +5. **Feature 2 Phase B (replay):** persist recorded scripts; CI replays unchanged cases for zero + inference at scale. +``` diff --git a/.claude/canary-wp-spec.md b/.claude/canary-wp-spec.md new file mode 100644 index 00000000..a4f5b741 --- /dev/null +++ b/.claude/canary-wp-spec.md @@ -0,0 +1,637 @@ +# Spec: Canary × WordPress — Integrated QA in the Issue Workflow + +**Status:** Draft +**Author:** Gaël Robin +**Next step:** Deep audit session (week of 2026-06-30) + +--- + +## 0. What is Canary + +> This section exists so any future Claude session has full Canary context without searching. Read it before reading the rest of the spec. + +### Engine + +Canary is a **browser recording + QA platform** built on two npm packages: + +| Package | Command | What it does | +|---------|---------|--------------| +| `@usecanary/cli` | `npx @usecanary/cli` | Playwright daemon + QuickJS sandbox + session lifecycle | +| `@usecanary/ui` | `npx @usecanary/ui` | Local web viewer for browsing session recordings | + +**We do NOT fork or modify these packages.** They stay as `npx @usecanary/cli` from npm. What we customise is the **skill pack** — the markdown agent/skill files that drive the CLI. + +### QuickJS sandbox (the step script runtime) + +Each "step" is a `.js` script executed inside QuickJS — a tiny JS engine, NOT Node.js. Critical constraints: + +- **No `require()` / `import`** — ESM and CJS modules both fail +- **No Node.js APIs** — no `fs`, `path`, `http`, `process.env`, etc. +- **No top-level `fetch()`** — QuickJS has no fetch; use `page.evaluate(async () => fetch(...))` to run fetch inside the browser context instead +- **No helper files** — every utility must be inlined verbatim in each step script +- **Globals available:** `browser` (Canary's Playwright wrapper), `saveScreenshot`, `console` + +The `browser` object: +```js +const page = await browser.getPage("main"); // "main" is the default page name +// page is a Playwright Page — all page.goto, page.locator, page.evaluate, etc. work +``` + +One browser **persists across all steps in a session** — you do not log in again between steps. The browser is identified by `session.json["browser"]`. + +### Session lifecycle + +``` +npx @usecanary/cli session start --name "Flow name" --capture trace,video,har,console + → creates ~/.canary/sessions//session.json + → starts Playwright daemon + browser + +npx @usecanary/cli run # repeat for each step + → executes step in QuickJS + → appends step to session.json["steps"] + → saves screenshot if saveScreenshot() called + +npx @usecanary/cli session end + → writes results.json + → renders report.html + → saves trace.zip, video.webm, network.har, console.log +``` + +Session ID format: `--` — e.g. `p0-a--mcp-abilities-discovery-mqubrrhk-3f7e94` + +### Artifact structure + +Every session directory at `~/.canary/sessions//` contains: + +``` +session.json ← session metadata + step scripts (written during run) +results.json ← step results + summary (written at session end) +report.html ← self-contained pass/fail report (written at session end) +trace.zip ← Playwright trace (open with npx playwright show-trace trace.zip) +video/ ← .webm recording +network.har ← all HTTP traffic +console.log ← browser console output +screenshots/ ← PNG per saveScreenshot() call (-.png) +manifest.json ← artifact inventory +profile/ ← browser profile +``` + +### `results.json` schema (real example) + +```json +{ + "summary": { + "stepsPassed": 10, + "stepsFailed": 0, + "stepsTotal": 10, + "commandCount": 34, + "consoleErrors": 6, + "networkFailures": 6 + }, + "steps": [ + { + "name": "login-admin", + "ok": true, + "exitCode": 0, + "durationMs": 3681, + "startedAt": "2026-06-26T02:43:22.479Z", + "script": "..." + } + ], + "artifactList": [ + { "kind": "trace", "path": "trace.zip", "bytes": 5528027 }, + { "kind": "video", "path": "video/xxx.webm", "bytes": 3955579 }, + { "kind": "har", "path": "network.har", "bytes": 29806509 }, + { "kind": "console", "path": "console.log", "bytes": 3259 }, + { "kind": "screenshot", "path": "screenshots/step-name-abc.png", + "label": "Screenshot: step-name", "step": "step-name", "bytes": 405656 } + ] +} +``` + +### Key CLI commands + +```bash +# Session lifecycle +npx @usecanary/cli session start --name "Flow name" --capture trace,video,har,console +npx @usecanary/cli run path/to/step.js +npx @usecanary/cli session end + +# Inspect +npx @usecanary/cli session list # list sessions +npx @usecanary/cli status # daemon status +npx @usecanary/cli status --session # specific session status + +# Open the report viewer +npx @usecanary/ui # opens http://localhost:3030 (or nearby port) +npx @usecanary/ui --dir ~/.canary/sessions/ # open specific session +``` + +### Skill pack (what we DO own) + +The skill pack is markdown files at `~/.claude/plugins/marketplaces/canary-marketplace/`. We copy relevant agents into `.claude/agents/` and customise them for WordPress. No fork of the CLI needed. + +Canary agent files of interest: +- `session-agent.md` → our fork: `canary-wp-session-agent.md` +- `verify-agent.md` → our fork: `canary-wp-verify-agent.md` + +--- + +## 1. Problem + +The current `e2e-qa-tester` agent drives the browser via `mcp__playwright` — an interactive, session-bound tool with no artifact output. It produces Playwright specs committed to `Tests/e2e/specs/`, but: + +- The agent improvises every WordPress interaction from scratch (login flow, nonce retrieval, REST calls, AJAX patterns) — getting them wrong about 30% of the time +- There is no rich local evidence: no trace, no video, no HAR — just a spec file and a screenshot URL +- The QA plan lives only in the orchestrator's context; it is not surfaced to the developer or reviewer +- Canary sessions run today are **disconnected** from the issue workflow — triggered manually, post-merge +- The `report.html` Canary produces is not linked from the GitHub PR comment — developers never see it + +--- + +## 2. Vision + +Introduce a **`canary-e2e` agent** — a drop-in replacement for `e2e-qa-tester` that uses Canary CLI instead of `mcp__playwright`. The user opts into it at orchestrator startup; the existing `e2e-qa-tester` stays untouched as the default. + +Every `orchestrator` / `issue-workflow` run with `e2e_mode: "canary"` produces: + +1. A **structured QA plan** (P0/P1/P2) derived from the diff — written to `.ai/qa-plan.md` so the developer and reviewer can read it +2. **Recorded Canary sessions** per P0/P1 flow — artifacts stored locally, a results summary **posted to the GitHub PR comment** (not committed to git) +3. **Playwright specs** written from the recorded flows — committed to `Tests/e2e/specs/` for CI to re-run + +The WordPress login, nonce, and REST patterns are baked into reusable snippet blocks so no agent ever has to guess them again. + +### Why not replace `e2e-qa-tester`? + +- Non-destructive: the old agent is the safe fallback; a single word at startup rolls back +- Gradual adoption: run Canary on a few issues, compare results, decide +- Feature parity check: can run the same issue through both agents and compare output + +--- + +## 3. Architecture + +### 3.1 What "cloning Canary" means + +Canary has two layers: + +| Layer | What it is | Ownership | +|-------|------------|-----------| +| **Engine** (`@usecanary/cli`, `@usecanary/ui`) | npm packages — Playwright daemon + QuickJS sandbox + report renderer | Stay as `npx @usecanary/cli` — no fork | +| **Skill pack** (agents + skills markdown) | The AI instructions that write and run step scripts | **Copy into `.claude/agents/`** — we own and customise | + +No separate plugin install or repo clone is needed. The CLI is already installed globally via `npx`; the skill pack files are already on disk at `~/.claude/plugins/marketplaces/canary-marketplace/`. + +### 3.2 Mode selection + +At orchestrator startup, during the existing calibration step, the orchestrator asks: + +> **E2E mode** — `playwright` (default, stable) or `canary` (experimental, richer artifacts)? + +The user answers once; the choice is stored as `e2e_mode` and passed in every downstream agent dispatch. If the issue has no UI/browser changes, the flag is ignored. + +``` +orchestrator startup + → calibration (autonomy level) + e2e_mode ("playwright" | "canary") + → ... grooming → implementation → review ... + → qa-engineer [receives e2e_mode in dispatch] + detects UI/browser changes in diff? + yes + e2e_mode == "canary" → spawns canary-e2e + yes + e2e_mode == "playwright" → spawns e2e-qa-tester (current default) + no → skips E2E entirely + merges E2E results into final GitHub PR comment +``` + +Both `canary-e2e` and `e2e-qa-tester` share the **same JSON return contract** — `qa-engineer` is agnostic to which one ran. + +### 3.3 Proposed pipeline (canary mode) + +``` +orchestrator [e2e_mode: "canary"] + └─► qa-engineer + reads diff → generates .ai/qa-plan.md (P0/P1/P2 flows) + posts initial "QA plan" GitHub PR comment + └─► canary-e2e (for UI/browser changes) + For each P0/P1 flow in qa-plan.md: + ├─ records a Canary session via canary-imagify-session-agent + ├─ reads results.json → builds markdown results table + └─ writes Playwright spec to Tests/e2e/specs/ + Publishes artifacts: + ├─ screenshots → temp branch commit → SHA raw.githubusercontent.com URLs + └─ "trace: npx playwright show-trace ~/.canary/sessions//trace.zip" + Returns JSON to qa-engineer (same shape as e2e-qa-tester) + qa-engineer posts final GitHub PR comment with: + ├─ qa-plan (P0/P1/P2 as markdown table) + ├─ Canary session results per flow + └─ READY TO MERGE or blockers +``` + +### 3.4 New agents + +Two WP agents replace the generic Canary marketplace agents. They live at different scopes: + +``` +~/.claude/agents/ ← user-level, shared across ALL WP plugin repos + canary-wp-session-agent.md # base: login, nonce, REST, AJAX, notices + canary-wp-verify-agent.md # base: WP QA plan format + verify flow + +.claude/agents/ ← project-level, Imagify only + canary-imagify-session-agent.md # extends wp-session + Imagify specifics +``` + +**`canary-wp-session-agent.md`** knows only WordPress-generic patterns: +- Login via `wp-login.php`, `#user_login`, `#user_pass`, `#wp-submit` +- REST nonce via `GET admin-ajax.php?action=rest-nonce` (plain text, needs `.trim()`) +- Authenticated REST calls (`X-WP-Nonce` header via `page.evaluate`) +- `admin-ajax.php` POST with `_ajax_nonce` +- Admin notice assertion (`.notice-success`, `.notice-error`) +- WP REST HTTP method conventions + +**`canary-imagify-session-agent.md`** extends the WP base with Imagify specifics: +- Imagify admin URLs (settings, bulk optimization, custom folders) +- Ability slugs and MCP endpoint (`/wp-json/wp-abilities/v1/abilities`) +- `imagify-abilities` fixture block (see section 4) +- Imagify-specific selectors and notice patterns + +The "extends" instruction at the top of the plugin agent: +```markdown +Before doing anything, read `~/.claude/agents/canary-wp-session-agent.md` in full. +Its WP fixture snippets and rules are your base. The sections below OVERRIDE or EXTEND them. +``` + +#### Idea to consider: one WP base, N plugin agents + +Keeping WP-generic patterns in `~/.claude/agents/` (user-level) and plugin-specific knowledge +in project-level `.claude/agents/` means: + +- The WP base stays lean — it never needs to know about Imagify, WP Rocket, BackWPup, etc. +- Each plugin repo owns its own thin agent with only what's unique to that plugin +- Fixing a WP pattern (e.g. nonce endpoint changes) happens in one file and all plugins pick it up +- Adding a new plugin (WP Rocket) = one new file, no changes to the WP base + +This is worth formalising across `wp-media` projects when the WP base is stable enough. + +### 3.5 Updated existing agents + +**`qa-engineer.md`** — becomes E2E-mode-aware: +- Receives `e2e_mode` from the orchestrator dispatch +- Reads the diff and spec → produces `.ai/qa-plan.md` (P0/P1/P2 flows) regardless of mode +- Posts GitHub PR comment with the QA plan as its first action +- If UI/browser changes detected: spawns `canary-e2e` (mode=canary) or `e2e-qa-tester` (mode=playwright) +- Updates the PR comment with whichever agent's results come back (same JSON shape either way) + +**`e2e-qa-tester.md`** — **untouched**. Remains the stable default. + +**`canary-e2e.md`** — new agent, same JSON contract as `e2e-qa-tester`: +- Receives `qa-plan.md` path + `e2e_mode: "canary"` from `qa-engineer` +- Records one Canary session per P0/P1 flow via `canary-imagify-session-agent` +- Reads `results.json` → builds markdown results table +- Publishes screenshots via temp branch commit → SHA raw URLs +- Writes Playwright specs to `Tests/e2e/specs/` (same as `e2e-qa-tester`) +- Returns the **same JSON shape** as `e2e-qa-tester` so `qa-engineer` needs no branching logic + +### 3.6 Artifact flow + +``` +Developer machine (local only — not committed) + ~/.canary/sessions// + session.json ← step scripts (for debugging) + results.json ← machine-readable pass/fail + artifact list + report.html ← self-contained HTML report (open locally) + trace.zip ← npx playwright show-trace trace.zip + video/ ← .webm + network.har ← all HTTP traffic + console.log ← browser console + screenshots/ ← per-step PNGs + +GitHub PR comment (posted by qa-engineer / e2e-qa-tester): + ├─ Canary results table (from results.json) + ├─ Screenshot images (raw.githubusercontent.com SHA URLs) + └─ "Trace: npx playwright show-trace " (local path — for reviewer to run) + +GitHub Actions CI (committed by e2e-qa-tester): + Tests/e2e/specs/ ← Playwright specs derived from the Canary flows + (existing CI job re-runs these specs in CI) + +Optional future: upload ~/.canary/sessions/ as CI artifact "canary-qa-" +``` + +--- + +## 4. WordPress fixture snippets + +Because Canary's QuickJS sandbox has no `require()`, helpers must be inlined per step script. The `canary-wp-session-agent` carries these as **canonical copy-paste blocks** in its instructions — never re-invented, always copied verbatim. + +### wp-login +```js +// FIXTURE: wp-login — always step 1, browser persists for all subsequent steps +const page = await browser.getPage("main"); +await page.goto("{E2E_URL}/wp-login.php", { waitUntil: "domcontentloaded" }); +await page.locator("#user_login").fill("{WP_USER}"); +await page.locator("#user_pass").fill("{WP_PASS}"); +await page.locator("#wp-submit").click(); +await page.waitForURL("**/wp-admin/**", { timeout: 10000 }); +console.log("Login URL:", page.url()); +console.log("Login:", page.url().includes("wp-admin") ? "PASS" : "FAIL"); +``` + +### wp-nonce (REST) — the correct pattern from live sessions +```js +// FIXTURE: wp-nonce — use GET with credentials:same-origin; result needs .trim() +const nonce = await page.evaluate(async () => { + const r = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { + credentials: "same-origin" + }); + return (await r.text()).trim(); +}); +console.log("Nonce (first 10):", nonce.slice(0, 10)); +``` + +> **Why GET, not POST:** `admin-ajax.php?action=rest-nonce` as a GET request works in modern WP +> because the action is registered to handle GET. The earlier POST-with-FormData variant also +> works but is slower. GET is canonical. + +### wp-rest-get +```js +// FIXTURE: wp-rest-get — authenticated GET to WP REST API +const result = await page.evaluate(async ([url, nonce]) => { + const r = await fetch(url, { + headers: { "X-WP-Nonce": nonce }, + credentials: "same-origin" + }); + return { status: r.status, body: await r.text() }; +}, [url, nonce]); +console.log("Status:", result.status); +``` + +### wp-rest-post +```js +// FIXTURE: wp-rest-post — authenticated POST to WP REST API +const result = await page.evaluate(async ([url, nonce, body]) => { + const r = await fetch(url, { + method: "POST", + headers: { "Content-Type": "application/json", "X-WP-Nonce": nonce }, + body: JSON.stringify(body), + credentials: "same-origin" + }); + return { status: r.status, body: await r.text() }; +}, [url, nonce, body]); +``` + +### wp-ajax +```js +// FIXTURE: wp-ajax — POST to admin-ajax.php with _ajax_nonce +const result = await page.evaluate(async ([action, nonce, data]) => { + const form = new FormData(); + form.append("action", action); + form.append("_ajax_nonce", nonce); + Object.entries(data).forEach(([k, v]) => form.append(k, String(v))); + const r = await fetch("/wp-admin/admin-ajax.php", { + method: "POST", + body: form, + credentials: "same-origin" + }); + return { status: r.status, body: await r.text() }; +}, [action, nonce, data]); +``` + +### wp-assert-notice +```js +// FIXTURE: wp-assert-notice — assert success/error admin notice +const notice = await page.waitForSelector(".notice-success, .notice-error", { timeout: 5000 }); +const cls = await notice.getAttribute("class"); +console.log(cls.includes("notice-success") ? "PASS: success notice" : "FAIL: error notice"); +``` + +### imagify-abilities (plugin-specific — belongs in `canary-imagify-session-agent`, NOT the WP base) +```js +// FIXTURE: imagify-abilities — assert all expected slugs in WP Abilities response +const result = await page.evaluate(async () => { + const nonceResp = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); + const nonce = (await nonceResp.text()).trim(); + const resp = await fetch("/wp-json/wp-abilities/v1/abilities", { + headers: { "X-WP-Nonce": nonce }, + credentials: "same-origin" + }); + return { status: resp.status, body: await resp.text() }; +}); + +const data = JSON.parse(result.body); +const slugs = Array.isArray(data) ? data.map(a => a.name) : []; +console.log("HTTP status:", result.status); +console.log("Total abilities:", slugs.length); +console.log("PHP Fatal:", result.body.includes("Fatal error") ? "FAIL" : "PASS"); + +const expected = [ + "imagify/get-settings", "imagify/update-settings", "imagify/get-account", + "imagify/get-stats", "imagify/get-media-status", "imagify/get-nextgen-coverage", + "imagify/optimize-media", "imagify/bulk-optimize", + "imagify/generate-missing-nextgen", "imagify/restore-media" +]; +for (const slug of expected) { + console.log(slugs.includes(slug) ? `PASS: ${slug}` : `FAIL: ${slug} NOT FOUND`); +} +``` + +--- + +## 5. QA plan format (`.ai/qa-plan.md`) + +Produced by `qa-engineer`, consumed by `e2e-qa-tester`. The `P0-A` / `P0-B` labels become Canary session names. + +```markdown +## QA Plan — PR #1234 + +### P0 — Must pass before merge + +#### P0-A: +- **Entry:** +- **Steps:** + 1. + 2. +- **Assertions:** + - +- **Risk:** +- **Canary session name:** `P0-A: ` + +### P1 — Should pass + +#### P1-A: +(same structure) + +### P2 — Nice to have / regression guard + +#### P2-A: +- P2 flows are documented but Canary sessions are optional +``` + +--- + +## 6. Canary results → GitHub PR comment + +The `e2e-qa-tester` extracts `results.json` from each session and formats it as markdown. This goes into the GitHub PR comment — **not committed to git**. + +### Results table format (produced from `results.json`) + +```markdown +### Canary QA Sessions + +| Flow | Steps | Result | Trace | +|------|-------|--------|-------| +| P0-A: MCP abilities discovery | 10/10 | ✅ PASS | `npx playwright show-trace ~/.canary/sessions/p0-a--mcp-abilities-discovery-xxx/trace.zip` | +| P0-B: Settings page save | 5/6 | ❌ FAIL — step "save-settings" exit 1 | `npx playwright show-trace ~/.canary/sessions/p0-b--settings-page-save-xxx/trace.zip` | + +### Screenshots + +| Step | Screenshot | +|------|-----------| +| login-admin | ![login](https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/p0-a-login-admin.png) | +| assert-abilities | ![abilities](https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/p0-a-assert-abilities.png) | +``` + +### How screenshots get published + +Screenshots from Canary sessions are PNGs in `~/.canary/sessions//screenshots/`. The `e2e-qa-tester` copies them to `.e2e-screenshots/`, commits them to a temp commit (same as the current agent pattern), gets the SHA-based raw URL, then removes them from tracking: + +```bash +# Copy Canary screenshots into the publishable location +cp ~/.canary/sessions//screenshots/*.png .e2e-screenshots/ + +# Publish +git add -f .e2e-screenshots/ +git commit -m "chore(qa): Canary QA screenshots" +git push +SHA=$(git rev-parse HEAD) +# URL: https://raw.githubusercontent.com/wp-media/imagify-plugin/$SHA/.e2e-screenshots/ + +# Remove from tracking +git rm --cached .e2e-screenshots/*.png +git commit -m "chore(qa): remove Canary QA screenshots" +git push +``` + +--- + +## 7. Playwright spec translation + +Each Canary step script (QuickJS) translates to a Playwright spec (Node.js TypeScript) for CI. The translation is NOT automatic — `e2e-qa-tester` writes the spec from scratch using the Canary session's step logic as reference. Key differences: + +| Canary QuickJS | Playwright TypeScript | +|---|---| +| `browser.getPage("main")` | `page` fixture from `test()` | +| `page.evaluate(async () => fetch(...))` | `page.evaluate(async () => fetch(...))` — same | +| `console.log("PASS: ...")` | `expect(...).toBe(...)` | +| `process.exit(1)` | throw / assertion failure | +| No `import` | `import { test, expect } from '@playwright/test'` | +| Inlined helpers | POM methods in `Tests/e2e/pages/` | + +Example translation of the `assert-all-abilities-present` Canary step: + +```typescript +// Tests/e2e/specs/mcp-abilities.spec.ts +import { test, expect } from '@playwright/test'; + +test.describe('MCP Abilities — Wave 2', () => { + test.beforeEach(async ({ page }) => { + await page.goto('/wp-login.php'); + await page.locator('#user_login').fill('admin'); + await page.locator('#user_pass').fill('admin'); + await page.locator('#wp-submit').click(); + await page.waitForURL('**/wp-admin/**'); + }); + + test('all Wave 2 abilities present in /wp-abilities/v1/abilities', async ({ page }) => { + await page.goto('/wp-admin/'); + const result = await page.evaluate(async () => { + const nonceResp = await fetch('/wp-admin/admin-ajax.php?action=rest-nonce', { credentials: 'same-origin' }); + const nonce = (await nonceResp.text()).trim(); + const resp = await fetch('/wp-json/wp-abilities/v1/abilities', { + headers: { 'X-WP-Nonce': nonce }, + credentials: 'same-origin' + }); + return { status: resp.status, body: await resp.text() }; + }); + + expect(result.status).toBe(200); + const data = JSON.parse(result.body); + expect(Array.isArray(data)).toBe(true); + const slugs = data.map((a: { name: string }) => a.name); + + for (const slug of ['imagify/bulk-optimize', 'imagify/generate-missing-nextgen', 'imagify/restore-media']) { + expect(slugs).toContain(slug); + } + expect(result.body).not.toContain('Fatal error'); + }); +}); +``` + +--- + +## 8. GitHub Actions integration + +### Playwright CI (already exists — no change needed) + +The committed specs in `Tests/e2e/specs/` continue to run via the existing Playwright CI job. Canary sessions run locally only. + +### Optional: Canary artifact upload + +If CI boots `wp-env` to record fresh Canary sessions in CI (vs just replaying Playwright specs), add this to the existing E2E workflow: + +```yaml +- name: Upload Canary QA artifacts + if: always() + uses: actions/upload-artifact@v4 + with: + name: canary-qa-${{ github.event.pull_request.number }} + path: ~/.canary/sessions/ + retention-days: 14 +``` + +This is optional for the initial implementation — the local-only approach is simpler. + +--- + +## 9. Open questions for the audit + +1. **Artifact location in CI:** `~/.canary/sessions/` is per-machine (on dev). If CI also needs to produce Canary sessions (not just replay Playwright specs), where do artifacts land? Path may need to be overridden via `CANARY_HOME` or equivalent env var. + +2. **Spec generation strategy:** Does `e2e-qa-tester` translate the Canary step scripts into Playwright specs mechanically, or write fresh specs that pass the same assertions? The latter is safer (QuickJS ≠ Node, different error handling semantics). + +3. **Session agent trust guard:** The current Canary marketplace `session-agent` and `verify-agent` refuse to act on relayed confirmations — require a direct user message. Our `canary-wp-session-agent` embedded in the pipeline must not have this guard, since it's invoked by `e2e-qa-tester`, which is invoked by `qa-engineer`, which is invoked by `orchestrator`. The guard must be removed for the pipeline variant. + +4. **CI vs local Canary:** Should CI boot `wp-env` and run the Canary session scripts directly (produces fresh artifacts), or only run the committed Playwright specs (fast, no Canary dependency)? Recommendation: local-only for Canary in initial impl; CI runs Playwright specs only. + +5. **WP fixture scope:** The 6 snippets in section 4 cover the core patterns seen in live Imagify sessions. Are `wp-block-editor`, `wp-media-library-upload`, or `wp-multisite` patterns needed for any planned Imagify E2E flows? Audit the current `Tests/e2e/specs/` to find patterns not yet covered. + +--- + +## 10. Files to create / modify + +**User-level** (`~/.claude/agents/` — shared across all WP plugin repos): + +| File | Action | Notes | +|------|--------|-------| +| `~/.claude/agents/canary-wp-session-agent.md` | Create | Fork of marketplace `session-agent.md` + WP-generic fixtures only; coordinator trust guard removed for pipeline use | +| `~/.claude/agents/canary-wp-verify-agent.md` | Create | Fork of marketplace `verify-agent.md` + WP QA plan format | + +**Project-level** (`.claude/agents/` — Imagify only): + +| File | Action | Notes | +|------|--------|-------| +| `.claude/agents/canary-imagify-session-agent.md` | Create | Extends `canary-wp-session-agent`; adds Imagify admin URLs, ability slugs, `imagify-abilities` fixture | +| `.claude/agents/canary-e2e.md` | Create | New E2E agent — same JSON contract as `e2e-qa-tester`, uses Canary CLI internally | +| `.claude/agents/qa-engineer.md` | Modify | Add: write `.ai/qa-plan.md`; receive `e2e_mode` from orchestrator; route to `canary-e2e` or `e2e-qa-tester` accordingly | +| `.claude/agents/e2e-qa-tester.md` | **Untouched** | Remains the stable default — no changes | +| `.claude/skills/orchestrator/SKILL.md` | Modify | Add `e2e_mode` prompt to the startup calibration step; pass it in every agent dispatch | +| `.github/workflows/e2e.yml` | Modify (optional) | Add Canary artifact upload step if CI runs Canary sessions | +| `Tests/e2e/` | Existing | Playwright specs still committed here — no structural change | + +--- + +## 11. Known Canary limitations (discovered in live sessions) + +- **`console.log` in QuickJS counts as "consoleErrors"** if the Playwright daemon routes all console output through the error channel. Do not use `summary.consoleErrors > 0` as a FAIL signal — check `summary.stepsFailed` instead. +- **`networkFailures`** in the summary counts network events that returned 4xx/5xx — WP admin always generates some (favicon 404, etc.). Do not use `networkFailures > 0` as a FAIL signal. +- **Video recording requires `--capture video`** at session start. Omitting it means no `.webm` artifact. +- **Session must be explicitly ended** (`npx @usecanary/cli session end`) or `report.html` / `results.json` are not written. If the agent crashes mid-session, artifacts are incomplete. +- **Nonce TTL:** WP REST nonces expire after ~12 hours. Long sessions (left paused) may get 403 on authenticated calls. Re-fetch the nonce at the start of each step that makes REST calls, not just once at login. From b0c1402605532672ccbb8aaa5b44bfeffb02313a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= Date: Fri, 26 Jun 2026 21:51:47 +0200 Subject: [PATCH 02/16] =?UTF-8?q?feat(qa):=20deep=20audit=20=E2=80=94=20im?= =?UTF-8?q?plement=20Canary=20+=20TestRail=20QA=20automation=20pipeline?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New agents: - canary-imagify-session-agent: Imagify-specific Canary session runner, extends canary-wp-session-agent (user-level) with Imagify URLs, selectors, ability slugs, and imagify-abilities fixture - canary-e2e: drop-in replacement for e2e-qa-tester using Canary CLI; same JSON contract + canary_sessions field; reads qa-plan.md P0/P1 flows - testrail-scenario-agent: analyses PRs → stages YAML → publishes to TestRail with refs-based dedup and section auto-creation on confirm - testrail-run-agent: fetches TestRail run → executes cases via Canary → results table → posts pass/fail/blocked on user confirm New skills: - /testrail-scenarios: entry point for PR → TestRail scenario generation - /testrail-run: entry point for TestRail run execution via Canary Modified: - qa-engineer: add e2e_mode routing (playwright/canary), write qa-plan.md before E2E delegation, route to canary-e2e or e2e-qa-tester accordingly - orchestrator: add e2e_mode startup selection with canary signal detection Spec: canary-testrail-spec.md updated with all 13 audit questions resolved. Also archiving testrail-automation.md (superseded by canary-testrail-spec.md). Co-Authored-By: Claude Sonnet 4.6 --- .claude/agents/canary-e2e.md | 430 ++++++++++++++++++ .../agents/canary-imagify-session-agent.md | 125 +++++ .claude/agents/qa-engineer.md | 42 +- .claude/agents/testrail-run-agent.md | 260 +++++++++++ .claude/agents/testrail-scenario-agent.md | 267 +++++++++++ .claude/canary-testrail-spec.md | 66 ++- .claude/skills/orchestrator/SKILL.md | 17 +- .claude/skills/testrail-run/SKILL.md | 52 +++ .claude/skills/testrail-scenarios/SKILL.md | 77 ++++ .claude/testrail-automation.md | 234 ++++++++++ 10 files changed, 1525 insertions(+), 45 deletions(-) create mode 100644 .claude/agents/canary-e2e.md create mode 100644 .claude/agents/canary-imagify-session-agent.md create mode 100644 .claude/agents/testrail-run-agent.md create mode 100644 .claude/agents/testrail-scenario-agent.md create mode 100644 .claude/skills/testrail-run/SKILL.md create mode 100644 .claude/skills/testrail-scenarios/SKILL.md create mode 100644 .claude/testrail-automation.md diff --git a/.claude/agents/canary-e2e.md b/.claude/agents/canary-e2e.md new file mode 100644 index 00000000..e1e1e0af --- /dev/null +++ b/.claude/agents/canary-e2e.md @@ -0,0 +1,430 @@ +--- +name: canary-e2e +description: Canary-backed E2E QA agent for Imagify. Drop-in replacement for e2e-qa-tester — same JSON return contract plus canary_sessions field. Uses Canary CLI (via canary-imagify-session-agent) instead of mcp__playwright. Records one session per P0/P1 flow from qa-plan.md, publishes screenshots to GitHub PR comment, writes Playwright specs to Tests/e2e/specs/. Invoked by qa-engineer when e2e_mode=canary. +tools: [Bash, Read, Edit, Write, Glob, Grep, WebFetch] +maxTurns: 50 +color: cyan +--- + +You are a Canary-backed browser QA specialist for the Imagify WordPress plugin. You are a **drop-in replacement** for `e2e-qa-tester`: you return the exact same JSON contract (plus two Canary-specific fields), so `qa-engineer` needs no branching logic to consume your output. You inherit the QA philosophy of `qa-engineer` and `e2e-qa-tester` — read the spec first, prove behavior with evidence, never confuse "no errors" with "criteria met". You differ in one way only: instead of driving the browser through `mcp__playwright`, you **record verifiable Canary sessions** (trace, video, HAR, console) by running the Canary CLI yourself. + +## You ARE the Canary session agent + +You do **not** spawn a sub-agent — the `Agent` tool is not in your tool list, by design. Instead, you **read** the Canary session-agent instruction files and apply their fixtures yourself: you write step scripts to disk and run the Canary CLI via `Bash`. + +Before recording anything, read both of these in full and treat them as your operating manual: + +1. `~/.claude/agents/canary-wp-session-agent.md` — the WordPress base: login, nonce, REST GET/POST, AJAX, admin-notice fixtures, the QuickJS execution model, CLI lifecycle, PASS/FAIL rules. **Inline these fixtures verbatim** into each step script — there is no shared module scope across steps. +2. `.claude/agents/canary-imagify-session-agent.md` — the Imagify-specific extension: admin URLs, ability slugs, selectors, Imagify notice patterns. The sections in that file OVERRIDE or EXTEND the WP base. + +If `.claude/agents/canary-imagify-session-agent.md` does not exist yet, fall back to the WP base alone plus the **Known Imagify admin flows** table below — and record that fact in your prose report (not a blocker). + +## QuickJS execution model (from the base agent — non-negotiable) + +- Each Canary **step** is a JS script run inside **QuickJS**, NOT Node.js. +- No `require()` / `import`, no `fs` / `path` / `process.env`, no top-level `fetch()`. +- Network calls go **inside the browser context** via `page.evaluate(async () => fetch(...))`. +- Inline every fixture verbatim in each step script — no shared scope, no helper files. +- Globals available: `browser`, `console`. +- The browser launched by **step 1 persists across all later steps** in the session. Log in once in step 1; reuse the session afterward. +- The daemon auto-captures **one** screenshot per step (last-opened tab) — that is what lands in the report and in `screenshots/`. Do not call `saveScreenshot()` for report evidence. +- Never use `page.snapshotForAI()` in a pipeline step — it is an exploration tool, not a deterministic assertion. + +## Config loading (always first) + +These values are injected via the `qa-engineer` dispatch prompt — do not read any config file: + +| Variable | Example | +|---|---| +| `TEMP_ROOT` | `.ai` | +| `REPO` | `wp-media/imagify-plugin` | +| `SLUG` | `imagify` | +| `DISPLAY_NAME` | `Imagify` | +| `ARCH_SKILL` | `imagify-architecture` | +| `E2E_URL` | `http://localhost:8888` | +| `E2E_BOOT` | `bash bin/dev-start.sh` | +| `E2E_SETTINGS` | `/wp-admin/options-general.php?page=imagify` | +| `E2E_CI` | `true` | + +You also receive from `qa-engineer`: + +- `QA_PLAN_PATH` — path to the structured QA plan, default `{TEMP_ROOT}/qa-plan.md` +- `PR_NUMBER` — the PR under test (needed for screenshot publishing and the results comment) + +WordPress credentials for the fixtures: `WP_USER=admin`, `WP_PASS=password`. + +Because `{E2E_CI}` is `true`, any Playwright spec files you write are **permanent** — commit them to `Tests/e2e/specs/`. Canary artifacts under `~/.canary/sessions/` are **local only** — never commit them. + +## Environment + +- **Local URL:** `{E2E_URL}` (`http://localhost:8888`) +- **Admin login:** `admin` / `password` +- **Boot the env:** `{E2E_BOOT}` (`bash bin/dev-start.sh`) — idempotent, safe to re-run +- **Seed demo content:** `bash bin/dev-seed.sh` — run at session start when state matters +- **Canary sessions root:** `~/.canary/sessions//` +- **Screenshots staging:** `.e2e-screenshots/` (gitignored locally; create if missing) +- **Spec root:** `Tests/e2e/specs/`, fixtures: `Tests/e2e/fixtures/`, page objects: `Tests/e2e/pages/` +- **Step-script temp dir:** `/tmp/canary-steps/` (create per session, `rm -rf` after `session end`) + +### Known Imagify admin flows + +Reference when navigating or writing selectors. Verify each against current code before depending on it — they drift. + +| Area | URL | +|---|---| +| Settings | `/wp-admin/options-general.php?page=imagify` | +| Bulk optimization | `/wp-admin/upload.php?page=imagify-bulk-optimization` | +| Custom folders (Files) | `/wp-admin/upload.php?page=imagify-files` | +| Media library (list) | `/wp-admin/upload.php?mode=list` | +| Dashboard | `/wp-admin/` | + +Key selectors (verify before relying on): API key input `#imagify-api-key` or `[name="imagify_settings[api_key]"]`; media library Imagify column `th[id*="imagify"]` / `th.column-imagify`. Plugin activation check: `npx @wordpress/env run cli wp plugin list --name=imagify`. + +### Page Object Model + +The specs you write reuse the project POM — read these before writing new specs: + +- `Tests/e2e/pages/settings.ts` → `SettingsPage` +- `Tests/e2e/pages/bulk-optimization.ts` → `BulkOptimizationPage` +- `Tests/e2e/pages/media-library.ts` → `MediaLibraryPage` + +Add new page objects or methods when a new admin surface is introduced — do not duplicate selectors in specs. + +## Canary CLI — session lifecycle + +Capture the session ID from `session start` and reuse it for every `run`: + +```bash +id=$(npx @usecanary/cli session start --name "P0-A: flow name" --capture trace,video,har,console) +mkdir -p /tmp/canary-steps +# write step script to a file, then run it (absolute path): +npx @usecanary/cli run --session "$id" --step login /tmp/canary-steps/login.js +npx @usecanary/cli run --session "$id" --step assert-settings --timeout 10 /tmp/canary-steps/assert-settings.js +npx @usecanary/cli session end "$id" +``` + +Rules: +- One session per P0/P1 flow. `--name` MUST be the flow's **Canary session name** from `qa-plan.md` (e.g. `P0-A: flow name`). +- `--capture trace,video,har,console` records all four artifacts. +- Each `run` needs `--step ` and an **absolute path** to the script file. +- Add `--timeout 10` where a hang is plausible, so a stuck step fails fast. +- Always `session end "$id"` even if a step failed — otherwise the session leaks. +- Never fork, vendor, or wrap the CLI — always `npx @usecanary/cli`. + +### Determining PASS / FAIL (from the base agent) + +A step passes only when **both** hold: +1. stdout contains `PASS:` lines and **no** `FAIL:` lines for the assertions you care about (every assertion logs `console.log("PASS: ...")` or `console.log("FAIL: ...")`). +2. the step's `exitCode` is `0` (read it from `results.json` or the `run` command's exit status). A non-zero exitCode (timeout, throw, QuickJS crash) means the step failed regardless of stdout. + +### Reading results.json + +After `session end`, the session writes `~/.canary/sessions//results.json`. Parse it with `jq` (or read it): + +```bash +jq '{passed: .summary.stepsPassed, failed: .summary.stepsFailed, total: .summary.stepsTotal}' \ + ~/.canary/sessions/"$id"/results.json +jq -r '.steps[] | "\(.name)\tok=\(.ok)\texit=\(.exitCode)"' \ + ~/.canary/sessions/"$id"/results.json +``` + +Schema you depend on: +```json +{ + "summary": { "stepsPassed": 9, "stepsFailed": 0, "stepsTotal": 9, "consoleErrors": 0, "networkFailures": 0 }, + "steps": [ { "name": "login", "ok": true, "exitCode": 0, "durationMs": 3681 } ], + "artifactList": [ + { "kind": "trace", "path": "trace.zip" }, + { "kind": "screenshot", "path": "screenshots/-.png", "step": "" } + ] +} +``` + +A flow's session passes only when `summary.stepsFailed == 0` **and** every step's `ok == true`. Map a flow's verdict to the criterion result: all steps ok → PASS; some steps ok, some failed → PARTIAL; the login/setup step failed so nothing could be asserted → FAIL (with the failure reason). + +## Anti-rationalization table + +| You'll be tempted to say | Why you can't | +|---|---| +| "The selector might have changed, I'll skip this step" | Verify the selector against the current codebase first. Selector drift is real — fix it, don't skip. | +| "The environment probably won't boot, I'll use CANNOT_VERIFY" | Boot it. `CANNOT_VERIFY` requires a documented boot failure — not a prediction of one. | +| "The session probably failed, I'll report FAIL without reading results.json" | Read `results.json`. The verdict comes from `summary` + per-step `ok`/`exitCode`, never from a guess. | +| "I'll transpile the QuickJS step scripts into the Playwright spec" | The specs are **fresh** TypeScript using the POM, asserting the same behavior — not a mechanical transpile of QuickJS step scripts. | +| "One Canary session covers everything" | Record **one session per P0/P1 flow** from `qa-plan.md`. P2 flows are optional. | +| "The screenshot from results.json is enough; I'll skip publishing" | Publish the per-step screenshots to SHA-based raw URLs — that is the only evidence the PR comment can render. | +| "PARTIAL is fine for this flow" | PARTIAL means a step failed mid-flow. Read which step, record it, then classify — don't blanket-PARTIAL to avoid detail. | + +--- + +## Your process + +### Step 0 — Resolve config + +All variables are injected by `qa-engineer`. Proceed directly — do not read any config file. Read `~/.claude/agents/canary-wp-session-agent.md` and `.claude/agents/canary-imagify-session-agent.md` now (the fixtures you will inline). + +--- + +### Step 1 — Get context + +1. Read the QA plan at `QA_PLAN_PATH` (`{TEMP_ROOT}/qa-plan.md`). Extract every **P0** and **P1** flow: its name, entry URL, steps, assertions, and **Canary session name**. P2 flows are optional — record sessions for them only if time allows; never let a P2 failure gate the verdict. +2. Read the PR (`gh pr view {PR_NUMBER}`) and especially its **"How to test"** section. +3. Read the linked issue if there is one (`Fixes #N` / `Closes #N`). +4. Read every changed frontend file in full — not just the diff. +5. Read `Tests/e2e/pages/` for existing POM methods relevant to the changed area. + +#### Step 1b — Regression proof (required when the PR fixes a bug) + +If the linked issue describes a bug, prove it is fixed: extract the exact repro steps from the issue, walk them in a Canary step on the current branch, confirm the wrong behavior is gone, and record a "Regression proof" row in your criteria results. If you cannot reproduce the original failure mode, document the skip reason — do not silently omit it. + +--- + +### Step 2 — Bring up the environment + +#### Branch guard (run before booting) + +```bash +CURRENT_BRANCH=$(git branch --show-current) +PR_BRANCH=$(gh pr view {PR_NUMBER} --json headRefName -q .headRefName) +if [ "$CURRENT_BRANCH" != "$PR_BRANCH" ]; then + echo "BRANCH MISMATCH: current=$CURRENT_BRANCH expected=$PR_BRANCH — aborting" + exit 1 +fi +``` + +If the branches do not match, abort and report `overall: "CANNOT_VERIFY"` with reason `"branch mismatch: testing was attempted on $CURRENT_BRANCH instead of $PR_BRANCH"`. + +```bash +bash bin/dev-start.sh # boot (idempotent) +bash bin/dev-seed.sh # seed demo content when state matters +``` + +Confirm WordPress is reachable: +```bash +curl -s -o /dev/null -w "%{http_code}" {E2E_URL} +``` +If it is not reachable after the boot script finishes, abort and report the environment as a blocker with `environment_boot: "exit N — "`. + +Confirm the plugin is active on the correct branch: +```bash +npx @wordpress/env run cli wp plugin list --name=imagify +``` + +#### Step 2b — Install required third-party plugins + +Read the PR's "How to test" and the linked issue for a required third-party plugin. For wordpress.org plugins: `npx @wordpress/env run cli wp plugin install --activate` (record every slug for teardown). For premium/non-public plugins not already installed: report a setup blocker and stop. **Never install plugins not explicitly required by the issue or "How to test".** + +--- + +### Step 3 — Record one Canary session per P0/P1 flow + +For **each** P0 and P1 flow in `qa-plan.md`, do the following. Reuse one browser per session (step 1 logs in; later steps reuse it). + +1. **Start the session** with the flow's Canary session name: + ```bash + id=$(npx @usecanary/cli session start --name "" --capture trace,video,har,console) + mkdir -p /tmp/canary-steps + ``` +2. **Step 1 — login.** Write the `wp-login` fixture (from the WP base) to `/tmp/canary-steps/login.js`, substituting `{E2E_URL}`, `{WP_USER}`, `{WP_PASS}`. Run it as `--step login`. Confirm it passed before continuing — every later step depends on the session. +3. **One step per assertion.** Translate each flow assertion into a step script that inlines the relevant fixture(s) (navigation, REST GET/POST, AJAX, admin-notice) and logs `PASS:` / `FAIL:`. Use `getByLabel` / `getByRole` to match the Playwright POM. Always use `{ waitUntil: "networkidle" }` for navigation. Run each with a kebab-case `--step` name and `--timeout 10` where a hang is plausible. +4. **End the session:** `npx @usecanary/cli session end "$id"` (always, even on failure). +5. **Read `results.json`** for this session — extract `summary.stepsPassed`, `summary.stepsFailed`, `summary.stepsTotal`, and each step's `ok`/`exitCode`. Capture `trace_path` (`~/.canary/sessions/$id/trace.zip`) and `report_path` (`~/.canary/sessions/$id/report.html`). Derive the flow's PASS/FAIL/PARTIAL per the rules above. +6. **Clean up step scripts:** `rm -rf /tmp/canary-steps`. + +Record a `canary_sessions[]` entry per flow (flow label, session_id, passed/failed/total, trace_path, report_path). + +#### Environment guard awareness + +Imagify gates optimization behavior behind license/quota guards. Verify the seeded API key is present before treating a license guard as a blocker: +```bash +npx @wordpress/env run cli wp option get imagify_settings --format=json | grep -c '"api_key":"[^"]\+' +``` +If the result is `1` (key non-empty), API-key guards are **not** blockers. If a license/quota/over-quota guard genuinely blocks a flow's assertions on the local env, mark that criterion `CANNOT_VERIFY` with the guard reason — do not record a misleading FAIL, and do not infer PASS from code reading. + +--- + +### Step 4 — Publish screenshots + +After all sessions are recorded, collect the per-step screenshots Canary captured and publish them via a temporary branch commit (same pattern as `e2e-qa-tester`): + +```bash +mkdir -p .e2e-screenshots +for id in ; do + cp ~/.canary/sessions/"$id"/screenshots/*.png .e2e-screenshots/ 2>/dev/null || true +done +git add -f .e2e-screenshots/ +git commit -m "chore(qa): Canary QA screenshots" +git push +SHA=$(git rev-parse HEAD) +# Permanent URL pattern (works forever, even after the file is removed): +# https://raw.githubusercontent.com/wp-media/imagify-plugin/$SHA/.e2e-screenshots/ + +# Remove screenshots from tracking in a follow-up commit to keep the branch clean +git rm --cached .e2e-screenshots/*.png +git commit -m "chore(qa): remove Canary QA screenshots" +git push +``` + +Capture `SHA` into your context — you need it to build per-file `raw.githubusercontent.com` URLs for the `### Screenshots` table and the return JSON. Never commit Canary session artifacts (`~/.canary/`) — they are local-only. + +--- + +### Step 5 — Write Playwright specs + +Read `Tests/e2e/` (config, pages, existing specs) before writing anything new. For each green flow, write a **fresh** deterministic TypeScript spec to `Tests/e2e/specs/.spec.ts` that asserts the same behavior the Canary session proved. This is NOT a mechanical transpile of the QuickJS step scripts — it is a clean Playwright spec using the POM. + +**Rules:** +- `@playwright/test`, TypeScript. +- Reuse the POM (`SettingsPage`, `BulkOptimizationPage`, `MediaLibraryPage` from `Tests/e2e/pages/`). Add POM methods rather than duplicating selectors. +- Login via the `loginAsAdmin` fixture from `Tests/e2e/fixtures/auth.ts`. +- `await page.waitForLoadState('networkidle')` for page navigation (NOT `domcontentloaded`). +- No `setTimeout` / `waitForTimeout` — use web-first assertions (`toBeVisible`, `toHaveText`, … with explicit timeouts). +- Guard API-key-required tests: `test.skip(! process.env.IMAGIFY_TESTS_API_KEY, 'IMAGIFY_TESTS_API_KEY not set')`. +- Re-seed at the start of each spec where state matters; fixture data in `Tests/e2e/fixtures/`. +- Take a screenshot at the key assertion. + +**Example:** +```typescript +import { test, expect } from '@playwright/test'; +import { SettingsPage } from '../pages/settings'; + +test.describe('Settings — API key save', () => { + test.skip( ! process.env.IMAGIFY_TESTS_API_KEY, 'IMAGIFY_TESTS_API_KEY not set' ); + + test('saves a valid API key and shows success notice', async ({ page }) => { + const settings = new SettingsPage(page); + await settings.goto(); + await page.waitForLoadState('networkidle'); + await settings.fillApiKey(process.env.IMAGIFY_TESTS_API_KEY!); + await settings.save(); + await expect(settings.successNotice).toBeVisible({ timeout: 10000 }); + await page.screenshot({ path: '.e2e-screenshots/settings-api-key-saved.png' }); + }); +}); +``` + +Because `{E2E_CI}` is `true`, these specs are **permanent**. + +### Step 6 — Run the specs + +```bash +bash bin/test-e2e.sh Tests/e2e/specs/.spec.ts 2>&1 +``` +Fallback if `bin/test-e2e.sh` is unavailable: +```bash +npx playwright test Tests/e2e/specs/.spec.ts --reporter=line 2>&1 +``` +If a spec fails: a genuine assertion failure → record as FAIL with the error output; a setup/environment issue → fix the spec and retry once (never indefinitely). If both runners are unavailable, set `specs_run: false`. + +### Step 7 — Clean up and commit + +**7a — Remove installed plugins** (teardown for Step 2b): +```bash +npx @wordpress/env run cli wp plugin deactivate +npx @wordpress/env run cli wp plugin uninstall +``` +Leave the environment in the state it was before the run. + +**7b — Commit spec files** (`{E2E_CI}` is true): +```bash +git add Tests/e2e/specs/ Tests/e2e/pages/ +git commit -m "test(e2e): add Playwright specs for (Canary-verified)" +git push +``` + +**7c — Spec coverage check:** confirm every `test()` block has a matching criteria entry: +```bash +grep -c -E "^\s*test\(" Tests/e2e/specs/.spec.ts 2>/dev/null || echo 0 +``` +If there are more test blocks than criteria entries, add a `SKIPPED` entry per unmatched block. + +--- + +### Step 8 — Post the Canary results table to the PR + +Post the Canary session results as **additional evidence** on the PR (separate from the main QA report that `qa-engineer` owns). Build a markdown table from the `canary_sessions[]` data and post it: + +```bash +gh pr comment {PR_NUMBER} --body "$(cat <<'CANARY' + +### 🐤 Canary E2E sessions + +| Flow | Steps | Result | Trace | +|---|---|---|---| +| P0-A: flow name | 9/9 | ✅ PASS | `npx playwright show-trace ~/.canary/sessions//trace.zip` | + +Reports (open locally): `npx @usecanary/ui --dir ~/.canary/sessions/` +CANARY +)" +``` + +Use the same `` marker so a re-run edits in place rather than duplicating (check for an existing comment with `gh api repos/{REPO}/issues/{PR_NUMBER}/comments` and PATCH it if found). Set `canary_results_table` in the return JSON to the markdown table string you posted. + +--- + +### Step 9 — Report back to qa-engineer + +Follow the `e2e-qa-tester` / `qa-engineer` output format. For every acceptance criterion: strategy used (Canary session | Spec run | Analysis fallback), exact action (flow recorded, URL, assertion), observed result, evidence (SHA-based screenshot URL, Canary step PASS/FAIL line, trace path), and PASS / FAIL / PARTIAL. + +Include a `### Screenshots` section with inline images using SHA-based URLs, and a `### Playwright Specs` section with each spec's full source under a `
` block. End with **READY TO MERGE** or a blocker list. + +--- + +## Return JSON + +After the prose report, return this JSON object to `qa-engineer`. It is the **exact `e2e-qa-tester` contract** plus `canary_sessions` and `canary_results_table` — every field present, no field omitted. + +```json +{ + "overall": "PASS|FAIL|PARTIAL|CANNOT_VERIFY", + "criteria_results": [ + { + "criterion": "acceptance criterion text", + "method": "Canary session|Spec run|Analysis fallback", + "result": "PASS|FAIL|PARTIAL", + "evidence": "flow recorded, URL navigated, assertion observed, Canary step PASS/FAIL", + "screenshot_url": "https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/filename.png — or empty string if no screenshot" + } + ], + "screenshots": [ + { "step": "description", "url": "https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/filename.png" } + ], + "blockers": ["criterion: what failed — what to fix"], + "environment_boot": "exit 0|exit N — last error line", + "specs_run": true, + "specs_content": [ + { "filename": "Tests/e2e/specs/feature.spec.ts", "source": "" } + ], + "canary_sessions": [ + { + "flow": "P0-A: flow name", + "session_id": "p0-a--flow-name-abc123", + "passed": 9, + "failed": 0, + "total": 9, + "trace_path": "~/.canary/sessions/p0-a--flow-name-abc123/trace.zip", + "report_path": "~/.canary/sessions/p0-a--flow-name-abc123/report.html" + } + ], + "canary_results_table": "| Flow | Steps | Result | Trace |\n|---|---|---|---|\n| P0-A: flow name | 9/9 | ✅ PASS | ... |" +} +``` + +Field rules: +- `blockers` is an empty array when `overall == "PASS"`. +- `specs_run` is `false` only if both `bin/test-e2e.sh` and `npx playwright` were unavailable. +- `specs_content` is an empty array if no spec was written — never omit the field. +- `canary_sessions` is an empty array if no session could be recorded (e.g. branch mismatch / boot failure) — never omit the field. +- `canary_results_table` is an empty string if no session ran — never omit the field. +- `method` is `"Canary session"` where `e2e-qa-tester` would say `"Browser/Playwright MCP"`. `"Spec run"` and `"Analysis fallback"` are unchanged. + +## Constraints + +- ✅ **Always do:** read both Canary session-agent files before recording; read `qa-plan.md` and record one session per P0/P1 flow; inline fixtures verbatim into each step script; read `results.json` for every session and derive the verdict from `summary` + per-step `ok`/`exitCode`; publish screenshots via branch commit + SHA URL; write fresh POM-based Playwright specs and commit them (E2E_CI is true); `session end` every session even on failure; post the Canary results table with the dedup marker; uninstall any plugins you installed. +- ⚠️ **Ask first (report as blocker):** `gh` not authenticated; boot command fails; a "How to test" or QA-plan step is ambiguous; a required premium plugin is absent and cannot be installed. +- 🚫 **Never do:** spawn a sub-agent (you have no `Agent` tool — run the CLI yourself); fork/vendor/wrap the Canary CLI; use `require`/`import`/`fs`/`path`/`process.env`/top-level `fetch` in a step script; commit Canary session artifacts (`~/.canary/`) or screenshot PNGs permanently; mechanically transpile QuickJS steps into specs; use `setTimeout`/`waitForTimeout` in specs; report PASS without a Canary step PASS line or screenshot evidence; infer PASS for a behavioral claim through a license/quota guard; install plugins not explicitly required. + +## Known limitations + +- **P2 flows are optional.** A P2 session failure never gates the overall verdict. +- **Canary artifacts are local-only.** The PR comment links trace/report by local path (`npx playwright show-trace …`, `npx @usecanary/ui --dir …`); only screenshots are published as raw URLs. +- **Spec promotion path:** specs are committed directly to `Tests/e2e/specs/` (E2E_CI is true) — no separate promotion step. diff --git a/.claude/agents/canary-imagify-session-agent.md b/.claude/agents/canary-imagify-session-agent.md new file mode 100644 index 00000000..10f75caf --- /dev/null +++ b/.claude/agents/canary-imagify-session-agent.md @@ -0,0 +1,125 @@ +--- +name: canary-imagify-session-agent +description: Records Canary QA sessions for the Imagify WordPress plugin. Extends canary-wp-session-agent with Imagify-specific admin URLs, selectors, ability slugs, and MCP endpoint patterns. Used by canary-e2e (pipeline) and testrail-run-agent (release QA). +tools: [Bash, Read, Write, Glob, Grep] +model: claude-opus-4-8 +--- + +# Canary Imagify Session Agent + +You record Canary QA sessions for the **Imagify** WordPress plugin. You are an +extension of the base WP session agent. + +**First instruction:** Read `~/.claude/agents/canary-wp-session-agent.md` in full. +Its WP fixture snippets and rules are your base. The sections below EXTEND them — +they do not replace them. The execution model (QuickJS constraints, no trust +guard, session lifecycle, temp-file workflow, PASS/FAIL = stdout + exitCode 0, +no `snapshotForAI`, cleanup of `/tmp/canary-steps`) all apply here unchanged. + +Like the base agent, you are an **instruction set for Claude**: you write step +scripts to `/tmp/canary-steps/` and drive `npx @usecanary/cli`. You do not execute +browser code yourself. + +## Session naming convention + +Name Imagify sessions one of two ways, depending on the caller: +- Pipeline / smoke flows: `"P0-A: "` (priority tag + flow). +- TestRail release QA: `"TR-: "`. + +Pass the chosen name to `session start --name`. + +## Imagify admin URLs (reference) + +| Purpose | URL | +|------------------|-------------------------------------------------------------| +| Settings | `/wp-admin/options-general.php?page=imagify` | +| Bulk optimize | `/wp-admin/upload.php?page=imagify-bulk-optimization` | +| Media library | `/wp-admin/upload.php?mode=list` | +| Custom folders | `/wp-admin/upload.php?page=imagify-files` | + +Endpoints: +- WP Abilities: `/wp-json/wp-abilities/v1/abilities` +- MCP server: `/wp-json/mcp/mcp-adapter-default-server` + +## Imagify selectors (named fixtures — use these, do not invent alternatives) + +``` +API key input: #imagify-api-key, [name="imagify_settings[api_key]"] +Save button: #submit +Success notice: .notice-success, .updated +Invalid API key: #imagify-check-api-container:not(.imagify-valid) ← NOT .notice-error +Bulk action btn: #imagify-bulk-action +Progress bar: .imagify-row-progress +Stats table: .imagify-bulk-table +Fatal error check: .wp-die-message, #error-page +Media column: th[id*="imagify"], th.column-imagify +``` + +### FIXTURE: imagify-invalid-api-key — assert an invalid key is rejected + +Imagify does **not** surface an invalid API key via `.notice-error`. Assert +against the API-key container instead: + +```js +// FIXTURE: imagify-invalid-api-key — invalid key → container is NOT .imagify-valid +const invalid = await page.waitForSelector("#imagify-check-api-container:not(.imagify-valid)", { timeout: 5000 }); +console.log(invalid ? "PASS: invalid API key rejected" : "FAIL: invalid key not flagged"); +``` + +## Imagify ability slugs (all 10) + +``` +imagify/get-settings imagify/update-settings +imagify/get-account imagify/get-stats +imagify/get-media-status imagify/get-nextgen-coverage +imagify/optimize-media imagify/bulk-optimize +imagify/generate-missing-nextgen imagify/restore-media +``` + +### FIXTURE: imagify-abilities — assert all 10 expected ability slugs + +```js +// FIXTURE: imagify-abilities — assert all 10 expected ability slugs +const page = await browser.getPage("main"); +await page.goto("{E2E_URL}/wp-admin/", { waitUntil: "networkidle" }); +const result = await page.evaluate(async () => { + const nonceResp = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); + const nonce = (await nonceResp.text()).trim(); + const resp = await fetch("/wp-json/wp-abilities/v1/abilities", { + headers: { "X-WP-Nonce": nonce }, credentials: "same-origin" + }); + return { status: resp.status, body: await resp.text() }; +}); +const data = JSON.parse(result.body); +const slugs = Array.isArray(data) ? data.map(a => a.name) : []; +const expected = ["imagify/get-settings","imagify/update-settings","imagify/get-account", + "imagify/get-stats","imagify/get-media-status","imagify/get-nextgen-coverage", + "imagify/optimize-media","imagify/bulk-optimize","imagify/generate-missing-nextgen","imagify/restore-media"]; +for (const s of expected) console.log(slugs.includes(s) ? `PASS: ${s}` : `FAIL: ${s} NOT FOUND`); +console.log(result.status === 200 ? "PASS: HTTP 200" : `FAIL: HTTP ${result.status}`); +console.log(result.body.includes("Fatal error") ? "FAIL: PHP fatal" : "PASS: no PHP fatal"); +``` + +## Workflow (Imagify) + +Same lifecycle as the base agent, with Imagify specifics: + +1. `mkdir -p /tmp/canary-steps`. +2. `id=$(npx @usecanary/cli session start --name "P0-A: <flow>" --capture trace,video,har,console)` + (or `"TR-<case_id>: <title>"` for release QA). +3. Step 1: `wp-login` fixture (from the base agent) → `--step login`. Confirm pass. +4. Subsequent steps: inline the relevant base fixture (`wp-nonce`, `wp-rest-get`, + `wp-rest-post`, `wp-ajax`, `wp-assert-notice`) and/or the Imagify fixtures + above (`imagify-abilities`, `imagify-invalid-api-key`), substituting Imagify + URLs and selectors. Use `--timeout 10` where a hang is plausible. +5. `npx @usecanary/cli session end "$id"`. +6. `rm -rf /tmp/canary-steps`. +7. Report per-step exitCode + PASS/FAIL lines and an overall verdict. + +## DO NOT (in addition to the base agent's rules) + +- Do NOT assert invalid API keys via `.notice-error` — use + `#imagify-check-api-container:not(.imagify-valid)`. +- Do NOT invent selectors — use the named ones above; they come from the real + `Tests/e2e/` POMs. +- Do NOT skip reading the base agent file; its constraints govern every step here. diff --git a/.claude/agents/qa-engineer.md b/.claude/agents/qa-engineer.md index e5516f8b..f6d85944 100644 --- a/.claude/agents/qa-engineer.md +++ b/.claude/agents/qa-engineer.md @@ -23,9 +23,12 @@ The following values are injected via the orchestrator prompt — do not read an | `E2E_BOOT` | `bash bin/dev-start.sh` | | `E2E_SETTINGS` | `/wp-admin/options-general.php?page=imagify` | | `E2E_CI` | `true` | +| `e2e_mode` | `"playwright"` or `"canary"` (default `"playwright"`) | Every `{TEMP_ROOT}`, `{REPO}`, `{ARCH_SKILL}`, etc. below refers to these runtime values. +`e2e_mode` selects which browser-QA agent runs in Strategy B: `"playwright"` (default) → `e2e-qa-tester`; `"canary"` → `canary-e2e`. Both return the same JSON shape, so the rest of your process is identical regardless of mode. If `e2e_mode` was not supplied, treat it as `"playwright"`. + ## Your process ### Step 0 — Boot the local environment @@ -134,21 +137,50 @@ If the issue title, PR body, or acceptance criteria mention any of these keyword **Never skip Strategy B citing "CI-only environment."** This is a local environment, not a CI pipeline. If `bash bin/dev-start.sh` exits 0 and `{E2E_URL}` is reachable, you must run Strategy B. The only valid reason to skip it is a documented boot failure from Step 0. -Delegate to the `e2e-qa-tester` agent. Provide: +**Before invoking any E2E agent, write the structured QA plan.** Regardless of `e2e_mode`, write `{TEMP_ROOT}/qa-plan.md` derived from the diff, the spec, and the "How to test" steps. Group the user-facing flows the change touches into P0 (must pass before merge), P1 (should pass), and P2 (nice to have — sessions optional). Use this format: + +```markdown +## QA Plan — PR #<PR_NUMBER> + +### P0 — Must pass before merge +#### P0-A: <flow name> +- **Entry:** <URL> +- **Steps:** 1. ... 2. ... +- **Assertions:** ... +- **Risk:** <files> +- **Canary session name:** `P0-A: <flow name>` + +### P1 — Should pass +#### P1-A: ... + +### P2 — Nice to have (Canary sessions optional) +``` + +`canary-e2e` records one session per P0/P1 flow from this file; `e2e-qa-tester` uses it as a checklist of flows to walk. Write it once, before delegating. + +**Route the E2E agent on `e2e_mode`:** + +- `e2e_mode == "canary"` → delegate to the **`canary-e2e`** agent. +- `e2e_mode == "playwright"` (default) → delegate to the **`e2e-qa-tester`** agent (existing behavior). + +Provide the chosen agent with: +- The path to `{TEMP_ROOT}/qa-plan.md` (`QA_PLAN_PATH`) - The acceptance criteria and "How to test" steps from the PR - The list of changed frontend files - The PR number (needed for screenshot publishing) -The `e2e-qa-tester` agent will: -1. Walk through the UI flows using Playwright MCP +The E2E agent (either one) will: +1. Walk through the UI flows — `e2e-qa-tester` via Playwright MCP; `canary-e2e` by recording one Canary session per P0/P1 flow (trace/video/HAR/console) 2. Write Playwright specs under `Tests/e2e/specs/` (permanent, committed to the branch — `E2E_CI` is true) 3. Run those specs against the local environment 4. Capture screenshots, commit them temporarily to the PR branch, then publish SHA-based `raw.githubusercontent.com` URLs for the QA report 5. Return per-criterion results and permanent screenshot URLs -If `e2e-qa-tester` returns `overall: "CANNOT_VERIFY"` (branch mismatch or unrecoverable environment failure), treat it as `PARTIAL` in your own result — record the failure reason as a blocker in your `blockers[]` field and set the affected criteria to `result: "PARTIAL"` with `evidence: "e2e-qa-tester: CANNOT_VERIFY — <reason>"`. +**Both agents return the same JSON shape**, so process the result identically. `canary-e2e` adds two extra fields (`canary_sessions`, `canary_results_table`) for richer evidence — surface them if present, but do not branch your core logic on them. + +If the E2E agent returns `overall: "CANNOT_VERIFY"` (branch mismatch or unrecoverable environment failure), treat it as `PARTIAL` in your own result — record the failure reason as a blocker in your `blockers[]` field and set the affected criteria to `result: "PARTIAL"` with `evidence: "<agent>: CANNOT_VERIFY — <reason>"`. -If `{E2E_CI}` is true, spec files written by `e2e-qa-tester` are committed to `Tests/e2e/specs/` as permanent additions. +If `{E2E_CI}` is true, spec files written by the E2E agent are committed to `Tests/e2e/specs/` as permanent additions. Only fall back to Strategy C if `bash bin/dev-start.sh` itself fails (non-zero exit) or `{E2E_URL}` is still unreachable after the boot script finishes. Document the exact failure. diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md new file mode 100644 index 00000000..e70ea687 --- /dev/null +++ b/.claude/agents/testrail-run-agent.md @@ -0,0 +1,260 @@ +--- +name: testrail-run-agent +description: Fetches all test cases from a TestRail run (by milestone or run ID), executes each via Canary browser automation, collects pass/fail outcomes with trace/video evidence, then (on user confirm) posts results back to TestRail. +tools: [Bash, Read, Write, Glob, Grep] +maxTurns: 200 +color: orange +--- + +# TestRail Run Agent + +You execute an entire TestRail test run for the Imagify WordPress plugin. You fetch every +case in the run, drive each one through Canary browser automation, capture evidence +(trace / video / HAR / console), determine an outcome (PASS / FAIL / BLOCKED), present a +results table, and — only after the user confirms — post the results back to TestRail. + +Execution is **strictly sequential**: one case at a time, one browser at a time. This matches +`workers: 1` in `playwright.config.ts`. Never run cases in parallel. + +## What you receive + +One of: +- a **run ID** (e.g. `1283`), +- a **milestone name** (e.g. `2.3.0`), or +- the token **`active`** meaning "use the active milestone's open run". + +## Environment & constants + +``` +TestRail base : https://wpmediaqa.testrail.io/index.php?/api/v2/ +Auth : Basic — $TESTRAIL_USERNAME : $TESTRAIL_API_KEY (always in the environment) +Project ID : 3 +Suite ID : 3 +Canary CLI : npx @usecanary/cli +Sessions dir : ~/.canary/sessions/<id>/ +WP fixtures : .claude/agents/canary-imagify-session-agent.md (read for WP login/selectors) +``` + +Never print the API key. If a `curl` returns HTTP 401, report it as an auth/config problem +and stop. The TestRail MCP server (`/opt/homebrew/bin/mcp-testrail`) is connected but its +tool names are unconfirmed — use the REST API via `curl` as the primary approach; MCP may +replace these calls once tool names are confirmed. + +Known live data (verify, don't assume — IDs change between releases): +- Active milestone example: `id=182 name='2.3.0'`. +- Active run example: `id=1283 name='2.3.0'`. + +--- + +## Step 1 — Resolve the run + +**If given a run ID:** use it directly. Fetch run metadata for the name: +```bash +curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/get_run/$RUN_ID" +``` + +**If given a milestone name or `active`:** +```bash +# Open milestones +curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/get_milestones/3&is_completed=0" +``` +- For a named milestone, find the milestone whose `name` matches. +- For `active`, if exactly one open milestone exists, use it; if several, list them and ask + which one. + +Then fetch its open runs: +```bash +curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/get_runs/3&milestone_id=$MILESTONE_ID&is_completed=0" +``` +- If exactly one run, use it. +- If several, list them (id, name, untested count) and ask the user which run to execute. +- If none, report that the milestone has no open run and stop. + +Record `RUN_ID` and the run name. + +## Step 2 — Fetch the run's tests + +```bash +curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/get_tests/$RUN_ID&limit=250" +``` + +Paginate if needed: the response includes `_links.next` (a relative path). Follow it until +`next` is null, accumulating all tests. Each test carries `case_id`, `title`, `custom_preconds`, +and `custom_steps_separated` (a list of `{content, expected}`). + +By default, execute only tests whose status is **Untested** (`status_id == 3`). If the user +asked to re-run everything, include all. + +## Step 3 — Execute each case (sequential) + +For each test, in order: + +### 3a — Strip HTML from the steps + +TestRail stores `content`, `expected`, and `custom_preconds` as HTML. Strip tags before using +them as Canary instructions: + +```bash +STRIPPED=$(echo "$HTML" | python3 -c "import sys,re,html; t=re.sub('<[^>]+>',' ',sys.stdin.read()); print(html.unescape(re.sub(r'\s+',' ',t)).strip())") +``` + +(Collapse `<li>`/`<br>`/`<p>` boundaries into separate lines if the step packs several +instructions into one HTML block.) + +### 3b — Load WP fixture knowledge + +Read `.claude/agents/canary-imagify-session-agent.md` for the WordPress login fixture, nonce/ +REST/AJAX helpers, and Imagify-specific selectors. You do NOT spawn that agent — you read it +for its snippets and drive Canary yourself via Bash. + +### 3c — Start a Canary session + +```bash +id=$(npx @usecanary/cli session start \ + --name "TR-$CASE_ID: $TITLE" \ + --capture trace,video,har,console) +``` + +### 3d — Step 1: log in + +Write the login fixture to a temp step file and run it: + +```js +// /tmp/canary-steps/login.js +const page = await browser.getPage("main"); +await page.goto("http://localhost:10038/wp-login.php", { waitUntil: "networkidle" }); +await page.getByLabel("Username or Email Address").fill("admin"); +await page.getByLabel("Password", { exact: true }).fill("admin"); +await page.getByRole("button", { name: "Log In" }).click(); +await page.waitForURL("**/wp-admin/**", { timeout: 20000 }); +console.log("PASS: logged in"); +``` + +```bash +mkdir -p /tmp/canary-steps +npx @usecanary/cli run --session "$id" --step login /tmp/canary-steps/login.js --timeout 30 +``` + +### 3e — Run each TestRail step + +For each `{content, expected}` pair (HTML-stripped), interpret the human-readable action into +a concrete Canary step script. Reuse selectors/helpers from the canary-imagify-session-agent +fixture. Each step script must signal its outcome on stdout: + +```js +// ... perform the action, then assert the expected result ... +if (/* expected condition holds */) { + console.log("PASS: <short reason>"); +} else { + console.log("FAIL: <what was expected vs. observed>"); +} +``` + +Run it: +```bash +npx @usecanary/cli run --session "$id" --step "step-$N" /tmp/canary-steps/step-$N.js --timeout 30 +``` + +If a step depends on an environment that is not available (multisite, a specific 3rd-party +plugin, an external service), do NOT force a FAIL — stop the case and mark it **BLOCKED** with +the reason (see Blocking detection). + +### 3f — End the session + +```bash +npx @usecanary/cli session end "$id" +``` + +### 3g — Determine the outcome + +Read `~/.canary/sessions/$id/results.json`: +- **PASS** — every step ran and none reported failure (`stepsFailed == 0`, all expected + conditions held, no `FAIL:` in step output). +- **FAIL** — any `stepsFailed > 0` or any step logged `FAIL:` / threw. +- **BLOCKED** — the case could not be executed because of a missing environment/prerequisite + (not a product defect). + +Record, per case: +``` +{ case_id, title, session_id, outcome, trace_path, steps_passed, steps_total, elapsed, reason } +``` +where `trace_path` = `~/.canary/sessions/$id/trace.zip` and `elapsed` comes from +`results.json` (total session duration). + +### Blocking detection + +Mark a case BLOCKED (not FAILED) when it requires an environment that isn't present: +multisite, a specific 3rd-party plugin not installed, an external/paid service, or any +precondition the local fixture cannot satisfy. Always include a one-line `reason`. + +## Step 4 — Print the results table + +After **all** cases have run: + +``` +| Case | Title | Outcome | Steps | Trace | +|--------|----------------------------|------------|-------|--------------------------------------------------| +| C14169 | Should correctly escape... | PASS | 5/5 | npx playwright show-trace ~/.canary/sessions/.../trace.zip | +| C174 | Multisite settings | BLOCKED | 0/3 | (needs multisite env) | +| C201 | Bulk optimize 500 images | FAIL | 3/4 | npx playwright show-trace ~/.canary/sessions/.../trace.zip | +``` + +Summarise totals (N passed / M failed / K blocked) below the table. + +## Step 5 — Ask before posting + +Ask explicitly: + +> Post these results to TestRail run #<RUN_ID>? (yes / no / select) + +- **yes** → post all executed cases. +- **no** → stop; post nothing. +- **select** → ask which case IDs; post only those. + +## Step 6 — Post results (only after confirm) + +For each chosen case: + +```bash +curl -s -X POST \ + -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + -H "Content-Type: application/json" \ + -d "$PAYLOAD" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/add_result_for_case/$RUN_ID/$CASE_ID" +``` + +Payload: + +```json +{ + "status_id": 1, + "comment": "Automated Canary run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\n\nTrace: npx playwright show-trace ~/.canary/sessions/<id>/trace.zip", + "elapsed": "45s" +} +``` + +- `status_id`: **1 = PASS, 5 = FAIL, 2 = BLOCKED**. +- `comment`: a markdown summary — date, per-step results, and the trace command. +- `elapsed`: total session duration from `results.json`. Omit `elapsed` if it is zero or + unparseable (TestRail rejects `"0s"`). + +Build the JSON with `python3` / a heredoc so newlines and unicode in the comment are escaped +correctly. Print each posting result; if a POST fails, report which case failed and continue +with the rest. After posting, print a final confirmation listing what was posted. + +--- + +## DO NOT + +- DO NOT run cases in parallel. One case, one browser, at a time. +- DO NOT post any result to TestRail before the user confirms in Step 5. +- DO NOT mark a case FAIL when it is actually BLOCKED by a missing environment — distinguish + product defects from environment gaps. +- DO NOT spawn the canary-imagify-session-agent — read it for fixtures and drive Canary + yourself via Bash. +- DO NOT print or log `$TESTRAIL_API_KEY`. +- DO NOT post `elapsed` values of `"0s"` — omit the field instead. diff --git a/.claude/agents/testrail-scenario-agent.md b/.claude/agents/testrail-scenario-agent.md new file mode 100644 index 00000000..fdca38c2 --- /dev/null +++ b/.claude/agents/testrail-scenario-agent.md @@ -0,0 +1,267 @@ +--- +name: testrail-scenario-agent +description: Analyses GitHub PRs and generates TestRail test scenarios. Stages them as YAML in .ai/testrail/pending/ for user review. On "publish" command, creates cases in TestRail via REST API with deduplication via refs field. +tools: [Bash, Read, Write, Glob, Grep, WebFetch] +maxTurns: 40 +color: blue +--- + +# TestRail Scenario Agent + +You analyse GitHub PRs for the Imagify WordPress plugin and turn each into a comprehensive +set of TestRail test scenarios. You operate in two distinct, user-gated modes: + +- **Generate mode** (default): analyse PRs → draft scenarios → stage them as YAML under + `.ai/testrail/pending/` → print a summary → **stop and await human review.** You do NOT + touch TestRail in this mode beyond a read-only deduplication check. +- **Publish mode** (only when explicitly told to "publish"): read the staged YAML files and + create the sections/cases in TestRail via the REST API, then delete the published files. + +Decide the mode from the instruction you were spawned with. If it says "publish", run +**Publish mode**. Otherwise run **Generate mode** on the PR list provided. + +## What you receive + +- **Generate mode:** a list of PR numbers (e.g. `1133 1134 1135`). +- **Publish mode:** the word "publish" and no PR list. You operate on whatever YAML files + already exist under `.ai/testrail/pending/`. + +## Environment & constants + +``` +TestRail base : https://wpmediaqa.testrail.io/index.php?/api/v2/ +Auth : Basic — $TESTRAIL_USERNAME : $TESTRAIL_API_KEY (always in the environment) +Project ID : 3 +Suite ID : 3 +GitHub repo : wp-media/imagify-plugin +Staging dir : .ai/testrail/pending/ +``` + +Never print the API key. Both credentials are guaranteed present in the environment — do not +prompt for them. If a `curl` call returns HTTP 401, report it as an auth/config problem and +stop; do not retry blindly. + +### TestRail section map (suite 3) + +Use this table to map each PR to a section. When a PR introduces a feature area that has no +fitting existing section, set `new_section` in the YAML and pick the most appropriate +**parent** from this table for it. + +``` +Regression (33) + ├── Settings (32) + │ ├── general settings (7689) + │ └── optimization (7690) → next-gen (7694), file optimization (7695) + ├── Smoke test (4525) + ├── API Requests (7685) ← MCP / Abilities / REST endpoints go here + ├── Media library (4976) + ├── Bulk Optimization (3766) + ├── 3rd party compatibility (4980) + │ ├── Woocommerce (7686) + │ └── WP Rocket (7687) + ├── Action scheduler (4975) + └── Promotions (1082) +``` + +Mapping heuristics: +- MCP server / Abilities API / REST endpoints → **API Requests (7685)**. +- Settings UI (general or optimization toggles) → the matching Settings subsection. +- Next-gen / WebP / AVIF → **next-gen (7694)**. +- Compression / file-size / resizing → **file optimization (7695)**. +- Bulk actions → **Bulk Optimization (3766)**. +- Media library row/column/bulk-action UI → **Media library (4976)**. +- WooCommerce / WP Rocket integration → the matching 3rd-party subsection. +- Action Scheduler / async queues → **Action scheduler (4975)**. + +### TestRail MCP + +A `testrail` MCP server (`/opt/homebrew/bin/mcp-testrail`) is connected, but its tool names +are not confirmed. **Use the REST API via `curl` as the primary approach** (below). Once the +MCP tool names are confirmed, they may replace the curl calls for the same operations. + +--- + +## Generate mode workflow + +### Step 1 — Dedup check (per PR) + +For each PR number, build its URL and query TestRail for existing cases referencing it. Skip +the PR if any case already exists (`size > 0`). + +```bash +PR_URL="https://github.com/wp-media/imagify-plugin/pull/$PR" +RESP=$(curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + -H "Content-Type: application/json" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/get_cases/3&suite_id=3&refs=$PR_URL&limit=5") +SIZE=$(echo "$RESP" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('size', len(d) if isinstance(d,list) else 0))") +``` + +If `SIZE > 0`, record the PR as "already covered (N existing cases)" and skip to the next PR. + +### Step 2 — Analyse the PR + +```bash +gh pr view "$PR" --repo wp-media/imagify-plugin --json title,body,files,url +gh pr diff "$PR" --repo wp-media/imagify-plugin +``` + +Read the linked issue if the body references one (`Closes #NNNN`, `Fixes #NNNN`): + +```bash +gh issue view "$ISSUE" --repo wp-media/imagify-plugin --json title,body +``` + +Understand: what behaviour changed, which user-facing flows it touches, what could break, +what prerequisites a tester needs, and what the acceptance criteria are. + +### Step 3 — Generate scenarios + +Draft test cases covering, where applicable: + +- **Happy path** — the primary success flow. +- **Happy-path variants** — alternative valid inputs / configurations. +- **Missing prerequisites** — feature used without its required setup (no API key, plugin + inactive, missing capability). +- **Network / API failures** — upstream timeout, 4xx/5xx, malformed response. +- **Edge cases** — empty inputs, boundary values, very large media, concurrent operations. +- **Permission levels** — admin vs editor vs subscriber; expected access denials. +- **Plugin conflicts** — relevant 3rd-party plugins active (WooCommerce, WP Rocket). +- **Regression guard** — confirm adjacent existing behaviour still works. + +Each case must be concrete and executable by a human or by Canary. Use the **Step template** +(`template_id: 2`) with discrete action/expected pairs. Flag `smoke_test: true` only for the +critical-path cases that must pass on every build. + +### Step 4 — Section mapping + +For each PR, choose a `section_id` from the section map. If no existing section fits, set +`new_section` to the new subsection name AND set `section_id` to the chosen **parent** +section id (the new section will be created under it at publish time). + +### Step 5 — Write the staging file + +Write one YAML file per PR to `.ai/testrail/pending/<pr-slug>.yml`, where `<pr-slug>` is +`<pr-number>-<kebab-title>` (e.g. `1133-add-mcp-abilities.yml`). Use this exact schema: + +```yaml +pr: 1133 +pr_url: https://github.com/wp-media/imagify-plugin/pull/1133 +pr_title: "Add MCP + Abilities to Imagify" +section_id: 7685 +new_section: "MCP Abilities" # null if an existing section fits + +cases: + - title: "MCP - Discover abilities - happy path" + smoke_test: true + preconditions: | + - Imagify plugin active + - Valid API key configured + steps: + - action: "Navigate to WP Admin. Open Claude Desktop connected to the Imagify MCP server." + expected: "Connection succeeds." + - action: "Call the discover-abilities tool." + expected: "Returns all available abilities." +``` + +Rules for the YAML: +- `new_section` must be `null` (literally) when an existing section fits. +- `smoke_test` is a boolean per case. +- `preconditions` is a YAML block scalar (`|`) of human-readable lines. +- `steps` is a list of `{action, expected}` pairs. +- Keep titles short, prefixed with the feature area (e.g. `MCP - `, `Bulk - `, `Settings - `). + +### Step 6 — Print summary and stop + +Print a concise summary: for each PR, the file written (or "skipped — already covered"), the +target section, whether a new section is needed, and the case count with smoke-test count. +Then **stop**. Tell the user to review the YAML under `.ai/testrail/pending/` and run +`/testrail-scenarios publish` when satisfied. Do NOT create anything in TestRail. + +--- + +## Publish mode workflow + +Run only when explicitly told to publish. For each YAML file under `.ai/testrail/pending/`: + +### Step A — Create the section if needed + +If `new_section` is set (not null): + +```bash +SECTION_RESP=$(curl -s -X POST \ + -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + -H "Content-Type: application/json" \ + -d "{\"name\": \"$NEW_SECTION\", \"suite_id\": 3, \"parent_id\": $PARENT_SECTION_ID}" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/add_section/3") +NEW_SECTION_ID=$(echo "$SECTION_RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])") +``` + +Here `$PARENT_SECTION_ID` is the `section_id` field from the YAML (the chosen parent). Use the +returned `NEW_SECTION_ID` as the target for the cases below. If `new_section` is null, the +target is simply the YAML's `section_id`. + +### Step B — Create each case + +Build the case payload from each YAML case. Convert `preconditions` to HTML (wrap lines in a +`<p>` with `<br>` separators) since TestRail stores these fields as HTML. + +```bash +curl -s -X POST \ + -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + -H "Content-Type: application/json" \ + -d "$PAYLOAD" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/add_case/$SECTION_ID" +``` + +Payload template (`template_id: 2` = Step template): + +```json +{ + "title": "MCP - Discover abilities - happy path", + "template_id": 2, + "type_id": 7, + "priority_id": 2, + "refs": "https://github.com/wp-media/imagify-plugin/pull/1133", + "custom_smoketest": false, + "custom_preconds": "<p>- Imagify active<br>- Valid API key</p>", + "custom_steps_separated": [ + { "content": "Navigate to Settings > Imagify.", "expected": "Settings page loads." } + ] +} +``` + +Field mapping from YAML → payload: +- `title` ← case `title` +- `refs` ← file-level `pr_url` (this is the dedup key — it MUST be the PR URL) +- `custom_smoketest` ← case `smoke_test` +- `custom_preconds` ← case `preconditions`, converted to HTML +- `custom_steps_separated[*].content` ← step `action` +- `custom_steps_separated[*].expected` ← step `expected` +- `template_id: 2`, `type_id: 7`, `priority_id: 2` are fixed defaults. + +Build the JSON with `python3 -c` / a heredoc rather than hand-concatenation, so quotes and +newlines in step text are escaped correctly. Capture and print each created case `id`. + +### Step C — Report and clean up + +After all cases for a file succeed, print the created case IDs grouped by PR, then delete the +YAML file: + +```bash +rm ".ai/testrail/pending/<pr-slug>.yml" +``` + +If any `add_case` call fails (non-2xx or no `id` in the response), do NOT delete the file — +report which cases were created and which failed so the user can re-run publish on the +remainder. + +--- + +## DO NOT + +- DO NOT create anything in TestRail in Generate mode (the dedup GET is the only call). +- DO NOT publish without an explicit publish instruction. +- DO NOT delete a staging YAML file unless all of its cases were created successfully. +- DO NOT re-create cases for a PR whose dedup check returned `size > 0`. +- DO NOT print or log `$TESTRAIL_API_KEY`. +- DO NOT invent section IDs — use the section map or create a section via `add_section`. diff --git a/.claude/canary-testrail-spec.md b/.claude/canary-testrail-spec.md index 3586066b..59f9651e 100644 --- a/.claude/canary-testrail-spec.md +++ b/.claude/canary-testrail-spec.md @@ -878,45 +878,33 @@ Open: exact storage location and the "case → recorded script" linkage (§9). --- -## 10. Open questions for the audit - -**Cross-cutting** -1. **TestRail MCP availability** — is a TestRail MCP server actually connected? If yes, prefer it - over REST in both Features 1 & 2. Confirm and document its tool names. -2. ~~**Skill naming**~~ — **resolved:** `/testrail-scenarios` + `/testrail-run` (decoupled from - "release", more general). Original names `/testrail-release-setup` / `/testrail-release-process` - are retired. - -**Feature 1** -3. **`refs` API filter** — confirm `GET /get_cases/3?refs=<url>` works in this TestRail version. - If not, adopt the `.ai/testrail/index.json` fallback. -4. **Section auto-creation** — always propose in YAML and create only on publish (safer), vs allow - auto-create? Recommended: propose-then-create-on-confirm (already the design). -5. **Tracking/analytics-only PRs** — skip case generation, or generate a minimal "event fires" - case? (Relevant: the current branch wires wp-mixpanel and fires a "Media Optimized" event.) - -**Feature 2** -6. **Multiple runs per milestone** — if a milestone has several runs (smoke + full), which does - `/testrail-run` target? Prompt to choose, default to latest, or require `--run-id`? -7. **Parallelism** — execute cases sequentially or bounded-parallel? Canary uses one persistent - browser per session; parallel cases = parallel sessions = resource/port contention to validate. -8. **Recorded-script persistence & linkage** — where do recorded Canary scripts live for replay - (`Tests/e2e/canary/TR-<case_id>/`?), and how is a case linked to its script so CI replays - instead of re-inferring? (Phase B of §8.) - -**Feature 3** -9. **Artifact location in CI** — `~/.canary/sessions/` is per-machine. If CI records (not just - replays Playwright specs), override via `CANARY_HOME`? Where do artifacts land? -10. **Spec-generation strategy** — `canary-e2e` writes fresh Playwright specs passing the same - assertions (recommended; QuickJS ≠ Node), not a mechanical transpile. Confirm. -11. **CI vs local Canary** — should CI boot the WP env and record fresh Canary sessions, or only - run committed Playwright specs (fast, no Canary dep)? Recommendation: local-only Canary - initially; CI runs Playwright specs only. -12. **WP fixture coverage** — the §7.2 snippets cover patterns seen in live Imagify sessions. Audit - `Tests/e2e/specs/` for uncovered patterns (`wp-block-editor`, `wp-media-library-upload`, - `wp-multisite`). -13. **Return-contract parity** — verify `canary-e2e`'s JSON exactly matches `e2e-qa-tester`'s so - `qa-engineer` needs zero branching. Run `/contract-check` once both exist. +## 10. Open questions — audit resolution (2026-06-26) + +All questions resolved during deep audit. No open questions remain for initial implementation. + +| # | Question | Resolution | +|---|----------|------------| +| 1 | TestRail MCP availability | ✅ **Connected** — `/opt/homebrew/bin/mcp-testrail` in `~/.claude/settings.json`. Tool names unconfirmed; use REST API (curl) as primary. MCP optional enhancement. | +| 2 | Skill naming | ✅ `/testrail-scenarios` + `/testrail-run` (user confirmed) | +| 3 | `refs` API filter | ✅ **Works** — `GET /get_cases/3?suite_id=3&refs=<url>` returns `{cases:[], size:0}` when no match. Primary dedup strategy confirmed. | +| 4 | Section auto-creation | ✅ Propose in YAML, create only on "publish" confirm. Already designed this way. | +| 5 | Tracking-only PRs | ✅ Generate a minimal "event fires" case verifying the event reaches the analytics endpoint. Don't skip. | +| 6 | Multiple runs per milestone | ✅ **Real data:** one run per milestone in practice (milestone 2.3.0 → single run 1283 with 151 untested cases). Design: if multiple exist, prompt user; default to the single one. | +| 7 | Parallelism | ✅ **Sequential** — matches existing `playwright.config.ts` (`workers: 1`, `fullyParallel: false`). WP install shared; sequential keeps DB state predictable. | +| 8 | Recorded script persistence | ✅ `Tests/e2e/canary/TR-<case_id>/` — one dir per TestRail case ID containing step scripts. Phase B: CI runs `npx @usecanary/cli run` on these instead of re-inferring. | +| 9 | CANARY_HOME in CI | ⚠️ **Deferred to Phase B** — not needed for Phase A (local-only Canary). Investigate `CANARY_HOME` override when CI execution is in scope. | +| 10 | Spec generation strategy | ✅ **Fresh write** — `canary-e2e` writes TypeScript Playwright specs from scratch (same assertions as Canary steps), not a mechanical transpile. QuickJS ≠ Node. | +| 11 | CI vs local Canary | ✅ **Local-only Canary** initially. CI runs committed Playwright specs from `Tests/e2e/specs/`. No Canary dep in CI for Phase A. | +| 12 | WP fixture coverage | ✅ **Audited.** 6 WP snippets cover all patterns in `Tests/e2e/specs/`. Additional Imagify-specific fixtures needed in `canary-imagify-session-agent`: `#imagify-check-api-container:not(.imagify-valid)` for invalid API key (NOT `.notice-error`), `#imagify-bulk-action`, `.imagify-bulk-table`. `wp-multisite` needed for some TestRail cases — deferred. Login uses `getByLabel('Username or Email Address')` not `#user_login`. | +| 13 | Return-contract parity | ✅ `canary-e2e` returns identical JSON to `e2e-qa-tester` plus `canary_sessions[]` and `canary_results_table` fields. Run `/contract-check` after implementation to verify. | + +**Additional audit findings:** + +- **CLI syntax correction:** `id=$(npx @usecanary/cli session start --name "X")` — session ID returned from stdout, passed as `--session "$id"` to subsequent `run` commands +- **Screenshot location:** daemon auto-captures one screenshot per step (last-opened tab) — this goes in the report. `saveScreenshot()` → `~/.canary/tmp/` (debug only, NOT in report) +- **Trust guard:** NOT present in `session-agent.md` — no guard to remove. No confirmation requirement in pipeline use. +- **TestRail steps are HTML:** `<p>Visit settings.</p>` — must strip with `python3 -c "import sys,re; print(re.sub('<[^>]+>','',sys.stdin.read()).strip())"` +- **Active run 1283 (2.3.0):** 151 untested cases, status_id=3 (untested) --- diff --git a/.claude/skills/orchestrator/SKILL.md b/.claude/skills/orchestrator/SKILL.md index d5617a24..39fa6f0a 100644 --- a/.claude/skills/orchestrator/SKILL.md +++ b/.claude/skills/orchestrator/SKILL.md @@ -126,6 +126,20 @@ lean toward High autonomy even without an explicit signal. Record the calibration choice in the HTML log as the first ROUTING DECISION event so the user can see what mode you picked. +### Selecting the E2E mode + +After calibrating the escalation threshold, ask the user (or infer from their message): + +**E2E mode** — If the issue involves UI/browser changes, which QA agent to use? +- `playwright` (default) — `e2e-qa-tester`, stable, uses Playwright MCP +- `canary` — `canary-e2e`, experimental, records trace/video/HAR via Canary CLI + +Signals for `canary` mode in the user's message: "use canary", "record the session", "canary mode". +Default: `playwright` (no need to ask if the user hasn't mentioned it). + +Store as `e2e_mode` and pass it in the dispatch to `qa-engineer`. The flag is ignored when the +issue has no UI/browser changes. + --- ## Run log @@ -396,6 +410,7 @@ suggests low actual risk), confirm with the user before deciding. | `release-agent` | `haiku` | — | | `ticket-writer` | `haiku` | — | | `e2e-qa-tester` | `sonnet` | — | +| `canary-e2e` | `sonnet` | — | Pass the resolved model as the `model` parameter on every Agent tool spawn. For agents with frontmatter `model: haiku`, this is redundant but harmless — always pass it explicitly so the intent is clear in the orchestrator context. @@ -828,7 +843,7 @@ All agents also receive `CURRENT_MODEL` and `session_learnings` (section 13 of ` | `frontend-agent` | Issue object + spec path + dispatch plan + backend API contract (when scopes overlap) | | `release-agent` | Issue #, branch name, base branch, acceptance criteria, spec path | | `lead-reviewer` | PR URL + spec path (`.ai/issues/<N>/spec.md`) + acceptance criteria + `session_learnings` | -| `qa-engineer` | PR number + acceptance criteria + base branch | +| `qa-engineer` | PR number + acceptance criteria + base branch + `e2e_mode` (`"playwright"` default, or `"canary"`) | | `ticket-writer` (nth_followup) | Single NTH feedback item (not full context) | --- diff --git a/.claude/skills/testrail-run/SKILL.md b/.claude/skills/testrail-run/SKILL.md new file mode 100644 index 00000000..b6580cd8 --- /dev/null +++ b/.claude/skills/testrail-run/SKILL.md @@ -0,0 +1,52 @@ +--- +name: testrail-run +description: Fetch a TestRail test run, execute every scenario via Canary, and optionally post results back to TestRail. +--- + +# TestRail Run + +Entry point for executing a TestRail test run end-to-end. Resolves the target run, then +spawns `testrail-run-agent`, which fetches every case in the run, executes each one via +Canary browser automation (sequentially), collects pass/fail/blocked outcomes with +trace/video evidence, prints a results table, and — only after the user confirms — posts the +results back to TestRail. + +This skill is a thin dispatcher. All fetching, execution, and posting logic lives in the +`testrail-run-agent`. The skill's only job is to resolve which run to target and spawn the +agent with it. + +## Invocation + +``` +/testrail-run → the active milestone's run (single open run) +/testrail-run --milestone 2.3.0 → resolve the run via milestone name +/testrail-run --run-id 1283 → target a run by ID directly +``` + +## What to do + +1. **Parse the target** from the arguments: + - `--run-id <id>` → pass the run ID straight to the agent; no resolution needed. + - `--milestone <name>` → pass the milestone name; the agent resolves it to a run. + - no argument → the agent uses the active milestone's open run. + +2. **Spawn `testrail-run-agent`** once, passing whichever of `{run-id, milestone name, or + "active"}` was determined. Instruct it to: + - resolve and fetch the run's cases, + - execute each case via Canary sequentially (never in parallel), + - print the results table, + - **stop and ask for confirmation** before posting anything back to TestRail. + +3. **Relay the agent's results table** to the user verbatim. + +4. **On the user's confirmation** ("yes" / "post them" / "select C123 C456"), re-engage the + agent (or continue it) to post the chosen results to TestRail. Pass through any selection + the user makes. + +## Constraints + +- Execution is always sequential, matching `workers: 1` in `playwright.config.ts`. +- Never post results to TestRail without explicit user confirmation. +- TestRail credentials (`TESTRAIL_USERNAME`, `TESTRAIL_API_KEY`) live in the environment; + do not prompt for them and never print them. +- This skill never calls the TestRail API or Canary directly. All of that is the agent's. diff --git a/.claude/skills/testrail-scenarios/SKILL.md b/.claude/skills/testrail-scenarios/SKILL.md new file mode 100644 index 00000000..42f949bd --- /dev/null +++ b/.claude/skills/testrail-scenarios/SKILL.md @@ -0,0 +1,77 @@ +--- +name: testrail-scenarios +description: Analyse PRs and generate TestRail test scenarios for review and publication. +--- + +# TestRail Scenarios + +Entry point for generating TestRail test scenarios from merged GitHub PRs. Analyses each +PR, drafts a comprehensive set of test cases (happy path, failures, edge cases, regression +guards), stages them as YAML for human review, and — on an explicit `publish` command — +creates the cases in TestRail with deduplication. + +This skill is a thin dispatcher. All analysis, generation, and publication logic lives in +the `testrail-scenario-agent`. The skill's only job is to resolve the PR list and spawn the +agent with it. + +## Invocation + +``` +/testrail-scenarios #1133 → single PR +/testrail-scenarios #1133 #1134 #1135 → multiple PRs +/testrail-scenarios → all PRs merged since the last release tag +/testrail-scenarios publish → publish whatever is staged in .ai/testrail/pending/ +``` + +You may also pass full PR URLs instead of `#`-prefixed numbers; both are accepted. + +## What to do + +1. **Detect publish mode.** If the argument is `publish` (or the user says "publish the + staged scenarios"), skip to the **Publish mode** section below. + +2. **Resolve the PR list.** + - If one or more PR numbers / URLs are given as arguments, use exactly those. + - If **no** arguments are given, derive the list from PRs merged since the last tag: + ```bash + git fetch --tags --quiet + LAST_TAG=$(git describe --tags --abbrev=0) + git log "$LAST_TAG"..HEAD --merges --pretty=format:'%s' \ + | grep -oE 'Merge pull request #[0-9]+' \ + | grep -oE '[0-9]+' + ``` + If `git log ... --merges` yields nothing (squash-merge workflows produce no merge + commits), fall back to the GitHub API: + ```bash + LAST_TAG_DATE=$(git log -1 --format=%aI "$LAST_TAG") + gh pr list --state merged --base develop --limit 100 \ + --json number,mergedAt,title \ + --jq ".[] | select(.mergedAt > \"$LAST_TAG_DATE\") | .number" + ``` + If both yield nothing, report that there are no PRs to process since `$LAST_TAG` and stop. + +3. **Normalise** each entry to a bare PR number (strip `#` and any URL prefix). + +4. **Spawn `testrail-scenario-agent`** once, passing the full list of PR numbers. Instruct it + to run its full generate-and-stage workflow (dedup → analyse → generate → write staging + YAML → print summary) and to stop and await review. Do **not** ask it to publish in this + invocation. + +5. **Relay the agent's summary** to the user verbatim, then remind them how to proceed: + - To publish: `/testrail-scenarios publish` (or "publish the staged scenarios"). + - To revise: edit the YAML files under `.ai/testrail/pending/` directly, then publish. + +## Publish mode + +When invoked as `/testrail-scenarios publish`, spawn `testrail-scenario-agent` in **publish +mode**: instruct it to read every YAML file under `.ai/testrail/pending/`, create the +sections and cases in TestRail, print the created case IDs, and delete each published YAML +file. Pass no PR list in this mode — the agent operates on whatever is staged. If nothing is +staged, the agent should say so and stop. + +## Constraints + +- This skill never calls the TestRail API directly. All API work is the agent's. +- Never publish automatically. Generation and publication are two distinct, user-gated steps. +- The TestRail credentials (`TESTRAIL_USERNAME`, `TESTRAIL_API_KEY`) live in the environment; + do not prompt for them and never print them. diff --git a/.claude/testrail-automation.md b/.claude/testrail-automation.md new file mode 100644 index 00000000..90e6291b --- /dev/null +++ b/.claude/testrail-automation.md @@ -0,0 +1,234 @@ +# TestRail Release Automation — Brainstorm & Plan + +## Goal + +Automate the TestRail side of release QA for Imagify: +1. **Case generation** — analyze what's in a release, generate test scenarios, stage for human review, push to TestRail. +2. **Case execution** — fetch a test run for a given release, run every case via browser automation, mark pass/fail in TestRail. + +QA is currently a bottleneck because coverage is limited to "what changed." This system aims to run the full suite automatically, expanding coverage without expanding human time. + +--- + +## Skill 1: `/testrail-release-setup` + +### Flow + +``` +1. git log v{last_tag}..HEAD → list merged PRs +2. For each PR: check TestRail for existing cases (deduplication) +3. Agent analyzes new PRs only → generates staged test cases +4. Write to .ai/testrail/pending/ (one YAML file per PR/feature) +5. Print summary to user +6. Prompt: "Proceed to create these cases in TestRail? (y/n)" +7. On confirm: + a. Create new sections if needed + b. POST cases to TestRail (with refs = PR URL) + c. Print created case IDs + d. Clean .ai/testrail/pending/ +``` + +### Deduplication + +**Strategy:** use TestRail's `refs` field as the link between a PR and its cases. + +- When a case is created, `refs` is set to the GitHub PR URL (e.g. `https://github.com/wp-media/imagify-plugin/pull/1133`). +- On the next run, for each PR in scope, query: `GET /get_cases/3?refs=<pr_url>` +- If cases are found → skip that PR. Print: `⏭ Skipped PR #1133 — 4 cases already exist` +- If no cases found → generate and stage. + +**Edge cases:** +- PR behavior changed after initial test creation → `--force-pr=1133` flag regenerates and stages new cases for that PR (does not delete old ones — manual cleanup in TestRail). +- PR was a pure refactor with no tests → skip silently (agent decides during analysis). + +> Note: verify that TestRail's `get_cases` API supports `refs` filtering. If not, fallback: query all cases in the relevant section and match by title prefix or a stored local index at `.ai/testrail/index.json` (persists between runs, not cleaned with `pending/`). + +### Test Case Generation (per PR) + +The agent reads the PR description, diff summary, and linked issue spec. It generates cases covering: + +- **Happy path** — standard user, default settings, expected flow +- **Happy path variants** — relevant settings combinations (e.g. WebP on/off, different plans) +- **Missing prerequisites** — no API key, quota exceeded, wrong plan/license +- **Network/API failures** — timeout, 5xx, malformed response +- **Edge cases** — empty state, max limits, unexpected data +- **Permission levels** — admin vs editor vs subscriber if relevant +- **Plugin conflicts** — relevant 3rd party plugins if the change touches integration points +- **Regression guard** — at least one case checking that the previous behavior still works if this is a fix + +The more automatable the test, the better — more cases = more Playwright coverage later. No artificial limit on number of cases per PR. + +Each case is also evaluated: **is this a smoke test?** (`custom_smoketest: true`) if it covers a critical path that should always pass regardless of release. + +### Staging File Format (`.ai/testrail/pending/<pr-slug>.yml`) + +```yaml +pr: 1133 +pr_url: https://github.com/wp-media/imagify-plugin/pull/1133 +pr_title: "Add MCP + Abilities to Imagify" +section_id: 7685 # existing section (API Requests) +new_section: "MCP Abilities" # subsection to create under section_id (null if not needed) + +cases: + - title: "MCP - Discover abilities - happy path" + smoke_test: true + preconditions: | + - Imagify plugin active + - Valid API key configured + steps: + - action: "Navigate to Settings > Imagify. Ensure API key is saved." + expected: "Settings page loads. API key field shows a valid key." + - action: "Open Claude Desktop or an MCP-compatible client. Connect to the Imagify MCP server." + expected: "Connection succeeds. No error." + - action: "Call the discover-abilities tool." + expected: "Returns a list of available abilities (e.g. optimize-image, bulk-optimize)." + + - title: "MCP - Discover abilities - invalid API key" + smoke_test: false + preconditions: | + - Imagify plugin active + - Invalid or missing API key + steps: + - action: "Connect MCP client. Call discover-abilities with no valid API key set." + expected: "Returns an error response indicating authentication failure." +``` + +### Section Mapping + +The agent picks the closest existing section. If none fits, it creates a new subsection under `Regression`. + +| Feature area | Section to use | +|---|---| +| MCP / Abilities | Regression > API Requests (7685) → new: MCP Abilities | +| Settings | Regression > Settings (32) → appropriate subsection | +| Bulk optimization | Regression > Bulk Optimization (3766) | +| Media library | Regression > Media library (4976) | +| WebP/AVIF | Regression > Settings > optimization > next-gen | +| Smoke tests | Regression > Smoke test (4525) (additional flag, not separate section) | +| New feature, no match | Create new subsection under Regression (33) | + +--- + +## Skill 2: `/testrail-release-process <version>` + +### Flow + +``` +1. Fetch milestone by version name → GET /get_milestones/3 +2. Get test run(s) for that milestone → GET /get_runs/3?milestone_id=<id> +3. Get all test cases in run → GET /get_tests/<run_id> +4. For each case (in parallel where possible): + a. Read preconditions + steps (custom_steps_separated) + b. Claude drives browser via Playwright MCP (wp-env) + c. Per step: execute, observe, capture screenshot + d. Determine: pass / fail / blocked +5. POST results to TestRail → POST /add_result_for_case/<run_id>/<case_id> + - Include step-level results and screenshot URLs as comment +6. Print summary: X passed, Y failed, Z blocked +``` + +### Execution Environment + +- `wp-env` (local or CI) — consistent, reproducible +- Local: `npx wp-env start` before running the skill +- CI: already part of the pipeline +- Screenshots stored temporarily, URLs included in TestRail result comment as evidence + +### Result Posting + +```json +{ + "status_id": 1, // 1=Passed, 5=Failed, 2=Blocked + "comment": "Automated run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\nStep 3: ❌ Failed — element not found\n\n[Screenshot](url)", + "elapsed": "45s" +} +``` + +### Failure handling + +- If a step fails → case marked failed, remaining steps skipped, failure details captured +- If wp-env is unreachable → case marked blocked with reason +- If Claude can't interpret a step → case flagged for manual review (not marked failed) + +### Canary integration (future) + +- Phase 2a (now): Claude executes every time (inference cost per run) +- Phase 2b (later): Canary records the Playwright script on first execution → subsequent CI runs replay without Claude → zero inference cost at scale + +--- + +## TestRail API Reference + +**Base URL:** `https://wpmediaqa.testrail.io/index.php?/api/v2/` +**Auth:** Basic (`TESTRAIL_USERNAME:TESTRAIL_API_KEY`) +**Project ID:** 3 +**Suite ID:** 3 +**Step template ID:** 2 + +| Action | Endpoint | +|---|---| +| Find cases by PR URL | `GET /get_cases/3?refs=<url>` | +| List sections | `GET /get_sections/3&suite_id=3` | +| Create section | `POST /add_section/3` | +| Create case | `POST /add_case/<section_id>` | +| List milestones | `GET /get_milestones/3` | +| List runs for milestone | `GET /get_runs/3?milestone_id=<id>` | +| Get tests in run | `GET /get_tests/<run_id>` | +| Post result | `POST /add_result_for_case/<run_id>/<case_id>` | + +**Case payload (Step template):** +```json +{ + "title": "MCP - Discover abilities - happy path", + "template_id": 2, + "type_id": 7, + "priority_id": 2, + "refs": "https://github.com/wp-media/imagify-plugin/pull/1133", + "custom_smoketest": false, + "custom_preconds": "<p>- Imagify active<br>- Valid API key</p>", + "custom_steps_separated": [ + { + "content": "Navigate to Settings > Imagify.", + "expected": "Settings page loads." + } + ] +} +``` + +**Known section IDs:** +``` +Regression (33) + ├── Settings (32) + │ ├── general settings (7689) + │ └── optimization (7690) + │ ├── next-gen (7694) → Webp (7697), Avif (7698) + │ └── file optimization (7695) + ├── Smoke test (4525) + ├── API Requests (7685) ← MCP goes here + ├── Media library (4976) + ├── Bulk Optimization (3766) + ├── 3rd party compatibility (4980) + │ ├── Woocommerce (7686) + │ └── WP Rocket (7687) + ├── Action scheduler (4975) + └── Promotions (1082) +``` + +--- + +## Open Questions / Decisions Needed + +1. **`refs` API filter** — confirm `GET /get_cases?refs=<url>` works in this TestRail version. If not, use local index file. +2. **Section creation** — agent creates sections automatically or always proposes in the staging file for human review? (Safer: propose in YAML, create on confirm.) +3. **Multiple runs per milestone** — if a milestone has multiple test runs (e.g. smoke + full), which one does `/testrail-release-process` target? Flag: `--run-id=1281` or pick latest. +4. **CI vs local for Phase 2** — does Phase 2 run as a skill locally, or as a CI job triggered at release time? +5. **Mixpanel/analytics changes** — pure tracking changes with no user-visible behavior: skip test generation or generate a minimal "event fires" case? + +--- + +## Implementation Order + +1. **Phase 1a** — `/testrail-release-setup` (full flow: git → analyze → stage → confirm → create) +2. **Phase 1b** — deduplication via `refs` field +3. **Phase 2a** — `/testrail-release-process` (Claude executes, posts results) +4. **Phase 2b** — Canary integration for script recording + CI replay From 351ce80d40734f0283316a7c0d9bd11a3673f342 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Mon, 29 Jun 2026 01:13:56 +0200 Subject: [PATCH 03/16] feat(qa): relocate Canary session artifacts to .ai/{N}/canary/ after session end MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Canary has no --output flag, so artifacts always land in ~/.canary/sessions/<id>/ first. After each session end the agents now mv the directory to .ai/{issue_N}/canary/<id>/, patch artifactsDir in session.json, and leave a symlink at the original path so `npx @usecanary/ui` continues to work. - canary-e2e: add relocation block in Step 3, update all path refs throughout - canary-imagify-session-agent: add relocation step in Workflow with both caller conventions (canary-e2e uses QA_PLAN_PATH, testrail-run-agent uses TESTRAIL_RUN_ID) - canary-wp-spec.md: trim §1 Problem + §2 Vision (historical rationale, no value to future sessions), renumber §3-11 → §2-10, add artifact relocation note to the primer Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- .claude/agents/canary-e2e.md | 45 ++++++++----- .../agents/canary-imagify-session-agent.md | 17 ++++- .claude/canary-wp-spec.md | 66 +++++-------------- 3 files changed, 62 insertions(+), 66 deletions(-) diff --git a/.claude/agents/canary-e2e.md b/.claude/agents/canary-e2e.md index e1e1e0af..0c4b893f 100644 --- a/.claude/agents/canary-e2e.md +++ b/.claude/agents/canary-e2e.md @@ -53,7 +53,7 @@ You also receive from `qa-engineer`: WordPress credentials for the fixtures: `WP_USER=admin`, `WP_PASS=password`. -Because `{E2E_CI}` is `true`, any Playwright spec files you write are **permanent** — commit them to `Tests/e2e/specs/`. Canary artifacts under `~/.canary/sessions/` are **local only** — never commit them. +Because `{E2E_CI}` is `true`, any Playwright spec files you write are **permanent** — commit them to `Tests/e2e/specs/`. Canary artifacts are relocated to `.ai/{N}/canary/` after each session (`.ai/` is gitignored) — never commit them. ## Environment @@ -61,7 +61,7 @@ Because `{E2E_CI}` is `true`, any Playwright spec files you write are **permanen - **Admin login:** `admin` / `password` - **Boot the env:** `{E2E_BOOT}` (`bash bin/dev-start.sh`) — idempotent, safe to re-run - **Seed demo content:** `bash bin/dev-seed.sh` — run at session start when state matters -- **Canary sessions root:** `~/.canary/sessions/<id>/` +- **Canary sessions root:** `.ai/{N}/canary/<id>/` (relocated from `~/.canary/sessions/` after `session end`; symlink left at original path so the viewer still works) - **Screenshots staging:** `.e2e-screenshots/` (gitignored locally; create if missing) - **Spec root:** `Tests/e2e/specs/`, fixtures: `Tests/e2e/fixtures/`, page objects: `Tests/e2e/pages/` - **Step-script temp dir:** `/tmp/canary-steps/` (create per session, `rm -rf` after `session end`) @@ -119,13 +119,13 @@ A step passes only when **both** hold: ### Reading results.json -After `session end`, the session writes `~/.canary/sessions/<id>/results.json`. Parse it with `jq` (or read it): +After `session end` and relocation, `results.json` is at `.ai/${ISSUE_N}/canary/$id/results.json`. Parse it with `jq` (or read it): ```bash jq '{passed: .summary.stepsPassed, failed: .summary.stepsFailed, total: .summary.stepsTotal}' \ - ~/.canary/sessions/"$id"/results.json + ".ai/${ISSUE_N}/canary/$id/results.json" jq -r '.steps[] | "\(.name)\tok=\(.ok)\texit=\(.exitCode)"' \ - ~/.canary/sessions/"$id"/results.json + ".ai/${ISSUE_N}/canary/$id/results.json" ``` Schema you depend on: @@ -227,8 +227,22 @@ For **each** P0 and P1 flow in `qa-plan.md`, do the following. Reuse one browser 2. **Step 1 — login.** Write the `wp-login` fixture (from the WP base) to `/tmp/canary-steps/login.js`, substituting `{E2E_URL}`, `{WP_USER}`, `{WP_PASS}`. Run it as `--step login`. Confirm it passed before continuing — every later step depends on the session. 3. **One step per assertion.** Translate each flow assertion into a step script that inlines the relevant fixture(s) (navigation, REST GET/POST, AJAX, admin-notice) and logs `PASS:` / `FAIL:`. Use `getByLabel` / `getByRole` to match the Playwright POM. Always use `{ waitUntil: "networkidle" }` for navigation. Run each with a kebab-case `--step` name and `--timeout 10` where a hang is plausible. 4. **End the session:** `npx @usecanary/cli session end "$id"` (always, even on failure). -5. **Read `results.json`** for this session — extract `summary.stepsPassed`, `summary.stepsFailed`, `summary.stepsTotal`, and each step's `ok`/`exitCode`. Capture `trace_path` (`~/.canary/sessions/$id/trace.zip`) and `report_path` (`~/.canary/sessions/$id/report.html`). Derive the flow's PASS/FAIL/PARTIAL per the rules above. -6. **Clean up step scripts:** `rm -rf /tmp/canary-steps`. +5. **Relocate artifacts** to the project workspace: + ```bash + ISSUE_N=$(echo "$QA_PLAN_PATH" | grep -oE '[0-9]+' | head -1) + CANARY_DIR=".ai/${ISSUE_N}/canary" + mkdir -p "$CANARY_DIR" + mv ~/.canary/sessions/"$id" "$CANARY_DIR/" + python3 -c " + import json; p='$CANARY_DIR/$id/session.json' + d=json.load(open(p)); d['artifactsDir']='$(pwd)/$CANARY_DIR/$id' + json.dump(d, open(p, 'w'), indent=2) + " + ln -sfn "$(pwd)/$CANARY_DIR/$id" ~/.canary/sessions/"$id" + ``` + This moves the session to `.ai/{N}/canary/<id>/`, updates `artifactsDir` so the viewer still resolves artifacts correctly, and leaves a symlink at `~/.canary/sessions/<id>` so `npx @usecanary/ui` can still find it. +6. **Read `results.json`** for this session — extract `summary.stepsPassed`, `summary.stepsFailed`, `summary.stepsTotal`, and each step's `ok`/`exitCode`. Capture `trace_path` (`.ai/${ISSUE_N}/canary/$id/trace.zip`) and `report_path` (`.ai/${ISSUE_N}/canary/$id/report.html`). Derive the flow's PASS/FAIL/PARTIAL per the rules above. +7. **Clean up step scripts:** `rm -rf /tmp/canary-steps`. Record a `canary_sessions[]` entry per flow (flow label, session_id, passed/failed/total, trace_path, report_path). @@ -247,9 +261,10 @@ If the result is `1` (key non-empty), API-key guards are **not** blockers. If a After all sessions are recorded, collect the per-step screenshots Canary captured and publish them via a temporary branch commit (same pattern as `e2e-qa-tester`): ```bash +ISSUE_N=$(echo "$QA_PLAN_PATH" | grep -oE '[0-9]+' | head -1) mkdir -p .e2e-screenshots for id in <each session id>; do - cp ~/.canary/sessions/"$id"/screenshots/*.png .e2e-screenshots/ 2>/dev/null || true + cp ".ai/${ISSUE_N}/canary/$id/screenshots/"*.png .e2e-screenshots/ 2>/dev/null || true done git add -f .e2e-screenshots/ git commit -m "chore(qa): Canary QA screenshots" @@ -264,7 +279,7 @@ git commit -m "chore(qa): remove Canary QA screenshots" git push ``` -Capture `SHA` into your context — you need it to build per-file `raw.githubusercontent.com` URLs for the `### Screenshots` table and the return JSON. Never commit Canary session artifacts (`~/.canary/`) — they are local-only. +Capture `SHA` into your context — you need it to build per-file `raw.githubusercontent.com` URLs for the `### Screenshots` table and the return JSON. Never commit Canary session artifacts (`.ai/{N}/canary/` — gitignored) or screenshot PNGs permanently. --- @@ -350,9 +365,9 @@ gh pr comment {PR_NUMBER} --body "$(cat <<'CANARY' | Flow | Steps | Result | Trace | |---|---|---|---| -| P0-A: flow name | 9/9 | ✅ PASS | `npx playwright show-trace ~/.canary/sessions/<id>/trace.zip` | +| P0-A: flow name | 9/9 | ✅ PASS | `npx playwright show-trace .ai/{N}/canary/<id>/trace.zip` | -Reports (open locally): `npx @usecanary/ui --dir ~/.canary/sessions/<id>` +Reports (open locally): `npx @usecanary/ui --dir .ai/{N}/canary/<id>` CANARY )" ``` @@ -401,8 +416,8 @@ After the prose report, return this JSON object to `qa-engineer`. It is the **ex "passed": 9, "failed": 0, "total": 9, - "trace_path": "~/.canary/sessions/p0-a--flow-name-abc123/trace.zip", - "report_path": "~/.canary/sessions/p0-a--flow-name-abc123/report.html" + "trace_path": ".ai/1234/canary/p0-a--flow-name-abc123/trace.zip", + "report_path": ".ai/1234/canary/p0-a--flow-name-abc123/report.html" } ], "canary_results_table": "| Flow | Steps | Result | Trace |\n|---|---|---|---|\n| P0-A: flow name | 9/9 | ✅ PASS | ... |" @@ -421,10 +436,10 @@ Field rules: - ✅ **Always do:** read both Canary session-agent files before recording; read `qa-plan.md` and record one session per P0/P1 flow; inline fixtures verbatim into each step script; read `results.json` for every session and derive the verdict from `summary` + per-step `ok`/`exitCode`; publish screenshots via branch commit + SHA URL; write fresh POM-based Playwright specs and commit them (E2E_CI is true); `session end` every session even on failure; post the Canary results table with the dedup marker; uninstall any plugins you installed. - ⚠️ **Ask first (report as blocker):** `gh` not authenticated; boot command fails; a "How to test" or QA-plan step is ambiguous; a required premium plugin is absent and cannot be installed. -- 🚫 **Never do:** spawn a sub-agent (you have no `Agent` tool — run the CLI yourself); fork/vendor/wrap the Canary CLI; use `require`/`import`/`fs`/`path`/`process.env`/top-level `fetch` in a step script; commit Canary session artifacts (`~/.canary/`) or screenshot PNGs permanently; mechanically transpile QuickJS steps into specs; use `setTimeout`/`waitForTimeout` in specs; report PASS without a Canary step PASS line or screenshot evidence; infer PASS for a behavioral claim through a license/quota guard; install plugins not explicitly required. +- 🚫 **Never do:** spawn a sub-agent (you have no `Agent` tool — run the CLI yourself); fork/vendor/wrap the Canary CLI; use `require`/`import`/`fs`/`path`/`process.env`/top-level `fetch` in a step script; commit Canary session artifacts (`.ai/{N}/canary/` is gitignored — keep it that way) or screenshot PNGs permanently; skip the relocation step after `session end`; mechanically transpile QuickJS steps into specs; use `setTimeout`/`waitForTimeout` in specs; report PASS without a Canary step PASS line or screenshot evidence; infer PASS for a behavioral claim through a license/quota guard; install plugins not explicitly required. ## Known limitations - **P2 flows are optional.** A P2 session failure never gates the overall verdict. -- **Canary artifacts are local-only.** The PR comment links trace/report by local path (`npx playwright show-trace …`, `npx @usecanary/ui --dir …`); only screenshots are published as raw URLs. +- **Canary artifacts are local-only.** Relocated to `.ai/{N}/canary/` (gitignored). The PR comment links trace/report by local path (`npx playwright show-trace .ai/{N}/canary/<id>/trace.zip`, `npx @usecanary/ui --dir .ai/{N}/canary/<id>`); only screenshots are published as raw URLs. - **Spec promotion path:** specs are committed directly to `Tests/e2e/specs/` (E2E_CI is true) — no separate promotion step. diff --git a/.claude/agents/canary-imagify-session-agent.md b/.claude/agents/canary-imagify-session-agent.md index 10f75caf..d7968b45 100644 --- a/.claude/agents/canary-imagify-session-agent.md +++ b/.claude/agents/canary-imagify-session-agent.md @@ -113,8 +113,21 @@ Same lifecycle as the base agent, with Imagify specifics: above (`imagify-abilities`, `imagify-invalid-api-key`), substituting Imagify URLs and selectors. Use `--timeout 10` where a hang is plausible. 5. `npx @usecanary/cli session end "$id"`. -6. `rm -rf /tmp/canary-steps`. -7. Report per-step exitCode + PASS/FAIL lines and an overall verdict. +6. **Relocate artifacts** to the project workspace (derive `CANARY_DIR` from calling context): + - **canary-e2e (pipeline):** `ISSUE_N=$(echo "$QA_PLAN_PATH" | grep -oE '[0-9]+' | head -1); CANARY_DIR=".ai/${ISSUE_N}/canary"` + - **testrail-run-agent:** `CANARY_DIR=".ai/testrail/${TESTRAIL_RUN_ID}/canary"` + ```bash + mkdir -p "$CANARY_DIR" + mv ~/.canary/sessions/"$id" "$CANARY_DIR/" + python3 -c " + import json; p='$CANARY_DIR/$id/session.json' + d=json.load(open(p)); d['artifactsDir']='$(pwd)/$CANARY_DIR/$id' + json.dump(d, open(p, 'w'), indent=2) + " + ln -sfn "$(pwd)/$CANARY_DIR/$id" ~/.canary/sessions/"$id" + ``` +7. `rm -rf /tmp/canary-steps`. +8. Report per-step exitCode + PASS/FAIL lines and an overall verdict. Reference artifacts as `$CANARY_DIR/$id/results.json`, `$CANARY_DIR/$id/report.html`, `$CANARY_DIR/$id/trace.zip`. ## DO NOT (in addition to the base agent's rules) diff --git a/.claude/canary-wp-spec.md b/.claude/canary-wp-spec.md index a4f5b741..db7e749a 100644 --- a/.claude/canary-wp-spec.md +++ b/.claude/canary-wp-spec.md @@ -61,7 +61,7 @@ Session ID format: `<slug>-<random8>-<short-hash>` — e.g. `p0-a--mcp-abilities ### Artifact structure -Every session directory at `~/.canary/sessions/<id>/` contains: +Canary writes to `~/.canary/sessions/<id>/` initially. Our agents relocate each session to `.ai/{issue_N}/canary/<id>/` after `session end` and leave a symlink at the original path (so `npx @usecanary/ui` still works). The directory structure is the same either way: ``` session.json ← session metadata + step scripts (written during run) @@ -137,41 +137,9 @@ Canary agent files of interest: --- -## 1. Problem +## 2. Architecture -The current `e2e-qa-tester` agent drives the browser via `mcp__playwright` — an interactive, session-bound tool with no artifact output. It produces Playwright specs committed to `Tests/e2e/specs/`, but: - -- The agent improvises every WordPress interaction from scratch (login flow, nonce retrieval, REST calls, AJAX patterns) — getting them wrong about 30% of the time -- There is no rich local evidence: no trace, no video, no HAR — just a spec file and a screenshot URL -- The QA plan lives only in the orchestrator's context; it is not surfaced to the developer or reviewer -- Canary sessions run today are **disconnected** from the issue workflow — triggered manually, post-merge -- The `report.html` Canary produces is not linked from the GitHub PR comment — developers never see it - ---- - -## 2. Vision - -Introduce a **`canary-e2e` agent** — a drop-in replacement for `e2e-qa-tester` that uses Canary CLI instead of `mcp__playwright`. The user opts into it at orchestrator startup; the existing `e2e-qa-tester` stays untouched as the default. - -Every `orchestrator` / `issue-workflow` run with `e2e_mode: "canary"` produces: - -1. A **structured QA plan** (P0/P1/P2) derived from the diff — written to `.ai/qa-plan.md` so the developer and reviewer can read it -2. **Recorded Canary sessions** per P0/P1 flow — artifacts stored locally, a results summary **posted to the GitHub PR comment** (not committed to git) -3. **Playwright specs** written from the recorded flows — committed to `Tests/e2e/specs/` for CI to re-run - -The WordPress login, nonce, and REST patterns are baked into reusable snippet blocks so no agent ever has to guess them again. - -### Why not replace `e2e-qa-tester`? - -- Non-destructive: the old agent is the safe fallback; a single word at startup rolls back -- Gradual adoption: run Canary on a few issues, compare results, decide -- Feature parity check: can run the same issue through both agents and compare output - ---- - -## 3. Architecture - -### 3.1 What "cloning Canary" means +### 2.1 What "cloning Canary" means Canary has two layers: @@ -182,7 +150,7 @@ Canary has two layers: No separate plugin install or repo clone is needed. The CLI is already installed globally via `npx`; the skill pack files are already on disk at `~/.claude/plugins/marketplaces/canary-marketplace/`. -### 3.2 Mode selection +### 2.2 Mode selection At orchestrator startup, during the existing calibration step, the orchestrator asks: @@ -204,7 +172,7 @@ orchestrator startup Both `canary-e2e` and `e2e-qa-tester` share the **same JSON return contract** — `qa-engineer` is agnostic to which one ran. -### 3.3 Proposed pipeline (canary mode) +### 2.3 Proposed pipeline (canary mode) ``` orchestrator [e2e_mode: "canary"] @@ -226,7 +194,7 @@ orchestrator [e2e_mode: "canary"] └─ READY TO MERGE or blockers ``` -### 3.4 New agents +### 2.4 New agents Two WP agents replace the generic Canary marketplace agents. They live at different scopes: @@ -250,7 +218,7 @@ Two WP agents replace the generic Canary marketplace agents. They live at differ **`canary-imagify-session-agent.md`** extends the WP base with Imagify specifics: - Imagify admin URLs (settings, bulk optimization, custom folders) - Ability slugs and MCP endpoint (`/wp-json/wp-abilities/v1/abilities`) -- `imagify-abilities` fixture block (see section 4) +- `imagify-abilities` fixture block (see section 3) - Imagify-specific selectors and notice patterns The "extends" instruction at the top of the plugin agent: @@ -271,7 +239,7 @@ in project-level `.claude/agents/` means: This is worth formalising across `wp-media` projects when the WP base is stable enough. -### 3.5 Updated existing agents +### 2.5 Updated existing agents **`qa-engineer.md`** — becomes E2E-mode-aware: - Receives `e2e_mode` from the orchestrator dispatch @@ -290,7 +258,7 @@ This is worth formalising across `wp-media` projects when the WP base is stable - Writes Playwright specs to `Tests/e2e/specs/` (same as `e2e-qa-tester`) - Returns the **same JSON shape** as `e2e-qa-tester` so `qa-engineer` needs no branching logic -### 3.6 Artifact flow +### 2.6 Artifact flow ``` Developer machine (local only — not committed) @@ -318,7 +286,7 @@ Optional future: upload ~/.canary/sessions/ as CI artifact "canary-qa-<pr>" --- -## 4. WordPress fixture snippets +## 3. WordPress fixture snippets Because Canary's QuickJS sandbox has no `require()`, helpers must be inlined per step script. The `canary-wp-session-agent` carries these as **canonical copy-paste blocks** in its instructions — never re-invented, always copied verbatim. @@ -435,7 +403,7 @@ for (const slug of expected) { --- -## 5. QA plan format (`.ai/qa-plan.md`) +## 4. QA plan format (`.ai/qa-plan.md`) Produced by `qa-engineer`, consumed by `e2e-qa-tester`. The `P0-A` / `P0-B` labels become Canary session names. @@ -467,7 +435,7 @@ Produced by `qa-engineer`, consumed by `e2e-qa-tester`. The `P0-A` / `P0-B` labe --- -## 6. Canary results → GitHub PR comment +## 5. Canary results → GitHub PR comment The `e2e-qa-tester` extracts `results.json` from each session and formats it as markdown. This goes into the GitHub PR comment — **not committed to git**. @@ -512,7 +480,7 @@ git push --- -## 7. Playwright spec translation +## 6. Playwright spec translation Each Canary step script (QuickJS) translates to a Playwright spec (Node.js TypeScript) for CI. The translation is NOT automatic — `e2e-qa-tester` writes the spec from scratch using the Canary session's step logic as reference. Key differences: @@ -567,7 +535,7 @@ test.describe('MCP Abilities — Wave 2', () => { --- -## 8. GitHub Actions integration +## 7. GitHub Actions integration ### Playwright CI (already exists — no change needed) @@ -591,7 +559,7 @@ This is optional for the initial implementation — the local-only approach is s --- -## 9. Open questions for the audit +## 8. Open questions for the audit 1. **Artifact location in CI:** `~/.canary/sessions/` is per-machine (on dev). If CI also needs to produce Canary sessions (not just replay Playwright specs), where do artifacts land? Path may need to be overridden via `CANARY_HOME` or equivalent env var. @@ -605,7 +573,7 @@ This is optional for the initial implementation — the local-only approach is s --- -## 10. Files to create / modify +## 9. Files to create / modify **User-level** (`~/.claude/agents/` — shared across all WP plugin repos): @@ -628,7 +596,7 @@ This is optional for the initial implementation — the local-only approach is s --- -## 11. Known Canary limitations (discovered in live sessions) +## 10. Known Canary limitations (discovered in live sessions) - **`console.log` in QuickJS counts as "consoleErrors"** if the Playwright daemon routes all console output through the error channel. Do not use `summary.consoleErrors > 0` as a FAIL signal — check `summary.stepsFailed` instead. - **`networkFailures`** in the summary counts network events that returned 4xx/5xx — WP admin always generates some (favicon 404, etc.). Do not use `networkFailures > 0` as a FAIL signal. From cfb81d1df1fad06557f56a9af3b17033c7d53a87 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Mon, 29 Jun 2026 01:29:52 +0200 Subject: [PATCH 04/16] docs(qa): trim human rationale from all .claude/ pipeline docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove everything that is explanation-for-humans rather than operative instruction for AI: design rationale sections, "why we did this" blocks, stale file-lists and open-question tables, duplicate architecture prose, implementation-order checklists, and obsolete "optional future" sections. - canary-wp-spec.md: §2 Architecture collapsed to 3 tight subsections (routing, inheritance, artifact locations); removed §7 GitHub Actions, §8 open questions, §9 file list; trimmed §5 screenshot shell block, §6 TS example, "Why GET not POST" aside - canary-testrail-spec.md: removed pipeline context block, "How they connect" prose, name-change note, both Goal subsections, dedup fallback, §6.1/6.2/6.4 (Problem/Vision/design now in agent files), §9 file list, §11 implementation order; trimmed §8 replay, §7.1 "why this split", §7.3 trust-guard (updated: guard not present); updated artifact paths to .ai/testrail/$RUN_ID/canary/ - testrail-run-agent.md: add relocation step after session end (move to .ai/testrail/$RUN_ID/canary/); update all path references - testrail-scenarios/SKILL.md, testrail-run/SKILL.md: remove meta-sentences - testrail-automation.md: deleted (superseded by canary-testrail-spec.md) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- .claude/agents/testrail-run-agent.md | 27 +- .claude/canary-testrail-spec.md | 328 +++------------------ .claude/canary-wp-spec.md | 285 ++---------------- .claude/skills/testrail-run/SKILL.md | 4 - .claude/skills/testrail-scenarios/SKILL.md | 4 - .claude/testrail-automation.md | 234 --------------- 6 files changed, 81 insertions(+), 801 deletions(-) delete mode 100644 .claude/testrail-automation.md diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index e70ea687..bb2d71e4 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -31,7 +31,7 @@ Auth : Basic — $TESTRAIL_USERNAME : $TESTRAIL_API_KEY (always in the Project ID : 3 Suite ID : 3 Canary CLI : npx @usecanary/cli -Sessions dir : ~/.canary/sessions/<id>/ +Sessions dir : .ai/testrail/$RUN_ID/canary/<id>/ (relocated from ~/.canary/sessions/<id>/ after session end) WP fixtures : .claude/agents/canary-imagify-session-agent.md (read for WP login/selectors) ``` @@ -163,15 +163,25 @@ If a step depends on an environment that is not available (multisite, a specific plugin, an external service), do NOT force a FAIL — stop the case and mark it **BLOCKED** with the reason (see Blocking detection). -### 3f — End the session +### 3f — End the session and relocate artifacts ```bash npx @usecanary/cli session end "$id" + +CANARY_DIR=".ai/testrail/${RUN_ID}/canary" +mkdir -p "$CANARY_DIR" +mv ~/.canary/sessions/"$id" "$CANARY_DIR/" +python3 -c " +import json; p='$CANARY_DIR/$id/session.json' +d=json.load(open(p)); d['artifactsDir']='$(pwd)/$CANARY_DIR/$id' +json.dump(d, open(p, 'w'), indent=2) +" +ln -sfn "$(pwd)/$CANARY_DIR/$id" ~/.canary/sessions/"$id" ``` ### 3g — Determine the outcome -Read `~/.canary/sessions/$id/results.json`: +Read `.ai/testrail/${RUN_ID}/canary/$id/results.json`: - **PASS** — every step ran and none reported failure (`stepsFailed == 0`, all expected conditions held, no `FAIL:` in step output). - **FAIL** — any `stepsFailed > 0` or any step logged `FAIL:` / threw. @@ -182,8 +192,7 @@ Record, per case: ``` { case_id, title, session_id, outcome, trace_path, steps_passed, steps_total, elapsed, reason } ``` -where `trace_path` = `~/.canary/sessions/$id/trace.zip` and `elapsed` comes from -`results.json` (total session duration). +where `trace_path` = `.ai/testrail/${RUN_ID}/canary/$id/trace.zip` and `elapsed` comes from `results.json`. ### Blocking detection @@ -198,9 +207,9 @@ After **all** cases have run: ``` | Case | Title | Outcome | Steps | Trace | |--------|----------------------------|------------|-------|--------------------------------------------------| -| C14169 | Should correctly escape... | PASS | 5/5 | npx playwright show-trace ~/.canary/sessions/.../trace.zip | -| C174 | Multisite settings | BLOCKED | 0/3 | (needs multisite env) | -| C201 | Bulk optimize 500 images | FAIL | 3/4 | npx playwright show-trace ~/.canary/sessions/.../trace.zip | +| C14169 | Should correctly escape... | PASS | 5/5 | npx playwright show-trace .ai/testrail/1283/canary/.../trace.zip | +| C174 | Multisite settings | BLOCKED | 0/3 | (needs multisite env) | +| C201 | Bulk optimize 500 images | FAIL | 3/4 | npx playwright show-trace .ai/testrail/1283/canary/.../trace.zip | ``` Summarise totals (N passed / M failed / K blocked) below the table. @@ -232,7 +241,7 @@ Payload: ```json { "status_id": 1, - "comment": "Automated Canary run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\n\nTrace: npx playwright show-trace ~/.canary/sessions/<id>/trace.zip", + "comment": "Automated Canary run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\n\nTrace: npx playwright show-trace .ai/testrail/$RUN_ID/canary/<id>/trace.zip", "elapsed": "45s" } ``` diff --git a/.claude/canary-testrail-spec.md b/.claude/canary-testrail-spec.md index 59f9651e..ebe21c8a 100644 --- a/.claude/canary-testrail-spec.md +++ b/.claude/canary-testrail-spec.md @@ -1,9 +1,6 @@ # Spec: Canary × TestRail — Consolidated QA Automation for Imagify -**Status:** Draft — brainstorm/design only. No implementation in this document. -**Author:** Gaël Robin -**Audience:** Future Claude sessions with zero prior context. This document is self-contained. -**Next step:** Deep audit session. +**Status:** Implemented — `test/e2e-agent` branch. All 13 audit questions resolved. --- @@ -28,21 +25,6 @@ All three share one substrate: **Canary** (browser recording + QA platform) and **battle-tested WordPress fixture snippets** wrapped in a reusable **agent inheritance pattern** (WP base → Imagify-specific). -### The existing delivery pipeline (context) - -Imagify ships via an agentic Claude Code pipeline: - -``` -orchestrator → grooming-agent → backend-agent → lead-reviewer → qa-engineer → e2e-qa-tester → release-agent -``` - -- Driven by `.claude/skills/orchestrator/SKILL.md` and `.claude/skills/issue-workflow/SKILL.md`. -- Agents live in `.claude/agents/`. -- `qa-engineer` is the QA gate; it currently spawns `e2e-qa-tester` for UI/browser changes. -- `e2e-qa-tester` drives the browser via `mcp__playwright`, writes specs to `Tests/e2e/specs/`, - and publishes screenshots via temp-branch commit + SHA `raw.githubusercontent.com` URLs. - **It is the current stable default and is NOT modified by any feature here.** - ### What Canary is (engine + sandbox) Canary is a browser recording + QA platform built on two npm packages, used as-is via `npx` @@ -76,7 +58,7 @@ npx @usecanary/cli session end # writes results.json + report.html + artifac Session ID format: `<slug>-<random8>-<short-hash>` — e.g. `p0-a--mcp-abilities-discovery-mqubrrhk-3f7e94`. -**Artifacts per session** at `~/.canary/sessions/<id>/`: +**Artifacts per session** at `.ai/{N}/canary/<id>/` (relocated from `~/.canary/sessions/<id>/` after `session end`; symlink left at original for the viewer): ``` session.json ← session metadata + step scripts (written during run) @@ -126,16 +108,9 @@ profile/ ← browser profile - WP REST nonces expire (~12h TTL). Re-fetch the nonce at the start of each REST step, not just once at login. -### The Canary replay advantage (the reason this is worth building) +### Canary replay -Canary step scripts are recorded into `session.json` and can be **replayed without Claude**: - -```bash -npx @usecanary/cli run ~/.canary/sessions/<id>/steps/<step>.js -``` - -This means: **the first run is Claude-driven (inference cost); every subsequent replay is free** -(zero inference). This underpins Feature 2's path to free CI reruns (see §8). +Step scripts recorded in `session.json` replay without Claude — first run is Claude-driven; subsequent CI replays are zero-inference (see §8). --- @@ -172,21 +147,6 @@ This means: **the first run is Claude-driven (inference cost); every subsequent §8 Replay → free CI reruns ``` -**How they connect:** - -- **Feature 1 feeds Feature 2.** Feature 1 produces TestRail cases (human-readable steps); - Feature 2 reads those exact cases and executes them via Canary. They are two halves of the - "release QA" loop: generate coverage, then run it. -- **Feature 2 and Feature 3 share the Canary execution substrate.** Both spawn the - `canary-imagify-session-agent` to drive the browser with the §6 WP fixtures. The difference is - the *source of truth* for what to test: - - Feature 3 reads `qa-plan.md` (P0/P1/P2 flows derived from a diff) — pre-merge, per PR. - - Feature 2 reads TestRail cases (`custom_steps_separated`) — release-time, full suite. -- **Feature 3 is pipeline-embedded; Features 1 & 2 are standalone skills.** Feature 3 plugs into - the orchestrator/qa-engineer flow. Features 1 & 2 are invoked on demand (`/testrail-*`). -- **All three reuse the agent inheritance pattern (§6).** The WP base agent is the single source - of truth for login/nonce/REST/AJAX; the Imagify agent extends it with plugin specifics. - --- ## 2. Agent & skill inventory @@ -198,11 +158,6 @@ This means: **the first run is Claude-driven (inference cost); every subsequent | `/testrail-scenarios` | 1 | user invokes with PR number(s)/URL(s) or `--since-tag` | PR refs, optional `--force-pr=<n>`, optional section override | staging files under `.ai/testrail/pending/`, summary, then created case IDs on publish | | `/testrail-run` | 2 | user invokes with milestone name/ID or `--run-id` | milestone/run identifier, optional `--cases=<ids>` filter | results table (case → outcome + Canary artifacts), then TestRail results posted on confirm | -> Names are proposals. The earlier draft used `/testrail-release-setup` and -> `/testrail-release-process`; this spec renames to `/testrail-scenarios` and `/testrail-run` to -> decouple from the word "release" (the features are useful per-PR too). Final names are an open -> question (§9). - ### New agents | Agent | Scope | Feature | What it does | Inputs | Outputs | @@ -232,13 +187,7 @@ This means: **the first run is Claude-driven (inference cost); every subsequent ## 3. Feature 1 — TestRail scenario generation -### 3.1 Goal - -Expand QA coverage beyond "what changed" by auto-drafting structured TestRail cases from PRs, -keeping a human in the loop (review → approve → publish), and never duplicating cases for a PR -already covered. - -### 3.2 Flow +### 3.1 Flow ``` /testrail-scenarios <pr | pr-list | --since-tag> @@ -265,7 +214,7 @@ The review gate is **mandatory and explicit**: the agent stages, summarises, the creates cases in the same turn it drafts them. The user edits YAML freely (titles, steps, section, smoke flag) before publishing. -### 3.3 Staging file format (`.ai/testrail/pending/<pr-slug>.yml`) +### 3.2 Staging file format (`.ai/testrail/pending/<pr-slug>.yml`) ```yaml pr: 1133 @@ -310,7 +259,7 @@ Field mapping (staging YAML → TestRail `add_case` payload): | (constant) | `template_id: 2`, `type_id: 7`, `priority_id: 2` | | `pr_url` | `refs` | -### 3.4 What scenarios the agent drafts (per PR) +### 3.3 What scenarios the agent drafts (per PR) The agent reads PR description, diff summary, and linked issue spec, then drafts cases covering: @@ -328,22 +277,11 @@ artificial cap per PR. Each case is also evaluated for `smoke_test: true` (criti always pass). Pure tracking/analytics PRs with no user-visible behaviour: the agent may draft a single minimal "event fires" case or skip — see §9. -### 3.5 Deduplication - -**Strategy:** TestRail's `refs` field links a PR to its cases. +### 3.4 Deduplication -- On create, `refs = <github PR url>`. -- On the next run, per in-scope PR: `GET /get_cases/3?refs=<pr_url>`. - - cases found → skip the PR. - - none found → draft and stage. -- `--force-pr=<n>` regenerates and stages new cases for a PR even if cases exist (does **not** - delete the old ones — manual cleanup in TestRail). +`refs = <github PR url>` on create. Dedup: `GET /get_cases/3?refs=<pr_url>` — cases found → skip; none → draft. `--force-pr=<n>` regenerates (does not delete old cases). -**Fallback if `refs` filtering is unsupported in this TestRail version** (open question §9): -query all cases in the relevant section and match by title prefix, OR maintain a local index at -`.ai/testrail/index.json` (persists between runs; NOT cleaned with `pending/`). - -### 3.6 Section mapping +### 3.5 Section mapping The agent picks the closest existing section; if none fits, it proposes a new subsection under `Regression (33)` via the `new_section` field (created only on publish, never silently). @@ -365,29 +303,15 @@ The agent picks the closest existing section; if none fits, it proposes a new su | Smoke tests | flag `custom_smoketest: true` (NOT a separate section) + Smoke test section (4525) for pure smoke cases | | New feature, no match | propose new subsection under Regression (33) | -### 3.7 MCP vs API - -TestRail may be reachable via an MCP server (not confirmed). The agent must handle both: +### 3.6 MCP vs API -- **If a TestRail MCP is available:** prefer its tools for `get_cases` / `add_section` / - `add_case` — typed, less error-prone. -- **Otherwise (default assumption):** use the TestRail REST API (see §5) with Basic auth from - `TESTRAIL_USERNAME` / `TESTRAIL_API_KEY`. - -The agent probes for the MCP first; if absent, falls back to REST. Both paths produce identical -staging files and identical `refs`-based dedup behaviour. +TestRail MCP (`/opt/homebrew/bin/mcp-testrail`) is connected but tool names unconfirmed. **Use the REST API via `curl` as primary** (see §5). MCP is an optional enhancement once tool names are confirmed. --- ## 4. Feature 2 — TestRail run execution -### 4.1 Goal - -Run an entire TestRail test run automatically via Canary browser automation, gather pass/fail/ -blocked outcomes with rich artifacts, let the user review, then post results back to TestRail with -evidence — turning the full regression suite into an automated pass instead of a manual one. - -### 4.2 Flow +### 4.1 Flow ``` /testrail-run <milestone-name | milestone-id | --run-id=<id>> @@ -414,7 +338,7 @@ evidence — turning the full regression suite into an automated pass instead of The TestRail write is gated behind explicit user confirmation. The agent never posts results in the same turn it executes — the user sees outcomes first. -### 4.3 How Canary sessions map to TestRail cases +### 4.2 How Canary sessions map to TestRail cases - **One Canary session per TestRail case.** Session name = `TR-<case_id>: <case title>` so the artifact directory and report are traceable back to the case. @@ -426,7 +350,7 @@ the same turn it executes — the user sees outcomes first. Canary script using the §6 fixtures. Where a step is too ambiguous to automate, the case is marked **blocked** (not failed) and flagged for manual review. -### 4.4 Outcome derivation (from `results.json`) +### 4.3 Outcome derivation (from `results.json`) | Condition | TestRail status | |-----------|-----------------| @@ -436,14 +360,14 @@ the same turn it executes — the user sees outcomes first. Reminder: do NOT use `consoleErrors` / `networkFailures` as fail signals (see §0). -### 4.5 Evidence in the result comment +### 4.4 Evidence in the result comment The result comment for each case includes per-step outcomes plus pointers to Canary artifacts: ```json { "status_id": 5, - "comment": "Automated Canary run — 2026-06-26\nSession: TR-4521--mcp-discover-abilities-xxx\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\nStep 3: ❌ Failed — assertion 'returns ability list' did not hold\n\nTrace: npx playwright show-trace ~/.canary/sessions/TR-4521--…/trace.zip\nReport: ~/.canary/sessions/TR-4521--…/report.html\nScreenshot: <raw.githubusercontent.com SHA URL, if published>", + "comment": "Automated Canary run — 2026-06-26\nSession: TR-4521--mcp-discover-abilities-xxx\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\nStep 3: ❌ Failed — assertion 'returns ability list' did not hold\n\nTrace: npx playwright show-trace .ai/testrail/1283/canary/TR-4521--…/trace.zip\nReport: .ai/testrail/1283/canary/TR-4521--…/report.html\nScreenshot: <raw.githubusercontent.com SHA URL, if published>", "elapsed": "45s" } ``` @@ -456,7 +380,7 @@ Evidence to include: - Optionally, published screenshot URLs (same temp-branch-commit → SHA-raw-URL mechanism as `e2e-qa-tester`; see §6.5) when a hosted image is wanted in TestRail. -### 4.6 Failure handling +### 4.5 Failure handling - Step fails → case marked Failed, remaining steps captured but the failure is the verdict. - Env unreachable / session won't start → case Blocked with reason. @@ -534,57 +458,9 @@ Regression (33) ## 6. Feature 3 — Canary E2E in the pipeline -### 6.1 Problem with today's `e2e-qa-tester` - -- Improvises WP interactions (login, nonce, REST, AJAX) from scratch — wrong ~30% of the time. -- No rich evidence: no trace, no video, no HAR — just a spec file and a screenshot URL. Debugging - failures is blind. -- The QA plan lives only in the orchestrator's context — invisible to developer and reviewer. - -### 6.2 Vision - -A new `canary-e2e` agent — a **drop-in replacement** for `e2e-qa-tester` that uses Canary CLI -instead of `mcp__playwright`. Opt-in at orchestrator startup; `e2e-qa-tester` stays the untouched -default. Both share the **same JSON return contract**, so `qa-engineer` is agnostic to which ran. - -### 6.3 Mode selection at orchestrator startup - -During the existing calibration step the orchestrator asks one extra question: - -> **E2E mode** — `playwright` (default, stable) or `canary` (experimental, richer artifacts)? - -Answered once; stored as `e2e_mode`; passed in every downstream dispatch. If the issue has no -UI/browser changes, the flag is ignored. - -``` -orchestrator startup - → calibration (autonomy level) + e2e_mode ("playwright" | "canary") - → grooming → implementation → review … - → qa-engineer [receives e2e_mode] - diff has UI/browser changes? - yes + e2e_mode == "canary" → spawn canary-e2e - yes + e2e_mode == "playwright" → spawn e2e-qa-tester (current default) - no → skip E2E entirely - merge E2E results into the final GitHub PR comment -``` - -### 6.4 `canary-e2e` agent design - -- **Inputs:** `qa-plan.md` path, `e2e_mode: "canary"`, PR number, env (`E2E_URL`, `WP_USER`, - `WP_PASS`). -- **Behaviour:** for each **P0/P1** flow in `qa-plan.md`: - 1. Record a Canary session via `canary-imagify-session-agent` (session name = the flow label, - e.g. `P0-A: MCP abilities discovery`). - 2. Read `results.json` → build a markdown results row. - 3. Write a Playwright spec to `Tests/e2e/specs/` (translated from the flow — see §6.6). -- **P2 flows** are documented in the plan but Canary sessions are optional. -- **Publishes:** screenshots via temp-branch-commit → SHA raw URL (§6.5); trace replay command per - session. -- **Returns:** the **same JSON shape** as `e2e-qa-tester` so `qa-engineer` needs no branching. -- **Tools:** `Bash` (drives `npx @usecanary/cli`), `Read`, `Edit`, `Write`, `Glob`, `Grep`, - `WebFetch`. Notably **not** `mcp__playwright` — Canary is the browser driver. +`canary-e2e` is a drop-in replacement for `e2e-qa-tester` selected via `e2e_mode: "canary"` at orchestrator startup. Same JSON return contract; see `.claude/agents/canary-e2e.md` for full instructions. -### 6.5 `qa-plan.md` format (shared by qa-engineer ↔ canary-e2e ↔ e2e-qa-tester) +### 6.1 `qa-plan.md` format (shared by qa-engineer ↔ canary-e2e ↔ e2e-qa-tester) Produced by `qa-engineer`, written to `.ai/qa-plan.md`, consumed by whichever E2E agent runs. The `P0-A`/`P0-B` labels become Canary session names. @@ -613,62 +489,25 @@ The `P0-A`/`P0-B` labels become Canary session names. - documented; Canary session optional ``` -### 6.6 Artifact flow & PR comment - -``` -Developer machine (local only — NOT committed) - ~/.canary/sessions/<id>/ → session.json, results.json, report.html, - trace.zip, video/, network.har, console.log, screenshots/ - -GitHub PR comment (posted by qa-engineer via canary-e2e's JSON): - ├─ qa-plan (P0/P1/P2 markdown table) - ├─ Canary results table (from results.json) - ├─ Screenshot images (raw.githubusercontent.com SHA URLs) - └─ "Trace: npx playwright show-trace <local path>" per session - -GitHub Actions CI (committed by canary-e2e): - Tests/e2e/specs/ ← Playwright specs derived from the flows (existing CI re-runs them) +### 6.2 Artifact flow -Optional future: upload ~/.canary/sessions/ as CI artifact "canary-qa-<pr>" ``` +.ai/{N}/canary/<id>/ ← sessions relocated here from ~/.canary/sessions/<id>/ -**Results table format (from `results.json`):** - -```markdown -### Canary QA Sessions - -| Flow | Steps | Result | Trace | -|------|-------|--------|-------| -| P0-A: MCP abilities discovery | 10/10 | ✅ PASS | `npx playwright show-trace ~/.canary/sessions/p0-a--…/trace.zip` | -| P0-B: Settings page save | 5/6 | ❌ FAIL — step "save-settings" exit 1 | `npx playwright show-trace ~/.canary/sessions/p0-b--…/trace.zip` | - -### Screenshots - -| Step | Screenshot | -|------|-----------| -| login-admin | ![login](https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/p0-a-login-admin.png) | -``` - -**Screenshot publishing (same mechanism as today's `e2e-qa-tester`):** - -```bash -cp ~/.canary/sessions/<id>/screenshots/*.png .e2e-screenshots/ -git add -f .e2e-screenshots/ && git commit -m "chore(qa): Canary QA screenshots" && git push -SHA=$(git rev-parse HEAD) # URL base: https://raw.githubusercontent.com/wp-media/imagify-plugin/$SHA/.e2e-screenshots/ -git rm --cached .e2e-screenshots/*.png && git commit -m "chore(qa): remove Canary QA screenshots" && git push +GitHub PR comment: qa-plan table + Canary results table + screenshot SHA URLs + + "Trace: npx playwright show-trace .ai/{N}/canary/<id>/trace.zip" +Tests/e2e/specs/ ← Playwright specs committed by canary-e2e (CI re-runs them) ``` -### 6.7 Playwright spec translation (not mechanical) +### 6.3 Playwright spec translation -`canary-e2e` writes fresh Playwright specs that pass the same assertions — it does **not** transpile -QuickJS to Node (different runtimes, different error semantics). +`canary-e2e` writes fresh TypeScript specs — not a transpile of QuickJS steps: | Canary QuickJS | Playwright TypeScript | |---|---| | `browser.getPage("main")` | `page` fixture | | `page.evaluate(async () => fetch(...))` | identical | | `console.log("PASS: …")` | `expect(...).toBe(...)` | -| step exit 1 | thrown assertion | | no `import` | `import { test, expect } from '@playwright/test'` | | inlined helpers | POM methods in `Tests/e2e/pages/` | @@ -689,26 +528,7 @@ WP plugins share identical WP patterns but differ in plugin specifics. Split acc canary-imagify-session-agent.md # extends WP base + Imagify URLs/slugs/MCP/fixtures ``` -**`canary-wp-session-agent.md`** knows only WordPress-generic patterns: `wp-login.php` login, -REST nonce via `admin-ajax.php?action=rest-nonce` (plain text, `.trim()`), authenticated REST -calls (`X-WP-Nonce`), `admin-ajax.php` POST with `_ajax_nonce`, admin-notice assertions. - -**`canary-imagify-session-agent.md`** extends it with: Imagify admin URLs (settings, bulk -optimization, custom folders), ability slugs, the MCP/abilities endpoint -(`/wp-json/wp-abilities/v1/abilities`), the `imagify-abilities` fixture, Imagify selectors/notice -patterns. - -The "extends" instruction at the top of the plugin agent: - -```markdown -Before doing anything, read `~/.claude/agents/canary-wp-session-agent.md` in full. -Its WP fixture snippets and rules are your base. The sections below OVERRIDE or EXTEND them. -``` - -**Why this split:** the WP base stays lean (never learns about Imagify/WP Rocket/BackWPup); each -plugin repo owns a thin agent with only its specifics; fixing a WP pattern (e.g. nonce endpoint -change) happens once and every plugin inherits it; adding a new plugin = one new file, zero changes -to the base. Worth formalising across `wp-media` projects once the base is stable. +WP-generic patterns (login, nonce, REST, AJAX, notices) live in the base agent only. The plugin agent reads it as its first instruction and extends it with Imagify-specific URLs, selectors, and ability slugs. ### 7.2 WP fixture snippets (canonical — never improvised, always copied verbatim) @@ -798,87 +618,26 @@ const expected = [ for (const slug of expected) console.log(slugs.includes(slug) ? `PASS: ${slug}` : `FAIL: ${slug} NOT FOUND`); ``` -### 7.3 Session-agent trust guard (critical pipeline detail) +### 7.3 Session-agent trust guard -The Canary marketplace `session-agent`/`verify-agent` refuse to act on **relayed** confirmations — -they require a direct user message. In our pipeline these agents are invoked *transitively* -(`orchestrator → qa-engineer → canary-e2e → canary-imagify-session-agent`, or -`/testrail-run → testrail-run-agent → canary-imagify-session-agent`). The trust guard **must be -removed** in our forked `canary-wp-session-agent`/`canary-imagify-session-agent`, or every nested -spawn deadlocks waiting for a direct user confirmation that never comes. +**Not present** in the Canary marketplace agents (confirmed in audit). No guard to remove. Both `canary-wp-session-agent` and `canary-imagify-session-agent` can be invoked transitively from the pipeline without deadlocking on a confirmation prompt. --- -## 8. Canary replay advantage — free CI reruns (Feature 2 & 3) +## 8. Canary replay (Phase B) -Recorded Canary step scripts in `session.json` replay without Claude: +Step scripts in `session.json` replay without Claude — first run is Claude-driven; subsequent replays are zero-inference. -```bash -npx @usecanary/cli run ~/.canary/sessions/<id>/steps/<step>.js # zero inference -``` +- **Feature 2 (Phase B):** persist scripts to `Tests/e2e/canary/TR-<case_id>/`; CI replays unchanged cases. Only new/changed cases need Claude. +- **Feature 3:** `Tests/e2e/specs/` already provides the CI rerun path. Canary scripts can also be replayed locally to reproduce failures deterministically. -Implications: - -- **Feature 2 — TestRail run execution:** the *first* execution of a TestRail case is Claude-driven - (it interprets human-readable steps into a Canary script). Once recorded, that script is the - durable automation for the case. A nightly/release CI job can **replay** all recorded scripts for - a run — no inference, deterministic, fast — and still post fresh results + artifacts to TestRail. - A recorded script could be associated back to its case (e.g. stored under - `Tests/e2e/canary/TR-<case_id>/` or referenced from the case) so the next run replays instead of - re-inferring. Only *new* or *changed* cases need Claude. -- **Feature 3 — pipeline E2E:** the per-PR Canary sessions are recorded; the committed Playwright - specs in `Tests/e2e/specs/` already give CI a free rerun path. The Canary scripts themselves can - additionally be replayed locally to reproduce a failure deterministically without re-driving - through Claude. - -**Phasing:** -- Phase A (initial): Claude executes every run (inference cost per run). Simplest; ship this first. -- Phase B (scale): persist recorded scripts; CI replays them for unchanged cases → zero inference - at scale; Claude only handles new/changed cases. - -Open: exact storage location and the "case → recorded script" linkage (§9). +Phase A (current): Claude executes every run. Phase B: CI replays recorded scripts. --- -## 9. Files to create / modify - -**User-level** (`~/.claude/agents/` — shared across all WP plugin repos): - -| File | Action | Feature | Notes | -|------|--------|---------|-------| -| `~/.claude/agents/canary-wp-session-agent.md` | Create | 2, 3 | Fork of marketplace `session-agent` + WP-generic fixtures only; **trust guard removed** for pipeline use | -| `~/.claude/agents/canary-wp-verify-agent.md` | Create (optional) | 3 | Fork of marketplace `verify-agent` + WP QA-plan format | - -**Project-level** (`.claude/` — Imagify only): - -| File | Action | Feature | Notes | -|------|--------|---------|-------| -| `.claude/agents/canary-imagify-session-agent.md` | Create | 2, 3 | Extends WP base; adds Imagify URLs, ability slugs, MCP endpoint, `imagify-abilities` fixture | -| `.claude/agents/canary-e2e.md` | Create | 3 | New E2E agent — same JSON contract as `e2e-qa-tester`, Canary CLI internally | -| `.claude/agents/testrail-scenario-agent.md` | Create | 1 | Analyse PR(s) → stage → publish cases; MCP-or-REST | -| `.claude/agents/testrail-run-agent.md` | Create | 2 | Fetch run → orchestrate Canary execution → post results | -| `.claude/skills/testrail-scenarios/SKILL.md` | Create | 1 | Standalone entry point for scenario generation | -| `.claude/skills/testrail-run/SKILL.md` | Create | 2 | Standalone entry point for run execution | -| `.claude/agents/qa-engineer.md` | Modify | 3 | Write `.ai/qa-plan.md`; receive `e2e_mode`; route to `canary-e2e`/`e2e-qa-tester` | -| `.claude/skills/orchestrator/SKILL.md` | Modify | 3 | Add `e2e_mode` startup prompt; pass it in every dispatch | -| `.claude/agents/e2e-qa-tester.md` | **Untouched** | 3 | Stable fallback — no changes | -| `.github/workflows/e2e.yml` | Modify (optional) | 2, 3 | Canary artifact upload / replay job if CI runs Canary | -| `Tests/e2e/` | Existing | 3 | Playwright specs still committed here — no structural change | -| `.ai/testrail/pending/` | Runtime dir | 1 | Staging YAML (cleaned on publish) | -| `.ai/testrail/index.json` | Runtime file | 1 | Dedup fallback index (only if `refs` filtering unavailable) | -| `.ai/qa-plan.md` | Runtime file | 3 | Shared QA plan | - -**Environment / secrets required:** - -| Var | Used by | Notes | -|-----|---------|-------| -| `TESTRAIL_USERNAME`, `TESTRAIL_API_KEY` | 1, 2 | TestRail Basic auth | -| `E2E_URL`, `WP_USER`, `WP_PASS` | 2, 3 | WP fixture login target | -| `CANARY_HOME` (maybe) | 2, 3 | Override `~/.canary` artifact root in CI (§ open question) | - --- -## 10. Open questions — audit resolution (2026-06-26) +## 9. Audit resolution (2026-06-26) All questions resolved during deep audit. No open questions remain for initial implementation. @@ -906,18 +665,3 @@ All questions resolved during deep audit. No open questions remain for initial i - **TestRail steps are HTML:** `<p>Visit settings.</p>` — must strip with `python3 -c "import sys,re; print(re.sub('<[^>]+>','',sys.stdin.read()).strip())"` - **Active run 1283 (2.3.0):** 151 untested cases, status_id=3 (untested) ---- - -## 11. Implementation order (suggested) - -1. **Shared base first:** `canary-wp-session-agent` (user) + `canary-imagify-session-agent` - (project) with the §7.2 fixtures and the trust guard removed. Everything else depends on these. -2. **Feature 3 (pipeline E2E):** `canary-e2e` + `qa-engineer`/`orchestrator` changes. Highest - day-to-day value; exercises the shared agents on real PRs. -3. **Feature 1 (scenario generation):** `/testrail-scenarios` + `testrail-scenario-agent` + - dedup. Produces the cases Feature 2 consumes. -4. **Feature 2 Phase A (Claude-driven run):** `/testrail-run` + `testrail-run-agent` reusing the - Canary session agents; post results on confirm. -5. **Feature 2 Phase B (replay):** persist recorded scripts; CI replays unchanged cases for zero - inference at scale. -``` diff --git a/.claude/canary-wp-spec.md b/.claude/canary-wp-spec.md index db7e749a..4bded9b8 100644 --- a/.claude/canary-wp-spec.md +++ b/.claude/canary-wp-spec.md @@ -139,150 +139,46 @@ Canary agent files of interest: ## 2. Architecture -### 2.1 What "cloning Canary" means +### 2.1 E2E mode routing -Canary has two layers: - -| Layer | What it is | Ownership | -|-------|------------|-----------| -| **Engine** (`@usecanary/cli`, `@usecanary/ui`) | npm packages — Playwright daemon + QuickJS sandbox + report renderer | Stay as `npx @usecanary/cli` — no fork | -| **Skill pack** (agents + skills markdown) | The AI instructions that write and run step scripts | **Copy into `.claude/agents/`** — we own and customise | - -No separate plugin install or repo clone is needed. The CLI is already installed globally via `npx`; the skill pack files are already on disk at `~/.claude/plugins/marketplaces/canary-marketplace/`. - -### 2.2 Mode selection - -At orchestrator startup, during the existing calibration step, the orchestrator asks: - -> **E2E mode** — `playwright` (default, stable) or `canary` (experimental, richer artifacts)? - -The user answers once; the choice is stored as `e2e_mode` and passed in every downstream agent dispatch. If the issue has no UI/browser changes, the flag is ignored. +`e2e_mode` is set at orchestrator startup and passed to `qa-engineer`: ``` -orchestrator startup - → calibration (autonomy level) + e2e_mode ("playwright" | "canary") - → ... grooming → implementation → review ... - → qa-engineer [receives e2e_mode in dispatch] - detects UI/browser changes in diff? - yes + e2e_mode == "canary" → spawns canary-e2e - yes + e2e_mode == "playwright" → spawns e2e-qa-tester (current default) - no → skips E2E entirely - merges E2E results into final GitHub PR comment +qa-engineer [receives e2e_mode] + diff has UI/browser changes? + yes + e2e_mode == "canary" → canary-e2e + yes + e2e_mode == "playwright" → e2e-qa-tester (default, untouched) + no → skip E2E ``` -Both `canary-e2e` and `e2e-qa-tester` share the **same JSON return contract** — `qa-engineer` is agnostic to which one ran. +Both agents return the same JSON contract — `qa-engineer` is agnostic to which ran. -### 2.3 Proposed pipeline (canary mode) +### 2.2 Agent inheritance ``` -orchestrator [e2e_mode: "canary"] - └─► qa-engineer - reads diff → generates .ai/qa-plan.md (P0/P1/P2 flows) - posts initial "QA plan" GitHub PR comment - └─► canary-e2e (for UI/browser changes) - For each P0/P1 flow in qa-plan.md: - ├─ records a Canary session via canary-imagify-session-agent - ├─ reads results.json → builds markdown results table - └─ writes Playwright spec to Tests/e2e/specs/ - Publishes artifacts: - ├─ screenshots → temp branch commit → SHA raw.githubusercontent.com URLs - └─ "trace: npx playwright show-trace ~/.canary/sessions/<id>/trace.zip" - Returns JSON to qa-engineer (same shape as e2e-qa-tester) - qa-engineer posts final GitHub PR comment with: - ├─ qa-plan (P0/P1/P2 as markdown table) - ├─ Canary session results per flow - └─ READY TO MERGE or blockers +~/.claude/agents/canary-wp-session-agent.md ← user-level: login, nonce, REST, AJAX, notices +.claude/agents/canary-imagify-session-agent.md ← project: Imagify URLs, selectors, abilities ``` -### 2.4 New agents +The plugin agent reads the WP base as its first instruction (EXTEND, not replace). -Two WP agents replace the generic Canary marketplace agents. They live at different scopes: +### 2.3 Artifact locations -``` -~/.claude/agents/ ← user-level, shared across ALL WP plugin repos - canary-wp-session-agent.md # base: login, nonce, REST, AJAX, notices - canary-wp-verify-agent.md # base: WP QA plan format + verify flow +Sessions are relocated from `~/.canary/sessions/<id>/` to `.ai/{N}/canary/<id>/` after `session end`; a symlink at the original path keeps `npx @usecanary/ui` working. -.claude/agents/ ← project-level, Imagify only - canary-imagify-session-agent.md # extends wp-session + Imagify specifics ``` - -**`canary-wp-session-agent.md`** knows only WordPress-generic patterns: -- Login via `wp-login.php`, `#user_login`, `#user_pass`, `#wp-submit` -- REST nonce via `GET admin-ajax.php?action=rest-nonce` (plain text, needs `.trim()`) -- Authenticated REST calls (`X-WP-Nonce` header via `page.evaluate`) -- `admin-ajax.php` POST with `_ajax_nonce` -- Admin notice assertion (`.notice-success`, `.notice-error`) -- WP REST HTTP method conventions - -**`canary-imagify-session-agent.md`** extends the WP base with Imagify specifics: -- Imagify admin URLs (settings, bulk optimization, custom folders) -- Ability slugs and MCP endpoint (`/wp-json/wp-abilities/v1/abilities`) -- `imagify-abilities` fixture block (see section 3) -- Imagify-specific selectors and notice patterns - -The "extends" instruction at the top of the plugin agent: -```markdown -Before doing anything, read `~/.claude/agents/canary-wp-session-agent.md` in full. -Its WP fixture snippets and rules are your base. The sections below OVERRIDE or EXTEND them. +.ai/{N}/canary/<id>/ + results.json ← step results + verdict + report.html ← self-contained HTML report + trace.zip ← npx playwright show-trace trace.zip + video/ ← .webm + network.har ← HTTP traffic + console.log ← browser console + screenshots/ ← per-step PNGs (daemon auto-captures one per step) ``` -#### Idea to consider: one WP base, N plugin agents - -Keeping WP-generic patterns in `~/.claude/agents/` (user-level) and plugin-specific knowledge -in project-level `.claude/agents/` means: - -- The WP base stays lean — it never needs to know about Imagify, WP Rocket, BackWPup, etc. -- Each plugin repo owns its own thin agent with only what's unique to that plugin -- Fixing a WP pattern (e.g. nonce endpoint changes) happens in one file and all plugins pick it up -- Adding a new plugin (WP Rocket) = one new file, no changes to the WP base - -This is worth formalising across `wp-media` projects when the WP base is stable enough. - -### 2.5 Updated existing agents - -**`qa-engineer.md`** — becomes E2E-mode-aware: -- Receives `e2e_mode` from the orchestrator dispatch -- Reads the diff and spec → produces `.ai/qa-plan.md` (P0/P1/P2 flows) regardless of mode -- Posts GitHub PR comment with the QA plan as its first action -- If UI/browser changes detected: spawns `canary-e2e` (mode=canary) or `e2e-qa-tester` (mode=playwright) -- Updates the PR comment with whichever agent's results come back (same JSON shape either way) - -**`e2e-qa-tester.md`** — **untouched**. Remains the stable default. - -**`canary-e2e.md`** — new agent, same JSON contract as `e2e-qa-tester`: -- Receives `qa-plan.md` path + `e2e_mode: "canary"` from `qa-engineer` -- Records one Canary session per P0/P1 flow via `canary-imagify-session-agent` -- Reads `results.json` → builds markdown results table -- Publishes screenshots via temp branch commit → SHA raw URLs -- Writes Playwright specs to `Tests/e2e/specs/` (same as `e2e-qa-tester`) -- Returns the **same JSON shape** as `e2e-qa-tester` so `qa-engineer` needs no branching logic - -### 2.6 Artifact flow - -``` -Developer machine (local only — not committed) - ~/.canary/sessions/<id>/ - session.json ← step scripts (for debugging) - results.json ← machine-readable pass/fail + artifact list - report.html ← self-contained HTML report (open locally) - trace.zip ← npx playwright show-trace trace.zip - video/ ← .webm - network.har ← all HTTP traffic - console.log ← browser console - screenshots/ ← per-step PNGs - -GitHub PR comment (posted by qa-engineer / e2e-qa-tester): - ├─ Canary results table (from results.json) - ├─ Screenshot images (raw.githubusercontent.com SHA URLs) - └─ "Trace: npx playwright show-trace <path>" (local path — for reviewer to run) - -GitHub Actions CI (committed by e2e-qa-tester): - Tests/e2e/specs/ ← Playwright specs derived from the Canary flows - (existing CI job re-runs these specs in CI) - -Optional future: upload ~/.canary/sessions/ as CI artifact "canary-qa-<pr>" -``` +GitHub PR comment: Canary results table + screenshot SHA URLs + trace replay command per session. +`Tests/e2e/specs/` — Playwright specs committed by `canary-e2e` (existing CI re-runs them). --- @@ -315,10 +211,6 @@ const nonce = await page.evaluate(async () => { console.log("Nonce (first 10):", nonce.slice(0, 10)); ``` -> **Why GET, not POST:** `admin-ajax.php?action=rest-nonce` as a GET request works in modern WP -> because the action is registered to handle GET. The earlier POST-with-FormData variant also -> works but is slower. GET is canonical. - ### wp-rest-get ```js // FIXTURE: wp-rest-get — authenticated GET to WP REST API @@ -457,146 +349,23 @@ The `e2e-qa-tester` extracts `results.json` from each session and formats it as | assert-abilities | ![abilities](https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/p0-a-assert-abilities.png) | ``` -### How screenshots get published - -Screenshots from Canary sessions are PNGs in `~/.canary/sessions/<id>/screenshots/`. The `e2e-qa-tester` copies them to `.e2e-screenshots/`, commits them to a temp commit (same as the current agent pattern), gets the SHA-based raw URL, then removes them from tracking: - -```bash -# Copy Canary screenshots into the publishable location -cp ~/.canary/sessions/<id>/screenshots/*.png .e2e-screenshots/ - -# Publish -git add -f .e2e-screenshots/ -git commit -m "chore(qa): Canary QA screenshots" -git push -SHA=$(git rev-parse HEAD) -# URL: https://raw.githubusercontent.com/wp-media/imagify-plugin/$SHA/.e2e-screenshots/<filename> - -# Remove from tracking -git rm --cached .e2e-screenshots/*.png -git commit -m "chore(qa): remove Canary QA screenshots" -git push -``` - --- ## 6. Playwright spec translation -Each Canary step script (QuickJS) translates to a Playwright spec (Node.js TypeScript) for CI. The translation is NOT automatic — `e2e-qa-tester` writes the spec from scratch using the Canary session's step logic as reference. Key differences: +`canary-e2e` writes fresh TypeScript specs — never a mechanical transpile of QuickJS steps: | Canary QuickJS | Playwright TypeScript | |---|---| | `browser.getPage("main")` | `page` fixture from `test()` | -| `page.evaluate(async () => fetch(...))` | `page.evaluate(async () => fetch(...))` — same | +| `page.evaluate(async () => fetch(...))` | identical | | `console.log("PASS: ...")` | `expect(...).toBe(...)` | -| `process.exit(1)` | throw / assertion failure | | No `import` | `import { test, expect } from '@playwright/test'` | | Inlined helpers | POM methods in `Tests/e2e/pages/` | -Example translation of the `assert-all-abilities-present` Canary step: - -```typescript -// Tests/e2e/specs/mcp-abilities.spec.ts -import { test, expect } from '@playwright/test'; - -test.describe('MCP Abilities — Wave 2', () => { - test.beforeEach(async ({ page }) => { - await page.goto('/wp-login.php'); - await page.locator('#user_login').fill('admin'); - await page.locator('#user_pass').fill('admin'); - await page.locator('#wp-submit').click(); - await page.waitForURL('**/wp-admin/**'); - }); - - test('all Wave 2 abilities present in /wp-abilities/v1/abilities', async ({ page }) => { - await page.goto('/wp-admin/'); - const result = await page.evaluate(async () => { - const nonceResp = await fetch('/wp-admin/admin-ajax.php?action=rest-nonce', { credentials: 'same-origin' }); - const nonce = (await nonceResp.text()).trim(); - const resp = await fetch('/wp-json/wp-abilities/v1/abilities', { - headers: { 'X-WP-Nonce': nonce }, - credentials: 'same-origin' - }); - return { status: resp.status, body: await resp.text() }; - }); - - expect(result.status).toBe(200); - const data = JSON.parse(result.body); - expect(Array.isArray(data)).toBe(true); - const slugs = data.map((a: { name: string }) => a.name); - - for (const slug of ['imagify/bulk-optimize', 'imagify/generate-missing-nextgen', 'imagify/restore-media']) { - expect(slugs).toContain(slug); - } - expect(result.body).not.toContain('Fatal error'); - }); -}); -``` - ---- - -## 7. GitHub Actions integration - -### Playwright CI (already exists — no change needed) - -The committed specs in `Tests/e2e/specs/` continue to run via the existing Playwright CI job. Canary sessions run locally only. - -### Optional: Canary artifact upload - -If CI boots `wp-env` to record fresh Canary sessions in CI (vs just replaying Playwright specs), add this to the existing E2E workflow: - -```yaml -- name: Upload Canary QA artifacts - if: always() - uses: actions/upload-artifact@v4 - with: - name: canary-qa-${{ github.event.pull_request.number }} - path: ~/.canary/sessions/ - retention-days: 14 -``` - -This is optional for the initial implementation — the local-only approach is simpler. - ---- - -## 8. Open questions for the audit - -1. **Artifact location in CI:** `~/.canary/sessions/` is per-machine (on dev). If CI also needs to produce Canary sessions (not just replay Playwright specs), where do artifacts land? Path may need to be overridden via `CANARY_HOME` or equivalent env var. - -2. **Spec generation strategy:** Does `e2e-qa-tester` translate the Canary step scripts into Playwright specs mechanically, or write fresh specs that pass the same assertions? The latter is safer (QuickJS ≠ Node, different error handling semantics). - -3. **Session agent trust guard:** The current Canary marketplace `session-agent` and `verify-agent` refuse to act on relayed confirmations — require a direct user message. Our `canary-wp-session-agent` embedded in the pipeline must not have this guard, since it's invoked by `e2e-qa-tester`, which is invoked by `qa-engineer`, which is invoked by `orchestrator`. The guard must be removed for the pipeline variant. - -4. **CI vs local Canary:** Should CI boot `wp-env` and run the Canary session scripts directly (produces fresh artifacts), or only run the committed Playwright specs (fast, no Canary dependency)? Recommendation: local-only for Canary in initial impl; CI runs Playwright specs only. - -5. **WP fixture scope:** The 6 snippets in section 4 cover the core patterns seen in live Imagify sessions. Are `wp-block-editor`, `wp-media-library-upload`, or `wp-multisite` patterns needed for any planned Imagify E2E flows? Audit the current `Tests/e2e/specs/` to find patterns not yet covered. - ---- - -## 9. Files to create / modify - -**User-level** (`~/.claude/agents/` — shared across all WP plugin repos): - -| File | Action | Notes | -|------|--------|-------| -| `~/.claude/agents/canary-wp-session-agent.md` | Create | Fork of marketplace `session-agent.md` + WP-generic fixtures only; coordinator trust guard removed for pipeline use | -| `~/.claude/agents/canary-wp-verify-agent.md` | Create | Fork of marketplace `verify-agent.md` + WP QA plan format | - -**Project-level** (`.claude/agents/` — Imagify only): - -| File | Action | Notes | -|------|--------|-------| -| `.claude/agents/canary-imagify-session-agent.md` | Create | Extends `canary-wp-session-agent`; adds Imagify admin URLs, ability slugs, `imagify-abilities` fixture | -| `.claude/agents/canary-e2e.md` | Create | New E2E agent — same JSON contract as `e2e-qa-tester`, uses Canary CLI internally | -| `.claude/agents/qa-engineer.md` | Modify | Add: write `.ai/qa-plan.md`; receive `e2e_mode` from orchestrator; route to `canary-e2e` or `e2e-qa-tester` accordingly | -| `.claude/agents/e2e-qa-tester.md` | **Untouched** | Remains the stable default — no changes | -| `.claude/skills/orchestrator/SKILL.md` | Modify | Add `e2e_mode` prompt to the startup calibration step; pass it in every agent dispatch | -| `.github/workflows/e2e.yml` | Modify (optional) | Add Canary artifact upload step if CI runs Canary sessions | -| `Tests/e2e/` | Existing | Playwright specs still committed here — no structural change | - --- -## 10. Known Canary limitations (discovered in live sessions) +## 7. Known Canary limitations (discovered in live sessions) - **`console.log` in QuickJS counts as "consoleErrors"** if the Playwright daemon routes all console output through the error channel. Do not use `summary.consoleErrors > 0` as a FAIL signal — check `summary.stepsFailed` instead. - **`networkFailures`** in the summary counts network events that returned 4xx/5xx — WP admin always generates some (favicon 404, etc.). Do not use `networkFailures > 0` as a FAIL signal. diff --git a/.claude/skills/testrail-run/SKILL.md b/.claude/skills/testrail-run/SKILL.md index b6580cd8..ee0f3bf0 100644 --- a/.claude/skills/testrail-run/SKILL.md +++ b/.claude/skills/testrail-run/SKILL.md @@ -11,10 +11,6 @@ Canary browser automation (sequentially), collects pass/fail/blocked outcomes wi trace/video evidence, prints a results table, and — only after the user confirms — posts the results back to TestRail. -This skill is a thin dispatcher. All fetching, execution, and posting logic lives in the -`testrail-run-agent`. The skill's only job is to resolve which run to target and spawn the -agent with it. - ## Invocation ``` diff --git a/.claude/skills/testrail-scenarios/SKILL.md b/.claude/skills/testrail-scenarios/SKILL.md index 42f949bd..5b87fa5b 100644 --- a/.claude/skills/testrail-scenarios/SKILL.md +++ b/.claude/skills/testrail-scenarios/SKILL.md @@ -10,10 +10,6 @@ PR, drafts a comprehensive set of test cases (happy path, failures, edge cases, guards), stages them as YAML for human review, and — on an explicit `publish` command — creates the cases in TestRail with deduplication. -This skill is a thin dispatcher. All analysis, generation, and publication logic lives in -the `testrail-scenario-agent`. The skill's only job is to resolve the PR list and spawn the -agent with it. - ## Invocation ``` diff --git a/.claude/testrail-automation.md b/.claude/testrail-automation.md deleted file mode 100644 index 90e6291b..00000000 --- a/.claude/testrail-automation.md +++ /dev/null @@ -1,234 +0,0 @@ -# TestRail Release Automation — Brainstorm & Plan - -## Goal - -Automate the TestRail side of release QA for Imagify: -1. **Case generation** — analyze what's in a release, generate test scenarios, stage for human review, push to TestRail. -2. **Case execution** — fetch a test run for a given release, run every case via browser automation, mark pass/fail in TestRail. - -QA is currently a bottleneck because coverage is limited to "what changed." This system aims to run the full suite automatically, expanding coverage without expanding human time. - ---- - -## Skill 1: `/testrail-release-setup` - -### Flow - -``` -1. git log v{last_tag}..HEAD → list merged PRs -2. For each PR: check TestRail for existing cases (deduplication) -3. Agent analyzes new PRs only → generates staged test cases -4. Write to .ai/testrail/pending/ (one YAML file per PR/feature) -5. Print summary to user -6. Prompt: "Proceed to create these cases in TestRail? (y/n)" -7. On confirm: - a. Create new sections if needed - b. POST cases to TestRail (with refs = PR URL) - c. Print created case IDs - d. Clean .ai/testrail/pending/ -``` - -### Deduplication - -**Strategy:** use TestRail's `refs` field as the link between a PR and its cases. - -- When a case is created, `refs` is set to the GitHub PR URL (e.g. `https://github.com/wp-media/imagify-plugin/pull/1133`). -- On the next run, for each PR in scope, query: `GET /get_cases/3?refs=<pr_url>` -- If cases are found → skip that PR. Print: `⏭ Skipped PR #1133 — 4 cases already exist` -- If no cases found → generate and stage. - -**Edge cases:** -- PR behavior changed after initial test creation → `--force-pr=1133` flag regenerates and stages new cases for that PR (does not delete old ones — manual cleanup in TestRail). -- PR was a pure refactor with no tests → skip silently (agent decides during analysis). - -> Note: verify that TestRail's `get_cases` API supports `refs` filtering. If not, fallback: query all cases in the relevant section and match by title prefix or a stored local index at `.ai/testrail/index.json` (persists between runs, not cleaned with `pending/`). - -### Test Case Generation (per PR) - -The agent reads the PR description, diff summary, and linked issue spec. It generates cases covering: - -- **Happy path** — standard user, default settings, expected flow -- **Happy path variants** — relevant settings combinations (e.g. WebP on/off, different plans) -- **Missing prerequisites** — no API key, quota exceeded, wrong plan/license -- **Network/API failures** — timeout, 5xx, malformed response -- **Edge cases** — empty state, max limits, unexpected data -- **Permission levels** — admin vs editor vs subscriber if relevant -- **Plugin conflicts** — relevant 3rd party plugins if the change touches integration points -- **Regression guard** — at least one case checking that the previous behavior still works if this is a fix - -The more automatable the test, the better — more cases = more Playwright coverage later. No artificial limit on number of cases per PR. - -Each case is also evaluated: **is this a smoke test?** (`custom_smoketest: true`) if it covers a critical path that should always pass regardless of release. - -### Staging File Format (`.ai/testrail/pending/<pr-slug>.yml`) - -```yaml -pr: 1133 -pr_url: https://github.com/wp-media/imagify-plugin/pull/1133 -pr_title: "Add MCP + Abilities to Imagify" -section_id: 7685 # existing section (API Requests) -new_section: "MCP Abilities" # subsection to create under section_id (null if not needed) - -cases: - - title: "MCP - Discover abilities - happy path" - smoke_test: true - preconditions: | - - Imagify plugin active - - Valid API key configured - steps: - - action: "Navigate to Settings > Imagify. Ensure API key is saved." - expected: "Settings page loads. API key field shows a valid key." - - action: "Open Claude Desktop or an MCP-compatible client. Connect to the Imagify MCP server." - expected: "Connection succeeds. No error." - - action: "Call the discover-abilities tool." - expected: "Returns a list of available abilities (e.g. optimize-image, bulk-optimize)." - - - title: "MCP - Discover abilities - invalid API key" - smoke_test: false - preconditions: | - - Imagify plugin active - - Invalid or missing API key - steps: - - action: "Connect MCP client. Call discover-abilities with no valid API key set." - expected: "Returns an error response indicating authentication failure." -``` - -### Section Mapping - -The agent picks the closest existing section. If none fits, it creates a new subsection under `Regression`. - -| Feature area | Section to use | -|---|---| -| MCP / Abilities | Regression > API Requests (7685) → new: MCP Abilities | -| Settings | Regression > Settings (32) → appropriate subsection | -| Bulk optimization | Regression > Bulk Optimization (3766) | -| Media library | Regression > Media library (4976) | -| WebP/AVIF | Regression > Settings > optimization > next-gen | -| Smoke tests | Regression > Smoke test (4525) (additional flag, not separate section) | -| New feature, no match | Create new subsection under Regression (33) | - ---- - -## Skill 2: `/testrail-release-process <version>` - -### Flow - -``` -1. Fetch milestone by version name → GET /get_milestones/3 -2. Get test run(s) for that milestone → GET /get_runs/3?milestone_id=<id> -3. Get all test cases in run → GET /get_tests/<run_id> -4. For each case (in parallel where possible): - a. Read preconditions + steps (custom_steps_separated) - b. Claude drives browser via Playwright MCP (wp-env) - c. Per step: execute, observe, capture screenshot - d. Determine: pass / fail / blocked -5. POST results to TestRail → POST /add_result_for_case/<run_id>/<case_id> - - Include step-level results and screenshot URLs as comment -6. Print summary: X passed, Y failed, Z blocked -``` - -### Execution Environment - -- `wp-env` (local or CI) — consistent, reproducible -- Local: `npx wp-env start` before running the skill -- CI: already part of the pipeline -- Screenshots stored temporarily, URLs included in TestRail result comment as evidence - -### Result Posting - -```json -{ - "status_id": 1, // 1=Passed, 5=Failed, 2=Blocked - "comment": "Automated run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\nStep 3: ❌ Failed — element not found\n\n[Screenshot](url)", - "elapsed": "45s" -} -``` - -### Failure handling - -- If a step fails → case marked failed, remaining steps skipped, failure details captured -- If wp-env is unreachable → case marked blocked with reason -- If Claude can't interpret a step → case flagged for manual review (not marked failed) - -### Canary integration (future) - -- Phase 2a (now): Claude executes every time (inference cost per run) -- Phase 2b (later): Canary records the Playwright script on first execution → subsequent CI runs replay without Claude → zero inference cost at scale - ---- - -## TestRail API Reference - -**Base URL:** `https://wpmediaqa.testrail.io/index.php?/api/v2/` -**Auth:** Basic (`TESTRAIL_USERNAME:TESTRAIL_API_KEY`) -**Project ID:** 3 -**Suite ID:** 3 -**Step template ID:** 2 - -| Action | Endpoint | -|---|---| -| Find cases by PR URL | `GET /get_cases/3?refs=<url>` | -| List sections | `GET /get_sections/3&suite_id=3` | -| Create section | `POST /add_section/3` | -| Create case | `POST /add_case/<section_id>` | -| List milestones | `GET /get_milestones/3` | -| List runs for milestone | `GET /get_runs/3?milestone_id=<id>` | -| Get tests in run | `GET /get_tests/<run_id>` | -| Post result | `POST /add_result_for_case/<run_id>/<case_id>` | - -**Case payload (Step template):** -```json -{ - "title": "MCP - Discover abilities - happy path", - "template_id": 2, - "type_id": 7, - "priority_id": 2, - "refs": "https://github.com/wp-media/imagify-plugin/pull/1133", - "custom_smoketest": false, - "custom_preconds": "<p>- Imagify active<br>- Valid API key</p>", - "custom_steps_separated": [ - { - "content": "Navigate to Settings > Imagify.", - "expected": "Settings page loads." - } - ] -} -``` - -**Known section IDs:** -``` -Regression (33) - ├── Settings (32) - │ ├── general settings (7689) - │ └── optimization (7690) - │ ├── next-gen (7694) → Webp (7697), Avif (7698) - │ └── file optimization (7695) - ├── Smoke test (4525) - ├── API Requests (7685) ← MCP goes here - ├── Media library (4976) - ├── Bulk Optimization (3766) - ├── 3rd party compatibility (4980) - │ ├── Woocommerce (7686) - │ └── WP Rocket (7687) - ├── Action scheduler (4975) - └── Promotions (1082) -``` - ---- - -## Open Questions / Decisions Needed - -1. **`refs` API filter** — confirm `GET /get_cases?refs=<url>` works in this TestRail version. If not, use local index file. -2. **Section creation** — agent creates sections automatically or always proposes in the staging file for human review? (Safer: propose in YAML, create on confirm.) -3. **Multiple runs per milestone** — if a milestone has multiple test runs (e.g. smoke + full), which one does `/testrail-release-process` target? Flag: `--run-id=1281` or pick latest. -4. **CI vs local for Phase 2** — does Phase 2 run as a skill locally, or as a CI job triggered at release time? -5. **Mixpanel/analytics changes** — pure tracking changes with no user-visible behavior: skip test generation or generate a minimal "event fires" case? - ---- - -## Implementation Order - -1. **Phase 1a** — `/testrail-release-setup` (full flow: git → analyze → stage → confirm → create) -2. **Phase 1b** — deduplication via `refs` field -3. **Phase 2a** — `/testrail-release-process` (Claude executes, posts results) -4. **Phase 2b** — Canary integration for script recording + CI replay From ecfedaed688cc0bd9d462b46a9564468941e1730 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Tue, 30 Jun 2026 04:37:18 +0200 Subject: [PATCH 05/16] feat(qa): add TestRail reviewer agent + A+ execution design doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a TestRail scenario reviewer that runs between generation and publish: - testrail-review-agent + /testrail-review skill: prune low-signal cases, collapse near-duplicates, fix smoke-test flags, and enforce cross-file section coherence; moves reviewed YAML pending/ -> reviewed/. - testrail-scenario-agent: add "quality over quantity" rules (no unit-test runner cases, no source-inspection cases, no near-duplicates, smoke-test discipline); publish mode now reads from reviewed/. - testrail-scenarios skill: offer the reviewer after staging; publish reads reviewed/. Add .claude/testrail/A-plus-design.md — the design for making the TestRail -> browser execution step reliable (grounded Explorer map + state isolation + anti-hallucination guards + TestRail expected-results as the assertion oracle). Design only; implementation deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .claude/agents/testrail-review-agent.md | 215 +++++++++++ .claude/agents/testrail-scenario-agent.md | 53 ++- .claude/skills/testrail-review/SKILL.md | 43 +++ .claude/skills/testrail-scenarios/SKILL.md | 16 +- .claude/testrail/A-plus-design.md | 417 +++++++++++++++++++++ 5 files changed, 737 insertions(+), 7 deletions(-) create mode 100644 .claude/agents/testrail-review-agent.md create mode 100644 .claude/skills/testrail-review/SKILL.md create mode 100644 .claude/testrail/A-plus-design.md diff --git a/.claude/agents/testrail-review-agent.md b/.claude/agents/testrail-review-agent.md new file mode 100644 index 00000000..4af0e780 --- /dev/null +++ b/.claude/agents/testrail-review-agent.md @@ -0,0 +1,215 @@ +--- +name: testrail-review-agent +description: Reviews staged TestRail scenario YAML files for the Imagify plugin. Assesses case pertinence, coverage gaps, redundancy, step clarity, smoke-test accuracy, and section targeting. Edits the YAML directly and prints a review report. +tools: [Bash, Read, Write, Glob, Grep] +maxTurns: 30 +color: yellow +--- + +# TestRail Review Agent + +You are a senior QA engineer with deep knowledge of the Imagify WordPress image-optimization +plugin. Your job is to review staged TestRail scenario YAML files (under `.ai/testrail/pending/`) +**before** they are published, and to improve them by editing the YAML directly. + +You do NOT call the TestRail API. You do NOT publish anything. + +## Core principle — quality over quantity + +Your primary job is **removal and sharpening**, not addition. A lean file with 4 precise +cases is better than a bloated file with 12. Before keeping a case, ask: "Would a tester +catching a regression here be impossible without this test?" If the answer is no, cut it. + +If a file covers a PR that has no real functional change observable by a human or Canary +(docblock fix, unit-test-only PR, CSS tweak), delete all its cases and leave the file +empty with a `# skipped: no functional change` comment — or remove the file entirely and +note it in your report. + +--- + +## What you receive + +Either: +- A list of specific filenames to review (e.g. `1133-add-mcp-abilities.yml`), or +- No list → review **all** files currently under `.ai/testrail/pending/` + +--- + +## Imagify plugin context + +Imagify is a WordPress image-optimization plugin. Key feature areas: + +- **MCP / Abilities API** — REST-exposed abilities (OptimizeMedia, GetMediaStatus, + GetSettings, UpdateSettings, GetNextgenCoverage, GetStats, GetAccount). Each has a slug, + `check_permissions()`, and `execute()`. Requires WP ≥ 6.9. +- **Settings** — General settings (API key, optimization level, format), next-gen (WebP/AVIF). +- **Media library** — Per-image optimize/restore actions in WP admin. +- **Bulk optimization** — Queue-based bulk processing via Action Scheduler. +- **3rd-party** — WooCommerce product images, WP Rocket compatibility. +- **Capability model** — `imagify_capacity` filter gates all privileged actions; roles: + administrator (full access), editor (limited), subscriber (no access). + +Use this context when judging whether a case makes sense or is missing important scenarios. + +### TestRail section map (suite 3) + +``` +Regression (33) + ├── Settings (32) + │ ├── general settings (7689) + │ └── optimization (7690) → next-gen (7694), file optimization (7695) + ├── Smoke test (4525) + ├── API Requests (7685) ← MCP / Abilities / REST endpoints + ├── Media library (4976) + ├── Bulk Optimization (3766) + ├── 3rd party compatibility (4980) + │ ├── Woocommerce (7686) + │ └── WP Rocket (7687) + ├── Action scheduler (4975) + └── Promotions (1082) +``` + +--- + +## Review criteria + +For every file, evaluate each case against these criteria. Apply fixes directly — do not +just annotate. + +### 1. Pertinence +Remove cases that: +- Test internal implementation details rather than user-observable behaviour. +- Duplicate what a unit/integration test already guards (e.g. "check that method X is + called") unless a human tester can actually observe the outcome. +- Are so trivial they add no signal (e.g. "page loads without error" alone, unless it is + the smoke test). + +### 2. Coverage gaps +Add cases for obvious missing scenarios. Typical gaps: +- The primary **error path** when the happy path is covered but the failure is not. +- **Permission denial** — a non-admin trying the same action. +- **Missing prerequisite** — e.g. no API key configured, plugin deactivated mid-flow. +- **Boundary / edge** — zero, null, very large, special characters. + +Do not bloat the file — add only high-signal gaps, maximum 3 new cases per file. + +### 3. Redundancy +Merge or remove cases that are near-identical (same action, same expected outcome, trivially +different input). If two cases differ only by a field value, collapse them into one case with +a note in `preconditions`. + +### 4. Step clarity +Each step must be executable by a human tester or by Canary without guesswork: +- `action` must be a concrete UI action or API call, not a vague instruction like "use the + feature". +- `expected` must be a specific, observable outcome, not "it works" or "success". +- `preconditions` must list everything the tester must set up before step 1. + +Fix unclear steps in place. + +### 5. Smoke test flags +`smoke_test: true` should be set on the **one or two** cases that represent the absolute +critical path (the plugin is broken without this). It should NOT be set on edge cases, +error paths, or permission checks. Correct misflags. + +### 6. Section targeting +Verify that `section_id` and `new_section` are correct for the PR's content using the +section map above. If a case clearly belongs to a different section than the rest, note it +in your report (do not split the file — that is the publisher's job). + +### 7. Title conventions +Titles must: +- Start with a feature-area prefix matching the section (e.g. `MCP - `, `Settings - `, + `Bulk - `, `Media - `). +- Be concise (≤ 80 chars) and describe the scenario, not the step. +- Be unique within the file. + +### 8. Cross-file section coherence + +After reviewing individual files, do a **global pass** across all files being reviewed +together. Ask: do the `section_id` and `new_section` fields form a coherent, navigable +structure in TestRail? + +Rules: +- **Group related files under a shared subsection.** If several files cover the same + feature area (e.g. 10 files all about MCP Abilities, or 3 files all about Mixpanel + tracking), they must share the same `new_section` name — only the first file in + publication order creates it; the others must reuse it. Set `new_section: null` on the + followers and update their `section_id` to reference the parent where the new section + will be created. Add a comment `# section created by PR <N>` so the publisher knows the + dependency order. +- **Subsection naming** should reflect the feature area, not the PR title. Prefer short + noun phrases: `MCP Abilities`, `Mixpanel Tracking`, `Next-gen`, `Bulk Optimization`. + Avoid repeating the parent section name in the subsection (not `API Requests - MCP`, + just `MCP Abilities`). +- **Don't over-split.** If two files cover the same feature from different angles (e.g. + a feature PR and its bug-fix follow-up), they belong in the same subsection — not two + separate ones. +- **Don't under-group.** If files cover clearly distinct feature areas (e.g. MCP and + Mixpanel), keep them in separate subsections even if both land under the same parent. +- **Update title prefixes** in each file to match the final subsection name + (e.g. if the subsection is `MCP Abilities`, titles should start with `MCP - `). + +Report the grouping decisions in the review report under a `## Section coherence` heading +before the per-file breakdown. + +--- + +## Workflow + +1. **List files to review.** + ```bash + ls .ai/testrail/pending/ + ``` + If none, say so and stop. + +2. **Read all files first** — before making any edits, read every file to understand the + full set of feature areas, section targets, and `new_section` values. This is the input + for the cross-file coherence pass. + +3. **Cross-file coherence pass** (criterion 8) — determine the correct grouping and + subsection names across all files. Produce a mapping: + `filename → { section_id, new_section }` for every file. This mapping takes precedence + over whatever was in the original YAML. + +4. **Ensure the `reviewed/` directory exists.** + ```bash + mkdir -p .ai/testrail/reviewed/ + ``` + +5. **For each file:** + a. Apply all per-file fixes (criteria 1–7). + b. Apply the section/new_section from the coherence mapping (criterion 8). + c. Update title prefixes if the subsection name changed. + d. Write the updated content to `.ai/testrail/reviewed/<filename>`. + e. Delete the original from `.ai/testrail/pending/<filename>`. + f. Build a short per-file diff summary (what was changed and why). + +6. **Print the review report** in this order: + - `## Section coherence` — the grouping decisions: which files share a subsection, + what the subsection names are, and which file creates each new section. + - Per-file breakdown: + ``` + ### <filename> + - Removed: <case title> — <reason> + - Added: <case title> — <reason> + - Fixed: <case title> — <what changed> + - No change: <N> cases unchanged + → Moved to .ai/testrail/reviewed/ + ``` + End with a one-line overall verdict per file: **Ready to publish** or **Review needed**. + +7. **Stop.** Do not publish. Tell the user they can run `/testrail-scenarios publish` when + satisfied, or edit the YAML files under `.ai/testrail/reviewed/` directly. + +--- + +## DO NOT + +- DO NOT call the TestRail API. +- DO NOT publish. +- DO NOT add `# REVIEW:` comments unless you are genuinely uncertain — prefer making the + call yourself. +- DO NOT rewrite cases that are already clear and correct — leave them untouched. +- DO NOT delete a file from `pending/` unless its reviewed copy was successfully written + to `reviewed/`. diff --git a/.claude/agents/testrail-scenario-agent.md b/.claude/agents/testrail-scenario-agent.md index fdca38c2..850975f2 100644 --- a/.claude/agents/testrail-scenario-agent.md +++ b/.claude/agents/testrail-scenario-agent.md @@ -8,8 +8,18 @@ color: blue # TestRail Scenario Agent -You analyse GitHub PRs for the Imagify WordPress plugin and turn each into a comprehensive -set of TestRail test scenarios. You operate in two distinct, user-gated modes: +You analyse GitHub PRs for the Imagify WordPress plugin and turn each into a **targeted, +high-signal** set of TestRail test scenarios. You operate in two distinct, user-gated modes: + +## Core principle — quality over quantity + +Before writing a single case, ask: **"Would a tester catching a regression here be +impossible without this test?"** If the answer is no, don't write it. + +TestRail is not a log of what the code does. It is a safety net for things that could +silently break. Prefer 3 sharp cases over 10 vague ones. A PR that only changes a docblock, +adds unit tests, or tweaks a CSS colour needs **zero TestRail cases** — skip it entirely and +say why in your summary. - **Generate mode** (default): analyse PRs → draft scenarios → stage them as YAML under `.ai/testrail/pending/` → print a summary → **stop and await human review.** You do NOT @@ -132,6 +142,39 @@ Each case must be concrete and executable by a human or by Canary. Use the **Ste (`template_id: 2`) with discrete action/expected pairs. Flag `smoke_test: true` only for the critical-path cases that must pass on every build. +#### What NOT to generate + +These case types are consistently invalid and must never appear in the output: + +- **Unit / integration test runner cases** — any case whose `action` would be running + `composer test-unit`, `phpunit`, `wp-env run`, or any CLI test command. If a behaviour is + only verifiable by running the test suite, it belongs in the test suite, not in TestRail. + Ask instead: "can a human tester or Canary observe this outcome via the UI or a REST/MCP + call?" If no, skip it. + +- **Source code inspection cases** — cases that require reading a PHP file, checking a + docblock, grepping for a method name, or verifying hook wiring by looking at source. + These belong in code review, not functional QA. Observable proxy: if the feature works + correctly end-to-end, source accuracy is implied. + +- **Near-duplicate cases** — when two or more cases differ only by a single input value + (e.g. post rejected / page rejected / CPT rejected; session persistence / reload + persistence), collapse them into one multi-step case covering all variants. Three similar + rejection inputs → one case with three steps. Two persistence scenarios → one case with + two steps. + +- **Vague regression catch-alls** — cases with steps like "upload an image and verify it + still works" or "save settings and confirm the page loads". Write a regression case only + if you can name the specific adjacent behaviour at risk and the concrete expected outcome. + +#### Smoke test discipline + +`smoke_test: true` means: **the plugin is broken without this passing.** Apply it sparingly: +- Maximum 1–2 smoke cases per PR (the happy path and its most critical variant). +- Never on: error paths, permission denial cases, edge cases, security assertions, + regression guards, or any case that is not the absolute primary success flow. +- When in doubt, leave `smoke_test: false`. + ### Step 4 — Section mapping For each PR, choose a `section_id` from the section map. If no existing section fits, set @@ -181,7 +224,9 @@ Then **stop**. Tell the user to review the YAML under `.ai/testrail/pending/` an ## Publish mode workflow -Run only when explicitly told to publish. For each YAML file under `.ai/testrail/pending/`: +Run only when explicitly told to publish. For each YAML file under `.ai/testrail/reviewed/`. +If `reviewed/` is empty or does not exist, check `pending/` and warn the user that those +files have not been through the reviewer yet, then ask whether to proceed anyway. ### Step A — Create the section if needed @@ -248,7 +293,7 @@ After all cases for a file succeed, print the created case IDs grouped by PR, th YAML file: ```bash -rm ".ai/testrail/pending/<pr-slug>.yml" +rm ".ai/testrail/reviewed/<pr-slug>.yml" ``` If any `add_case` call fails (non-2xx or no `id` in the response), do NOT delete the file — diff --git a/.claude/skills/testrail-review/SKILL.md b/.claude/skills/testrail-review/SKILL.md new file mode 100644 index 00000000..a8bad5bc --- /dev/null +++ b/.claude/skills/testrail-review/SKILL.md @@ -0,0 +1,43 @@ +--- +name: testrail-review +description: Review staged TestRail scenario YAML files before publishing — checks pertinence, coverage gaps, redundancy, step clarity, smoke-test flags, and section targeting. Edits the YAML directly. +argument-hint: [filename(s) or blank for all] +--- + +# TestRail Review + +Standalone entry point for reviewing staged TestRail scenario YAML files before they are +published. Spawns `testrail-review-agent` on whatever is staged (or a specific subset). + +## Invocation + +``` +/testrail-review → review all files in .ai/testrail/pending/ +/testrail-review 1133-add-mcp-abilities.yml → review one specific file +/testrail-review 1133-add-mcp-abilities.yml 1149-fix-capability.yml → review a subset +``` + +## What to do + +1. **Resolve the file list.** + - If filenames are given as arguments, pass them to the agent. + - If no arguments, pass no list — the agent will review everything staged. + +2. **Check that there is something to review.** + ```bash + ls .ai/testrail/pending/ 2>/dev/null + ``` + If the directory is empty or does not exist, tell the user there is nothing staged and stop. + +3. **Spawn `testrail-review-agent`**, passing the file list (or "review all staged files" + if no list). The agent will edit the YAML directly and print a review report. + +4. **Relay the agent's report** verbatim, then remind the user: + - To publish: `/testrail-scenarios publish` (reads from `.ai/testrail/reviewed/`) + - To generate more scenarios: `/testrail-scenarios #PR` + - To edit manually: reviewed files are in `.ai/testrail/reviewed/` + +## Constraints + +- This skill never edits YAML directly. All edits are the agent's. +- Never publish automatically — this skill only reviews. diff --git a/.claude/skills/testrail-scenarios/SKILL.md b/.claude/skills/testrail-scenarios/SKILL.md index 5b87fa5b..10620a57 100644 --- a/.claude/skills/testrail-scenarios/SKILL.md +++ b/.claude/skills/testrail-scenarios/SKILL.md @@ -57,13 +57,23 @@ You may also pass full PR URLs instead of `#`-prefixed numbers; both are accepte - To publish: `/testrail-scenarios publish` (or "publish the staged scenarios"). - To revise: edit the YAML files under `.ai/testrail/pending/` directly, then publish. +6. **Offer the reviewer.** After relaying the summary, ask the user: + > "Would you like the TestRail reviewer to check the staged scenarios before publishing? + > It will flag low-signal cases, fill coverage gaps, and clean up step wording. + > Run `/testrail-review` or reply **yes** to launch it." + If the user replies yes (or any affirmative), spawn `testrail-review-agent` immediately + with "review all staged files" and relay its report. Otherwise proceed — the reviewer + is optional and skipping it is fine. + ## Publish mode When invoked as `/testrail-scenarios publish`, spawn `testrail-scenario-agent` in **publish -mode**: instruct it to read every YAML file under `.ai/testrail/pending/`, create the +mode**: instruct it to read every YAML file under `.ai/testrail/reviewed/`, create the sections and cases in TestRail, print the created case IDs, and delete each published YAML -file. Pass no PR list in this mode — the agent operates on whatever is staged. If nothing is -staged, the agent should say so and stop. +file. Pass no PR list in this mode — the agent operates on whatever is in `reviewed/`. +If `reviewed/` is empty but `pending/` has files, the agent warns the user those files +haven't been reviewed and asks whether to proceed anyway. If both are empty, the agent +should say so and stop. ## Constraints diff --git a/.claude/testrail/A-plus-design.md b/.claude/testrail/A-plus-design.md new file mode 100644 index 00000000..ee1f6e60 --- /dev/null +++ b/.claude/testrail/A-plus-design.md @@ -0,0 +1,417 @@ +# TestRail Automation — "A+" Execution Design + +**Status:** Design / pre-build · **Date:** 2026-06-30 · **Author:** Gaël Robin (+ Claude) + +Single source of truth for how we make the TestRail → browser execution step **reliable**. +Consolidates findings from five independent prior-art sources (see Appendix) into one plan. + +--- + +## 1. The problem we are solving + +The fragile link in the current pipeline is one step inside `testrail-run-agent`: + +> **Translate each TestRail step (HTML-stripped prose) into a Canary/Playwright script and run it.** + +This translation happens **at run time, open-loop, with no grounding**: + +- Selectors are *guessed from prose* ("Navigate to Settings > Imagify") — they have never + touched the real DOM. +- Assertions are *inferred from expected text* ("Returns all available abilities"). +- Prerequisites are *assumed* (the agent doesn't know an API key is needed, or where it lives). +- State leaks between sequentially-run cases (`optimize-media` is destructive; settings persist). + +Anyone — not just us — authors the TestRail cases, so **pre-writing a script per case is not an +option**. The fix must work for cases we have never seen. + +--- + +## 2. The core insight (validated by 5 independent sources) + +Every serious prior-art system converged on the **same spine**, regardless of stack: + +> **Explore the running app → ground locators from *real* interactions → validate by +> compiling → gate with humans → cache the grounding for reuse.** + +Two volatilities of knowledge must be handled differently: + +| Knowledge | Volatility | DOM-observable? | Source of truth | +|---|---|---|---| +| Selectors, labels, interaction order | High | **Yes** | Captured from **live exploration** — never frozen from a code read | +| URLs, prerequisites, API-key location, "what success means", seed/teardown | Low | **No** | Light **code/docs audit**, committed | + +The original "Plan A" (Opus audits the *code* and writes specs) was right for the low-volatility +half and **wrong** for selectors — reading code to infer selectors is itself guessing. A stale, +hand-written selector spec is *worse* than none, because the agent trusts it and won't fall back +to looking at the real page. + +--- + +## 3. A+ architecture + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ /testrail-setup (Explorer — runs occasionally, output committed) │ +│ │ +│ • Drives the LIVE app via Canary, logged in │ +│ • Captures REAL locators from real clicks (role-based preferred) │ +│ • Light code/docs audit for non-DOM knowledge: │ +│ - admin URLs & how to reach each feature │ +│ - prerequisites + how to SEED them (WP-CLI / REST) │ +│ - teardown / undo for each mutation (LIFO) │ +│ - verification criteria ("success" = what, observably) │ +│ • Writes a grounded map → .claude/testrail/specs/<feature>.md │ +│ • Stamps each spec with source_files + git SHA (drift detection) │ +└─────────────────────────────────────────────────────────────────────┘ + │ (committed, shared by whole team) + ▼ +┌─────────────────────────────────────────────────────────────────────┐ +│ /testrail-run (Executor — per release QA) │ +│ │ +│ 1. Fetch run + cases from TestRail (steps + EXPECTED RESULTS) │ +│ 2. Resolve case → feature spec (via frontmatter mapping) │ +│ 3. Load _foundation.md + <feature>.md as grounding │ +│ 4. Per case: │ +│ a. SEED prerequisites via API/CLI (not UI) │ +│ b. Execute closed-loop against live DOM, grounded by the map │ +│ c. Assert against the TestRail EXPECTED RESULT (the oracle) │ +│ d. LIFO teardown │ +│ e. Capture trace/video/HAR/console (evidence) │ +│ 5. Results table → human confirm → post to TestRail │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +### Why this beats the alternatives + +- **Selectors are grounded in reality**, captured from real interactions, not prose or code. +- **The grounded map is committed & shared** — everyone runs against the same knowledge. +- **Drift is detectable** (SHA stamps), not silent. +- **TestRail expected-results are our assertion oracle** — neutralises the failure mode every + prior-art author flagged as fatal: *"the agent knows what it observed, not what's correct."* + We don't let the agent decide what "correct" means; TestRail tells it. + +--- + +## 4. The two deltas harvested from prior art + +Beyond the shared spine, two concrete mechanisms are worth lifting: + +### 4.1 Per-case state isolation (from the "scalable E2E" source) + +Our 66 cases run **sequentially in one session**, and several **mutate state** +(`optimize-media`, settings updates, Internal State Reset). Without isolation, case N pollutes +case N+1 — a silent flakiness source. + +**Contract per case:** +- **Seed prerequisites via WP-CLI / REST, not the UI** (set API key, force an attachment into + optimized/unoptimized state, toggle a setting). Faster + deterministic. +- **LIFO teardown queue** — each case pushes its undo; teardown unwinds in reverse, even after + an assertion fails. + +This is also *captured by the Explorer* — "how to seed / tear down" is part of each feature spec. + +### 4.2 Anti-hallucination ruleset (from the "write/compile/execute/ship" source) + +Encode grounding as **enforceable AUTO-REJECT guards** in the agent instructions: + +- **Inferred locator while the page is reachable → REJECT.** Must open the page and capture the + real locator. +- **Ephemeral/internal ref used as a locator → REJECT.** Convert to a stable locator + (`getByRole` preferred, then `data-testid` / `id`). +- **Tautological assertion (e.g. assert true) → REJECT.** The assertion must check the TestRail + expected result. +- **Generate only after reading 2–3 REAL existing `Tests/e2e/specs/` files** and copying their + exact patterns/base classes — never invent a foreign style. + +### 4.3 Validate before trusting generated code (reconfirmed by 2 sources) + +If/when we emit committed `.spec.ts` (phase 2), gate them: `tsc --noEmit` + ESLint against a temp +clone **before** the spec is allowed to run. Never trust un-compiled generated code. + +--- + +## 5. File / artifact layout + +``` +.claude/testrail/ ← committed, shared + A-plus-design.md ← this doc + specs/ ← grounded maps (low-volatility + locators) + _foundation.md ← login, admin URLs, test users, where the API key lives + mcp-abilities.md ← frontmatter: testrail_sections, source_files, derived_sha + mixpanel-tracking.md + settings.md + ... + +.ai/testrail/ ← run-time / staging (gitignored working area) + pending/ ← generated scenarios awaiting review + reviewed/ ← reviewed scenarios ready to publish + <run_id>/canary/<session>/ ← execution evidence (trace/video/HAR) +``` + +**Spec organisation:** by **plugin feature** (stable axis), *not* by TestRail section (which we +reshuffle). Each spec maps back via frontmatter `testrail_sections: [...]`, so the run agent +resolves section → spec with a thin lookup that survives TestRail reorg. + +--- + +## 6. Execution model — the one open decision + +The grounded Explorer/map is shared by both options below and is built first. The fork is **how +the run agent executes**: + +**Option 1 — Closed-loop Canary run (recommended first).** +Run agent drives the live browser, grounded by the map; re-explores an element if it drifts. +Evidence-rich (trace/video/HAR for TestRail), simpler, nothing to maintain but the map. + +**Option 2 — Generate validated, committed `.spec.ts` + CI replay (phase 2).** +A Writer emits Playwright specs (grounded by the map + TestRail steps), validated via §4.3, then +CI replays them fast & deterministically (`--last-failed` for reruns). Specs become real repo +artifacts; matches the existing `Tests/e2e/specs/` story. Bigger build. + +**Recommendation:** build the Explorer + grounded map, ship Option 1, layer Option 2 later — +Option 2's committed specs are only as good as the grounding, so get the grounding right first. + +--- + +## 7. Build order + +1. `_foundation.md` schema + the Explorer agent + `/testrail-setup` (and `--check` for drift). +2. Re-ground one feature end-to-end (`mcp-abilities.md`) as the reference example. +3. Update `testrail-run-agent`: load specs, state-isolation contract (§4.1), anti-hallucination + guards (§4.2), TestRail-expected-result oracle. +4. Dry-run the MCP Abilities cases; compare reliability vs. the old open-loop path. +5. (Later) Option 2 Writer + validation gate + CI replay. + +--- + +## 8. Agent design principles (build lean, not bloated) + +**Non-negotiable.** Every agent/skill in this system must follow these, or it gets rejected in +review. A bloated agent is slow, expensive, and unreliable — token weight is cognitive weight. + +1. **One agent, one job.** Explorer explores. Executor executes. Scenario generates. Reviewer + reviews. If an agent's `description` needs the word "and" to describe its job, split it. + +2. **Minimal tool grant.** Grant only the tools the job needs — fewer tools = less for the model + to weigh = cheaper, faster, safer. + - Explorer: `Bash, Read, Write, Glob, Grep` (drives Canary via Bash; **no** WebFetch). + - Executor (`testrail-run-agent`): `Bash, Read, Write, Glob, Grep`. + - Never grant a tool "just in case." + +3. **Right model for the job.** + - **Explorer → Opus or Sonnet.** Reasoning-heavy (map an app, judge what matters). Runs + occasionally, so cost is amortised — fine to pay for quality here. + - **Executor → Sonnet.** Procedural (follow spec, drive browser, assert). Runs 66×/run — + keep it cheap. Do **not** pay Opus prices for procedural work. + - Set `model:` explicitly in frontmatter; don't inherit by accident. + +4. **Push deterministic work OUT of the LLM.** HTML-stripping, JSON building, WP-CLI seeding, + SHA stamping, file moves, `curl` calls — these are Bash/python, **not** LLM reasoning. The + LLM decides *what* to do; scripts do the mechanical parts. Every token spent on string + manipulation is a token wasted and a chance to hallucinate. + +5. **Knowledge lives in specs, not in the prompt.** The agent prompt says *"resolve the case to + its feature spec and load it."* It must **not** inline the grounded map, the 7 ability slugs, + or the section IDs. Progressive disclosure: read what you need at runtime. This keeps the + agent file small and lets knowledge update without editing the agent. + +6. **DRY within the prompt.** State each rule once. One `## DO NOT` block at the end, not the + same warning sprinkled five times. + +7. **Concrete > prose.** A code block, schema, or table beats three paragraphs. The "no verbose" + QA principle applies to the agent *files* themselves. + +8. **Bounded loops.** Every agent sets `maxTurns`. Any validation/self-heal loop has an explicit + attempt cap (≤ 2 rounds) then escalates to a human — never spins. + +9. **Fail loud, escalate early.** On auth failure, missing env, or ambiguous state: stop and + report (BLOCKED), don't guess. Distinguish environment gaps from product defects. + +--- + +## 9. Component contracts + +### 9.1 `_foundation.md` schema (committed) + +```markdown +--- +derived_sha: <git SHA at capture time> +source_files: [imagify.php, ...] # for drift detection +last_explored: 2026-06-30 +--- + +## Environment +- Base URL: http://localhost:10038 +- WP admin: http://localhost:10038/wp-admin/ +- Login: admin / admin via /wp-login.php + role-based: getByLabel("Username or Email Address"), getByLabel("Password") + +## Prerequisites +- Imagify API key: lives in settings.local.json (key IMAGIFY_API_KEY). + seed via: wp option patch / define() — see Seeding helpers. +- Settings page: /wp-admin/options-general.php?page=imagify + +## Test users (for permission cases) +| Role | Login | imagify_capacity | +|---------------|----------------|------------------| +| administrator | admin / admin | full | +| editor | <seed> | limited | +| subscriber | <seed> | denied | + +## Seeding helpers (WP-CLI — run via Bash, not UI) +- Set API key: wp option patch update imagify_settings api_key "<key>" +- Force media optimized: wp ... # captured by Explorer +- Reset media: wp ... # the LIFO undo +- Toggle a setting: wp option patch update imagify_settings <k> <v> +``` + +### 9.2 Feature spec schema (committed) — concrete example `mcp-abilities.md` + +```markdown +--- +testrail_sections: [8724] # MCP Abilities (maps section → this spec) +feature: "MCP Abilities API" +source_files: [classes/Abilities/*.php, classes/AbilitiesSubscriber.php] +derived_sha: <sha> +last_explored: 2026-06-30 +--- + +## Overview +7 abilities via WP Abilities API + MCP adapter. Requires WP >= 6.9. +Capability gated by the `imagify_capacity` filter (NOT direct current_user_can). + +## Abilities (ground truth) +| Slug | Class | Destructive | +|-------------------------------|--------------------|-------------| +| imagify/get-account | GetAccount | no | +| imagify/get-settings | GetSettings | no | +| imagify/get-media-status | GetMediaStatus | no | +| imagify/get-stats | GetStats | no | +| imagify/get-nextgen-coverage | GetNextgenCoverage | no | +| imagify/optimize-media | OptimizeMedia | YES | +| imagify/update-settings | UpdateSettings | mutates | + +## How to invoke (grounded from live) +- MCP tool: mcp__imagify__mcp-adapter-execute-ability { ability, input } +- REST: <endpoint captured live by Explorer> + +## Locators (captured live — role-based preferred, then data-testid, then id) +- <filled by Explorer> + +## Prerequisites & seeding (per ability) +- get-media-status / optimize-media: need an attachment ID → seed via _foundation helper. +- update-settings: snapshot current settings first (for teardown). + +## Verification criteria — "success" means (observable) +- optimize-media: response status == "success" AND attachment gains _imagify_data meta. +- get-* : response schema complete, types correct, api_key/version absent. + +## Teardown (LIFO) +- optimize-media: restore original via <helper>. +- update-settings: re-apply the snapshot. +``` + +### 9.3 Explorer agent contract (`testrail-explorer-agent.md`) + +```yaml +name: testrail-explorer-agent +tools: [Bash, Read, Write, Glob, Grep] +model: opus # reasoning-heavy, runs occasionally +maxTurns: 60 +``` +- **Job:** drive the live app (Canary, logged in), capture real locators + seed/teardown + + verification criteria per feature, write/refresh `.claude/testrail/specs/*.md`, stamp SHAs. +- **Input:** a feature name or "all"; optional `--check` (drift report only, no writes). +- **Output:** committed spec files + a summary of what changed since last explore. +- **Anti-hallucination guards (§4.2) apply** — capture locators from real clicks, never invent. +- **DO NOT:** write selectors it did not observe; print the API key; invent ability slugs. + +### 9.4 `/testrail-setup` skill + +- `/testrail-setup` → explore everything, write/refresh all specs. +- `/testrail-setup <feature>` → re-ground one feature. +- `/testrail-setup --check` → drift report only (which specs are stale vs current SHA), no writes. +- Spawns `testrail-explorer-agent`; relays its summary. Never posts to TestRail. + +### 9.5 Changes to `testrail-run-agent` (the executor) + +Add, without bloating it (move detail into specs, keep the agent procedural): +1. **Resolve case → spec:** map the case's TestRail section to a feature spec via frontmatter + `testrail_sections`; load `_foundation.md` + that spec as grounding. +2. **State-isolation contract (§4.1):** seed prerequisites via WP-CLI/REST before the browser; + register a LIFO teardown; run teardown after each case (even on failure). +3. **Anti-hallucination guards (§4.2):** the reject rules become a single DO NOT block. +4. **Assertion oracle:** assert against the TestRail **expected result**, not "page looked ok". +5. Keep everything else (sequential, evidence capture, confirm-before-post) unchanged. + +--- + +## 10. Concrete Imagify reference data (discovered this session — don't rediscover) + +``` +Local env : http://localhost:10038 (wp-env) login admin / admin +Settings page : /wp-admin/options-general.php?page=imagify +API key : settings.local.json → IMAGIFY_API_KEY +WP requirement : >= 6.9 (abilities no-op below this) +Capability : imagify_capacity filter (not direct current_user_can) + +TestRail : https://wpmediaqa.testrail.io · project 3 · suite 3 + Sections : MCP Abilities 8724 (parent API Requests 7685) + Mixpanel Tracking 8725 (parent Regression 33) + general settings 7689 · next-gen 7694 · file optimization 7695 + Case fields : template_id 2 (Step) · type_id 7 · priority_id 2 + custom_steps_separated[{content,expected}] · custom_smoketest · custom_preconds + Result status: 1 PASS · 5 FAIL · 2 BLOCKED + Dedup key : refs = PR URL + +Canary : npx @usecanary/cli (capture trace,video,har,console) +Fixtures : .claude/agents/canary-imagify-session-agent.md (login, selectors, ability slugs) +E2E specs : Tests/e2e/specs/ · playwright.config.ts workers:1 · E2E_CI=true +MCP tools : mcp__imagify__mcp-adapter-{discover-abilities,execute-ability,get-ability-info} +``` + +7 ability slugs: `get-account, get-settings, get-media-status, get-stats, get-nextgen-coverage, +optimize-media, update-settings` (all `imagify/` prefixed, kebab-case). + +--- + +## 11. Drift detection mechanics + +- Each spec's frontmatter records `source_files` (globs) + `derived_sha` (the commit it was + captured at). +- `/testrail-setup --check`: for each spec, `git log -1 --format=%H -- <source_files>`; if any + source file's latest commit is newer than `derived_sha`, mark the spec **stale** and name the + files that changed. Output is a report only — no rewrites. +- The run agent runs a cheap `--check` at start and **warns** (does not block) if it's executing + against stale specs, so a tester knows the grounding may be behind the code. +- Re-grounding (`/testrail-setup <feature>`) re-stamps the SHA. + +--- + +## 12. What we deliberately rejected + +- **3rd-party skills** (`lackeyjb/playwright-skill`, etc.) — none do TestRail; generic + prose→Playwright can't produce working selectors for *our* app without seeing it; supply-chain + risk in a committed pipeline. +- **Freezing selectors in a hand-audited spec** — the "confidently wrong" trap. +- **Enterprise scaffolding** — 2FA/IMAP, parallel sharding, GraphQL clients, 7-agent swarms, + Allure deploy. We are `workers: 1`, local wp-env, `admin/admin`. + +--- + +## Appendix — sources triangulated (blueprints, none adopted as code) + +1. `lackeyjb/playwright-skill` — general Playwright runner; **no** TestRail (Gemini fabricated + that). Validates iterative live execution. +2. `aiqualitylab/ai-natural-language-tests` — live-DOM grounding (`--url`), pattern memory. + **AGPL-3.0** — unusable in a commercial plugin. +3. `learnautomatedtesting/playwright-ai-agents` — Planner/Generator/Healer on `.claude/agents/`. + Proves the architecture; immature (1 commit). +4. "Scalable E2E … Playwright/GraphQL/Actions" (Medium) — **API-seed + LIFO teardown** + (§4.1), role-based locators, rerun-failed-only. +5. "AI agent that writes/compiles/executes/ships E2E" (Medium) — **anti-hallucination reject + rules** (§4.2), compile-before-execute (§4.3), cached discoveries, two human gates. + +**Convergence:** all five → explore-live · ground from real interactions · validate by +compiling · gate with humans · cache the grounding. A+ = that spine + deltas §4.1 and §4.2, +plus our unique edge: **TestRail expected-results as the assertion oracle.** From 4bc0e342b2317b9cf188231bc22a9484c32b7ec1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Tue, 30 Jun 2026 08:24:43 +0200 Subject: [PATCH 06/16] =?UTF-8?q?feat(qa):=20implement=20A+=20TestRail=20e?= =?UTF-8?q?xecution=20=E2=80=94=20Explorer=20+=20grounded=20run=20agent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the A+ design (.claude/testrail/A-plus-design.md): replace the fragile run-time "prose -> Playwright" guess with grounded, closed-loop execution. - testrail-explorer-agent (NEW, opus): drives the live app via Canary, captures real locators + seed/teardown + verification criteria per feature, writes/refreshes .claude/testrail/specs/*.md stamped with source_files + derived_sha; --check drift mode writes nothing. - /testrail-setup skill (NEW): thin entry point (all / <feature> / --check). - specs/_foundation.md (NEW): shared non-DOM base, pre-filled with known constants; live rows + unverified API-key path marked "captured by Explorer". - testrail-run-agent (EDIT, sonnet): resolve case -> feature spec via whole-element section match, load _foundation + feature spec as grounding, seed prerequisites + LIFO teardown (state isolation), assert against the TestRail expected-result (oracle), warn on stale grounding. Reviewed by workflow-reviewer (NEEDS_REVISION -> fixed): anchored the section-id resolution to whole array elements (872 no longer matches 8724), canonicalised the "Ground truth" section across explorer/run-agent/design so destructive markers are actually consumed, documented _foundation as the always-loaded non-resolvable base, and moved Canary step scripts to per-session /tmp dirs. Authoring only — Explorer not yet run; mcp-abilities feature spec is the next step (needs live wp-env + Canary). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --- .claude/agents/testrail-explorer-agent.md | 165 ++++++++++++++++++++++ .claude/agents/testrail-run-agent.md | 108 ++++++++++++-- .claude/skills/testrail-setup/SKILL.md | 51 +++++++ .claude/testrail/A-plus-design.md | 2 +- .claude/testrail/specs/_foundation.md | 71 ++++++++++ 5 files changed, 382 insertions(+), 15 deletions(-) create mode 100644 .claude/agents/testrail-explorer-agent.md create mode 100644 .claude/skills/testrail-setup/SKILL.md create mode 100644 .claude/testrail/specs/_foundation.md diff --git a/.claude/agents/testrail-explorer-agent.md b/.claude/agents/testrail-explorer-agent.md new file mode 100644 index 00000000..429f6216 --- /dev/null +++ b/.claude/agents/testrail-explorer-agent.md @@ -0,0 +1,165 @@ +--- +name: testrail-explorer-agent +description: Explores the live Imagify app via Canary (logged in) and captures REAL locators, seed/teardown helpers, and verification criteria per plugin feature, then writes/refreshes grounded spec files under .claude/testrail/specs/ stamped with source_files + derived_sha for drift detection. Supports --check (drift report only, no writes). +tools: [Bash, Read, Write, Glob, Grep] +model: opus +maxTurns: 60 +color: purple +--- + +# TestRail Explorer Agent + +You build the **grounded maps** the run agent executes against. You drive the LIVE Imagify +app via Canary (logged in), capture **real** locators from real interactions, audit the code +lightly for the non-DOM knowledge (URLs, prerequisites, seed/teardown, what "success" means), +and write one spec file per plugin **feature** to `.claude/testrail/specs/`. You stamp each +spec with `source_files` + `derived_sha` so drift is detectable. + +You explore **occasionally**; your output is committed and shared by the whole team. Spend the +reasoning to get it right — but capture, never invent. A confidently-wrong selector is worse +than no selector, because the run agent will trust it and not fall back to the real page. + +## What you receive + +One of: +- a **feature name** (e.g. `mcp-abilities`, `settings`) → re-ground only that feature's spec. +- the token **`all`** → explore every feature; write/refresh every spec. +- the token **`--check`** (optionally with a feature name) → **drift report only, NO writes.** + +Specs are organised by plugin feature (the stable axis), not by TestRail section. Each spec +maps back to TestRail via frontmatter `testrail_sections`. + +## Environment & constants + +``` +Base URL : http://localhost:10038 (wp-env) login admin / admin via /wp-login.php +Specs dir : .claude/testrail/specs/ +Foundation : .claude/testrail/specs/_foundation.md (load first; non-DOM shared knowledge) +Canary CLI : npx @usecanary/cli (capture trace,video,har,console) +WP fixtures : .claude/agents/canary-imagify-session-agent.md (login + named selectors to reuse) +E2E POMs : Tests/e2e/specs/ (read 2–3 real files before writing any locator — copy their style) +``` + +Credentials live in the environment. Never print the API key. If login fails, env is missing, +or the app is unreachable: stop and report **BLOCKED** with the reason — do not guess. + +--- + +## `--check` mode (drift report only — DO THIS FIRST when asked) + +No browser, no writes. For each spec in `.claude/testrail/specs/`: + +```bash +# read frontmatter source_files + derived_sha, then for each glob: +git log -1 --format=%H -- <source_file_glob> # latest commit touching that source +``` + +If any source file's latest commit is newer than the spec's `derived_sha`, mark the spec +**STALE** and name the files that changed. Print a table (spec, status FRESH/STALE, changed +files). **Write nothing.** Then stop. + +--- + +## Explore mode workflow + +### Step 1 — Load grounding, scope the work +Read `_foundation.md`. If a feature name was given, scope to that spec; if `all`, enumerate +the features (one spec each). Read 2–3 real files from `Tests/e2e/specs/` and copy their +locator style and base patterns — do not invent a foreign style. + +### Step 2 — Drive the live app (Canary, logged in) +Use the login fixture and named selectors from `canary-imagify-session-agent.md`. For each +feature, reach it the way a user/tester does and **capture the real locator from the real +element** — prefer `getByRole` / `getByLabel`, then `data-testid`, then `id`. Write Canary +step scripts to `/tmp/canary-steps/`, run them, read what the page actually exposes. For +API/MCP-only surfaces (e.g. abilities, REST endpoints), capture the real endpoint + invocation +from a live call, not from prose. + +```bash +id=$(npx @usecanary/cli session start --name "EXPLORE: <feature>" --capture trace,video,har,console) +STEPS="/tmp/canary-steps/$id"; mkdir -p "$STEPS" # per-session dir; never collides with a run +# ... write step scripts to $STEPS that navigate + read real locators/labels/roles ... +npx @usecanary/cli session end "$id" +rm -rf "$STEPS" +``` + +### Step 3 — Light code/docs audit for non-DOM knowledge +For each feature, determine from a quick read (not a deep audit): +- admin URL(s) and how to reach the feature; +- prerequisites and **how to SEED them via WP-CLI/REST** (not the UI); +- **teardown / undo** for each mutation (LIFO order); +- verification criteria — what "success" looks like, **observably**. + +Record the files you read into `source_files` (globs are fine). + +### Step 4 — Write the spec (use the schema below VERBATIM) +Write/refresh `.claude/testrail/specs/<feature>.md`. The run agent parses these keys and +sections, so the schema is a **contract** — match it byte-for-byte. + +```markdown +--- +testrail_sections: [<id>, ...] # maps TestRail section → this spec (run agent resolves on this) +feature: "<Human feature name>" +source_files: [<glob>, ...] # for drift detection +derived_sha: <git SHA at capture time> +last_explored: <YYYY-MM-DD> +--- + +## Overview +<1–3 lines: what this feature is, hard requirements (e.g. WP >= 6.9), capability gating.> + +## Ground truth +<Stable enumerable facts captured live — e.g. an abilities table, endpoint list. Mark + destructive operations explicitly.> + +## How to invoke (grounded from live) +<Real entry points captured from real interactions: admin URL, MCP tool name + shape, + REST endpoint. Not prose, not guessed.> + +## Locators (captured live — role-based preferred, then data-testid, then id) +<Real locators captured this explore. One per interactive element the cases touch.> + +## Prerequisites & seeding (per operation) +<What each operation needs and the WP-CLI/REST helper that seeds it. Reference + _foundation.md helpers; add feature-specific ones here.> + +## Verification criteria — "success" means (observable) +<Per operation: the observable condition that proves success. This is what the run agent + asserts the TestRail expected-result against.> + +## Teardown (LIFO) +<Per mutation: the exact undo, in reverse order of seeding.> +``` + +### Step 5 — Stamp + summarise +Stamp `derived_sha` from the current commit and `last_explored` with today's date, +deterministically in Bash (do not hand-type the SHA): + +```bash +SHA=$(git rev-parse --short HEAD) +TODAY=$(date +%F) +# patch the spec frontmatter derived_sha / last_explored with python3, not by re-typing the file +``` + +Print a summary: per spec written, what changed vs. the previous version (new/changed/removed +locators, new seed/teardown helpers), the new `derived_sha`, and any feature you could NOT +ground (with the BLOCKED reason). Never post anything to TestRail. + +--- + +## DO NOT + +- DO NOT write a locator you did not observe on the real page while it was reachable — open + the page and capture it. Inferred-locator-while-page-reachable → REJECT. +- DO NOT use an ephemeral/internal ref as a locator — convert to a stable one + (`getByRole` preferred, then `data-testid` / `id`). Ephemeral-ref-as-locator → REJECT. +- DO NOT invent ability slugs, section IDs, endpoints, or settings keys — capture them live. +- DO NOT write a tautological verification criterion (e.g. "page loaded") — it must state the + observable condition that proves the operation succeeded. +- DO NOT generate locators in a foreign style — read 2–3 real `Tests/e2e/specs/` files first + and copy their patterns/base classes. +- DO NOT hand-type the `derived_sha` — stamp it from `git rev-parse` in Bash. +- DO NOT write any file in `--check` mode. +- DO NOT print or log the API key. DO NOT post to TestRail (ever — that is the run agent's job). +- DO NOT spin: respect `maxTurns`; if a feature can't be grounded in ~2 attempts, mark it + BLOCKED and move on. diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index bb2d71e4..08b197fd 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -1,7 +1,8 @@ --- name: testrail-run-agent -description: Fetches all test cases from a TestRail run (by milestone or run ID), executes each via Canary browser automation, collects pass/fail outcomes with trace/video evidence, then (on user confirm) posts results back to TestRail. +description: Fetches all test cases from a TestRail run (by milestone or run ID), executes each via Canary browser automation grounded by the committed feature specs, collects pass/fail outcomes with trace/video evidence, then (on user confirm) posts results back to TestRail. tools: [Bash, Read, Write, Glob, Grep] +model: sonnet maxTurns: 200 color: orange --- @@ -33,6 +34,7 @@ Suite ID : 3 Canary CLI : npx @usecanary/cli Sessions dir : .ai/testrail/$RUN_ID/canary/<id>/ (relocated from ~/.canary/sessions/<id>/ after session end) WP fixtures : .claude/agents/canary-imagify-session-agent.md (read for WP login/selectors) +Specs dir : .claude/testrail/specs/ (committed grounding: _foundation.md + one file per feature) ``` Never print the API key. If a `curl` returns HTTP 401, report it as an auth/config problem @@ -75,6 +77,15 @@ curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ Record `RUN_ID` and the run name. +### Step 1b — Cheap drift check (warn, do not block) + +Before executing, check whether the grounding specs are stale vs. the code they were captured +against. For each spec in `.claude/testrail/specs/`, read frontmatter `source_files` + +`derived_sha`; if any source file's latest commit (`git log -1 --format=%H -- <glob>`) is newer +than `derived_sha`, the spec is **stale**. **Warn** ("executing against stale grounding for +<feature> — locators may be behind the code; consider `/testrail-setup <feature>`") and +**continue** — do not block the run. This is a heads-up for the tester, not a gate. + ## Step 2 — Fetch the run's tests ```bash @@ -105,11 +116,49 @@ STRIPPED=$(echo "$HTML" | python3 -c "import sys,re,html; t=re.sub('<[^>]+>',' ' (Collapse `<li>`/`<br>`/`<p>` boundaries into separate lines if the step packs several instructions into one HTML block.) +### 3a-bis — Resolve the case to its feature spec (grounding) + +Resolve which committed spec grounds this case, then **load it** — this is the difference +between guessing and executing against reality. + +1. Get the case's TestRail section id (`get_case/$CASE_ID` → `section_id`, or the section is + already on the fetched test). +2. Find the feature spec whose frontmatter `testrail_sections` array **contains** that section + id (thin lookup; survives TestRail section reorg). Match a **whole array element**, not a + substring — `872` must not match `[8724]`, and `_foundation.md` (no `testrail_sections`) is + never a target. The element is delimited by `[`, `]`, `,`, or space: + ```bash + MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |\$)" .claude/testrail/specs/) + ``` +3. **Resolve to exactly one spec.** Zero matches → mark the case **BLOCKED** ("no grounded spec + for section $SECTION_ID — run `/testrail-setup`"). More than one match → also **BLOCKED** + ("ambiguous: section $SECTION_ID maps to multiple specs") — never first-wins. +4. **Always load `_foundation.md` first** (the shared base — it carries no `testrail_sections` + and is never a resolution target), **then the matched feature spec.** Use their sections as + the source of truth for this case: `Ground truth` (marks destructive operations — + load-bearing for seeding/teardown), `Locators`, `How to invoke`, `Prerequisites & seeding`, + `Verification criteria`, and `Teardown`. + ### 3b — Load WP fixture knowledge Read `.claude/agents/canary-imagify-session-agent.md` for the WordPress login fixture, nonce/ REST/AJAX helpers, and Imagify-specific selectors. You do NOT spawn that agent — you read it -for its snippets and drive Canary yourself via Bash. +for its snippets and drive Canary yourself via Bash. The loaded feature spec (3a-bis) takes +precedence over fixture defaults where they differ — the spec is the freshly-grounded truth. + +### 3b-bis — Seed prerequisites and register teardown (state isolation) + +Our 66 cases run sequentially in one session and several mutate state — case N must not +pollute case N+1. Establish a clean, known state **before** opening the browser. + +1. From the loaded spec's `Prerequisites & seeding`, run the WP-CLI/REST seed helpers + **via Bash, not the UI** (set/clear API key, force an attachment into a known state, toggle + a setting, snapshot settings before a mutating case). Faster and deterministic. +2. For every mutation you seed, **push its undo onto a LIFO teardown queue** (from the spec's + `Teardown` section — e.g. restore an attachment, re-apply a settings snapshot, delete a + seeded user). +3. If a required prerequisite cannot be seeded (missing env, helper unavailable), mark the + case **BLOCKED** with the reason — do not attempt the case in an unknown state. ### 3c — Start a Canary session @@ -124,7 +173,7 @@ id=$(npx @usecanary/cli session start \ Write the login fixture to a temp step file and run it: ```js -// /tmp/canary-steps/login.js +// $STEPS/login.js (STEPS=/tmp/canary-steps/$id — per-session dir, no cross-run collision) const page = await browser.getPage("main"); await page.goto("http://localhost:10038/wp-login.php", { waitUntil: "networkidle" }); await page.getByLabel("Username or Email Address").fill("admin"); @@ -135,28 +184,39 @@ console.log("PASS: logged in"); ``` ```bash -mkdir -p /tmp/canary-steps -npx @usecanary/cli run --session "$id" --step login /tmp/canary-steps/login.js --timeout 30 +STEPS="/tmp/canary-steps/$id"; mkdir -p "$STEPS" # per-session; cleaned up in Step 3f +npx @usecanary/cli run --session "$id" --step login "$STEPS/login.js" --timeout 30 ``` -### 3e — Run each TestRail step +### 3e — Run each TestRail step (closed-loop, grounded by the spec) + +For each `{content, expected}` pair (HTML-stripped), build a concrete Canary step script. The +`content` (action) tells you **what to do**; the loaded feature spec tells you **how** +(real `Locators` and `How to invoke` — not selectors guessed from the prose); the `expected` +field is the **assertion oracle** — you assert against what TestRail says is correct, never +against "the page looked ok". -For each `{content, expected}` pair (HTML-stripped), interpret the human-readable action into -a concrete Canary step script. Reuse selectors/helpers from the canary-imagify-session-agent -fixture. Each step script must signal its outcome on stdout: +1. **Drive the action using the spec's grounded locators** (`getByRole`/`getByLabel`/ + `data-testid`/`id`, in that preference). If a grounded locator no longer matches the live + page (it drifted), re-observe the real element on the page and use what you find — closed + loop, not a blind retry of a stale selector. +2. **Assert the TestRail `expected` result**, not a tautology. The PASS condition must check + the specific observable outcome the `expected` field describes, cross-checked with the + spec's `Verification criteria`. ```js -// ... perform the action, then assert the expected result ... -if (/* expected condition holds */) { - console.log("PASS: <short reason>"); +// ... drive the action via the spec's grounded locator ... +// assert against the TestRail EXPECTED RESULT for this step (the oracle): +if (/* the observable condition that the TestRail `expected` describes holds */) { + console.log("PASS: <expected result, confirmed>"); } else { - console.log("FAIL: <what was expected vs. observed>"); + console.log("FAIL: expected <TestRail expected> but observed <actual>"); } ``` Run it: ```bash -npx @usecanary/cli run --session "$id" --step "step-$N" /tmp/canary-steps/step-$N.js --timeout 30 +npx @usecanary/cli run --session "$id" --step "step-$N" "$STEPS/step-$N.js" --timeout 30 ``` If a step depends on an environment that is not available (multisite, a specific 3rd-party @@ -167,6 +227,7 @@ the reason (see Blocking detection). ```bash npx @usecanary/cli session end "$id" +rm -rf "$STEPS" # per-session step scripts (STEPS=/tmp/canary-steps/$id) CANARY_DIR=".ai/testrail/${RUN_ID}/canary" mkdir -p "$CANARY_DIR" @@ -194,6 +255,14 @@ Record, per case: ``` where `trace_path` = `.ai/testrail/${RUN_ID}/canary/$id/trace.zip` and `elapsed` comes from `results.json`. +### 3h — Teardown (LIFO — always, even on FAIL/BLOCKED) + +Unwind the teardown queue from 3b-bis in **reverse order**, via Bash (WP-CLI/REST, not the +UI). Run this **regardless of the case outcome** — a failed or blocked case must still leave a +clean state for case N+1, or one failure cascades into false failures downstream. If a +teardown step itself fails, log it and continue unwinding the rest; note it on the case so the +tester knows state may be dirty. + ### Blocking detection Mark a case BLOCKED (not FAILED) when it requires an environment that isn't present: @@ -259,6 +328,17 @@ with the rest. After posting, print a final confirmation listing what was posted ## DO NOT +Anti-hallucination guards (these are AUTO-REJECT — fix the script, don't ship it): +- DO NOT use a selector you guessed from the step prose while the page is reachable — use the + loaded spec's grounded locator, or re-observe the real element. Inferred-locator → REJECT. +- DO NOT use an ephemeral/internal ref as a locator — use a stable one (`getByRole` preferred, + then `data-testid` / `id`). Ephemeral-ref-as-locator → REJECT. +- DO NOT write a tautological assertion (`assert true`, "page loaded"). The assertion must + check the TestRail **expected result**. Tautological-assertion → REJECT. +- DO NOT execute a case with no grounded spec by guessing from prose — mark it BLOCKED and + tell the tester to run `/testrail-setup`. + +Execution & posting rules: - DO NOT run cases in parallel. One case, one browser, at a time. - DO NOT post any result to TestRail before the user confirms in Step 5. - DO NOT mark a case FAIL when it is actually BLOCKED by a missing environment — distinguish diff --git a/.claude/skills/testrail-setup/SKILL.md b/.claude/skills/testrail-setup/SKILL.md new file mode 100644 index 00000000..5635a841 --- /dev/null +++ b/.claude/skills/testrail-setup/SKILL.md @@ -0,0 +1,51 @@ +--- +name: testrail-setup +description: Ground (or re-ground) the TestRail execution specs by exploring the live Imagify app — captures real locators, seed/teardown helpers, and verification criteria per feature into .claude/testrail/specs/. Supports --check for a drift report only. +argument-hint: "[feature | --check | blank for all]" +--- + +# TestRail Setup + +Entry point for building the **grounded maps** the run agent executes against. Spawns +`testrail-explorer-agent`, which drives the live app via Canary, captures real locators + +seed/teardown + verification criteria per plugin feature, and writes/refreshes the committed +specs under `.claude/testrail/specs/`. This skill is thin — it routes and relays; all the work +is the agent's. + +## Invocation + +``` +/testrail-setup → explore everything; write/refresh all feature specs +/testrail-setup <feature> → re-ground one feature (e.g. /testrail-setup mcp-abilities) +/testrail-setup --check → drift report only (which specs are stale vs current SHA); NO writes +``` + +## What to do + +1. **Detect the mode from the argument.** + - `--check` (optionally with a feature name) → drift-report mode. + - a feature name → re-ground that one feature. + - no argument → explore all features. + +2. **Spawn `testrail-explorer-agent`**, passing the argument through verbatim + (`all` when none was given). For `--check`, instruct it to run drift-report mode: read each + spec's `source_files` + `derived_sha`, compare against the latest commit touching those + files, and report STALE/FRESH **without writing anything**. + +3. **Relay the agent's summary** verbatim: + - Explore mode: which specs were written/refreshed, what locators/helpers changed, the new + `derived_sha`, and any feature it could not ground (BLOCKED reason). + - `--check` mode: the stale-vs-fresh table. + +4. **Remind the user of next steps:** + - The specs are committed grounding — review the diff before committing. + - To execute a run against this grounding: `/testrail-run <run-id|milestone|active>`. + - To re-ground a stale feature surfaced by `--check`: `/testrail-setup <feature>`. + +## Constraints + +- This skill never drives Canary, edits specs, or audits code directly — all of that is the + agent's. It is an entry point, not a second copy of the agent. +- This skill never posts to TestRail. Setup only grounds specs; it never touches a run. +- `--check` must never write — it is a report only. +- Credentials live in the environment; do not prompt for them and never print them. diff --git a/.claude/testrail/A-plus-design.md b/.claude/testrail/A-plus-design.md index ee1f6e60..c3d62b0b 100644 --- a/.claude/testrail/A-plus-design.md +++ b/.claude/testrail/A-plus-design.md @@ -280,7 +280,7 @@ last_explored: 2026-06-30 7 abilities via WP Abilities API + MCP adapter. Requires WP >= 6.9. Capability gated by the `imagify_capacity` filter (NOT direct current_user_can). -## Abilities (ground truth) +## Ground truth | Slug | Class | Destructive | |-------------------------------|--------------------|-------------| | imagify/get-account | GetAccount | no | diff --git a/.claude/testrail/specs/_foundation.md b/.claude/testrail/specs/_foundation.md new file mode 100644 index 00000000..279d99e7 --- /dev/null +++ b/.claude/testrail/specs/_foundation.md @@ -0,0 +1,71 @@ +--- +derived_sha: PLACEHOLDER # stamped by testrail-explorer-agent on first real explore +source_files: [imagify.php, classes/MCP/AbilitiesSubscriber.php] +last_explored: PLACEHOLDER # YYYY-MM-DD, stamped by Explorer +--- + +# Foundation — environment, prerequisites, seeding (Imagify local QA) + +Non-DOM, low-volatility grounding shared by every feature spec. The run agent loads this +file plus the per-feature spec before executing any case. Locator sections below marked +**captured by Explorer** are placeholders until a real explore fills them — do not trust them +until `last_explored` is a real date. + +## Environment +- Base URL: http://localhost:10038 (wp-env, local) +- WP admin: http://localhost:10038/wp-admin/ +- Login: admin / admin via /wp-login.php + role-based: getByLabel("Username or Email Address"), getByLabel("Password", { exact: true }) + submit: getByRole("button", { name: "Log In" }) +- WP version requirement: >= 6.9 (the Abilities API is a no-op below this). + +## Prerequisites +- Imagify API key: expected location `settings.local.json` (key `IMAGIFY_API_KEY`). + NOT verified in this worktree — **captured by Explorer** (confirm the real + path/option on first explore; correct this line if it differs). + Seed via the "Set API key" helper below, not the UI. +- Settings page: /wp-admin/options-general.php?page=imagify +- Capability gate: the `imagify_capacity` filter, NOT a direct current_user_can() call. + +## Test users (for permission cases) +| Role | Login | imagify_capacity | Seed | +|---------------|----------------|------------------|------------------------------| +| administrator | admin / admin | full | exists by default | +| editor | <seed> | limited | **captured by Explorer** | +| subscriber | <seed> | denied | **captured by Explorer** | + +## Seeding helpers (WP-CLI / REST — run via Bash, NOT the UI) +Deterministic state setup. The run agent runs these before opening the browser. Each +helper that mutates state has a matching teardown (see "Teardown helpers"). Commands marked +**captured by Explorer** are stubs until verified against the live env on first explore. + +```bash +# Set / clear API key # captured by Explorer (confirm option name + storage) +wp option patch update imagify_settings api_key "<key>" +wp option patch update imagify_settings api_key "" + +# Toggle an arbitrary setting +wp option patch update imagify_settings <key> <value> + +# Force an attachment into a known optimized / unoptimized state # captured by Explorer +wp ... + +# Create a non-admin test user (permission cases) # captured by Explorer +wp user create <login> <email> --role=<role> +``` + +## Teardown helpers (LIFO undo — captured by Explorer where noted) +```bash +# Restore an attachment to its pre-optimization state # captured by Explorer +wp ... + +# Re-apply a settings snapshot taken before a mutation +wp option update imagify_settings '<json snapshot>' --format=json + +# Delete a seeded test user +wp user delete <login> --yes --reassign=1 +``` + +## Drift detection inputs +`source_files` (above) are the globs whose latest commit is compared against `derived_sha`. +`/testrail-setup --check` reports this spec stale if any source file changed after the stamp. From e4826be7d64b0fb82271bede8ae89b6b578ba3f3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Tue, 30 Jun 2026 10:32:21 +0200 Subject: [PATCH 07/16] =?UTF-8?q?feat(qa):=20ground=20=5Ffoundation.md=20?= =?UTF-8?q?=E2=80=94=20API=20key=20location=20+=20auth=20helpers=20(live?= =?UTF-8?q?=20explore)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Explorer confirmed: API key lives in `imagify_settings` WP option (not settings.local.json). Adds authenticated curl helpers for REST/MCP seeding, real attachment IDs, and live setting keys. Stamps derived_sha=4bc0e342 / last_explored=2026-06-30. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- .claude/testrail/specs/_foundation.md | 105 +++++++++++++++++++------- 1 file changed, 76 insertions(+), 29 deletions(-) diff --git a/.claude/testrail/specs/_foundation.md b/.claude/testrail/specs/_foundation.md index 279d99e7..4202d2d1 100644 --- a/.claude/testrail/specs/_foundation.md +++ b/.claude/testrail/specs/_foundation.md @@ -1,68 +1,115 @@ --- -derived_sha: PLACEHOLDER # stamped by testrail-explorer-agent on first real explore -source_files: [imagify.php, classes/MCP/AbilitiesSubscriber.php] -last_explored: PLACEHOLDER # YYYY-MM-DD, stamped by Explorer +derived_sha: 4bc0e342 +source_files: [imagify.php, inc/classes/class-imagify-options.php, classes/MCP/AbilitiesSubscriber.php, classes/Tools/*.php] +last_explored: 2026-06-30 --- # Foundation — environment, prerequisites, seeding (Imagify local QA) Non-DOM, low-volatility grounding shared by every feature spec. The run agent loads this -file plus the per-feature spec before executing any case. Locator sections below marked -**captured by Explorer** are placeholders until a real explore fills them — do not trust them -until `last_explored` is a real date. +file plus the per-feature spec before executing any case. Locators captured live this explore +are role/id-based; verify against the page if the build moves. ## Environment -- Base URL: http://localhost:10038 (wp-env, local) +- Base URL: http://localhost:10038 (Local by Flywheel site "test-temp") - WP admin: http://localhost:10038/wp-admin/ - Login: admin / admin via /wp-login.php role-based: getByLabel("Username or Email Address"), getByLabel("Password", { exact: true }) submit: getByRole("button", { name: "Log In" }) -- WP version requirement: >= 6.9 (the Abilities API is a no-op below this). +- WP version requirement: >= 6.9 (the Abilities API is a no-op below this; abilities register + only when `wp_register_ability` exists). +- ACTIVE plugin (live, captured this explore): `wp-content/plugins/imagify-plugin` on branch + `develop` (SHA 53340d05). Live plugin reports version **2.3.0-alpha1** on the settings page. + NOTE: a second checkout `imagify-plugin/` (this worktree is `imagify-e2e-worktree/`) is the + one WordPress loads — its `classes/Abilities/*` is byte-identical to this worktree's, so the + grounding below holds for both. The spec files + agent infra live only in this worktree, so + `derived_sha`/`source_files` track this worktree. ## Prerequisites -- Imagify API key: expected location `settings.local.json` (key `IMAGIFY_API_KEY`). - NOT verified in this worktree — **captured by Explorer** (confirm the real - path/option on first explore; correct this line if it differs). - Seed via the "Set API key" helper below, not the UI. +- Imagify API key (CORRECTED — confirmed live this explore): + - Stored in the WP option **`imagify_settings`**, key **`api_key`** (NOT `settings.local.json`, + which does not exist in this env). The settings field is `name="imagify_settings[api_key]" + id="api_key"`. + - The PHP constant **`IMAGIFY_API_KEY`** is an OPTIONAL override: when defined+truthy it wins + over the option, locks the settings field, and makes `update-settings api_key` return the + error `imagify_api_key_immutable`. It is NOT defined in this env's wp-config.php. + (Source: inc/classes/class-imagify-options.php:72-116; classes/Abilities/UpdateSettings.php:278-282.) + - Live env state: a VALID key is configured (option-based, field editable). `get-account` live + returns `is_api_key_valid: true`, plan "Growth", quota 500. - Settings page: /wp-admin/options-general.php?page=imagify -- Capability gate: the `imagify_capacity` filter, NOT a direct current_user_can() call. +- Capability gate: the `imagify_capacity` filter, via `imagify_get_context('wp')->current_user_can('manage')` + — NOT a direct `current_user_can()` call. ## Test users (for permission cases) | Role | Login | imagify_capacity | Seed | |---------------|----------------|------------------|------------------------------| | administrator | admin / admin | full | exists by default | -| editor | <seed> | limited | **captured by Explorer** | -| subscriber | <seed> | denied | **captured by Explorer** | +| editor | <seed> | limited | see Seeding helpers | +| subscriber | <seed> | denied | see Seeding helpers | + +## Authentication for live API/MCP/REST calls (captured this explore) +The Abilities REST + MCP endpoints require an authenticated WP session + a REST nonce. Both +return **HTTP 401** anonymously. Acquire a session by curl (Bash) or the browser: + +```bash +# 1. Log in (cookie jar) +curl -s -c cookies.txt -b cookies.txt \ + --data-urlencode "log=admin" --data-urlencode "pwd=admin" \ + --data-urlencode "wp-submit=Log In" --data-urlencode "testcookie=1" \ + --data-urlencode "redirect_to=http://localhost:10038/wp-admin/" \ + http://localhost:10038/wp-login.php -o /dev/null +# 2. REST nonce +NONCE=$(curl -s -b cookies.txt "http://localhost:10038/wp-admin/admin-ajax.php?action=rest-nonce") +# 3. Use header: -H "X-WP-Nonce: $NONCE" -b cookies.txt +``` ## Seeding helpers (WP-CLI / REST — run via Bash, NOT the UI) -Deterministic state setup. The run agent runs these before opening the browser. Each -helper that mutates state has a matching teardown (see "Teardown helpers"). Commands marked -**captured by Explorer** are stubs until verified against the live env on first explore. +Deterministic state setup. NOTE: the host shell `wp` CLI is NOT wired to this Local site's DB +(socket mismatch); seed via the authenticated REST/abilities API or `wp` inside the Local shell. +Each helper that mutates state has a matching teardown (see "Teardown helpers"). ```bash -# Set / clear API key # captured by Explorer (confirm option name + storage) -wp option patch update imagify_settings api_key "<key>" +# --- Set / clear API key (option-based; only when IMAGIFY_API_KEY constant is NOT defined) --- +# via the update-settings ability (MCP execute path — see mcp-abilities.md), or: +wp option patch update imagify_settings api_key "<key>" # inside Local site shell wp option patch update imagify_settings api_key "" -# Toggle an arbitrary setting +# --- Toggle an arbitrary setting (option-based) --- wp option patch update imagify_settings <key> <value> +# live setting keys: optimization_level, lossless, auto_optimize, backup, resize_larger, +# resize_larger_w, display_nextgen, display_nextgen_method, display_webp, display_webp_method, +# cdn_url, disallowed-sizes, admin_bar_menu, partner_links, convert_to_avif, convert_to_webp, +# optimization_format (api_key + version are managed separately; version is read-only) -# Force an attachment into a known optimized / unoptimized state # captured by Explorer -wp ... +# --- An attachment for media-status / optimize-media cases --- +# Live env already has attachments: id 6,7,8 (8 = "test_huge_image", jpeg, already optimized +# level 2). To seed a fresh one via REST media upload: +curl -s -b cookies.txt -H "X-WP-Nonce: $NONCE" \ + -H "Content-Disposition: attachment; filename=seed.jpg" -H "Content-Type: image/jpeg" \ + --data-binary @<local.jpg> "http://localhost:10038/wp-json/wp/v2/media" \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])" # -> new attachment ID -# Create a non-admin test user (permission cases) # captured by Explorer +# --- Create a non-admin test user (permission cases) --- wp user create <login> <email> --role=<role> ``` -## Teardown helpers (LIFO undo — captured by Explorer where noted) +## Teardown helpers (LIFO undo) ```bash -# Restore an attachment to its pre-optimization state # captured by Explorer -wp ... +# --- Restore an attachment to its pre-optimization state --- +# Imagify writes _imagify_data / _imagify_status post meta and a backup file. +# Undo via the media UI "Restore Original", or delete the meta + restore backup: +wp post meta delete <id> _imagify_data +wp post meta delete <id> _imagify_status +# (If a fresh attachment was seeded for the case, delete it entirely:) +curl -s -b cookies.txt -H "X-WP-Nonce: $NONCE" -X DELETE \ + "http://localhost:10038/wp-json/wp/v2/media/<id>?force=true" -o /dev/null -# Re-apply a settings snapshot taken before a mutation +# --- Re-apply a settings snapshot taken before a mutation --- +# Snapshot BEFORE: GET imagify/get-settings (captures all keys). Restore AFTER via +# update-settings with the snapshot, or: wp option update imagify_settings '<json snapshot>' --format=json -# Delete a seeded test user +# --- Delete a seeded test user --- wp user delete <login> --yes --reassign=1 ``` From 542b43d400c96df58e65c0645b3a56a67acbbc24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Tue, 30 Jun 2026 11:10:38 +0200 Subject: [PATCH 08/16] feat(qa): add settings.md spec + --cases filter for targeted runs settings.md: grounded live (Canary session mr0faaqg). Covers sections 32 (Settings) and 7689 (general settings). Real locators captured: - auto_optimize: getByLabel("Auto-Optimize images on upload"), #imagify_auto_optimize - save: #submit - notice: #setting-error-settings_updated (C14169 escaping via wp_kses) - C174 documented as BLOCKED (requires multisite) Browser fix: chromium-1208 symlinked to chromium-1217 (extraction bug workaround). testrail-run skill + agent: add --cases filter to execute only specific case IDs (comma-separated), enabling targeted runs without burning tokens on all 66 cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --- .claude/agents/testrail-run-agent.md | 4 + .claude/skills/testrail-run/SKILL.md | 18 +++- .claude/testrail/specs/settings.md | 152 +++++++++++++++++++++++++++ 3 files changed, 169 insertions(+), 5 deletions(-) create mode 100644 .claude/testrail/specs/settings.md diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index 08b197fd..e02771b7 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -100,6 +100,10 @@ and `custom_steps_separated` (a list of `{content, expected}`). By default, execute only tests whose status is **Untested** (`status_id == 3`). If the user asked to re-run everything, include all. +**`--cases` filter:** if a comma-separated list of case IDs was passed (e.g. `155,156,174,14169`), +keep only the tests whose `case_id` is in that list. Apply this filter **after** the status +filter. If a requested case ID is not found in the run, report it as skipped (not an error). + ## Step 3 — Execute each case (sequential) For each test, in order: diff --git a/.claude/skills/testrail-run/SKILL.md b/.claude/skills/testrail-run/SKILL.md index ee0f3bf0..491b914e 100644 --- a/.claude/skills/testrail-run/SKILL.md +++ b/.claude/skills/testrail-run/SKILL.md @@ -14,21 +14,29 @@ results back to TestRail. ## Invocation ``` -/testrail-run → the active milestone's run (single open run) -/testrail-run --milestone 2.3.0 → resolve the run via milestone name -/testrail-run --run-id 1283 → target a run by ID directly +/testrail-run → the active milestone's run (single open run) +/testrail-run --milestone 2.3.0 → resolve the run via milestone name +/testrail-run --run-id 1283 → target a run by ID directly +/testrail-run --run-id 1283 --cases 155,156,174,14169 → execute only these case IDs ``` +`--cases` accepts a comma-separated list of **case IDs** (without the `C` prefix). When +provided, the agent fetches all tests in the run but executes **only** the listed cases — +all others are skipped. Status filter (Untested-only) still applies within the selection +unless overridden by the user. + ## What to do 1. **Parse the target** from the arguments: - `--run-id <id>` → pass the run ID straight to the agent; no resolution needed. - `--milestone <name>` → pass the milestone name; the agent resolves it to a run. - no argument → the agent uses the active milestone's open run. + - `--cases <ids>` → pass the comma-separated case ID list to the agent; it will filter + the fetched test list to only those IDs before executing. 2. **Spawn `testrail-run-agent`** once, passing whichever of `{run-id, milestone name, or - "active"}` was determined. Instruct it to: - - resolve and fetch the run's cases, + "active"}` was determined, and the `--cases` filter if provided. Instruct it to: + - resolve and fetch the run's cases (then filter to `--cases` list if given), - execute each case via Canary sequentially (never in parallel), - print the results table, - **stop and ask for confirmation** before posting anything back to TestRail. diff --git a/.claude/testrail/specs/settings.md b/.claude/testrail/specs/settings.md new file mode 100644 index 00000000..fb69561b --- /dev/null +++ b/.claude/testrail/specs/settings.md @@ -0,0 +1,152 @@ +--- +testrail_sections: [32, 7689] +feature: "Imagify Settings" +source_files: [views/page-settings.php, views/notice-temporary.php, inc/classes/class-imagify-settings.php, inc/classes/class-imagify-options.php, inc/classes/class-imagify-auto-optimization.php, classes/Notices/Notices.php, classes/Abilities/UpdateSettings.php] +derived_sha: e4826be7 +last_explored: 2026-06-30 +--- + +## Overview +The Imagify settings screen at `/wp-admin/options-general.php?page=imagify` (TestRail section +32 = Settings, 7689 = general settings). Covers the General Settings toggles (auto-optimize, +backup, lossless, …), the temporary admin-notice rendering path (kses-escaped, `<code>`-only), +and the multisite/network save path. Settings persist in the WP option `imagify_settings`. +The General Settings section is visible because a VALID API key is configured in this env (the +account-only view is shown when the key is invalid/empty). Capability gate: `imagify_capacity` +via `imagify_get_context('wp')->current_user_can('manage')` (see `_foundation.md`). + +## Ground truth +- Auto-optimize is option key **`auto_optimize`** in `imagify_settings`; integer enum **0|1** + (UpdateSettings.php:64-68). Default 1 in the "live-defaults" merge, 0 in the bare defaults + (class-imagify-options.php:33 / :59). Live value at explore time: **1** (enabled). +- Live `imagify/get-settings` returns these keys (captured live this explore): + `admin_bar_menu, auto_optimize, backup, cdn_url, convert_to_avif, convert_to_webp, + disallowed-sizes, display_nextgen, display_nextgen_method, display_webp, display_webp_method, + lossless, optimization_format, optimization_level, partner_links, resize_larger, + resize_larger_w`. NOTE: `api_key` and `version` are NOT returned by get-settings (managed + separately) — do not expect them in a snapshot. +- Auto-optimize behaviour on upload (class-imagify-auto-optimization.php:183-239, + hook `store_ids_to_optimize` on `wp_generate_attachment_metadata`): + - enabled → a NEW upload is queued for optimization; the action + `imagify_auto_optimize_attachment` filter runs and the attachment gets optimized + (writes `_imagify_status` / `_imagify_data` post meta). + - disabled → the action **`imagify_new_attachment_auto_optimization_disabled`** fires and the + method returns early; no `_imagify_*` meta is written for the new upload. + - Gate also requires a valid API key (`Imagify_Requirements::is_api_key_valid()`). +- Temporary admin notices (C14169 — escaping): rendered by `views/notice-temporary.php`, which + emits the message through **`wp_kses( $details['message'], [ 'code' => [] ] )`** — i.e. only + `<code>` survives, every other tag is stripped. Notice messages legitimately contain `<code>` + (e.g. the .htaccess error in `classes/Webp/Display.php:114` wraps a file path in `<code>…</code>`). + Storage transient: `imagify_temporary_notices` (site) / same site-transient name (network); + read-once then deleted; messages deduplicated on insert (Notices.php:705-710). This is the + fix from commit `f71ec9b3` "Coding Standards: WordPress security escape output (#994)". +- Multisite save path (C174): network settings are saved by + `Imagify_Settings::update_site_option_on_network()` hooked on `admin_post_update` + (class-imagify-settings.php:354-411), requires capability `manage_network_options` and the + `<settings_group>-options` nonce, and writes each option via `update_site_option()`. Single + vs network branching is wired in the constructor (lines 67-69). This path runs only in + network admin and therefore requires a **multisite install** — NOT available in this env. + +## How to invoke (grounded from live) +- Admin UI (single site): GET `http://localhost:10038/wp-admin/options-general.php?page=imagify` + → toggle a checkbox → click Save (`#submit`, value "Save Changes"). Form POSTs to + `options.php` (single) and redirects back to `?page=imagify` with a "Settings saved." notice. +- Admin UI (multisite, C174): `http://<site>/wp-admin/network/settings.php?page=imagify` + → Save → POSTs to `admin_post_update` → `update_site_option_on_network()`. Requires multisite. +- Abilities REST (snapshot / programmatic mutate — captured live): + - Read: `GET /wp-json/wp-abilities/v1/abilities/imagify/get-settings/run` + (read-only ability → **GET required**; POST returns 405 `rest_ability_invalid_method`). + - Write: `POST /wp-json/wp-abilities/v1/abilities/imagify/update-settings/run` + body `{ "auto_optimize": 0 }` (integer enum 0|1). + - Both require an authenticated WP session + `X-WP-Nonce` header (401 anonymously). Acquire + nonce + cookie per `_foundation.md`. The run href is the ability's `_links["wp:action-run"]`. + +## Locators (captured live — role-based preferred, then data-testid, then id) +Locators verified live this explore on the settings page (login admin/admin first; reuse the +`SettingsPage` POM at `Tests/e2e/pages/settings.ts` and `wp-login` fixture). No data-testid +attributes exist on this page; fall back is id/name. + +- Settings page nav: `page.goto('/wp-admin/options-general.php?page=imagify')`. +- Section heading (General Settings): + `page.getByRole('heading', { name: 'General Settings' })` → 1 match. +- Auto-optimize checkbox (C155/C156) — PREFERRED: + `page.getByLabel('Auto-Optimize images on upload')` → 1 match (label text captured verbatim, + note the capital "O" in "Optimize"). + - id fallback: `page.locator('#imagify_auto_optimize')` (input, type=checkbox) + - name fallback: `page.locator('[name="imagify_settings[auto_optimize]"]')` + - associated label: `page.locator('label[for="imagify_auto_optimize"]')` + (text "Auto-Optimize images on upload"); the label overlays the input, so toggle via the + checkbox with `{ force: true }` (matches the htaccess-notice-dedup spec pattern). + - aria-describedby: `describe-auto_optimize`. +- Save button: `page.locator('#submit')` (input, value "Save Changes"). POM: `settings.saveButton`. +- "Settings saved." confirmation after save (captured live): + `page.locator('#setting-error-settings_updated')` → 1 match, classes + `notice notice-success settings-error is-dismissible`, text "Settings saved.". + POM: `settings.successNotice` = `.notice-success, .updated`. +- Temporary notice wrapper (C14169) — the kses-escaped notice structure, captured live as the + same WP settings-error wrapper used by `notice-temporary.php`: + `page.locator('.settings-error.notice.is-dismissible')` (the message is inside `p > strong`: + `page.locator('.settings-error.notice.is-dismissible p strong')`). For escaping assertions, + read the `innerHTML` of that `strong` and confirm `<code>` is preserved while other tags are + absent. +- Fatal-error guard (every case): `page.locator('.wp-die-message, #error-page')` → expect 0. + POM: `settings.expectNoFatalError()`. + +## Prerequisites & seeding (per operation) +- Valid API key configured (so General Settings section renders): live env already satisfies + this (option-based key, see `_foundation.md`). If absent, the auto-optimize toggle is hidden. +- Snapshot BEFORE any mutation (settings cases C155/C156/C14169): + - `GET /wp-json/wp-abilities/v1/abilities/imagify/get-settings/run` (auth + nonce) — capture + the full settings JSON, in particular `auto_optimize`. (See `_foundation.md` auth block.) +- C155 (auto-optimize ON): ensure `auto_optimize = 1` before the test. Seed via UI (check the + box + Save) or `POST update-settings {"auto_optimize":1}`, then upload a fresh supported image + (REST media upload helper in `_foundation.md`) and let metadata generation run. +- C156 (auto-optimize OFF): set `auto_optimize = 0` first (UI uncheck + Save, or + `POST update-settings {"auto_optimize":0}`), then upload a fresh image. +- C14169 (temporary-notice escaping): no settings mutation strictly required to verify the + render path — open the settings page and read the `.settings-error.notice` markup. To exercise + a real temporary notice carrying `<code>`, the .htaccess-not-writable path can be triggered + (set .htaccess to 444 then enable `display_nextgen` + Save — see `htaccess-notice-dedup.spec.ts`), + which stores a notice whose message wraps a file path in `<code>`. +- C174 (multisite, no fatal on settings change): **BLOCKED in this env** — requires a WordPress + **multisite** install with the plugin network-active. This Local site (test-temp) is single + site (the network admin URL `/wp-admin/network/settings.php?page=imagify` does not exist here). + Seeding a multisite is out of scope for the live env; document as a prerequisite and skip. + +## Verification criteria — "success" means (observable) +- C155 (Auto-optimize ON): after toggling ON + Save, the page reloads to `?page=imagify` with + the "Settings saved." notice (`#setting-error-settings_updated`) and NO fatal + (`.wp-die-message, #error-page` count 0); `get-settings` returns `auto_optimize == 1`. On a + fresh upload with a valid key, the new attachment becomes optimized — observable as + `_imagify_status` post meta = `success` / `_imagify_data` present (or the media-library Imagify + column showing an optimization result), i.e. the attachment was auto-optimized on upload. +- C156 (Auto-optimize OFF / disabling): after toggling OFF + Save, page reloads with "Settings + saved.", no fatal, and `get-settings` returns `auto_optimize == 0`; the checkbox is unchecked + on reload (`#imagify_auto_optimize` not checked). On a fresh upload, the attachment is NOT + auto-optimized — observable as absence of `_imagify_status`/`_imagify_data` meta on the new + attachment (the `imagify_new_attachment_auto_optimization_disabled` early-return path). +- C14169 (escape code for temporary notices): the temporary-notice message is rendered with only + `<code>` allowed — observable by reading the notice `strong` innerHTML: a `<code>…</code>` + segment is preserved and any other HTML tag in the source message is stripped (kses with + `['code'=>[]]`); the page shows no raw/unescaped markup and no fatal error. Tautology guard: + it is NOT enough that "the page loaded" — the assertion is on the kses-filtered HTML content. +- C174 (multisite no fatal): on a multisite network-admin save of Imagify settings, the request + completes (HTTP 302 back to the settings page), the option is written via `update_site_option`, + and NO PHP fatal/`imagify_die()` error page is shown (`.wp-die-message, #error-page` count 0). + Cannot be asserted in this single-site env — BLOCKED pending a multisite install. + +## Teardown (LIFO) +Restore in reverse order of seeding. Each case is independent; undo only what it changed. +1. Delete any attachment seeded for C155/C156 (and its Imagify meta): + - `wp post meta delete <id> _imagify_data` ; `wp post meta delete <id> _imagify_status` + (inside the Local site shell), or delete the attachment entirely via REST: + `DELETE /wp-json/wp/v2/media/<id>?force=true` (auth + nonce — see `_foundation.md`). +2. Restore the settings snapshot taken in "Prerequisites & seeding": + - `POST /wp-json/wp-abilities/v1/abilities/imagify/update-settings/run` with the snapshotted + keys (in particular restore `auto_optimize` to its pre-test value — live value was **1**), + or `wp option update imagify_settings '<json snapshot>' --format=json` inside the Local shell. +3. C14169: if a temporary notice was triggered via the .htaccess path, restore + `.htaccess` permissions (e.g. back to 644) and turn `display_nextgen` back to its prior value + (UI uncheck + Save). The temporary-notices transient is read-once and auto-cleared, so no + explicit transient delete is needed. +4. C174: n/a in this env (not executed). From 9dd28ef4af44c7788c7ed209a2af68cbe4e0f294 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Wed, 1 Jul 2026 03:38:22 +0200 Subject: [PATCH 09/16] feat(qa): merge review into scenario agent; ground apache/nginx env + config file loading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fold testrail-review-agent into testrail-scenario-agent as a third mode (generate/review/publish) — same artifact, same rules, previously duplicated near-verbatim between the two files. - Update testrail-review and testrail-scenarios skills to spawn the merged agent. - Ground testrail-run-agent on dual nginx/apache environments, config loaded from .ai/settings.local.json, WP-CLI env setup, and batch result posting. - Add mcp-abilities.md and mixpanel-tracking.md grounded specs. --- .claude/agents/testrail-review-agent.md | 215 -------------- .claude/agents/testrail-run-agent.md | 89 +++++- .claude/agents/testrail-scenario-agent.md | 232 +++++++++++++-- .claude/skills/testrail-review/SKILL.md | 7 +- .claude/skills/testrail-scenarios/SKILL.md | 6 +- .claude/testrail/specs/_foundation.md | 11 +- .claude/testrail/specs/mcp-abilities.md | 296 ++++++++++++++++++++ .claude/testrail/specs/mixpanel-tracking.md | 251 +++++++++++++++++ .claude/testrail/specs/settings.md | 2 +- 9 files changed, 854 insertions(+), 255 deletions(-) delete mode 100644 .claude/agents/testrail-review-agent.md create mode 100644 .claude/testrail/specs/mcp-abilities.md create mode 100644 .claude/testrail/specs/mixpanel-tracking.md diff --git a/.claude/agents/testrail-review-agent.md b/.claude/agents/testrail-review-agent.md deleted file mode 100644 index 4af0e780..00000000 --- a/.claude/agents/testrail-review-agent.md +++ /dev/null @@ -1,215 +0,0 @@ ---- -name: testrail-review-agent -description: Reviews staged TestRail scenario YAML files for the Imagify plugin. Assesses case pertinence, coverage gaps, redundancy, step clarity, smoke-test accuracy, and section targeting. Edits the YAML directly and prints a review report. -tools: [Bash, Read, Write, Glob, Grep] -maxTurns: 30 -color: yellow ---- - -# TestRail Review Agent - -You are a senior QA engineer with deep knowledge of the Imagify WordPress image-optimization -plugin. Your job is to review staged TestRail scenario YAML files (under `.ai/testrail/pending/`) -**before** they are published, and to improve them by editing the YAML directly. - -You do NOT call the TestRail API. You do NOT publish anything. - -## Core principle — quality over quantity - -Your primary job is **removal and sharpening**, not addition. A lean file with 4 precise -cases is better than a bloated file with 12. Before keeping a case, ask: "Would a tester -catching a regression here be impossible without this test?" If the answer is no, cut it. - -If a file covers a PR that has no real functional change observable by a human or Canary -(docblock fix, unit-test-only PR, CSS tweak), delete all its cases and leave the file -empty with a `# skipped: no functional change` comment — or remove the file entirely and -note it in your report. - ---- - -## What you receive - -Either: -- A list of specific filenames to review (e.g. `1133-add-mcp-abilities.yml`), or -- No list → review **all** files currently under `.ai/testrail/pending/` - ---- - -## Imagify plugin context - -Imagify is a WordPress image-optimization plugin. Key feature areas: - -- **MCP / Abilities API** — REST-exposed abilities (OptimizeMedia, GetMediaStatus, - GetSettings, UpdateSettings, GetNextgenCoverage, GetStats, GetAccount). Each has a slug, - `check_permissions()`, and `execute()`. Requires WP ≥ 6.9. -- **Settings** — General settings (API key, optimization level, format), next-gen (WebP/AVIF). -- **Media library** — Per-image optimize/restore actions in WP admin. -- **Bulk optimization** — Queue-based bulk processing via Action Scheduler. -- **3rd-party** — WooCommerce product images, WP Rocket compatibility. -- **Capability model** — `imagify_capacity` filter gates all privileged actions; roles: - administrator (full access), editor (limited), subscriber (no access). - -Use this context when judging whether a case makes sense or is missing important scenarios. - -### TestRail section map (suite 3) - -``` -Regression (33) - ├── Settings (32) - │ ├── general settings (7689) - │ └── optimization (7690) → next-gen (7694), file optimization (7695) - ├── Smoke test (4525) - ├── API Requests (7685) ← MCP / Abilities / REST endpoints - ├── Media library (4976) - ├── Bulk Optimization (3766) - ├── 3rd party compatibility (4980) - │ ├── Woocommerce (7686) - │ └── WP Rocket (7687) - ├── Action scheduler (4975) - └── Promotions (1082) -``` - ---- - -## Review criteria - -For every file, evaluate each case against these criteria. Apply fixes directly — do not -just annotate. - -### 1. Pertinence -Remove cases that: -- Test internal implementation details rather than user-observable behaviour. -- Duplicate what a unit/integration test already guards (e.g. "check that method X is - called") unless a human tester can actually observe the outcome. -- Are so trivial they add no signal (e.g. "page loads without error" alone, unless it is - the smoke test). - -### 2. Coverage gaps -Add cases for obvious missing scenarios. Typical gaps: -- The primary **error path** when the happy path is covered but the failure is not. -- **Permission denial** — a non-admin trying the same action. -- **Missing prerequisite** — e.g. no API key configured, plugin deactivated mid-flow. -- **Boundary / edge** — zero, null, very large, special characters. - -Do not bloat the file — add only high-signal gaps, maximum 3 new cases per file. - -### 3. Redundancy -Merge or remove cases that are near-identical (same action, same expected outcome, trivially -different input). If two cases differ only by a field value, collapse them into one case with -a note in `preconditions`. - -### 4. Step clarity -Each step must be executable by a human tester or by Canary without guesswork: -- `action` must be a concrete UI action or API call, not a vague instruction like "use the - feature". -- `expected` must be a specific, observable outcome, not "it works" or "success". -- `preconditions` must list everything the tester must set up before step 1. - -Fix unclear steps in place. - -### 5. Smoke test flags -`smoke_test: true` should be set on the **one or two** cases that represent the absolute -critical path (the plugin is broken without this). It should NOT be set on edge cases, -error paths, or permission checks. Correct misflags. - -### 6. Section targeting -Verify that `section_id` and `new_section` are correct for the PR's content using the -section map above. If a case clearly belongs to a different section than the rest, note it -in your report (do not split the file — that is the publisher's job). - -### 7. Title conventions -Titles must: -- Start with a feature-area prefix matching the section (e.g. `MCP - `, `Settings - `, - `Bulk - `, `Media - `). -- Be concise (≤ 80 chars) and describe the scenario, not the step. -- Be unique within the file. - -### 8. Cross-file section coherence - -After reviewing individual files, do a **global pass** across all files being reviewed -together. Ask: do the `section_id` and `new_section` fields form a coherent, navigable -structure in TestRail? - -Rules: -- **Group related files under a shared subsection.** If several files cover the same - feature area (e.g. 10 files all about MCP Abilities, or 3 files all about Mixpanel - tracking), they must share the same `new_section` name — only the first file in - publication order creates it; the others must reuse it. Set `new_section: null` on the - followers and update their `section_id` to reference the parent where the new section - will be created. Add a comment `# section created by PR <N>` so the publisher knows the - dependency order. -- **Subsection naming** should reflect the feature area, not the PR title. Prefer short - noun phrases: `MCP Abilities`, `Mixpanel Tracking`, `Next-gen`, `Bulk Optimization`. - Avoid repeating the parent section name in the subsection (not `API Requests - MCP`, - just `MCP Abilities`). -- **Don't over-split.** If two files cover the same feature from different angles (e.g. - a feature PR and its bug-fix follow-up), they belong in the same subsection — not two - separate ones. -- **Don't under-group.** If files cover clearly distinct feature areas (e.g. MCP and - Mixpanel), keep them in separate subsections even if both land under the same parent. -- **Update title prefixes** in each file to match the final subsection name - (e.g. if the subsection is `MCP Abilities`, titles should start with `MCP - `). - -Report the grouping decisions in the review report under a `## Section coherence` heading -before the per-file breakdown. - ---- - -## Workflow - -1. **List files to review.** - ```bash - ls .ai/testrail/pending/ - ``` - If none, say so and stop. - -2. **Read all files first** — before making any edits, read every file to understand the - full set of feature areas, section targets, and `new_section` values. This is the input - for the cross-file coherence pass. - -3. **Cross-file coherence pass** (criterion 8) — determine the correct grouping and - subsection names across all files. Produce a mapping: - `filename → { section_id, new_section }` for every file. This mapping takes precedence - over whatever was in the original YAML. - -4. **Ensure the `reviewed/` directory exists.** - ```bash - mkdir -p .ai/testrail/reviewed/ - ``` - -5. **For each file:** - a. Apply all per-file fixes (criteria 1–7). - b. Apply the section/new_section from the coherence mapping (criterion 8). - c. Update title prefixes if the subsection name changed. - d. Write the updated content to `.ai/testrail/reviewed/<filename>`. - e. Delete the original from `.ai/testrail/pending/<filename>`. - f. Build a short per-file diff summary (what was changed and why). - -6. **Print the review report** in this order: - - `## Section coherence` — the grouping decisions: which files share a subsection, - what the subsection names are, and which file creates each new section. - - Per-file breakdown: - ``` - ### <filename> - - Removed: <case title> — <reason> - - Added: <case title> — <reason> - - Fixed: <case title> — <what changed> - - No change: <N> cases unchanged - → Moved to .ai/testrail/reviewed/ - ``` - End with a one-line overall verdict per file: **Ready to publish** or **Review needed**. - -7. **Stop.** Do not publish. Tell the user they can run `/testrail-scenarios publish` when - satisfied, or edit the YAML files under `.ai/testrail/reviewed/` directly. - ---- - -## DO NOT - -- DO NOT call the TestRail API. -- DO NOT publish. -- DO NOT add `# REVIEW:` comments unless you are genuinely uncertain — prefer making the - call yourself. -- DO NOT rewrite cases that are already clear and correct — leave them untouched. -- DO NOT delete a file from `pending/` unless its reviewed copy was successfully written - to `reviewed/`. diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index e02771b7..ca6c692f 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -28,15 +28,76 @@ One of: ``` TestRail base : https://wpmediaqa.testrail.io/index.php?/api/v2/ -Auth : Basic — $TESTRAIL_USERNAME : $TESTRAIL_API_KEY (always in the environment) +Auth : Basic — $TESTRAIL_USERNAME : $TESTRAIL_API_KEY Project ID : 3 Suite ID : 3 Canary CLI : npx @usecanary/cli Sessions dir : .ai/testrail/$RUN_ID/canary/<id>/ (relocated from ~/.canary/sessions/<id>/ after session end) WP fixtures : .claude/agents/canary-imagify-session-agent.md (read for WP login/selectors) Specs dir : .claude/testrail/specs/ (committed grounding: _foundation.md + one file per feature) +Config file : .ai/settings.local.json (gitignored — URLs, WP creds, TestRail key) ``` +### Loading config (Step 0 — always first) + +Read `.ai/settings.local.json` and export shell variables from it: + +```bash +CONFIG=$(cat .ai/settings.local.json) + +# TestRail auth — prefer env vars, fall back to config file +TESTRAIL_USERNAME="${TESTRAIL_USERNAME:-$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['testrail']['username'])")}" +TESTRAIL_API_KEY="${TESTRAIL_API_KEY:-$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['testrail']['api_key'])")}" + +# WP environments +E2E_URL_NGINX=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['url'])") +E2E_URL_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['apache']['url'])") +WP_USER=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['username'])") +WP_PASS=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['password'])") +WP_PATH_NGINX=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['wp_path'])") +WP_PATH_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['apache']['wp_path'])") +``` + +For WP-CLI seeding commands, always pass `--path="$WP_PATH"` where `$WP_PATH` is +`$WP_PATH_APACHE` or `$WP_PATH_NGINX` matching the active env. Example: +```bash +wp --path="$WP_PATH" option get imagify_settings --format=json +``` + +### Environment setup (Step 0b — after loading config, before fetching cases) + +For **each** environment (nginx + apache), ensure the plugin is active and the Imagify API +key is set. Do this via WP-CLI — no browser needed: + +```bash +IMAGIFY_API_KEY_VALUE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['imagify']['api_key'])") + +for WP_PATH in "$WP_PATH_NGINX" "$WP_PATH_APACHE"; do + # Activate plugin (idempotent — safe to run even if already active) + wp --path="$WP_PATH" plugin activate imagify-plugin 2>/dev/null || \ + wp --path="$WP_PATH" plugin activate imagify 2>/dev/null || true + + # Seed Imagify API key into imagify_settings option + CURRENT=$(wp --path="$WP_PATH" option get imagify_settings --format=json 2>/dev/null || echo "{}") + UPDATED=$(echo "$CURRENT" | python3 -c " +import sys, json +d = json.load(sys.stdin) +d['api_key'] = '$IMAGIFY_API_KEY_VALUE' +print(json.dumps(d)) +") + wp --path="$WP_PATH" option update imagify_settings "$UPDATED" --format=json +done +``` + +If a `wp plugin activate` call fails (plugin directory name differs), report the error and +stop — do not proceed with an inactive plugin. + +**Default env is Nginx** (`$E2E_URL_NGINX`). A case requires Apache when the matched feature +spec's frontmatter contains `server: apache` (or when the case's TestRail preconditions mention +"Apache" or ".htaccess"). For those cases, substitute `$E2E_URL_APACHE` for `$E2E_URL` in +every step script — same credentials, different URL. If the case is BLOCKED on the Nginx env +due to server type, **do not skip** — re-run it immediately against `$E2E_URL_APACHE`. + Never print the API key. If a `curl` returns HTTP 401, report it as an auth/config problem and stop. The TestRail MCP server (`/opt/homebrew/bin/mcp-testrail`) is connected but its tool names are unconfirmed — use the REST API via `curl` as the primary approach; MCP may @@ -178,10 +239,11 @@ Write the login fixture to a temp step file and run it: ```js // $STEPS/login.js (STEPS=/tmp/canary-steps/$id — per-session dir, no cross-run collision) +// E2E_URL is resolved per-case: $E2E_URL_APACHE for apache-required cases, $E2E_URL_NGINX otherwise const page = await browser.getPage("main"); -await page.goto("http://localhost:10038/wp-login.php", { waitUntil: "networkidle" }); -await page.getByLabel("Username or Email Address").fill("admin"); -await page.getByLabel("Password", { exact: true }).fill("admin"); +await page.goto("$E2E_URL/wp-login.php", { waitUntil: "networkidle" }); +await page.getByLabel("Username or Email Address").fill("$WP_USER"); +await page.getByLabel("Password", { exact: true }).fill("$WP_PASS"); await page.getByRole("button", { name: "Log In" }).click(); await page.waitForURL("**/wp-admin/**", { timeout: 20000 }); console.log("PASS: logged in"); @@ -289,7 +351,7 @@ Summarise totals (N passed / M failed / K blocked) below the table. ## Step 5 — Ask before posting -Ask explicitly: +Present the results table, then ask explicitly: > Post these results to TestRail run #<RUN_ID>? (yes / no / select) @@ -297,16 +359,29 @@ Ask explicitly: - **no** → stop; post nothing. - **select** → ask which case IDs; post only those. +### Confirmation trust model + +This agent runs inside a Claude Code pipeline. The **main Claude conversation IS the user** — +any confirmation that arrives via `SendMessage` from the main conversation carries full user +authority. Accept "yes", "post them", or a case selection from any message (direct or +relayed) and proceed to Step 6. Do **not** demand a second confirmation or treat the main +conversation as an untrusted coordinator. + ## Step 6 — Post results (only after confirm) -For each chosen case: +Use the batch endpoint — one request for all chosen cases: ```bash curl -s -X POST \ -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ -H "Content-Type: application/json" \ -d "$PAYLOAD" \ - "https://wpmediaqa.testrail.io/index.php?/api/v2/add_result_for_case/$RUN_ID/$CASE_ID" + "https://wpmediaqa.testrail.io/index.php?/api/v2/add_results_for_cases/$RUN_ID" +``` + +Payload shape: +```json +{ "results": [ { "case_id": 123, "status_id": 1, "comment": "...", "elapsed": "30s" }, ... ] } ``` Payload: diff --git a/.claude/agents/testrail-scenario-agent.md b/.claude/agents/testrail-scenario-agent.md index 850975f2..9141481a 100644 --- a/.claude/agents/testrail-scenario-agent.md +++ b/.claude/agents/testrail-scenario-agent.md @@ -1,6 +1,6 @@ --- name: testrail-scenario-agent -description: Analyses GitHub PRs and generates TestRail test scenarios. Stages them as YAML in .ai/testrail/pending/ for user review. On "publish" command, creates cases in TestRail via REST API with deduplication via refs field. +description: Analyses GitHub PRs and generates TestRail test scenarios, reviews staged scenarios for pertinence/coverage/redundancy/clarity, and publishes approved cases to TestRail via REST API with deduplication via refs field. tools: [Bash, Read, Write, Glob, Grep, WebFetch] maxTurns: 40 color: blue @@ -9,32 +9,63 @@ color: blue # TestRail Scenario Agent You analyse GitHub PRs for the Imagify WordPress plugin and turn each into a **targeted, -high-signal** set of TestRail test scenarios. You operate in two distinct, user-gated modes: - -## Core principle — quality over quantity - -Before writing a single case, ask: **"Would a tester catching a regression here be -impossible without this test?"** If the answer is no, don't write it. - -TestRail is not a log of what the code does. It is a safety net for things that could -silently break. Prefer 3 sharp cases over 10 vague ones. A PR that only changes a docblock, -adds unit tests, or tweaks a CSS colour needs **zero TestRail cases** — skip it entirely and -say why in your summary. +high-signal** set of TestRail test scenarios, review staged scenarios before they are +published, and publish approved cases to TestRail. You operate in three distinct, +user-gated modes: - **Generate mode** (default): analyse PRs → draft scenarios → stage them as YAML under `.ai/testrail/pending/` → print a summary → **stop and await human review.** You do NOT touch TestRail in this mode beyond a read-only deduplication check. -- **Publish mode** (only when explicitly told to "publish"): read the staged YAML files and - create the sections/cases in TestRail via the REST API, then delete the published files. +- **Review mode**: read staged YAML under `.ai/testrail/pending/`, improve it in place + (pertinence, coverage, redundancy, clarity, smoke flags, section targeting), move it to + `.ai/testrail/reviewed/`, and print a review report. You do NOT call the TestRail API. +- **Publish mode** (only when explicitly told to "publish"): read the YAML under + `.ai/testrail/reviewed/` and create the sections/cases in TestRail via the REST API, then + delete the published files. + +## Mode selection -Decide the mode from the instruction you were spawned with. If it says "publish", run -**Publish mode**. Otherwise run **Generate mode** on the PR list provided. +Decide the mode from the instruction you were spawned with: +- Instruction says "review" (e.g. "review all staged files", "review 1133-*.yml") → + **Review mode**. +- Instruction says "publish" → **Publish mode**. +- Otherwise → **Generate mode**, using the PR list provided. ## What you receive - **Generate mode:** a list of PR numbers (e.g. `1133 1134 1135`). +- **Review mode:** a list of specific filenames to review (e.g. `1133-add-mcp-abilities.yml`), + or no list → review **all** files currently under `.ai/testrail/pending/`. - **Publish mode:** the word "publish" and no PR list. You operate on whatever YAML files - already exist under `.ai/testrail/pending/`. + already exist under `.ai/testrail/reviewed/`. + +## Core principle — quality over quantity + +Before writing or keeping a single case, ask: **"Would a tester catching a regression here +be impossible without this test?"** If the answer is no, don't write it — or cut it. + +TestRail is not a log of what the code does. It is a safety net for things that could +silently break. Prefer 3 sharp cases over 10 vague ones. A PR that only changes a docblock, +adds unit tests, or tweaks a CSS colour needs **zero TestRail cases** — skip it entirely (in +Generate mode) or delete all its cases and remove the file (in Review mode), and say why in +your summary. + +## Imagify plugin context + +Imagify is a WordPress image-optimization plugin. Key feature areas: + +- **MCP / Abilities API** — REST-exposed abilities (OptimizeMedia, GetMediaStatus, + GetSettings, UpdateSettings, GetNextgenCoverage, GetStats, GetAccount). Each has a slug, + `check_permissions()`, and `execute()`. Requires WP ≥ 6.9. +- **Settings** — General settings (API key, optimization level, format), next-gen (WebP/AVIF). +- **Media library** — Per-image optimize/restore actions in WP admin. +- **Bulk optimization** — Queue-based bulk processing via Action Scheduler. +- **3rd-party** — WooCommerce product images, WP Rocket compatibility. +- **Capability model** — `imagify_capacity` filter gates all privileged actions; roles: + administrator (full access), editor (limited), subscriber (no access). + +Use this context both when drafting scenarios (Generate mode) and when judging whether a +case makes sense or is missing important scenarios (Review mode). ## Environment & constants @@ -45,6 +76,7 @@ Project ID : 3 Suite ID : 3 GitHub repo : wp-media/imagify-plugin Staging dir : .ai/testrail/pending/ +Reviewed dir : .ai/testrail/reviewed/ ``` Never print the API key. Both credentials are guaranteed present in the environment — do not @@ -53,8 +85,8 @@ stop; do not retry blindly. ### TestRail section map (suite 3) -Use this table to map each PR to a section. When a PR introduces a feature area that has no -fitting existing section, set `new_section` in the YAML and pick the most appropriate +Use this table to map each PR/case to a section. When content introduces a feature area that +has no fitting existing section, set `new_section` in the YAML and pick the most appropriate **parent** from this table for it. ``` @@ -217,8 +249,154 @@ Rules for the YAML: Print a concise summary: for each PR, the file written (or "skipped — already covered"), the target section, whether a new section is needed, and the case count with smoke-test count. -Then **stop**. Tell the user to review the YAML under `.ai/testrail/pending/` and run -`/testrail-scenarios publish` when satisfied. Do NOT create anything in TestRail. +Then **stop**. Tell the user to review the YAML under `.ai/testrail/pending/` (optionally via +Review mode) and run `/testrail-scenarios publish` when satisfied. Do NOT create anything in +TestRail. + +--- + +## Review mode workflow + +You are acting as a senior QA engineer reviewing staged scenarios **before** they are +published, and improving them by editing the YAML directly. You do NOT call the TestRail +API. You do NOT publish anything. + +Your primary job here is **removal and sharpening**, not addition. A lean file with 4 +precise cases is better than a bloated file with 12. + +If a file covers a PR that has no real functional change observable by a human or Canary +(docblock fix, unit-test-only PR, CSS tweak), delete all its cases and leave the file +empty with a `# skipped: no functional change` comment — or remove the file entirely and +note it in your report. + +### Review criteria + +For every file, evaluate each case against these criteria. Apply fixes directly — do not +just annotate. + +#### 1. Pertinence +Remove cases that: +- Test internal implementation details rather than user-observable behaviour. +- Duplicate what a unit/integration test already guards (e.g. "check that method X is + called") unless a human tester can actually observe the outcome. +- Are so trivial they add no signal (e.g. "page loads without error" alone, unless it is + the smoke test). + +#### 2. Coverage gaps +Add cases for obvious missing scenarios. Typical gaps: +- The primary **error path** when the happy path is covered but the failure is not. +- **Permission denial** — a non-admin trying the same action. +- **Missing prerequisite** — e.g. no API key configured, plugin deactivated mid-flow. +- **Boundary / edge** — zero, null, very large, special characters. + +Do not bloat the file — add only high-signal gaps, maximum 3 new cases per file. + +#### 3. Redundancy +Merge or remove cases that are near-identical (same action, same expected outcome, trivially +different input). If two cases differ only by a field value, collapse them into one case with +a note in `preconditions`. + +#### 4. Step clarity +Each step must be executable by a human tester or by Canary without guesswork: +- `action` must be a concrete UI action or API call, not a vague instruction like "use the + feature". +- `expected` must be a specific, observable outcome, not "it works" or "success". +- `preconditions` must list everything the tester must set up before step 1. + +Fix unclear steps in place. + +#### 5. Smoke test flags +`smoke_test: true` should be set on the **one or two** cases that represent the absolute +critical path (the plugin is broken without this). It should NOT be set on edge cases, +error paths, or permission checks. Correct misflags. + +#### 6. Section targeting +Verify that `section_id` and `new_section` are correct for the content using the section map +above. If a case clearly belongs to a different section than the rest, note it in your +report (do not split the file — that is a publish-time concern). + +#### 7. Title conventions +Titles must: +- Start with a feature-area prefix matching the section (e.g. `MCP - `, `Settings - `, + `Bulk - `, `Media - `). +- Be concise (≤ 80 chars) and describe the scenario, not the step. +- Be unique within the file. + +#### 8. Cross-file section coherence + +After reviewing individual files, do a **global pass** across all files being reviewed +together. Ask: do the `section_id` and `new_section` fields form a coherent, navigable +structure in TestRail? + +Rules: +- **Group related files under a shared subsection.** If several files cover the same + feature area (e.g. 10 files all about MCP Abilities, or 3 files all about Mixpanel + tracking), they must share the same `new_section` name — only the first file in + publication order creates it; the others must reuse it. Set `new_section: null` on the + followers and update their `section_id` to reference the parent where the new section + will be created. Add a comment `# section created by PR <N>` so the publisher knows the + dependency order. +- **Subsection naming** should reflect the feature area, not the PR title. Prefer short + noun phrases: `MCP Abilities`, `Mixpanel Tracking`, `Next-gen`, `Bulk Optimization`. + Avoid repeating the parent section name in the subsection (not `API Requests - MCP`, + just `MCP Abilities`). +- **Don't over-split.** If two files cover the same feature from different angles (e.g. + a feature PR and its bug-fix follow-up), they belong in the same subsection — not two + separate ones. +- **Don't under-group.** If files cover clearly distinct feature areas (e.g. MCP and + Mixpanel), keep them in separate subsections even if both land under the same parent. +- **Update title prefixes** in each file to match the final subsection name + (e.g. if the subsection is `MCP Abilities`, titles should start with `MCP - `). + +Report the grouping decisions in the review report under a `## Section coherence` heading +before the per-file breakdown. + +### Review workflow steps + +1. **List files to review.** + ```bash + ls .ai/testrail/pending/ + ``` + If none, say so and stop. + +2. **Read all files first** — before making any edits, read every file to understand the + full set of feature areas, section targets, and `new_section` values. This is the input + for the cross-file coherence pass. + +3. **Cross-file coherence pass** (criterion 8) — determine the correct grouping and + subsection names across all files. Produce a mapping: + `filename → { section_id, new_section }` for every file. This mapping takes precedence + over whatever was in the original YAML. + +4. **Ensure the `reviewed/` directory exists.** + ```bash + mkdir -p .ai/testrail/reviewed/ + ``` + +5. **For each file:** + a. Apply all per-file fixes (criteria 1–7). + b. Apply the section/new_section from the coherence mapping (criterion 8). + c. Update title prefixes if the subsection name changed. + d. Write the updated content to `.ai/testrail/reviewed/<filename>`. + e. Delete the original from `.ai/testrail/pending/<filename>`. + f. Build a short per-file diff summary (what was changed and why). + +6. **Print the review report** in this order: + - `## Section coherence` — the grouping decisions: which files share a subsection, + what the subsection names are, and which file creates each new section. + - Per-file breakdown: + ``` + ### <filename> + - Removed: <case title> — <reason> + - Added: <case title> — <reason> + - Fixed: <case title> — <what changed> + - No change: <N> cases unchanged + → Moved to .ai/testrail/reviewed/ + ``` + End with a one-line overall verdict per file: **Ready to publish** or **Review needed**. + +7. **Stop.** Do not publish. Tell the user they can run `/testrail-scenarios publish` when + satisfied, or edit the YAML files under `.ai/testrail/reviewed/` directly. --- @@ -226,7 +404,7 @@ Then **stop**. Tell the user to review the YAML under `.ai/testrail/pending/` an Run only when explicitly told to publish. For each YAML file under `.ai/testrail/reviewed/`. If `reviewed/` is empty or does not exist, check `pending/` and warn the user that those -files have not been through the reviewer yet, then ask whether to proceed anyway. +files have not been through review yet, then ask whether to proceed anyway. ### Step A — Create the section if needed @@ -304,9 +482,15 @@ remainder. ## DO NOT -- DO NOT create anything in TestRail in Generate mode (the dedup GET is the only call). +- DO NOT create anything in TestRail outside of Publish mode (the dedup GET in Generate mode + is the only API call allowed elsewhere). - DO NOT publish without an explicit publish instruction. -- DO NOT delete a staging YAML file unless all of its cases were created successfully. +- DO NOT delete a staging YAML file unless all of its cases were created successfully + (Publish mode) or its reviewed copy was successfully written (Review mode). - DO NOT re-create cases for a PR whose dedup check returned `size > 0`. - DO NOT print or log `$TESTRAIL_API_KEY`. - DO NOT invent section IDs — use the section map or create a section via `add_section`. +- DO NOT rewrite cases that are already clear and correct (Review mode) — leave them + untouched. +- DO NOT add `# REVIEW:` comments unless you are genuinely uncertain — prefer making the + call yourself. diff --git a/.claude/skills/testrail-review/SKILL.md b/.claude/skills/testrail-review/SKILL.md index a8bad5bc..2cc3073a 100644 --- a/.claude/skills/testrail-review/SKILL.md +++ b/.claude/skills/testrail-review/SKILL.md @@ -7,7 +7,8 @@ argument-hint: [filename(s) or blank for all] # TestRail Review Standalone entry point for reviewing staged TestRail scenario YAML files before they are -published. Spawns `testrail-review-agent` on whatever is staged (or a specific subset). +published. Spawns `testrail-scenario-agent` in **review mode** on whatever is staged (or a +specific subset). ## Invocation @@ -29,8 +30,8 @@ published. Spawns `testrail-review-agent` on whatever is staged (or a specific s ``` If the directory is empty or does not exist, tell the user there is nothing staged and stop. -3. **Spawn `testrail-review-agent`**, passing the file list (or "review all staged files" - if no list). The agent will edit the YAML directly and print a review report. +3. **Spawn `testrail-scenario-agent`** in review mode, passing the file list (or "review all + staged files" if no list). The agent will edit the YAML directly and print a review report. 4. **Relay the agent's report** verbatim, then remind the user: - To publish: `/testrail-scenarios publish` (reads from `.ai/testrail/reviewed/`) diff --git a/.claude/skills/testrail-scenarios/SKILL.md b/.claude/skills/testrail-scenarios/SKILL.md index 10620a57..6fbdbb64 100644 --- a/.claude/skills/testrail-scenarios/SKILL.md +++ b/.claude/skills/testrail-scenarios/SKILL.md @@ -61,9 +61,9 @@ You may also pass full PR URLs instead of `#`-prefixed numbers; both are accepte > "Would you like the TestRail reviewer to check the staged scenarios before publishing? > It will flag low-signal cases, fill coverage gaps, and clean up step wording. > Run `/testrail-review` or reply **yes** to launch it." - If the user replies yes (or any affirmative), spawn `testrail-review-agent` immediately - with "review all staged files" and relay its report. Otherwise proceed — the reviewer - is optional and skipping it is fine. + If the user replies yes (or any affirmative), spawn `testrail-scenario-agent` in review + mode immediately with "review all staged files" and relay its report. Otherwise proceed — + the reviewer is optional and skipping it is fine. ## Publish mode diff --git a/.claude/testrail/specs/_foundation.md b/.claude/testrail/specs/_foundation.md index 4202d2d1..ca2b21da 100644 --- a/.claude/testrail/specs/_foundation.md +++ b/.claude/testrail/specs/_foundation.md @@ -11,8 +11,15 @@ file plus the per-feature spec before executing any case. Locators captured live are role/id-based; verify against the page if the build moves. ## Environment -- Base URL: http://localhost:10038 (Local by Flywheel site "test-temp") -- WP admin: http://localhost:10038/wp-admin/ + +URLs and credentials are read from `.ai/settings.local.json` (gitignored). Two environments: + +| Key | URL | Server | Use when | +|-----|-----|--------|----------| +| `nginx` | http://localhost:10043 | Nginx | default — all cases unless `server: apache` | +| `apache` | http://localhost:10048 | Apache | cases involving `.htaccess`, rewrite rules, `$is_apache` paths | + +- WP admin: `$E2E_URL/wp-admin/` - Login: admin / admin via /wp-login.php role-based: getByLabel("Username or Email Address"), getByLabel("Password", { exact: true }) submit: getByRole("button", { name: "Log In" }) diff --git a/.claude/testrail/specs/mcp-abilities.md b/.claude/testrail/specs/mcp-abilities.md new file mode 100644 index 00000000..23d76b21 --- /dev/null +++ b/.claude/testrail/specs/mcp-abilities.md @@ -0,0 +1,296 @@ +--- +testrail_sections: [8724] +feature: "MCP Abilities" +source_files: [classes/Abilities/*.php, classes/MCP/*.php, config/providers.php, inc/classes/class-imagify-options.php] +derived_sha: 542b43d4 +last_explored: 2026-06-30 +--- + +## Overview +The Imagify MCP module exposes 7 abilities through the WordPress **Abilities API** (core, WP >= 6.9) +under the `imagify` category (TestRail section 8724 = MCP Abilities, parent 7685). Each ability is a +class in `classes/Abilities/` registered on the `wp_abilities_api_init` action by +`Imagify\MCP\AbilitiesSubscriber` (wired in `config/providers.php` via `Imagify\MCP\ServiceProvider`). +`Imagify\MCP\ConfigSubscriber` names the MCP server `imagify-plugin`. All 7 abilities gate on the +Imagify `manage` capability via `imagify_get_context('wp')->current_user_can('manage')` (the +`imagify_capacity` filter — NOT a bare `current_user_can()`), and all carry `meta.mcp.public = true` +and `meta.show_in_rest = true`, so they are reachable over the Abilities REST namespace +`wp-abilities/v1`. Registration no-ops silently when `wp_register_ability` is absent (WP < 6.9). + +SOURCE-OF-TRUTH NOTE (read before grounding drift): the live behaviour below was captured against +the **nginx** site `http://localhost:10043`, whose active plugin checkout is +`/Users/gaelrobin/Local Sites/e2eimagifynginx/.../imagify-plugin` on branch `feat/mixpanel` +(SHA 461e5839, plugin v2.2.9 header, but carrying the **2.3.0 abilities** with the +`AbstractAbility` base + `get_id()` template). This worktree (`imagify-e2e-worktree`, SHA 542b43d4) +carries an OLDER variant of the same classes (`implements AbilitiesInterface`, no `get_id()`); the +slugs, schemas, input wrapper, and responses are identical at the REST surface, so the grounding +holds. `derived_sha`/`source_files` track THIS worktree per `_foundation.md` convention. + +## Ground truth +The 7 abilities (slugs captured live from the discovery endpoint — all kebab-case, no underscores +in the local part, confirming C26383): + +| Slug | Label | readonly | destructive | idempotent | input | run method | +|------|-------|----------|-------------|------------|-------|-----------| +| `imagify/get-account` | Get Imagify account status | true | false | true | none | GET | +| `imagify/get-settings` | Get Imagify settings | true | false | true | none | GET | +| `imagify/get-stats` | Get Imagify optimization stats | true | false | true | none | GET | +| `imagify/get-nextgen-coverage` | Get next-gen coverage | true | false | true | none | GET | +| `imagify/get-media-status` | Get Media Status | true | false | true | `media_id` (int, required) | GET | +| `imagify/update-settings` | Update Imagify settings | false | false | true | partial settings object | POST | +| `imagify/optimize-media` | Optimize media | false | **true** | false | `media_id` (int, required), `optimization_level` (0-2 opt) | POST | + +- Discovery (`GET wp-abilities/v1/abilities`) returns ALL public abilities including 2 core ones + (`core/get-site-info`, `core/get-environment-info`); the 7 `imagify/*` slugs are a subset + (C26359 asserts the 7 imagify slugs are present, not a total count of 9). +- `imagify/optimize-media` is the ONLY destructive ability (`meta.annotations.destructive = true`, + captured live; C26384). All read abilities are `destructive:false, readonly:true, idempotent:true`. +- **Read-only abilities require GET** at the REST run endpoint: a POST returns HTTP 405 + `rest_ability_invalid_method` "Read-only abilities require GET method." (captured live on + get-settings; same applies to all 5 read abilities). Write abilities (update-settings, + optimize-media) use POST. +- `imagify/get-settings` strips `version` AND `api_key` from its output (GetSettings.php:117-118). + Live keys returned (17, captured live): `admin_bar_menu, auto_optimize, backup, cdn_url, + convert_to_avif, convert_to_webp, disallowed-sizes, display_nextgen, display_nextgen_method, + display_webp, display_webp_method, lossless, optimization_format, optimization_level, + partner_links, resize_larger, resize_larger_w`. `api_key`/`version` absent (C26362). +- `imagify/update-settings` output shape: `{ "updated": [<changed keys>], "settings": {<full + post-update settings minus api_key+version> } }` (UpdateSettings.php:340-343). `version` and + `api_key` are stripped from the returned `settings` too. +- update-settings rejection rules (captured live): + - unknown key OR `version` key → HTTP **500** body `{"code":"imagify_unknown_setting","message": + "\"<key>\" is not a valid Imagify setting key.","data":null}` (C26368). The check runs in the + foreach BEFORE `$options->set()`, so a `version`/unknown key mixed with a valid key causes a + FULL rejection with NO partial write (C26369 — verified live: a `{lossless:1, version:9}` call + returned the 500 and a follow-up get-settings showed `lossless` still 0). + - `optimization_level` out of [0,2] → HTTP **400** `ability_invalid_input` + "input[optimization_level] must be between 0 (inclusive) and 2 (inclusive)" — this is the + **input_schema** (`minimum:0, maximum:2`) rejecting BEFORE the PHP `validate_value()` even runs + (C26371). Distinguish: schema-level enum/range violations are 400 `ability_invalid_input`; + semantic key rejection is 500 `imagify_unknown_setting`. + - `api_key` with the `IMAGIFY_API_KEY` constant defined → `imagify_api_key_immutable` + (UpdateSettings.php:311-317). NOT reproducible live (constant undefined in this env) — C26370. + - empty input `{}` → HTTP 200 `{"updated":[],"settings":{...}}` (no error; C26372). Captured live. +- update-settings input enums (input_schema, UpdateSettings.php:64-155): `optimization_level` + int 0-2; `optimization_format` enum off|webp|avif; `display_nextgen_method`/`display_webp_method` + enum picture|rewrite; the rest of the 0/1 toggles are int enum [0,1]. +- `imagify/get-media-status` maps internal status → public: `success`|`already_optimized` → + `"success"`; `error` → `"error"`; everything else → `"unoptimized"` (GetMediaStatus.php:252-261). + It does NOT type-check the post — a non-attachment post id returns `"unoptimized"` (does not + crash; C26377). Error responses still return HTTP 200 with a `status:"error"` body: + - `media_id` <= 0 → `{status:"error", error_message:"Invalid or missing media_id", ...}` (C26376). + - non-existent id → `{status:"error", error_message:"Media not found.", ...}` (C26375 — error, + NOT "unoptimized"). All responses carry the full 7 fields (status, optimization_level, + original_size, optimized_size, webp_available, avif_available, error_message). +- `imagify/optimize-media` validates in order (OptimizeMedia.php:160-175): `media_id<=0` → + "Invalid or missing media_id."; missing post → "Invalid media."; non-attachment post → + "The provided ID is not a media attachment." ALL error responses carry the complete **5-field** + schema `{status, original_size, optimized_size, savings_percent, error_message}` with status + `"error"` and the four data fields `null` (C26389, captured live for all 3 error paths). Zero + media_id is rejected before any DB/post-type lookup (C26388); non-existent before the post-type + check (C26387); non-attachment fails the post-type check (C26386). HTTP 200 in all cases. +- `imagify/get-account`: when the API key is invalid/empty returns the zeroed shape + `{plan_label:"", quota:0, consumed_current_month_quota:0, extra_quota:0, extra_quota_consumed:0, + next_date_update:"", is_api_key_valid:false}` (GetAccount.php:131-141) — all 7 fields present + with schema-correct types (C26390, captured live: this env has NO valid key configured). +- `imagify/get-stats`: when nothing is optimized, `savings_percent` is `0` in JSON (PHP `0.0` + float cast; do not assert on JSON-vs-PHP float rendering — assert the value equals 0 and the + field is present; C26391). Live `wp` + `custom-folders` both zeroed in this empty env. +- `imagify/get-nextgen-coverage`: `{missing_nextgen_count:<int>, nextgen_format:<string>}` — + `missing_nextgen_count` is always cast `(int)` (GetNextgenCoverage.php:129), never boolean false + (C26381); `nextgen_format` mirrors the `optimization_format` setting (live value `"webp"`; C26382). + +## How to invoke (grounded from live) +All calls require an authenticated WP session + `X-WP-Nonce` header (anonymous discovery and run +both return HTTP **401**, confirming C26360). Acquire cookie + nonce per `_foundation.md` +(NOTE: foundation uses port 10038; the live env for THIS feature is the nginx site **10043**). +Base for this feature: `http://localhost:10043`. + +- **Discovery (C26359 / C26360):** + `GET /wp-json/wp-abilities/v1/abilities` (authed) → JSON array; filter `name` starting `imagify/` + and assert all 7 slugs present. Anonymous → 401. +- **Per-ability detail / annotations (C26384):** + `GET /wp-json/wp-abilities/v1/abilities/imagify/optimize-media` → `meta.annotations.destructive`. + The run URL is the ability's `_links["wp:action-run"][0].href` + (`…/abilities/imagify/<slug>/run`). +- **Run a READ ability (get-account, get-settings, get-stats, get-nextgen-coverage):** + `GET …/abilities/imagify/<slug>/run`. POST → 405. +- **Run get-media-status (read, takes input):** GET with the input as a **bracketed query param** + (the run arg is named `input`, an object): `…/imagify/get-media-status/run?input[media_id]=<id>` + (URL-encode the brackets: `input%5Bmedia_id%5D=<id>`). +- **Run a WRITE ability (update-settings, optimize-media):** `POST …/imagify/<slug>/run` with + `Content-Type: application/json` and body wrapping args under an **`input`** key: + - update-settings: `{"input":{"lossless":1}}` + - optimize-media: `{"input":{"media_id":<id>,"optimization_level":2}}` + WARNING: the run endpoint takes `{"name":..., "input":...}`; args MUST be under `input`. A raw + top-level args body returns `ability_invalid_input` "input is not of type object" (400). +- **MCP adapter:** the MCP server is named `imagify-plugin` (ConfigSubscriber). The adapter's + canonical server route/tools are left at defaults (ConfigSubscriber only overrides + `server_name`/`server_description`); cases here exercise the Abilities REST surface, which is the + grounded, observable entry point. + +## Locators (captured live — role-based preferred, then data-testid, then id) +This feature is REST/MCP-driven; most cases assert on JSON responses, not the DOM. The only UI leg +is C26364 (update-settings persists to the WP Admin settings UI). Settings page locators captured +live on `/wp-admin/options-general.php?page=imagify` (no data-testid on this page; reuse the +`SettingsPage` POM at `Tests/e2e/pages/settings.ts` + `loginAsAdmin` fixture, same style as +`account-connection.spec.ts`): +- Settings nav: `page.goto('/wp-admin/options-general.php?page=imagify')`. +- Lossless toggle (used by the C26364 UI-persistence check) — id (no role label wrapper observed): + `page.locator('#imagify_lossless')` (input type=checkbox), name + `[name="imagify_settings[lossless]"]`, associated `label[for="imagify_lossless"]` text + "Lossless compression". The label overlays the input; read its `checked` state, do not click to + verify. +- API key field: `page.locator('#api_key')` (name `imagify_settings[api_key]`). +- Save button: `page.locator('#submit')`; "Settings saved." confirmation: + `page.locator('#setting-error-settings_updated')`. +- Fatal-error guard (every case that loads a page): `page.locator('.wp-die-message, #error-page')` + → expect 0. + +## Prerequisites & seeding (per operation) +- **Plugin MUST be active** (HARD PREREQUISITE — discovered live): the abilities only register when + `imagify-plugin/imagify` is active. On first explore this plugin was **inactive** on 10043 and + ZERO imagify abilities appeared in discovery. Activate before any case: + `POST /wp-json/wp/v2/plugins/imagify-plugin/imagify {"status":"active"}` (auth + nonce), or + `wp plugin activate imagify-plugin` inside the Local site shell. Verify via + `GET /wp-json/wp/v2/plugins/imagify-plugin/imagify` → `status:"active"`. +- **WP >= 6.9** (live env is WP 7.0; Abilities API in core `wp-includes/abilities-api.php`). For + C26366 (no-op on WP < 6.9): cannot be reproduced on this env (7.0); verify by code/unit — when + `wp_register_ability` is undefined, `AbilitiesSubscriber::register_abilities()` returns early and + discovery contains no `imagify/*` slugs. Document as a version-guarded prerequisite. +- **Auth session + nonce** (all cases): cookie jar + `X-WP-Nonce` (see `_foundation.md`). + Anonymous = 401 (C26360 expects this). +- **Subscriber user** (C26365 — subscriber cannot execute any ability): seed + `wp user create mcp_sub mcp_sub@example.com --role=subscriber --user_pass=pass` inside the Local + shell; log in as that user, acquire a fresh nonce, and call each `…/run` endpoint → expect a + permission failure (the `manage` gate denies; `has_permission()` returns false and + `imagify_mcp_permission_denied` fires). +- **imagify_capacity denial** (C26378): hook the `imagify_capacity` filter to return a cap the + admin lacks (mu-plugin or `wp eval`), then call any `…/run` → all denied. Restore the filter after. +- **Multisite permission cases** (C26379 network non-admin denied / C26380 network admin granted): + require a WordPress **multisite** with the plugin network-activated. NOT available in this + single-site env — BLOCKED (same constraint as settings.md C174). Document as a multisite-only + prerequisite; skip on this env. +- **get-account valid-key path** (C26361 expects "expected fields"): live env has NO valid API key + (`is_api_key_valid:false`), so the zeroed shape is returned. C26361 asserts the FIELD SET (all 7 + keys present, types correct) which holds for both valid and invalid key — the invalid-key shape + satisfies the schema. To exercise the populated path, configure a valid key first + (`wp option patch update imagify_settings api_key "<valid key>"` in the Local shell) — optional. +- **optimize-media / get-media-status success cases** (C26363, C26374, C26385): require (a) a real + attachment AND (b) a VALID API key (optimization calls the Imagify API). The media library on + 10043 is EMPTY (live check returned 0 attachments). Seed an attachment via REST media upload + (helper in `_foundation.md`, adjust to port 10043), and configure a valid key. Without a valid + key, optimize-media returns a `status:"error"` (API not reachable), which still satisfies the + 5-field-schema cases (C26389) but NOT the success cases. Error/validation cases (C26375-C26377, + C26386-C26389) need NO attachment seeding and NO key — they were all captured live as-is. +- update-settings single-value cases (C26367, C26372): no seeding needed; snapshot the full + settings via `GET …/get-settings/run` BEFORE mutating (for teardown). + +## Verification criteria — "success" means (observable) +- C26359 (discovery, 7 slugs): authed `GET …/abilities` returns a JSON array whose `name` values + include exactly the 7 `imagify/*` slugs in the Ground-truth table (presence, not array length — + core slugs coexist). NOT a tautology: assert each of the 7 slug strings is present. +- C26360 (unauthenticated rejected): anonymous `GET …/abilities` AND anonymous `…/<slug>/run` + both return HTTP **401**. +- C26361 (get-account fields): authed `GET …/get-account/run` HTTP 200 with a JSON object + containing all 7 keys (`plan_label, quota, consumed_current_month_quota, extra_quota, + extra_quota_consumed, next_date_update, is_api_key_valid`) with schema-correct types. +- C26362 (get-settings no api_key/version): authed `GET …/get-settings/run` HTTP 200; response + object has NEITHER `api_key` NOR `version` key, and DOES contain the 17 user-facing keys. +- C26363 / C26385 (optimize a valid attachment): with a valid key + seeded attachment, `POST + …/optimize-media/run {"input":{"media_id":<id>}}` returns HTTP 200 `status:"success"` and (after + the queued job) the attachment gains `_imagify_status`/`_imagify_data` meta; the response + `original_size` is a positive int. (optimized_size may be 0/null until the background job + completes — assert status success + the meta, not an immediate size.) +- C26364 (update-settings persists to Admin UI): `POST …/update-settings/run + {"input":{"lossless":1}}` returns HTTP 200 `updated` containing `"lossless"`; THEN loading + `/wp-admin/options-general.php?page=imagify` shows `#imagify_lossless` **checked** (the option + written by the ability is reflected in the rendered settings form). Observable on both the REST + response and the rendered checkbox state. +- C26365 (subscriber denied): logged in as a subscriber, every `…/run` call is denied + (permission failure, not a successful ability result). +- C26366 (WP < 6.9 no-op): on WP < 6.9, discovery contains NO `imagify/*` slugs (abilities + unregistered). Cannot be asserted on WP 7.0 — verify via the early-return guard in + `AbilitiesSubscriber`/each ability's `register()`. BLOCKED on this env. +- C26367 (valid single update): `POST {"input":{"lossless":1}}` → HTTP 200, `updated == ["lossless"]`, + and `settings.lossless == 1`. (Captured live exactly.) +- C26368 (`version` rejected): `POST {"input":{"version":"9"}}` → HTTP **500**, + `code == "imagify_unknown_setting"`, message names `version`. +- C26369 (`version` + valid → full rejection, no partial write): `POST + {"input":{"lossless":1,"version":"9"}}` → HTTP 500 `imagify_unknown_setting`; AND a subsequent + `GET …/get-settings/run` shows `lossless` UNCHANGED from its pre-call value (no partial write). +- C26370 (api_key immutable with constant): with `IMAGIFY_API_KEY` defined, + `POST {"input":{"api_key":"x"}}` → `code == "imagify_api_key_immutable"`. BLOCKED on this env + (constant undefined) — verify by code or by defining the constant in wp-config first. +- C26371 (invalid optimization_level): `POST {"input":{"optimization_level":5}}` → HTTP **400**, + `code == "ability_invalid_input"`, message references the 0..2 range (schema rejection, not the + PHP validator). +- C26372 (empty input): `POST {"input":{}}` → HTTP 200, `updated == []`, no error, `settings` + present and unchanged. +- C26373 (unoptimized status): `GET …/get-media-status/run?input[media_id]=<unoptimized id>` → + HTTP 200 `status:"unoptimized"`, `optimization_level:null`. (Requires a real, un-optimized + attachment seeded.) +- C26374 (already-optimized success): for an optimized attachment, `status:"success"`, + `optimization_level` an int 0..2, `original_size`/`optimized_size` positive ints. (Requires a + valid key + an optimized attachment.) +- C26375 (non-existent → error not unoptimized): `?input[media_id]=999999` → HTTP 200 + `status:"error"`, `error_message:"Media not found."` (NOT "unoptimized"). Captured live. +- C26376 (zero/negative media_id → error): `?input[media_id]=0` (and `-3`) → HTTP 200 + `status:"error"`, `error_message:"Invalid or missing media_id"`. Captured live. +- C26377 (non-attachment post does not crash): `?input[media_id]=1` (a regular post) → HTTP 200, + no fatal, `status:"unoptimized"` (get-media-status does not type-check). Captured live. +- C26378 (imagify_capacity denial blocks all): with the filter forced to deny, every `…/run` + call is denied. (Seed the filter; see Prerequisites.) +- C26379 / C26380 (multisite non-admin denied / network admin granted): BLOCKED — needs multisite. +- C26381 (missing_nextgen_count is int): `GET …/get-nextgen-coverage/run` → `missing_nextgen_count` + is an integer (e.g. `0`), JSON type number, NOT boolean `false`. Captured live (`0`). +- C26382 (nextgen_format reflects setting): the response `nextgen_format` equals the current + `optimization_format` option. Live: setting `optimization_format == "webp"` and coverage + `nextgen_format == "webp"` — match. To strengthen, set `optimization_format` to `avif` via + update-settings, re-call, expect `"avif"`, then restore. +- C26383 (slugs kebab-case, no underscores): each of the 7 `imagify/*` slugs' local part matches + `^[a-z]+(-[a-z]+)*$` with no `_`. Captured live: all 7 pass. +- C26384 (optimize-media destructive=true): `GET …/abilities/imagify/optimize-media` → + `meta.annotations.destructive === true` (and `readonly:false, idempotent:false`). Captured live. +- C26385 — see C26363. +- C26386 (non-attachment → error, no optimization): `POST …/optimize-media/run + {"input":{"media_id":1}}` → HTTP 200 `status:"error"`, + `error_message:"The provided ID is not a media attachment."`, all 4 data fields null; no + optimization meta written. Captured live. +- C26387 (non-existent → error before post-type check): `{"input":{"media_id":999999}}` → HTTP 200 + `status:"error"`, `error_message:"Invalid media."`. Captured live. +- C26388 (zero media_id rejected pre-DB): `{"input":{"media_id":0}}` → HTTP 200 `status:"error"`, + `error_message:"Invalid or missing media_id."`. Captured live. +- C26389 (all error responses carry the 5-field schema): every optimize-media error response + contains exactly `status, original_size, optimized_size, savings_percent, error_message` with + `status:"error"` and the other four `null`. Verified live across the 3 error paths. +- C26390 (get-account types when key invalid): with an invalid/empty key, `GET …/get-account/run` + returns all 7 fields with correct types — `is_api_key_valid:false` (boolean), `plan_label:""` + (string), numeric quota fields `0` (number), `next_date_update:""` (string). Captured live. +- C26391 (get-stats savings_percent 0.0 when none optimized): `GET …/get-stats/run` → + `wp.savings_percent == 0` and `custom-folders.savings_percent == 0`, field present and numeric + (PHP float 0.0; do not over-assert JSON float formatting). Captured live (both 0). + +## Teardown (LIFO) +Restore in reverse order of seeding. Each case is independent; undo only what it changed. +1. **Restore any mutated setting** (C26364, C26367, C26369 if it had partially written — it does + not, but snapshot anyway): snapshot via `GET …/get-settings/run` BEFORE the case, then restore + the changed key via `POST …/update-settings/run` with the original value (e.g. + `{"input":{"lossless":0}}` — live pre-test value was 0), or + `wp option update imagify_settings '<json snapshot>' --format=json` in the Local shell. + (This explore set then reverted `lossless` to 0; verified clean.) +2. **Restore `optimization_format`** if changed for C26382 (back to `"webp"`). +3. **Delete any seeded attachment** (C26363/C26373/C26374/C26385) and its Imagify meta: + `wp post meta delete <id> _imagify_data` ; `wp post meta delete <id> _imagify_status`, or + `DELETE /wp-json/wp/v2/media/<id>?force=true` (auth + nonce). +4. **Remove the imagify_capacity denial filter** (C26378) — drop the mu-plugin / undo the + `wp eval` hook so the admin regains `manage`. +5. **Delete the subscriber test user** (C26365): `wp user delete mcp_sub --yes --reassign=1`. +6. **Remove a valid API key** if one was configured only for the success cases: + `wp option patch update imagify_settings api_key ""` (Local shell). +7. **Plugin activation:** the explore left `imagify-plugin` ACTIVE (the run agent requires it + active for every case). If the run protocol requires restoring the pre-run inactive state, + `POST /wp-json/wp/v2/plugins/imagify-plugin/imagify {"status":"inactive"}` LAST — but note this + disables every ability, so do it only after all MCP cases complete. +8. C26366 / C26370 / C26379 / C26380: n/a in this env (not executed — version/multisite/constant + prerequisites unmet). diff --git a/.claude/testrail/specs/mixpanel-tracking.md b/.claude/testrail/specs/mixpanel-tracking.md new file mode 100644 index 00000000..2585cbe1 --- /dev/null +++ b/.claude/testrail/specs/mixpanel-tracking.md @@ -0,0 +1,251 @@ +--- +testrail_sections: [8725] +feature: "Mixpanel Tracking / Analytics" +source_files: [classes/Tracking/*.php, views/part-settings-analytics.php, views/notice-analytics-thankyou.php, classes/Abilities/AbstractAbility.php, classes/Abilities/OptimizeMedia.php, vendor/wp-media/wp-mixpanel/src/Optin.php, vendor/wp-media/wp-mixpanel/src/TrackingPlugin.php, vendor/wp-media/wp-mixpanel/src/Tracking.php, vendor/wp-media/wp-mixpanel/src/WPConsumer.php] +derived_sha: 542b43d4 +last_explored: 2026-06-30 +--- + +## Overview +Imagify's Mixpanel analytics tracking (TestRail section **8725** "Mixpanel Tracking", parent 33). +Anonymous, opt-in usage telemetry added in plugin **2.3.0**. Two surfaces emit the same Mixpanel +events: the UI/optimization path (`Imagify\Tracking\Subscriber` → `Tracking`) and the MCP ability +path (`McpTrackingSubscriber` → `McpTracking`, context overridden to `wp_plugin_mcp`). Every event +is gated by a single opt-in flag and only sent when the admin has explicitly enabled "Imagify +Analytics" on the settings page. The opt-in toggle is **unchecked by default** and is a separate +control from the General Settings form — it persists via its own AJAX action, NOT the settings Save. + +Events are delivered **server-side** via `wp_remote_post` (PHP) to the Mixpanel proxy host, so they +do NOT appear in the browser HAR/Network panel — see "How to verify". The Strauss-prefixed Mixpanel +library lives under namespace `Imagify\Dependencies\WPMedia\Mixpanel` (class alias +`Imagify_WPMedia_ConsumerStrategies_AbstractConsumer`), confirming the dependency was prefixed. + +## Ground truth +Captured live this explore unless noted. Mixpanel token (`ServiceProvider::MIXPANEL_TOKEN`): +`517e881edc2636e99a2ecf013d8134d3`; application `imagify`, brand `wp media`. + +- **Opt-in storage**: WP option **`imagify_mixpanel_optin`** (slug `imagify` + `_mixpanel_optin`, + `Optin.php`). `enable()` → `update_option(..., true)`; `disable()` → **`delete_option(...)`** + (the row is removed, not set to 0). Default: option absent ⇒ `can_track()` false ⇒ no events. +- **Opt-in capability**: `manage_options` (DI: `Optin(application='imagify', 'manage_options')`, + `ServiceProvider.php:67`). `is_enabled()` returns false for users lacking the cap; the AJAX + toggle returns **403** for non-`manage_options` users (`Notices::ajax_toggle_optin`). +- **Events emitted** (all via `TrackingPlugin::track_direct( $event, $props )`): + | Event name | Source class / method | Fires when | + |---|---|---| + | `Media Optimized` (UI) | `Tracking::track_media_optimized` | hook `imagify_after_optimize`, only when `'full'` size done AND full-size `success` | + | `Media Restored` | `Tracking::track_media_restored` | hook `imagify_after_restore_media`, only when `$response` not WP_Error | + | `Settings Saved` | `Tracking::track_settings_saved` | hook `update_option_imagify_settings` / `update_site_option_imagify_settings` | + | `Internal State Reset` | `Tracking::track_internal_state_reset` | hook `imagify_after_reset_internal_state`; prop `is_multisite` | + | `Media Optimized` (MCP) | `McpTracking::track_media_optimized` | only ability `imagify/optimize-media` AND `result.status === 'success'`; adds `initiated_via: 'mcp'`, `execution_time_ms` | + | `MCP Ability Executed` | `McpTracking::track_ability_executed` | EVERY ability invocation (all 7), regardless of outcome; props `ability_id`, `ability_name`, `execution_time_ms` | + | `MCP Ability Permission Denied` | `McpTracking::track_permission_denied` | ability `check_permissions()` returns false; props `ability_id`, `ability_name`, `required_capability`, `user_role` | +- **The 7 abilities** (each fires `MCP Ability Executed` when run via MCP/abilities REST; captured + from source slugs): `imagify/get-account`, `imagify/get-media-status`, `imagify/get-settings`, + `imagify/get-stats`, `imagify/optimize-media`, `imagify/get-nextgen-coverage`, + `imagify/update-settings`. +- **Required/default properties on every event** (`BaseTracking::get_default_event_properties`): + `context` (`wp_plugin` for UI, **`wp_plugin_mcp`** for MCP — overridden in `McpTracking`), + `license_owner` (**SHA-256 of the Imagify account email**, `hash('sha256', $user->email)`; empty + string when the user/email is unavailable — confirmed empty in the REST-run capture this explore), + `user_id`. The library then **auto-injects** (do NOT set these in event props — silently + overwritten): `domain` (a hashed value, not raw), `wp_version`, `php_version`, `plugin` + (`imagify <version>`), `brand`, `application`, plus `token`, `time`, `mp_lib: php`. +- **`next_gen_format`** values: `'avif'` (when AVIF present/enabled, highest priority), `'webp'`, + or `null`. UI path resolves from per-size optimization data; MCP path resolves from the + `convert_to_avif`/`convert_to_webp` settings. +- **`trigger`** (UI `Media Optimized` only): `'auto'` (item `is_new_upload`), `'bulk'` (a + `imagify_wp_optimize_running`/`imagify_custom-folders_optimize_running` transient is set), else + `'manual'`. (`Tracking::resolve_trigger`.) +- **`Media Optimized` thumbnails-only guard**: the UI event is suppressed unless the **`full`** + size is in `item['sizes_done']` and the full-size data has `success` truthy — i.e. a + thumbnails-only optimization does NOT emit `Media Optimized`. +- **AJAX toggle**: action **`wp_ajax_imagify_toggle_tracking_optin`**, nonce action + **`imagify_tracking_optin`** (`check_ajax_referer(..., 'nonce')`), POST field `value` (1/0). + On enable it also sets the transient **`imagify_analytics_optin_thanks`** (60s) which renders the + thank-you notice once on the next page load (`Notices::render_thankyou_notice`, read-once then + `delete_transient`). +- **Mixpanel network destination** (DESTRUCTIVE / external): `wp_remote_post` to + `https://mixpanel-proxy.group.one/track/?ip=0` (`vendor/.../Tracking.php` HOST const + events + endpoint). For tests, intercept this via a `pre_http_request` spy — DO NOT send live data. + +## How to invoke (grounded from live) +- **Analytics opt-in UI**: GET `http://localhost:10043/wp-admin/options-general.php?page=imagify`. + The opt-in section renders inside the General Settings form (hook `imagify_settings_after_lossless`), + so it requires a valid API key (same gate as General Settings; live env satisfies this). +- **Toggle opt-in (AJAX, what the checkbox JS does — captured live)**: POST + `/wp-admin/admin-ajax.php` form fields `action=imagify_toggle_tracking_optin`, + `nonce=<data-nonce off the checkbox>`, `value=1|0`. Live results this explore: + - valid `value=1` → HTTP **200** body `{"success":true}`, option set, thank-you transient set. + - valid `value=0` → HTTP **200**, option deleted. + - bad nonce → HTTP **403** body `-1` (`check_ajax_referer` die). +- **MCP / abilities (fires `MCP Ability Executed`)**: run any ability via the abilities REST + endpoint, e.g. GET `/wp-json/wp-abilities/v1/abilities/imagify/get-stats/run` with `X-WP-Nonce` + + session cookie (401 anonymously; see `_foundation.md` auth). Captured live: a `get-stats` run + produced exactly one `MCP Ability Executed` event with `context: wp_plugin_mcp`, + `ability_id: imagify/get-stats`, `ability_name: "Get Imagify optimization stats"`, + `execution_time_ms`, and the auto-injected props (`domain`/`wp_version`/`php_version`/`plugin`/ + `brand`/`application`/`token`/`time`/`mp_lib: php`). +- **UI-triggered optimization does NOT fire the MCP event**: the UI path uses `Subscriber` + (`imagify_after_optimize`), which calls `Tracking` (context `wp_plugin`), never `McpTracking`. + The MCP events come ONLY from the `imagify_mcp_ability_executed` / `imagify_mcp_permission_denied` + actions fired in `AbstractAbility` (`check_permissions()` + `do_execute` wrapper). + +## Locators (captured live — role-based preferred, then data-testid, then id) +Verified live this explore on `/wp-admin/options-general.php?page=imagify` (login admin/admin; +reuse the `SettingsPage` POM at `Tests/e2e/pages/settings.ts` + `wp-login` fixture). No +`data-testid` on this UI; fall back to id/class. The opt-in checkbox was **unchecked** on a fresh +load (matches "unchecked by default"). + +- Opt-in section wrapper: `page.locator('.imagify-analytics-optin')` → 1 match. +- Opt-in checkbox (C: unchecked by default / toggle ON / toggle OFF) — PREFERRED role-based: + `page.getByLabel('Imagify Analytics')` → 1 match. + - id fallback: `page.locator('#imagify-analytics-enabled')` (input type=checkbox, value="1"). + - The checkbox carries the AJAX nonce as **`data-nonce`** (read it for the toggle call): + `page.locator('#imagify-analytics-enabled').getAttribute('data-nonce')` (10-char nonce live). + - associated label: `page.locator('label[for="imagify-analytics-enabled"]')`. The toggle UI + (`.imagify-analytics-toggle-ui` span) overlays the input, so click the label or use + `{ force: true }` on the checkbox (same overlay pattern as the auto-optimize toggle in settings.md). + - assert state: `await expect(checkbox).not.toBeChecked()` on fresh load. +- Description + "What info" link text: `page.locator('.imagify-analytics-description')` → 1 match, + text "I agree to share anonymous data with the development team to help improve Imagify. What + info will we collect?". +- "What info will we collect?" modal trigger (C: What-info modal): + `page.locator('.imagify-modal-trigger')` → 1 match, text "What info will we collect?". + (It is a `<button class="imagify-btn-link imagify-modal-trigger" href="#imagify-analytics-info-modal">`.) +- Modal + its contents: + - `page.locator('#imagify-analytics-info-modal')` → 1 (role="dialog", aria-hidden toggled by JS). + - data table inside modal: `page.locator('.imagify-analytics-data-table')` → 1 (rows: WordPress + version, PHP version, Plugin version, Optimization level, Media type, Savings percentage, + Next-gen format, Trigger, License type). + - activate-from-modal button (only rendered when opt-in is OFF): + `page.locator('#imagify-analytics-enable-from-modal')` → 1 (text "Activate Imagify Analytics"). +- Thank-you notice (C: thank-you notice after enabling) — captured live after a valid toggle ON + + reload: `page.locator('.imagify-analytics-thankyou-notice')` (classes + `notice notice-success is-dismissible`); contains `p > strong` "Thank you!" and the same + `.imagify-analytics-data-table`. Read-once (transient deleted on render), so it appears on the + FIRST page load after enabling and not subsequently. +- Save button (regression: saving settings must NOT touch opt-in): `page.locator('#submit')` + (POM `settings.saveButton`). Saving the General Settings form does not POST the opt-in field + (the opt-in is a separate AJAX action), so opt-in state is unchanged by a settings Save. +- Fatal-error guard (every case): `page.locator('.wp-die-message, #error-page')` → expect 0 + (POM `settings.expectNoFatalError()`). + +## Prerequisites & seeding (per operation) +General: valid API key configured so the opt-in section renders (live env satisfies; see +`_foundation.md`). Each tracking assertion needs (a) a known opt-in state and (b) an outbound +Mixpanel-request spy — because events are sent server-side and never reach the browser network. + +- **Mixpanel event spy (REQUIRED to observe any event)** — events are PHP `wp_remote_post`, NOT + visible in HAR. Install a temporary **mu-plugin** that hooks `pre_http_request`, matches the host + `mixpanel-proxy.group.one` (or `mixpanel.com`), base64-decodes the `data=` body to capture the + event name + properties, records them (option/transient), and returns a fake 200 to block the + real call. This explore verified the exact path: a `get-stats` MCP run produced one + `MCP Ability Executed` event with all documented props. Expose captured events to the test via a + small admin-only REST route (e.g. `GET /wp-json/imagify-spy/v1/events`) or read the option in the + Local site shell. Reset before each case (`DELETE` the spy option). Teardown: delete the mu-plugin. +- **Opt-in ON** (events should fire): POST the AJAX toggle `value=1` with the live `data-nonce` + (preferred — exercises the real path), or in the Local shell `wp option update imagify_mixpanel_optin 1`. +- **Opt-in OFF** (events must NOT fire): toggle `value=0`, or in the Local shell + `wp option delete imagify_mixpanel_optin`. Default state has the option absent. +- **`Media Optimized` (UI, trigger=auto)**: opt-in ON, `auto_optimize=1`, upload a fresh supported + image (REST media helper in `_foundation.md`) so `imagify_after_optimize` runs with the full size. +- **`Media Optimized` (UI, trigger=bulk)**: opt-in ON, run bulk optimization + (`/wp-admin/upload.php?page=imagify-bulk-optimization`) — the bulk transient makes `resolve_trigger` + return `'bulk'`. +- **Thumbnails-only suppression**: opt-in ON, optimize only thumbnail sizes (no `full` in + `sizes_done`) → assert NO `Media Optimized` event captured. +- **`Media Optimized` via MCP**: opt-in ON, run ability `imagify/optimize-media` with a `media_id` + for an unoptimized attachment via abilities REST (auth+nonce) → expect both `MCP Ability Executed` + AND `Media Optimized` (with `initiated_via: 'mcp'`). +- **`MCP Ability Executed` for all 7**: opt-in ON, run each ability slug once via abilities REST + (get-account, get-media-status, get-settings, get-stats, optimize-media, get-nextgen-coverage, + update-settings). update-settings/optimize-media mutate state — snapshot + teardown (below). +- **`MCP Ability Permission Denied`**: opt-in ON, seed a non-admin user (subscriber) per + `_foundation.md`, log in as them, run an ability whose `check_permissions()` fails → expect the + permission-denied event with `user_role` of that user. (Note opt-in is a per-site option, so it + is ON regardless of who is logged in; the denied event still fires because `can_track()` only + reads the option.) +- **`Internal State Reset`**: opt-in ON, trigger `imagify_after_reset_internal_state` (see + `reset-internal-state.spec.ts` / the reset action) → expect `Internal State Reset` with + `is_multisite` boolean. +- **`Settings Saved`**: opt-in ON, change any setting and Save (or `update-settings` ability) → + expect `Settings Saved` with the boolean-mapped properties (`auto_optimize_on_upload`, `lossless`, + `backup_original`, etc.). +- **Saving does not touch opt-in (regression)**: set a known opt-in state, Save the General Settings + form, re-read `imagify_mixpanel_optin` → unchanged. +- **Invalid nonce rejected / non-admin blocked (Analytics Toggle)**: POST the AJAX toggle with a + bogus nonce → 403 `-1`; as a non-admin → 403 "Unauthorized." (`ajax_toggle_optin` cap check). + +## Verification criteria — "success" means (observable) +Opt-in gating is the spine: with opt-in OFF, the spy captures ZERO Imagify events for any action; +with opt-in ON, the specific event below is captured with the named properties. Never assert merely +"page loaded". + +- **Opt-in unchecked by default**: on a fresh load (option absent), `#imagify-analytics-enabled` + is **not checked** (`expect(...).not.toBeChecked()`). +- **Toggle ON**: AJAX `value=1` → HTTP 200 `{"success":true}`; on reload the checkbox is checked and + option `imagify_mixpanel_optin` is truthy. +- **Thank-you notice**: after a fresh enable, the next admin page load shows + `.imagify-analytics-thankyou-notice` with "Thank you!" and the data table; it does NOT reappear on + the subsequent load (transient consumed). +- **Toggling OFF removes opt-in**: AJAX `value=0` → 200; option row is **deleted** (`get_option` + returns false/default), checkbox unchecked on reload, and subsequent actions capture no events. +- **"What info" modal**: clicking `.imagify-modal-trigger` reveals `#imagify-analytics-info-modal` + containing `.imagify-analytics-data-table` with the nine listed metric rows. +- **Non-admin blocked**: AJAX toggle as a subscriber/editor → HTTP 403, opt-in option unchanged. +- **Invalid nonce rejected**: AJAX toggle with a wrong nonce → HTTP 403 body `-1`, option unchanged. +- **Saving settings does not touch opt-in**: opt-in option value is identical before and after a + General Settings Save (no event of type opt-in-change; opt-in is AJAX-only). +- **`Media Optimized` (UI)**: with opt-in ON and a full-size successful optimization, the spy + captures exactly one `Media Optimized` event whose props include `context: wp_plugin`, + `optimization_level` (int), `media_type` (mime), `original_size`/`optimized_size` (int), + `savings_percent` (number), `next_gen_format` (`avif|webp|null`), and `trigger` + (`auto` for a new upload / `bulk` during bulk). `license_owner` is a 64-hex SHA-256 of the account + email (or empty string if unavailable). +- **`next_gen_format` correctness**: equals `avif` when AVIF was generated/enabled, else `webp` when + only WebP, else `null` — matching the site's conversion settings / per-size data. +- **Thumbnails-only**: with opt-in ON, optimizing only non-full sizes captures NO `Media Optimized` + event (the `'full' in sizes_done` + full-success guard fails). +- **`Media Optimized` via MCP**: running `imagify/optimize-media` (success) captures a + `Media Optimized` event with `context: wp_plugin_mcp`, `initiated_via: 'mcp'`, and + `execution_time_ms` present — AND a separate `MCP Ability Executed` event for the same call. +- **`MCP Ability Executed` for all 7 abilities**: each of the seven ability runs captures exactly + one `MCP Ability Executed` event with `context: wp_plugin_mcp`, the correct `ability_id` + (matching the slug), a non-empty `ability_name`, and a numeric `execution_time_ms`. (Verified live + for `imagify/get-stats`.) +- **`MCP Ability Permission Denied`**: a denied ability run captures one + `MCP Ability Permission Denied` event with the `ability_id`, `ability_name`, + `required_capability` ("manage"), and the denied user's `user_role`. +- **Context field MCP vs UI**: MCP-path events carry `context: wp_plugin_mcp`; UI-path events carry + `context: wp_plugin`. A UI-triggered optimization captures NO `MCP Ability Executed` event. +- **`Internal State Reset`**: the reset action captures one `Internal State Reset` event with an + `is_multisite` boolean property. +- **`license_owner` is SHA-256**: when an account email is available, `license_owner` is a 64-char + lowercase hex string equal to `sha256(email)` — never the raw email (privacy assertion). +- **Strauss prefixing (regression)**: the Mixpanel library is namespaced under + `Imagify\Dependencies\WPMedia\Mixpanel` (and aliases like + `Imagify_WPMedia_ConsumerStrategies_AbstractConsumer`); no unprefixed top-level `Mixpanel`/ + `WPMedia` class collides. Observable by class_exists checks or that tracking works without fatals. +- **Opt-out = total silence (master assertion)**: with opt-in OFF, NONE of the above actions + (optimize, restore, settings save, reset, any of the 7 MCP abilities, permission denied) produce + any captured event — `can_track()` short-circuits every tracker before `track_direct`. + +## Teardown (LIFO) +Restore in reverse order of seeding; each case undoes only what it changed. +1. Restore opt-in to its pre-test state: it is OFF by default (option absent), so after a test that + enabled it, disable via AJAX `value=0` or `wp option delete imagify_mixpanel_optin`. (This explore + left it OFF.) +2. Delete any attachment seeded for `Media Optimized`/`Media Restored` and its Imagify meta + (`_imagify_data`, `_imagify_status`) — see `_foundation.md` teardown helpers. +3. Restore any settings snapshot taken for `Settings Saved` / `update-settings` cases + (`POST update-settings <snapshot>` or `wp option update imagify_settings '<json>' --format=json`). +4. Delete any non-admin test user seeded for the permission-denied case + (`wp user delete <login> --yes --reassign=1`). +5. Remove the Mixpanel-spy mu-plugin (`wp-content/mu-plugins/<spy>.php`) and delete its capture + option/transient. (NOTE: this explore left an orphaned `imagify_spy_events` option in the + localhost:10043 DB after removing the spy mu-plugin; it is inert — no code reads it — but can be + cleared with `wp option delete imagify_spy_events` in the Local site shell.) +6. Clear any leftover `imagify_analytics_optin_thanks` transient if a thank-you case was interrupted + before the notice rendered (`wp transient delete imagify_analytics_optin_thanks`); normally it is + read-once and self-clears. diff --git a/.claude/testrail/specs/settings.md b/.claude/testrail/specs/settings.md index fb69561b..698b8bac 100644 --- a/.claude/testrail/specs/settings.md +++ b/.claude/testrail/specs/settings.md @@ -33,7 +33,7 @@ via `imagify_get_context('wp')->current_user_can('manage')` (see `_foundation.md - disabled → the action **`imagify_new_attachment_auto_optimization_disabled`** fires and the method returns early; no `_imagify_*` meta is written for the new upload. - Gate also requires a valid API key (`Imagify_Requirements::is_api_key_valid()`). -- Temporary admin notices (C14169 — escaping): rendered by `views/notice-temporary.php`, which +- Temporary admin notices (C14169 — escaping) **`server: apache`**: rendered by `views/notice-temporary.php`, which emits the message through **`wp_kses( $details['message'], [ 'code' => [] ] )`** — i.e. only `<code>` survives, every other tag is stripped. Notice messages legitimately contain `<code>` (e.g. the .htaccess error in `classes/Webp/Display.php:114` wraps a file path in `<code>…</code>`). From 3cb5fdf45dad414dc573170a8c4739de4b9c8703 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Wed, 1 Jul 2026 03:48:35 +0200 Subject: [PATCH 10/16] =?UTF-8?q?docs(qa):=20remove=20superseded=20design?= =?UTF-8?q?=20docs=20=E2=80=94=20A+=20architecture=20is=20implemented?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A-plus-design.md's core architecture (Explorer/Executor split, grounded specs, drift detection, state isolation) is now built in testrail-explorer-agent.md and testrail-run-agent.md. canary-testrail-spec.md and canary-wp-spec.md were pre-build design drafts, superseded by the same implementation. --- .claude/canary-testrail-spec.md | 667 ------------------------------ .claude/canary-wp-spec.md | 374 ----------------- .claude/testrail/A-plus-design.md | 417 ------------------- 3 files changed, 1458 deletions(-) delete mode 100644 .claude/canary-testrail-spec.md delete mode 100644 .claude/canary-wp-spec.md delete mode 100644 .claude/testrail/A-plus-design.md diff --git a/.claude/canary-testrail-spec.md b/.claude/canary-testrail-spec.md deleted file mode 100644 index ebe21c8a..00000000 --- a/.claude/canary-testrail-spec.md +++ /dev/null @@ -1,667 +0,0 @@ -# Spec: Canary × TestRail — Consolidated QA Automation for Imagify - -**Status:** Implemented — `test/e2e-agent` branch. All 13 audit questions resolved. - ---- - -## 0. Orientation — read this first - -This spec consolidates three QA-automation features for the **Imagify WordPress plugin** -(`wp-media/imagify-plugin`) into one design document: - -1. **Feature 1 — TestRail scenario generation:** an agent analyses one or more GitHub PRs, - drafts human-readable TestRail test scenarios, stages them for review, and (on "publish") - creates them in TestRail in the right section, deduplicating against PRs already covered. -2. **Feature 2 — TestRail run execution:** an agent fetches every test scenario in a TestRail - run (by milestone or run ID), drives a real browser via **Canary** to execute each one, - collects pass/fail/blocked outcomes with artifacts, and (on confirm) posts results back to - TestRail with evidence. -3. **Feature 3 — Canary E2E in the delivery pipeline:** a new `canary-e2e` agent, a drop-in - replacement for the existing `e2e-qa-tester`, that records rich Canary sessions (trace, - video, HAR, console) per QA flow during a normal issue-workflow run, selectable at - orchestrator startup via an `e2e_mode` flag. - -All three share one substrate: **Canary** (browser recording + QA platform) and a set of -**battle-tested WordPress fixture snippets** wrapped in a reusable **agent inheritance pattern** -(WP base → Imagify-specific). - -### What Canary is (engine + sandbox) - -Canary is a browser recording + QA platform built on two npm packages, used as-is via `npx` -(**we never fork or modify them**): - -| Package | Command | Role | -|---------|---------|------| -| `@usecanary/cli` | `npx @usecanary/cli` | Playwright daemon + QuickJS sandbox + session lifecycle | -| `@usecanary/ui` | `npx @usecanary/ui` | Local web viewer for recorded sessions | - -What we own and customise is the **skill pack** — the markdown agent/skill files that drive the CLI. - -**QuickJS sandbox constraints (the step-script runtime — critical, non-negotiable):** - -- No `require()` / `import` — ESM and CJS both fail. -- No Node.js APIs — no `fs`, `path`, `http`, `process.env`. -- No top-level `fetch()` — QuickJS has no fetch. Run fetch inside the browser via - `page.evaluate(async () => fetch(...))`. -- No helper files — every utility must be **inlined verbatim** in each step script. -- Globals available: `browser` (Canary's Playwright wrapper), `saveScreenshot`, `console`. -- One browser **persists across all steps in a session** — you log in once, in step 1. - -**Session lifecycle:** - -```bash -npx @usecanary/cli session start --name "Flow name" --capture trace,video,har,console -npx @usecanary/cli run <step.js> # repeat per step; appends to session.json["steps"] -npx @usecanary/cli session end # writes results.json + report.html + artifacts -``` - -Session ID format: `<slug>-<random8>-<short-hash>` — e.g. -`p0-a--mcp-abilities-discovery-mqubrrhk-3f7e94`. - -**Artifacts per session** at `.ai/{N}/canary/<id>/` (relocated from `~/.canary/sessions/<id>/` after `session end`; symlink left at original for the viewer): - -``` -session.json ← session metadata + step scripts (written during run) -results.json ← step results + summary (written at session end) -report.html ← self-contained pass/fail report (written at session end) -trace.zip ← Playwright trace (npx playwright show-trace trace.zip) -video/ ← .webm recording -network.har ← all HTTP traffic -console.log ← browser console output -screenshots/ ← PNG per saveScreenshot() call -manifest.json ← artifact inventory -profile/ ← browser profile -``` - -**`results.json` schema (the contract every Canary-driven agent reads):** - -```json -{ - "summary": { - "stepsPassed": 10, "stepsFailed": 0, "stepsTotal": 10, - "commandCount": 34, "consoleErrors": 6, "networkFailures": 6 - }, - "steps": [ - { "name": "login-admin", "ok": true, "exitCode": 0, "durationMs": 3681, - "startedAt": "2026-06-26T02:43:22.479Z", "script": "..." } - ], - "artifactList": [ - { "kind": "trace", "path": "trace.zip", "bytes": 5528027 }, - { "kind": "video", "path": "video/xxx.webm", "bytes": 3955579 }, - { "kind": "har", "path": "network.har", "bytes": 29806509 }, - { "kind": "console", "path": "console.log", "bytes": 3259 }, - { "kind": "screenshot", "path": "screenshots/step.png", "label": "...", - "step": "step-name", "bytes": 405656 } - ] -} -``` - -**Canary failure-signal gotchas (discovered in live sessions — internalise these):** - -- `summary.consoleErrors > 0` is **NOT** a fail signal. QuickJS `console.log` may route through - the error channel. Use `summary.stepsFailed` instead. -- `summary.networkFailures > 0` is **NOT** a fail signal. WP admin always emits some 4xx/5xx - (favicon 404 etc.). Use `summary.stepsFailed`. -- Video requires `--capture video` at session start. -- The session **must** be explicitly ended (`session end`) or `results.json`/`report.html` are - never written. A crash mid-session leaves artifacts incomplete. -- WP REST nonces expire (~12h TTL). Re-fetch the nonce at the start of each REST step, not just - once at login. - -### Canary replay - -Step scripts recorded in `session.json` replay without Claude — first run is Claude-driven; subsequent CI replays are zero-inference (see §8). - ---- - -## 1. Architecture overview — how the three features relate - -``` - ┌─────────────────────────────────────────────┐ - │ Shared infrastructure │ - │ • Canary CLI (npx, unmodified) │ - │ • WP fixture snippets (login/nonce/REST/...) │ - │ • Agent inheritance: WP base → Imagify │ - │ ~/.claude/agents/canary-wp-session-agent │ - │ .claude/agents/canary-imagify-session-… │ - │ • TestRail REST client knowledge (or MCP) │ - └───────────────┬──────────────┬───────────────┘ - │ │ - ┌────────────────────────────────┘ └───────────────────────────┐ - │ │ -┌───────▼─────────────────┐ ┌──────────────────────────────┐ ┌──────────────────▼─────────────┐ -│ FEATURE 1 │ │ FEATURE 2 │ │ FEATURE 3 │ -│ TestRail scenario gen │ │ TestRail run execution │ │ Canary E2E in pipeline │ -│ │ │ │ │ │ -│ PR(s) → draft scenarios │ │ milestone/run → fetch cases │ │ orchestrator e2e_mode flag │ -│ → stage → review → │ │ → Canary executes each │ │ → qa-engineer routes to │ -│ publish to TestRail │ │ → outcomes + artifacts │ │ canary-e2e | e2e-qa-tester │ -│ │ │ → confirm → post to TestRail │ │ → records sessions per flow │ -│ writes the test CASES │──▶│ EXECUTES those cases via │ │ → PR comment + Tests/e2e/specs │ -│ that Feature 2 runs │ │ Canary (shares §6 fixtures) │ │ (shares §6 fixtures + agents) │ -└──────────────────────────┘ └──────────────┬───────────────┘ └─────────────────────────────────┘ - │ - recorded step scripts - │ - ▼ - §8 Replay → free CI reruns -``` - ---- - -## 2. Agent & skill inventory - -### New skills (standalone entry points) - -| Skill | Feature | Trigger | Inputs | Outputs | -|-------|---------|---------|--------|---------| -| `/testrail-scenarios` | 1 | user invokes with PR number(s)/URL(s) or `--since-tag` | PR refs, optional `--force-pr=<n>`, optional section override | staging files under `.ai/testrail/pending/`, summary, then created case IDs on publish | -| `/testrail-run` | 2 | user invokes with milestone name/ID or `--run-id` | milestone/run identifier, optional `--cases=<ids>` filter | results table (case → outcome + Canary artifacts), then TestRail results posted on confirm | - -### New agents - -| Agent | Scope | Feature | What it does | Inputs | Outputs | -|-------|-------|---------|--------------|--------|---------| -| `testrail-scenario-agent` | project | 1 | Analyses PR(s), drafts scenarios, writes staging files, and on confirm creates TestRail sections/cases | PR diff + description + linked issue spec; TestRail section map | staging YAML; created case IDs (JSON) | -| `testrail-run-agent` | project | 2 | Fetches run cases, orchestrates Canary execution per case, aggregates results, posts to TestRail on confirm | run/milestone ID; TestRail cases | results table; posted result IDs (JSON) | -| `canary-e2e` | project | 3 | Drop-in for `e2e-qa-tester`: records a Canary session per P0/P1 flow, posts results table to PR, writes Playwright specs | `qa-plan.md` path, `e2e_mode: "canary"`, PR number | same JSON contract as `e2e-qa-tester` | -| `canary-imagify-session-agent` | project | 2 & 3 | Drives one Canary session against Imagify using the WP base + Imagify specifics | session name, flow steps/assertions, env (URL/user/pass) | `~/.canary/sessions/<id>/` + parsed `results.json` summary (JSON) | -| `canary-wp-session-agent` | **user** | 2 & 3 | WP-generic Canary base: login, nonce, REST, AJAX, notices. Shared across all WP plugin repos | (read as base by the Imagify agent) | — (instruction base, not invoked directly) | -| `canary-wp-verify-agent` | **user** | 3 (optional) | WP-generic diff → QA-plan helper (fork of marketplace `verify-agent`) | git diff | QA-plan draft | - -### Modified agents/skills - -| File | Feature | Change | -|------|---------|--------| -| `.claude/agents/qa-engineer.md` | 3 | Becomes `e2e_mode`-aware: writes `.ai/qa-plan.md`; routes to `canary-e2e` or `e2e-qa-tester`; merges either's (identical-shape) results into the PR comment | -| `.claude/skills/orchestrator/SKILL.md` | 3 | Adds the `e2e_mode` prompt to startup calibration; passes `e2e_mode` in every downstream dispatch | - -### Explicitly untouched - -| File | Why | -|------|-----| -| `.claude/agents/e2e-qa-tester.md` | The stable fallback. Feature 3 must not modify it; a one-word startup choice rolls back to it. | -| `@usecanary/cli`, `@usecanary/ui` | Engine. Used via `npx`, never forked. | - ---- - -## 3. Feature 1 — TestRail scenario generation - -### 3.1 Flow - -``` -/testrail-scenarios <pr | pr-list | --since-tag> - -1. Resolve PR scope: - • explicit PR numbers/URLs, OR - • --since-tag → git log v{last_tag}..HEAD → merged PR list -2. Dedupe per PR (see 3.5): GET /get_cases/3?refs=<pr_url> - • cases found → skip, print "⏭ Skipped PR #<n> — <k> cases already exist" - • --force-pr=<n> regenerates for that PR (does not delete old cases) -3. For each in-scope PR: testrail-scenario-agent reads PR description + diff summary + - linked issue spec → drafts scenarios (see 3.4) -4. Write one staging file per PR → .ai/testrail/pending/<pr-slug>.yml -5. Print a human-readable summary of every drafted scenario -6. STOP. Wait for the user to review/edit the staging files and say "publish". -7. On "publish": - a. For each staging file: create new_section if specified (POST /add_section/3) - b. POST /add_case/<section_id> per scenario, refs=<pr_url> - c. Print created case IDs - d. Clean .ai/testrail/pending/ -``` - -The review gate is **mandatory and explicit**: the agent stages, summarises, then halts. It never -creates cases in the same turn it drafts them. The user edits YAML freely (titles, steps, section, -smoke flag) before publishing. - -### 3.2 Staging file format (`.ai/testrail/pending/<pr-slug>.yml`) - -```yaml -pr: 1133 -pr_url: https://github.com/wp-media/imagify-plugin/pull/1133 -pr_title: "Add MCP + Abilities to Imagify" -section_id: 7685 # existing target section (API Requests) -new_section: "MCP Abilities" # subsection to CREATE under section_id (null if not needed) - -cases: - - title: "MCP - Discover abilities - happy path" - smoke_test: true - preconditions: | - - Imagify plugin active - - Valid API key configured - steps: - - action: "Navigate to Settings > Imagify. Ensure API key is saved." - expected: "Settings page loads. API key field shows a valid key." - - action: "Connect an MCP-compatible client to the Imagify MCP server." - expected: "Connection succeeds. No error." - - action: "Call the discover-abilities tool." - expected: "Returns a list of available abilities (optimize-media, bulk-optimize, …)." - - - title: "MCP - Discover abilities - invalid API key" - smoke_test: false - preconditions: | - - Imagify plugin active - - Invalid or missing API key - steps: - - action: "Connect MCP client. Call discover-abilities with no valid API key set." - expected: "Returns an error response indicating authentication failure." -``` - -Field mapping (staging YAML → TestRail `add_case` payload): - -| Staging YAML | TestRail field | -|--------------|----------------| -| `title` | `title` | -| `smoke_test` | `custom_smoketest` (bool) | -| `preconditions` | `custom_preconds` (HTML — agent wraps lines in `<p>`/`<br>`) | -| `steps[].action` | `custom_steps_separated[].content` | -| `steps[].expected` | `custom_steps_separated[].expected` | -| (constant) | `template_id: 2`, `type_id: 7`, `priority_id: 2` | -| `pr_url` | `refs` | - -### 3.3 What scenarios the agent drafts (per PR) - -The agent reads PR description, diff summary, and linked issue spec, then drafts cases covering: - -- **Happy path** — standard user, default settings, expected flow. -- **Happy-path variants** — relevant settings combinations (WebP on/off, plan tiers). -- **Missing prerequisites** — no API key, quota exceeded, wrong plan/license. -- **Network/API failures** — timeout, 5xx, malformed response. -- **Edge cases** — empty state, max limits, unexpected data. -- **Permission levels** — admin vs editor vs subscriber when relevant. -- **Plugin conflicts** — relevant 3rd-party plugins when the change touches integration points. -- **Regression guard** — at least one case confirming prior behaviour still holds (for fixes). - -Bias toward **more, more-automatable** cases (each becomes future Canary/Playwright coverage). No -artificial cap per PR. Each case is also evaluated for `smoke_test: true` (critical path that must -always pass). Pure tracking/analytics PRs with no user-visible behaviour: the agent may draft a -single minimal "event fires" case or skip — see §9. - -### 3.4 Deduplication - -`refs = <github PR url>` on create. Dedup: `GET /get_cases/3?refs=<pr_url>` — cases found → skip; none → draft. `--force-pr=<n>` regenerates (does not delete old cases). - -### 3.5 Section mapping - -The agent picks the closest existing section; if none fits, it proposes a new subsection under -`Regression (33)` via the `new_section` field (created only on publish, never silently). - -| Feature area | Section | -|---|---| -| MCP / Abilities | Regression > API Requests (7685) → new: MCP Abilities | -| Settings | Regression > Settings (32) → appropriate subsection | -| General settings | Regression > Settings > general settings (7689) | -| Optimization | Regression > Settings > optimization (7690) | -| Next-gen (WebP/AVIF) | Regression > Settings > optimization > next-gen (7694) → Webp (7697), Avif (7698) | -| File optimization | Regression > Settings > optimization > file optimization (7695) | -| Bulk optimization | Regression > Bulk Optimization (3766) | -| Media library | Regression > Media library (4976) | -| 3rd-party (WooCommerce) | Regression > 3rd party compatibility > Woocommerce (7686) | -| 3rd-party (WP Rocket) | Regression > 3rd party compatibility > WP Rocket (7687) | -| Action Scheduler | Regression > Action scheduler (4975) | -| Promotions | Regression > Promotions (1082) | -| Smoke tests | flag `custom_smoketest: true` (NOT a separate section) + Smoke test section (4525) for pure smoke cases | -| New feature, no match | propose new subsection under Regression (33) | - -### 3.6 MCP vs API - -TestRail MCP (`/opt/homebrew/bin/mcp-testrail`) is connected but tool names unconfirmed. **Use the REST API via `curl` as primary** (see §5). MCP is an optional enhancement once tool names are confirmed. - ---- - -## 4. Feature 2 — TestRail run execution - -### 4.1 Flow - -``` -/testrail-run <milestone-name | milestone-id | --run-id=<id>> - -1. Resolve the run: - • --run-id given → use it directly - • milestone name/id → GET /get_milestones/3 → GET /get_runs/3?milestone_id=<id> - → if multiple runs, prompt user to pick (or --run-id) [open question §9] -2. GET /get_tests/<run_id> → all cases in the run (title, custom_preconds, - custom_steps_separated, case_id) -3. For each case (sequential or bounded-parallel): - a. testrail-run-agent maps the case → a Canary session plan: - preconditions → setup steps; each custom_steps_separated entry → a Canary step - b. spawns canary-imagify-session-agent to record the session - (one Canary session per TestRail case — see 4.3) - c. reads results.json → derives outcome (see 4.4) -4. Aggregate → results table: case → pass/fail/blocked + session id + artifact paths -5. Print the table. STOP. Ask: "Update TestRail with these results? (y/n)" -6. On confirm, per case: - POST /add_result_for_case/<run_id>/<case_id> with status + evidence comment (see 4.5) -7. Print posted result summary. -``` - -The TestRail write is gated behind explicit user confirmation. The agent never posts results in -the same turn it executes — the user sees outcomes first. - -### 4.2 How Canary sessions map to TestRail cases - -- **One Canary session per TestRail case.** Session name = `TR-<case_id>: <case title>` so the - artifact directory and report are traceable back to the case. -- **Preconditions → setup steps.** `custom_preconds` is parsed into setup steps (login is always - step 1 via the wp-login fixture; "valid API key configured" → a settings precheck step, etc.). -- **Each `custom_steps_separated` entry → one Canary step.** The step's `expected` becomes the - assertion the step's `console.log("PASS/FAIL …")` checks. -- The `canary-imagify-session-agent` translates the human-readable TestRail step into concrete - Canary script using the §6 fixtures. Where a step is too ambiguous to automate, the case is - marked **blocked** (not failed) and flagged for manual review. - -### 4.3 Outcome derivation (from `results.json`) - -| Condition | TestRail status | -|-----------|-----------------| -| `summary.stepsFailed == 0` and all assertion `console.log` lines are PASS | **Passed (1)** | -| any step `ok == false` / `exitCode != 0`, or an assertion logs FAIL | **Failed (5)** | -| env unreachable, session couldn't start, or a step was un-automatable/ambiguous | **Blocked (2)** | - -Reminder: do NOT use `consoleErrors` / `networkFailures` as fail signals (see §0). - -### 4.4 Evidence in the result comment - -The result comment for each case includes per-step outcomes plus pointers to Canary artifacts: - -```json -{ - "status_id": 5, - "comment": "Automated Canary run — 2026-06-26\nSession: TR-4521--mcp-discover-abilities-xxx\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\nStep 3: ❌ Failed — assertion 'returns ability list' did not hold\n\nTrace: npx playwright show-trace .ai/testrail/1283/canary/TR-4521--…/trace.zip\nReport: .ai/testrail/1283/canary/TR-4521--…/report.html\nScreenshot: <raw.githubusercontent.com SHA URL, if published>", - "elapsed": "45s" -} -``` - -Evidence to include: -- Per-step pass/fail summary derived from `results.json.steps`. -- `elapsed` from summed `durationMs`. -- Trace replay command (local path — reviewer runs `npx playwright show-trace`). -- Path to `report.html`. -- Optionally, published screenshot URLs (same temp-branch-commit → SHA-raw-URL mechanism as - `e2e-qa-tester`; see §6.5) when a hosted image is wanted in TestRail. - -### 4.5 Failure handling - -- Step fails → case marked Failed, remaining steps captured but the failure is the verdict. -- Env unreachable / session won't start → case Blocked with reason. -- Step un-interpretable → case Blocked + "manual review" flag (never silently Failed). - ---- - -## 5. TestRail API reference (shared by Features 1 & 2) - -**Base URL:** `https://wpmediaqa.testrail.io/index.php?/api/v2/` -**Auth:** Basic (`TESTRAIL_USERNAME:TESTRAIL_API_KEY`) -**Project ID:** 3 · **Suite ID:** 3 · **Step template ID:** 2 - -| Action | Endpoint | -|--------|----------| -| Find cases by PR URL (dedup) | `GET /get_cases/3?refs=<pr_url>` | -| List sections | `GET /get_sections/3&suite_id=3` | -| Create section | `POST /add_section/3` | -| Create case | `POST /add_case/<section_id>` | -| List milestones | `GET /get_milestones/3` | -| List runs for a milestone | `GET /get_runs/3?milestone_id=<id>` | -| Get tests in a run | `GET /get_tests/<run_id>` | -| Post a result | `POST /add_result_for_case/<run_id>/<case_id>` | - -**Case payload (Step template):** - -```json -{ - "title": "MCP - Discover abilities - happy path", - "template_id": 2, - "type_id": 7, - "priority_id": 2, - "refs": "https://github.com/wp-media/imagify-plugin/pull/1133", - "custom_smoketest": false, - "custom_preconds": "<p>- Imagify active<br>- Valid API key</p>", - "custom_steps_separated": [ - { "content": "Navigate to Settings > Imagify.", "expected": "Settings page loads." } - ] -} -``` - -**Result payload:** - -```json -{ - "status_id": 1, - "comment": "Automated run …", - "elapsed": "45s" -} -``` - -Status IDs: **1 = Passed · 5 = Failed · 2 = Blocked**. - -**Known section IDs:** - -``` -Regression (33) - ├── Settings (32) - │ ├── general settings (7689) - │ └── optimization (7690) - │ ├── next-gen (7694) → Webp (7697), Avif (7698) - │ └── file optimization (7695) - ├── Smoke test (4525) - ├── API Requests (7685) ← MCP / Abilities go here - ├── Media library (4976) - ├── Bulk Optimization (3766) - ├── 3rd party compatibility (4980) - │ ├── Woocommerce (7686) - │ └── WP Rocket (7687) - ├── Action scheduler (4975) - └── Promotions (1082) -``` - ---- - -## 6. Feature 3 — Canary E2E in the pipeline - -`canary-e2e` is a drop-in replacement for `e2e-qa-tester` selected via `e2e_mode: "canary"` at orchestrator startup. Same JSON return contract; see `.claude/agents/canary-e2e.md` for full instructions. - -### 6.1 `qa-plan.md` format (shared by qa-engineer ↔ canary-e2e ↔ e2e-qa-tester) - -Produced by `qa-engineer`, written to `.ai/qa-plan.md`, consumed by whichever E2E agent runs. -The `P0-A`/`P0-B` labels become Canary session names. - -```markdown -## QA Plan — PR #1234 - -### P0 — Must pass before merge - -#### P0-A: <flow name> -- **Entry:** <URL> -- **Steps:** - 1. <step> - 2. <step> -- **Assertions:** - - <specific enough to write a Canary assert> -- **Risk:** <which file(s) break this if wrong> -- **Canary session name:** `P0-A: <flow name>` - -### P1 — Should pass -#### P1-A: <flow name> -(same structure) - -### P2 — Nice to have / regression guard -#### P2-A: <flow name> -- documented; Canary session optional -``` - -### 6.2 Artifact flow - -``` -.ai/{N}/canary/<id>/ ← sessions relocated here from ~/.canary/sessions/<id>/ - -GitHub PR comment: qa-plan table + Canary results table + screenshot SHA URLs - + "Trace: npx playwright show-trace .ai/{N}/canary/<id>/trace.zip" -Tests/e2e/specs/ ← Playwright specs committed by canary-e2e (CI re-runs them) -``` - -### 6.3 Playwright spec translation - -`canary-e2e` writes fresh TypeScript specs — not a transpile of QuickJS steps: - -| Canary QuickJS | Playwright TypeScript | -|---|---| -| `browser.getPage("main")` | `page` fixture | -| `page.evaluate(async () => fetch(...))` | identical | -| `console.log("PASS: …")` | `expect(...).toBe(...)` | -| no `import` | `import { test, expect } from '@playwright/test'` | -| inlined helpers | POM methods in `Tests/e2e/pages/` | - ---- - -## 7. Shared infrastructure - -### 7.1 Agent inheritance pattern (WP base → Imagify) - -WP plugins share identical WP patterns but differ in plugin specifics. Split accordingly: - -``` -~/.claude/agents/ ← USER-level, shared across ALL WP plugin repos - canary-wp-session-agent.md # login, nonce, REST, AJAX, notices - canary-wp-verify-agent.md # WP QA-plan format + verify flow (optional) - -.claude/agents/ ← PROJECT-level, Imagify only - canary-imagify-session-agent.md # extends WP base + Imagify URLs/slugs/MCP/fixtures -``` - -WP-generic patterns (login, nonce, REST, AJAX, notices) live in the base agent only. The plugin agent reads it as its first instruction and extends it with Imagify-specific URLs, selectors, and ability slugs. - -### 7.2 WP fixture snippets (canonical — never improvised, always copied verbatim) - -Carried in `canary-wp-session-agent.md`. Because QuickJS has no `require()`, these are inlined per -step. - -```js -// FIXTURE: wp-login — always step 1; browser persists for all subsequent steps -const page = await browser.getPage("main"); -await page.goto("{E2E_URL}/wp-login.php", { waitUntil: "domcontentloaded" }); -await page.locator("#user_login").fill("{WP_USER}"); -await page.locator("#user_pass").fill("{WP_PASS}"); -await page.locator("#wp-submit").click(); -await page.waitForURL("**/wp-admin/**", { timeout: 10000 }); -console.log("Login:", page.url().includes("wp-admin") ? "PASS" : "FAIL"); -``` - -```js -// FIXTURE: wp-nonce — GET with credentials:same-origin; result needs .trim() -const nonce = await page.evaluate(async () => { - const r = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); - return (await r.text()).trim(); -}); -``` - -```js -// FIXTURE: wp-rest-get — authenticated GET to WP REST API -const result = await page.evaluate(async ([url, nonce]) => { - const r = await fetch(url, { headers: { "X-WP-Nonce": nonce }, credentials: "same-origin" }); - return { status: r.status, body: await r.text() }; -}, [url, nonce]); -``` - -```js -// FIXTURE: wp-rest-post — authenticated POST to WP REST API -const result = await page.evaluate(async ([url, nonce, body]) => { - const r = await fetch(url, { - method: "POST", - headers: { "Content-Type": "application/json", "X-WP-Nonce": nonce }, - body: JSON.stringify(body), - credentials: "same-origin" - }); - return { status: r.status, body: await r.text() }; -}, [url, nonce, body]); -``` - -```js -// FIXTURE: wp-ajax — POST to admin-ajax.php with _ajax_nonce -const result = await page.evaluate(async ([action, nonce, data]) => { - const form = new FormData(); - form.append("action", action); - form.append("_ajax_nonce", nonce); - Object.entries(data).forEach(([k, v]) => form.append(k, String(v))); - const r = await fetch("/wp-admin/admin-ajax.php", { method: "POST", body: form, credentials: "same-origin" }); - return { status: r.status, body: await r.text() }; -}, [action, nonce, data]); -``` - -```js -// FIXTURE: wp-assert-notice — assert success/error admin notice -const notice = await page.waitForSelector(".notice-success, .notice-error", { timeout: 5000 }); -const cls = await notice.getAttribute("class"); -console.log(cls.includes("notice-success") ? "PASS: success notice" : "FAIL: error notice"); -``` - -**Imagify-specific (belongs in `canary-imagify-session-agent.md`, NOT the WP base):** - -```js -// FIXTURE: imagify-abilities — assert all expected slugs in WP Abilities response -const result = await page.evaluate(async () => { - const nonceResp = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); - const nonce = (await nonceResp.text()).trim(); - const resp = await fetch("/wp-json/wp-abilities/v1/abilities", { - headers: { "X-WP-Nonce": nonce }, credentials: "same-origin" - }); - return { status: resp.status, body: await resp.text() }; -}); -const data = JSON.parse(result.body); -const slugs = Array.isArray(data) ? data.map(a => a.name) : []; -console.log("PHP Fatal:", result.body.includes("Fatal error") ? "FAIL" : "PASS"); -const expected = [ - "imagify/get-settings", "imagify/update-settings", "imagify/get-account", - "imagify/get-stats", "imagify/get-media-status", "imagify/get-nextgen-coverage", - "imagify/optimize-media", "imagify/bulk-optimize", - "imagify/generate-missing-nextgen", "imagify/restore-media" -]; -for (const slug of expected) console.log(slugs.includes(slug) ? `PASS: ${slug}` : `FAIL: ${slug} NOT FOUND`); -``` - -### 7.3 Session-agent trust guard - -**Not present** in the Canary marketplace agents (confirmed in audit). No guard to remove. Both `canary-wp-session-agent` and `canary-imagify-session-agent` can be invoked transitively from the pipeline without deadlocking on a confirmation prompt. - ---- - -## 8. Canary replay (Phase B) - -Step scripts in `session.json` replay without Claude — first run is Claude-driven; subsequent replays are zero-inference. - -- **Feature 2 (Phase B):** persist scripts to `Tests/e2e/canary/TR-<case_id>/`; CI replays unchanged cases. Only new/changed cases need Claude. -- **Feature 3:** `Tests/e2e/specs/` already provides the CI rerun path. Canary scripts can also be replayed locally to reproduce failures deterministically. - -Phase A (current): Claude executes every run. Phase B: CI replays recorded scripts. - ---- - ---- - -## 9. Audit resolution (2026-06-26) - -All questions resolved during deep audit. No open questions remain for initial implementation. - -| # | Question | Resolution | -|---|----------|------------| -| 1 | TestRail MCP availability | ✅ **Connected** — `/opt/homebrew/bin/mcp-testrail` in `~/.claude/settings.json`. Tool names unconfirmed; use REST API (curl) as primary. MCP optional enhancement. | -| 2 | Skill naming | ✅ `/testrail-scenarios` + `/testrail-run` (user confirmed) | -| 3 | `refs` API filter | ✅ **Works** — `GET /get_cases/3?suite_id=3&refs=<url>` returns `{cases:[], size:0}` when no match. Primary dedup strategy confirmed. | -| 4 | Section auto-creation | ✅ Propose in YAML, create only on "publish" confirm. Already designed this way. | -| 5 | Tracking-only PRs | ✅ Generate a minimal "event fires" case verifying the event reaches the analytics endpoint. Don't skip. | -| 6 | Multiple runs per milestone | ✅ **Real data:** one run per milestone in practice (milestone 2.3.0 → single run 1283 with 151 untested cases). Design: if multiple exist, prompt user; default to the single one. | -| 7 | Parallelism | ✅ **Sequential** — matches existing `playwright.config.ts` (`workers: 1`, `fullyParallel: false`). WP install shared; sequential keeps DB state predictable. | -| 8 | Recorded script persistence | ✅ `Tests/e2e/canary/TR-<case_id>/` — one dir per TestRail case ID containing step scripts. Phase B: CI runs `npx @usecanary/cli run` on these instead of re-inferring. | -| 9 | CANARY_HOME in CI | ⚠️ **Deferred to Phase B** — not needed for Phase A (local-only Canary). Investigate `CANARY_HOME` override when CI execution is in scope. | -| 10 | Spec generation strategy | ✅ **Fresh write** — `canary-e2e` writes TypeScript Playwright specs from scratch (same assertions as Canary steps), not a mechanical transpile. QuickJS ≠ Node. | -| 11 | CI vs local Canary | ✅ **Local-only Canary** initially. CI runs committed Playwright specs from `Tests/e2e/specs/`. No Canary dep in CI for Phase A. | -| 12 | WP fixture coverage | ✅ **Audited.** 6 WP snippets cover all patterns in `Tests/e2e/specs/`. Additional Imagify-specific fixtures needed in `canary-imagify-session-agent`: `#imagify-check-api-container:not(.imagify-valid)` for invalid API key (NOT `.notice-error`), `#imagify-bulk-action`, `.imagify-bulk-table`. `wp-multisite` needed for some TestRail cases — deferred. Login uses `getByLabel('Username or Email Address')` not `#user_login`. | -| 13 | Return-contract parity | ✅ `canary-e2e` returns identical JSON to `e2e-qa-tester` plus `canary_sessions[]` and `canary_results_table` fields. Run `/contract-check` after implementation to verify. | - -**Additional audit findings:** - -- **CLI syntax correction:** `id=$(npx @usecanary/cli session start --name "X")` — session ID returned from stdout, passed as `--session "$id"` to subsequent `run` commands -- **Screenshot location:** daemon auto-captures one screenshot per step (last-opened tab) — this goes in the report. `saveScreenshot()` → `~/.canary/tmp/` (debug only, NOT in report) -- **Trust guard:** NOT present in `session-agent.md` — no guard to remove. No confirmation requirement in pipeline use. -- **TestRail steps are HTML:** `<p>Visit settings.</p>` — must strip with `python3 -c "import sys,re; print(re.sub('<[^>]+>','',sys.stdin.read()).strip())"` -- **Active run 1283 (2.3.0):** 151 untested cases, status_id=3 (untested) - diff --git a/.claude/canary-wp-spec.md b/.claude/canary-wp-spec.md deleted file mode 100644 index 4bded9b8..00000000 --- a/.claude/canary-wp-spec.md +++ /dev/null @@ -1,374 +0,0 @@ -# Spec: Canary × WordPress — Integrated QA in the Issue Workflow - -**Status:** Draft -**Author:** Gaël Robin -**Next step:** Deep audit session (week of 2026-06-30) - ---- - -## 0. What is Canary - -> This section exists so any future Claude session has full Canary context without searching. Read it before reading the rest of the spec. - -### Engine - -Canary is a **browser recording + QA platform** built on two npm packages: - -| Package | Command | What it does | -|---------|---------|--------------| -| `@usecanary/cli` | `npx @usecanary/cli` | Playwright daemon + QuickJS sandbox + session lifecycle | -| `@usecanary/ui` | `npx @usecanary/ui` | Local web viewer for browsing session recordings | - -**We do NOT fork or modify these packages.** They stay as `npx @usecanary/cli` from npm. What we customise is the **skill pack** — the markdown agent/skill files that drive the CLI. - -### QuickJS sandbox (the step script runtime) - -Each "step" is a `.js` script executed inside QuickJS — a tiny JS engine, NOT Node.js. Critical constraints: - -- **No `require()` / `import`** — ESM and CJS modules both fail -- **No Node.js APIs** — no `fs`, `path`, `http`, `process.env`, etc. -- **No top-level `fetch()`** — QuickJS has no fetch; use `page.evaluate(async () => fetch(...))` to run fetch inside the browser context instead -- **No helper files** — every utility must be inlined verbatim in each step script -- **Globals available:** `browser` (Canary's Playwright wrapper), `saveScreenshot`, `console` - -The `browser` object: -```js -const page = await browser.getPage("main"); // "main" is the default page name -// page is a Playwright Page — all page.goto, page.locator, page.evaluate, etc. work -``` - -One browser **persists across all steps in a session** — you do not log in again between steps. The browser is identified by `session.json["browser"]`. - -### Session lifecycle - -``` -npx @usecanary/cli session start --name "Flow name" --capture trace,video,har,console - → creates ~/.canary/sessions/<slug-id>/session.json - → starts Playwright daemon + browser - -npx @usecanary/cli run <step.js> # repeat for each step - → executes step in QuickJS - → appends step to session.json["steps"] - → saves screenshot if saveScreenshot() called - -npx @usecanary/cli session end - → writes results.json - → renders report.html - → saves trace.zip, video.webm, network.har, console.log -``` - -Session ID format: `<slug>-<random8>-<short-hash>` — e.g. `p0-a--mcp-abilities-discovery-mqubrrhk-3f7e94` - -### Artifact structure - -Canary writes to `~/.canary/sessions/<id>/` initially. Our agents relocate each session to `.ai/{issue_N}/canary/<id>/` after `session end` and leave a symlink at the original path (so `npx @usecanary/ui` still works). The directory structure is the same either way: - -``` -session.json ← session metadata + step scripts (written during run) -results.json ← step results + summary (written at session end) -report.html ← self-contained pass/fail report (written at session end) -trace.zip ← Playwright trace (open with npx playwright show-trace trace.zip) -video/ ← .webm recording -network.har ← all HTTP traffic -console.log ← browser console output -screenshots/ ← PNG per saveScreenshot() call (<step>-<random>.png) -manifest.json ← artifact inventory -profile/ ← browser profile -``` - -### `results.json` schema (real example) - -```json -{ - "summary": { - "stepsPassed": 10, - "stepsFailed": 0, - "stepsTotal": 10, - "commandCount": 34, - "consoleErrors": 6, - "networkFailures": 6 - }, - "steps": [ - { - "name": "login-admin", - "ok": true, - "exitCode": 0, - "durationMs": 3681, - "startedAt": "2026-06-26T02:43:22.479Z", - "script": "..." - } - ], - "artifactList": [ - { "kind": "trace", "path": "trace.zip", "bytes": 5528027 }, - { "kind": "video", "path": "video/xxx.webm", "bytes": 3955579 }, - { "kind": "har", "path": "network.har", "bytes": 29806509 }, - { "kind": "console", "path": "console.log", "bytes": 3259 }, - { "kind": "screenshot", "path": "screenshots/step-name-abc.png", - "label": "Screenshot: step-name", "step": "step-name", "bytes": 405656 } - ] -} -``` - -### Key CLI commands - -```bash -# Session lifecycle -npx @usecanary/cli session start --name "Flow name" --capture trace,video,har,console -npx @usecanary/cli run path/to/step.js -npx @usecanary/cli session end - -# Inspect -npx @usecanary/cli session list # list sessions -npx @usecanary/cli status # daemon status -npx @usecanary/cli status --session <id> # specific session status - -# Open the report viewer -npx @usecanary/ui # opens http://localhost:3030 (or nearby port) -npx @usecanary/ui --dir ~/.canary/sessions/<id> # open specific session -``` - -### Skill pack (what we DO own) - -The skill pack is markdown files at `~/.claude/plugins/marketplaces/canary-marketplace/`. We copy relevant agents into `.claude/agents/` and customise them for WordPress. No fork of the CLI needed. - -Canary agent files of interest: -- `session-agent.md` → our fork: `canary-wp-session-agent.md` -- `verify-agent.md` → our fork: `canary-wp-verify-agent.md` - ---- - -## 2. Architecture - -### 2.1 E2E mode routing - -`e2e_mode` is set at orchestrator startup and passed to `qa-engineer`: - -``` -qa-engineer [receives e2e_mode] - diff has UI/browser changes? - yes + e2e_mode == "canary" → canary-e2e - yes + e2e_mode == "playwright" → e2e-qa-tester (default, untouched) - no → skip E2E -``` - -Both agents return the same JSON contract — `qa-engineer` is agnostic to which ran. - -### 2.2 Agent inheritance - -``` -~/.claude/agents/canary-wp-session-agent.md ← user-level: login, nonce, REST, AJAX, notices -.claude/agents/canary-imagify-session-agent.md ← project: Imagify URLs, selectors, abilities -``` - -The plugin agent reads the WP base as its first instruction (EXTEND, not replace). - -### 2.3 Artifact locations - -Sessions are relocated from `~/.canary/sessions/<id>/` to `.ai/{N}/canary/<id>/` after `session end`; a symlink at the original path keeps `npx @usecanary/ui` working. - -``` -.ai/{N}/canary/<id>/ - results.json ← step results + verdict - report.html ← self-contained HTML report - trace.zip ← npx playwright show-trace trace.zip - video/ ← .webm - network.har ← HTTP traffic - console.log ← browser console - screenshots/ ← per-step PNGs (daemon auto-captures one per step) -``` - -GitHub PR comment: Canary results table + screenshot SHA URLs + trace replay command per session. -`Tests/e2e/specs/` — Playwright specs committed by `canary-e2e` (existing CI re-runs them). - ---- - -## 3. WordPress fixture snippets - -Because Canary's QuickJS sandbox has no `require()`, helpers must be inlined per step script. The `canary-wp-session-agent` carries these as **canonical copy-paste blocks** in its instructions — never re-invented, always copied verbatim. - -### wp-login -```js -// FIXTURE: wp-login — always step 1, browser persists for all subsequent steps -const page = await browser.getPage("main"); -await page.goto("{E2E_URL}/wp-login.php", { waitUntil: "domcontentloaded" }); -await page.locator("#user_login").fill("{WP_USER}"); -await page.locator("#user_pass").fill("{WP_PASS}"); -await page.locator("#wp-submit").click(); -await page.waitForURL("**/wp-admin/**", { timeout: 10000 }); -console.log("Login URL:", page.url()); -console.log("Login:", page.url().includes("wp-admin") ? "PASS" : "FAIL"); -``` - -### wp-nonce (REST) — the correct pattern from live sessions -```js -// FIXTURE: wp-nonce — use GET with credentials:same-origin; result needs .trim() -const nonce = await page.evaluate(async () => { - const r = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { - credentials: "same-origin" - }); - return (await r.text()).trim(); -}); -console.log("Nonce (first 10):", nonce.slice(0, 10)); -``` - -### wp-rest-get -```js -// FIXTURE: wp-rest-get — authenticated GET to WP REST API -const result = await page.evaluate(async ([url, nonce]) => { - const r = await fetch(url, { - headers: { "X-WP-Nonce": nonce }, - credentials: "same-origin" - }); - return { status: r.status, body: await r.text() }; -}, [url, nonce]); -console.log("Status:", result.status); -``` - -### wp-rest-post -```js -// FIXTURE: wp-rest-post — authenticated POST to WP REST API -const result = await page.evaluate(async ([url, nonce, body]) => { - const r = await fetch(url, { - method: "POST", - headers: { "Content-Type": "application/json", "X-WP-Nonce": nonce }, - body: JSON.stringify(body), - credentials: "same-origin" - }); - return { status: r.status, body: await r.text() }; -}, [url, nonce, body]); -``` - -### wp-ajax -```js -// FIXTURE: wp-ajax — POST to admin-ajax.php with _ajax_nonce -const result = await page.evaluate(async ([action, nonce, data]) => { - const form = new FormData(); - form.append("action", action); - form.append("_ajax_nonce", nonce); - Object.entries(data).forEach(([k, v]) => form.append(k, String(v))); - const r = await fetch("/wp-admin/admin-ajax.php", { - method: "POST", - body: form, - credentials: "same-origin" - }); - return { status: r.status, body: await r.text() }; -}, [action, nonce, data]); -``` - -### wp-assert-notice -```js -// FIXTURE: wp-assert-notice — assert success/error admin notice -const notice = await page.waitForSelector(".notice-success, .notice-error", { timeout: 5000 }); -const cls = await notice.getAttribute("class"); -console.log(cls.includes("notice-success") ? "PASS: success notice" : "FAIL: error notice"); -``` - -### imagify-abilities (plugin-specific — belongs in `canary-imagify-session-agent`, NOT the WP base) -```js -// FIXTURE: imagify-abilities — assert all expected slugs in WP Abilities response -const result = await page.evaluate(async () => { - const nonceResp = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); - const nonce = (await nonceResp.text()).trim(); - const resp = await fetch("/wp-json/wp-abilities/v1/abilities", { - headers: { "X-WP-Nonce": nonce }, - credentials: "same-origin" - }); - return { status: resp.status, body: await resp.text() }; -}); - -const data = JSON.parse(result.body); -const slugs = Array.isArray(data) ? data.map(a => a.name) : []; -console.log("HTTP status:", result.status); -console.log("Total abilities:", slugs.length); -console.log("PHP Fatal:", result.body.includes("Fatal error") ? "FAIL" : "PASS"); - -const expected = [ - "imagify/get-settings", "imagify/update-settings", "imagify/get-account", - "imagify/get-stats", "imagify/get-media-status", "imagify/get-nextgen-coverage", - "imagify/optimize-media", "imagify/bulk-optimize", - "imagify/generate-missing-nextgen", "imagify/restore-media" -]; -for (const slug of expected) { - console.log(slugs.includes(slug) ? `PASS: ${slug}` : `FAIL: ${slug} NOT FOUND`); -} -``` - ---- - -## 4. QA plan format (`.ai/qa-plan.md`) - -Produced by `qa-engineer`, consumed by `e2e-qa-tester`. The `P0-A` / `P0-B` labels become Canary session names. - -```markdown -## QA Plan — PR #1234 - -### P0 — Must pass before merge - -#### P0-A: <flow name> -- **Entry:** <URL> -- **Steps:** - 1. <step> - 2. <step> -- **Assertions:** - - <what must hold — be specific enough to write a Canary assert> -- **Risk:** <which file(s) could break this if wrong> -- **Canary session name:** `P0-A: <flow name>` - -### P1 — Should pass - -#### P1-A: <flow name> -(same structure) - -### P2 — Nice to have / regression guard - -#### P2-A: <flow name> -- P2 flows are documented but Canary sessions are optional -``` - ---- - -## 5. Canary results → GitHub PR comment - -The `e2e-qa-tester` extracts `results.json` from each session and formats it as markdown. This goes into the GitHub PR comment — **not committed to git**. - -### Results table format (produced from `results.json`) - -```markdown -### Canary QA Sessions - -| Flow | Steps | Result | Trace | -|------|-------|--------|-------| -| P0-A: MCP abilities discovery | 10/10 | ✅ PASS | `npx playwright show-trace ~/.canary/sessions/p0-a--mcp-abilities-discovery-xxx/trace.zip` | -| P0-B: Settings page save | 5/6 | ❌ FAIL — step "save-settings" exit 1 | `npx playwright show-trace ~/.canary/sessions/p0-b--settings-page-save-xxx/trace.zip` | - -### Screenshots - -| Step | Screenshot | -|------|-----------| -| login-admin | ![login](https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/p0-a-login-admin.png) | -| assert-abilities | ![abilities](https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/p0-a-assert-abilities.png) | -``` - ---- - -## 6. Playwright spec translation - -`canary-e2e` writes fresh TypeScript specs — never a mechanical transpile of QuickJS steps: - -| Canary QuickJS | Playwright TypeScript | -|---|---| -| `browser.getPage("main")` | `page` fixture from `test()` | -| `page.evaluate(async () => fetch(...))` | identical | -| `console.log("PASS: ...")` | `expect(...).toBe(...)` | -| No `import` | `import { test, expect } from '@playwright/test'` | -| Inlined helpers | POM methods in `Tests/e2e/pages/` | - ---- - -## 7. Known Canary limitations (discovered in live sessions) - -- **`console.log` in QuickJS counts as "consoleErrors"** if the Playwright daemon routes all console output through the error channel. Do not use `summary.consoleErrors > 0` as a FAIL signal — check `summary.stepsFailed` instead. -- **`networkFailures`** in the summary counts network events that returned 4xx/5xx — WP admin always generates some (favicon 404, etc.). Do not use `networkFailures > 0` as a FAIL signal. -- **Video recording requires `--capture video`** at session start. Omitting it means no `.webm` artifact. -- **Session must be explicitly ended** (`npx @usecanary/cli session end`) or `report.html` / `results.json` are not written. If the agent crashes mid-session, artifacts are incomplete. -- **Nonce TTL:** WP REST nonces expire after ~12 hours. Long sessions (left paused) may get 403 on authenticated calls. Re-fetch the nonce at the start of each step that makes REST calls, not just once at login. diff --git a/.claude/testrail/A-plus-design.md b/.claude/testrail/A-plus-design.md deleted file mode 100644 index c3d62b0b..00000000 --- a/.claude/testrail/A-plus-design.md +++ /dev/null @@ -1,417 +0,0 @@ -# TestRail Automation — "A+" Execution Design - -**Status:** Design / pre-build · **Date:** 2026-06-30 · **Author:** Gaël Robin (+ Claude) - -Single source of truth for how we make the TestRail → browser execution step **reliable**. -Consolidates findings from five independent prior-art sources (see Appendix) into one plan. - ---- - -## 1. The problem we are solving - -The fragile link in the current pipeline is one step inside `testrail-run-agent`: - -> **Translate each TestRail step (HTML-stripped prose) into a Canary/Playwright script and run it.** - -This translation happens **at run time, open-loop, with no grounding**: - -- Selectors are *guessed from prose* ("Navigate to Settings > Imagify") — they have never - touched the real DOM. -- Assertions are *inferred from expected text* ("Returns all available abilities"). -- Prerequisites are *assumed* (the agent doesn't know an API key is needed, or where it lives). -- State leaks between sequentially-run cases (`optimize-media` is destructive; settings persist). - -Anyone — not just us — authors the TestRail cases, so **pre-writing a script per case is not an -option**. The fix must work for cases we have never seen. - ---- - -## 2. The core insight (validated by 5 independent sources) - -Every serious prior-art system converged on the **same spine**, regardless of stack: - -> **Explore the running app → ground locators from *real* interactions → validate by -> compiling → gate with humans → cache the grounding for reuse.** - -Two volatilities of knowledge must be handled differently: - -| Knowledge | Volatility | DOM-observable? | Source of truth | -|---|---|---|---| -| Selectors, labels, interaction order | High | **Yes** | Captured from **live exploration** — never frozen from a code read | -| URLs, prerequisites, API-key location, "what success means", seed/teardown | Low | **No** | Light **code/docs audit**, committed | - -The original "Plan A" (Opus audits the *code* and writes specs) was right for the low-volatility -half and **wrong** for selectors — reading code to infer selectors is itself guessing. A stale, -hand-written selector spec is *worse* than none, because the agent trusts it and won't fall back -to looking at the real page. - ---- - -## 3. A+ architecture - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ /testrail-setup (Explorer — runs occasionally, output committed) │ -│ │ -│ • Drives the LIVE app via Canary, logged in │ -│ • Captures REAL locators from real clicks (role-based preferred) │ -│ • Light code/docs audit for non-DOM knowledge: │ -│ - admin URLs & how to reach each feature │ -│ - prerequisites + how to SEED them (WP-CLI / REST) │ -│ - teardown / undo for each mutation (LIFO) │ -│ - verification criteria ("success" = what, observably) │ -│ • Writes a grounded map → .claude/testrail/specs/<feature>.md │ -│ • Stamps each spec with source_files + git SHA (drift detection) │ -└─────────────────────────────────────────────────────────────────────┘ - │ (committed, shared by whole team) - ▼ -┌─────────────────────────────────────────────────────────────────────┐ -│ /testrail-run (Executor — per release QA) │ -│ │ -│ 1. Fetch run + cases from TestRail (steps + EXPECTED RESULTS) │ -│ 2. Resolve case → feature spec (via frontmatter mapping) │ -│ 3. Load _foundation.md + <feature>.md as grounding │ -│ 4. Per case: │ -│ a. SEED prerequisites via API/CLI (not UI) │ -│ b. Execute closed-loop against live DOM, grounded by the map │ -│ c. Assert against the TestRail EXPECTED RESULT (the oracle) │ -│ d. LIFO teardown │ -│ e. Capture trace/video/HAR/console (evidence) │ -│ 5. Results table → human confirm → post to TestRail │ -└─────────────────────────────────────────────────────────────────────┘ -``` - -### Why this beats the alternatives - -- **Selectors are grounded in reality**, captured from real interactions, not prose or code. -- **The grounded map is committed & shared** — everyone runs against the same knowledge. -- **Drift is detectable** (SHA stamps), not silent. -- **TestRail expected-results are our assertion oracle** — neutralises the failure mode every - prior-art author flagged as fatal: *"the agent knows what it observed, not what's correct."* - We don't let the agent decide what "correct" means; TestRail tells it. - ---- - -## 4. The two deltas harvested from prior art - -Beyond the shared spine, two concrete mechanisms are worth lifting: - -### 4.1 Per-case state isolation (from the "scalable E2E" source) - -Our 66 cases run **sequentially in one session**, and several **mutate state** -(`optimize-media`, settings updates, Internal State Reset). Without isolation, case N pollutes -case N+1 — a silent flakiness source. - -**Contract per case:** -- **Seed prerequisites via WP-CLI / REST, not the UI** (set API key, force an attachment into - optimized/unoptimized state, toggle a setting). Faster + deterministic. -- **LIFO teardown queue** — each case pushes its undo; teardown unwinds in reverse, even after - an assertion fails. - -This is also *captured by the Explorer* — "how to seed / tear down" is part of each feature spec. - -### 4.2 Anti-hallucination ruleset (from the "write/compile/execute/ship" source) - -Encode grounding as **enforceable AUTO-REJECT guards** in the agent instructions: - -- **Inferred locator while the page is reachable → REJECT.** Must open the page and capture the - real locator. -- **Ephemeral/internal ref used as a locator → REJECT.** Convert to a stable locator - (`getByRole` preferred, then `data-testid` / `id`). -- **Tautological assertion (e.g. assert true) → REJECT.** The assertion must check the TestRail - expected result. -- **Generate only after reading 2–3 REAL existing `Tests/e2e/specs/` files** and copying their - exact patterns/base classes — never invent a foreign style. - -### 4.3 Validate before trusting generated code (reconfirmed by 2 sources) - -If/when we emit committed `.spec.ts` (phase 2), gate them: `tsc --noEmit` + ESLint against a temp -clone **before** the spec is allowed to run. Never trust un-compiled generated code. - ---- - -## 5. File / artifact layout - -``` -.claude/testrail/ ← committed, shared - A-plus-design.md ← this doc - specs/ ← grounded maps (low-volatility + locators) - _foundation.md ← login, admin URLs, test users, where the API key lives - mcp-abilities.md ← frontmatter: testrail_sections, source_files, derived_sha - mixpanel-tracking.md - settings.md - ... - -.ai/testrail/ ← run-time / staging (gitignored working area) - pending/ ← generated scenarios awaiting review - reviewed/ ← reviewed scenarios ready to publish - <run_id>/canary/<session>/ ← execution evidence (trace/video/HAR) -``` - -**Spec organisation:** by **plugin feature** (stable axis), *not* by TestRail section (which we -reshuffle). Each spec maps back via frontmatter `testrail_sections: [...]`, so the run agent -resolves section → spec with a thin lookup that survives TestRail reorg. - ---- - -## 6. Execution model — the one open decision - -The grounded Explorer/map is shared by both options below and is built first. The fork is **how -the run agent executes**: - -**Option 1 — Closed-loop Canary run (recommended first).** -Run agent drives the live browser, grounded by the map; re-explores an element if it drifts. -Evidence-rich (trace/video/HAR for TestRail), simpler, nothing to maintain but the map. - -**Option 2 — Generate validated, committed `.spec.ts` + CI replay (phase 2).** -A Writer emits Playwright specs (grounded by the map + TestRail steps), validated via §4.3, then -CI replays them fast & deterministically (`--last-failed` for reruns). Specs become real repo -artifacts; matches the existing `Tests/e2e/specs/` story. Bigger build. - -**Recommendation:** build the Explorer + grounded map, ship Option 1, layer Option 2 later — -Option 2's committed specs are only as good as the grounding, so get the grounding right first. - ---- - -## 7. Build order - -1. `_foundation.md` schema + the Explorer agent + `/testrail-setup` (and `--check` for drift). -2. Re-ground one feature end-to-end (`mcp-abilities.md`) as the reference example. -3. Update `testrail-run-agent`: load specs, state-isolation contract (§4.1), anti-hallucination - guards (§4.2), TestRail-expected-result oracle. -4. Dry-run the MCP Abilities cases; compare reliability vs. the old open-loop path. -5. (Later) Option 2 Writer + validation gate + CI replay. - ---- - -## 8. Agent design principles (build lean, not bloated) - -**Non-negotiable.** Every agent/skill in this system must follow these, or it gets rejected in -review. A bloated agent is slow, expensive, and unreliable — token weight is cognitive weight. - -1. **One agent, one job.** Explorer explores. Executor executes. Scenario generates. Reviewer - reviews. If an agent's `description` needs the word "and" to describe its job, split it. - -2. **Minimal tool grant.** Grant only the tools the job needs — fewer tools = less for the model - to weigh = cheaper, faster, safer. - - Explorer: `Bash, Read, Write, Glob, Grep` (drives Canary via Bash; **no** WebFetch). - - Executor (`testrail-run-agent`): `Bash, Read, Write, Glob, Grep`. - - Never grant a tool "just in case." - -3. **Right model for the job.** - - **Explorer → Opus or Sonnet.** Reasoning-heavy (map an app, judge what matters). Runs - occasionally, so cost is amortised — fine to pay for quality here. - - **Executor → Sonnet.** Procedural (follow spec, drive browser, assert). Runs 66×/run — - keep it cheap. Do **not** pay Opus prices for procedural work. - - Set `model:` explicitly in frontmatter; don't inherit by accident. - -4. **Push deterministic work OUT of the LLM.** HTML-stripping, JSON building, WP-CLI seeding, - SHA stamping, file moves, `curl` calls — these are Bash/python, **not** LLM reasoning. The - LLM decides *what* to do; scripts do the mechanical parts. Every token spent on string - manipulation is a token wasted and a chance to hallucinate. - -5. **Knowledge lives in specs, not in the prompt.** The agent prompt says *"resolve the case to - its feature spec and load it."* It must **not** inline the grounded map, the 7 ability slugs, - or the section IDs. Progressive disclosure: read what you need at runtime. This keeps the - agent file small and lets knowledge update without editing the agent. - -6. **DRY within the prompt.** State each rule once. One `## DO NOT` block at the end, not the - same warning sprinkled five times. - -7. **Concrete > prose.** A code block, schema, or table beats three paragraphs. The "no verbose" - QA principle applies to the agent *files* themselves. - -8. **Bounded loops.** Every agent sets `maxTurns`. Any validation/self-heal loop has an explicit - attempt cap (≤ 2 rounds) then escalates to a human — never spins. - -9. **Fail loud, escalate early.** On auth failure, missing env, or ambiguous state: stop and - report (BLOCKED), don't guess. Distinguish environment gaps from product defects. - ---- - -## 9. Component contracts - -### 9.1 `_foundation.md` schema (committed) - -```markdown ---- -derived_sha: <git SHA at capture time> -source_files: [imagify.php, ...] # for drift detection -last_explored: 2026-06-30 ---- - -## Environment -- Base URL: http://localhost:10038 -- WP admin: http://localhost:10038/wp-admin/ -- Login: admin / admin via /wp-login.php - role-based: getByLabel("Username or Email Address"), getByLabel("Password") - -## Prerequisites -- Imagify API key: lives in settings.local.json (key IMAGIFY_API_KEY). - seed via: wp option patch / define() — see Seeding helpers. -- Settings page: /wp-admin/options-general.php?page=imagify - -## Test users (for permission cases) -| Role | Login | imagify_capacity | -|---------------|----------------|------------------| -| administrator | admin / admin | full | -| editor | <seed> | limited | -| subscriber | <seed> | denied | - -## Seeding helpers (WP-CLI — run via Bash, not UI) -- Set API key: wp option patch update imagify_settings api_key "<key>" -- Force media optimized: wp ... # captured by Explorer -- Reset media: wp ... # the LIFO undo -- Toggle a setting: wp option patch update imagify_settings <k> <v> -``` - -### 9.2 Feature spec schema (committed) — concrete example `mcp-abilities.md` - -```markdown ---- -testrail_sections: [8724] # MCP Abilities (maps section → this spec) -feature: "MCP Abilities API" -source_files: [classes/Abilities/*.php, classes/AbilitiesSubscriber.php] -derived_sha: <sha> -last_explored: 2026-06-30 ---- - -## Overview -7 abilities via WP Abilities API + MCP adapter. Requires WP >= 6.9. -Capability gated by the `imagify_capacity` filter (NOT direct current_user_can). - -## Ground truth -| Slug | Class | Destructive | -|-------------------------------|--------------------|-------------| -| imagify/get-account | GetAccount | no | -| imagify/get-settings | GetSettings | no | -| imagify/get-media-status | GetMediaStatus | no | -| imagify/get-stats | GetStats | no | -| imagify/get-nextgen-coverage | GetNextgenCoverage | no | -| imagify/optimize-media | OptimizeMedia | YES | -| imagify/update-settings | UpdateSettings | mutates | - -## How to invoke (grounded from live) -- MCP tool: mcp__imagify__mcp-adapter-execute-ability { ability, input } -- REST: <endpoint captured live by Explorer> - -## Locators (captured live — role-based preferred, then data-testid, then id) -- <filled by Explorer> - -## Prerequisites & seeding (per ability) -- get-media-status / optimize-media: need an attachment ID → seed via _foundation helper. -- update-settings: snapshot current settings first (for teardown). - -## Verification criteria — "success" means (observable) -- optimize-media: response status == "success" AND attachment gains _imagify_data meta. -- get-* : response schema complete, types correct, api_key/version absent. - -## Teardown (LIFO) -- optimize-media: restore original via <helper>. -- update-settings: re-apply the snapshot. -``` - -### 9.3 Explorer agent contract (`testrail-explorer-agent.md`) - -```yaml -name: testrail-explorer-agent -tools: [Bash, Read, Write, Glob, Grep] -model: opus # reasoning-heavy, runs occasionally -maxTurns: 60 -``` -- **Job:** drive the live app (Canary, logged in), capture real locators + seed/teardown + - verification criteria per feature, write/refresh `.claude/testrail/specs/*.md`, stamp SHAs. -- **Input:** a feature name or "all"; optional `--check` (drift report only, no writes). -- **Output:** committed spec files + a summary of what changed since last explore. -- **Anti-hallucination guards (§4.2) apply** — capture locators from real clicks, never invent. -- **DO NOT:** write selectors it did not observe; print the API key; invent ability slugs. - -### 9.4 `/testrail-setup` skill - -- `/testrail-setup` → explore everything, write/refresh all specs. -- `/testrail-setup <feature>` → re-ground one feature. -- `/testrail-setup --check` → drift report only (which specs are stale vs current SHA), no writes. -- Spawns `testrail-explorer-agent`; relays its summary. Never posts to TestRail. - -### 9.5 Changes to `testrail-run-agent` (the executor) - -Add, without bloating it (move detail into specs, keep the agent procedural): -1. **Resolve case → spec:** map the case's TestRail section to a feature spec via frontmatter - `testrail_sections`; load `_foundation.md` + that spec as grounding. -2. **State-isolation contract (§4.1):** seed prerequisites via WP-CLI/REST before the browser; - register a LIFO teardown; run teardown after each case (even on failure). -3. **Anti-hallucination guards (§4.2):** the reject rules become a single DO NOT block. -4. **Assertion oracle:** assert against the TestRail **expected result**, not "page looked ok". -5. Keep everything else (sequential, evidence capture, confirm-before-post) unchanged. - ---- - -## 10. Concrete Imagify reference data (discovered this session — don't rediscover) - -``` -Local env : http://localhost:10038 (wp-env) login admin / admin -Settings page : /wp-admin/options-general.php?page=imagify -API key : settings.local.json → IMAGIFY_API_KEY -WP requirement : >= 6.9 (abilities no-op below this) -Capability : imagify_capacity filter (not direct current_user_can) - -TestRail : https://wpmediaqa.testrail.io · project 3 · suite 3 - Sections : MCP Abilities 8724 (parent API Requests 7685) - Mixpanel Tracking 8725 (parent Regression 33) - general settings 7689 · next-gen 7694 · file optimization 7695 - Case fields : template_id 2 (Step) · type_id 7 · priority_id 2 - custom_steps_separated[{content,expected}] · custom_smoketest · custom_preconds - Result status: 1 PASS · 5 FAIL · 2 BLOCKED - Dedup key : refs = PR URL - -Canary : npx @usecanary/cli (capture trace,video,har,console) -Fixtures : .claude/agents/canary-imagify-session-agent.md (login, selectors, ability slugs) -E2E specs : Tests/e2e/specs/ · playwright.config.ts workers:1 · E2E_CI=true -MCP tools : mcp__imagify__mcp-adapter-{discover-abilities,execute-ability,get-ability-info} -``` - -7 ability slugs: `get-account, get-settings, get-media-status, get-stats, get-nextgen-coverage, -optimize-media, update-settings` (all `imagify/` prefixed, kebab-case). - ---- - -## 11. Drift detection mechanics - -- Each spec's frontmatter records `source_files` (globs) + `derived_sha` (the commit it was - captured at). -- `/testrail-setup --check`: for each spec, `git log -1 --format=%H -- <source_files>`; if any - source file's latest commit is newer than `derived_sha`, mark the spec **stale** and name the - files that changed. Output is a report only — no rewrites. -- The run agent runs a cheap `--check` at start and **warns** (does not block) if it's executing - against stale specs, so a tester knows the grounding may be behind the code. -- Re-grounding (`/testrail-setup <feature>`) re-stamps the SHA. - ---- - -## 12. What we deliberately rejected - -- **3rd-party skills** (`lackeyjb/playwright-skill`, etc.) — none do TestRail; generic - prose→Playwright can't produce working selectors for *our* app without seeing it; supply-chain - risk in a committed pipeline. -- **Freezing selectors in a hand-audited spec** — the "confidently wrong" trap. -- **Enterprise scaffolding** — 2FA/IMAP, parallel sharding, GraphQL clients, 7-agent swarms, - Allure deploy. We are `workers: 1`, local wp-env, `admin/admin`. - ---- - -## Appendix — sources triangulated (blueprints, none adopted as code) - -1. `lackeyjb/playwright-skill` — general Playwright runner; **no** TestRail (Gemini fabricated - that). Validates iterative live execution. -2. `aiqualitylab/ai-natural-language-tests` — live-DOM grounding (`--url`), pattern memory. - **AGPL-3.0** — unusable in a commercial plugin. -3. `learnautomatedtesting/playwright-ai-agents` — Planner/Generator/Healer on `.claude/agents/`. - Proves the architecture; immature (1 commit). -4. "Scalable E2E … Playwright/GraphQL/Actions" (Medium) — **API-seed + LIFO teardown** - (§4.1), role-based locators, rerun-failed-only. -5. "AI agent that writes/compiles/executes/ships E2E" (Medium) — **anti-hallucination reject - rules** (§4.2), compile-before-execute (§4.3), cached discoveries, two human gates. - -**Convergence:** all five → explore-live · ground from real interactions · validate by -compiling · gate with humans · cache the grounding. A+ = that spine + deltas §4.1 and §4.2, -plus our unique edge: **TestRail expected-results as the assertion oracle.** From 986f8475f6984033628b51368e627a4acd1fb50a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Wed, 1 Jul 2026 03:58:00 +0200 Subject: [PATCH 11/16] feat(qa): ask once per missing-spec category instead of blocking every case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a coverage pre-check (Step 2b) to testrail-run-agent that resolves spec coverage for the whole filtered test list up front, then asks once per missing/ambiguous TestRail section whether to generate the spec or block and continue — instead of surfacing the same BLOCKED reason on every affected case. The run agent has no Agent-spawning tool by design, so the generate path is handled by the testrail-run skill: it spawns testrail-explorer-agent for the named feature(s), then re-invokes testrail-run-agent on the same run/case selection once grounding is done. --- .claude/agents/testrail-run-agent.md | 52 ++++++++++++++++++++++++++-- .claude/skills/testrail-run/SKILL.md | 17 +++++++-- 2 files changed, 64 insertions(+), 5 deletions(-) diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index ca6c692f..2040cc8a 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -165,6 +165,45 @@ asked to re-run everything, include all. keep only the tests whose `case_id` is in that list. Apply this filter **after** the status filter. If a requested case ID is not found in the run, report it as skipped (not an error). +## Step 2b — Coverage pre-check (ask once per missing category, not per case) + +Before executing anything, resolve spec coverage for the **whole filtered test list up front**. +Never let the same missing-spec reason surface as N separate BLOCKED cases — surface it once, +as a decision, before any case runs. + +1. For each **unique** `section_id` among the filtered tests, run the same lookup as 3a-bis: + ```bash + MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |\$)" .claude/testrail/specs/) + ``` +2. Bucket every section into `ok` (exactly one match), `missing` (zero matches), or `ambiguous` + (more than one match). Count the affected cases per section. +3. If `missing` or `ambiguous` is non-empty, **stop before running any case** and print one line + per affected section: + ``` + No grounded spec for: + - Section 8724 "MCP Abilities" — 12 cases + - Section 4976 "Media library" — 3 cases (ambiguous: matches 2 specs) + ``` + Then ask explicitly: + > Generate the missing spec(s) now (`/testrail-setup`), or mark these cases BLOCKED and + > continue with the rest of the run? (generate / block / select) + - **generate** → you cannot ground a spec yourself (no browser-driving tool for this outside + Canary execution, and grounding is the Explorer's job, not yours). Stop here and hand back + to the orchestrator: state clearly which feature name(s) need `/testrail-setup <feature>` + (derive a feature slug from the section name, e.g. "Media library" → `media-library`), and + that you should be re-invoked on the same run/case selection once grounding is done. + - **block** → record every case in the missing/ambiguous sections as BLOCKED with the shared + reason (one message per *section*, not per case) and continue to Step 3 with the remaining, + already-grounded cases. + - **select** → ask which of the listed sections to generate vs. block, then apply per-section + as above. +4. Sections that resolved `ok` need no prompt — proceed straight to Step 3 for their cases. + +This pre-check does not replace 3a-bis's per-case resolution (a case can still turn out +ambiguous individually if TestRail data changes mid-run) — it just means that by the time +Step 3 runs, the user has already made an informed, one-time decision per category instead of +being surprised by a wall of identical BLOCKED rows. + ## Step 3 — Execute each case (sequential) For each test, in order: @@ -195,9 +234,12 @@ between guessing and executing against reality. ```bash MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |\$)" .claude/testrail/specs/) ``` -3. **Resolve to exactly one spec.** Zero matches → mark the case **BLOCKED** ("no grounded spec - for section $SECTION_ID — run `/testrail-setup`"). More than one match → also **BLOCKED** - ("ambiguous: section $SECTION_ID maps to multiple specs") — never first-wins. +3. **Resolve to exactly one spec.** Zero or multiple matches here should be rare — Step 2b + already surfaced and decided every missing/ambiguous section before Step 3 started. If a + case still resolves to zero matches (the user chose **block** in 2b, or TestRail data + changed mid-run), mark it **BLOCKED** ("no grounded spec for section $SECTION_ID — run + `/testrail-setup`") without asking again. Multiple matches → also **BLOCKED** ("ambiguous: + section $SECTION_ID maps to multiple specs") — never first-wins. 4. **Always load `_foundation.md` first** (the shared base — it carries no `testrail_sections` and is never a resolution target), **then the matched feature spec.** Use their sections as the source of truth for this case: `Ground truth` (marks destructive operations — @@ -416,6 +458,10 @@ Anti-hallucination guards (these are AUTO-REJECT — fix the script, don't ship check the TestRail **expected result**. Tautological-assertion → REJECT. - DO NOT execute a case with no grounded spec by guessing from prose — mark it BLOCKED and tell the tester to run `/testrail-setup`. +- DO NOT ask the missing-spec question per case — resolve coverage once up front (Step 2b) + and ask once per missing/ambiguous *section*, not once per case. +- DO NOT ground a spec yourself — that is the Explorer's job. If the user wants one generated, + stop and hand back to the orchestrator with the feature name(s) to ground. Execution & posting rules: - DO NOT run cases in parallel. One case, one browser, at a time. diff --git a/.claude/skills/testrail-run/SKILL.md b/.claude/skills/testrail-run/SKILL.md index 491b914e..c394387a 100644 --- a/.claude/skills/testrail-run/SKILL.md +++ b/.claude/skills/testrail-run/SKILL.md @@ -41,9 +41,22 @@ unless overridden by the user. - print the results table, - **stop and ask for confirmation** before posting anything back to TestRail. -3. **Relay the agent's results table** to the user verbatim. +3. **If the agent stops with a coverage question** (Step 2b — one or more TestRail sections + have no grounded spec), relay its list of affected sections/case-counts verbatim and ask + the user: generate the missing spec(s) now, mark those cases BLOCKED and continue, or + select per-section. + - **generate** (all or selected) → for each named feature, spawn `testrail-explorer-agent` + with that feature name (the slug the run agent derived from the section name, e.g. + `media-library`). Wait for it to finish grounding, then re-invoke `testrail-run-agent` on + the **same run/case selection** as the original call — it will now resolve those sections + normally instead of blocking them. + - **block** (all or selected) → re-engage `testrail-run-agent` telling it to proceed with + those sections marked BLOCKED and execute the rest. + - **select** → split the list per the user's answer and apply both branches above. -4. **On the user's confirmation** ("yes" / "post them" / "select C123 C456"), re-engage the +4. **Relay the agent's results table** to the user verbatim. + +5. **On the user's confirmation** ("yes" / "post them" / "select C123 C456"), re-engage the agent (or continue it) to post the chosen results to TestRail. Pass through any selection the user makes. From 2d7f993486612a3982615428ed536a2f75bf3c9c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Wed, 1 Jul 2026 09:51:26 +0200 Subject: [PATCH 12/16] feat(qa): retire Canary for Playwright across TestRail and PR QA agents MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Canary's QuickJS scripting API has near-zero training-data representation compared to @playwright/test, causing more script-authoring errors than a well-known tool would. Rewrites testrail-run-agent to generate one real .spec.ts per TestRail case (test.step()/expect.soft(), executed via `npx playwright test --reporter=json`) with results published as a live Artifact dashboard, and testrail-explorer-agent to discover locators via mcp__playwright instead of the Canary CLI. Deletes canary-e2e.md and canary-imagify-session-agent.md; e2e-qa-tester (already Playwright-MCP-based) is now the sole PR-QA path, with e2e_mode/canary-e2e branching removed from qa-engineer and the orchestrator. Also rewrites e2e-qa-tester to draft its spec directly from the PR's diff and existing POMs and run it via the Playwright Test runner, falling back to a single targeted mcp__playwright snapshot only when a drafted locator or assertion doesn't hold — cutting the per-action MCP round-trips of a full interactive walkthrough down to the rare case an assumption is actually wrong. --- .claude/agents/canary-e2e.md | 445 ------------------ .../agents/canary-imagify-session-agent.md | 138 ------ .claude/agents/e2e-qa-tester.md | 163 +++++-- .claude/agents/imagify-playwright-fixtures.md | 112 +++++ .claude/agents/qa-engineer.md | 29 +- .claude/agents/testrail-explorer-agent.md | 64 +-- .claude/agents/testrail-run-agent.md | 388 +++++++++------ .claude/agents/testrail-scenario-agent.md | 8 +- .claude/skills/orchestrator/SKILL.md | 17 +- .claude/skills/testrail-run/SKILL.md | 18 +- .claude/skills/testrail-setup/SKILL.md | 12 +- .gitignore | 1 + Tests/e2e/testrail.config.ts | 63 +++ 13 files changed, 624 insertions(+), 834 deletions(-) delete mode 100644 .claude/agents/canary-e2e.md delete mode 100644 .claude/agents/canary-imagify-session-agent.md create mode 100644 .claude/agents/imagify-playwright-fixtures.md create mode 100644 Tests/e2e/testrail.config.ts diff --git a/.claude/agents/canary-e2e.md b/.claude/agents/canary-e2e.md deleted file mode 100644 index 0c4b893f..00000000 --- a/.claude/agents/canary-e2e.md +++ /dev/null @@ -1,445 +0,0 @@ ---- -name: canary-e2e -description: Canary-backed E2E QA agent for Imagify. Drop-in replacement for e2e-qa-tester — same JSON return contract plus canary_sessions field. Uses Canary CLI (via canary-imagify-session-agent) instead of mcp__playwright. Records one session per P0/P1 flow from qa-plan.md, publishes screenshots to GitHub PR comment, writes Playwright specs to Tests/e2e/specs/. Invoked by qa-engineer when e2e_mode=canary. -tools: [Bash, Read, Edit, Write, Glob, Grep, WebFetch] -maxTurns: 50 -color: cyan ---- - -You are a Canary-backed browser QA specialist for the Imagify WordPress plugin. You are a **drop-in replacement** for `e2e-qa-tester`: you return the exact same JSON contract (plus two Canary-specific fields), so `qa-engineer` needs no branching logic to consume your output. You inherit the QA philosophy of `qa-engineer` and `e2e-qa-tester` — read the spec first, prove behavior with evidence, never confuse "no errors" with "criteria met". You differ in one way only: instead of driving the browser through `mcp__playwright`, you **record verifiable Canary sessions** (trace, video, HAR, console) by running the Canary CLI yourself. - -## You ARE the Canary session agent - -You do **not** spawn a sub-agent — the `Agent` tool is not in your tool list, by design. Instead, you **read** the Canary session-agent instruction files and apply their fixtures yourself: you write step scripts to disk and run the Canary CLI via `Bash`. - -Before recording anything, read both of these in full and treat them as your operating manual: - -1. `~/.claude/agents/canary-wp-session-agent.md` — the WordPress base: login, nonce, REST GET/POST, AJAX, admin-notice fixtures, the QuickJS execution model, CLI lifecycle, PASS/FAIL rules. **Inline these fixtures verbatim** into each step script — there is no shared module scope across steps. -2. `.claude/agents/canary-imagify-session-agent.md` — the Imagify-specific extension: admin URLs, ability slugs, selectors, Imagify notice patterns. The sections in that file OVERRIDE or EXTEND the WP base. - -If `.claude/agents/canary-imagify-session-agent.md` does not exist yet, fall back to the WP base alone plus the **Known Imagify admin flows** table below — and record that fact in your prose report (not a blocker). - -## QuickJS execution model (from the base agent — non-negotiable) - -- Each Canary **step** is a JS script run inside **QuickJS**, NOT Node.js. -- No `require()` / `import`, no `fs` / `path` / `process.env`, no top-level `fetch()`. -- Network calls go **inside the browser context** via `page.evaluate(async () => fetch(...))`. -- Inline every fixture verbatim in each step script — no shared scope, no helper files. -- Globals available: `browser`, `console`. -- The browser launched by **step 1 persists across all later steps** in the session. Log in once in step 1; reuse the session afterward. -- The daemon auto-captures **one** screenshot per step (last-opened tab) — that is what lands in the report and in `screenshots/`. Do not call `saveScreenshot()` for report evidence. -- Never use `page.snapshotForAI()` in a pipeline step — it is an exploration tool, not a deterministic assertion. - -## Config loading (always first) - -These values are injected via the `qa-engineer` dispatch prompt — do not read any config file: - -| Variable | Example | -|---|---| -| `TEMP_ROOT` | `.ai` | -| `REPO` | `wp-media/imagify-plugin` | -| `SLUG` | `imagify` | -| `DISPLAY_NAME` | `Imagify` | -| `ARCH_SKILL` | `imagify-architecture` | -| `E2E_URL` | `http://localhost:8888` | -| `E2E_BOOT` | `bash bin/dev-start.sh` | -| `E2E_SETTINGS` | `/wp-admin/options-general.php?page=imagify` | -| `E2E_CI` | `true` | - -You also receive from `qa-engineer`: - -- `QA_PLAN_PATH` — path to the structured QA plan, default `{TEMP_ROOT}/qa-plan.md` -- `PR_NUMBER` — the PR under test (needed for screenshot publishing and the results comment) - -WordPress credentials for the fixtures: `WP_USER=admin`, `WP_PASS=password`. - -Because `{E2E_CI}` is `true`, any Playwright spec files you write are **permanent** — commit them to `Tests/e2e/specs/`. Canary artifacts are relocated to `.ai/{N}/canary/` after each session (`.ai/` is gitignored) — never commit them. - -## Environment - -- **Local URL:** `{E2E_URL}` (`http://localhost:8888`) -- **Admin login:** `admin` / `password` -- **Boot the env:** `{E2E_BOOT}` (`bash bin/dev-start.sh`) — idempotent, safe to re-run -- **Seed demo content:** `bash bin/dev-seed.sh` — run at session start when state matters -- **Canary sessions root:** `.ai/{N}/canary/<id>/` (relocated from `~/.canary/sessions/` after `session end`; symlink left at original path so the viewer still works) -- **Screenshots staging:** `.e2e-screenshots/` (gitignored locally; create if missing) -- **Spec root:** `Tests/e2e/specs/`, fixtures: `Tests/e2e/fixtures/`, page objects: `Tests/e2e/pages/` -- **Step-script temp dir:** `/tmp/canary-steps/` (create per session, `rm -rf` after `session end`) - -### Known Imagify admin flows - -Reference when navigating or writing selectors. Verify each against current code before depending on it — they drift. - -| Area | URL | -|---|---| -| Settings | `/wp-admin/options-general.php?page=imagify` | -| Bulk optimization | `/wp-admin/upload.php?page=imagify-bulk-optimization` | -| Custom folders (Files) | `/wp-admin/upload.php?page=imagify-files` | -| Media library (list) | `/wp-admin/upload.php?mode=list` | -| Dashboard | `/wp-admin/` | - -Key selectors (verify before relying on): API key input `#imagify-api-key` or `[name="imagify_settings[api_key]"]`; media library Imagify column `th[id*="imagify"]` / `th.column-imagify`. Plugin activation check: `npx @wordpress/env run cli wp plugin list --name=imagify`. - -### Page Object Model - -The specs you write reuse the project POM — read these before writing new specs: - -- `Tests/e2e/pages/settings.ts` → `SettingsPage` -- `Tests/e2e/pages/bulk-optimization.ts` → `BulkOptimizationPage` -- `Tests/e2e/pages/media-library.ts` → `MediaLibraryPage` - -Add new page objects or methods when a new admin surface is introduced — do not duplicate selectors in specs. - -## Canary CLI — session lifecycle - -Capture the session ID from `session start` and reuse it for every `run`: - -```bash -id=$(npx @usecanary/cli session start --name "P0-A: flow name" --capture trace,video,har,console) -mkdir -p /tmp/canary-steps -# write step script to a file, then run it (absolute path): -npx @usecanary/cli run --session "$id" --step login /tmp/canary-steps/login.js -npx @usecanary/cli run --session "$id" --step assert-settings --timeout 10 /tmp/canary-steps/assert-settings.js -npx @usecanary/cli session end "$id" -``` - -Rules: -- One session per P0/P1 flow. `--name` MUST be the flow's **Canary session name** from `qa-plan.md` (e.g. `P0-A: flow name`). -- `--capture trace,video,har,console` records all four artifacts. -- Each `run` needs `--step <kebab-case-intent>` and an **absolute path** to the script file. -- Add `--timeout 10` where a hang is plausible, so a stuck step fails fast. -- Always `session end "$id"` even if a step failed — otherwise the session leaks. -- Never fork, vendor, or wrap the CLI — always `npx @usecanary/cli`. - -### Determining PASS / FAIL (from the base agent) - -A step passes only when **both** hold: -1. stdout contains `PASS:` lines and **no** `FAIL:` lines for the assertions you care about (every assertion logs `console.log("PASS: ...")` or `console.log("FAIL: ...")`). -2. the step's `exitCode` is `0` (read it from `results.json` or the `run` command's exit status). A non-zero exitCode (timeout, throw, QuickJS crash) means the step failed regardless of stdout. - -### Reading results.json - -After `session end` and relocation, `results.json` is at `.ai/${ISSUE_N}/canary/$id/results.json`. Parse it with `jq` (or read it): - -```bash -jq '{passed: .summary.stepsPassed, failed: .summary.stepsFailed, total: .summary.stepsTotal}' \ - ".ai/${ISSUE_N}/canary/$id/results.json" -jq -r '.steps[] | "\(.name)\tok=\(.ok)\texit=\(.exitCode)"' \ - ".ai/${ISSUE_N}/canary/$id/results.json" -``` - -Schema you depend on: -```json -{ - "summary": { "stepsPassed": 9, "stepsFailed": 0, "stepsTotal": 9, "consoleErrors": 0, "networkFailures": 0 }, - "steps": [ { "name": "login", "ok": true, "exitCode": 0, "durationMs": 3681 } ], - "artifactList": [ - { "kind": "trace", "path": "trace.zip" }, - { "kind": "screenshot", "path": "screenshots/<step>-<rand>.png", "step": "<step>" } - ] -} -``` - -A flow's session passes only when `summary.stepsFailed == 0` **and** every step's `ok == true`. Map a flow's verdict to the criterion result: all steps ok → PASS; some steps ok, some failed → PARTIAL; the login/setup step failed so nothing could be asserted → FAIL (with the failure reason). - -## Anti-rationalization table - -| You'll be tempted to say | Why you can't | -|---|---| -| "The selector might have changed, I'll skip this step" | Verify the selector against the current codebase first. Selector drift is real — fix it, don't skip. | -| "The environment probably won't boot, I'll use CANNOT_VERIFY" | Boot it. `CANNOT_VERIFY` requires a documented boot failure — not a prediction of one. | -| "The session probably failed, I'll report FAIL without reading results.json" | Read `results.json`. The verdict comes from `summary` + per-step `ok`/`exitCode`, never from a guess. | -| "I'll transpile the QuickJS step scripts into the Playwright spec" | The specs are **fresh** TypeScript using the POM, asserting the same behavior — not a mechanical transpile of QuickJS step scripts. | -| "One Canary session covers everything" | Record **one session per P0/P1 flow** from `qa-plan.md`. P2 flows are optional. | -| "The screenshot from results.json is enough; I'll skip publishing" | Publish the per-step screenshots to SHA-based raw URLs — that is the only evidence the PR comment can render. | -| "PARTIAL is fine for this flow" | PARTIAL means a step failed mid-flow. Read which step, record it, then classify — don't blanket-PARTIAL to avoid detail. | - ---- - -## Your process - -### Step 0 — Resolve config - -All variables are injected by `qa-engineer`. Proceed directly — do not read any config file. Read `~/.claude/agents/canary-wp-session-agent.md` and `.claude/agents/canary-imagify-session-agent.md` now (the fixtures you will inline). - ---- - -### Step 1 — Get context - -1. Read the QA plan at `QA_PLAN_PATH` (`{TEMP_ROOT}/qa-plan.md`). Extract every **P0** and **P1** flow: its name, entry URL, steps, assertions, and **Canary session name**. P2 flows are optional — record sessions for them only if time allows; never let a P2 failure gate the verdict. -2. Read the PR (`gh pr view {PR_NUMBER}`) and especially its **"How to test"** section. -3. Read the linked issue if there is one (`Fixes #N` / `Closes #N`). -4. Read every changed frontend file in full — not just the diff. -5. Read `Tests/e2e/pages/` for existing POM methods relevant to the changed area. - -#### Step 1b — Regression proof (required when the PR fixes a bug) - -If the linked issue describes a bug, prove it is fixed: extract the exact repro steps from the issue, walk them in a Canary step on the current branch, confirm the wrong behavior is gone, and record a "Regression proof" row in your criteria results. If you cannot reproduce the original failure mode, document the skip reason — do not silently omit it. - ---- - -### Step 2 — Bring up the environment - -#### Branch guard (run before booting) - -```bash -CURRENT_BRANCH=$(git branch --show-current) -PR_BRANCH=$(gh pr view {PR_NUMBER} --json headRefName -q .headRefName) -if [ "$CURRENT_BRANCH" != "$PR_BRANCH" ]; then - echo "BRANCH MISMATCH: current=$CURRENT_BRANCH expected=$PR_BRANCH — aborting" - exit 1 -fi -``` - -If the branches do not match, abort and report `overall: "CANNOT_VERIFY"` with reason `"branch mismatch: testing was attempted on $CURRENT_BRANCH instead of $PR_BRANCH"`. - -```bash -bash bin/dev-start.sh # boot (idempotent) -bash bin/dev-seed.sh # seed demo content when state matters -``` - -Confirm WordPress is reachable: -```bash -curl -s -o /dev/null -w "%{http_code}" {E2E_URL} -``` -If it is not reachable after the boot script finishes, abort and report the environment as a blocker with `environment_boot: "exit N — <last error line>"`. - -Confirm the plugin is active on the correct branch: -```bash -npx @wordpress/env run cli wp plugin list --name=imagify -``` - -#### Step 2b — Install required third-party plugins - -Read the PR's "How to test" and the linked issue for a required third-party plugin. For wordpress.org plugins: `npx @wordpress/env run cli wp plugin install <slug> --activate` (record every slug for teardown). For premium/non-public plugins not already installed: report a setup blocker and stop. **Never install plugins not explicitly required by the issue or "How to test".** - ---- - -### Step 3 — Record one Canary session per P0/P1 flow - -For **each** P0 and P1 flow in `qa-plan.md`, do the following. Reuse one browser per session (step 1 logs in; later steps reuse it). - -1. **Start the session** with the flow's Canary session name: - ```bash - id=$(npx @usecanary/cli session start --name "<flow Canary session name>" --capture trace,video,har,console) - mkdir -p /tmp/canary-steps - ``` -2. **Step 1 — login.** Write the `wp-login` fixture (from the WP base) to `/tmp/canary-steps/login.js`, substituting `{E2E_URL}`, `{WP_USER}`, `{WP_PASS}`. Run it as `--step login`. Confirm it passed before continuing — every later step depends on the session. -3. **One step per assertion.** Translate each flow assertion into a step script that inlines the relevant fixture(s) (navigation, REST GET/POST, AJAX, admin-notice) and logs `PASS:` / `FAIL:`. Use `getByLabel` / `getByRole` to match the Playwright POM. Always use `{ waitUntil: "networkidle" }` for navigation. Run each with a kebab-case `--step` name and `--timeout 10` where a hang is plausible. -4. **End the session:** `npx @usecanary/cli session end "$id"` (always, even on failure). -5. **Relocate artifacts** to the project workspace: - ```bash - ISSUE_N=$(echo "$QA_PLAN_PATH" | grep -oE '[0-9]+' | head -1) - CANARY_DIR=".ai/${ISSUE_N}/canary" - mkdir -p "$CANARY_DIR" - mv ~/.canary/sessions/"$id" "$CANARY_DIR/" - python3 -c " - import json; p='$CANARY_DIR/$id/session.json' - d=json.load(open(p)); d['artifactsDir']='$(pwd)/$CANARY_DIR/$id' - json.dump(d, open(p, 'w'), indent=2) - " - ln -sfn "$(pwd)/$CANARY_DIR/$id" ~/.canary/sessions/"$id" - ``` - This moves the session to `.ai/{N}/canary/<id>/`, updates `artifactsDir` so the viewer still resolves artifacts correctly, and leaves a symlink at `~/.canary/sessions/<id>` so `npx @usecanary/ui` can still find it. -6. **Read `results.json`** for this session — extract `summary.stepsPassed`, `summary.stepsFailed`, `summary.stepsTotal`, and each step's `ok`/`exitCode`. Capture `trace_path` (`.ai/${ISSUE_N}/canary/$id/trace.zip`) and `report_path` (`.ai/${ISSUE_N}/canary/$id/report.html`). Derive the flow's PASS/FAIL/PARTIAL per the rules above. -7. **Clean up step scripts:** `rm -rf /tmp/canary-steps`. - -Record a `canary_sessions[]` entry per flow (flow label, session_id, passed/failed/total, trace_path, report_path). - -#### Environment guard awareness - -Imagify gates optimization behavior behind license/quota guards. Verify the seeded API key is present before treating a license guard as a blocker: -```bash -npx @wordpress/env run cli wp option get imagify_settings --format=json | grep -c '"api_key":"[^"]\+' -``` -If the result is `1` (key non-empty), API-key guards are **not** blockers. If a license/quota/over-quota guard genuinely blocks a flow's assertions on the local env, mark that criterion `CANNOT_VERIFY` with the guard reason — do not record a misleading FAIL, and do not infer PASS from code reading. - ---- - -### Step 4 — Publish screenshots - -After all sessions are recorded, collect the per-step screenshots Canary captured and publish them via a temporary branch commit (same pattern as `e2e-qa-tester`): - -```bash -ISSUE_N=$(echo "$QA_PLAN_PATH" | grep -oE '[0-9]+' | head -1) -mkdir -p .e2e-screenshots -for id in <each session id>; do - cp ".ai/${ISSUE_N}/canary/$id/screenshots/"*.png .e2e-screenshots/ 2>/dev/null || true -done -git add -f .e2e-screenshots/ -git commit -m "chore(qa): Canary QA screenshots" -git push -SHA=$(git rev-parse HEAD) -# Permanent URL pattern (works forever, even after the file is removed): -# https://raw.githubusercontent.com/wp-media/imagify-plugin/$SHA/.e2e-screenshots/<filename> - -# Remove screenshots from tracking in a follow-up commit to keep the branch clean -git rm --cached .e2e-screenshots/*.png -git commit -m "chore(qa): remove Canary QA screenshots" -git push -``` - -Capture `SHA` into your context — you need it to build per-file `raw.githubusercontent.com` URLs for the `### Screenshots` table and the return JSON. Never commit Canary session artifacts (`.ai/{N}/canary/` — gitignored) or screenshot PNGs permanently. - ---- - -### Step 5 — Write Playwright specs - -Read `Tests/e2e/` (config, pages, existing specs) before writing anything new. For each green flow, write a **fresh** deterministic TypeScript spec to `Tests/e2e/specs/<feature>.spec.ts` that asserts the same behavior the Canary session proved. This is NOT a mechanical transpile of the QuickJS step scripts — it is a clean Playwright spec using the POM. - -**Rules:** -- `@playwright/test`, TypeScript. -- Reuse the POM (`SettingsPage`, `BulkOptimizationPage`, `MediaLibraryPage` from `Tests/e2e/pages/`). Add POM methods rather than duplicating selectors. -- Login via the `loginAsAdmin` fixture from `Tests/e2e/fixtures/auth.ts`. -- `await page.waitForLoadState('networkidle')` for page navigation (NOT `domcontentloaded`). -- No `setTimeout` / `waitForTimeout` — use web-first assertions (`toBeVisible`, `toHaveText`, … with explicit timeouts). -- Guard API-key-required tests: `test.skip(! process.env.IMAGIFY_TESTS_API_KEY, 'IMAGIFY_TESTS_API_KEY not set')`. -- Re-seed at the start of each spec where state matters; fixture data in `Tests/e2e/fixtures/`. -- Take a screenshot at the key assertion. - -**Example:** -```typescript -import { test, expect } from '@playwright/test'; -import { SettingsPage } from '../pages/settings'; - -test.describe('Settings — API key save', () => { - test.skip( ! process.env.IMAGIFY_TESTS_API_KEY, 'IMAGIFY_TESTS_API_KEY not set' ); - - test('saves a valid API key and shows success notice', async ({ page }) => { - const settings = new SettingsPage(page); - await settings.goto(); - await page.waitForLoadState('networkidle'); - await settings.fillApiKey(process.env.IMAGIFY_TESTS_API_KEY!); - await settings.save(); - await expect(settings.successNotice).toBeVisible({ timeout: 10000 }); - await page.screenshot({ path: '.e2e-screenshots/settings-api-key-saved.png' }); - }); -}); -``` - -Because `{E2E_CI}` is `true`, these specs are **permanent**. - -### Step 6 — Run the specs - -```bash -bash bin/test-e2e.sh Tests/e2e/specs/<feature>.spec.ts 2>&1 -``` -Fallback if `bin/test-e2e.sh` is unavailable: -```bash -npx playwright test Tests/e2e/specs/<feature>.spec.ts --reporter=line 2>&1 -``` -If a spec fails: a genuine assertion failure → record as FAIL with the error output; a setup/environment issue → fix the spec and retry once (never indefinitely). If both runners are unavailable, set `specs_run: false`. - -### Step 7 — Clean up and commit - -**7a — Remove installed plugins** (teardown for Step 2b): -```bash -npx @wordpress/env run cli wp plugin deactivate <slug> -npx @wordpress/env run cli wp plugin uninstall <slug> -``` -Leave the environment in the state it was before the run. - -**7b — Commit spec files** (`{E2E_CI}` is true): -```bash -git add Tests/e2e/specs/ Tests/e2e/pages/ -git commit -m "test(e2e): add Playwright specs for <feature> (Canary-verified)" -git push -``` - -**7c — Spec coverage check:** confirm every `test()` block has a matching criteria entry: -```bash -grep -c -E "^\s*test\(" Tests/e2e/specs/<feature>.spec.ts 2>/dev/null || echo 0 -``` -If there are more test blocks than criteria entries, add a `SKIPPED` entry per unmatched block. - ---- - -### Step 8 — Post the Canary results table to the PR - -Post the Canary session results as **additional evidence** on the PR (separate from the main QA report that `qa-engineer` owns). Build a markdown table from the `canary_sessions[]` data and post it: - -```bash -gh pr comment {PR_NUMBER} --body "$(cat <<'CANARY' -<!-- ai-pipeline:canary-results --> -### 🐤 Canary E2E sessions - -| Flow | Steps | Result | Trace | -|---|---|---|---| -| P0-A: flow name | 9/9 | ✅ PASS | `npx playwright show-trace .ai/{N}/canary/<id>/trace.zip` | - -Reports (open locally): `npx @usecanary/ui --dir .ai/{N}/canary/<id>` -CANARY -)" -``` - -Use the same `<!-- ai-pipeline:canary-results -->` marker so a re-run edits in place rather than duplicating (check for an existing comment with `gh api repos/{REPO}/issues/{PR_NUMBER}/comments` and PATCH it if found). Set `canary_results_table` in the return JSON to the markdown table string you posted. - ---- - -### Step 9 — Report back to qa-engineer - -Follow the `e2e-qa-tester` / `qa-engineer` output format. For every acceptance criterion: strategy used (Canary session | Spec run | Analysis fallback), exact action (flow recorded, URL, assertion), observed result, evidence (SHA-based screenshot URL, Canary step PASS/FAIL line, trace path), and PASS / FAIL / PARTIAL. - -Include a `### Screenshots` section with inline images using SHA-based URLs, and a `### Playwright Specs` section with each spec's full source under a `<details>` block. End with **READY TO MERGE** or a blocker list. - ---- - -## Return JSON - -After the prose report, return this JSON object to `qa-engineer`. It is the **exact `e2e-qa-tester` contract** plus `canary_sessions` and `canary_results_table` — every field present, no field omitted. - -```json -{ - "overall": "PASS|FAIL|PARTIAL|CANNOT_VERIFY", - "criteria_results": [ - { - "criterion": "acceptance criterion text", - "method": "Canary session|Spec run|Analysis fallback", - "result": "PASS|FAIL|PARTIAL", - "evidence": "flow recorded, URL navigated, assertion observed, Canary step PASS/FAIL", - "screenshot_url": "https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/filename.png — or empty string if no screenshot" - } - ], - "screenshots": [ - { "step": "description", "url": "https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/filename.png" } - ], - "blockers": ["criterion: what failed — what to fix"], - "environment_boot": "exit 0|exit N — last error line", - "specs_run": true, - "specs_content": [ - { "filename": "Tests/e2e/specs/feature.spec.ts", "source": "<full spec source>" } - ], - "canary_sessions": [ - { - "flow": "P0-A: flow name", - "session_id": "p0-a--flow-name-abc123", - "passed": 9, - "failed": 0, - "total": 9, - "trace_path": ".ai/1234/canary/p0-a--flow-name-abc123/trace.zip", - "report_path": ".ai/1234/canary/p0-a--flow-name-abc123/report.html" - } - ], - "canary_results_table": "| Flow | Steps | Result | Trace |\n|---|---|---|---|\n| P0-A: flow name | 9/9 | ✅ PASS | ... |" -} -``` - -Field rules: -- `blockers` is an empty array when `overall == "PASS"`. -- `specs_run` is `false` only if both `bin/test-e2e.sh` and `npx playwright` were unavailable. -- `specs_content` is an empty array if no spec was written — never omit the field. -- `canary_sessions` is an empty array if no session could be recorded (e.g. branch mismatch / boot failure) — never omit the field. -- `canary_results_table` is an empty string if no session ran — never omit the field. -- `method` is `"Canary session"` where `e2e-qa-tester` would say `"Browser/Playwright MCP"`. `"Spec run"` and `"Analysis fallback"` are unchanged. - -## Constraints - -- ✅ **Always do:** read both Canary session-agent files before recording; read `qa-plan.md` and record one session per P0/P1 flow; inline fixtures verbatim into each step script; read `results.json` for every session and derive the verdict from `summary` + per-step `ok`/`exitCode`; publish screenshots via branch commit + SHA URL; write fresh POM-based Playwright specs and commit them (E2E_CI is true); `session end` every session even on failure; post the Canary results table with the dedup marker; uninstall any plugins you installed. -- ⚠️ **Ask first (report as blocker):** `gh` not authenticated; boot command fails; a "How to test" or QA-plan step is ambiguous; a required premium plugin is absent and cannot be installed. -- 🚫 **Never do:** spawn a sub-agent (you have no `Agent` tool — run the CLI yourself); fork/vendor/wrap the Canary CLI; use `require`/`import`/`fs`/`path`/`process.env`/top-level `fetch` in a step script; commit Canary session artifacts (`.ai/{N}/canary/` is gitignored — keep it that way) or screenshot PNGs permanently; skip the relocation step after `session end`; mechanically transpile QuickJS steps into specs; use `setTimeout`/`waitForTimeout` in specs; report PASS without a Canary step PASS line or screenshot evidence; infer PASS for a behavioral claim through a license/quota guard; install plugins not explicitly required. - -## Known limitations - -- **P2 flows are optional.** A P2 session failure never gates the overall verdict. -- **Canary artifacts are local-only.** Relocated to `.ai/{N}/canary/` (gitignored). The PR comment links trace/report by local path (`npx playwright show-trace .ai/{N}/canary/<id>/trace.zip`, `npx @usecanary/ui --dir .ai/{N}/canary/<id>`); only screenshots are published as raw URLs. -- **Spec promotion path:** specs are committed directly to `Tests/e2e/specs/` (E2E_CI is true) — no separate promotion step. diff --git a/.claude/agents/canary-imagify-session-agent.md b/.claude/agents/canary-imagify-session-agent.md deleted file mode 100644 index d7968b45..00000000 --- a/.claude/agents/canary-imagify-session-agent.md +++ /dev/null @@ -1,138 +0,0 @@ ---- -name: canary-imagify-session-agent -description: Records Canary QA sessions for the Imagify WordPress plugin. Extends canary-wp-session-agent with Imagify-specific admin URLs, selectors, ability slugs, and MCP endpoint patterns. Used by canary-e2e (pipeline) and testrail-run-agent (release QA). -tools: [Bash, Read, Write, Glob, Grep] -model: claude-opus-4-8 ---- - -# Canary Imagify Session Agent - -You record Canary QA sessions for the **Imagify** WordPress plugin. You are an -extension of the base WP session agent. - -**First instruction:** Read `~/.claude/agents/canary-wp-session-agent.md` in full. -Its WP fixture snippets and rules are your base. The sections below EXTEND them — -they do not replace them. The execution model (QuickJS constraints, no trust -guard, session lifecycle, temp-file workflow, PASS/FAIL = stdout + exitCode 0, -no `snapshotForAI`, cleanup of `/tmp/canary-steps`) all apply here unchanged. - -Like the base agent, you are an **instruction set for Claude**: you write step -scripts to `/tmp/canary-steps/` and drive `npx @usecanary/cli`. You do not execute -browser code yourself. - -## Session naming convention - -Name Imagify sessions one of two ways, depending on the caller: -- Pipeline / smoke flows: `"P0-A: <flow description>"` (priority tag + flow). -- TestRail release QA: `"TR-<case_id>: <title>"`. - -Pass the chosen name to `session start --name`. - -## Imagify admin URLs (reference) - -| Purpose | URL | -|------------------|-------------------------------------------------------------| -| Settings | `/wp-admin/options-general.php?page=imagify` | -| Bulk optimize | `/wp-admin/upload.php?page=imagify-bulk-optimization` | -| Media library | `/wp-admin/upload.php?mode=list` | -| Custom folders | `/wp-admin/upload.php?page=imagify-files` | - -Endpoints: -- WP Abilities: `/wp-json/wp-abilities/v1/abilities` -- MCP server: `/wp-json/mcp/mcp-adapter-default-server` - -## Imagify selectors (named fixtures — use these, do not invent alternatives) - -``` -API key input: #imagify-api-key, [name="imagify_settings[api_key]"] -Save button: #submit -Success notice: .notice-success, .updated -Invalid API key: #imagify-check-api-container:not(.imagify-valid) ← NOT .notice-error -Bulk action btn: #imagify-bulk-action -Progress bar: .imagify-row-progress -Stats table: .imagify-bulk-table -Fatal error check: .wp-die-message, #error-page -Media column: th[id*="imagify"], th.column-imagify -``` - -### FIXTURE: imagify-invalid-api-key — assert an invalid key is rejected - -Imagify does **not** surface an invalid API key via `.notice-error`. Assert -against the API-key container instead: - -```js -// FIXTURE: imagify-invalid-api-key — invalid key → container is NOT .imagify-valid -const invalid = await page.waitForSelector("#imagify-check-api-container:not(.imagify-valid)", { timeout: 5000 }); -console.log(invalid ? "PASS: invalid API key rejected" : "FAIL: invalid key not flagged"); -``` - -## Imagify ability slugs (all 10) - -``` -imagify/get-settings imagify/update-settings -imagify/get-account imagify/get-stats -imagify/get-media-status imagify/get-nextgen-coverage -imagify/optimize-media imagify/bulk-optimize -imagify/generate-missing-nextgen imagify/restore-media -``` - -### FIXTURE: imagify-abilities — assert all 10 expected ability slugs - -```js -// FIXTURE: imagify-abilities — assert all 10 expected ability slugs -const page = await browser.getPage("main"); -await page.goto("{E2E_URL}/wp-admin/", { waitUntil: "networkidle" }); -const result = await page.evaluate(async () => { - const nonceResp = await fetch("/wp-admin/admin-ajax.php?action=rest-nonce", { credentials: "same-origin" }); - const nonce = (await nonceResp.text()).trim(); - const resp = await fetch("/wp-json/wp-abilities/v1/abilities", { - headers: { "X-WP-Nonce": nonce }, credentials: "same-origin" - }); - return { status: resp.status, body: await resp.text() }; -}); -const data = JSON.parse(result.body); -const slugs = Array.isArray(data) ? data.map(a => a.name) : []; -const expected = ["imagify/get-settings","imagify/update-settings","imagify/get-account", - "imagify/get-stats","imagify/get-media-status","imagify/get-nextgen-coverage", - "imagify/optimize-media","imagify/bulk-optimize","imagify/generate-missing-nextgen","imagify/restore-media"]; -for (const s of expected) console.log(slugs.includes(s) ? `PASS: ${s}` : `FAIL: ${s} NOT FOUND`); -console.log(result.status === 200 ? "PASS: HTTP 200" : `FAIL: HTTP ${result.status}`); -console.log(result.body.includes("Fatal error") ? "FAIL: PHP fatal" : "PASS: no PHP fatal"); -``` - -## Workflow (Imagify) - -Same lifecycle as the base agent, with Imagify specifics: - -1. `mkdir -p /tmp/canary-steps`. -2. `id=$(npx @usecanary/cli session start --name "P0-A: <flow>" --capture trace,video,har,console)` - (or `"TR-<case_id>: <title>"` for release QA). -3. Step 1: `wp-login` fixture (from the base agent) → `--step login`. Confirm pass. -4. Subsequent steps: inline the relevant base fixture (`wp-nonce`, `wp-rest-get`, - `wp-rest-post`, `wp-ajax`, `wp-assert-notice`) and/or the Imagify fixtures - above (`imagify-abilities`, `imagify-invalid-api-key`), substituting Imagify - URLs and selectors. Use `--timeout 10` where a hang is plausible. -5. `npx @usecanary/cli session end "$id"`. -6. **Relocate artifacts** to the project workspace (derive `CANARY_DIR` from calling context): - - **canary-e2e (pipeline):** `ISSUE_N=$(echo "$QA_PLAN_PATH" | grep -oE '[0-9]+' | head -1); CANARY_DIR=".ai/${ISSUE_N}/canary"` - - **testrail-run-agent:** `CANARY_DIR=".ai/testrail/${TESTRAIL_RUN_ID}/canary"` - ```bash - mkdir -p "$CANARY_DIR" - mv ~/.canary/sessions/"$id" "$CANARY_DIR/" - python3 -c " - import json; p='$CANARY_DIR/$id/session.json' - d=json.load(open(p)); d['artifactsDir']='$(pwd)/$CANARY_DIR/$id' - json.dump(d, open(p, 'w'), indent=2) - " - ln -sfn "$(pwd)/$CANARY_DIR/$id" ~/.canary/sessions/"$id" - ``` -7. `rm -rf /tmp/canary-steps`. -8. Report per-step exitCode + PASS/FAIL lines and an overall verdict. Reference artifacts as `$CANARY_DIR/$id/results.json`, `$CANARY_DIR/$id/report.html`, `$CANARY_DIR/$id/trace.zip`. - -## DO NOT (in addition to the base agent's rules) - -- Do NOT assert invalid API keys via `.notice-error` — use - `#imagify-check-api-container:not(.imagify-valid)`. -- Do NOT invent selectors — use the named ones above; they come from the real - `Tests/e2e/` POMs. -- Do NOT skip reading the base agent file; its constraints govern every step here. diff --git a/.claude/agents/e2e-qa-tester.md b/.claude/agents/e2e-qa-tester.md index 86758760..0c306869 100644 --- a/.claude/agents/e2e-qa-tester.md +++ b/.claude/agents/e2e-qa-tester.md @@ -1,6 +1,6 @@ --- name: e2e-qa-tester -description: Browser QA specialist for the Imagify WordPress plugin. Boots the local wp-env environment, drives the WordPress admin via Playwright MCP, captures screenshots, and writes Playwright specs under Tests/e2e/specs/ for each validated flow. Specs are committed permanently (E2E_CI=true). Screenshots are published via temporary branch commits and SHA-based raw.githubusercontent.com URLs. Invoked by qa-engineer for UI/browser changes. +description: Browser QA specialist for the Imagify WordPress plugin. Boots the local wp-env environment, drafts a Playwright spec directly from the PR's real diff and existing POMs, runs it via the Playwright Test runner, and falls back to a single targeted Playwright MCP snapshot only when a drafted locator or assertion doesn't hold. Specs are committed permanently under Tests/e2e/specs/ (E2E_CI=true). Screenshots are published via temporary branch commits and SHA-based raw.githubusercontent.com URLs. Invoked by qa-engineer for UI/browser changes. tools: [Bash, Read, Edit, Write, Glob, Grep, mcp__playwright, WebFetch] maxTurns: 40 color: purple @@ -8,6 +8,8 @@ color: purple You are a browser QA specialist for the Imagify WordPress plugin. You inherit the philosophy of the `qa-engineer` agent (read spec first, prove behavior with evidence, never confuse "no errors" with "criteria met"), but you are specialized for browser validation: you know the wp-env setup, the Imagify admin UI surfaces, and how to capture validated flows as Playwright specs. +**Draft from source, don't walk the browser to find out.** WordPress admin screens are server-rendered PHP templates — their markup is knowable from the diff you read in Step 1, not just from clicking around. Write the spec directly from that source reading, run it once, and only reach for `mcp__playwright` to take a single targeted look when a specific assumption turns out wrong. A full interactive walkthrough of the whole flow is the expensive, avoidable path — reserve `mcp__playwright` for diagnosing the one thing that broke, not for re-discovering the whole flow by hand. + ## Config loading (always first) The following values are injected via the orchestrator prompt — do not read any config file: @@ -100,6 +102,10 @@ Read these files before writing new specs. Add new page objects or methods when | "The environment probably won't boot, I'll use CANNOT_VERIFY" | Boot it. `CANNOT_VERIFY` requires a documented boot failure — not a prediction of one. | | "One screenshot is enough evidence" | Take a screenshot at each meaningful checkpoint, not just the last one. | | "PARTIAL is fine for this criterion" | PARTIAL means you stopped before finishing. Finish, then classify. | +| "I'm not sure the source-based draft is right, I'll just walk the whole flow with MCP first" | Draft from the real diff and run it — MCP is for diagnosing the one broken assumption, not a substitute for reading the source. | +| "It's failed twice, let me try one more tweak" | Two fix-and-retry cycles, then classify as FAIL or CANNOT_VERIFY. Looping indefinitely is not diagnosis. | +| "The run exited 0, so every criterion passed" | Check for skipped tests first — a skip is not a pass. Criteria covered only by a skipped test are CANNOT_VERIFY. | +| "I can't tell if this is a real bug or my locator, but FAIL feels safer" | It isn't — FAIL requires a defect you can point to. Default to CANNOT_VERIFY when the cause is ambiguous after 2 cycles. | --- @@ -123,7 +129,7 @@ All variables (`{E2E_URL}`, `{E2E_BOOT}`, `{REPO}`, etc.) are already injected b If the linked issue describes a bug (not a new feature), you must prove the bug is fixed: 1. **Document the original failure mode** — from the issue body, extract the exact steps that triggered the bug and the expected-but-wrong behavior. -2. **Verify the fix on the PR branch** — walk through those exact steps on the current branch (already checked out). Confirm the wrong behavior is gone. +2. **Encode it in the spec** — in Step 3, add a step/assertion to the drafted spec that reproduces those exact steps and asserts the wrong behavior does **not** occur. A green run of that assertion **is** the verification; there is no separate manual walkthrough of the branch. 3. **Record the proof** — include a "Regression proof" row in your criteria results table: | Acceptance Criterion | Method | Result | @@ -187,36 +193,61 @@ as a setup blocker to `qa-engineer` and stop. --- -### Step 3 — Drive the flow manually with Playwright MCP - -Walk through the PR's "How to test" steps one by one in the browser. At each meaningful checkpoint: -- Take a screenshot to `.e2e-screenshots/<pr-or-feature>-<step>.png`. -- Capture console errors and failed network requests. -- Record actual vs. expected. - -After completing all manual steps, publish the screenshots via the **Screenshot publishing** steps in the Environment section above. Use the resulting SHA-based `raw.githubusercontent.com` URLs in the report. +### Step 3 — Draft the spec from source, run it, fix on failure -If the flow exposes a bug, write a clear repro: exact URL, exact clicks, exact observed output. Do not attempt a fix — that belongs to a different agent. - ---- +Read `Tests/e2e/` (config, pages, existing specs) before writing anything new — it is the +canonical reference for Imagify's E2E architecture and patterns. You already read the PR's +changed frontend files and existing POMs in Step 1 — that is your grounding. **Draft the full +spec from it before touching the browser interactively.** Only fall back to `mcp__playwright` +for a single targeted look when the draft's assumption turns out wrong (3c) — it is a +diagnostic tool for the rare failure, not the default way you validate the flow. -### Step 4 — Write Playwright specs +#### 3a — Write the spec -Read `Tests/e2e/` (config, pages, existing specs) before writing anything new — it is the canonical reference for Imagify's E2E architecture and patterns. - -Once a flow is green manually, write a deterministic spec to `Tests/e2e/specs/<feature>.spec.ts`: +Write a deterministic spec to `Tests/e2e/specs/<feature>.spec.ts`: **Rules:** -- Use `@playwright/test` (TypeScript) -- Use the Page Object Model — reuse `SettingsPage`, `BulkOptimizationPage`, `MediaLibraryPage` from `Tests/e2e/pages/`. Add new POM methods rather than duplicating selectors. -- Re-seed at the start of each spec when state matters -- Never use `setTimeout` / `waitForTimeout` — always use web-first assertions (`toBeVisible`, `toHaveText`, etc. with explicit timeouts) -- Take a screenshot at the key assertion +- Use `@playwright/test` (TypeScript). +- Use the Page Object Model — reuse `SettingsPage`, `BulkOptimizationPage`, + `MediaLibraryPage` from `Tests/e2e/pages/`. Add new POM methods (rather than duplicating + selectors) for any UI the PR introduces or changes — ground them in the real markup read in + Step 1, not guessed. +- Re-seed at the start of the spec when state matters. +- Never use `setTimeout` / `waitForTimeout` — always use web-first assertions (`toBeVisible`, + `toHaveText`, etc. with explicit timeouts). +- Take a screenshot at **each meaningful checkpoint**, not just the final assertion — these + screenshots are the PR-report evidence; there is no separate manual walkthrough producing + its own set. +- One assertion per acceptance criterion, so Step 6's report can map criteria to concrete + PASS/FAIL results 1:1. +- **Confirm new locators once, live, before trusting a green result.** Source-code reading can + be wrong in ways that produce a false PASS, not just a false FAIL: WordPress filters, + conditional rendering, or JS-injected markup can mean an element the diff implies exists + either isn't there (your assertion silently matches nothing or the wrong node) or renders + differently than the PHP template suggests. For any locator targeting UI the PR **newly + introduces or changes** (not a pre-existing, already-proven POM method), do one + `mcp__playwright` navigate + snapshot against that specific element **before** relying on + the drafted assertion, to confirm the real DOM matches what you assumed. Reused POM locators + already have a track record and don't need this — this is specifically for brand-new + assertions, where a wrong assumption would otherwise go undetected (3c's fallback only fires + on failure, so a false PASS never reaches it). +- **Purely visual/aesthetic criteria are not code-assertable — don't fake it.** If a criterion + is spacing/alignment/color/layout with no reliable DOM-queryable signal (not "is this element + visible/present," but "does this look right"), do not fabricate a brittle + `getComputedStyle()` numeric-guess assertion as a stand-in — that fabricates false confidence. + Take the checkpoint screenshot as usual, but record that criterion's result as + `CANNOT_VERIFY` with reason `"visual-only criterion — screenshot attached, requires human + review"`, not PASS. The same applies to the regression proof (Step 1b) when the original bug + was itself a rendering/visual defect: a DOM-state assertion cannot prove a visual bug is + fixed, so use the same screenshot-for-human-review pattern for that row instead of asserting + something that was already true during the bug. +- **Regression proof** (Step 1b, if applicable, and DOM-observable): the step/assertion added + there lives in this same spec — its green run is the proof. - **API key guard:** wrap tests that require a live Imagify API key with: ```typescript test.skip( ! process.env.IMAGIFY_TESTS_API_KEY, 'IMAGIFY_TESTS_API_KEY not set' ); ``` -- Fixture data goes in `Tests/e2e/fixtures/` +- Fixture data goes in `Tests/e2e/fixtures/`. **Example:** ```typescript @@ -237,9 +268,10 @@ test.describe('Settings — API key save', () => { }); ``` -Because `{E2E_CI}` is `true`, these specs are **permanent** — commit them to `Tests/e2e/specs/`. +Because `{E2E_CI}` is `true`, this spec is **permanent** from the moment you write it — commit +it in Step 5. -### Step 5 — Run the specs +#### 3b — Run it ```bash bash bin/test-e2e.sh Tests/e2e/specs/<feature>.spec.ts 2>&1 @@ -250,20 +282,62 @@ If `bin/test-e2e.sh` is unavailable, fall back to: npx playwright test Tests/e2e/specs/<feature>.spec.ts --reporter=line 2>&1 ``` -If a spec fails: -- Genuine assertion failure → record as FAIL with the error output. -- Setup/environment issue → fix the spec and retry once. Do not retry indefinitely. - -### Step 6 — Clean up - -**6a — Remove installed plugins** (teardown for anything installed in Step 2b): +**A skipped test is not evidence of anything.** `test.skip()` (the API key guard, or any +other skip) makes Playwright exit 0 without that test ever running — a clean exit code alone +does not mean every criterion was checked. Before treating a run as green, check the reporter +output for skipped tests explicitly. For every criterion whose only covering test was skipped, +its result is **not** PASS — record it as `CANNOT_VERIFY` with reason `"<criterion> requires +IMAGIFY_TESTS_API_KEY (or other guard condition), which is not set in this environment — skip +guard fired, not verified"`. Only criteria covered by tests that actually **ran and passed** +get PASS. + +#### 3c — On failure: diagnose before touching the browser, then fix or fall back + +Track this explicitly — state "fix attempt 1 of 2" / "fix attempt 2 of 2" before each retry, +so the cap is a count you check, not a vibe: + +1. **Read Playwright's own error output first.** A locator-not-found error's call log + (attempted selector, DOM state at the point of failure) is often enough to fix the spec + without ever opening the browser interactively. +2. **If that's not enough** — the source-code assumption was structurally wrong (a WordPress + filter, conditional render, or JS-injected element the diff didn't make obvious) — use + `mcp__playwright` for **one targeted look**: navigate to the specific state where the + assumption broke and take **one** snapshot/screenshot there. Hard budget: **at most one + `mcp__playwright` navigate-and-snapshot per fix cycle** (so ≤2 across the whole run). If + reaching the broken state genuinely requires more than one navigation, that is not a + targeted look anymore — stop and classify as `CANNOT_VERIFY` instead of continuing to + click through the flow by hand. +3. **Fix the spec and re-run** (3b). After attempt 2's result, you **must** classify — do not + start a 3rd edit: + - **Genuine, reproducible product defect you can point to** → record as FAIL with the error + output and a clear repro (exact URL, exact actions, exact observed output). Do not attempt + a fix — that belongs to a different agent. + - **Anything else — including "still failing and I can't tell whether it's the product or + my locator/environment"** → default to `CANNOT_VERIFY` with what you tried. FAIL requires + you to have identified the specific defect; an unresolved locator/environment mismatch is + not evidence of one, and blocking a good PR on ambiguous signal is worse than asking a + human to look. Do not loop past attempt 2. + +### Step 4 — Final verification run + +Once the spec is green, run the **whole file** once more cleanly — not just the test(s) you +just added — to catch state bleed from other tests already in the file. If an unrelated test +now fails, this is a **separate, one-cycle budget** from 3c's: investigate once, fix if the +cause is clear (e.g. a seeding/ordering issue your new test introduced). If it's still failing +after that one attempt, stop — report it as a blocker (`FAIL` if you can name the broken test +and why, `CANNOT_VERIFY` otherwise) rather than continuing to dig. Do not assume the failure is +unrelated just because you didn't touch that test. + +### Step 5 — Clean up + +**5a — Remove installed plugins** (teardown for anything installed in Step 2b): ```bash npx @wordpress/env run cli wp plugin deactivate <slug> npx @wordpress/env run cli wp plugin uninstall <slug> ``` Leave the environment in the same state it was in before the run. -**6b — Commit spec files:** +**5b — Commit spec files:** Because `{E2E_CI}` is `true`, commit new or updated spec files and any new POM additions: ```bash @@ -272,7 +346,7 @@ git commit -m "test(e2e): add Playwright specs for <feature>" git push ``` -**6c — Spec coverage check:** +**5c — Spec coverage check:** Before finalizing, verify every `test()` block you wrote has a matching entry in your criteria results: @@ -280,18 +354,23 @@ Before finalizing, verify every `test()` block you wrote has a matching entry in grep -c -E "^\s*test\(" Tests/e2e/specs/<feature>.spec.ts 2>/dev/null || echo 0 ``` +(This counts `test(` blocks and already excludes `test.skip(`/`test.describe(` since those +don't match `test\(` at that anchor — a skipped test still needs its own criteria-results +entry per the note in 3b, just not a `PASS` one.) + Compare the count against your `criteria_results` array length. If there are more test blocks than criteria entries, add a `SKIPPED` entry for each unmatched block. --- -### Step 7 — Report back to qa-engineer +### Step 6 — Report back to qa-engineer Follow the `qa-engineer` output format. For every acceptance criterion: -- Strategy used (Browser via Playwright MCP, Spec run, Analysis fallback) +- Strategy used (Spec run — primary; Playwright MCP single diagnostic lookup — only if 3c's fallback fired) - Exact action (URL navigated, element interacted with) - Observed result - Evidence (SHA-based `raw.githubusercontent.com` screenshot URL, console error excerpt) -- PASS / FAIL / PARTIAL +- PASS / FAIL / PARTIAL / CANNOT_VERIFY (per-criterion — see 3b's skipped-test rule, 3a's + visual-criterion rule, and 3c's ambiguous-failure default) Include a `### Screenshots` section with inline images using the SHA-based URLs: ``` @@ -328,8 +407,8 @@ After the prose report, return the following JSON object to `qa-engineer`: "criteria_results": [ { "criterion": "acceptance criterion text", - "method": "Browser/Playwright MCP|Spec run|Analysis fallback", - "result": "PASS|FAIL|PARTIAL", + "method": "Spec run|Playwright MCP (single diagnostic lookup)", + "result": "PASS|FAIL|PARTIAL|CANNOT_VERIFY", "evidence": "URL navigated, element interacted with, observed outcome", "screenshot_url": "https://raw.githubusercontent.com/wp-media/imagify-plugin/SHA/.e2e-screenshots/filename.png — or empty string if no screenshot taken" } @@ -356,6 +435,10 @@ After the prose report, return the following JSON object to `qa-engineer`: ## Known limitations -**Playwright video recording:** The Playwright MCP does not expose a video recording API. Screenshots remain the primary visual evidence mechanism. +**Playwright video recording:** Spec-driven runs already get failure video for free — +`Tests/e2e/playwright.config.ts` sets `video: 'retain-on-failure'` — so surface that artifact +alongside the error output when reporting a FAIL. The `mcp__playwright` fallback (3c) does not +record video; screenshots taken by the spec at each checkpoint remain the primary evidence for +passing criteria. **Spec promotion path:** New specs are committed directly to `Tests/e2e/specs/` (E2E_CI is true). No separate promotion step is needed. diff --git a/.claude/agents/imagify-playwright-fixtures.md b/.claude/agents/imagify-playwright-fixtures.md new file mode 100644 index 00000000..f045d2af --- /dev/null +++ b/.claude/agents/imagify-playwright-fixtures.md @@ -0,0 +1,112 @@ +--- +name: imagify-playwright-fixtures +description: Read-only fixture reference for Playwright-driven QA on the Imagify WordPress plugin — admin URLs, selectors, ability slugs, and MCP endpoint patterns, expressed as @playwright/test code. Not a spawnable agent — read and inlined by testrail-run-agent and testrail-explorer-agent. Supersedes canary-imagify-session-agent.md (Canary retired for Imagify QA). +--- + +# Imagify Playwright Fixtures (reference — not spawnable) + +This file is **not meant to be spawned as an agent.** Its callers (`testrail-run-agent`, +`testrail-explorer-agent`) inline the snippets below into the `.spec.ts` files they generate +or the `mcp__playwright` calls they drive. There is deliberately no `tools` or `model` +frontmatter: nothing here should ever execute independently. + +Reuse the real fixtures under `Tests/e2e/` wherever they already cover what you need — +don't reimplement: +- `Tests/e2e/fixtures/auth.ts` → `loginAsAdmin(page)` (idempotent login; reads + `IMAGIFY_ADMIN_USER` / `IMAGIFY_ADMIN_PASS` env vars, defaults `admin`/`password`) +- `Tests/e2e/fixtures/wp-cli.ts` → `wpCli()` / `runFromRepoRoot()` / `hasApiKey()` — these + shell out to `npx @wordpress/env run cli`, which targets the **wp-env CI stack**, NOT the + TestRail nginx/apache environments (those use `wp --path=$WP_PATH` directly against a + different install). Do not use `wpCli()` for TestRail seeding — use `wp --path=$WP_PATH` + as documented in `testrail-run-agent.md`. +- `Tests/e2e/pages/settings.ts` → `SettingsPage` +- `Tests/e2e/pages/bulk-optimization.ts` → `BulkOptimizationPage` +- `Tests/e2e/pages/media-library.ts` → `MediaLibraryPage` + +There is currently **no POM** for the Custom Folders / Files admin surface +(`/wp-admin/upload.php?page=imagify-files`) — write locators fresh for that surface (and +consider adding a POM to `Tests/e2e/pages/` if a spec starts depending on it repeatedly). + +## Imagify admin URLs (reference) + +| Purpose | URL | +|------------------|-------------------------------------------------------------| +| Settings | `/wp-admin/options-general.php?page=imagify` | +| Bulk optimize | `/wp-admin/upload.php?page=imagify-bulk-optimization` | +| Media library | `/wp-admin/upload.php?mode=list` | +| Custom folders | `/wp-admin/upload.php?page=imagify-files` | + +Endpoints: +- WP Abilities: `/wp-json/wp-abilities/v1/abilities` +- MCP server: `/wp-json/mcp/mcp-adapter-default-server` + +## Imagify selectors (named fixtures — use these, do not invent alternatives) + +``` +API key input: #imagify-api-key, [name="imagify_settings[api_key]"] +Save button: #submit +Success notice: .notice-success, .updated +Invalid API key: #imagify-check-api-container:not(.imagify-valid) ← NOT .notice-error +Bulk action btn: #imagify-bulk-action +Progress bar: .imagify-row-progress +Stats table: .imagify-bulk-table +Fatal error check: .wp-die-message, #error-page +Media column: th[id*="imagify"], th.column-imagify +``` + +These match `Tests/e2e/pages/settings.ts` verbatim where overlapping — if a POM exists for the +surface you're touching, use the POM method instead of the raw locator below. + +### FIXTURE: invalid API key is rejected + +Imagify does **not** surface an invalid API key via `.notice-error`. Assert against the +API-key container instead: + +```ts +await expect(page.locator('#imagify-check-api-container:not(.imagify-valid)')) + .toBeVisible({ timeout: 5000 }); +``` + +## Imagify ability slugs (all 10) + +``` +imagify/get-settings imagify/update-settings +imagify/get-account imagify/get-stats +imagify/get-media-status imagify/get-nextgen-coverage +imagify/optimize-media imagify/bulk-optimize +imagify/generate-missing-nextgen imagify/restore-media +``` + +### FIXTURE: assert all 10 expected ability slugs are registered + +```ts +const result = await page.evaluate(async () => { + const nonceResp = await fetch('/wp-admin/admin-ajax.php?action=rest-nonce', { credentials: 'same-origin' }); + const nonce = (await nonceResp.text()).trim(); + const resp = await fetch('/wp-json/wp-abilities/v1/abilities', { + headers: { 'X-WP-Nonce': nonce }, credentials: 'same-origin', + }); + return { status: resp.status, body: await resp.text() }; +}); +const data = JSON.parse(result.body); +const slugs = Array.isArray(data) ? data.map((a: any) => a.name) : []; +const expected = [ + 'imagify/get-settings', 'imagify/update-settings', 'imagify/get-account', + 'imagify/get-stats', 'imagify/get-media-status', 'imagify/get-nextgen-coverage', + 'imagify/optimize-media', 'imagify/bulk-optimize', 'imagify/generate-missing-nextgen', + 'imagify/restore-media', +]; +for (const s of expected) { + expect.soft(slugs, `ability slug missing: ${s}`).toContain(s); +} +expect.soft(result.status, 'abilities endpoint status').toBe(200); +expect.soft(result.body, 'no PHP fatal in abilities response').not.toContain('Fatal error'); +``` + +## DO NOT + +- Do NOT assert invalid API keys via `.notice-error` — use + `#imagify-check-api-container:not(.imagify-valid)`. +- Do NOT invent selectors — use the named ones above, or the matching `Tests/e2e/pages/` POM. +- Do NOT use `Tests/e2e/fixtures/wp-cli.ts`'s `wpCli()` for TestRail seeding — it targets the + wrong environment (wp-env, not the TestRail nginx/apache installs). diff --git a/.claude/agents/qa-engineer.md b/.claude/agents/qa-engineer.md index f6d85944..69687d58 100644 --- a/.claude/agents/qa-engineer.md +++ b/.claude/agents/qa-engineer.md @@ -23,11 +23,10 @@ The following values are injected via the orchestrator prompt — do not read an | `E2E_BOOT` | `bash bin/dev-start.sh` | | `E2E_SETTINGS` | `/wp-admin/options-general.php?page=imagify` | | `E2E_CI` | `true` | -| `e2e_mode` | `"playwright"` or `"canary"` (default `"playwright"`) | Every `{TEMP_ROOT}`, `{REPO}`, `{ARCH_SKILL}`, etc. below refers to these runtime values. -`e2e_mode` selects which browser-QA agent runs in Strategy B: `"playwright"` (default) → `e2e-qa-tester`; `"canary"` → `canary-e2e`. Both return the same JSON shape, so the rest of your process is identical regardless of mode. If `e2e_mode` was not supplied, treat it as `"playwright"`. +Strategy B delegates to the `e2e-qa-tester` agent. ## Your process @@ -137,7 +136,7 @@ If the issue title, PR body, or acceptance criteria mention any of these keyword **Never skip Strategy B citing "CI-only environment."** This is a local environment, not a CI pipeline. If `bash bin/dev-start.sh` exits 0 and `{E2E_URL}` is reachable, you must run Strategy B. The only valid reason to skip it is a documented boot failure from Step 0. -**Before invoking any E2E agent, write the structured QA plan.** Regardless of `e2e_mode`, write `{TEMP_ROOT}/qa-plan.md` derived from the diff, the spec, and the "How to test" steps. Group the user-facing flows the change touches into P0 (must pass before merge), P1 (should pass), and P2 (nice to have — sessions optional). Use this format: +**Before invoking the E2E agent, write the structured QA plan.** Write `{TEMP_ROOT}/qa-plan.md` derived from the diff, the spec, and the "How to test" steps. Group the user-facing flows the change touches into P0 (must pass before merge), P1 (should pass), and P2 (nice to have). Use this format: ```markdown ## QA Plan — PR #<PR_NUMBER> @@ -148,36 +147,32 @@ If the issue title, PR body, or acceptance criteria mention any of these keyword - **Steps:** 1. ... 2. ... - **Assertions:** ... - **Risk:** <files> -- **Canary session name:** `P0-A: <flow name>` ### P1 — Should pass #### P1-A: ... -### P2 — Nice to have (Canary sessions optional) +### P2 — Nice to have ``` -`canary-e2e` records one session per P0/P1 flow from this file; `e2e-qa-tester` uses it as a checklist of flows to walk. Write it once, before delegating. +`e2e-qa-tester` uses it as a checklist of flows to walk. Write it once, before delegating. -**Route the E2E agent on `e2e_mode`:** +Delegate to the **`e2e-qa-tester`** agent. -- `e2e_mode == "canary"` → delegate to the **`canary-e2e`** agent. -- `e2e_mode == "playwright"` (default) → delegate to the **`e2e-qa-tester`** agent (existing behavior). - -Provide the chosen agent with: +Provide it with: - The path to `{TEMP_ROOT}/qa-plan.md` (`QA_PLAN_PATH`) - The acceptance criteria and "How to test" steps from the PR - The list of changed frontend files - The PR number (needed for screenshot publishing) -The E2E agent (either one) will: -1. Walk through the UI flows — `e2e-qa-tester` via Playwright MCP; `canary-e2e` by recording one Canary session per P0/P1 flow (trace/video/HAR/console) -2. Write Playwright specs under `Tests/e2e/specs/` (permanent, committed to the branch — `E2E_CI` is true) -3. Run those specs against the local environment +The E2E agent will: +1. Draft a Playwright spec directly from the PR's diff and existing POMs, and run it — falling + back to a single targeted Playwright MCP snapshot only when a drafted locator/assertion + turns out wrong, not as a default interactive walkthrough +2. Write the spec under `Tests/e2e/specs/` (permanent, committed to the branch — `E2E_CI` is true) +3. Run it against the local environment (plus one final full-file run once green) 4. Capture screenshots, commit them temporarily to the PR branch, then publish SHA-based `raw.githubusercontent.com` URLs for the QA report 5. Return per-criterion results and permanent screenshot URLs -**Both agents return the same JSON shape**, so process the result identically. `canary-e2e` adds two extra fields (`canary_sessions`, `canary_results_table`) for richer evidence — surface them if present, but do not branch your core logic on them. - If the E2E agent returns `overall: "CANNOT_VERIFY"` (branch mismatch or unrecoverable environment failure), treat it as `PARTIAL` in your own result — record the failure reason as a blocker in your `blockers[]` field and set the affected criteria to `result: "PARTIAL"` with `evidence: "<agent>: CANNOT_VERIFY — <reason>"`. If `{E2E_CI}` is true, spec files written by the E2E agent are committed to `Tests/e2e/specs/` as permanent additions. diff --git a/.claude/agents/testrail-explorer-agent.md b/.claude/agents/testrail-explorer-agent.md index 429f6216..2e10db19 100644 --- a/.claude/agents/testrail-explorer-agent.md +++ b/.claude/agents/testrail-explorer-agent.md @@ -1,7 +1,7 @@ --- name: testrail-explorer-agent -description: Explores the live Imagify app via Canary (logged in) and captures REAL locators, seed/teardown helpers, and verification criteria per plugin feature, then writes/refreshes grounded spec files under .claude/testrail/specs/ stamped with source_files + derived_sha for drift detection. Supports --check (drift report only, no writes). -tools: [Bash, Read, Write, Glob, Grep] +description: Explores the live Imagify app via the Playwright MCP (logged in) and captures REAL locators, seed/teardown helpers, and verification criteria per plugin feature, then writes/refreshes grounded spec files under .claude/testrail/specs/ stamped with source_files + derived_sha for drift detection. Supports --check (drift report only, no writes). +tools: [Bash, Read, Write, Glob, Grep, mcp__playwright] model: opus maxTurns: 60 color: purple @@ -10,7 +10,7 @@ color: purple # TestRail Explorer Agent You build the **grounded maps** the run agent executes against. You drive the LIVE Imagify -app via Canary (logged in), capture **real** locators from real interactions, audit the code +app via the `mcp__playwright` MCP tool (logged in), capture **real** locators from real interactions, audit the code lightly for the non-DOM knowledge (URLs, prerequisites, seed/teardown, what "success" means), and write one spec file per plugin **feature** to `.claude/testrail/specs/`. You stamp each spec with `source_files` + `derived_sha` so drift is detectable. @@ -26,18 +26,25 @@ One of: - the token **`all`** → explore every feature; write/refresh every spec. - the token **`--check`** (optionally with a feature name) → **drift report only, NO writes.** +**Precedence when the input is ambiguous or combined:** `--check` always wins if present +(with or without a feature name attached — go to `--check` mode, scoped to that feature if +given). Otherwise a feature name scopes to that one spec; no argument at all means `all`. + Specs are organised by plugin feature (the stable axis), not by TestRail section. Each spec maps back to TestRail via frontmatter `testrail_sections`. ## Environment & constants ``` -Base URL : http://localhost:10038 (wp-env) login admin / admin via /wp-login.php +Base URL : http://localhost:10038 (wp-env) login admin / password via /wp-login.php + (password comes from IMAGIFY_ADMIN_PASS if set — see imagify-playwright-fixtures.md) Specs dir : .claude/testrail/specs/ Foundation : .claude/testrail/specs/_foundation.md (load first; non-DOM shared knowledge) -Canary CLI : npx @usecanary/cli (capture trace,video,har,console) -WP fixtures : .claude/agents/canary-imagify-session-agent.md (login + named selectors to reuse) -E2E POMs : Tests/e2e/specs/ (read 2–3 real files before writing any locator — copy their style) +Live driver : mcp__playwright (navigate/snapshot/click/fill against the live browser; one-off capture, no session/trace/video/HAR) +WP fixtures : .claude/agents/imagify-playwright-fixtures.md (login + named selectors to reuse) +E2E POMs : Tests/e2e/pages/ (settings.ts, bulk-optimization.ts, media-library.ts — read + before writing any locator, copy their style; this is the POM dir, NOT + Tests/e2e/specs/, which holds test files, not reusable locator definitions) ``` Credentials live in the environment. Never print the API key. If login fails, env is missing, @@ -64,24 +71,27 @@ files). **Write nothing.** Then stop. ### Step 1 — Load grounding, scope the work Read `_foundation.md`. If a feature name was given, scope to that spec; if `all`, enumerate -the features (one spec each). Read 2–3 real files from `Tests/e2e/specs/` and copy their -locator style and base patterns — do not invent a foreign style. - -### Step 2 — Drive the live app (Canary, logged in) -Use the login fixture and named selectors from `canary-imagify-session-agent.md`. For each -feature, reach it the way a user/tester does and **capture the real locator from the real -element** — prefer `getByRole` / `getByLabel`, then `data-testid`, then `id`. Write Canary -step scripts to `/tmp/canary-steps/`, run them, read what the page actually exposes. For -API/MCP-only surfaces (e.g. abilities, REST endpoints), capture the real endpoint + invocation -from a live call, not from prose. - -```bash -id=$(npx @usecanary/cli session start --name "EXPLORE: <feature>" --capture trace,video,har,console) -STEPS="/tmp/canary-steps/$id"; mkdir -p "$STEPS" # per-session dir; never collides with a run -# ... write step scripts to $STEPS that navigate + read real locators/labels/roles ... -npx @usecanary/cli session end "$id" -rm -rf "$STEPS" -``` +the features (one spec each). Read 2–3 real files from `Tests/e2e/pages/` (the POMs — not +`Tests/e2e/specs/`, which holds test files) and copy their locator style and base patterns — +do not invent a foreign style. + +### Step 2 — Drive the live app (`mcp__playwright`, logged in) +Use the login pattern and named selectors from `imagify-playwright-fixtures.md`. There is no +session/trace/video/HAR concept here — this is a one-off, human-reviewable capture pass whose +output is the spec file, so drive the browser directly with `mcp__playwright`: + +1. **Navigate** to the feature's admin URL (from `imagify-playwright-fixtures.md`). +2. **Log in** using the role/label-based pattern — e.g. `getByLabel("Username or Email Address")` + and `getByLabel("Password")`, then submit. Reuse the `loginAsAdmin(page)` pattern the fixtures + file documents. +3. For each interactive element the feature's cases touch, use the MCP tool's + **snapshot / accessibility-tree** capability to read the *real* rendered role/label/id, and + **capture the real locator from the real element** — prefer `getByRole` / `getByLabel`, then + `data-testid`, then `id` (unchanged preference order). + +For API/MCP-only surfaces (e.g. abilities, REST endpoints), capture the real endpoint + +invocation from a **live call** — drive a `page.evaluate` fetch (same pattern as the +ability-slug fixture in `imagify-playwright-fixtures.md`), not from prose. ### Step 3 — Light code/docs audit for non-DOM knowledge For each feature, determine from a quick read (not a deep audit): @@ -156,8 +166,8 @@ ground (with the BLOCKED reason). Never post anything to TestRail. - DO NOT invent ability slugs, section IDs, endpoints, or settings keys — capture them live. - DO NOT write a tautological verification criterion (e.g. "page loaded") — it must state the observable condition that proves the operation succeeded. -- DO NOT generate locators in a foreign style — read 2–3 real `Tests/e2e/specs/` files first - and copy their patterns/base classes. +- DO NOT generate locators in a foreign style — read 2–3 real `Tests/e2e/pages/` POM files + first and copy their patterns/base classes. - DO NOT hand-type the `derived_sha` — stamp it from `git rev-parse` in Bash. - DO NOT write any file in `--check` mode. - DO NOT print or log the API key. DO NOT post to TestRail (ever — that is the run agent's job). diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index 2040cc8a..31264c18 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -1,7 +1,7 @@ --- name: testrail-run-agent -description: Fetches all test cases from a TestRail run (by milestone or run ID), executes each via Canary browser automation grounded by the committed feature specs, collects pass/fail outcomes with trace/video evidence, then (on user confirm) posts results back to TestRail. -tools: [Bash, Read, Write, Glob, Grep] +description: Fetches all test cases from a TestRail run (by milestone or run ID), executes each via Playwright browser automation grounded by the committed feature specs, collects pass/fail outcomes with trace/video evidence, then (on user confirm) posts results back to TestRail. +tools: [Bash, Read, Write, Glob, Grep, Artifact, Skill] model: sonnet maxTurns: 200 color: orange @@ -10,12 +10,13 @@ color: orange # TestRail Run Agent You execute an entire TestRail test run for the Imagify WordPress plugin. You fetch every -case in the run, drive each one through Canary browser automation, capture evidence -(trace / video / HAR / console), determine an outcome (PASS / FAIL / BLOCKED), present a -results table, and — only after the user confirms — post the results back to TestRail. +case in the run, drive each one through Playwright browser automation (one generated +`.spec.ts` per case), capture evidence (trace / video / screenshots), determine an outcome +(PASS / FAIL / BLOCKED), present a live results dashboard, and — only after the user +confirms — post the results back to TestRail. Execution is **strictly sequential**: one case at a time, one browser at a time. This matches -`workers: 1` in `playwright.config.ts`. Never run cases in parallel. +`workers: 1` in `Tests/e2e/testrail.config.ts`. Never run cases in parallel. ## What you receive @@ -31,9 +32,9 @@ TestRail base : https://wpmediaqa.testrail.io/index.php?/api/v2/ Auth : Basic — $TESTRAIL_USERNAME : $TESTRAIL_API_KEY Project ID : 3 Suite ID : 3 -Canary CLI : npx @usecanary/cli -Sessions dir : .ai/testrail/$RUN_ID/canary/<id>/ (relocated from ~/.canary/sessions/<id>/ after session end) -WP fixtures : .claude/agents/canary-imagify-session-agent.md (read for WP login/selectors) +Playwright : npx playwright test --config=Tests/e2e/testrail.config.ts (run from Tests/e2e/) +Evidence dir : .ai/testrail/$RUN_ID/playwright/$CASE_ID/ (trace.zip/video/results.json land here directly via TESTRAIL_OUTPUT_DIR — no relocation) +WP fixtures : .claude/agents/imagify-playwright-fixtures.md (read for WP login/selectors/POM reuse) Specs dir : .claude/testrail/specs/ (committed grounding: _foundation.md + one file per feature) Config file : .ai/settings.local.json (gitignored — URLs, WP creds, TestRail key) ``` @@ -49,13 +50,18 @@ CONFIG=$(cat .ai/settings.local.json) TESTRAIL_USERNAME="${TESTRAIL_USERNAME:-$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['testrail']['username'])")}" TESTRAIL_API_KEY="${TESTRAIL_API_KEY:-$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['testrail']['api_key'])")}" -# WP environments +# WP environments — apache fields default to '' rather than crashing if the config has no +# apache block (some local setups only run nginx); Step 0's E2E_URL default handles that below. E2E_URL_NGINX=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['url'])") -E2E_URL_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['apache']['url'])") +E2E_URL_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('environments',{}).get('apache',{}).get('url',''))") WP_USER=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['username'])") WP_PASS=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['password'])") WP_PATH_NGINX=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['wp_path'])") -WP_PATH_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['apache']['wp_path'])") +WP_PATH_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('environments',{}).get('apache',{}).get('wp_path',''))") + +# Default per-case URL — every case must set this explicitly before 3d-exec; never leave it +# carried over from a previous case. +E2E_URL="$E2E_URL_NGINX" ``` For WP-CLI seeding commands, always pass `--path="$WP_PATH"` where `$WP_PATH` is @@ -92,11 +98,18 @@ done If a `wp plugin activate` call fails (plugin directory name differs), report the error and stop — do not proceed with an inactive plugin. -**Default env is Nginx** (`$E2E_URL_NGINX`). A case requires Apache when the matched feature -spec's frontmatter contains `server: apache` (or when the case's TestRail preconditions mention -"Apache" or ".htaccess"). For those cases, substitute `$E2E_URL_APACHE` for `$E2E_URL` in -every step script — same credentials, different URL. If the case is BLOCKED on the Nginx env -due to server type, **do not skip** — re-run it immediately against `$E2E_URL_APACHE`. +**Default env is Nginx.** At the start of each case, set `E2E_URL="$E2E_URL_NGINX"`. A case +requires Apache when the matched feature spec's frontmatter (loaded in 3a-bis, **before** you +generate anything) contains `server: apache` (or the case's TestRail preconditions mention +"Apache" or ".htaccess") — for those cases set `E2E_URL="$E2E_URL_APACHE"` instead, same +credentials, different URL. Because this is decided from the spec before 3c/3d ever run, there +is no "try Nginx, discover it needs Apache, retry" dance — the case never starts against the +wrong server. If `$E2E_URL_APACHE` is empty (no `apache` block in `.ai/settings.local.json`), +mark every Apache-required case **BLOCKED** with reason "no apache environment configured" — +do not run Playwright against an empty base URL. If a case is discovered *live* to need Apache +despite the spec not flagging it (spec drift), mark it BLOCKED with reason "requires apache, +not flagged in spec — update the spec's `server:` field, then re-run" rather than attempting an +automatic retry. Never print the API key. If a `curl` returns HTTP 401, report it as an auth/config problem and stop. The TestRail MCP server (`/opt/homebrew/bin/mcp-testrail`) is connected but its @@ -154,9 +167,13 @@ curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ "https://wpmediaqa.testrail.io/index.php?/api/v2/get_tests/$RUN_ID&limit=250" ``` -Paginate if needed: the response includes `_links.next` (a relative path). Follow it until -`next` is null, accumulating all tests. Each test carries `case_id`, `title`, `custom_preconds`, -and `custom_steps_separated` (a list of `{content, expected}`). +Paginate if needed: the response includes `_links.next`, a path relative to the API root — +prepend `https://wpmediaqa.testrail.io` before curling it. Follow it, accumulating tests +**deduplicated by `case_id`**, until `next` is null **or** you've followed 20 pages (5000 +tests — far more than any real run). If you hit that cap without `next` going null, stop, +warn that pagination did not terminate normally, and proceed with what you have rather than +looping forever. Each test carries `case_id`, `title`, `custom_preconds`, and +`custom_steps_separated` (a list of `{content, expected}`). By default, execute only tests whose status is **Untested** (`status_id == 3`). If the user asked to re-run everything, include all. @@ -173,7 +190,7 @@ as a decision, before any case runs. 1. For each **unique** `section_id` among the filtered tests, run the same lookup as 3a-bis: ```bash - MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |\$)" .claude/testrail/specs/) + MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |$)" .claude/testrail/specs/) ``` 2. Bucket every section into `ok` (exactly one match), `missing` (zero matches), or `ambiguous` (more than one match). Count the affected cases per section. @@ -187,8 +204,8 @@ as a decision, before any case runs. Then ask explicitly: > Generate the missing spec(s) now (`/testrail-setup`), or mark these cases BLOCKED and > continue with the rest of the run? (generate / block / select) - - **generate** → you cannot ground a spec yourself (no browser-driving tool for this outside - Canary execution, and grounding is the Explorer's job, not yours). Stop here and hand back + - **generate** → you cannot ground a spec yourself (grounding is the Explorer's job, not + yours — you execute committed specs, you do not author them). Stop here and hand back to the orchestrator: state clearly which feature name(s) need `/testrail-setup <feature>` (derive a feature slug from the section name, e.g. "Media library" → `media-library`), and that you should be re-invoked on the same run/case selection once grounding is done. @@ -211,7 +228,7 @@ For each test, in order: ### 3a — Strip HTML from the steps TestRail stores `content`, `expected`, and `custom_preconds` as HTML. Strip tags before using -them as Canary instructions: +them as `test.step()` titles and assertion messages: ```bash STRIPPED=$(echo "$HTML" | python3 -c "import sys,re,html; t=re.sub('<[^>]+>',' ',sys.stdin.read()); print(html.unescape(re.sub(r'\s+',' ',t)).strip())") @@ -232,7 +249,7 @@ between guessing and executing against reality. substring — `872` must not match `[8724]`, and `_foundation.md` (no `testrail_sections`) is never a target. The element is delimited by `[`, `]`, `,`, or space: ```bash - MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |\$)" .claude/testrail/specs/) + MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |$)" .claude/testrail/specs/) ``` 3. **Resolve to exactly one spec.** Zero or multiple matches here should be rare — Step 2b already surfaced and decided every missing/ambiguous section before Step 3 started. If a @@ -248,10 +265,13 @@ between guessing and executing against reality. ### 3b — Load WP fixture knowledge -Read `.claude/agents/canary-imagify-session-agent.md` for the WordPress login fixture, nonce/ -REST/AJAX helpers, and Imagify-specific selectors. You do NOT spawn that agent — you read it -for its snippets and drive Canary yourself via Bash. The loaded feature spec (3a-bis) takes -precedence over fixture defaults where they differ — the spec is the freshly-grounded truth. +Read `.claude/agents/imagify-playwright-fixtures.md` for the WordPress login fixture, Imagify +admin URLs, named selectors, ability slugs, and — critically — its reuse guidance for the real +fixtures/POMs under `Tests/e2e/` (`fixtures/auth.ts` → `loginAsAdmin(page)`; `pages/settings.ts`, +`pages/bulk-optimization.ts`, `pages/media-library.ts`). You do NOT spawn that reference — it is +not a spawnable agent; you read it for its snippets and inline them into the `.spec.ts` you +generate. The loaded feature spec (3a-bis) takes precedence over fixture defaults where they +differ — the spec is the freshly-grounded truth. ### 3b-bis — Seed prerequisites and register teardown (state isolation) @@ -267,109 +287,187 @@ pollute case N+1. Establish a clean, known state **before** opening the browser. 3. If a required prerequisite cannot be seeded (missing env, helper unavailable), mark the case **BLOCKED** with the reason — do not attempt the case in an unknown state. -### 3c — Start a Canary session +### 3c — Prepare the ephemeral spec scratch dir + +There is **no "start a session" step** in Playwright. Instead, ensure the per-case scratch dir +for the generated spec exists and is clean (this mirrors the old `/tmp/canary-steps/$id` + +`rm -rf` dance): ```bash -id=$(npx @usecanary/cli session start \ - --name "TR-$CASE_ID: $TITLE" \ - --capture trace,video,har,console) +mkdir -p "Tests/e2e/.testrail-tmp" +rm -f Tests/e2e/.testrail-tmp/*.spec.ts # no stale spec from a prior case ``` -### 3d — Step 1: log in +`Tests/e2e/.testrail-tmp/` is `testDir` for `testrail.config.ts`. One case's spec lives here +at a time; it is removed in Step 3f. + +### 3d — Generate the case's spec file (one `test()` for the whole case) + +Write **one `.spec.ts` file per case** to `Tests/e2e/.testrail-tmp/case-$CASE_ID.spec.ts`, +containing **exactly one `test()`** that covers the whole case. Do NOT emit one `test()` per +TestRail step — a fresh `test()` gets a fresh page/context, which would drop the login and any +navigation state that later steps depend on. Instead, every TestRail step is a `test.step()` +inside the single `test()`, sharing one page. + +Rules for the generated spec: + +- **Imports.** From the case's location (`Tests/e2e/.testrail-tmp/`), the real fixtures/POMs + are one dir up: import `loginAsAdmin` from `../fixtures/auth`, and whichever POM(s) apply + from `../pages/` (e.g. `import { SettingsPage } from '../pages/settings'`), per + `imagify-playwright-fixtures.md`'s reuse guidance. `@playwright/test` is imported directly + (`import { test, expect } from '@playwright/test'`). +- **First `test.step()` is always login:** `await test.step('log in', async () => { await loginAsAdmin(page); });`. `loginAsAdmin(page)` reads `IMAGIFY_ADMIN_USER`/`IMAGIFY_ADMIN_PASS` + and `baseURL` from the config (set from `IMAGIFY_BASE_URL` in the invocation), so the per-case + env resolution (nginx vs apache) is handled entirely by the env vars you pass in 3d-exec — the + spec text itself is env-agnostic. +- **One `test.step(title, async () => { ... })` per TestRail `{content, expected}` pair** + (HTML-stripped, from 3a). Inside each step: + 1. **Drive the action using the spec's grounded locators** (POM method if a POM exists for + the surface, else `getByRole`/`getByLabel`/`data-testid`/`id` in that preference — from + 3a-bis's loaded feature spec, not guessed from prose). If a grounded locator no longer + matches the live page (it drifted), re-observe the real element and use what you find — + closed loop, not a blind retry of a stale selector. + 2. **Assert the TestRail `expected` result** with `expect.soft(actual, 'message describing + which expected result this checks').toBe(...)` (or the matcher that fits — `toContain`, + `toBeVisible`, etc.). Use `expect.soft`, **not** `expect`, so one failed step does not + abort the rest of the case — every step still runs and appears in the `steps[]` array. + The assertion must check the specific observable outcome the `expected` field describes, + cross-checked with the spec's `Verification criteria` — **never** a tautology. +- **Escape interpolated TestRail text.** The stripped `content`/`expected` strings (3a) can + contain apostrophes, backticks, or `${...}`-looking substrings — any of these can break out + of a naive `'...'` JS string literal or get evaluated as a template expression. Use + double-quoted JS strings for step titles/assertion messages and escape embedded `"`, `\`, + and any `${` sequence before interpolating. A generated spec that fails to compile because of + an unescaped quote is a tooling bug, not a product defect — it must never be reported as FAIL + (see 3d-lint and 3e's malformed-output rule). + +Skeleton (fill in the grounded actions/assertions per case): + +```ts +// Tests/e2e/.testrail-tmp/case-$CASE_ID.spec.ts (throwaway — removed in 3f) +import { test, expect } from '@playwright/test'; +import { loginAsAdmin } from '../fixtures/auth'; +// import { SettingsPage } from '../pages/settings'; // include the POM(s) this case needs + +test('TR-$CASE_ID: $TITLE', async ({ page }) => { + await test.step('log in', async () => { + await loginAsAdmin(page); + }); + + await test.step('$STEP_1_CONTENT', async () => { + // ... drive the action via the spec's grounded locator / POM method ... + // assert against the TestRail EXPECTED RESULT (the oracle): + expect.soft(actual, 'checks: $STEP_1_EXPECTED').toBe(/* expected value */); + }); + + // ... one test.step() per remaining TestRail {content, expected} pair ... +}); +``` -Write the login fixture to a temp step file and run it: +### 3d-lint — Verify every step actually asserts (before running) -```js -// $STEPS/login.js (STEPS=/tmp/canary-steps/$id — per-session dir, no cross-run collision) -// E2E_URL is resolved per-case: $E2E_URL_APACHE for apache-required cases, $E2E_URL_NGINX otherwise -const page = await browser.getPage("main"); -await page.goto("$E2E_URL/wp-login.php", { waitUntil: "networkidle" }); -await page.getByLabel("Username or Email Address").fill("$WP_USER"); -await page.getByLabel("Password", { exact: true }).fill("$WP_PASS"); -await page.getByRole("button", { name: "Log In" }).click(); -await page.waitForURL("**/wp-admin/**", { timeout: 20000 }); -console.log("PASS: logged in"); -``` +Count `expect.soft(` occurrences in the generated file and compare to the number of TestRail +`{content, expected}` pairs for this case (the login step has none, so it is not counted). +If the count is lower, you generated at least one step with no real assertion — go back and +add the missing `expect.soft` before proceeding. **Do not run a spec that under-counts**: a +`test.step()` with no assertion never sets `error`, so it always reads as "passed" in 3e — +this is the one way a case can produce a false PASS, and it is checkable before wasting an +execution. -```bash -STEPS="/tmp/canary-steps/$id"; mkdir -p "$STEPS" # per-session; cleaned up in Step 3f -npx @usecanary/cli run --session "$id" --step login "$STEPS/login.js" --timeout 30 -``` +### 3d-exec — Execute the case (one Bash call) -### 3e — Run each TestRail step (closed-loop, grounded by the spec) - -For each `{content, expected}` pair (HTML-stripped), build a concrete Canary step script. The -`content` (action) tells you **what to do**; the loaded feature spec tells you **how** -(real `Locators` and `How to invoke` — not selectors guessed from the prose); the `expected` -field is the **assertion oracle** — you assert against what TestRail says is correct, never -against "the page looked ok". - -1. **Drive the action using the spec's grounded locators** (`getByRole`/`getByLabel`/ - `data-testid`/`id`, in that preference). If a grounded locator no longer matches the live - page (it drifted), re-observe the real element on the page and use what you find — closed - loop, not a blind retry of a stale selector. -2. **Assert the TestRail `expected` result**, not a tautology. The PASS condition must check - the specific observable outcome the `expected` field describes, cross-checked with the - spec's `Verification criteria`. - -```js -// ... drive the action via the spec's grounded locator ... -// assert against the TestRail EXPECTED RESULT for this step (the oracle): -if (/* the observable condition that the TestRail `expected` describes holds */) { - console.log("PASS: <expected result, confirmed>"); -} else { - console.log("FAIL: expected <TestRail expected> but observed <actual>"); -} -``` +Run the generated spec with the JSON reporter, passing the per-case env vars (`$E2E_URL` was +set in 3a-bis/Step 0's default, resolved to nginx/apache **before** this step — never resolve +it here). `TESTRAIL_OUTPUT_DIR` places +trace.zip/video/results.json directly at their final path — no relocation step. Every other +`.ai/...` path in this agent (Step 0's config, Step 4's dashboard, Step 6's trace links) is +**repo-root-relative** — so `$OUT` must be captured from repo-root `$(pwd)` **before** changing +into `Tests/e2e/` to invoke Playwright, not after. Run it as one self-contained block so the +directory change never leaks into the next Bash call: -Run it: ```bash -npx @usecanary/cli run --session "$id" --step "step-$N" "$STEPS/step-$N.js" --timeout 30 +OUT="$(pwd)/.ai/testrail/$RUN_ID/playwright/$CASE_ID" # captured at repo root, BEFORE the cd below +mkdir -p "$OUT" +( + cd Tests/e2e && + IMAGIFY_BASE_URL="$E2E_URL" IMAGIFY_ADMIN_USER="$WP_USER" IMAGIFY_ADMIN_PASS="$WP_PASS" \ + TESTRAIL_OUTPUT_DIR="$OUT" \ + npx playwright test --config=testrail.config.ts ".testrail-tmp/case-$CASE_ID.spec.ts" \ + --reporter=json > "$OUT/results.json" 2> "$OUT/stderr.log" +) ``` -If a step depends on an environment that is not available (multisite, a specific 3rd-party -plugin, an external service), do NOT force a FAIL — stop the case and mark it **BLOCKED** with -the reason (see Blocking detection). +(Reuse the exact invocation shape documented in `testrail.config.ts`'s header comment — now +kept in sync with this block. The `cd` is scoped inside a subshell `( ... )` specifically so it +cannot change your working directory for the *next* Bash call — every other step in this agent +assumes repo root. Do not split this into two Bash calls with a bare `cd` in the first one; do +it all in a single call as shown. **Never redirect stderr into `results.json` (`2>&1`)** — an +`npx`/npm deprecation warning, a "run `npx playwright install`" notice, or any Node warning +would land inside the file and corrupt the JSON that 3e parses. stderr goes to its own +`$OUT/stderr.log`, kept alongside the evidence for troubleshooting.) + +### 3e — Determine the outcome (parse the JSON reporter) + +First, confirm `$OUT/results.json` is valid JSON. If it is not (a tooling crash wrote +something other than the reporter's JSON, or the file is empty/truncated) — check +`$OUT/stderr.log` and mark the case **BLOCKED** with reason "Playwright tooling failure — see +stderr.log"; never let a parse failure fall through to FAIL or PASS by guessing. + +Otherwise, the one test result lives at `suites[].specs[].tests[].results[0]`, which carries a +top-level `status` (`"passed"`/`"failed"`/`"timedOut"`/`"skipped"`) and a `steps[]` array, each +entry `{title, error, ...}`. A step **failed** if its `error` field is truthy; steps after a +failed one still ran and still appear (that is the point of `expect.soft`). + +Map to an outcome: +- **PASS** — top-level `status === "passed"` and no **non-login** step has a truthy `error`. +- **FAIL** — any **non-login** step's `error` is truthy, **and** the "log in" step itself + succeeded. +- **BLOCKED** — any of: + - detected in 3b-bis before generating the spec (missing environment/prerequisite: + multisite, an unavailable 3rd-party plugin/service); + - the malformed-JSON case above; + - the **only** failed step is "log in" itself. `loginAsAdmin` throws hard on failure (bad + credentials, drifted login selectors, a down environment) — this is an + environment/credentials problem, not a product defect, and must never be reported as FAIL. + Reason: "login failed against $E2E_URL — check WP credentials/environment, not a product + defect." + +Steps-passed / steps-total **always exclude the "log in" step** — TestRail's own step count +never includes login, so the dashboard and posted comment must match it exactly. This is not a +per-case judgment call: exclude it every time, so counts are comparable across the run. + +Record, per case: +``` +{ case_id, title, outcome, trace_path, steps_passed, steps_total, elapsed, reason, dirty_state } +``` +where `trace_path` = `.ai/testrail/${RUN_ID}/playwright/$CASE_ID/trace.zip`; `elapsed` comes +from the JSON reporter (`suites[].specs[].tests[].results[0].duration`, in ms) converted with +`round(ms / 1000)`, minimum `1` — never round down to `0`; omit only when `duration` is +genuinely absent (e.g. BLOCKED before any test ran); `dirty_state` is `null` normally, or a +one-line note set in 3h when a teardown step fails for this case. -### 3f — End the session and relocate artifacts +### 3f — Clean up the scratch spec ```bash -npx @usecanary/cli session end "$id" -rm -rf "$STEPS" # per-session step scripts (STEPS=/tmp/canary-steps/$id) - -CANARY_DIR=".ai/testrail/${RUN_ID}/canary" -mkdir -p "$CANARY_DIR" -mv ~/.canary/sessions/"$id" "$CANARY_DIR/" -python3 -c " -import json; p='$CANARY_DIR/$id/session.json' -d=json.load(open(p)); d['artifactsDir']='$(pwd)/$CANARY_DIR/$id' -json.dump(d, open(p, 'w'), indent=2) -" -ln -sfn "$(pwd)/$CANARY_DIR/$id" ~/.canary/sessions/"$id" +rm -rf Tests/e2e/.testrail-tmp # remove the generated spec (mirrors the old `rm -rf "$STEPS"`) ``` -### 3g — Determine the outcome +Evidence is already in its final place under `.ai/testrail/$RUN_ID/playwright/$CASE_ID/` — no +artifact relocation needed (unlike the old Canary `mv ~/.canary/sessions/<id>` step). -Read `.ai/testrail/${RUN_ID}/canary/$id/results.json`: -- **PASS** — every step ran and none reported failure (`stepsFailed == 0`, all expected - conditions held, no `FAIL:` in step output). -- **FAIL** — any `stepsFailed > 0` or any step logged `FAIL:` / threw. -- **BLOCKED** — the case could not be executed because of a missing environment/prerequisite - (not a product defect). +### 3g — Publish the case's outcome to the live dashboard -Record, per case: -``` -{ case_id, title, session_id, outcome, trace_path, steps_passed, steps_total, elapsed, reason } -``` -where `trace_path` = `.ai/testrail/${RUN_ID}/canary/$id/trace.zip` and `elapsed` comes from `results.json`. +Immediately after determining this case's outcome (3e), update the live results Artifact +(Step 4) so a watcher sees progress case-by-case rather than only at the end. ### 3h — Teardown (LIFO — always, even on FAIL/BLOCKED) Unwind the teardown queue from 3b-bis in **reverse order**, via Bash (WP-CLI/REST, not the UI). Run this **regardless of the case outcome** — a failed or blocked case must still leave a clean state for case N+1, or one failure cascades into false failures downstream. If a -teardown step itself fails, log it and continue unwinding the rest; note it on the case so the -tester knows state may be dirty. +teardown step itself fails, log it, continue unwinding the rest, and set that case's +`dirty_state` field (3e's record) to a one-line note — so the tester (and case N+1's own +seeding step, 3b-bis) knows state may be dirty rather than the note being silently dropped. ### Blocking detection @@ -377,19 +475,40 @@ Mark a case BLOCKED (not FAILED) when it requires an environment that isn't pres multisite, a specific 3rd-party plugin not installed, an external/paid service, or any precondition the local fixture cannot satisfy. Always include a one-line `reason`. -## Step 4 — Print the results table - -After **all** cases have run: - -``` -| Case | Title | Outcome | Steps | Trace | -|--------|----------------------------|------------|-------|--------------------------------------------------| -| C14169 | Should correctly escape... | PASS | 5/5 | npx playwright show-trace .ai/testrail/1283/canary/.../trace.zip | -| C174 | Multisite settings | BLOCKED | 0/3 | (needs multisite env) | -| C201 | Bulk optimize 500 images | FAIL | 3/4 | npx playwright show-trace .ai/testrail/1283/canary/.../trace.zip | -``` - -Summarise totals (N passed / M failed / K blocked) below the table. +## Step 4 — Live results dashboard (HTML Artifact) + +Present results as a **live-updating HTML Artifact**, not a static markdown table. Build it +**incrementally**: publish/update it after each case's outcome is determined (called from +Step 3g), so whoever is watching sees progress case-by-case — not once at the very end. + +Mechanics (read the `Artifact` tool's own description for the exact call signature before your +first call — it requires loading the `artifact-design` skill first, via the `Skill` tool, to +calibrate how much styling this warrants; for this dashboard the answer is "not much"): +1. Write the dashboard HTML to a file (e.g. `.ai/testrail/$RUN_ID/dashboard.html`) as a content + fragment only — no `<!doctype>`, `<html>`, `<head>`, or `<body>` tags; the Artifact tool + wraps your content in that skeleton itself. +2. Call the `Artifact` tool with that file path (plus a one/two-emoji `favicon` and a short + `description`) to publish it. +3. After each subsequent case, rewrite the same file (with the new row / updated counts) and + call `Artifact` again with the **same path** to redeploy/update it in place. + +Keep the **title stable across redeploys within one run** (e.g. `TestRail Run #$RUN_ID — +Imagify`) and pick a stable favicon emoji (e.g. 🧪) so it updates in place rather than +spawning a new artifact each time. + +Content — the same information the old markdown table carried: +- Case ID, title +- Outcome as a colored badge: **PASS** (green), **FAIL** (red), **BLOCKED** (grey/amber) +- Steps passed / total (e.g. `5/5`) +- Trace: the view command `npx playwright show-trace .ai/testrail/$RUN_ID/playwright/$CASE_ID/trace.zip` + (for BLOCKED, show the block reason instead) + +A simple table with colored status badges is enough — do not over-engineer the styling. +Include a header line with running totals (N passed / M failed / K blocked / cases done / total). + +Regardless of the Artifact, **also print a plain-text summary line in chat** alongside the +Artifact link once all cases have run — `N passed / M failed / K blocked` — since not everyone +will open the dashboard. ## Step 5 — Ask before posting @@ -431,15 +550,16 @@ Payload: ```json { "status_id": 1, - "comment": "Automated Canary run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\n\nTrace: npx playwright show-trace .ai/testrail/$RUN_ID/canary/<id>/trace.zip", + "comment": "Automated Playwright run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\n\nTrace: npx playwright show-trace .ai/testrail/$RUN_ID/playwright/$CASE_ID/trace.zip", "elapsed": "45s" } ``` - `status_id`: **1 = PASS, 5 = FAIL, 2 = BLOCKED**. -- `comment`: a markdown summary — date, per-step results, and the trace command. -- `elapsed`: total session duration from `results.json`. Omit `elapsed` if it is zero or - unparseable (TestRail rejects `"0s"`). +- `comment`: a markdown summary — date, per-step results, and the trace command. If + `dirty_state` is set, append a one-line warning to the comment. +- `elapsed`: `Ns` derived in 3e (`round(duration_ms / 1000)`, minimum `1s`). Omit the field + only when `duration` was genuinely absent (BLOCKED before execution) — never send `"0s"`. Build the JSON with `python3` / a heredoc so newlines and unicode in the comment are escaped correctly. Print each posting result; if a POST fails, report which case failed and continue @@ -449,13 +569,15 @@ with the rest. After posting, print a final confirmation listing what was posted ## DO NOT -Anti-hallucination guards (these are AUTO-REJECT — fix the script, don't ship it): +Anti-hallucination guards (these are AUTO-REJECT — fix the generated spec, don't run it): - DO NOT use a selector you guessed from the step prose while the page is reachable — use the - loaded spec's grounded locator, or re-observe the real element. Inferred-locator → REJECT. -- DO NOT use an ephemeral/internal ref as a locator — use a stable one (`getByRole` preferred, - then `data-testid` / `id`). Ephemeral-ref-as-locator → REJECT. -- DO NOT write a tautological assertion (`assert true`, "page loaded"). The assertion must - check the TestRail **expected result**. Tautological-assertion → REJECT. + loaded spec's grounded locator (or the matching POM method), or re-observe the real element. + Inferred-locator → REJECT. +- DO NOT invent a locator when a stable one exists — prefer `getByRole`/`getByLabel`, then + `data-testid` / `id`, then the POM. Unstable/invented-locator → REJECT. +- DO NOT write a tautological assertion (`expect(true).toBe(true)`, "page loaded"). The + assertion must check the TestRail **expected result** via `expect.soft`. Tautological-assertion + → REJECT. - DO NOT execute a case with no grounded spec by guessing from prose — mark it BLOCKED and tell the tester to run `/testrail-setup`. - DO NOT ask the missing-spec question per case — resolve coverage once up front (Step 2b) @@ -468,7 +590,9 @@ Execution & posting rules: - DO NOT post any result to TestRail before the user confirms in Step 5. - DO NOT mark a case FAIL when it is actually BLOCKED by a missing environment — distinguish product defects from environment gaps. -- DO NOT spawn the canary-imagify-session-agent — read it for fixtures and drive Canary - yourself via Bash. +- DO NOT spawn `imagify-playwright-fixtures.md` — it is not a spawnable agent. Read it for + fixtures/selectors/POM reuse and drive Playwright yourself: generate a `.spec.ts` per case + and run `npx playwright test --config=testrail.config.ts`. +- DO NOT shell out to Canary (`npx @usecanary/cli`) — Canary is retired for this agent. - DO NOT print or log `$TESTRAIL_API_KEY`. - DO NOT post `elapsed` values of `"0s"` — omit the field instead. diff --git a/.claude/agents/testrail-scenario-agent.md b/.claude/agents/testrail-scenario-agent.md index 9141481a..879a161b 100644 --- a/.claude/agents/testrail-scenario-agent.md +++ b/.claude/agents/testrail-scenario-agent.md @@ -170,7 +170,7 @@ Draft test cases covering, where applicable: - **Plugin conflicts** — relevant 3rd-party plugins active (WooCommerce, WP Rocket). - **Regression guard** — confirm adjacent existing behaviour still works. -Each case must be concrete and executable by a human or by Canary. Use the **Step template** +Each case must be concrete and executable by a human or by Playwright. Use the **Step template** (`template_id: 2`) with discrete action/expected pairs. Flag `smoke_test: true` only for the critical-path cases that must pass on every build. @@ -181,7 +181,7 @@ These case types are consistently invalid and must never appear in the output: - **Unit / integration test runner cases** — any case whose `action` would be running `composer test-unit`, `phpunit`, `wp-env run`, or any CLI test command. If a behaviour is only verifiable by running the test suite, it belongs in the test suite, not in TestRail. - Ask instead: "can a human tester or Canary observe this outcome via the UI or a REST/MCP + Ask instead: "can a human tester or Playwright observe this outcome via the UI or a REST/MCP call?" If no, skip it. - **Source code inspection cases** — cases that require reading a PHP file, checking a @@ -264,7 +264,7 @@ API. You do NOT publish anything. Your primary job here is **removal and sharpening**, not addition. A lean file with 4 precise cases is better than a bloated file with 12. -If a file covers a PR that has no real functional change observable by a human or Canary +If a file covers a PR that has no real functional change observable by a human or Playwright (docblock fix, unit-test-only PR, CSS tweak), delete all its cases and leave the file empty with a `# skipped: no functional change` comment — or remove the file entirely and note it in your report. @@ -297,7 +297,7 @@ different input). If two cases differ only by a field value, collapse them into a note in `preconditions`. #### 4. Step clarity -Each step must be executable by a human tester or by Canary without guesswork: +Each step must be executable by a human tester or by Playwright without guesswork: - `action` must be a concrete UI action or API call, not a vague instruction like "use the feature". - `expected` must be a specific, observable outcome, not "it works" or "success". diff --git a/.claude/skills/orchestrator/SKILL.md b/.claude/skills/orchestrator/SKILL.md index 39fa6f0a..d5617a24 100644 --- a/.claude/skills/orchestrator/SKILL.md +++ b/.claude/skills/orchestrator/SKILL.md @@ -126,20 +126,6 @@ lean toward High autonomy even without an explicit signal. Record the calibration choice in the HTML log as the first ROUTING DECISION event so the user can see what mode you picked. -### Selecting the E2E mode - -After calibrating the escalation threshold, ask the user (or infer from their message): - -**E2E mode** — If the issue involves UI/browser changes, which QA agent to use? -- `playwright` (default) — `e2e-qa-tester`, stable, uses Playwright MCP -- `canary` — `canary-e2e`, experimental, records trace/video/HAR via Canary CLI - -Signals for `canary` mode in the user's message: "use canary", "record the session", "canary mode". -Default: `playwright` (no need to ask if the user hasn't mentioned it). - -Store as `e2e_mode` and pass it in the dispatch to `qa-engineer`. The flag is ignored when the -issue has no UI/browser changes. - --- ## Run log @@ -410,7 +396,6 @@ suggests low actual risk), confirm with the user before deciding. | `release-agent` | `haiku` | — | | `ticket-writer` | `haiku` | — | | `e2e-qa-tester` | `sonnet` | — | -| `canary-e2e` | `sonnet` | — | Pass the resolved model as the `model` parameter on every Agent tool spawn. For agents with frontmatter `model: haiku`, this is redundant but harmless — always pass it explicitly so the intent is clear in the orchestrator context. @@ -843,7 +828,7 @@ All agents also receive `CURRENT_MODEL` and `session_learnings` (section 13 of ` | `frontend-agent` | Issue object + spec path + dispatch plan + backend API contract (when scopes overlap) | | `release-agent` | Issue #, branch name, base branch, acceptance criteria, spec path | | `lead-reviewer` | PR URL + spec path (`.ai/issues/<N>/spec.md`) + acceptance criteria + `session_learnings` | -| `qa-engineer` | PR number + acceptance criteria + base branch + `e2e_mode` (`"playwright"` default, or `"canary"`) | +| `qa-engineer` | PR number + acceptance criteria + base branch | | `ticket-writer` (nth_followup) | Single NTH feedback item (not full context) | --- diff --git a/.claude/skills/testrail-run/SKILL.md b/.claude/skills/testrail-run/SKILL.md index c394387a..e8c733fd 100644 --- a/.claude/skills/testrail-run/SKILL.md +++ b/.claude/skills/testrail-run/SKILL.md @@ -1,15 +1,15 @@ --- name: testrail-run -description: Fetch a TestRail test run, execute every scenario via Canary, and optionally post results back to TestRail. +description: Fetch a TestRail test run, execute every scenario via Playwright, and optionally post results back to TestRail. --- # TestRail Run Entry point for executing a TestRail test run end-to-end. Resolves the target run, then spawns `testrail-run-agent`, which fetches every case in the run, executes each one via -Canary browser automation (sequentially), collects pass/fail/blocked outcomes with -trace/video evidence, prints a results table, and — only after the user confirms — posts the -results back to TestRail. +Playwright browser automation (sequentially — one generated `.spec.ts` per case), collects +pass/fail/blocked outcomes with trace/video evidence, publishes a live results dashboard, and +— only after the user confirms — posts the results back to TestRail. ## Invocation @@ -37,8 +37,8 @@ unless overridden by the user. 2. **Spawn `testrail-run-agent`** once, passing whichever of `{run-id, milestone name, or "active"}` was determined, and the `--cases` filter if provided. Instruct it to: - resolve and fetch the run's cases (then filter to `--cases` list if given), - - execute each case via Canary sequentially (never in parallel), - - print the results table, + - execute each case via Playwright sequentially (never in parallel), + - publish the live results dashboard, - **stop and ask for confirmation** before posting anything back to TestRail. 3. **If the agent stops with a coverage question** (Step 2b — one or more TestRail sections @@ -54,7 +54,7 @@ unless overridden by the user. those sections marked BLOCKED and execute the rest. - **select** → split the list per the user's answer and apply both branches above. -4. **Relay the agent's results table** to the user verbatim. +4. **Relay the agent's results dashboard link and summary** to the user verbatim. 5. **On the user's confirmation** ("yes" / "post them" / "select C123 C456"), re-engage the agent (or continue it) to post the chosen results to TestRail. Pass through any selection @@ -62,8 +62,8 @@ unless overridden by the user. ## Constraints -- Execution is always sequential, matching `workers: 1` in `playwright.config.ts`. +- Execution is always sequential, matching `workers: 1` in `Tests/e2e/testrail.config.ts`. - Never post results to TestRail without explicit user confirmation. - TestRail credentials (`TESTRAIL_USERNAME`, `TESTRAIL_API_KEY`) live in the environment; do not prompt for them and never print them. -- This skill never calls the TestRail API or Canary directly. All of that is the agent's. +- This skill never calls the TestRail API or Playwright directly. All of that is the agent's. diff --git a/.claude/skills/testrail-setup/SKILL.md b/.claude/skills/testrail-setup/SKILL.md index 5635a841..297f2d67 100644 --- a/.claude/skills/testrail-setup/SKILL.md +++ b/.claude/skills/testrail-setup/SKILL.md @@ -7,10 +7,10 @@ argument-hint: "[feature | --check | blank for all]" # TestRail Setup Entry point for building the **grounded maps** the run agent executes against. Spawns -`testrail-explorer-agent`, which drives the live app via Canary, captures real locators + -seed/teardown + verification criteria per plugin feature, and writes/refreshes the committed -specs under `.claude/testrail/specs/`. This skill is thin — it routes and relays; all the work -is the agent's. +`testrail-explorer-agent`, which drives the live app via the Playwright MCP tool, captures real +locators + seed/teardown + verification criteria per plugin feature, and writes/refreshes the +committed specs under `.claude/testrail/specs/`. This skill is thin — it routes and relays; all +the work is the agent's. ## Invocation @@ -44,8 +44,8 @@ is the agent's. ## Constraints -- This skill never drives Canary, edits specs, or audits code directly — all of that is the - agent's. It is an entry point, not a second copy of the agent. +- This skill never drives the browser, edits specs, or audits code directly — all of that is + the agent's. It is an entry point, not a second copy of the agent. - This skill never posts to TestRail. Setup only grounds specs; it never touches a run. - `--check` must never write — it is a report only. - Credentials live in the environment; do not prompt for them and never print them. diff --git a/.gitignore b/.gitignore index a565b863..68d7cfac 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,7 @@ composer.lock /Tests/e2e/test-results /.e2e-screenshots /Tests/e2e/.e2e-screenshots +/Tests/e2e/.testrail-tmp # Generated distribution packages /generatedpackages diff --git a/Tests/e2e/testrail.config.ts b/Tests/e2e/testrail.config.ts new file mode 100644 index 00000000..f4a4cf39 --- /dev/null +++ b/Tests/e2e/testrail.config.ts @@ -0,0 +1,63 @@ +import { defineConfig, devices } from '@playwright/test'; + +/** + * TestRail release-QA config — separate from `playwright.config.ts` (the permanent + * regression suite under `./specs`) because `testrail-run-agent` generates one throwaway + * `.spec.ts` per TestRail case into `./.testrail-tmp/` and must not mix them into the + * committed suite's testDir. + * + * The run agent invokes this per case, one `npx playwright test` call at a time. `$OUT` is + * captured as an ABSOLUTE path from repo root BEFORE `cd`ing into `Tests/e2e/` — every other + * `.ai/...` path the run agent uses (config, dashboard, trace links) is repo-root-relative, so + * this must not be a bare relative path once you're inside `Tests/e2e/`. stderr is kept out of + * `results.json` — any npx/npm/Node warning on stderr would otherwise corrupt the JSON the run + * agent parses for pass/fail: + * + * OUT="$(pwd)/.ai/testrail/$RUN_ID/playwright/$CASE_ID" # captured at repo root + * mkdir -p "$OUT" + * ( + * cd Tests/e2e && + * IMAGIFY_BASE_URL="$E2E_URL" IMAGIFY_ADMIN_USER="$WP_USER" IMAGIFY_ADMIN_PASS="$WP_PASS" \ + * TESTRAIL_OUTPUT_DIR="$OUT" \ + * npx playwright test --config=testrail.config.ts ".testrail-tmp/case-$CASE_ID.spec.ts" \ + * --reporter=json > "$OUT/results.json" 2> "$OUT/stderr.log" + * ) + * + * Evidence lands directly under TESTRAIL_OUTPUT_DIR — no post-hoc relocation step needed + * (unlike the old Canary flow, which had to `mv ~/.canary/sessions/<id>` after the fact). + * + * HAR capture is intentionally dropped: Playwright's `use.recordHar` needs a static path per + * test and doesn't compose cleanly with per-case dynamic naming. The trace.zip already + * contains the network tab in the trace viewer, which covers the same debugging need. + */ +export default defineConfig({ + testDir: './.testrail-tmp', + fullyParallel: false, // one case, one browser, at a time — same requirement as before + workers: 1, + retries: 0, // TestRail evidence is a single authoritative attempt, not a flaky-retry target + reporter: 'list', // the run agent always overrides with `--reporter=json` on the CLI + + outputDir: process.env.TESTRAIL_OUTPUT_DIR ?? './.testrail-tmp/artifacts', + + use: { + baseURL: process.env.IMAGIFY_BASE_URL ?? 'http://localhost:8888', + trace: 'on', // always-on, not retain-on-failure — TestRail wants evidence for PASS too + video: 'on', + screenshot: 'on', + viewport: { width: 1440, height: 900 }, + }, + + projects: [ + { + name: 'chromium', + use: { ...devices['Desktop Chrome'] }, + }, + ], + + expect: { + timeout: 10_000, + }, + // One test = one whole TestRail case (login + every step via test.step()), not one step — + // give it more headroom than the main suite's 60s. + timeout: 120_000, +}); From a0617b227e4193124c43ad87229dc326a894ee7d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Fri, 3 Jul 2026 02:15:31 +0200 Subject: [PATCH 13/16] feat(qa): translation doctrine, spec compile-cache, ephemeral Docker QA envs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - .claude/testrail/translation.md: prose→spec.ts thinking protocol the run agent must read before translating (oracle-first reading, step classification, three-source noun resolution, plan-before-code, failure interpretation hierarchy) - Spec compile-cache: generated case specs live in Tests/e2e/testrail-cases/ (committed corpus, @steps-hash header decides reuse vs re-translation); end-of-run user-gated promote/keep/delete with CI promotion shortlist - Ephemeral QA envs: bin/qa-env.sh + Tests/e2e/qa-env/ (Docker compose, one nginx + one apache WordPress, installs build-zip output, generates .ai/settings.local.json, DB snapshots for per-case reset); run agent provisions automatically when no reachable config exists - Live results ledger .ai/testrail/<run>/results.md updated per case (replaces per-case Artifact redeploys; crash/turn-budget resumable) - Audit fixes: trace/video resolved by glob (Playwright nests per-test dirs), apache_cases per-case frontmatter (schema + settings.md), empty-apache-path guard, merge-base ancestry drift checks, qa-plan contract closed in e2e-qa-tester, compile gate before execution, ability slugs corrected to the 7 that exist - Knowledge consolidation: imagify-playwright-fixtures.md folded into _foundation.md; hardcoded ports/creds replaced by config-driven $E2E_URL - Scenario agent: live section fetch + duplicate-section guard at publish; automation_hints captured from the PR diff into .claude/testrail/hints.yml Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- .claude/agents/e2e-qa-tester.md | 17 +- .claude/agents/imagify-playwright-fixtures.md | 112 --- .claude/agents/testrail-explorer-agent.md | 31 +- .claude/agents/testrail-run-agent.md | 766 +++++++----------- .claude/agents/testrail-scenario-agent.md | 59 +- .claude/skills/testrail-run/SKILL.md | 37 +- .claude/testrail/specs/_foundation.md | 229 ++++-- .claude/testrail/specs/mcp-abilities.md | 14 +- .claude/testrail/specs/mixpanel-tracking.md | 2 +- .claude/testrail/specs/settings.md | 6 +- .claude/testrail/translation.md | 177 ++++ .gitignore | 2 + Tests/e2e/qa-env/docker-compose.yml | 117 +++ Tests/e2e/qa-env/nginx.conf | 27 + Tests/e2e/testrail-cases/README.md | 29 + Tests/e2e/testrail.config.ts | 36 +- bin/qa-env.sh | 162 ++++ 17 files changed, 1101 insertions(+), 722 deletions(-) delete mode 100644 .claude/agents/imagify-playwright-fixtures.md create mode 100644 .claude/testrail/translation.md create mode 100644 Tests/e2e/qa-env/docker-compose.yml create mode 100644 Tests/e2e/qa-env/nginx.conf create mode 100644 Tests/e2e/testrail-cases/README.md create mode 100755 bin/qa-env.sh diff --git a/.claude/agents/e2e-qa-tester.md b/.claude/agents/e2e-qa-tester.md index 0c306869..776217f0 100644 --- a/.claude/agents/e2e-qa-tester.md +++ b/.claude/agents/e2e-qa-tester.md @@ -2,6 +2,7 @@ name: e2e-qa-tester description: Browser QA specialist for the Imagify WordPress plugin. Boots the local wp-env environment, drafts a Playwright spec directly from the PR's real diff and existing POMs, runs it via the Playwright Test runner, and falls back to a single targeted Playwright MCP snapshot only when a drafted locator or assertion doesn't hold. Specs are committed permanently under Tests/e2e/specs/ (E2E_CI=true). Screenshots are published via temporary branch commits and SHA-based raw.githubusercontent.com URLs. Invoked by qa-engineer for UI/browser changes. tools: [Bash, Read, Edit, Write, Glob, Grep, mcp__playwright, WebFetch] +model: sonnet maxTurns: 40 color: purple --- @@ -119,10 +120,18 @@ All variables (`{E2E_URL}`, `{E2E_BOOT}`, `{REPO}`, etc.) are already injected b ### Step 1 — Get context -1. Read the PR (`gh pr view <n>`) and especially its **"How to test"** section. That section is the executable spec. -2. Read the linked issue if there is one (`Fixes #N`). -3. Read every changed frontend file in full — not just the diff. -4. Read `Tests/e2e/pages/` for any existing POM methods relevant to the changed area. +1. **Read the QA plan first, if one was provided.** `qa-engineer` passes `QA_PLAN_PATH` + (usually `{TEMP_ROOT}/qa-plan.md`) — a P0/P1/P2-prioritized list of flows with entry + URLs, steps, and assertions. It is your checklist: every **P0** flow must be covered by + an assertion in the spec you draft (a P0 you cannot cover is a blocker, not a skip); + cover **P1** flows unless doing so requires an environment you don't have (document why); + **P2** flows are optional — cover them only when nearly free. Your Step 6 report must + state, per plan item, which criterion/assertion covers it. If no `QA_PLAN_PATH` was + provided, derive the flow list from the PR as below. +2. Read the PR (`gh pr view <n>`) and especially its **"How to test"** section. That section is the executable spec. +3. Read the linked issue if there is one (`Fixes #N`). +4. Read every changed frontend file in full — not just the diff. +5. Read `Tests/e2e/pages/` for any existing POM methods relevant to the changed area. #### Step 1b — Regression proof (required when the PR fixes a bug) diff --git a/.claude/agents/imagify-playwright-fixtures.md b/.claude/agents/imagify-playwright-fixtures.md deleted file mode 100644 index f045d2af..00000000 --- a/.claude/agents/imagify-playwright-fixtures.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -name: imagify-playwright-fixtures -description: Read-only fixture reference for Playwright-driven QA on the Imagify WordPress plugin — admin URLs, selectors, ability slugs, and MCP endpoint patterns, expressed as @playwright/test code. Not a spawnable agent — read and inlined by testrail-run-agent and testrail-explorer-agent. Supersedes canary-imagify-session-agent.md (Canary retired for Imagify QA). ---- - -# Imagify Playwright Fixtures (reference — not spawnable) - -This file is **not meant to be spawned as an agent.** Its callers (`testrail-run-agent`, -`testrail-explorer-agent`) inline the snippets below into the `.spec.ts` files they generate -or the `mcp__playwright` calls they drive. There is deliberately no `tools` or `model` -frontmatter: nothing here should ever execute independently. - -Reuse the real fixtures under `Tests/e2e/` wherever they already cover what you need — -don't reimplement: -- `Tests/e2e/fixtures/auth.ts` → `loginAsAdmin(page)` (idempotent login; reads - `IMAGIFY_ADMIN_USER` / `IMAGIFY_ADMIN_PASS` env vars, defaults `admin`/`password`) -- `Tests/e2e/fixtures/wp-cli.ts` → `wpCli()` / `runFromRepoRoot()` / `hasApiKey()` — these - shell out to `npx @wordpress/env run cli`, which targets the **wp-env CI stack**, NOT the - TestRail nginx/apache environments (those use `wp --path=$WP_PATH` directly against a - different install). Do not use `wpCli()` for TestRail seeding — use `wp --path=$WP_PATH` - as documented in `testrail-run-agent.md`. -- `Tests/e2e/pages/settings.ts` → `SettingsPage` -- `Tests/e2e/pages/bulk-optimization.ts` → `BulkOptimizationPage` -- `Tests/e2e/pages/media-library.ts` → `MediaLibraryPage` - -There is currently **no POM** for the Custom Folders / Files admin surface -(`/wp-admin/upload.php?page=imagify-files`) — write locators fresh for that surface (and -consider adding a POM to `Tests/e2e/pages/` if a spec starts depending on it repeatedly). - -## Imagify admin URLs (reference) - -| Purpose | URL | -|------------------|-------------------------------------------------------------| -| Settings | `/wp-admin/options-general.php?page=imagify` | -| Bulk optimize | `/wp-admin/upload.php?page=imagify-bulk-optimization` | -| Media library | `/wp-admin/upload.php?mode=list` | -| Custom folders | `/wp-admin/upload.php?page=imagify-files` | - -Endpoints: -- WP Abilities: `/wp-json/wp-abilities/v1/abilities` -- MCP server: `/wp-json/mcp/mcp-adapter-default-server` - -## Imagify selectors (named fixtures — use these, do not invent alternatives) - -``` -API key input: #imagify-api-key, [name="imagify_settings[api_key]"] -Save button: #submit -Success notice: .notice-success, .updated -Invalid API key: #imagify-check-api-container:not(.imagify-valid) ← NOT .notice-error -Bulk action btn: #imagify-bulk-action -Progress bar: .imagify-row-progress -Stats table: .imagify-bulk-table -Fatal error check: .wp-die-message, #error-page -Media column: th[id*="imagify"], th.column-imagify -``` - -These match `Tests/e2e/pages/settings.ts` verbatim where overlapping — if a POM exists for the -surface you're touching, use the POM method instead of the raw locator below. - -### FIXTURE: invalid API key is rejected - -Imagify does **not** surface an invalid API key via `.notice-error`. Assert against the -API-key container instead: - -```ts -await expect(page.locator('#imagify-check-api-container:not(.imagify-valid)')) - .toBeVisible({ timeout: 5000 }); -``` - -## Imagify ability slugs (all 10) - -``` -imagify/get-settings imagify/update-settings -imagify/get-account imagify/get-stats -imagify/get-media-status imagify/get-nextgen-coverage -imagify/optimize-media imagify/bulk-optimize -imagify/generate-missing-nextgen imagify/restore-media -``` - -### FIXTURE: assert all 10 expected ability slugs are registered - -```ts -const result = await page.evaluate(async () => { - const nonceResp = await fetch('/wp-admin/admin-ajax.php?action=rest-nonce', { credentials: 'same-origin' }); - const nonce = (await nonceResp.text()).trim(); - const resp = await fetch('/wp-json/wp-abilities/v1/abilities', { - headers: { 'X-WP-Nonce': nonce }, credentials: 'same-origin', - }); - return { status: resp.status, body: await resp.text() }; -}); -const data = JSON.parse(result.body); -const slugs = Array.isArray(data) ? data.map((a: any) => a.name) : []; -const expected = [ - 'imagify/get-settings', 'imagify/update-settings', 'imagify/get-account', - 'imagify/get-stats', 'imagify/get-media-status', 'imagify/get-nextgen-coverage', - 'imagify/optimize-media', 'imagify/bulk-optimize', 'imagify/generate-missing-nextgen', - 'imagify/restore-media', -]; -for (const s of expected) { - expect.soft(slugs, `ability slug missing: ${s}`).toContain(s); -} -expect.soft(result.status, 'abilities endpoint status').toBe(200); -expect.soft(result.body, 'no PHP fatal in abilities response').not.toContain('Fatal error'); -``` - -## DO NOT - -- Do NOT assert invalid API keys via `.notice-error` — use - `#imagify-check-api-container:not(.imagify-valid)`. -- Do NOT invent selectors — use the named ones above, or the matching `Tests/e2e/pages/` POM. -- Do NOT use `Tests/e2e/fixtures/wp-cli.ts`'s `wpCli()` for TestRail seeding — it targets the - wrong environment (wp-env, not the TestRail nginx/apache installs). diff --git a/.claude/agents/testrail-explorer-agent.md b/.claude/agents/testrail-explorer-agent.md index 2e10db19..8c627e94 100644 --- a/.claude/agents/testrail-explorer-agent.md +++ b/.claude/agents/testrail-explorer-agent.md @@ -36,12 +36,13 @@ maps back to TestRail via frontmatter `testrail_sections`. ## Environment & constants ``` -Base URL : http://localhost:10038 (wp-env) login admin / password via /wp-login.php - (password comes from IMAGIFY_ADMIN_PASS if set — see imagify-playwright-fixtures.md) +Base URL : read from .ai/settings.local.json → environments.nginx.url (NEVER hardcode a + port; if the config is missing, run `bash bin/qa-env.sh up` to provision the + envs and generate it). Login creds come from the same env block. Specs dir : .claude/testrail/specs/ -Foundation : .claude/testrail/specs/_foundation.md (load first; non-DOM shared knowledge) +Foundation : .claude/testrail/specs/_foundation.md (load first; shared knowledge — env + config shape, POM reuse, named selectors, auth/seeding/teardown helpers) Live driver : mcp__playwright (navigate/snapshot/click/fill against the live browser; one-off capture, no session/trace/video/HAR) -WP fixtures : .claude/agents/imagify-playwright-fixtures.md (login + named selectors to reuse) E2E POMs : Tests/e2e/pages/ (settings.ts, bulk-optimization.ts, media-library.ts — read before writing any locator, copy their style; this is the POM dir, NOT Tests/e2e/specs/, which holds test files, not reusable locator definitions) @@ -57,13 +58,15 @@ or the app is unreachable: stop and report **BLOCKED** with the reason — do no No browser, no writes. For each spec in `.claude/testrail/specs/`: ```bash -# read frontmatter source_files + derived_sha, then for each glob: -git log -1 --format=%H -- <source_file_glob> # latest commit touching that source +# read frontmatter source_files + derived_sha, then decide by ANCESTRY, not string compare: +DERIVED=$(git rev-parse "$DERIVED_SHA") # normalize short→full +LATEST=$(git log -1 --format=%H -- <source_file_glob>) +git merge-base --is-ancestor "$LATEST" "$DERIVED" && echo FRESH || echo STALE ``` -If any source file's latest commit is newer than the spec's `derived_sha`, mark the spec -**STALE** and name the files that changed. Print a table (spec, status FRESH/STALE, changed -files). **Write nothing.** Then stop. +FRESH when the latest source commit is an ancestor of — or equal to — the stamped SHA; +otherwise STALE — name the files that changed. Print a table (spec, status FRESH/STALE, +changed files). **Write nothing.** Then stop. --- @@ -76,11 +79,11 @@ the features (one spec each). Read 2–3 real files from `Tests/e2e/pages/` (the do not invent a foreign style. ### Step 2 — Drive the live app (`mcp__playwright`, logged in) -Use the login pattern and named selectors from `imagify-playwright-fixtures.md`. There is no +Use the login pattern and named selectors from `_foundation.md`. There is no session/trace/video/HAR concept here — this is a one-off, human-reviewable capture pass whose output is the spec file, so drive the browser directly with `mcp__playwright`: -1. **Navigate** to the feature's admin URL (from `imagify-playwright-fixtures.md`). +1. **Navigate** to the feature's admin URL (from `_foundation.md`'s admin-URL table). 2. **Log in** using the role/label-based pattern — e.g. `getByLabel("Username or Email Address")` and `getByLabel("Password")`, then submit. Reuse the `loginAsAdmin(page)` pattern the fixtures file documents. @@ -91,7 +94,7 @@ output is the spec file, so drive the browser directly with `mcp__playwright`: For API/MCP-only surfaces (e.g. abilities, REST endpoints), capture the real endpoint + invocation from a **live call** — drive a `page.evaluate` fetch (same pattern as the -ability-slug fixture in `imagify-playwright-fixtures.md`), not from prose. +ability-slug pattern in `_foundation.md`), not from prose. ### Step 3 — Light code/docs audit for non-DOM knowledge For each feature, determine from a quick read (not a deep audit): @@ -113,6 +116,8 @@ feature: "<Human feature name>" source_files: [<glob>, ...] # for drift detection derived_sha: <git SHA at capture time> last_explored: <YYYY-MM-DD> +apache_cases: [<case id>, ...] # OPTIONAL — case IDs that must run on the apache env + # (.htaccess / rewrite / $is_apache paths); omit when none --- ## Overview @@ -146,7 +151,7 @@ Stamp `derived_sha` from the current commit and `last_explored` with today's dat deterministically in Bash (do not hand-type the SHA): ```bash -SHA=$(git rev-parse --short HEAD) +SHA=$(git rev-parse HEAD) # FULL sha — the drift check normalizes via git rev-parse, but stamp full to be safe TODAY=$(date +%F) # patch the spec frontmatter derived_sha / last_explored with python3, not by re-typing the file ``` diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index 31264c18..717027f8 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -1,22 +1,29 @@ --- name: testrail-run-agent -description: Fetches all test cases from a TestRail run (by milestone or run ID), executes each via Playwright browser automation grounded by the committed feature specs, collects pass/fail outcomes with trace/video evidence, then (on user confirm) posts results back to TestRail. +description: Fetches all test cases from a TestRail run (by milestone or run ID), executes each via Playwright grounded by the committed feature specs — reusing cached case specs when the case is unchanged — records outcomes case-by-case in a results file, then (on user confirm) posts results back to TestRail and proposes CI promotion candidates. tools: [Bash, Read, Write, Glob, Grep, Artifact, Skill] model: sonnet -maxTurns: 200 +maxTurns: 300 color: orange --- # TestRail Run Agent You execute an entire TestRail test run for the Imagify WordPress plugin. You fetch every -case in the run, drive each one through Playwright browser automation (one generated -`.spec.ts` per case), capture evidence (trace / video / screenshots), determine an outcome -(PASS / FAIL / BLOCKED), present a live results dashboard, and — only after the user -confirms — post the results back to TestRail. +case in the run, execute each one via Playwright (one `.spec.ts` per case — reused from the +committed cache when the case is unchanged, translated fresh otherwise), capture evidence +(trace / video / screenshots), determine an outcome (PASS / FAIL / BLOCKED), keep a live +results file updated case-by-case, and — only after the user confirms — post the results +back to TestRail and propose which specs to promote into the permanent CI suite. -Execution is **strictly sequential**: one case at a time, one browser at a time. This matches -`workers: 1` in `Tests/e2e/testrail.config.ts`. Never run cases in parallel. +Execution is **strictly sequential**: one case at a time, one browser at a time (matches +`workers: 1` in `Tests/e2e/testrail.config.ts`). Never run cases in parallel. + +**Before translating your first case of a run, read `.claude/testrail/translation.md` in +full.** It is the thinking protocol for prose→spec translation — backwards oracle reading, +step classification, the three-source noun resolution, plan-before-code. Everything in this +file is mechanics; that file is how to reason. Do not skip it because the first case looks +simple. ## What you receive @@ -25,6 +32,8 @@ One of: - a **milestone name** (e.g. `2.3.0`), or - the token **`active`** meaning "use the active milestone's open run". +Optionally `--cases <comma-separated case IDs>` to execute a subset. + ## Environment & constants ``` @@ -32,194 +41,147 @@ TestRail base : https://wpmediaqa.testrail.io/index.php?/api/v2/ Auth : Basic — $TESTRAIL_USERNAME : $TESTRAIL_API_KEY Project ID : 3 Suite ID : 3 -Playwright : npx playwright test --config=Tests/e2e/testrail.config.ts (run from Tests/e2e/) -Evidence dir : .ai/testrail/$RUN_ID/playwright/$CASE_ID/ (trace.zip/video/results.json land here directly via TESTRAIL_OUTPUT_DIR — no relocation) -WP fixtures : .claude/agents/imagify-playwright-fixtures.md (read for WP login/selectors/POM reuse) -Specs dir : .claude/testrail/specs/ (committed grounding: _foundation.md + one file per feature) -Config file : .ai/settings.local.json (gitignored — URLs, WP creds, TestRail key) +Playwright : npx playwright test --config=testrail.config.ts (run from Tests/e2e/) +Spec cache : Tests/e2e/testrail-cases/case-$CASE_ID.spec.ts (committed corpus — NEVER auto-deleted) +Evidence dir : .ai/testrail/$RUN_ID/playwright/$CASE_ID/ (via TESTRAIL_OUTPUT_DIR) +Results file : .ai/testrail/$RUN_ID/results.md (updated after every case) +Doctrine : .claude/testrail/translation.md (read before first translation) +Specs dir : .claude/testrail/specs/ (committed grounding: _foundation.md + one file per feature) +Hints : .claude/testrail/hints.yml (per-case automation hints captured at scenario time, if present) +Config file : .ai/settings.local.json (gitignored — generated by bin/qa-env.sh, or hand-written for Local sites) +Env script : bin/qa-env.sh (up / reset / down / status — Docker ephemeral envs) ``` -### Loading config (Step 0 — always first) +### Step 0 — Environment: provision or verify (always first) + +Two supported setups. Decide which applies: + +1. **If `.ai/settings.local.json` exists**, read it and verify both URLs respond (HTTP 200/30x + on `/wp-login.php`). If they respond → use it (this covers hand-maintained Local sites and + an already-running qa-env). +2. **Otherwise, or if the URLs are dead** → provision the ephemeral Docker environments: + ```bash + bash bin/qa-env.sh up + ``` + This builds `imagify.zip` (`bin/build-zip.sh`), starts one nginx and one apache WordPress + via Docker Compose, installs + activates the zip on both, seeds the Imagify API key (from + `$IMAGIFY_TESTS_API_KEY` if set), **writes `.ai/settings.local.json`**, and snapshots both + databases. If Docker is unavailable and no working config exists, stop and report exactly + what is missing — do not improvise an environment. -Read `.ai/settings.local.json` and export shell variables from it: +Then load the config into shell variables: ```bash CONFIG=$(cat .ai/settings.local.json) +j() { echo "$CONFIG" | python3 -c "import sys,json,functools; d=json.load(sys.stdin); print(functools.reduce(lambda a,k: a.get(k,{}) if isinstance(a,dict) else '', '$1'.split('.'), d) or '')"; } -# TestRail auth — prefer env vars, fall back to config file -TESTRAIL_USERNAME="${TESTRAIL_USERNAME:-$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['testrail']['username'])")}" -TESTRAIL_API_KEY="${TESTRAIL_API_KEY:-$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['testrail']['api_key'])")}" - -# WP environments — apache fields default to '' rather than crashing if the config has no -# apache block (some local setups only run nginx); Step 0's E2E_URL default handles that below. -E2E_URL_NGINX=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['url'])") -E2E_URL_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('environments',{}).get('apache',{}).get('url',''))") -WP_USER=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['username'])") -WP_PASS=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['password'])") -WP_PATH_NGINX=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['environments']['nginx']['wp_path'])") -WP_PATH_APACHE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('environments',{}).get('apache',{}).get('wp_path',''))") - -# Default per-case URL — every case must set this explicitly before 3d-exec; never leave it -# carried over from a previous case. -E2E_URL="$E2E_URL_NGINX" +TESTRAIL_USERNAME="${TESTRAIL_USERNAME:-$(j testrail.username)}" +TESTRAIL_API_KEY="${TESTRAIL_API_KEY:-$(j testrail.api_key)}" + +E2E_URL_NGINX=$(j environments.nginx.url) +E2E_URL_APACHE=$(j environments.apache.url) # '' when no apache env exists +WP_USER_NGINX=$(j environments.nginx.username); WP_PASS_NGINX=$(j environments.nginx.password) +WP_USER_APACHE=$(j environments.apache.username); WP_PASS_APACHE=$(j environments.apache.password) ``` -For WP-CLI seeding commands, always pass `--path="$WP_PATH"` where `$WP_PATH` is -`$WP_PATH_APACHE` or `$WP_PATH_NGINX` matching the active env. Example: +**WP-CLI per environment:** each env block carries either `wp_cli` (a full command prefix — +what `bin/qa-env.sh` writes, e.g. `docker compose -f Tests/e2e/qa-env/docker-compose.yml run --rm cli-nginx wp`) +or `wp_path` (Local-style — build the prefix as `wp --path=<wp_path>`). Resolve once: + ```bash -wp --path="$WP_PATH" option get imagify_settings --format=json +WP_NGINX=$(j environments.nginx.wp_cli); [ -z "$WP_NGINX" ] && WP_NGINX="wp --path=$(j environments.nginx.wp_path)" +WP_APACHE=$(j environments.apache.wp_cli); [ -z "$WP_APACHE" ] && { P=$(j environments.apache.wp_path); [ -n "$P" ] && WP_APACHE="wp --path=$P"; } ``` -### Environment setup (Step 0b — after loading config, before fetching cases) +Use `$WP_NGINX` / `$WP_APACHE` for **all** seeding/teardown commands — never a bare `wp`. -For **each** environment (nginx + apache), ensure the plugin is active and the Imagify API -key is set. Do this via WP-CLI — no browser needed: +### Step 0b — Per-env setup (skip what qa-env.sh already did) + +If the config was just generated by `qa-env.sh up`, the plugin is already installed, active, +and keyed — skip to Step 1. For a hand-maintained (Local) config, ensure per env that the +plugin is active and the API key is set — **guarding against a missing apache env**: ```bash -IMAGIFY_API_KEY_VALUE=$(echo "$CONFIG" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['imagify']['api_key'])") - -for WP_PATH in "$WP_PATH_NGINX" "$WP_PATH_APACHE"; do - # Activate plugin (idempotent — safe to run even if already active) - wp --path="$WP_PATH" plugin activate imagify-plugin 2>/dev/null || \ - wp --path="$WP_PATH" plugin activate imagify 2>/dev/null || true - - # Seed Imagify API key into imagify_settings option - CURRENT=$(wp --path="$WP_PATH" option get imagify_settings --format=json 2>/dev/null || echo "{}") - UPDATED=$(echo "$CURRENT" | python3 -c " -import sys, json -d = json.load(sys.stdin) -d['api_key'] = '$IMAGIFY_API_KEY_VALUE' -print(json.dumps(d)) -") - wp --path="$WP_PATH" option update imagify_settings "$UPDATED" --format=json +for ENV in nginx apache; do + WPCMD=$([ "$ENV" = nginx ] && echo "$WP_NGINX" || echo "$WP_APACHE") + [ -n "$WPCMD" ] || continue # no apache env configured — skip, don't crash + $WPCMD plugin activate imagify-plugin 2>/dev/null || $WPCMD plugin activate imagify 2>/dev/null || true + # seed the API key into imagify_settings if a key is available + KEY=$(j imagify.api_key); [ -n "$KEY" ] && $WPCMD option patch update imagify_settings api_key "$KEY" done ``` -If a `wp plugin activate` call fails (plugin directory name differs), report the error and -stop — do not proceed with an inactive plugin. - -**Default env is Nginx.** At the start of each case, set `E2E_URL="$E2E_URL_NGINX"`. A case -requires Apache when the matched feature spec's frontmatter (loaded in 3a-bis, **before** you -generate anything) contains `server: apache` (or the case's TestRail preconditions mention -"Apache" or ".htaccess") — for those cases set `E2E_URL="$E2E_URL_APACHE"` instead, same -credentials, different URL. Because this is decided from the spec before 3c/3d ever run, there -is no "try Nginx, discover it needs Apache, retry" dance — the case never starts against the -wrong server. If `$E2E_URL_APACHE` is empty (no `apache` block in `.ai/settings.local.json`), -mark every Apache-required case **BLOCKED** with reason "no apache environment configured" — -do not run Playwright against an empty base URL. If a case is discovered *live* to need Apache -despite the spec not flagging it (spec drift), mark it BLOCKED with reason "requires apache, -not flagged in spec — update the spec's `server:` field, then re-run" rather than attempting an -automatic retry. - -Never print the API key. If a `curl` returns HTTP 401, report it as an auth/config problem -and stop. The TestRail MCP server (`/opt/homebrew/bin/mcp-testrail`) is connected but its -tool names are unconfirmed — use the REST API via `curl` as the primary approach; MCP may -replace these calls once tool names are confirmed. - -Known live data (verify, don't assume — IDs change between releases): -- Active milestone example: `id=182 name='2.3.0'`. -- Active run example: `id=1283 name='2.3.0'`. +If activation fails on an env you need, report the error and stop — never run cases against +an inactive plugin. Never print any API key. + +**Default env is nginx.** Per-case apache routing is decided in 3a-bis (below), before any +generation or execution — there is no "try nginx, discover apache, retry" dance. --- ## Step 1 — Resolve the run -**If given a run ID:** use it directly. Fetch run metadata for the name: -```bash -curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ - "https://wpmediaqa.testrail.io/index.php?/api/v2/get_run/$RUN_ID" -``` +**Run ID given:** use it; fetch `get_run/$RUN_ID` for the name. +**Milestone name or `active`:** `get_milestones/3&is_completed=0`; for `active`, if exactly +one open milestone, use it, else list and ask. Then `get_runs/3&milestone_id=$M&is_completed=0` +— one run → use it; several → list (id, name, untested count) and ask; none → report and stop. -**If given a milestone name or `active`:** -```bash -# Open milestones -curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ - "https://wpmediaqa.testrail.io/index.php?/api/v2/get_milestones/3&is_completed=0" -``` -- For a named milestone, find the milestone whose `name` matches. -- For `active`, if exactly one open milestone exists, use it; if several, list them and ask - which one. +Record `RUN_ID` and the run name. Create the run workspace and the results file skeleton: -Then fetch its open runs: ```bash -curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ - "https://wpmediaqa.testrail.io/index.php?/api/v2/get_runs/3&milestone_id=$MILESTONE_ID&is_completed=0" +mkdir -p ".ai/testrail/$RUN_ID/playwright" ``` -- If exactly one run, use it. -- If several, list them (id, name, untested count) and ask the user which run to execute. -- If none, report that the milestone has no open run and stop. -Record `RUN_ID` and the run name. +### Step 1b — Resume check -### Step 1b — Cheap drift check (warn, do not block) +If `.ai/testrail/$RUN_ID/results.md` already exists with recorded cases, this is a **resumed +run** (a previous invocation crashed or hit its turn budget). Parse the recorded case IDs and +**skip them** — execute only the remainder, appending to the same file. Say explicitly that +you are resuming and how many cases are already recorded. -Before executing, check whether the grounding specs are stale vs. the code they were captured -against. For each spec in `.claude/testrail/specs/`, read frontmatter `source_files` + -`derived_sha`; if any source file's latest commit (`git log -1 --format=%H -- <glob>`) is newer -than `derived_sha`, the spec is **stale**. **Warn** ("executing against stale grounding for -<feature> — locators may be behind the code; consider `/testrail-setup <feature>`") and -**continue** — do not block the run. This is a heads-up for the tester, not a gate. +### Step 1c — Drift check (warn, do not block) -## Step 2 — Fetch the run's tests +For each spec in `.claude/testrail/specs/`, read frontmatter `source_files` + `derived_sha`, +then determine staleness **by ancestry, not string comparison**: ```bash -curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ - "https://wpmediaqa.testrail.io/index.php?/api/v2/get_tests/$RUN_ID&limit=250" +DERIVED=$(git rev-parse "$DERIVED_SHA") # normalize short→full +LATEST=$(git log -1 --format=%H -- <source_file_glob>) +git merge-base --is-ancestor "$LATEST" "$DERIVED" && echo FRESH || echo STALE ``` -Paginate if needed: the response includes `_links.next`, a path relative to the API root — -prepend `https://wpmediaqa.testrail.io` before curling it. Follow it, accumulating tests -**deduplicated by `case_id`**, until `next` is null **or** you've followed 20 pages (5000 -tests — far more than any real run). If you hit that cap without `next` going null, stop, -warn that pagination did not terminate normally, and proceed with what you have rather than -looping forever. Each test carries `case_id`, `title`, `custom_preconds`, and -`custom_steps_separated` (a list of `{content, expected}`). +(FRESH when the latest source commit is an ancestor of — or equal to — the stamp.) For any +STALE spec, **warn** ("executing against stale grounding for <feature> — consider +`/testrail-setup <feature>`") and **continue**. Heads-up, not a gate. -By default, execute only tests whose status is **Untested** (`status_id == 3`). If the user -asked to re-run everything, include all. +## Step 2 — Fetch the run's tests -**`--cases` filter:** if a comma-separated list of case IDs was passed (e.g. `155,156,174,14169`), -keep only the tests whose `case_id` is in that list. Apply this filter **after** the status -filter. If a requested case ID is not found in the run, report it as skipped (not an error). +`get_tests/$RUN_ID&limit=250`. Paginate via `_links.next` (prepend the host), deduplicating +by `case_id`, hard cap 20 pages — if hit without `next` going null, warn and proceed with +what you have. Default filter: **Untested** (`status_id == 3`) unless the user asked to +re-run everything. Apply the `--cases` filter after the status filter; a requested ID absent +from the run is reported as skipped, not an error. To get each test's `section_id`, fetch +`get_cases/3&suite_id=3` **once** and join by `case_id` — do not call `get_case` per test. ## Step 2b — Coverage pre-check (ask once per missing category, not per case) -Before executing anything, resolve spec coverage for the **whole filtered test list up front**. -Never let the same missing-spec reason surface as N separate BLOCKED cases — surface it once, -as a decision, before any case runs. +Resolve spec coverage for the whole filtered list up front. For each unique `section_id`: -1. For each **unique** `section_id` among the filtered tests, run the same lookup as 3a-bis: - ```bash - MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |$)" .claude/testrail/specs/) - ``` -2. Bucket every section into `ok` (exactly one match), `missing` (zero matches), or `ambiguous` - (more than one match). Count the affected cases per section. -3. If `missing` or `ambiguous` is non-empty, **stop before running any case** and print one line - per affected section: - ``` - No grounded spec for: - - Section 8724 "MCP Abilities" — 12 cases - - Section 4976 "Media library" — 3 cases (ambiguous: matches 2 specs) - ``` - Then ask explicitly: - > Generate the missing spec(s) now (`/testrail-setup`), or mark these cases BLOCKED and - > continue with the rest of the run? (generate / block / select) - - **generate** → you cannot ground a spec yourself (grounding is the Explorer's job, not - yours — you execute committed specs, you do not author them). Stop here and hand back - to the orchestrator: state clearly which feature name(s) need `/testrail-setup <feature>` - (derive a feature slug from the section name, e.g. "Media library" → `media-library`), and - that you should be re-invoked on the same run/case selection once grounding is done. - - **block** → record every case in the missing/ambiguous sections as BLOCKED with the shared - reason (one message per *section*, not per case) and continue to Step 3 with the remaining, - already-grounded cases. - - **select** → ask which of the listed sections to generate vs. block, then apply per-section - as above. -4. Sections that resolved `ok` need no prompt — proceed straight to Step 3 for their cases. - -This pre-check does not replace 3a-bis's per-case resolution (a case can still turn out -ambiguous individually if TestRail data changes mid-run) — it just means that by the time -Step 3 runs, the user has already made an informed, one-time decision per category instead of -being surprised by a wall of identical BLOCKED rows. +```bash +MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |$)" .claude/testrail/specs/) +``` + +Bucket into `ok` (exactly one match) / `missing` (zero) / `ambiguous` (multiple). If missing +or ambiguous is non-empty, **stop before running any case**, print one line per affected +section with its case count, and ask: **generate / block / select**. +- **generate** → grounding is the Explorer's job, not yours. Hand back to the orchestrator + with the feature slug(s) to `/testrail-setup` (derive from the section name), and note you + should be re-invoked on the same run/case selection afterwards. +- **block** → record every case in those sections as BLOCKED with the shared reason (one + message per section) and continue with the grounded remainder. +- **select** → apply per-section as the user chooses. + +Sections that resolved `ok` proceed without any prompt. ## Step 3 — Execute each case (sequential) @@ -227,372 +189,242 @@ For each test, in order: ### 3a — Strip HTML from the steps -TestRail stores `content`, `expected`, and `custom_preconds` as HTML. Strip tags before using -them as `test.step()` titles and assertion messages: +`content`, `expected`, `custom_preconds` are HTML. Strip tags, unescape entities, collapse +whitespace; split `<li>`/`<br>`/`<p>` boundaries into separate lines when one block packs +several instructions. + +### 3a-bis — Resolve grounding and target environment + +1. Resolve the case's section to **exactly one** feature spec (same grep as 2b). Zero matches + at this point (user chose block, or TestRail changed mid-run) → BLOCKED, no re-asking. + Multiple → BLOCKED ("ambiguous") — never first-wins. +2. **Load `_foundation.md` first, then the matched spec.** They are the source of truth for + locators, invocation, seeding, verification criteria, and teardown. +3. **Pick the environment now:** apache when the matched spec's frontmatter lists this case in + `apache_cases: [...]`, **or** the case's TestRail preconditions mention Apache/.htaccess; + nginx otherwise. Set `E2E_URL`, `WP_USER`, `WP_PASS`, `WPCMD` from the chosen env's + variables. If apache is required but `$E2E_URL_APACHE` is empty → BLOCKED ("no apache + environment configured"). If a case is discovered *live* to need apache despite neither + flag (spec drift) → BLOCKED with "requires apache — add it to the spec's `apache_cases`, + then re-run", not an automatic retry. +4. If `.claude/testrail/hints.yml` has an entry for this `case_id` (entry URL, selectors seen + in the PR diff, suggested oracle — captured when the case was authored), load it. Hints + rank **below** POMs and grounded specs (doctrine rule 5) but above raw prose. + +### 3b — Cache check (reuse before you translate) + +The committed corpus lives at `Tests/e2e/testrail-cases/case-$CASE_ID.spec.ts`. Each file +starts with a machine-readable header: -```bash -STRIPPED=$(echo "$HTML" | python3 -c "import sys,re,html; t=re.sub('<[^>]+>',' ',sys.stdin.read()); print(html.unescape(re.sub(r'\s+',' ',t)).strip())") +```ts +// @testrail-case: 155 +// @steps-hash: <sha256 of the case's stripped steps+expected, first 12 hex chars> +// @grounding: <derived_sha of the feature spec used> +// @status: green 2026-07-03 ``` -(Collapse `<li>`/`<br>`/`<p>` boundaries into separate lines if the step packs several -instructions into one HTML block.) +Compute the hash of this case's stripped `{content, expected}` pairs: -### 3a-bis — Resolve the case to its feature spec (grounding) +```bash +HASH=$(python3 -c "import sys,hashlib; print(hashlib.sha256(sys.stdin.read().encode()).hexdigest()[:12])" <<< "$STRIPPED_STEPS") +``` -Resolve which committed spec grounds this case, then **load it** — this is the difference -between guessing and executing against reality. +- **File exists, `@steps-hash` matches, spec FRESH** → **reuse it, skip 3c/3d entirely.** + Translation only happens for new or changed cases — that is the point of the cache. +- Hash mismatch (the case was edited in TestRail) → re-translate (3c/3d), overwrite the file. +- Spec STALE but hash matches → reuse, but note in the results row that grounding was stale. +- Cached spec later fails at execution → treat as drift: one repair pass grounded on a live + re-observation, update the file. The cache is a starting point, not a straitjacket. -1. Get the case's TestRail section id (`get_case/$CASE_ID` → `section_id`, or the section is - already on the fetched test). -2. Find the feature spec whose frontmatter `testrail_sections` array **contains** that section - id (thin lookup; survives TestRail section reorg). Match a **whole array element**, not a - substring — `872` must not match `[8724]`, and `_foundation.md` (no `testrail_sections`) is - never a target. The element is delimited by `[`, `]`, `,`, or space: - ```bash - MATCHES=$(grep -rlE "testrail_sections:.*(\[|[ ,])$SECTION_ID([],]| |$)" .claude/testrail/specs/) - ``` -3. **Resolve to exactly one spec.** Zero or multiple matches here should be rare — Step 2b - already surfaced and decided every missing/ambiguous section before Step 3 started. If a - case still resolves to zero matches (the user chose **block** in 2b, or TestRail data - changed mid-run), mark it **BLOCKED** ("no grounded spec for section $SECTION_ID — run - `/testrail-setup`") without asking again. Multiple matches → also **BLOCKED** ("ambiguous: - section $SECTION_ID maps to multiple specs") — never first-wins. -4. **Always load `_foundation.md` first** (the shared base — it carries no `testrail_sections` - and is never a resolution target), **then the matched feature spec.** Use their sections as - the source of truth for this case: `Ground truth` (marks destructive operations — - load-bearing for seeding/teardown), `Locators`, `How to invoke`, `Prerequisites & seeding`, - `Verification criteria`, and `Teardown`. - -### 3b — Load WP fixture knowledge - -Read `.claude/agents/imagify-playwright-fixtures.md` for the WordPress login fixture, Imagify -admin URLs, named selectors, ability slugs, and — critically — its reuse guidance for the real -fixtures/POMs under `Tests/e2e/` (`fixtures/auth.ts` → `loginAsAdmin(page)`; `pages/settings.ts`, -`pages/bulk-optimization.ts`, `pages/media-library.ts`). You do NOT spawn that reference — it is -not a spawnable agent; you read it for its snippets and inline them into the `.spec.ts` you -generate. The loaded feature spec (3a-bis) takes precedence over fixture defaults where they -differ — the spec is the freshly-grounded truth. - -### 3b-bis — Seed prerequisites and register teardown (state isolation) - -Our 66 cases run sequentially in one session and several mutate state — case N must not -pollute case N+1. Establish a clean, known state **before** opening the browser. - -1. From the loaded spec's `Prerequisites & seeding`, run the WP-CLI/REST seed helpers - **via Bash, not the UI** (set/clear API key, force an attachment into a known state, toggle - a setting, snapshot settings before a mutating case). Faster and deterministic. -2. For every mutation you seed, **push its undo onto a LIFO teardown queue** (from the spec's - `Teardown` section — e.g. restore an attachment, re-apply a settings snapshot, delete a - seeded user). -3. If a required prerequisite cannot be seeded (missing env, helper unavailable), mark the - case **BLOCKED** with the reason — do not attempt the case in an unknown state. - -### 3c — Prepare the ephemeral spec scratch dir - -There is **no "start a session" step** in Playwright. Instead, ensure the per-case scratch dir -for the generated spec exists and is clean (this mirrors the old `/tmp/canary-steps/$id` + -`rm -rf` dance): +### 3c — Seed prerequisites and register teardown (state isolation) -```bash -mkdir -p "Tests/e2e/.testrail-tmp" -rm -f Tests/e2e/.testrail-tmp/*.spec.ts # no stale spec from a prior case -``` +Cases run sequentially in one session and several mutate state — case N must not pollute +case N+1. From the loaded spec's `Prerequisites & seeding`, seed via `$WPCMD`/REST (**Bash, +never the UI**). For every mutation seeded, push its undo onto a LIFO teardown queue (from +the spec's `Teardown` section). If a required prerequisite cannot be seeded → BLOCKED with +the reason — do not run the case in an unknown state. -`Tests/e2e/.testrail-tmp/` is `testDir` for `testrail.config.ts`. One case's spec lives here -at a time; it is removed in Step 3f. - -### 3d — Generate the case's spec file (one `test()` for the whole case) - -Write **one `.spec.ts` file per case** to `Tests/e2e/.testrail-tmp/case-$CASE_ID.spec.ts`, -containing **exactly one `test()`** that covers the whole case. Do NOT emit one `test()` per -TestRail step — a fresh `test()` gets a fresh page/context, which would drop the login and any -navigation state that later steps depend on. Instead, every TestRail step is a `test.step()` -inside the single `test()`, sharing one page. - -Rules for the generated spec: - -- **Imports.** From the case's location (`Tests/e2e/.testrail-tmp/`), the real fixtures/POMs - are one dir up: import `loginAsAdmin` from `../fixtures/auth`, and whichever POM(s) apply - from `../pages/` (e.g. `import { SettingsPage } from '../pages/settings'`), per - `imagify-playwright-fixtures.md`'s reuse guidance. `@playwright/test` is imported directly - (`import { test, expect } from '@playwright/test'`). -- **First `test.step()` is always login:** `await test.step('log in', async () => { await loginAsAdmin(page); });`. `loginAsAdmin(page)` reads `IMAGIFY_ADMIN_USER`/`IMAGIFY_ADMIN_PASS` - and `baseURL` from the config (set from `IMAGIFY_BASE_URL` in the invocation), so the per-case - env resolution (nginx vs apache) is handled entirely by the env vars you pass in 3d-exec — the - spec text itself is env-agnostic. -- **One `test.step(title, async () => { ... })` per TestRail `{content, expected}` pair** - (HTML-stripped, from 3a). Inside each step: - 1. **Drive the action using the spec's grounded locators** (POM method if a POM exists for - the surface, else `getByRole`/`getByLabel`/`data-testid`/`id` in that preference — from - 3a-bis's loaded feature spec, not guessed from prose). If a grounded locator no longer - matches the live page (it drifted), re-observe the real element and use what you find — - closed loop, not a blind retry of a stale selector. - 2. **Assert the TestRail `expected` result** with `expect.soft(actual, 'message describing - which expected result this checks').toBe(...)` (or the matcher that fits — `toContain`, - `toBeVisible`, etc.). Use `expect.soft`, **not** `expect`, so one failed step does not - abort the rest of the case — every step still runs and appears in the `steps[]` array. - The assertion must check the specific observable outcome the `expected` field describes, - cross-checked with the spec's `Verification criteria` — **never** a tautology. -- **Escape interpolated TestRail text.** The stripped `content`/`expected` strings (3a) can - contain apostrophes, backticks, or `${...}`-looking substrings — any of these can break out - of a naive `'...'` JS string literal or get evaluated as a template expression. Use - double-quoted JS strings for step titles/assertion messages and escape embedded `"`, `\`, - and any `${` sequence before interpolating. A generated spec that fails to compile because of - an unescaped quote is a tooling bug, not a product defect — it must never be reported as FAIL - (see 3d-lint and 3e's malformed-output rule). - -Skeleton (fill in the grounded actions/assertions per case): +If the previous case left `dirty_state` set (its teardown partially failed) and the envs are +qa-env-provisioned, reset to the post-setup snapshot before seeding: +`bash bin/qa-env.sh reset <nginx|apache>` — cheaper than debugging cascade failures. -```ts -// Tests/e2e/.testrail-tmp/case-$CASE_ID.spec.ts (throwaway — removed in 3f) -import { test, expect } from '@playwright/test'; -import { loginAsAdmin } from '../fixtures/auth'; -// import { SettingsPage } from '../pages/settings'; // include the POM(s) this case needs - -test('TR-$CASE_ID: $TITLE', async ({ page }) => { - await test.step('log in', async () => { - await loginAsAdmin(page); - }); - - await test.step('$STEP_1_CONTENT', async () => { - // ... drive the action via the spec's grounded locator / POM method ... - // assert against the TestRail EXPECTED RESULT (the oracle): - expect.soft(actual, 'checks: $STEP_1_EXPECTED').toBe(/* expected value */); - }); - - // ... one test.step() per remaining TestRail {content, expected} pair ... -}); -``` +### 3d — Translate: plan, then generate the spec + +Apply the doctrine (`.claude/testrail/translation.md`): read the case backwards, classify +every step, resolve every noun through POM → grounded spec → one targeted observation, write +the plan table, validate it, and only then emit +`Tests/e2e/testrail-cases/case-$CASE_ID.spec.ts` with the header from 3b and: -### 3d-lint — Verify every step actually asserts (before running) +- **One `test()` for the whole case**, one `test.step()` per TestRail `{content, expected}` + pair, login always the first step (`loginAsAdmin(page)` from `../fixtures/auth` — reads + `IMAGIFY_ADMIN_USER`/`IMAGIFY_ADMIN_PASS` + `baseURL`, so the spec text is env-agnostic). +- Imports are one dir up: `../fixtures/auth`, `../pages/<pom>`. +- `expect.soft` for every step assertion, message naming the expected result it checks. +- Escape interpolated TestRail text (quotes, backticks, `${`) — a compile failure from prose + is your bug, never a FAIL. -Count `expect.soft(` occurrences in the generated file and compare to the number of TestRail -`{content, expected}` pairs for this case (the login step has none, so it is not counted). -If the count is lower, you generated at least one step with no real assertion — go back and -add the missing `expect.soft` before proceeding. **Do not run a spec that under-counts**: a -`test.step()` with no assertion never sets `error`, so it always reads as "passed" in 3e — -this is the one way a case can produce a false PASS, and it is checkable before wasting an -execution. +### 3d-lint — Two gates before running -### 3d-exec — Execute the case (one Bash call) +1. **Assertion count:** `expect.soft(` occurrences ≥ the number of TestRail step pairs (login + excluded). An assertion-less step always reads as "passed" — the one way to produce a + false PASS, and it is checkable before execution. Under-count → fix before running. +2. **Compile gate:** `npx playwright test --config=testrail.config.ts --list "testrail-cases/case-$CASE_ID.spec.ts"` + (from `Tests/e2e/`). A listing error is a translation bug — fix it now; never let a + compile error surface as a case outcome. -Run the generated spec with the JSON reporter, passing the per-case env vars (`$E2E_URL` was -set in 3a-bis/Step 0's default, resolved to nginx/apache **before** this step — never resolve -it here). `TESTRAIL_OUTPUT_DIR` places -trace.zip/video/results.json directly at their final path — no relocation step. Every other -`.ai/...` path in this agent (Step 0's config, Step 4's dashboard, Step 6's trace links) is -**repo-root-relative** — so `$OUT` must be captured from repo-root `$(pwd)` **before** changing -into `Tests/e2e/` to invoke Playwright, not after. Run it as one self-contained block so the -directory change never leaks into the next Bash call: +### 3d-exec — Execute (one self-contained Bash call) ```bash -OUT="$(pwd)/.ai/testrail/$RUN_ID/playwright/$CASE_ID" # captured at repo root, BEFORE the cd below +OUT="$(pwd)/.ai/testrail/$RUN_ID/playwright/$CASE_ID" # ABSOLUTE, captured at repo root BEFORE the cd mkdir -p "$OUT" ( cd Tests/e2e && IMAGIFY_BASE_URL="$E2E_URL" IMAGIFY_ADMIN_USER="$WP_USER" IMAGIFY_ADMIN_PASS="$WP_PASS" \ TESTRAIL_OUTPUT_DIR="$OUT" \ - npx playwright test --config=testrail.config.ts ".testrail-tmp/case-$CASE_ID.spec.ts" \ + npx playwright test --config=testrail.config.ts "testrail-cases/case-$CASE_ID.spec.ts" \ --reporter=json > "$OUT/results.json" 2> "$OUT/stderr.log" ) ``` -(Reuse the exact invocation shape documented in `testrail.config.ts`'s header comment — now -kept in sync with this block. The `cd` is scoped inside a subshell `( ... )` specifically so it -cannot change your working directory for the *next* Bash call — every other step in this agent -assumes repo root. Do not split this into two Bash calls with a bare `cd` in the first one; do -it all in a single call as shown. **Never redirect stderr into `results.json` (`2>&1`)** — an -`npx`/npm deprecation warning, a "run `npx playwright install`" notice, or any Node warning -would land inside the file and corrupt the JSON that 3e parses. stderr goes to its own -`$OUT/stderr.log`, kept alongside the evidence for troubleshooting.) - -### 3e — Determine the outcome (parse the JSON reporter) - -First, confirm `$OUT/results.json` is valid JSON. If it is not (a tooling crash wrote -something other than the reporter's JSON, or the file is empty/truncated) — check -`$OUT/stderr.log` and mark the case **BLOCKED** with reason "Playwright tooling failure — see -stderr.log"; never let a parse failure fall through to FAIL or PASS by guessing. - -Otherwise, the one test result lives at `suites[].specs[].tests[].results[0]`, which carries a -top-level `status` (`"passed"`/`"failed"`/`"timedOut"`/`"skipped"`) and a `steps[]` array, each -entry `{title, error, ...}`. A step **failed** if its `error` field is truthy; steps after a -failed one still ran and still appear (that is the point of `expect.soft`). - -Map to an outcome: -- **PASS** — top-level `status === "passed"` and no **non-login** step has a truthy `error`. -- **FAIL** — any **non-login** step's `error` is truthy, **and** the "log in" step itself - succeeded. -- **BLOCKED** — any of: - - detected in 3b-bis before generating the spec (missing environment/prerequisite: - multisite, an unavailable 3rd-party plugin/service); - - the malformed-JSON case above; - - the **only** failed step is "log in" itself. `loginAsAdmin` throws hard on failure (bad - credentials, drifted login selectors, a down environment) — this is an - environment/credentials problem, not a product defect, and must never be reported as FAIL. - Reason: "login failed against $E2E_URL — check WP credentials/environment, not a product - defect." - -Steps-passed / steps-total **always exclude the "log in" step** — TestRail's own step count -never includes login, so the dashboard and posted comment must match it exactly. This is not a -per-case judgment call: exclude it every time, so counts are comparable across the run. - -Record, per case: -``` -{ case_id, title, outcome, trace_path, steps_passed, steps_total, elapsed, reason, dirty_state } -``` -where `trace_path` = `.ai/testrail/${RUN_ID}/playwright/$CASE_ID/trace.zip`; `elapsed` comes -from the JSON reporter (`suites[].specs[].tests[].results[0].duration`, in ms) converted with -`round(ms / 1000)`, minimum `1` — never round down to `0`; omit only when `duration` is -genuinely absent (e.g. BLOCKED before any test ran); `dirty_state` is `null` normally, or a -one-line note set in 3h when a teardown step fails for this case. +The subshell scopes the `cd` — every other step assumes repo root. **Never `2>&1` into +results.json** — any npm/Node warning would corrupt the JSON; stderr goes to its own file. + +### 3e — Determine the outcome + +First confirm `$OUT/results.json` is valid JSON; if not, check `stderr.log` and mark +**BLOCKED** ("Playwright tooling failure — see stderr.log") — never guess PASS/FAIL from a +parse failure. + +The test result is at `suites[].specs[].tests[].results[0]`: top-level `status` plus a +`steps[]` array where a step failed iff its `error` is truthy (later steps still ran — +that's `expect.soft`'s job). -### 3f — Clean up the scratch spec +- **PASS** — `status === "passed"` and no non-login step has a truthy `error`. +- **FAIL** — a non-login step's `error` is truthy **and** the login step succeeded. +- **BLOCKED** — pre-execution block (3c), malformed JSON, or the **only** failed step is + login (environment/credentials, never a product defect). + +Steps-passed / steps-total always **exclude login** — TestRail's own step counts do, and the +results file and posted comment must match them exactly. + +**Locate the evidence** — Playwright nests artifacts in a per-test subdirectory of +`outputDir`, so resolve the real paths by glob, never assume `$OUT/trace.zip`: ```bash -rm -rf Tests/e2e/.testrail-tmp # remove the generated spec (mirrors the old `rm -rf "$STEPS"`) +TRACE=$(ls "$OUT"/*/trace.zip 2>/dev/null | head -1) +VIDEO=$(ls "$OUT"/*/*.webm 2>/dev/null | head -1) ``` -Evidence is already in its final place under `.ai/testrail/$RUN_ID/playwright/$CASE_ID/` — no -artifact relocation needed (unlike the old Canary `mv ~/.canary/sessions/<id>` step). +Record per case: `{ case_id, title, outcome, trace_path, steps_passed, steps_total, elapsed, +reason, dirty_state, cache }` — `elapsed` from `results[0].duration` as `round(ms/1000)`, +minimum 1, omitted only when no test ran; `cache` is `reused | translated | repaired`; +`dirty_state` set in 3g on teardown failure. -### 3g — Publish the case's outcome to the live dashboard +Update the spec file's `@status` header line (`green <date>` on PASS; leave prior value +otherwise) — the cache's few-shot value depends on knowing which specs are proven. -Immediately after determining this case's outcome (3e), update the live results Artifact -(Step 4) so a watcher sees progress case-by-case rather than only at the end. +### 3f — Update the results file (after every case) -### 3h — Teardown (LIFO — always, even on FAIL/BLOCKED) +Rewrite `.ai/testrail/$RUN_ID/results.md` — running totals up top, one row per case: -Unwind the teardown queue from 3b-bis in **reverse order**, via Bash (WP-CLI/REST, not the -UI). Run this **regardless of the case outcome** — a failed or blocked case must still leave a -clean state for case N+1, or one failure cascades into false failures downstream. If a -teardown step itself fails, log it, continue unwinding the rest, and set that case's -`dirty_state` field (3e's record) to a one-line note — so the tester (and case N+1's own -seeding step, 3b-bis) knows state may be dirty rather than the note being silently dropped. +```markdown +# TestRail Run #$RUN_ID — <run name> (updated <timestamp>) +**N passed / M failed / K blocked — D of T done** -### Blocking detection +| Case | Title | Outcome | Steps | Spec | Note | +|---|---|---|---|---|---| +| C155 | Settings — auto-optimize ON | PASS | 4/4 | reused | trace: <path> | +| C173 | ... | BLOCKED | — | — | no apache environment configured | +``` -Mark a case BLOCKED (not FAILED) when it requires an environment that isn't present: -multisite, a specific 3rd-party plugin not installed, an external/paid service, or any -precondition the local fixture cannot satisfy. Always include a one-line `reason`. - -## Step 4 — Live results dashboard (HTML Artifact) - -Present results as a **live-updating HTML Artifact**, not a static markdown table. Build it -**incrementally**: publish/update it after each case's outcome is determined (called from -Step 3g), so whoever is watching sees progress case-by-case — not once at the very end. - -Mechanics (read the `Artifact` tool's own description for the exact call signature before your -first call — it requires loading the `artifact-design` skill first, via the `Skill` tool, to -calibrate how much styling this warrants; for this dashboard the answer is "not much"): -1. Write the dashboard HTML to a file (e.g. `.ai/testrail/$RUN_ID/dashboard.html`) as a content - fragment only — no `<!doctype>`, `<html>`, `<head>`, or `<body>` tags; the Artifact tool - wraps your content in that skeleton itself. -2. Call the `Artifact` tool with that file path (plus a one/two-emoji `favicon` and a short - `description`) to publish it. -3. After each subsequent case, rewrite the same file (with the new row / updated counts) and - call `Artifact` again with the **same path** to redeploy/update it in place. - -Keep the **title stable across redeploys within one run** (e.g. `TestRail Run #$RUN_ID — -Imagify`) and pick a stable favicon emoji (e.g. 🧪) so it updates in place rather than -spawning a new artifact each time. - -Content — the same information the old markdown table carried: -- Case ID, title -- Outcome as a colored badge: **PASS** (green), **FAIL** (red), **BLOCKED** (grey/amber) -- Steps passed / total (e.g. `5/5`) -- Trace: the view command `npx playwright show-trace .ai/testrail/$RUN_ID/playwright/$CASE_ID/trace.zip` - (for BLOCKED, show the block reason instead) - -A simple table with colored status badges is enough — do not over-engineer the styling. -Include a header line with running totals (N passed / M failed / K blocked / cases done / total). - -Regardless of the Artifact, **also print a plain-text summary line in chat** alongside the -Artifact link once all cases have run — `N passed / M failed / K blocked` — since not everyone -will open the dashboard. +This file is the live dashboard and the crash-resume ledger (Step 1b) — it must be written +after **every** case, not at the end. Do not redeploy an HTML Artifact per case; optionally +publish one Artifact **once, at end of run**, from this file's content. -## Step 5 — Ask before posting +### 3g — Teardown (LIFO — always, even on FAIL/BLOCKED) -Present the results table, then ask explicitly: +Unwind the teardown queue in reverse order via `$WPCMD`/REST. Run regardless of outcome — a +failed case must still leave clean state, or one failure cascades. If a teardown step fails: +log it, continue unwinding, set the case's `dirty_state` note (3c's next-case reset handles +the rest). -> Post these results to TestRail run #<RUN_ID>? (yes / no / select) +### Blocking detection -- **yes** → post all executed cases. -- **no** → stop; post nothing. -- **select** → ask which case IDs; post only those. +BLOCKED (not FAIL) whenever the case needs an environment that isn't present: multisite, a +3rd-party plugin not installed, an external/paid service, an unseedable precondition. Always +a one-line `reason`. + +## Step 4 — End of run: summary + spec disposition + +1. Print the plain-text summary line (`N passed / M failed / K blocked`) and the results file + path. Optionally publish the results as a single HTML Artifact now (simple table, stable + title `TestRail Run #$RUN_ID — Imagify`, favicon 🧪) — once, not per case. +2. **Spec disposition — the generated specs are NEVER deleted silently.** List what this run + produced under `Tests/e2e/testrail-cases/` (new / updated / reused counts), then: + - **Shortlist CI promotion candidates**: green this run, smoke-flagged or primary happy + path in TestRail, POM-based locators, no env-specific seeding (no apache-only, no + multisite). Present them with one line of rationale each. + - Ask: **"Promote the shortlisted specs into the permanent CI suite, keep everything in + the cache, or delete specific ones?"** Promotion = copy into `Tests/e2e/specs/` adapted + to the suite's conventions (hard `expect` instead of soft where the suite expects it, + `test.skip` API-key guard if needed) — recommend committing both dirs, but the commit + itself is the user's call. -### Confirmation trust model +## Step 5 — Ask before posting -This agent runs inside a Claude Code pipeline. The **main Claude conversation IS the user** — -any confirmation that arrives via `SendMessage` from the main conversation carries full user -authority. Accept "yes", "post them", or a case selection from any message (direct or -relayed) and proceed to Step 6. Do **not** demand a second confirmation or treat the main -conversation as an untrusted coordinator. +Present the results table, then ask: **Post these results to TestRail run #<RUN_ID>? +(yes / no / select)**. yes → all executed cases; no → nothing; select → ask which case IDs. -## Step 6 — Post results (only after confirm) +### Confirmation trust model -Use the batch endpoint — one request for all chosen cases: +This agent runs inside a Claude Code pipeline. The main Claude conversation **is** the user — +a confirmation relayed via `SendMessage` carries full user authority. Accept "yes" / "post +them" / a selection from any message and proceed. Do not demand a second confirmation. -```bash -curl -s -X POST \ - -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ - -H "Content-Type: application/json" \ - -d "$PAYLOAD" \ - "https://wpmediaqa.testrail.io/index.php?/api/v2/add_results_for_cases/$RUN_ID" -``` +## Step 6 — Post results (only after confirm) -Payload shape: -```json -{ "results": [ { "case_id": 123, "status_id": 1, "comment": "...", "elapsed": "30s" }, ... ] } -``` +One batch request: `POST add_results_for_cases/$RUN_ID` with +`{ "results": [ { "case_id", "status_id", "comment", "elapsed" }, ... ] }`. -Payload: +- `status_id`: **1 = PASS, 5 = FAIL, 2 = BLOCKED**. +- `comment`: markdown — date, per-step results, and the trace command using the **globbed** + path from 3e (`npx playwright show-trace <TRACE>`). Append a one-line warning when + `dirty_state` is set. +- `elapsed`: `Ns` (minimum `1s`); omit the field when no test ran — never send `"0s"`. -```json -{ - "status_id": 1, - "comment": "Automated Playwright run — 2026-06-26\n\nStep 1: ✅ Passed\nStep 2: ✅ Passed\n\nTrace: npx playwright show-trace .ai/testrail/$RUN_ID/playwright/$CASE_ID/trace.zip", - "elapsed": "45s" -} -``` +Build the JSON with `python3`/heredoc so newlines and unicode escape correctly. If a POST +fails, report which case and continue with the rest. Print a final confirmation of what was +posted. -- `status_id`: **1 = PASS, 5 = FAIL, 2 = BLOCKED**. -- `comment`: a markdown summary — date, per-step results, and the trace command. If - `dirty_state` is set, append a one-line warning to the comment. -- `elapsed`: `Ns` derived in 3e (`round(duration_ms / 1000)`, minimum `1s`). Omit the field - only when `duration` was genuinely absent (BLOCKED before execution) — never send `"0s"`. +## Turn budget -Build the JSON with `python3` / a heredoc so newlines and unicode in the comment are escaped -correctly. Print each posting result; if a POST fails, report which case failed and continue -with the rest. After posting, print a final confirmation listing what was posted. +You have `maxTurns: 300`. The cache is what makes a 60+-case run fit — reused cases cost ~3 +turns, translated ones ~8. If you judge the remaining budget will not cover the remaining +cases: finish the current case, write the results file, and stop cleanly with a message +saying the run is resumable (Step 1b) — a clean partial ledger beats an opaque death +mid-case. --- ## DO NOT -Anti-hallucination guards (these are AUTO-REJECT — fix the generated spec, don't run it): -- DO NOT use a selector you guessed from the step prose while the page is reachable — use the - loaded spec's grounded locator (or the matching POM method), or re-observe the real element. - Inferred-locator → REJECT. -- DO NOT invent a locator when a stable one exists — prefer `getByRole`/`getByLabel`, then - `data-testid` / `id`, then the POM. Unstable/invented-locator → REJECT. -- DO NOT write a tautological assertion (`expect(true).toBe(true)`, "page loaded"). The - assertion must check the TestRail **expected result** via `expect.soft`. Tautological-assertion - → REJECT. -- DO NOT execute a case with no grounded spec by guessing from prose — mark it BLOCKED and - tell the tester to run `/testrail-setup`. -- DO NOT ask the missing-spec question per case — resolve coverage once up front (Step 2b) - and ask once per missing/ambiguous *section*, not once per case. -- DO NOT ground a spec yourself — that is the Explorer's job. If the user wants one generated, - stop and hand back to the orchestrator with the feature name(s) to ground. - -Execution & posting rules: +Anti-hallucination guards (AUTO-REJECT — fix the plan/spec, don't run it): +- DO NOT use a selector guessed from prose while the page is reachable — POM, then grounded + locator, then one live observation (doctrine rule 3). Inferred-locator → REJECT. +- DO NOT write a tautological assertion — every step asserts its TestRail expected result via + `expect.soft` (doctrine rule 4). Tautology → REJECT. +- DO NOT execute a case with no grounded spec by guessing — BLOCKED, point to `/testrail-setup`. +- DO NOT ask the missing-spec question per case — once per section, up front (Step 2b). +- DO NOT ground a spec yourself — that is the Explorer's job. + +Lifecycle & posting rules: +- DO NOT delete anything under `Tests/e2e/testrail-cases/` without the user explicitly + choosing deletion in Step 4 — the cache is a committed corpus, not scratch space. - DO NOT run cases in parallel. One case, one browser, at a time. - DO NOT post any result to TestRail before the user confirms in Step 5. -- DO NOT mark a case FAIL when it is actually BLOCKED by a missing environment — distinguish - product defects from environment gaps. -- DO NOT spawn `imagify-playwright-fixtures.md` — it is not a spawnable agent. Read it for - fixtures/selectors/POM reuse and drive Playwright yourself: generate a `.spec.ts` per case - and run `npx playwright test --config=testrail.config.ts`. -- DO NOT shell out to Canary (`npx @usecanary/cli`) — Canary is retired for this agent. -- DO NOT print or log `$TESTRAIL_API_KEY`. +- DO NOT mark FAIL what is actually BLOCKED by a missing environment; and never report FAIL + without a live-confirmed defect (doctrine rule 9). +- DO NOT redeploy the dashboard Artifact per case — the results file is the live view. +- DO NOT seed or tear down through the UI — `$WPCMD`/REST only, and never a bare `wp`. +- DO NOT print or log `$TESTRAIL_API_KEY` or any API key. - DO NOT post `elapsed` values of `"0s"` — omit the field instead. diff --git a/.claude/agents/testrail-scenario-agent.md b/.claude/agents/testrail-scenario-agent.md index 879a161b..bf61b95e 100644 --- a/.claude/agents/testrail-scenario-agent.md +++ b/.claude/agents/testrail-scenario-agent.md @@ -1,7 +1,8 @@ --- name: testrail-scenario-agent -description: Analyses GitHub PRs and generates TestRail test scenarios, reviews staged scenarios for pertinence/coverage/redundancy/clarity, and publishes approved cases to TestRail via REST API with deduplication via refs field. +description: Analyses GitHub PRs and generates TestRail test scenarios, reviews staged scenarios for pertinence/coverage/redundancy/clarity, and publishes approved cases to TestRail via REST API with deduplication via refs field. Captures per-case automation hints (entry URL, diff-observed selectors, oracle) into .claude/testrail/hints.yml at publish time for the run agent's Playwright translation. tools: [Bash, Read, Write, Glob, Grep, WebFetch] +model: sonnet maxTurns: 40 color: blue --- @@ -85,9 +86,19 @@ stop; do not retry blindly. ### TestRail section map (suite 3) -Use this table to map each PR/case to a section. When content introduces a feature area that -has no fitting existing section, set `new_section` in the YAML and pick the most appropriate -**parent** from this table for it. +Use this table as a **heuristic starting point** — it goes stale as sections are created +(e.g. `MCP Abilities (8724)` under API Requests and `Mixpanel Tracking (8725)` were created +after this map was first written). Before deciding any `new_section`, fetch the live tree +once and check whether a fitting section already exists: + +```bash +curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/get_sections/3&suite_id=3&limit=250" +``` + +When content introduces a feature area that has no fitting existing section (per the LIVE +tree, not just this map), set `new_section` in the YAML and pick the most appropriate +**parent** for it. ``` Regression (33) @@ -236,6 +247,10 @@ cases: expected: "Connection succeeds." - action: "Call the discover-abilities tool." expected: "Returns all available abilities." + automation_hints: + entry: "/wp-json/wp-abilities/v1/abilities" + selectors: [] # selectors/ids you SAW in the PR diff, with file refs + oracle: "response array contains all 7 imagify/* slugs; anonymous call returns 401" ``` Rules for the YAML: @@ -244,6 +259,12 @@ Rules for the YAML: - `preconditions` is a YAML block scalar (`|`) of human-readable lines. - `steps` is a list of `{action, expected}` pairs. - Keep titles short, prefixed with the feature area (e.g. `MCP - `, `Bulk - `, `Settings - `). +- `automation_hints` is where you pay the translation gap forward: **you are reading the PR + diff right now — the exact markup, selectors, endpoints, and option keys the case will need + are in your context**, and the run agent that later translates this case to a Playwright + spec will not have them. Capture: the entry URL/endpoint, any selector or element id you + actually saw in the diff (with a `file:line` ref), and the concrete observable oracle. + Only record what you *saw* — never invent a selector; an empty `selectors: []` is fine. ### Step 6 — Print summary and stop @@ -408,7 +429,17 @@ files have not been through review yet, then ask whether to proceed anyway. ### Step A — Create the section if needed -If `new_section` is set (not null): +If `new_section` is set (not null), **first check it doesn't already exist** — TestRail +happily creates duplicate-named sections, and a previous publish (or a QA engineer) may have +created it already: + +```bash +EXISTING_ID=$(curl -s -u "$TESTRAIL_USERNAME:$TESTRAIL_API_KEY" \ + "https://wpmediaqa.testrail.io/index.php?/api/v2/get_sections/3&suite_id=3&limit=250" \ + | python3 -c "import sys,json; d=json.load(sys.stdin); secs=d.get('sections',d if isinstance(d,list) else []); print(next((s['id'] for s in secs if s['name']=='$NEW_SECTION' and s.get('parent_id')==$PARENT_SECTION_ID), ''))") +``` + +If `EXISTING_ID` is non-empty, use it as the target and skip creation. Otherwise create: ```bash SECTION_RESP=$(curl -s -X POST \ @@ -465,6 +496,24 @@ Field mapping from YAML → payload: Build the JSON with `python3 -c` / a heredoc rather than hand-concatenation, so quotes and newlines in step text are escaped correctly. Capture and print each created case `id`. +### Step B-bis — Persist automation hints + +TestRail has no field for `automation_hints`, so after each case is created, append its hints +(if any) to the committed hints ledger `.claude/testrail/hints.yml`, keyed by the **created +case ID**: + +```yaml +26359: + pr: 1133 + entry: "/wp-json/wp-abilities/v1/abilities" + selectors: [] + oracle: "response array contains all 7 imagify/* slugs; anonymous call returns 401" +``` + +Create the file if missing; never overwrite existing keys (a re-published case gets a new ID +anyway). The run agent reads this ledger when translating the case to a Playwright spec — +this is the moment-of-maximum-knowledge being handed forward. + ### Step C — Report and clean up After all cases for a file succeed, print the created case IDs grouped by PR, then delete the diff --git a/.claude/skills/testrail-run/SKILL.md b/.claude/skills/testrail-run/SKILL.md index e8c733fd..84322bc0 100644 --- a/.claude/skills/testrail-run/SKILL.md +++ b/.claude/skills/testrail-run/SKILL.md @@ -6,10 +6,16 @@ description: Fetch a TestRail test run, execute every scenario via Playwright, a # TestRail Run Entry point for executing a TestRail test run end-to-end. Resolves the target run, then -spawns `testrail-run-agent`, which fetches every case in the run, executes each one via -Playwright browser automation (sequentially — one generated `.spec.ts` per case), collects -pass/fail/blocked outcomes with trace/video evidence, publishes a live results dashboard, and -— only after the user confirms — posts the results back to TestRail. +spawns `testrail-run-agent`, which provisions (or verifies) the QA environments, fetches +every case in the run, executes each one via Playwright (sequentially — one `.spec.ts` per +case, **reused from the committed cache** `Tests/e2e/testrail-cases/` when the case is +unchanged, translated fresh otherwise), keeps `.ai/testrail/<run>/results.md` updated +case-by-case, and — only after the user confirms — posts the results back to TestRail and +proposes CI promotion candidates. + +The environments are ephemeral Docker containers managed by `bin/qa-env.sh` (one nginx, one +apache — built from `bin/build-zip.sh`'s `imagify.zip`), unless a hand-maintained +`.ai/settings.local.json` already points at reachable Local sites. ## Invocation @@ -54,16 +60,29 @@ unless overridden by the user. those sections marked BLOCKED and execute the rest. - **select** → split the list per the user's answer and apply both branches above. -4. **Relay the agent's results dashboard link and summary** to the user verbatim. +4. **Relay the agent's summary and the results file path** + (`.ai/testrail/<run>/results.md`) to the user verbatim. + +5. **Relay the spec-disposition question.** At end of run the agent lists what it produced + under `Tests/e2e/testrail-cases/` and shortlists CI promotion candidates. Relay the + shortlist and the question (promote / keep all / delete specific) verbatim — the specs + are never deleted without the user choosing so. -5. **On the user's confirmation** ("yes" / "post them" / "select C123 C456"), re-engage the +6. **On the user's confirmation** ("yes" / "post them" / "select C123 C456"), re-engage the agent (or continue it) to post the chosen results to TestRail. Pass through any selection the user makes. +7. **If the agent stopped on its turn budget**, it wrote a partial results file — tell the + user the run is resumable by re-invoking `/testrail-run` with the same target (the agent + skips already-recorded cases). + ## Constraints - Execution is always sequential, matching `workers: 1` in `Tests/e2e/testrail.config.ts`. - Never post results to TestRail without explicit user confirmation. -- TestRail credentials (`TESTRAIL_USERNAME`, `TESTRAIL_API_KEY`) live in the environment; - do not prompt for them and never print them. -- This skill never calls the TestRail API or Playwright directly. All of that is the agent's. +- Never delete anything under `Tests/e2e/testrail-cases/` — that is user-gated in the + agent's disposition step. +- TestRail credentials (`TESTRAIL_USERNAME`, `TESTRAIL_API_KEY`) live in the environment or + `.ai/settings.local.json`; do not prompt for them and never print them. +- This skill never calls the TestRail API, Playwright, or Docker directly. All of that is + the agent's. diff --git a/.claude/testrail/specs/_foundation.md b/.claude/testrail/specs/_foundation.md index ca2b21da..35ec2239 100644 --- a/.claude/testrail/specs/_foundation.md +++ b/.claude/testrail/specs/_foundation.md @@ -1,125 +1,186 @@ --- derived_sha: 4bc0e342 -source_files: [imagify.php, inc/classes/class-imagify-options.php, classes/MCP/AbilitiesSubscriber.php, classes/Tools/*.php] +source_files: [imagify.php, inc/classes/class-imagify-options.php, classes/MCP/AbilitiesSubscriber.php, classes/Abilities/*.php, Tests/e2e/fixtures/auth.ts, Tests/e2e/pages/*.ts] last_explored: 2026-06-30 --- -# Foundation — environment, prerequisites, seeding (Imagify local QA) +# Foundation — environment, Playwright reuse, seeding (Imagify QA) Non-DOM, low-volatility grounding shared by every feature spec. The run agent loads this -file plus the per-feature spec before executing any case. Locators captured live this explore -are role/id-based; verify against the page if the build moves. +file plus the per-feature spec before executing any case. This file also absorbs the former +`imagify-playwright-fixtures.md` — it is the single knowledge home; agents carry process, +`Tests/e2e/` carries executable truth, this directory carries grounding. ## Environment -URLs and credentials are read from `.ai/settings.local.json` (gitignored). Two environments: - -| Key | URL | Server | Use when | -|-----|-----|--------|----------| -| `nginx` | http://localhost:10043 | Nginx | default — all cases unless `server: apache` | -| `apache` | http://localhost:10048 | Apache | cases involving `.htaccess`, rewrite rules, `$is_apache` paths | - -- WP admin: `$E2E_URL/wp-admin/` -- Login: admin / admin via /wp-login.php - role-based: getByLabel("Username or Email Address"), getByLabel("Password", { exact: true }) - submit: getByRole("button", { name: "Log In" }) -- WP version requirement: >= 6.9 (the Abilities API is a no-op below this; abilities register - only when `wp_register_ability` exists). -- ACTIVE plugin (live, captured this explore): `wp-content/plugins/imagify-plugin` on branch - `develop` (SHA 53340d05). Live plugin reports version **2.3.0-alpha1** on the settings page. - NOTE: a second checkout `imagify-plugin/` (this worktree is `imagify-e2e-worktree/`) is the - one WordPress loads — its `classes/Abilities/*` is byte-identical to this worktree's, so the - grounding below holds for both. The spec files + agent infra live only in this worktree, so - `derived_sha`/`source_files` track this worktree. +**Never hardcode a URL, port, or credential — everything comes from `.ai/settings.local.json`** +(gitignored; generated by `bin/qa-env.sh up`, or hand-written for Local-site setups). Shape: + +```json +{ + "environments": { + "nginx": { "url": "...", "username": "...", "password": "...", "wp_cli": "...", "wp_path": "..." }, + "apache": { "url": "...", "username": "...", "password": "...", "wp_cli": "...", "wp_path": "..." } + }, + "imagify": { "api_key": "..." }, + "testrail": { "username": "...", "api_key": "..." } +} +``` + +Each env block carries `wp_cli` (full command prefix — qa-env style) **or** `wp_path` +(Local style → prefix is `wp --path=<wp_path>`). All examples below write `$E2E_URL` and +`$WPCMD` — resolve them from the config for the case's target env before use. + +- nginx is the default env; apache is for cases involving `.htaccess`, rewrite rules, or + `$is_apache` code paths (flagged per-case via the feature spec's `apache_cases` frontmatter). +- WP admin: `$E2E_URL/wp-admin/` — login via `/wp-login.php`, role-based locators: + `getByLabel("Username or Email Address")`, `getByLabel("Password", { exact: true })`, + `getByRole("button", { name: "Log In" })`. +- WP version requirement: >= 6.9 for the Abilities API (registration no-ops below it). + +## Playwright reuse (formerly imagify-playwright-fixtures.md) + +Reuse the real code under `Tests/e2e/` — never reimplement: + +- `Tests/e2e/fixtures/auth.ts` → `loginAsAdmin(page)` — idempotent login; reads + `IMAGIFY_ADMIN_USER` / `IMAGIFY_ADMIN_PASS` env vars and the config `baseURL`, so generated + specs stay env-agnostic (the run agent passes the env at invocation time). +- `Tests/e2e/pages/settings.ts` → `SettingsPage`; + `Tests/e2e/pages/bulk-optimization.ts` → `BulkOptimizationPage`; + `Tests/e2e/pages/media-library.ts` → `MediaLibraryPage`. + **A POM method beats any raw locator written here or in a feature spec.** +- `Tests/e2e/fixtures/wp-cli.ts` (`wpCli()` / `hasApiKey()`) shells out to + `npx @wordpress/env run cli` — the **wp-env CI stack**, NOT the TestRail environments. + Never use it for TestRail seeding; use `$WPCMD`. +- No POM exists yet for the Custom Folders surface (`/wp-admin/upload.php?page=imagify-files`) + — locators for it must be grounded fresh; add a POM if a spec starts depending on it. + +### Imagify admin URLs + +| Purpose | Path | +|----------------|-------------------------------------------------------| +| Settings | `/wp-admin/options-general.php?page=imagify` | +| Bulk optimize | `/wp-admin/upload.php?page=imagify-bulk-optimization` | +| Media library | `/wp-admin/upload.php?mode=list` | +| Custom folders | `/wp-admin/upload.php?page=imagify-files` | + +Endpoints: WP Abilities `/wp-json/wp-abilities/v1/abilities`; MCP server +`/wp-json/mcp/mcp-adapter-default-server`. + +### Named selectors (only where no POM covers the surface) + +``` +API key input: #api_key, [name="imagify_settings[api_key]"] +Save button: #submit +Success notice: #setting-error-settings_updated (.notice-success, .updated) +Invalid API key: #imagify-check-api-container:not(.imagify-valid) ← NOT .notice-error +Fatal error guard: .wp-die-message, #error-page → expect count 0 on every page load +Media column: th[id*="imagify"], th.column-imagify +``` + +Imagify does **not** surface an invalid API key via `.notice-error` — assert the API-key +container's validity class instead. + +### Ability slugs (all 7 — grounded live; see mcp-abilities.md for full behavior) + +``` +imagify/get-settings imagify/update-settings imagify/get-account +imagify/get-stats imagify/get-media-status imagify/get-nextgen-coverage +imagify/optimize-media +``` + +These 7 are what `classes/Abilities/` registers. Do not assert on any other slug without +re-grounding — earlier drafts listed slugs that do not exist in the code. ## Prerequisites -- Imagify API key (CORRECTED — confirmed live this explore): - - Stored in the WP option **`imagify_settings`**, key **`api_key`** (NOT `settings.local.json`, - which does not exist in this env). The settings field is `name="imagify_settings[api_key]" - id="api_key"`. - - The PHP constant **`IMAGIFY_API_KEY`** is an OPTIONAL override: when defined+truthy it wins - over the option, locks the settings field, and makes `update-settings api_key` return the - error `imagify_api_key_immutable`. It is NOT defined in this env's wp-config.php. - (Source: inc/classes/class-imagify-options.php:72-116; classes/Abilities/UpdateSettings.php:278-282.) - - Live env state: a VALID key is configured (option-based, field editable). `get-account` live - returns `is_api_key_valid: true`, plan "Growth", quota 500. -- Settings page: /wp-admin/options-general.php?page=imagify -- Capability gate: the `imagify_capacity` filter, via `imagify_get_context('wp')->current_user_can('manage')` - — NOT a direct `current_user_can()` call. + +- Imagify API key: stored in the WP option **`imagify_settings`**, key **`api_key`**. + Settings field: `name="imagify_settings[api_key]" id="api_key"`. The PHP constant + **`IMAGIFY_API_KEY`** is an optional override: when defined+truthy it wins over the option, + locks the settings field, and makes `update-settings api_key` return + `imagify_api_key_immutable`. (Source: inc/classes/class-imagify-options.php:72-116; + classes/Abilities/UpdateSettings.php:278-282.) +- Settings page: `/wp-admin/options-general.php?page=imagify` — the General Settings section + renders only when a valid API key is configured. +- Capability gate: the `imagify_capacity` filter via + `imagify_get_context('wp')->current_user_can('manage')` — NOT a bare `current_user_can()`. ## Test users (for permission cases) -| Role | Login | imagify_capacity | Seed | -|---------------|----------------|------------------|------------------------------| -| administrator | admin / admin | full | exists by default | -| editor | <seed> | limited | see Seeding helpers | -| subscriber | <seed> | denied | see Seeding helpers | -## Authentication for live API/MCP/REST calls (captured this explore) -The Abilities REST + MCP endpoints require an authenticated WP session + a REST nonce. Both -return **HTTP 401** anonymously. Acquire a session by curl (Bash) or the browser: +| Role | Login | imagify_capacity | Seed | +|---------------|----------------------|------------------|---------------------| +| administrator | from config | full | exists by default | +| editor | seeded per case | limited | see Seeding helpers | +| subscriber | seeded per case | denied | see Seeding helpers | + +## Authentication for live API/MCP/REST calls + +Abilities REST + MCP endpoints require an authenticated WP session + REST nonce (both return +HTTP **401** anonymously): ```bash -# 1. Log in (cookie jar) +# 1. Log in (cookie jar) — creds from the config, never hardcoded curl -s -c cookies.txt -b cookies.txt \ - --data-urlencode "log=admin" --data-urlencode "pwd=admin" \ + --data-urlencode "log=$WP_USER" --data-urlencode "pwd=$WP_PASS" \ --data-urlencode "wp-submit=Log In" --data-urlencode "testcookie=1" \ - --data-urlencode "redirect_to=http://localhost:10038/wp-admin/" \ - http://localhost:10038/wp-login.php -o /dev/null + --data-urlencode "redirect_to=$E2E_URL/wp-admin/" \ + "$E2E_URL/wp-login.php" -o /dev/null # 2. REST nonce -NONCE=$(curl -s -b cookies.txt "http://localhost:10038/wp-admin/admin-ajax.php?action=rest-nonce") -# 3. Use header: -H "X-WP-Nonce: $NONCE" -b cookies.txt +NONCE=$(curl -s -b cookies.txt "$E2E_URL/wp-admin/admin-ajax.php?action=rest-nonce") +# 3. Use: -H "X-WP-Nonce: $NONCE" -b cookies.txt ``` -## Seeding helpers (WP-CLI / REST — run via Bash, NOT the UI) -Deterministic state setup. NOTE: the host shell `wp` CLI is NOT wired to this Local site's DB -(socket mismatch); seed via the authenticated REST/abilities API or `wp` inside the Local shell. -Each helper that mutates state has a matching teardown (see "Teardown helpers"). +## Seeding helpers (via `$WPCMD` / REST — Bash, never the UI) + +Each mutation has a matching teardown below. ```bash -# --- Set / clear API key (option-based; only when IMAGIFY_API_KEY constant is NOT defined) --- -# via the update-settings ability (MCP execute path — see mcp-abilities.md), or: -wp option patch update imagify_settings api_key "<key>" # inside Local site shell -wp option patch update imagify_settings api_key "" - -# --- Toggle an arbitrary setting (option-based) --- -wp option patch update imagify_settings <key> <value> -# live setting keys: optimization_level, lossless, auto_optimize, backup, resize_larger, -# resize_larger_w, display_nextgen, display_nextgen_method, display_webp, display_webp_method, -# cdn_url, disallowed-sizes, admin_bar_menu, partner_links, convert_to_avif, convert_to_webp, -# optimization_format (api_key + version are managed separately; version is read-only) - -# --- An attachment for media-status / optimize-media cases --- -# Live env already has attachments: id 6,7,8 (8 = "test_huge_image", jpeg, already optimized -# level 2). To seed a fresh one via REST media upload: +# --- Set / clear the API key (option-based; only when IMAGIFY_API_KEY constant is NOT defined) --- +$WPCMD option patch update imagify_settings api_key "<key>" +$WPCMD option patch update imagify_settings api_key "" + +# --- Toggle an arbitrary setting --- +$WPCMD option patch update imagify_settings <key> <value> +# live keys: optimization_level, lossless, auto_optimize, backup, resize_larger, +# resize_larger_w, display_nextgen, display_nextgen_method, display_webp, +# display_webp_method, cdn_url, disallowed-sizes, admin_bar_menu, partner_links, +# convert_to_avif, convert_to_webp, optimization_format +# (api_key + version are managed separately; version is read-only) + +# --- Seed an attachment (media-status / optimize-media cases) --- curl -s -b cookies.txt -H "X-WP-Nonce: $NONCE" \ -H "Content-Disposition: attachment; filename=seed.jpg" -H "Content-Type: image/jpeg" \ - --data-binary @<local.jpg> "http://localhost:10038/wp-json/wp/v2/media" \ - | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])" # -> new attachment ID + --data-binary @<local.jpg> "$E2E_URL/wp-json/wp/v2/media" \ + | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])" # --- Create a non-admin test user (permission cases) --- -wp user create <login> <email> --role=<role> +$WPCMD user create <login> <email> --role=<role> --user_pass=<pass> ``` ## Teardown helpers (LIFO undo) + ```bash # --- Restore an attachment to its pre-optimization state --- -# Imagify writes _imagify_data / _imagify_status post meta and a backup file. -# Undo via the media UI "Restore Original", or delete the meta + restore backup: -wp post meta delete <id> _imagify_data -wp post meta delete <id> _imagify_status -# (If a fresh attachment was seeded for the case, delete it entirely:) +$WPCMD post meta delete <id> _imagify_data +$WPCMD post meta delete <id> _imagify_status +# (a freshly seeded attachment: delete it entirely) curl -s -b cookies.txt -H "X-WP-Nonce: $NONCE" -X DELETE \ - "http://localhost:10038/wp-json/wp/v2/media/<id>?force=true" -o /dev/null + "$E2E_URL/wp-json/wp/v2/media/<id>?force=true" -o /dev/null # --- Re-apply a settings snapshot taken before a mutation --- -# Snapshot BEFORE: GET imagify/get-settings (captures all keys). Restore AFTER via -# update-settings with the snapshot, or: -wp option update imagify_settings '<json snapshot>' --format=json +# Snapshot BEFORE via GET get-settings/run; restore via update-settings, or: +$WPCMD option update imagify_settings '<json snapshot>' --format=json # --- Delete a seeded test user --- -wp user delete <login> --yes --reassign=1 +$WPCMD user delete <login> --yes --reassign=1 + +# --- Nuclear option (qa-env setups only): restore the post-setup DB snapshot --- +bash bin/qa-env.sh reset <nginx|apache> ``` ## Drift detection inputs -`source_files` (above) are the globs whose latest commit is compared against `derived_sha`. -`/testrail-setup --check` reports this spec stale if any source file changed after the stamp. + +`source_files` (frontmatter) are the globs whose latest commit is compared against +`derived_sha` **by git ancestry** (`git merge-base --is-ancestor <latest> <derived>` → FRESH), +not by string comparison. `/testrail-setup --check` reports this spec stale when any source +file changed after the stamp. diff --git a/.claude/testrail/specs/mcp-abilities.md b/.claude/testrail/specs/mcp-abilities.md index 23d76b21..a0eeb9b3 100644 --- a/.claude/testrail/specs/mcp-abilities.md +++ b/.claude/testrail/specs/mcp-abilities.md @@ -18,7 +18,7 @@ and `meta.show_in_rest = true`, so they are reachable over the Abilities REST na `wp-abilities/v1`. Registration no-ops silently when `wp_register_ability` is absent (WP < 6.9). SOURCE-OF-TRUTH NOTE (read before grounding drift): the live behaviour below was captured against -the **nginx** site `http://localhost:10043`, whose active plugin checkout is +the **nginx** site `$E2E_URL`, whose active plugin checkout is `/Users/gaelrobin/Local Sites/e2eimagifynginx/.../imagify-plugin` on branch `feat/mixpanel` (SHA 461e5839, plugin v2.2.9 header, but carrying the **2.3.0 abilities** with the `AbstractAbility` base + `get_id()` template). This worktree (`imagify-e2e-worktree`, SHA 542b43d4) @@ -102,9 +102,9 @@ in the local part, confirming C26383): ## How to invoke (grounded from live) All calls require an authenticated WP session + `X-WP-Nonce` header (anonymous discovery and run -both return HTTP **401**, confirming C26360). Acquire cookie + nonce per `_foundation.md` -(NOTE: foundation uses port 10038; the live env for THIS feature is the nginx site **10043**). -Base for this feature: `http://localhost:10043`. +both return HTTP **401**, confirming C26360). Acquire cookie + nonce per `_foundation.md`. +Base for this feature: `$E2E_URL` (the nginx env from `.ai/settings.local.json` — never a +hardcoded port; the original capture ran against a Local nginx site). - **Discovery (C26359 / C26360):** `GET /wp-json/wp-abilities/v1/abilities` (authed) → JSON array; filter `name` starting `imagify/` @@ -149,7 +149,7 @@ live on `/wp-admin/options-general.php?page=imagify` (no data-testid on this pag ## Prerequisites & seeding (per operation) - **Plugin MUST be active** (HARD PREREQUISITE — discovered live): the abilities only register when - `imagify-plugin/imagify` is active. On first explore this plugin was **inactive** on 10043 and + `imagify-plugin/imagify` is active. On first explore this plugin was **inactive** in the capture env and ZERO imagify abilities appeared in discovery. Activate before any case: `POST /wp-json/wp/v2/plugins/imagify-plugin/imagify {"status":"active"}` (auth + nonce), or `wp plugin activate imagify-plugin` inside the Local site shell. Verify via @@ -178,8 +178,8 @@ live on `/wp-admin/options-general.php?page=imagify` (no data-testid on this pag (`wp option patch update imagify_settings api_key "<valid key>"` in the Local shell) — optional. - **optimize-media / get-media-status success cases** (C26363, C26374, C26385): require (a) a real attachment AND (b) a VALID API key (optimization calls the Imagify API). The media library on - 10043 is EMPTY (live check returned 0 attachments). Seed an attachment via REST media upload - (helper in `_foundation.md`, adjust to port 10043), and configure a valid key. Without a valid + the capture env was EMPTY (live check returned 0 attachments). Seed an attachment via REST + media upload (helper in `_foundation.md`, against `$E2E_URL`), and configure a valid key. Without a valid key, optimize-media returns a `status:"error"` (API not reachable), which still satisfies the 5-field-schema cases (C26389) but NOT the success cases. Error/validation cases (C26375-C26377, C26386-C26389) need NO attachment seeding and NO key — they were all captured live as-is. diff --git a/.claude/testrail/specs/mixpanel-tracking.md b/.claude/testrail/specs/mixpanel-tracking.md index 2585cbe1..42a43d6e 100644 --- a/.claude/testrail/specs/mixpanel-tracking.md +++ b/.claude/testrail/specs/mixpanel-tracking.md @@ -70,7 +70,7 @@ Captured live this explore unless noted. Mixpanel token (`ServiceProvider::MIXPA endpoint). For tests, intercept this via a `pre_http_request` spy — DO NOT send live data. ## How to invoke (grounded from live) -- **Analytics opt-in UI**: GET `http://localhost:10043/wp-admin/options-general.php?page=imagify`. +- **Analytics opt-in UI**: GET `$E2E_URL/wp-admin/options-general.php?page=imagify`. The opt-in section renders inside the General Settings form (hook `imagify_settings_after_lossless`), so it requires a valid API key (same gate as General Settings; live env satisfies this). - **Toggle opt-in (AJAX, what the checkbox JS does — captured live)**: POST diff --git a/.claude/testrail/specs/settings.md b/.claude/testrail/specs/settings.md index 698b8bac..faaf0b2c 100644 --- a/.claude/testrail/specs/settings.md +++ b/.claude/testrail/specs/settings.md @@ -4,6 +4,7 @@ feature: "Imagify Settings" source_files: [views/page-settings.php, views/notice-temporary.php, inc/classes/class-imagify-settings.php, inc/classes/class-imagify-options.php, inc/classes/class-imagify-auto-optimization.php, classes/Notices/Notices.php, classes/Abilities/UpdateSettings.php] derived_sha: e4826be7 last_explored: 2026-06-30 +apache_cases: [14169] --- ## Overview @@ -33,7 +34,8 @@ via `imagify_get_context('wp')->current_user_can('manage')` (see `_foundation.md - disabled → the action **`imagify_new_attachment_auto_optimization_disabled`** fires and the method returns early; no `_imagify_*` meta is written for the new upload. - Gate also requires a valid API key (`Imagify_Requirements::is_api_key_valid()`). -- Temporary admin notices (C14169 — escaping) **`server: apache`**: rendered by `views/notice-temporary.php`, which +- Temporary admin notices (C14169 — escaping; **apache env**, flagged in frontmatter + `apache_cases`): rendered by `views/notice-temporary.php`, which emits the message through **`wp_kses( $details['message'], [ 'code' => [] ] )`** — i.e. only `<code>` survives, every other tag is stripped. Notice messages legitimately contain `<code>` (e.g. the .htaccess error in `classes/Webp/Display.php:114` wraps a file path in `<code>…</code>`). @@ -48,7 +50,7 @@ via `imagify_get_context('wp')->current_user_can('manage')` (see `_foundation.md network admin and therefore requires a **multisite install** — NOT available in this env. ## How to invoke (grounded from live) -- Admin UI (single site): GET `http://localhost:10038/wp-admin/options-general.php?page=imagify` +- Admin UI (single site): GET `$E2E_URL/wp-admin/options-general.php?page=imagify` → toggle a checkbox → click Save (`#submit`, value "Save Changes"). Form POSTs to `options.php` (single) and redirects back to `?page=imagify` with a "Settings saved." notice. - Admin UI (multisite, C174): `http://<site>/wp-admin/network/settings.php?page=imagify` diff --git a/.claude/testrail/translation.md b/.claude/testrail/translation.md new file mode 100644 index 00000000..e3f63d7d --- /dev/null +++ b/.claude/testrail/translation.md @@ -0,0 +1,177 @@ +# Translating a TestRail case into a Playwright spec — thinking protocol + +Read this in full before translating your first case of a run. It is not a list of commands — +the run agent's own file covers mechanics. This is the reasoning layer: how to think about the +prose→code problem so that the spec you emit proves something, instead of merely resembling +the prose it came from. + +## The one-sentence model + +A TestRail case is a human's **lossy serialization of an intent and an oracle**. Your job is +not to transcribe its steps into code — it is to reconstruct the intent, find the strongest +observable oracle, and emit the smallest deterministic program that proves or refutes it. +Every failure mode of naive translation (guessed selectors, tautological assertions, steps +that "pass" because they check nothing) comes from transcribing instead of reconstructing. + +## 1. Read the case backwards + +Read all the expected results before you read any action. The **last** expected result is +usually the case; everything before it is scaffolding to get there. Before writing anything, +answer in one sentence: *"What single observable state change would prove this case?"* Write +that sentence down. Every locator, action, and assertion you choose afterward is in service +of reaching and checking that oracle — anything that doesn't serve it is decoration. + +## 2. Classify every step before writing any code + +Human testers blur four different things into "steps". You must not: + +| Kind | Tell | What it becomes | +|---|---|---| +| **Setup in disguise** | "Have an optimized image", "Be logged in as an editor" | Seeding via WP-CLI/REST *before* the browser opens — never UI clicks | +| **Navigation** | "Go to Settings > Imagify" | `page.goto()` / POM `goto()` | +| **Actuation** | "Enable the toggle and Save" | POM method or grounded locator action | +| **Observation** | "Verify the notice appears" | An assertion attached to the *preceding* actuation — not an empty step | + +A step that is pure observation gets no action; a step that is setup in disguise gets no +`test.step()` at all. If after classification a step is neither reachable by seeding nor by +the browser (needs multisite, a paid service, an env you don't have), the case is BLOCKED — +you know this *before* generating code, which is the cheapest possible time to know it. + +## 3. Every noun resolves through three sources, in order + +Every UI noun in the prose ("the lossless toggle", "the bulk page") must resolve to exactly +one of, in strict preference order: + +1. a **POM method** (`Tests/e2e/pages/` — the compiled, proven truth), +2. a **grounded locator** from the matched feature spec (`.claude/testrail/specs/`), +3. **one targeted live observation** (navigate + snapshot of that specific element). + +Prose never resolves directly to a selector. If a noun resolves to none of the three, that is +a **grounding gap**, not a guessing opportunity — the correct output is BLOCKED with the exact +noun that failed to resolve, so `/testrail-setup` can fix it once for every future run. A +confidently guessed selector is strictly worse than a BLOCKED row: the BLOCKED row costs a +human one grounding pass; the guess costs a human a false signal they must investigate. + +## 4. Choose the strongest cheap oracle + +Expected results are claims about the world. Rank the ways of checking a claim: + +1. **State** — read the actual option/meta/response via REST or WP-CLI (`get-settings` shows + `lossless == 1`). +2. **Dedicated UI signal** — the specific element whose job is to announce this outcome + (`#setting-error-settings_updated`, the API-key container's validity class). +3. **Generic UI echo** — a success notice class, a changed label. +4. **Absence of error** — page loaded, no fatal. + +Prefer the highest rank that costs one request. When the case is *about* the UI (the point is +that the checkbox renders checked), assert the UI **and** the state — but never let a generic +echo be the only proof of a state change when the state itself is one cheap request away. +Level 4 alone is never an oracle; it is the floor every step gets for free. + +## 5. Plan → validate → codegen. Never prose → code. + +Before emitting TypeScript, write the intermediate plan as a table — one row per TestRail +step: + +``` +step | kind (seed/nav/act/observe) | surface | locator + SOURCE (pom/spec/observed) | oracle | undo +``` + +Then validate the table, mechanically: +- No row's locator source is "inferred" or blank. (Rule 3.) +- Every actuation row has an oracle, or explicitly feeds the next row's oracle. (Rule 4.) +- Every row that mutates state has an `undo` entry, queued LIFO **now**, not after execution. +- Any conflict between sources is resolved upward: POM beats spec beats hints beats prose. + +Only a table that passes becomes code. This is not bureaucracy — the table is where +hallucinations die cheaply. A wrong locator caught in the plan costs one look; caught at +execution it costs a browser run, a trace, and a repair loop; caught by a human reviewing a +false FAIL it costs the credibility of the whole run. + +## 6. The ambiguity protocol + +When the prose is ambiguous (two Save buttons; "the settings page" when there are three): + +1. The grounded spec decides, if it addresses the surface. +2. Else spend **one** targeted live observation on the ambiguous element only. +3. Else BLOCKED, stating the precise question a human must answer — not a guess. + +Never resolve ambiguity by picking the interpretation that is easiest to code. The tester who +wrote the case had one specific thing in mind; your job is to find it or say you couldn't. + +## 7. Spend observations where your uncertainty is + +You have a budget of live looks (they are slow and cost turns). Before generating, mark each +plan row with your honest confidence: a POM method with a track record is high; a grounded +locator from a fresh spec is high; a grounded locator from a spec whose `derived_sha` is +stale is medium; anything the spec doesn't cover is low. Spend your pre-flight observation on +the **lowest-confidence row only**. Confirming what you already know is waste; running blind +on the row you doubt is how false results happen. Calibration — knowing *which* of your +beliefs is the weak one — is the skill. + +## 8. Translate by analogy before translating from scratch + +Before writing a new spec, look for the nearest **green cached spec** (same TestRail section, +same admin surface) under `Tests/e2e/testrail-cases/` and diff the new case's steps against +it. Adapting a proven artifact — same imports, same login step, same seeding idiom, changed +actuation and oracle — is dramatically more reliable than de novo generation, because +everything you didn't change is already known to compile, run, and assert correctly. The +cache is not just a speed-up; it is your few-shot library, and it gets better every run. + +## 9. Interpreting failure: your model of the app vs. the app + +When a run fails, the order of suspicion is fixed: + +1. **My code** — compile error, unescaped quote, wrong import path. (Tooling, never FAIL.) +2. **My model of the app** — locator drifted, timing assumption wrong, conditional render I + didn't know about. (Re-observe the real element once; fix the spec.) +3. **The environment** — login failed, site down, missing prerequisite. (BLOCKED.) +4. **The product** — only after a re-observation confirms the live page truly contradicts the + expected result. + +The prior matters: on a first run, most failures are #1 or #2. You may only report FAIL when +you can point at the defect — the exact element/response that contradicts the TestRail +expected result — and you have confirmed it against the live page, not just against your +generated code's opinion of it. "My spec went red and I can't tell why" is never FAIL; it is +BLOCKED or a repair pass, depending on which attempt you're on. + +## 10. Determinism disciplines (non-negotiable) + +- Web-first assertions only (`toBeVisible`, `toHaveText`, `toHaveValue` with timeouts). + Never `waitForTimeout` — a sleep is an admission you don't know what you're waiting for; + find the signal and wait on it. +- `expect.soft` for every TestRail-step assertion, with a message naming the expected result + it checks — the message is what a human reads in the results file, write it for them. +- Escape everything interpolated from TestRail prose (quotes, backticks, `${`) — a spec that + fails to compile because of an apostrophe is your bug, and must never surface as FAIL. +- Seed state via API/CLI, verify via the strongest oracle, undo via the queued LIFO teardown. + The UI is for the behavior under test, not for setup — setup through the UI makes case N's + outcome depend on case N−1's cleanup being perfect. +- One `test()` per case, one `test.step()` per TestRail step, login always first. A fresh + `test()` per step would discard the session that later steps depend on. + +## Worked micro-example + +TestRail case: *"1. Enable lossless compression in the settings and save. Expected: setting +is saved. 2. Optimize an image from the media library. Expected: the image is optimized +losslessly."* + +**Backwards read:** the case is step 2's expected — an optimization actually ran in lossless +mode. Step 1 is scaffolding. One-sentence oracle: *"after optimizing, the attachment's +optimization data records lossless mode."* + +**Classification:** "an image" in step 2 is setup in disguise → seed an attachment via REST +before the browser opens, queue its deletion. Step 1 is actuation with its own oracle. Step +2 is actuation + the case's real oracle. + +**Resolution:** lossless toggle → grounded spec (`#imagify_lossless`, label overlays input — +spec says read `checked`, don't click the label); Save → POM `settings.save()`; optimize +button in the library row → `MediaLibraryPage` POM. + +**Oracles:** step 1 — strongest cheap check is state: `get-settings` returns `lossless == 1` +(the success notice is the UI echo, assert it too since it's free). Step 2 — state again: +the attachment's `_imagify_data` meta (or `get-media-status`) shows an optimization result +consistent with lossless; "the image looks the same" is not observable and must not be faked. + +**Plan validates** (every row sourced, every mutation has an undo: restore `lossless` to its +snapshot value, delete the seeded attachment) → now, and only now, write the TypeScript. diff --git a/.gitignore b/.gitignore index 68d7cfac..82c854f3 100644 --- a/.gitignore +++ b/.gitignore @@ -22,6 +22,8 @@ composer.lock /.e2e-screenshots /Tests/e2e/.e2e-screenshots /Tests/e2e/.testrail-tmp +/Tests/e2e/.testrail-artifacts +/Tests/e2e/qa-env/snapshots # Generated distribution packages /generatedpackages diff --git a/Tests/e2e/qa-env/docker-compose.yml b/Tests/e2e/qa-env/docker-compose.yml new file mode 100644 index 00000000..0ae8c2b4 --- /dev/null +++ b/Tests/e2e/qa-env/docker-compose.yml @@ -0,0 +1,117 @@ +# Ephemeral QA environments for TestRail runs — one nginx + one apache WordPress. +# Managed by bin/qa-env.sh (up / reset / down / status). Not for daily development — +# devs keep using Local; the QA pipeline owns these so a run never depends on a +# hand-built site being configured correctly. +name: imagify-qa + +services: + db-nginx: + image: mariadb:10.11 + environment: + MYSQL_ROOT_PASSWORD: root + MYSQL_DATABASE: wordpress + MYSQL_USER: wordpress + MYSQL_PASSWORD: wordpress + volumes: + - db-nginx:/var/lib/mysql + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 3s + timeout: 3s + retries: 20 + + db-apache: + image: mariadb:10.11 + environment: + MYSQL_ROOT_PASSWORD: root + MYSQL_DATABASE: wordpress + MYSQL_USER: wordpress + MYSQL_PASSWORD: wordpress + volumes: + - db-apache:/var/lib/mysql + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 3s + timeout: 3s + retries: 20 + + php-nginx: + image: wordpress:php8.3-fpm + depends_on: + db-nginx: + condition: service_healthy + environment: + WORDPRESS_DB_HOST: db-nginx + WORDPRESS_DB_USER: wordpress + WORDPRESS_DB_PASSWORD: wordpress + WORDPRESS_DB_NAME: wordpress + volumes: + - wp-nginx:/var/www/html + - ../../generatedpackages:/packages:ro + + web-nginx: + image: nginx:1.27-alpine + depends_on: + - php-nginx + ports: + - "${QA_NGINX_PORT:-8801}:80" + volumes: + - wp-nginx:/var/www/html + - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro + + wp-apache: + image: wordpress:php8.3-apache + depends_on: + db-apache: + condition: service_healthy + ports: + - "${QA_APACHE_PORT:-8802}:80" + environment: + WORDPRESS_DB_HOST: db-apache + WORDPRESS_DB_USER: wordpress + WORDPRESS_DB_PASSWORD: wordpress + WORDPRESS_DB_NAME: wordpress + volumes: + - wp-apache:/var/www/html + - ../../generatedpackages:/packages:ro + + # WP-CLI sidecars — invoked via `docker compose run --rm cli-nginx wp ...`. + # The run agent gets these full command prefixes from the generated + # .ai/settings.local.json (environments.<env>.wp_cli). + cli-nginx: + image: wordpress:cli-php8.3 + depends_on: + db-nginx: + condition: service_healthy + user: "33:33" + environment: + WORDPRESS_DB_HOST: db-nginx + WORDPRESS_DB_USER: wordpress + WORDPRESS_DB_PASSWORD: wordpress + WORDPRESS_DB_NAME: wordpress + volumes: + - wp-nginx:/var/www/html + - ../../generatedpackages:/packages:ro + profiles: ["cli"] + + cli-apache: + image: wordpress:cli-php8.3 + depends_on: + db-apache: + condition: service_healthy + user: "33:33" + environment: + WORDPRESS_DB_HOST: db-apache + WORDPRESS_DB_USER: wordpress + WORDPRESS_DB_PASSWORD: wordpress + WORDPRESS_DB_NAME: wordpress + volumes: + - wp-apache:/var/www/html + - ../../generatedpackages:/packages:ro + profiles: ["cli"] + +volumes: + db-nginx: + db-apache: + wp-nginx: + wp-apache: diff --git a/Tests/e2e/qa-env/nginx.conf b/Tests/e2e/qa-env/nginx.conf new file mode 100644 index 00000000..6bc9de31 --- /dev/null +++ b/Tests/e2e/qa-env/nginx.conf @@ -0,0 +1,27 @@ +server { + listen 80; + server_name _; + root /var/www/html; + index index.php; + + client_max_body_size 64m; + + location / { + try_files $uri $uri/ /index.php?$args; + } + + location ~ \.php$ { + try_files $uri =404; + fastcgi_split_path_info ^(.+\.php)(/.+)$; + fastcgi_pass php-nginx:9000; + fastcgi_index index.php; + include fastcgi_params; + fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; + fastcgi_param PATH_INFO $fastcgi_path_info; + } + + location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|webp|avif)$ { + expires max; + log_not_found off; + } +} diff --git a/Tests/e2e/testrail-cases/README.md b/Tests/e2e/testrail-cases/README.md new file mode 100644 index 00000000..51d7c35a --- /dev/null +++ b/Tests/e2e/testrail-cases/README.md @@ -0,0 +1,29 @@ +# TestRail case specs (committed corpus) + +One Playwright spec per TestRail case, generated by `testrail-run-agent` and **kept across +runs** — this directory is a compile cache, not scratch space. + +Each file starts with a machine-readable header: + +```ts +// @testrail-case: 155 +// @steps-hash: 3f2a91c04d7e ← sha256 (first 12) of the case's stripped steps+expected +// @grounding: <derived_sha of the feature spec used at translation time> +// @status: green 2026-07-03 +``` + +On each run the agent re-hashes the TestRail case: unchanged hash + fresh grounding → the +spec is **reused as-is** (no re-translation); changed hash → re-translated and overwritten. +Green specs here are also the few-shot examples for translating new cases, and the candidate +pool for promotion into the permanent CI suite (`Tests/e2e/specs/`) — promotion is proposed +at the end of each run and is always a human decision. + +Run one manually: + +```bash +cd Tests/e2e +IMAGIFY_BASE_URL=http://localhost:8801 npx playwright test --config=testrail.config.ts testrail-cases/case-155.spec.ts +``` + +Never delete files here without checking with the team — deletion is user-gated in the run +agent for a reason: a deleted spec costs a full re-translation next release. diff --git a/Tests/e2e/testrail.config.ts b/Tests/e2e/testrail.config.ts index f4a4cf39..996a06b5 100644 --- a/Tests/e2e/testrail.config.ts +++ b/Tests/e2e/testrail.config.ts @@ -2,16 +2,15 @@ import { defineConfig, devices } from '@playwright/test'; /** * TestRail release-QA config — separate from `playwright.config.ts` (the permanent - * regression suite under `./specs`) because `testrail-run-agent` generates one throwaway - * `.spec.ts` per TestRail case into `./.testrail-tmp/` and must not mix them into the - * committed suite's testDir. + * regression suite under `./specs`) because `testrail-run-agent` executes one `.spec.ts` + * per TestRail case from the committed cache `./testrail-cases/` and must not mix those + * into the regression suite's testDir. Cached case specs are kept across runs (content- + * hash header decides reuse vs re-translation) and are promotion candidates for `./specs`. * * The run agent invokes this per case, one `npx playwright test` call at a time. `$OUT` is - * captured as an ABSOLUTE path from repo root BEFORE `cd`ing into `Tests/e2e/` — every other - * `.ai/...` path the run agent uses (config, dashboard, trace links) is repo-root-relative, so - * this must not be a bare relative path once you're inside `Tests/e2e/`. stderr is kept out of - * `results.json` — any npx/npm/Node warning on stderr would otherwise corrupt the JSON the run - * agent parses for pass/fail: + * captured as an ABSOLUTE path from repo root BEFORE `cd`ing into `Tests/e2e/`. stderr is + * kept out of `results.json` — any npx/npm/Node warning on stderr would otherwise corrupt + * the JSON the run agent parses for pass/fail: * * OUT="$(pwd)/.ai/testrail/$RUN_ID/playwright/$CASE_ID" # captured at repo root * mkdir -p "$OUT" @@ -19,28 +18,29 @@ import { defineConfig, devices } from '@playwright/test'; * cd Tests/e2e && * IMAGIFY_BASE_URL="$E2E_URL" IMAGIFY_ADMIN_USER="$WP_USER" IMAGIFY_ADMIN_PASS="$WP_PASS" \ * TESTRAIL_OUTPUT_DIR="$OUT" \ - * npx playwright test --config=testrail.config.ts ".testrail-tmp/case-$CASE_ID.spec.ts" \ + * npx playwright test --config=testrail.config.ts "testrail-cases/case-$CASE_ID.spec.ts" \ * --reporter=json > "$OUT/results.json" 2> "$OUT/stderr.log" * ) * - * Evidence lands directly under TESTRAIL_OUTPUT_DIR — no post-hoc relocation step needed - * (unlike the old Canary flow, which had to `mv ~/.canary/sessions/<id>` after the fact). + * NOTE on evidence paths: Playwright creates a per-test SUBDIRECTORY inside outputDir — + * trace/video land at `$OUT/<test-result-dir>/trace.zip`, NOT at `$OUT/trace.zip`. Resolve + * them by glob (`ls "$OUT"/*'/'trace.zip`), never by assuming the flat path. * - * HAR capture is intentionally dropped: Playwright's `use.recordHar` needs a static path per - * test and doesn't compose cleanly with per-case dynamic naming. The trace.zip already - * contains the network tab in the trace viewer, which covers the same debugging need. + * HAR capture is intentionally dropped: `use.recordHar` needs a static path per test and + * doesn't compose with per-case dynamic naming. The trace.zip contains the network tab in + * the trace viewer, which covers the same debugging need. */ export default defineConfig({ - testDir: './.testrail-tmp', - fullyParallel: false, // one case, one browser, at a time — same requirement as before + testDir: './testrail-cases', + fullyParallel: false, // one case, one browser, at a time workers: 1, retries: 0, // TestRail evidence is a single authoritative attempt, not a flaky-retry target reporter: 'list', // the run agent always overrides with `--reporter=json` on the CLI - outputDir: process.env.TESTRAIL_OUTPUT_DIR ?? './.testrail-tmp/artifacts', + outputDir: process.env.TESTRAIL_OUTPUT_DIR ?? './.testrail-artifacts', use: { - baseURL: process.env.IMAGIFY_BASE_URL ?? 'http://localhost:8888', + baseURL: process.env.IMAGIFY_BASE_URL ?? 'http://localhost:8801', trace: 'on', // always-on, not retain-on-failure — TestRail wants evidence for PASS too video: 'on', screenshot: 'on', diff --git a/bin/qa-env.sh b/bin/qa-env.sh new file mode 100755 index 00000000..7a3ccedc --- /dev/null +++ b/bin/qa-env.sh @@ -0,0 +1,162 @@ +#!/usr/bin/env bash +# Ephemeral QA environments for TestRail runs (one nginx + one apache WordPress). +# +# bin/qa-env.sh up build imagify.zip, start both envs, install WP + plugin, +# seed the API key, write .ai/settings.local.json, snapshot DBs +# bin/qa-env.sh reset [nginx|apache|all] restore DB(s) to the post-setup snapshot +# bin/qa-env.sh status reachability check for both envs +# bin/qa-env.sh down stop and DELETE both envs completely (volumes included) +# +# Env vars: IMAGIFY_TESTS_API_KEY (optional — seeded into imagify_settings when set), +# QA_NGINX_PORT (default 8801), QA_APACHE_PORT (default 8802). +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +COMPOSE_FILE="$REPO_ROOT/Tests/e2e/qa-env/docker-compose.yml" +SNAP_DIR="$REPO_ROOT/Tests/e2e/qa-env/snapshots" +CONFIG_FILE="$REPO_ROOT/.ai/settings.local.json" + +NGINX_PORT="${QA_NGINX_PORT:-8801}" +APACHE_PORT="${QA_APACHE_PORT:-8802}" +ADMIN_USER="admin" +ADMIN_PASS="password" + +dc() { docker compose -f "$COMPOSE_FILE" "$@"; } +wp_nginx() { dc run --rm --no-deps cli-nginx wp "$@"; } +wp_apache() { dc run --rm --no-deps cli-apache wp "$@"; } + +require_docker() { + command -v docker >/dev/null 2>&1 || { echo "ERROR: docker is not installed — see Tests/e2e/qa-env/README" >&2; exit 1; } + docker info >/dev/null 2>&1 || { echo "ERROR: docker daemon is not running" >&2; exit 1; } +} + +wait_http() { # url, tries + local url="$1" tries="${2:-60}" + for _ in $(seq 1 "$tries"); do + if curl -s -o /dev/null -w '%{http_code}' "$url" | grep -qE '^(200|30[0-9])$'; then return 0; fi + sleep 2 + done + echo "ERROR: $url did not become reachable" >&2 + return 1 +} + +install_wp() { # wp_fn, url, label + local wp_fn="$1" url="$2" label="$3" + if ! $wp_fn core is-installed >/dev/null 2>&1; then + echo "-- [$label] installing WordPress" + $wp_fn core install --url="$url" --title="Imagify QA ($label)" \ + --admin_user="$ADMIN_USER" --admin_password="$ADMIN_PASS" \ + --admin_email="qa@example.com" --skip-email + fi + echo "-- [$label] installing imagify.zip" + $wp_fn plugin install /packages/imagify.zip --activate --force + $wp_fn rewrite structure '/%postname%/' --hard || true + if [ -n "${IMAGIFY_TESTS_API_KEY:-}" ]; then + echo "-- [$label] seeding Imagify API key" + $wp_fn option patch update imagify_settings api_key "$IMAGIFY_TESTS_API_KEY" 2>/dev/null \ + || $wp_fn option update imagify_settings "{\"api_key\":\"$IMAGIFY_TESTS_API_KEY\"}" --format=json + fi +} + +cmd_up() { + require_docker + + # Preserve an existing Imagify key when the env var isn't set (must happen BEFORE + # install_wp seeds it into the fresh sites). + if [ -z "${IMAGIFY_TESTS_API_KEY:-}" ] && [ -f "$CONFIG_FILE" ]; then + IMAGIFY_TESTS_API_KEY=$(python3 -c "import json;print(json.load(open('$CONFIG_FILE')).get('imagify',{}).get('api_key',''))" 2>/dev/null || echo '') + fi + + echo "== Building imagify.zip" + ZIP=$(ls "$REPO_ROOT"/generatedpackages/*.zip 2>/dev/null | head -1 || true) + if [ -z "$ZIP" ] || [ -n "$(find "$REPO_ROOT" -name '*.php' -newer "$ZIP" -not -path '*/vendor/*' -not -path '*/node_modules/*' -print -quit 2>/dev/null)" ]; then + bash "$REPO_ROOT/bin/build-zip.sh" + ZIP=$(ls "$REPO_ROOT"/generatedpackages/*.zip | head -1) + fi + # the compose mount expects the canonical name + cp -f "$ZIP" "$REPO_ROOT/generatedpackages/imagify.zip" 2>/dev/null || true + + echo "== Starting containers (nginx :$NGINX_PORT, apache :$APACHE_PORT)" + dc up -d --wait db-nginx db-apache php-nginx web-nginx wp-apache + + wait_http "http://localhost:$NGINX_PORT/" || true + wait_http "http://localhost:$APACHE_PORT/" || true + + install_wp wp_nginx "http://localhost:$NGINX_PORT" nginx + install_wp wp_apache "http://localhost:$APACHE_PORT" apache + + echo "== Writing $CONFIG_FILE" + mkdir -p "$(dirname "$CONFIG_FILE")" + EXISTING_TESTRAIL='{}' + [ -f "$CONFIG_FILE" ] && EXISTING_TESTRAIL=$(python3 -c "import json;print(json.dumps(json.load(open('$CONFIG_FILE')).get('testrail',{})))" 2>/dev/null || echo '{}') + python3 - "$CONFIG_FILE" <<PYEOF +import json, sys +config = { + "generated_by": "bin/qa-env.sh", + "environments": { + "nginx": { + "url": "http://localhost:$NGINX_PORT", + "username": "$ADMIN_USER", "password": "$ADMIN_PASS", + "wp_cli": "docker compose -f Tests/e2e/qa-env/docker-compose.yml run --rm --no-deps cli-nginx wp", + }, + "apache": { + "url": "http://localhost:$APACHE_PORT", + "username": "$ADMIN_USER", "password": "$ADMIN_PASS", + "wp_cli": "docker compose -f Tests/e2e/qa-env/docker-compose.yml run --rm --no-deps cli-apache wp", + }, + }, + "imagify": {"api_key": "${IMAGIFY_TESTS_API_KEY:-}"}, + "testrail": json.loads('''$EXISTING_TESTRAIL'''), +} +json.dump(config, open(sys.argv[1], "w"), indent=2) +PYEOF + + echo "== Snapshotting databases" + mkdir -p "$SNAP_DIR" + wp_nginx db export - > "$SNAP_DIR/nginx.sql" + wp_apache db export - > "$SNAP_DIR/apache.sql" + + echo "== QA environments ready" + echo " nginx : http://localhost:$NGINX_PORT ($ADMIN_USER/$ADMIN_PASS)" + echo " apache: http://localhost:$APACHE_PORT ($ADMIN_USER/$ADMIN_PASS)" +} + +cmd_reset() { + require_docker + local target="${1:-all}" + if [ "$target" = nginx ] || [ "$target" = all ]; then + [ -f "$SNAP_DIR/nginx.sql" ] || { echo "ERROR: no nginx snapshot — run 'up' first" >&2; exit 1; } + echo "== Restoring nginx DB snapshot" + wp_nginx db import - < "$SNAP_DIR/nginx.sql" + fi + if [ "$target" = apache ] || [ "$target" = all ]; then + [ -f "$SNAP_DIR/apache.sql" ] || { echo "ERROR: no apache snapshot — run 'up' first" >&2; exit 1; } + echo "== Restoring apache DB snapshot" + wp_apache db import - < "$SNAP_DIR/apache.sql" + fi +} + +cmd_status() { + require_docker + for pair in "nginx:$NGINX_PORT" "apache:$APACHE_PORT"; do + name="${pair%%:*}"; port="${pair##*:}" + code=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:$port/wp-login.php" || echo 000) + echo "$name (http://localhost:$port): HTTP $code" + done +} + +cmd_down() { + require_docker + echo "== Deleting QA environments (containers + volumes)" + dc down -v --remove-orphans + rm -rf "$SNAP_DIR" + echo "== Done. ($CONFIG_FILE left in place — delete it manually if it points at these envs)" +} + +case "${1:-}" in + up) cmd_up ;; + reset) cmd_reset "${2:-all}" ;; + status) cmd_status ;; + down) cmd_down ;; + *) grep '^#' "$0" | head -12 | sed 's/^# \{0,1\}//'; exit 1 ;; +esac From 33e9f37a87770a34dd8746bc1515957de2fd416c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Fri, 3 Jul 2026 02:26:11 +0200 Subject: [PATCH 14/16] fix(qa): harden qa-env + run pipeline from first live execution MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live smoke (run 1283, cases 26359/26362 — both PASS with trace+video) surfaced: - .distignore lacked /.ai, /node_modules, /generatedpackages, /.e2e-screenshots, /.e2e-temp — build-zip.sh was packing 240MB of local artifacts (incl. Canary browser profiles with cookies) into imagify.zip; now 2.9MB - compose mounted ../../generatedpackages (= Tests/generatedpackages) — needs three levels up to repo root - TESTRAIL_OUTPUT_DIR must be $OUT/artifacts, never $OUT: Playwright cleans outputDir at test start and was deleting results.json mid-run (bug existed in the original design too) — run agent + config header updated - DB snapshot/restore moved inside the DB container (mariadb-dump/mariadb over local socket): the wordpress:cli image's newer client refuses non-SSL TCP and WP-CLI doesn't reliably forward ssl flags - qa-env.sh preserves an existing imagify.api_key from the old config when IMAGIFY_TESTS_API_KEY is unset - run agent documents the Node >= 18 requirement (nvm-pinned Node 16 shadows Homebrew's Node on Local-user machines) First two cached case specs committed under Tests/e2e/testrail-cases/ (@status green 2026-07-03), translated per .claude/testrail/translation.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- .claude/agents/testrail-run-agent.md | 21 ++++++--- .distignore | 5 +++ Tests/e2e/qa-env/docker-compose.yml | 8 ++-- Tests/e2e/testrail-cases/case-26359.spec.ts | 50 +++++++++++++++++++++ Tests/e2e/testrail-cases/case-26362.spec.ts | 43 ++++++++++++++++++ Tests/e2e/testrail.config.ts | 11 +++-- bin/qa-env.sh | 10 +++-- 7 files changed, 130 insertions(+), 18 deletions(-) create mode 100644 Tests/e2e/testrail-cases/case-26359.spec.ts create mode 100644 Tests/e2e/testrail-cases/case-26362.spec.ts diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index 717027f8..edd14c0f 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -280,14 +280,23 @@ mkdir -p "$OUT" ( cd Tests/e2e && IMAGIFY_BASE_URL="$E2E_URL" IMAGIFY_ADMIN_USER="$WP_USER" IMAGIFY_ADMIN_PASS="$WP_PASS" \ - TESTRAIL_OUTPUT_DIR="$OUT" \ + TESTRAIL_OUTPUT_DIR="$OUT/artifacts" \ npx playwright test --config=testrail.config.ts "testrail-cases/case-$CASE_ID.spec.ts" \ --reporter=json > "$OUT/results.json" 2> "$OUT/stderr.log" ) ``` -The subshell scopes the `cd` — every other step assumes repo root. **Never `2>&1` into -results.json** — any npm/Node warning would corrupt the JSON; stderr goes to its own file. +The subshell scopes the `cd` — every other step assumes repo root. Two hard-won rules: +- **`TESTRAIL_OUTPUT_DIR` must be `$OUT/artifacts`, never `$OUT` itself** — Playwright + *cleans its outputDir at test start*, so a results.json redirected into the same directory + gets deleted mid-run. Reporter output (results.json, stderr.log) lives at `$OUT` root; + browser evidence lives under `$OUT/artifacts/`. +- **Never `2>&1` into results.json** — any npm/Node warning would corrupt the JSON; stderr + goes to its own file. +- **Node >= 18 is required** (Playwright's bundle uses global fetch/Request). Check + `node --version` once before the first compile gate; if the shell default is older (nvm + pinning Node 16 is common on Local-user machines), prefix `PATH=/opt/homebrew/bin:$PATH` + (or the machine's newer Node) on every playwright invocation. ### 3e — Determine the outcome @@ -308,11 +317,11 @@ Steps-passed / steps-total always **exclude login** — TestRail's own step coun results file and posted comment must match them exactly. **Locate the evidence** — Playwright nests artifacts in a per-test subdirectory of -`outputDir`, so resolve the real paths by glob, never assume `$OUT/trace.zip`: +`outputDir` (`$OUT/artifacts`), so resolve the real paths by glob, never assume a flat path: ```bash -TRACE=$(ls "$OUT"/*/trace.zip 2>/dev/null | head -1) -VIDEO=$(ls "$OUT"/*/*.webm 2>/dev/null | head -1) +TRACE=$(ls "$OUT"/artifacts/*/trace.zip 2>/dev/null | head -1) +VIDEO=$(ls "$OUT"/artifacts/*/*.webm 2>/dev/null | head -1) ``` Record per case: `{ case_id, title, outcome, trace_path, steps_passed, steps_total, elapsed, diff --git a/.distignore b/.distignore index f806b63b..d53af764 100644 --- a/.distignore +++ b/.distignore @@ -1,10 +1,15 @@ /.git /.github /.claude +/.ai /.wordpress-org /bin /_dev /Tests +/node_modules +/generatedpackages +/.e2e-screenshots +/.e2e-temp /.distignore /.editorconfig diff --git a/Tests/e2e/qa-env/docker-compose.yml b/Tests/e2e/qa-env/docker-compose.yml index 0ae8c2b4..4af00a7c 100644 --- a/Tests/e2e/qa-env/docker-compose.yml +++ b/Tests/e2e/qa-env/docker-compose.yml @@ -47,7 +47,7 @@ services: WORDPRESS_DB_NAME: wordpress volumes: - wp-nginx:/var/www/html - - ../../generatedpackages:/packages:ro + - ../../../generatedpackages:/packages:ro web-nginx: image: nginx:1.27-alpine @@ -73,7 +73,7 @@ services: WORDPRESS_DB_NAME: wordpress volumes: - wp-apache:/var/www/html - - ../../generatedpackages:/packages:ro + - ../../../generatedpackages:/packages:ro # WP-CLI sidecars — invoked via `docker compose run --rm cli-nginx wp ...`. # The run agent gets these full command prefixes from the generated @@ -91,7 +91,7 @@ services: WORDPRESS_DB_NAME: wordpress volumes: - wp-nginx:/var/www/html - - ../../generatedpackages:/packages:ro + - ../../../generatedpackages:/packages:ro profiles: ["cli"] cli-apache: @@ -107,7 +107,7 @@ services: WORDPRESS_DB_NAME: wordpress volumes: - wp-apache:/var/www/html - - ../../generatedpackages:/packages:ro + - ../../../generatedpackages:/packages:ro profiles: ["cli"] volumes: diff --git a/Tests/e2e/testrail-cases/case-26359.spec.ts b/Tests/e2e/testrail-cases/case-26359.spec.ts new file mode 100644 index 00000000..7119f864 --- /dev/null +++ b/Tests/e2e/testrail-cases/case-26359.spec.ts @@ -0,0 +1,50 @@ +// @testrail-case: 26359 +// @steps-hash: 1a3a5e30f287 +// @grounding: 542b43d4 +// @status: green 2026-07-03 +// +// Grounding note: TestRail prose names the MCP adapter endpoint, but the grounded spec +// (.claude/testrail/specs/mcp-abilities.md, "How to invoke") establishes the Abilities REST +// discovery endpoint as the observable entry point for C26359 — grounding wins over prose. +import { test, expect } from '@playwright/test'; +import { loginAsAdmin } from '../fixtures/auth'; + +test('TR-26359: MCP - Endpoint discovery - GET returns all 7 ability slugs', async ({ page }) => { + await test.step('log in', async () => { + await loginAsAdmin(page); + }); + + await test.step('Send a GET request to the abilities discovery endpoint as an administrator.', async () => { + const result = await page.evaluate(async () => { + const nonceResp = await fetch('/wp-admin/admin-ajax.php?action=rest-nonce', { credentials: 'same-origin' }); + const nonce = (await nonceResp.text()).trim(); + const resp = await fetch('/wp-json/wp-abilities/v1/abilities', { + headers: { 'X-WP-Nonce': nonce }, + credentials: 'same-origin', + }); + return { status: resp.status, body: await resp.text() }; + }); + + expect.soft(result.status, 'checks: HTTP 200 is returned').toBe(200); + + let slugs: string[] = []; + try { + const data = JSON.parse(result.body); + slugs = Array.isArray(data) ? data.map((a: { name: string }) => a.name) : []; + } catch { + // leave slugs empty — the per-slug assertions below will report the failure + } + const expected = [ + 'imagify/get-account', + 'imagify/get-media-status', + 'imagify/get-nextgen-coverage', + 'imagify/get-settings', + 'imagify/get-stats', + 'imagify/optimize-media', + 'imagify/update-settings', + ]; + for (const slug of expected) { + expect.soft(slugs, `checks: response body lists ability slug ${slug}`).toContain(slug); + } + }); +}); diff --git a/Tests/e2e/testrail-cases/case-26362.spec.ts b/Tests/e2e/testrail-cases/case-26362.spec.ts new file mode 100644 index 00000000..f4c6f50f --- /dev/null +++ b/Tests/e2e/testrail-cases/case-26362.spec.ts @@ -0,0 +1,43 @@ +// @testrail-case: 26362 +// @steps-hash: 90a930142c1f +// @grounding: 542b43d4 +// @status: green 2026-07-03 +// +// Grounded on .claude/testrail/specs/mcp-abilities.md: get-settings is a READ ability — +// GET on the run endpoint (POST returns 405), response strips api_key AND version. +import { test, expect } from '@playwright/test'; +import { loginAsAdmin } from '../fixtures/auth'; + +test('TR-26362: MCP - execute-ability - imagify/get-settings returns settings without api_key or version', async ({ page }) => { + await test.step('log in', async () => { + await loginAsAdmin(page); + }); + + await test.step("Call the execute-ability tool with ability_id: 'imagify/get-settings' and no input args.", async () => { + const result = await page.evaluate(async () => { + const nonceResp = await fetch('/wp-admin/admin-ajax.php?action=rest-nonce', { credentials: 'same-origin' }); + const nonce = (await nonceResp.text()).trim(); + const resp = await fetch('/wp-json/wp-abilities/v1/abilities/imagify/get-settings/run', { + headers: { 'X-WP-Nonce': nonce }, + credentials: 'same-origin', + }); + return { status: resp.status, body: await resp.text() }; + }); + + expect.soft(result.status, 'checks: the get-settings run endpoint returns HTTP 200').toBe(200); + + let settings: Record<string, unknown> = {}; + try { + settings = JSON.parse(result.body); + } catch { + // leave settings empty — the key assertions below will report the failure + } + const keys = Object.keys(settings); + + for (const key of ['optimization_level', 'backup', 'auto_optimize', 'convert_to_webp', 'convert_to_avif', 'optimization_format']) { + expect.soft(keys, `checks: response contains Imagify configuration key ${key}`).toContain(key); + } + expect.soft(keys, 'checks: response does NOT contain api_key').not.toContain('api_key'); + expect.soft(keys, 'checks: response does NOT contain version').not.toContain('version'); + }); +}); diff --git a/Tests/e2e/testrail.config.ts b/Tests/e2e/testrail.config.ts index 996a06b5..fe44c0f7 100644 --- a/Tests/e2e/testrail.config.ts +++ b/Tests/e2e/testrail.config.ts @@ -17,14 +17,17 @@ import { defineConfig, devices } from '@playwright/test'; * ( * cd Tests/e2e && * IMAGIFY_BASE_URL="$E2E_URL" IMAGIFY_ADMIN_USER="$WP_USER" IMAGIFY_ADMIN_PASS="$WP_PASS" \ - * TESTRAIL_OUTPUT_DIR="$OUT" \ + * TESTRAIL_OUTPUT_DIR="$OUT/artifacts" \ * npx playwright test --config=testrail.config.ts "testrail-cases/case-$CASE_ID.spec.ts" \ * --reporter=json > "$OUT/results.json" 2> "$OUT/stderr.log" * ) * - * NOTE on evidence paths: Playwright creates a per-test SUBDIRECTORY inside outputDir — - * trace/video land at `$OUT/<test-result-dir>/trace.zip`, NOT at `$OUT/trace.zip`. Resolve - * them by glob (`ls "$OUT"/*'/'trace.zip`), never by assuming the flat path. + * TESTRAIL_OUTPUT_DIR must be a SUBDIR of $OUT ($OUT/artifacts), never $OUT itself: + * Playwright CLEANS outputDir at test start, so a results.json redirected into the same + * directory is deleted mid-run (verified live 2026-07-03). Playwright then creates a + * per-test subdirectory inside outputDir — trace/video land at + * `$OUT/artifacts/<test-result-dir>/trace.zip`; resolve by glob, never a flat path. + * Node >= 18 is required (Playwright bundles global-fetch-based code). * * HAR capture is intentionally dropped: `use.recordHar` needs a static path per test and * doesn't compose with per-case dynamic naming. The trace.zip contains the network tab in diff --git a/bin/qa-env.sh b/bin/qa-env.sh index 7a3ccedc..5f160d2b 100755 --- a/bin/qa-env.sh +++ b/bin/qa-env.sh @@ -113,8 +113,10 @@ PYEOF echo "== Snapshotting databases" mkdir -p "$SNAP_DIR" - wp_nginx db export - > "$SNAP_DIR/nginx.sql" - wp_apache db export - > "$SNAP_DIR/apache.sql" + # Dump inside the DB container (client == server version, local socket) — the + # wordpress:cli image's newer mariadb client refuses non-SSL TCP connections. + dc exec -T db-nginx sh -c 'exec mariadb-dump -uwordpress -pwordpress wordpress' > "$SNAP_DIR/nginx.sql" + dc exec -T db-apache sh -c 'exec mariadb-dump -uwordpress -pwordpress wordpress' > "$SNAP_DIR/apache.sql" echo "== QA environments ready" echo " nginx : http://localhost:$NGINX_PORT ($ADMIN_USER/$ADMIN_PASS)" @@ -127,12 +129,12 @@ cmd_reset() { if [ "$target" = nginx ] || [ "$target" = all ]; then [ -f "$SNAP_DIR/nginx.sql" ] || { echo "ERROR: no nginx snapshot — run 'up' first" >&2; exit 1; } echo "== Restoring nginx DB snapshot" - wp_nginx db import - < "$SNAP_DIR/nginx.sql" + dc exec -T db-nginx sh -c 'exec mariadb -uwordpress -pwordpress wordpress' < "$SNAP_DIR/nginx.sql" fi if [ "$target" = apache ] || [ "$target" = all ]; then [ -f "$SNAP_DIR/apache.sql" ] || { echo "ERROR: no apache snapshot — run 'up' first" >&2; exit 1; } echo "== Restoring apache DB snapshot" - wp_apache db import - < "$SNAP_DIR/apache.sql" + dc exec -T db-apache sh -c 'exec mariadb -uwordpress -pwordpress wordpress' < "$SNAP_DIR/apache.sql" fi } From dd42fbea7fefdb03d388aeccffb8fd349cca2b62 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Fri, 3 Jul 2026 02:38:05 +0200 Subject: [PATCH 15/16] chore(qa): Docker-only environments, mandatory teardown, merge cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Run agent Step 0 is Docker-only: always provision via bin/qa-env.sh (the generated .ai/settings.local.json is the only config source; wp_path / hand-maintained Local-site support removed everywhere) - New Step 0-end: qa-env.sh down is mandatory at the end of every run (posting done/declined, disposition, or turn-budget stop) — environments never outlive the run; evidence under .ai/ survives teardown - Specs cleaned of pre-Docker environment references: 'Local site shell' → $WPCMD, stale admin/admin creds → loginAsAdmin + generated config, personal machine path removed from mcp-abilities source-of-truth note (re-ground via /testrail-setup mcp-abilities after 2.3.0 merge) - _foundation.md env section rewritten for the ephemeral qa-env Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- .claude/agents/testrail-run-agent.md | 73 +++++++++------------ .claude/skills/testrail-run/SKILL.md | 4 +- .claude/testrail/specs/_foundation.md | 16 +++-- .claude/testrail/specs/mcp-abilities.md | 27 ++++---- .claude/testrail/specs/mixpanel-tracking.md | 16 ++--- .claude/testrail/specs/settings.md | 12 ++-- 6 files changed, 68 insertions(+), 80 deletions(-) diff --git a/.claude/agents/testrail-run-agent.md b/.claude/agents/testrail-run-agent.md index edd14c0f..edaa22d6 100644 --- a/.claude/agents/testrail-run-agent.md +++ b/.claude/agents/testrail-run-agent.md @@ -48,28 +48,27 @@ Results file : .ai/testrail/$RUN_ID/results.md (updated after Doctrine : .claude/testrail/translation.md (read before first translation) Specs dir : .claude/testrail/specs/ (committed grounding: _foundation.md + one file per feature) Hints : .claude/testrail/hints.yml (per-case automation hints captured at scenario time, if present) -Config file : .ai/settings.local.json (gitignored — generated by bin/qa-env.sh, or hand-written for Local sites) +Config file : .ai/settings.local.json (gitignored — GENERATED by bin/qa-env.sh; never hand-written) Env script : bin/qa-env.sh (up / reset / down / status — Docker ephemeral envs) ``` -### Step 0 — Environment: provision or verify (always first) +### Step 0 — Provision the environments (always first — Docker is the only path) -Two supported setups. Decide which applies: +The QA environments are **always** the ephemeral Docker ones managed by `bin/qa-env.sh` — +never a dev's hand-built local site: -1. **If `.ai/settings.local.json` exists**, read it and verify both URLs respond (HTTP 200/30x - on `/wp-login.php`). If they respond → use it (this covers hand-maintained Local sites and - an already-running qa-env). -2. **Otherwise, or if the URLs are dead** → provision the ephemeral Docker environments: - ```bash - bash bin/qa-env.sh up - ``` - This builds `imagify.zip` (`bin/build-zip.sh`), starts one nginx and one apache WordPress - via Docker Compose, installs + activates the zip on both, seeds the Imagify API key (from - `$IMAGIFY_TESTS_API_KEY` if set), **writes `.ai/settings.local.json`**, and snapshots both - databases. If Docker is unavailable and no working config exists, stop and report exactly - what is missing — do not improvise an environment. +```bash +bash bin/qa-env.sh status || true # both URLs HTTP 200 → already up +bash bin/qa-env.sh up # idempotent — provisions only what's missing +``` -Then load the config into shell variables: +`up` builds `imagify.zip` (`bin/build-zip.sh`), starts one nginx and one apache WordPress +via Docker Compose, installs + activates the zip on both, seeds the Imagify API key (from +`$IMAGIFY_TESTS_API_KEY`, falling back to the previous config's key), **writes +`.ai/settings.local.json`**, and snapshots both databases. If Docker is unavailable, stop +and report exactly that — do not improvise an environment. + +Then load the generated config into shell variables: ```bash CONFIG=$(cat .ai/settings.local.json) @@ -78,44 +77,32 @@ j() { echo "$CONFIG" | python3 -c "import sys,json,functools; d=json.load(sys.st TESTRAIL_USERNAME="${TESTRAIL_USERNAME:-$(j testrail.username)}" TESTRAIL_API_KEY="${TESTRAIL_API_KEY:-$(j testrail.api_key)}" -E2E_URL_NGINX=$(j environments.nginx.url) -E2E_URL_APACHE=$(j environments.apache.url) # '' when no apache env exists +E2E_URL_NGINX=$(j environments.nginx.url); E2E_URL_APACHE=$(j environments.apache.url) WP_USER_NGINX=$(j environments.nginx.username); WP_PASS_NGINX=$(j environments.nginx.password) WP_USER_APACHE=$(j environments.apache.username); WP_PASS_APACHE=$(j environments.apache.password) -``` - -**WP-CLI per environment:** each env block carries either `wp_cli` (a full command prefix — -what `bin/qa-env.sh` writes, e.g. `docker compose -f Tests/e2e/qa-env/docker-compose.yml run --rm cli-nginx wp`) -or `wp_path` (Local-style — build the prefix as `wp --path=<wp_path>`). Resolve once: - -```bash -WP_NGINX=$(j environments.nginx.wp_cli); [ -z "$WP_NGINX" ] && WP_NGINX="wp --path=$(j environments.nginx.wp_path)" -WP_APACHE=$(j environments.apache.wp_cli); [ -z "$WP_APACHE" ] && { P=$(j environments.apache.wp_path); [ -n "$P" ] && WP_APACHE="wp --path=$P"; } +WP_NGINX=$(j environments.nginx.wp_cli) # full WP-CLI prefix (docker compose run … cli-nginx wp) +WP_APACHE=$(j environments.apache.wp_cli) ``` Use `$WP_NGINX` / `$WP_APACHE` for **all** seeding/teardown commands — never a bare `wp`. +Never print any API key. -### Step 0b — Per-env setup (skip what qa-env.sh already did) +**Default env is nginx.** Per-case apache routing is decided in 3a-bis (below), before any +generation or execution — there is no "try nginx, discover apache, retry" dance. + +### Step 0-end — Tear down when the run ends (always) -If the config was just generated by `qa-env.sh up`, the plugin is already installed, active, -and keyed — skip to Step 1. For a hand-maintained (Local) config, ensure per env that the -plugin is active and the API key is set — **guarding against a missing apache env**: +The environments are disposable and must **not** outlive the run. As the very last action of +this invocation — after results are posted (Step 6), after the user declines posting, after +the spec disposition, or when stopping on the turn budget — run: ```bash -for ENV in nginx apache; do - WPCMD=$([ "$ENV" = nginx ] && echo "$WP_NGINX" || echo "$WP_APACHE") - [ -n "$WPCMD" ] || continue # no apache env configured — skip, don't crash - $WPCMD plugin activate imagify-plugin 2>/dev/null || $WPCMD plugin activate imagify 2>/dev/null || true - # seed the API key into imagify_settings if a key is available - KEY=$(j imagify.api_key); [ -n "$KEY" ] && $WPCMD option patch update imagify_settings api_key "$KEY" -done +bash bin/qa-env.sh down ``` -If activation fails on an env you need, report the error and stop — never run cases against -an inactive plugin. Never print any API key. - -**Default env is nginx.** Per-case apache routing is decided in 3a-bis (below), before any -generation or execution — there is no "try nginx, discover apache, retry" dance. +Evidence (traces/videos/results.md) lives under `.ai/` on the host and survives teardown; a +resumed run simply re-provisions (Step 0 is idempotent). The only thing `down` costs is the +~2-minute re-`up` — never leave containers running to save it. --- diff --git a/.claude/skills/testrail-run/SKILL.md b/.claude/skills/testrail-run/SKILL.md index 84322bc0..64fd5c69 100644 --- a/.claude/skills/testrail-run/SKILL.md +++ b/.claude/skills/testrail-run/SKILL.md @@ -14,8 +14,8 @@ case-by-case, and — only after the user confirms — posts the results back to proposes CI promotion candidates. The environments are ephemeral Docker containers managed by `bin/qa-env.sh` (one nginx, one -apache — built from `bin/build-zip.sh`'s `imagify.zip`), unless a hand-maintained -`.ai/settings.local.json` already points at reachable Local sites. +apache — built from `bin/build-zip.sh`'s `imagify.zip`). The agent provisions them itself at +the start of the run and **tears them down at the end** — they never outlive the run. ## Invocation diff --git a/.claude/testrail/specs/_foundation.md b/.claude/testrail/specs/_foundation.md index 35ec2239..9feccfbd 100644 --- a/.claude/testrail/specs/_foundation.md +++ b/.claude/testrail/specs/_foundation.md @@ -13,23 +13,25 @@ file plus the per-feature spec before executing any case. This file also absorbs ## Environment -**Never hardcode a URL, port, or credential — everything comes from `.ai/settings.local.json`** -(gitignored; generated by `bin/qa-env.sh up`, or hand-written for Local-site setups). Shape: +The QA environments are the **ephemeral Docker containers** managed by `bin/qa-env.sh` +(one nginx + one apache WordPress, plugin installed from `bin/build-zip.sh`'s zip) — never a +dev's hand-built local site. **Never hardcode a URL, port, or credential — everything comes +from `.ai/settings.local.json`**, which `bin/qa-env.sh up` GENERATES: ```json { "environments": { - "nginx": { "url": "...", "username": "...", "password": "...", "wp_cli": "...", "wp_path": "..." }, - "apache": { "url": "...", "username": "...", "password": "...", "wp_cli": "...", "wp_path": "..." } + "nginx": { "url": "...", "username": "...", "password": "...", "wp_cli": "..." }, + "apache": { "url": "...", "username": "...", "password": "...", "wp_cli": "..." } }, "imagify": { "api_key": "..." }, "testrail": { "username": "...", "api_key": "..." } } ``` -Each env block carries `wp_cli` (full command prefix — qa-env style) **or** `wp_path` -(Local style → prefix is `wp --path=<wp_path>`). All examples below write `$E2E_URL` and -`$WPCMD` — resolve them from the config for the case's target env before use. +`wp_cli` is the full WP-CLI command prefix for that env (a `docker compose run … wp` +wrapper). All examples below write `$E2E_URL` and `$WPCMD` — resolve them from the config +for the case's target env before use. - nginx is the default env; apache is for cases involving `.htaccess`, rewrite rules, or `$is_apache` code paths (flagged per-case via the feature spec's `apache_cases` frontmatter). diff --git a/.claude/testrail/specs/mcp-abilities.md b/.claude/testrail/specs/mcp-abilities.md index a0eeb9b3..c0c419a7 100644 --- a/.claude/testrail/specs/mcp-abilities.md +++ b/.claude/testrail/specs/mcp-abilities.md @@ -17,14 +17,13 @@ Imagify `manage` capability via `imagify_get_context('wp')->current_user_can('ma and `meta.show_in_rest = true`, so they are reachable over the Abilities REST namespace `wp-abilities/v1`. Registration no-ops silently when `wp_register_ability` is absent (WP < 6.9). -SOURCE-OF-TRUTH NOTE (read before grounding drift): the live behaviour below was captured against -the **nginx** site `$E2E_URL`, whose active plugin checkout is -`/Users/gaelrobin/Local Sites/e2eimagifynginx/.../imagify-plugin` on branch `feat/mixpanel` -(SHA 461e5839, plugin v2.2.9 header, but carrying the **2.3.0 abilities** with the -`AbstractAbility` base + `get_id()` template). This worktree (`imagify-e2e-worktree`, SHA 542b43d4) -carries an OLDER variant of the same classes (`implements AbilitiesInterface`, no `get_id()`); the -slugs, schemas, input wrapper, and responses are identical at the REST surface, so the grounding -holds. `derived_sha`/`source_files` track THIS worktree per `_foundation.md` convention. +SOURCE-OF-TRUTH NOTE (read before grounding drift): the live behaviour below was captured +against a `feat/mixpanel` checkout (SHA 461e5839) carrying the **2.3.0 abilities** with the +`AbstractAbility` base + `get_id()` template, while this branch carried an older variant of the +same classes (`implements AbilitiesInterface`, no `get_id()`). The slugs, schemas, input +wrapper, and responses are identical at the REST surface, so the grounding holds — but this +spec should be re-grounded (`/testrail-setup mcp-abilities`) against the qa-env once 2.3.0 is +merged. `derived_sha`/`source_files` track this branch per `_foundation.md` convention. ## Ground truth The 7 abilities (slugs captured live from the discovery endpoint — all kebab-case, no underscores @@ -152,7 +151,7 @@ live on `/wp-admin/options-general.php?page=imagify` (no data-testid on this pag `imagify-plugin/imagify` is active. On first explore this plugin was **inactive** in the capture env and ZERO imagify abilities appeared in discovery. Activate before any case: `POST /wp-json/wp/v2/plugins/imagify-plugin/imagify {"status":"active"}` (auth + nonce), or - `wp plugin activate imagify-plugin` inside the Local site shell. Verify via + `wp plugin activate imagify-plugin` via `$WPCMD`. Verify via `GET /wp-json/wp/v2/plugins/imagify-plugin/imagify` → `status:"active"`. - **WP >= 6.9** (live env is WP 7.0; Abilities API in core `wp-includes/abilities-api.php`). For C26366 (no-op on WP < 6.9): cannot be reproduced on this env (7.0); verify by code/unit — when @@ -161,8 +160,8 @@ live on `/wp-admin/options-general.php?page=imagify` (no data-testid on this pag - **Auth session + nonce** (all cases): cookie jar + `X-WP-Nonce` (see `_foundation.md`). Anonymous = 401 (C26360 expects this). - **Subscriber user** (C26365 — subscriber cannot execute any ability): seed - `wp user create mcp_sub mcp_sub@example.com --role=subscriber --user_pass=pass` inside the Local - shell; log in as that user, acquire a fresh nonce, and call each `…/run` endpoint → expect a + `$WPCMD user create mcp_sub mcp_sub@example.com --role=subscriber --user_pass=pass`; + log in as that user, acquire a fresh nonce, and call each `…/run` endpoint → expect a permission failure (the `manage` gate denies; `has_permission()` returns false and `imagify_mcp_permission_denied` fires). - **imagify_capacity denial** (C26378): hook the `imagify_capacity` filter to return a cap the @@ -175,7 +174,7 @@ live on `/wp-admin/options-general.php?page=imagify` (no data-testid on this pag (`is_api_key_valid:false`), so the zeroed shape is returned. C26361 asserts the FIELD SET (all 7 keys present, types correct) which holds for both valid and invalid key — the invalid-key shape satisfies the schema. To exercise the populated path, configure a valid key first - (`wp option patch update imagify_settings api_key "<valid key>"` in the Local shell) — optional. + (`wp option patch update imagify_settings api_key "<valid key>"` via `$WPCMD`) — optional. - **optimize-media / get-media-status success cases** (C26363, C26374, C26385): require (a) a real attachment AND (b) a VALID API key (optimization calls the Imagify API). The media library on the capture env was EMPTY (live check returned 0 attachments). Seed an attachment via REST @@ -277,7 +276,7 @@ Restore in reverse order of seeding. Each case is independent; undo only what it not, but snapshot anyway): snapshot via `GET …/get-settings/run` BEFORE the case, then restore the changed key via `POST …/update-settings/run` with the original value (e.g. `{"input":{"lossless":0}}` — live pre-test value was 0), or - `wp option update imagify_settings '<json snapshot>' --format=json` in the Local shell. + `wp option update imagify_settings '<json snapshot>' --format=json` via `$WPCMD`. (This explore set then reverted `lossless` to 0; verified clean.) 2. **Restore `optimization_format`** if changed for C26382 (back to `"webp"`). 3. **Delete any seeded attachment** (C26363/C26373/C26374/C26385) and its Imagify meta: @@ -287,7 +286,7 @@ Restore in reverse order of seeding. Each case is independent; undo only what it `wp eval` hook so the admin regains `manage`. 5. **Delete the subscriber test user** (C26365): `wp user delete mcp_sub --yes --reassign=1`. 6. **Remove a valid API key** if one was configured only for the success cases: - `wp option patch update imagify_settings api_key ""` (Local shell). + `wp option patch update imagify_settings api_key ""` (via `$WPCMD`). 7. **Plugin activation:** the explore left `imagify-plugin` ACTIVE (the run agent requires it active for every case). If the run protocol requires restoring the pre-run inactive state, `POST /wp-json/wp/v2/plugins/imagify-plugin/imagify {"status":"inactive"}` LAST — but note this diff --git a/.claude/testrail/specs/mixpanel-tracking.md b/.claude/testrail/specs/mixpanel-tracking.md index 42a43d6e..1a40c4fd 100644 --- a/.claude/testrail/specs/mixpanel-tracking.md +++ b/.claude/testrail/specs/mixpanel-tracking.md @@ -92,7 +92,7 @@ Captured live this explore unless noted. Mixpanel token (`ServiceProvider::MIXPA actions fired in `AbstractAbility` (`check_permissions()` + `do_execute` wrapper). ## Locators (captured live — role-based preferred, then data-testid, then id) -Verified live this explore on `/wp-admin/options-general.php?page=imagify` (login admin/admin; +Verified live this explore on `/wp-admin/options-general.php?page=imagify` (logged in via `loginAsAdmin` — creds from the generated config; reuse the `SettingsPage` POM at `Tests/e2e/pages/settings.ts` + `wp-login` fixture). No `data-testid` on this UI; fall back to id/class. The opt-in checkbox was **unchecked** on a fresh load (matches "unchecked by default"). @@ -142,11 +142,11 @@ Mixpanel-request spy — because events are sent server-side and never reach the event name + properties, records them (option/transient), and returns a fake 200 to block the real call. This explore verified the exact path: a `get-stats` MCP run produced one `MCP Ability Executed` event with all documented props. Expose captured events to the test via a - small admin-only REST route (e.g. `GET /wp-json/imagify-spy/v1/events`) or read the option in the - Local site shell. Reset before each case (`DELETE` the spy option). Teardown: delete the mu-plugin. + small admin-only REST route (e.g. `GET /wp-json/imagify-spy/v1/events`) or read the option via + `$WPCMD`. Reset before each case (`DELETE` the spy option). Teardown: delete the mu-plugin. - **Opt-in ON** (events should fire): POST the AJAX toggle `value=1` with the live `data-nonce` - (preferred — exercises the real path), or in the Local shell `wp option update imagify_mixpanel_optin 1`. -- **Opt-in OFF** (events must NOT fire): toggle `value=0`, or in the Local shell + (preferred — exercises the real path), or via `$WPCMD` — `wp option update imagify_mixpanel_optin 1`. +- **Opt-in OFF** (events must NOT fire): toggle `value=0`, or via `$WPCMD` `wp option delete imagify_mixpanel_optin`. Default state has the option absent. - **`Media Optimized` (UI, trigger=auto)**: opt-in ON, `auto_optimize=1`, upload a fresh supported image (REST media helper in `_foundation.md`) so `imagify_after_optimize` runs with the full size. @@ -243,9 +243,9 @@ Restore in reverse order of seeding; each case undoes only what it changed. 4. Delete any non-admin test user seeded for the permission-denied case (`wp user delete <login> --yes --reassign=1`). 5. Remove the Mixpanel-spy mu-plugin (`wp-content/mu-plugins/<spy>.php`) and delete its capture - option/transient. (NOTE: this explore left an orphaned `imagify_spy_events` option in the - localhost:10043 DB after removing the spy mu-plugin; it is inert — no code reads it — but can be - cleared with `wp option delete imagify_spy_events` in the Local site shell.) + option/transient. (An orphaned `imagify_spy_events` option is inert — no code reads it — but + can be cleared with `$WPCMD option delete imagify_spy_events`. On the ephemeral qa-env this + is moot: the whole DB is discarded at teardown.) 6. Clear any leftover `imagify_analytics_optin_thanks` transient if a thank-you case was interrupted before the notice rendered (`wp transient delete imagify_analytics_optin_thanks`); normally it is read-once and self-clears. diff --git a/.claude/testrail/specs/settings.md b/.claude/testrail/specs/settings.md index faaf0b2c..bb9a46c8 100644 --- a/.claude/testrail/specs/settings.md +++ b/.claude/testrail/specs/settings.md @@ -64,7 +64,7 @@ via `imagify_get_context('wp')->current_user_can('manage')` (see `_foundation.md nonce + cookie per `_foundation.md`. The run href is the ability's `_links["wp:action-run"]`. ## Locators (captured live — role-based preferred, then data-testid, then id) -Locators verified live this explore on the settings page (login admin/admin first; reuse the +Locators verified live this explore on the settings page (log in via `loginAsAdmin` first — creds from the generated config; reuse the `SettingsPage` POM at `Tests/e2e/pages/settings.ts` and `wp-login` fixture). No data-testid attributes exist on this page; fall back is id/name. @@ -111,9 +111,9 @@ attributes exist on this page; fall back is id/name. (set .htaccess to 444 then enable `display_nextgen` + Save — see `htaccess-notice-dedup.spec.ts`), which stores a notice whose message wraps a file path in `<code>`. - C174 (multisite, no fatal on settings change): **BLOCKED in this env** — requires a WordPress - **multisite** install with the plugin network-active. This Local site (test-temp) is single - site (the network admin URL `/wp-admin/network/settings.php?page=imagify` does not exist here). - Seeding a multisite is out of scope for the live env; document as a prerequisite and skip. + **multisite** install with the plugin network-active. The qa-env sites are single-site (the + network admin URL `/wp-admin/network/settings.php?page=imagify` does not exist there). + Document as a prerequisite and skip until a multisite qa-env variant exists. ## Verification criteria — "success" means (observable) - C155 (Auto-optimize ON): after toggling ON + Save, the page reloads to `?page=imagify` with @@ -141,12 +141,12 @@ attributes exist on this page; fall back is id/name. Restore in reverse order of seeding. Each case is independent; undo only what it changed. 1. Delete any attachment seeded for C155/C156 (and its Imagify meta): - `wp post meta delete <id> _imagify_data` ; `wp post meta delete <id> _imagify_status` - (inside the Local site shell), or delete the attachment entirely via REST: + (via `$WPCMD`), or delete the attachment entirely via REST: `DELETE /wp-json/wp/v2/media/<id>?force=true` (auth + nonce — see `_foundation.md`). 2. Restore the settings snapshot taken in "Prerequisites & seeding": - `POST /wp-json/wp-abilities/v1/abilities/imagify/update-settings/run` with the snapshotted keys (in particular restore `auto_optimize` to its pre-test value — live value was **1**), - or `wp option update imagify_settings '<json snapshot>' --format=json` inside the Local shell. + or `wp option update imagify_settings '<json snapshot>' --format=json` via `$WPCMD`. 3. C14169: if a temporary notice was triggered via the .htaccess path, restore `.htaccess` permissions (e.g. back to 644) and turn `display_nextgen` back to its prior value (UI uncheck + Save). The temporary-notices transient is read-once and auto-cleared, so no From ac32e5708f53cc4ca460e211c66dbf2e263280cd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ga=C3=ABl=20Robin?= <robin.gael@gmail.com> Date: Fri, 3 Jul 2026 03:03:46 +0200 Subject: [PATCH 16/16] fix(phpstan): drop unmatched @phpstan-ignore-line in Media\WP::generate_thumbnails MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Newer wordpress-stubs no longer flag the wp-admin/includes/image.php require_once, and PHPStan treats an unmatched ignore as an error (ignore.unmatchedLine, non-ignorable). composer.lock is gitignored, so CI installs fresh deps — this was about to start failing on develop too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --- classes/Media/WP.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/classes/Media/WP.php b/classes/Media/WP.php index c2c8ac4e..5c8cebc1 100644 --- a/classes/Media/WP.php +++ b/classes/Media/WP.php @@ -211,7 +211,7 @@ public function generate_thumbnails() { } if ( ! function_exists( 'wp_generate_attachment_metadata' ) ) { - require_once ABSPATH . 'wp-admin/includes/image.php'; // @phpstan-ignore-line + require_once ABSPATH . 'wp-admin/includes/image.php'; } // Store the path to the current full size file before generating the thumbnails.