ci: wait out the stopping-state warm-pool race; move nightly drain to 23:07 UTC

[kurok]

What

Two changes to the warm-pool runner lifecycle, plus a serialization audit:

1. Close the stopping-state race (workflow side)

The previous run's stop-runner returns while its instance is still stopping (the action's StopInstances call is fire-and-forget), but every warm-pool lookup filters instance-state-name=stopped. A run queued tightly behind it saw an "empty" pool and cold-launched a duplicate instance that nothing drained until the nightly pass.

Log-proven on 2026-07-10 (UTC): run A's stop-runner logged is stopped (warm pool) 0.2 ms after StopInstances and exited 08:45:51 → run B's queue released 08:45:55 → 08:46:04 {"step":"warm_start","outcome":"pool_empty"} → second instance; every later run warned Found 2 stopped warm-pool instances (…); the reaper is not draining them. Same two-instance pattern on 2026-07-09.

The "Pre-attach sandbox EIP" step now also matches stopping and runs aws ec2 wait instance-stopped before the action's own lookup (preferring an already-stopped instance when both exist; falling back to a cold launch with a warning if the wait times out). namecheap/ec2-github-runner#69 fixes the same race action-side (mode: stop waits for stopped); this consumer-side belt also covers older action pins. After #69 merges, bumping the action pin here makes the pair complete.

2. Nightly drain at 23:07 UTC instead of 02:37 UTC

End-of-day destruction should happen at end of day. GitHub delays these crons by hours (the 2026-07-10 "02:37" drain actually fired at 06:27), so the earlier slot also buys headroom: even a badly delayed drain lands before the next workday. :07 offset keeps the top-of-hour-congestion convention the leak reaper already follows (the file documents it). The Determine pass cron-string comparison and the SECURITY_COMPLIANCE.md reference are updated byte-for-byte, as the workflow comment requires.

3. Single-runner / queueing audit (no change needed)

Verified from the same logs: only EC2-bound runs enter the FIFO queue (start-runner's gate excludes forks/Dependabot/branch pushes; mock acceptance stays parallel on hosted runners), the queue held run B until run A fully completed (waiting for run #29080496277: current status: in_progress), and no two runs ever held the runner concurrently. The only defect was the boundary itself — "run finished" preceded "instance reusable" — which items 1 and #69 close from both sides.

Validation

• actionlint clean on both workflows; YAML parses.
• Warm path: next same-day CI pair should log Pre-attached … to stopped warm-pool instance on the second run with no Found 2 warning.
• Drain: first scheduled fire after merge should show Determine pass → kind=nightly-drain from EVENT_SCHEDULE=7 23 * * * and terminate the day's instance; workflow_dispatch with mode: nightly-drain, dry_run: true previews it safely.

[github-actions]

🔐 Security scan summary

Commit ed1de6e4 · branch ci/warm-pool-stopping-race · generated 2026-07-10 10:34 UTC

Scan	Gate	Findings
Trivy — vulnerabilities (SCA)	✅	🔴 0 critical · 🟠 0 high · 🟡 0 medium · ⚪ 0 low · 1 unknown
Trivy — IaC misconfig	✅	0 failing
Trivy — secrets	✅	0 detected
Trivy — licenses	✅	0 policy finding(s)
govulncheck (reachability)	✅	0 reachable · 0 imported-only
CodeQL (code scanning, open)	—	0 open · 🔴 0 critical · 🟠 0 high

Gate (blocks merge): HIGH/CRITICAL fixable CVEs · reachable Go vulnerabilities · forbidden licenses · committed secrets. Counts above are the full picture (incl. medium/low and unfixed), so they can exceed what actually gates. CodeQL is informational here. ✅ = gate job passed, ❌ = failed, ⚪ = skipped.

📦 Dependencies

• SBOM components (CycloneDX): 64
• Go modules in go.mod: 6 direct · 56 indirect

Vulnerabilities (Trivy, top 50)

Severity	ID	Package	Installed	Fixed
UNKNOWN	GO-2026-5932	golang.org/x/crypto	v0.53.0	—

---

📎 Full machine-readable reports are attached to this run as artifacts: security-summary, trivy-json, govulncheck-json, sbom-cyclonedx.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.22%. Comparing base (517b525) to head (ed1de6e).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #290   +/-   ##
=======================================
  Coverage   89.22%   89.22%           
=======================================
  Files          15       15           
  Lines        1643     1643           
=======================================
  Hits         1466     1466           
  Misses        111      111           
  Partials       66       66           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.