Capacity telemetry: record runner/host utilization to right-size fleet capacity

[Nickfost]

Feature Description

Record per-runner and host resource utilization (CPU, memory, disk, inodes) during job execution, retain a rolling history, and surface a weekly aggregate so capacity decisions (max_runners, per-runner size, pool budget) are driven by data instead of guesswork.

Motivation

VM 130 (32 CPU / 64 GB) currently runs 6 runners × (4 CPU / 8 GB) — a reviewed but unmeasured number. Whether the right ceiling is 6, 7, or 8 runners, and whether per-runner RAM should be 6 GB or 8 GB, is unknown without utilization history. The same question recurs for every future host. Today the fleet only has point-in-time health checks and one-off proof-run snapshots; there is no time-series data.

The 6-runner concurrency proof and the first weeks of real application load are exactly the data needed to make the next capacity change confidently — but only if it's recorded.

Proposed Solution

Cheap version first, upgrade path noted:

1. Extend the existing host health collector to snapshot CPU/RAM/swap/disk/inodes and per-runner-container stats while any runner is active (samples already exist in proof-run tooling).
2. Persist samples locally (append-only JSON or SQLite on the host), redacted per existing health-report rules.
3. Aggregate weekly (p50/p95 per metric, per pool) into the health report or a small CLI report.
4. Use that report as required evidence in any rd-delivery-config PR that changes max_runners, per-runner resources, or pool budget.

Upgrade path if this outgrows the collector: node_exporter + Prometheus scraping the host, same aggregation contract. Do not build this first.

Non-goals

• No telemetry off the host. This stays local; it is operational evidence, not phone-home metrics.
• No per-job content recording (no logs, env, source — resource counters only), consistent with existing health-report redaction rules.

Acceptance

• Host records utilization samples automatically whenever runners are active
• Weekly p50/p95 CPU/RAM/disk report available on-host
• Report contains no job content or secrets
• One rd-delivery-config capacity PR cites the report as evidence

[Nickfost]

[lane-a:phase2:issue-50-outcome]

Repository implementation is clean at draft PR #73 head 71f56fb85bd2281ad669edaaf2fda6009cbca5f8: exact-head CI passed, Codex reported no major issues, all 23 findings have evidence replies and +1, and no useful thread remains unresolved.

The issue remains open for a reviewed capacity-planning configuration change and separately authorized acceptance evidence. No live capacity, host, runner, or private configuration was changed.