test(opal-server): git leak/resilience test environment (PR1)

[dshoen619]

PR1 — Git Leak/Resilience Test Environment

Builds the test bed and the single diagnostics hook needed to exercise the git-fetcher memory leak, the offline-repo hang, the slow serial boot, and the broadcaster-disconnect path. The leak/offline tests fail on master and become the regression gates for the follow-up fix PRs; the boot and bounce tests are tunable/recovery guards (see below).

This is intentionally one large PR — it's the foundation every later PR's gate depends on. The actual fixes land in the PRs this one blocks:

• PER-15156 (PR2) — memory-leak fix → gated by test WIP: Opal refactor (policy fetcher, tree structure) #1 (test_churn_releases_caches); Data fetcher #2 is an RSS guard, not a leak gate
• PER-15157 (PR3) — offline-repo resilience → gated by test Update and fetch #4
• PER-15065 — broadcaster in-process reconnect → relates to test remove singletons from server #5
• PR4 (boot parallelism) → gated by test Refactor: Opal Server, Repo Watcher, Cloner, Tracker, LeaderElection, BundleMaker, and bundle diffs #3

Linear: PER-15155

opal-server: an off-by-default stats endpoint

• opal_server/debug_stats.py — read-only git_fetcher_cache_stats() (sizes of the three process-global GitPolicyFetcher caches + process RSS) and a register_internal_stats_route() registrar.
• opal_server/config.py — new OPAL_DEBUG_INTERNAL_STATS flag, default False.
• opal_server/server.py — mounts GET /internal/git-fetcher-cache-stats beside /healthcheck, gated by the flag.
• Unit tests for the helper, the flag default, and the gating.

No production behavior change when the flag is off (the default).

app-tests/git-leak/: a self-contained stack + pytest harness

• docker-compose.yml — opal-server (single worker, OPAL_SCOPES=1, Postgres broadcaster) built from the repo's own docker/Dockerfile server target, plus Redis, Postgres, Gitea, and one-shot gitea-admin + seed sidecars.
• seed/ — idempotent Gitea seeding sidecar (N policy repos) + Dockerfile.
• helpers.py — OpalServerClient, GiteaAdmin (host-side admin client), make_repo_unreachable, bounce_postgres, compose.
• conftest.py — session-scoped stack + opal + gitea_admin fixtures; --boot-scopes=N, --keep-stack.
• test_leak.py (WIP: Opal refactor (policy fetcher, tree structure) #1 churn, Data fetcher #2 repeat-sync), test_resilience.py (Update and fetch #4 offline repo, remove singletons from server #5 postgres bounce), test_boot.py (Refactor: Opal Server, Repo Watcher, Cloner, Tracker, LeaderElection, BundleMaker, and bundle diffs #3 boot timing, tunable BOOT_TARGET_SECONDS).
• README.md.

The five flagship tests, and their baseline behavior on master

#	Test	On master	Gate
1	test_churn_releases_caches	FAILS (caches never drain)	PR2
2	test_repeat_sync_rss_stays_bounded	PASSES (RSS guard; the URL-keyed cache count can't grow for any impl)	RSS guard
3	test_boot_loads_all_scopes	passes; FAILS with BOOT_TARGET_SECONDS low	PR4
4	test_offline_repo_does_not_block_healthy_scopes	FAILS (no fetch timeout)	PR3
5	test_server_recovers_after_postgres_bounce	PASSES (2-worker; worker-PID-unchanged across bounce = in-place reconnect)	PER-15065

Test #5 — verified (2-worker, PID guard). Runs 2 workers (OPAL_TEST_WORKERS) so the Postgres backbone is actually fanned out cross-worker (debug-pubsub.md §3-4); across a transient bounce it asserts the gunicorn worker PIDs are unchanged — the in-place-reconnect signal that distinguishes #915 (PER-15065) from a plain gunicorn respawn — plus a servable post-bounce scope (GET /scopes/{id}/policy == 200, not /internal counts, which are per-process on 2 workers). Re-validated live on this branch: passes (~54s). Updated per the gate-coverage review — see the latest reply for the full fail/pass-for-the-right-reason matrix.

How to run

cd app-tests/git-leak
python -m pytest -v --boot-scopes=50              # full set
python -m pytest test_leak.py -v --boot-scopes=20 # just the leak gates

Requires Docker + compose v2 and host Python with pytest pytest-timeout requests GitPython.

Verification done

• All 4 opal-server unit tests pass inside the repo's Docker server image; import opal_server.server clean; flag defaults False with a description.
• docker compose config -q valid; all 5 flagship tests collect.
• Full-stack smoke: stack boots, Gitea seeds repos, opal healthy, /internal/git-fetcher-cache-stats live, PUT /scopes returns 201 with auth disabled.
• GiteaAdmin exercised live (list/exists/create/delete over the published port); test_server_recovers_after_postgres_bounce run to a green pass.

Notes for reviewers

• Auth disabled deliberately: OPAL_AUTH_PUBLIC_KEY left unset → JWT verifier disabled so the harness can drive scope routes without minting JWTs (require_peer_type becomes a no-op). Test bed only.
• Scope create body sets auth: {auth_type: "none"} (required pydantic-v1 discriminated union).
• Host ports: opal_server:7002 and gitea:13000 (uncommon, to avoid the :3000 clash; used by GiteaAdmin); Postgres is internal-only.

Cache-read determinism (how the gates stay trustworthy)

Two properties of master would make naive cache reads non-deterministic. Both are handled in this harness — they are not open issues:

1. Per-worker caches + round-robin reads. The GitPolicyFetcher caches are per-process, so with >1 worker a round-robin /internal read can miss the worker that fetched and report 0. Resolved: the stack runs a single uvicorn worker (UVICORN_NUM_WORKERS=1), so every cache read is deterministic.
2. repos populates on the second sync, not the first. The first-sync clone path fills only repo_locks; repos / repos_last_fetched are filled by the discover/fetch path on a subsequent sync. Resolved: the load helpers issue refresh_all() and wait before asserting on repos, so the gates assert against a cache the sync path has actually filled.

These were called out as caveats in the original design doc; the implementation above closes both, so the leak/drain assertions are deterministic single-worker.

🤖 Generated with Claude Code

[linear-code]

PER-15155

[netlify]

✅ Deploy Preview for opal-docs ready!

Name	Link
🔨 Latest commit	9b61027
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/6a5383a26d07050008ddf2bd
😎 Deploy Preview	https://deploy-preview-922--opal-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

Adds an off-by-default OPAL server internal diagnostics endpoint for git-fetcher cache/RSS stats, and introduces a docker-compose-based “git leak/resilience” integration testbed intended to reproduce/regress several production issues (leak, offline repo hang, boot slowness, broadcaster disconnect recovery).

Changes:

• Add /internal/git-fetcher-cache-stats endpoint gated by OPAL_DEBUG_INTERNAL_STATS (default off), plus unit tests.
• Add app-tests/git-leak/ docker-compose stack (opal-server + redis/postgres/gitea) and pytest harness (boot/leak/resilience tests).
• Add .claude/ to .gitignore.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
packages/opal-server/opal_server/debug_stats.py	New helper to read git-fetcher cache sizes + RSS and register a gated internal route.
packages/opal-server/opal_server/config.py	Adds DEBUG_INTERNAL_STATS config flag (default False).
packages/opal-server/opal_server/server.py	Mounts the internal stats route when enabled.
packages/opal-server/opal_server/tests/debug_stats_test.py	Unit tests for stats dict sizing + flag default.
packages/opal-server/opal_server/tests/debug_stats_endpoint_test.py	Unit tests for endpoint presence/absence when gated.
app-tests/git-leak/docker-compose.yml	Compose stack for OPAL + dependencies + seeding sidecars.
app-tests/git-leak/helpers.py	Host-side HTTP/compose helpers for the test harness.
app-tests/git-leak/conftest.py	Pytest fixtures to boot/teardown the stack and provide clients.
app-tests/git-leak/test_leak.py	Leak regression tests (churn + repeat sync).
app-tests/git-leak/test_resilience.py	Offline-repo hang test + Postgres bounce recovery guard.
app-tests/git-leak/test_boot.py	Boot timing/baseline test.
app-tests/git-leak/seed/seed_gitea.py	Idempotent Gitea seeding script for N repos.
app-tests/git-leak/seed/Dockerfile	Container image for the seeding sidecar.
app-tests/git-leak/README.md	Documentation for running the new testbed locally.
.gitignore	Ignores .claude/ working artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zeevmoney]

Requesting changes — the regression gates do not function as gates yet

Scope check first, because it's the reassuring part: the production surface is safe. The only always-on change is a benign import plus a default-off OPAL_DEBUG_INTERNAL_STATS config key; with the flag off (the default) there is zero behavior change, and when on the handler can't crash (the class dicts always exist, len() is GIL-atomic against the event-loop mutations, and _read_rss_kb swallows errors). testpaths = packages drops no existing tests. Git hygiene is clean (nothing private leaked into the diff, no rebase needed). This PR will not crash production.

The problem is the actual deliverable — the five flagship tests meant to gate PR2–PR5.

What's blocking (must fix before this can be trusted as the gate)

1. The load/drain gates assert on GitPolicyFetcher.repos, which the first-sync clone path never populates (see the CRITICAL inline comment on test_leak.py:26). On a fresh scope, fetch_and_notify_on_changes takes the _clone() branch, which only sets repo_locks — repos is filled solely by _get_repo() on a second sync or the bundle path, and periodic re-sync is disabled in the stack (OPAL_POLICY_REFRESH_INTERVAL: "0"). As a result test_churn_releases_caches, test_repeat_sync_does_not_grow, and test_offline_repo_does_not_block_healthy_scopes hang at their _wait_until(repos >= n) load gate for the full timeout and then fail at the load stage — they never reach the leak/offline logic they exist to test, and they would not flip green after PR2/PR3 land. Only test_boot_loads_all_scopes works, and only incidentally (compose restart → preload_scopes re-discovers the on-disk clones). This is the same root cause the PR description flags as "Known caveat #2" — it needs to be fixed, not just noted.

2. Even once load is fixed, the cache assertions are not reliable on a 2-worker stack. stats()'s max-merge cannot prevent a == 0 drain assertion from falsely passing when the samples miss the leader worker (HIGH on helpers.py:50). The cache tests should run single-worker, or the endpoint should aggregate across workers.

3. Failure modes are invisible. compose() swallows stdout/stderr on any compose failure (HIGH on helpers.py:218), and partial seeding is never detected before the suite runs (HIGH on conftest.py:40) — so a broken stack or a half-seeded Gitea looks like a test failure for the wrong reason.

Secondary correctness (should fix)

• test_repeat_sync_does_not_grow is tautological — len(repos) can't grow on repeat URL-keyed sync (test_leak.py:64).
• The offline test's TEST-NET-1 address likely fails fast instead of hanging, so it may not exercise starvation (helpers.py:205).
• The opal fixture clears _created_scopes at setup, orphaning scopes from a failed prior test and contaminating the next (conftest.py:55).
• The postgres-bounce test asserts only HTTP liveness, not broadcaster recovery, and doesn't --wait for Postgres (test_resilience.py:51).
• README/docstrings say "fails on master," but the suite requires this PR's /internal endpoint (README.md:30).
• Dead/misleading 404 branch in refresh_all() (helpers.py:128).

Minor

• Enabling the flag exposes an unauthenticated /internal route (non-sensitive payload, off by default) (debug_stats.py:39).
• || true masks genuine gitea-admin failures (docker-compose.yml:58).
• Boot-timing clock starts after wait_healthy, undercounting boot-sync time (test_boot.py:24).

Bottom line

All 11 plan tasks are implemented and the production change is safe, but the suite needs to (1) re-key the load/drain assertions to a cache the sync path actually populates (or force a policy fetch), (2) make the cache reads worker-deterministic, (3) surface compose/seed failures, and then (4) be validated end-to-end against a live stack to confirm each test fails/passes for the right reason. Until then it can't be relied on as the regression gate for PR2–PR5.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

[dshoen619]

Review addressed — 6046a10

Thanks for the thorough pass. All blocking items, both HIGHs from @Zivxx / @zeevmoney, and the secondary/minor items are addressed in 6046a10. Per-comment replies are inline; summary below.

Root cause behind most findings (and why the gates didn't gate): a fresh scope's first sync takes the _clone() branch, which only fills repo_locks; repos/repos_last_fetched are filled on a second sync. The load gates on repos therefore hung, and the 2-worker per-process caches made == 0 drains unsafe.

Blocking

1. Load gate (CRITICAL + @Zivxx HIGH) — _load_scopes gates on repo_locks (first-sync signal), then refresh_all() forces the second sync so repos/repos_last_fetched are populated before any drain/purge assertion. Churn now asserts all three caches drain to 0 (so a broken repos_last_fetched purge fails instead of passing vacuously).
2. Worker determinism — stack is now UVICORN_NUM_WORKERS=1; removes the false-== 0-drain class outright (no more reliance on max-merge to paper over it).
3. Failure visibility — compose() re-raises with captured stdout/stderr; seed completeness is asserted in conftest; seed_gitea.py isolates per-repo failures and exits non-zero with a count.

Secondary

• test_repeat_sync now asserts an rss_kb bound (count can't grow for a URL-keyed set).
• Offline test uses a blackhole socat sidecar (deterministic hang, verified) and saturates the fetch executor with 40 hung clones; recovers the session stack via hard_reset() (stop → redis FLUSHALL → start).
• Per-test clean slate deletes all server scopes (fixes the orphan-scope leak).
• Postgres-bounce proves broadcast recovery (PUT post-bounce → assert sync), uses up -d --wait.
• Dead 404 branch removed; boot clock starts at restart; gitea-admin || true → "already exists"-only guard; README/docstrings reworded to "fails on this branch without PR2/PR3", noting the suite needs this PR's /internal endpoint.

Minor

• /internal route now carries the JWTAuthenticator dependency (protected when JWT on, no-op in the test bed); added a unit test asserting a rejecting dependency → 401.

Validation done: 5/5 opal-server unit tests pass in the Docker server image (incl. the new auth test); docker compose config valid; blackhole hang and the gitea-admin guard verified directly.

Still outstanding (flagging honestly): I have not yet re-run the full live integration suite end-to-end to confirm each flagship test fails/passes for the right reason against a live stack — that's the "validate end-to-end" item from your bottom line. I can kick that off next; it's a ~30 min run and several tests are expected-fail gates on this branch until PR2/PR3.

approvded

[zeevmoney]

Approving — the test bed is sound and this PR has already been through several review rounds (Copilot / @Zivxx / @zeevmoney) with the substantive issues raised and resolved. A fresh python-pro pass (planned against the opal-development references) surfaced only non-blocking items; none are CRITICAL/HIGH, so they don't gate landing the test bed.

Issues to fix (tracked under PER-15155) — inline comments above:

Worth fixing before/with merge

• MEDIUM pytest.ini:10 — testpaths = packages makes the README's documented cd app-tests/git-leak && pytest --boot-scopes=50 error with unrecognized arguments: --boot-scopes. Self-root the suite + use a targeted ignore.
• MEDIUM test_resilience.py:123 — the bounce test's repo_locks > baseline signal is order-dependent across files (shared policy-repo-0000 + non-purging caches on master); assert GET /scopes/post-bounce/policy == 200 instead.

Follow-ups (LOW)

• helpers.py:274 — add a timeout= to compose() so a wedged setup fails fast instead of hitting the CI job limit.
• helpers.py:148 — delete_all_scopes burns ~40s/test waiting on a drain that can't happen on master; short-circuit it.
• seed_gitea.py:159 — remove the unused /seed-output/token artifact + volume.
• seed_gitea.py:108 — make push_url credential injection scheme-agnostic (urllib.parse).

Doc nit (not posted inline): the bounce-test docstring (test_resilience.py:85-93) still describes pre-PER-15065 "graceful shutdown + gunicorn respawn over the recovered broadcaster"; reconnect is now in-place — worth a 3-line update.

Recommend merging PR1 first so PR2 (#923) / PR3 (#924) can build on the test bed + /internal endpoint, then addressing the two MEDIUMs.

[dshoen619]

Gate coverage lifted — 8e24cb0b

Thanks, the gate-coverage matrix was exactly the right lens. You're right that as committed only #1/#4 actually gated. Addressed all three suggestions; left #1 churn and #4 offline (the load-bearing gates) untouched.

#5 broadcaster — now a real PER-15065 guard. A new opal_multiworker fixture recreates opal_server with 2 workers (UVICORN_NUM_WORKERS: ${OPAL_TEST_WORKERS:-1}) so the Postgres backbone is actually fanned out cross-worker (debug-pubsub.md §3-4), then restores single-worker on teardown so the cache gates keep their determinism. The test now asserts the gunicorn worker PIDs are unchanged across the bounce — the in-place-reconnect signal that tells #915 apart from a plain respawn. With the retry-forever default a transient blip never gives up, so the worker keeps its PID; a revert of #915 would escalate to a graceful shutdown → gunicorn respawn → new PID. Recovery is proven via a servable post-bounce scope (GET /scopes/{id}/policy == 200), not /internal counts (per-process, non-deterministic on 2 workers). worker_pids() reads /proc and matches host-side so the scan can't count its own sh -c wrapper (whose command line literally contains "gunicorn").

#3 boot — measures the right thing. Completion now keys on all scopes served (GET /scopes/{id}/policy == 200) instead of repo_locks (set at fetch start, so it undercut the final clone). Default BOOT_TARGET_SECONDS stays loose (baseline recorder); the docstring + README carry forward that PR4 must run it with a tight target (120s @ 50).

#2 repeat-sync — relabeled. Renamed test_repeat_sync_does_not_grow → test_repeat_sync_rss_stays_bounded; dropped the tautological len(repos) assertion (with a comment so it isn't re-added); RSS is the sole gate.

Live validation (--boot-scopes=50, this branch):

Test	Result	Reason
#3 boot	✅ PASS	served 50/50 in 7.4s (all-served probe)
#2 rss	✅ PASS	RSS guard
#1 churn	❌ FAIL (right reason)	caches stay at {repo_locks:50, repos:50, repos_last_fetched:50} — PR2 not landed
#4 offline	❌ FAIL (right reason)	healthy probe ReadTimeout while 40 clones hang — PR3 not landed
#5 broadcaster	✅ PASS	2 workers, worker PIDs unchanged across bounce, post-bounce scope served

Net: gates PR2 + PR3 today, PR5 is now guarded (in-place reconnect), and PR4 is ready to gate the moment its branch sets a tight BOOT_TARGET_SECONDS.

[zeevmoney]

Re-review (PER-15155) — post-approval commits

What this PR does. Adds the app-tests/git-leak/ docker-compose test bed (Gitea seed + blackhole + redis + postgres + opal_server) plus three regression suites — boot-sync timing (test_boot.py), cache-drain / RSS leak (test_leak.py), and offline-repo / postgres-bounce resilience (test_resilience.py) — that are red on master by design and turn green as PR2–PR5 land. It also adds one production surface those tests observe through: the off-by-default GET /internal/git-fetcher-cache-stats endpoint (debug_stats.py), its OPAL_DEBUG_INTERNAL_STATS config key (default False), and its wiring in server.py, plus two collected unit tests. Root pytest.ini gains testpaths = packages to keep the docker bed out of the unit CI run.

Verdict: APPROVE. Per the severity rule (verdict is driven by Postable findings only), there are zero Postable HIGH/CRITICAL findings. I already approved at a502f2e4dd (2026-06-30); the three commits pushed since (8e24cb0b, 6f002fda, 5aae85ca) touch only the app-tests/git-leak/ harness and are all gate-strengthening / cleanup-robustness changes. Re-verified below.

Prior review status. No prior finding of mine remains open (there are no unresolved review threads on the PR). The post-approval commits address earlier review rounds and strengthen the gates rather than regress them:

• test_boot.py now keys completion on every scope's bundle being served (GET /scopes/{id}/policy == 200) and uses --force-recreate for a deterministic cold boot — stronger and less flaky than the previous repo_locks-count signal.
• test_leak.py removed the tautological repos <= baseline_repos assertion (a cache-count that cannot grow for same-URL re-syncs) and renamed the test to reflect that RSS is the load-bearing guard — directly fixing the vacuous-gate concern. Added transient-error retry in _wait_until (fail-safe: a persistent failure still surfaces via the caller's assert).
• test_resilience.py moved the postgres-bounce test to a 2-worker stack (opal_multiworker) so the cross-worker Postgres backbone is actually exercised, and added an anti-vacuous assertion (b): the broadcaster reconnect-log count must strictly increase, so the PID-unchanged in-place-reconnect check can't pass because the bounce failed to break anything. Offline-repo PUTs moved inside try/finally so hard_reset() always cleans up.

Verification performed.

• The load-bearing log marker "Broadcaster listener connected to channel" in broadcaster_connect_count() matches real emitted output at packages/opal-server/opal_server/pubsub_resilience.py:277 — the guarded fix (#915 / PER-15065) is already on master, so this is a genuine regression gate, not a permanently-red test.
• The two reconnect knobs the compose file sets (OPAL_BROADCAST_RECONNECT_BACKOFF_MAX_SECONDS, OPAL_BROADCAST_RESYNC_SETTLE_SECONDS) are real Confi keys (config.py:73-77, 92-97) consumed by pubsub.py:157,159,269; the compose comment's stated defaults (30s max backoff / 2s settle) match the declarations.
• debug_stats.py's three caches are class-level dicts on GitPolicyFetcher (git_fetcher.py:118-120), so the "process-global" docstring is accurate. Config key DEBUG_INTERNAL_STATS has a description, no env-name collision, and no OPAL_-double-prefix.
• Genuine gates: each suite fails on master for the stated reason (no delete-purge; no scopes-path fetch timeout) and cleans up (down -v, hard_reset, delete_all_scopes, multiworker→single-worker restore with a fail-loud assert). Determinism is handled (single worker for per-process cache reads; connect-count for the broadcaster; --force-recreate cold boot).

Findings

Postable: none.

Informational (LOW — deliberate, well-documented test-bed tradeoffs; not blocking):

#	Severity	File:Line	Category	Description
1	LOW	app-tests/git-leak/helpers.py:305-346	Out-of-scope (test infra)	worker_pids() treats the lowest gunicorn PID as the master. Correct for a fresh, short-lived container; technically fragile only under PID wraparound. Fine for this bed.
2	LOW	app-tests/git-leak/conftest.py:84-86	Out-of-scope (test infra)	The opal fixture runs a docker compose exec (worker_pids()) on every test setup as a single-worker invariant guard — minor per-test overhead, intentional.

Blast radius: Contained. New production code is the off-by-default, schema-hidden, auth-gated-when-auth-enabled read-only stats endpoint. Root pytest.ini testpaths = packages scopes the default CI pytest run to packages/ — no test files exist outside packages/, so nothing is dropped, and the intentionally-red git-leak bed (not wired into the app-tests/run.sh e2e job) stays out of CI.

Isolation / scope: Isolated and complete. The lone production addition exists to support the test bed, is off by default, and is fully finished (no half-done side-quests). The harness lands first; gates go green as later PRs merge.