Skip to content
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
52d92c1
fix(codex): stop runtime helpers from leaking past their idle timeout
possibilities Aug 11, 2026
f5bf873
fix(codex): address CodeRabbit round 1 on the helper-leak fix
possibilities Aug 11, 2026
1590cd1
test(codex): pin the sweep-retry test to the four-attempt budget, ord…
possibilities Aug 11, 2026
6bb2049
fix(codex): reap app helpers stranded by the detach grace
possibilities Aug 11, 2026
c18a5df
fix(codex): degrade an unreadable connection count to zero
possibilities Aug 11, 2026
c03af14
fix(codex): correct the connection-count comment and cover the reap o…
possibilities Aug 11, 2026
e57da41
fix(codex): accept only a positive socket count as evidence of a cons…
possibilities Aug 11, 2026
16a335f
test: add owned-PID probes for helper-lifecycle fixtures (#668)
ndycode Aug 13, 2026
bf0a749
fix(codex): reap only helpers that were never handed to a consumer
ndycode Aug 13, 2026
040e0c1
fix(runtime): stop unbind from orphaning helper owner files (#666)
ndycode Aug 13, 2026
7c10722
refactor(runtime): one identity-checked helper selector for both read…
ndycode Aug 13, 2026
109db04
docs: correct the runtime app helper status path in the README
ndycode Aug 13, 2026
51c5ca4
refactor(runtime): narrow the helper PID instead of casting it
ndycode Aug 13, 2026
c42bf54
fix(runtime): address the review on the helper-lifecycle follow-ups
ndycode Aug 13, 2026
e087c43
test: stress the helper lifecycle at the scale the leak report described
ndycode Aug 13, 2026
8f0b578
test: fix the review round on the stress suite
ndycode Aug 13, 2026
6b9f327
test: make the spawn-failure coverage exercise the helpers, not node
ndycode Aug 13, 2026
02e0c9b
test: cover the failed-spawn branch in withLivePid
ndycode Aug 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,8 @@ npm run vendor:verify # vendored dependency provenance check
- Global accounts: `~/.codex/multi-auth/openai-codex-accounts.json`.
- Official Codex state: `~/.codex/auth.json`, `~/.codex/accounts.json`, `~/.codex/config.toml`.
- Runtime observability: `~/.codex/multi-auth/runtime-observability.json`.
- App helper status: `~/.codex/multi-auth/runtime-rotation-app-helper.json`.
- App helper status: `~/.codex/multi-auth/runtime-rotation-app-helper.<pid>.json` (per helper; legacy un-suffixed file still read).
- App helper owner identity: `~/.codex/multi-auth/runtime-rotation-app-helper-owner.<pid>.json` (per helper; removed on exit, swept once the helper PID is dead).
- App bind state/logs: `~/.codex/multi-auth/app-bind/`.
- Prompt templates sync from Codex CLI GitHub releases with ETag caching.
- Historical audit evidence under `docs/audits/evidence/` is snapshot evidence, not current architecture guidance.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ For remote or headless shells, prefer `codex-multi-auth login --device-auth`.
| Routing profiles | `~/.codex/multi-auth/routing-profiles.json` |
| Budget guards | `~/.codex/multi-auth/budget-guards.json` |
| Local client tokens | `~/.codex/multi-auth/local-client-tokens.json` |
| Runtime app helper status | `~/.codex/multi-auth/runtime-rotation-app-helper.json` |
| Runtime app helper status | `~/.codex/multi-auth/runtime-rotation-app-helper.<pid>.json` |
| Runtime app helper owner metadata | `~/.codex/multi-auth/runtime-rotation-app-helper-owner.<pid>.json` |
| Persistent app bind state/logs | `~/.codex/multi-auth/app-bind/` |
| Logs | `~/.codex/multi-auth/logs/codex-plugin/` |
Expand Down
2 changes: 2 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ These are safe for most operators and frequently used in day-to-day workflows.
| `CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY=0/1` | Opt out/in of live Codex Responses routing through the localhost account-rotation proxy |
| `CODEX_MULTI_AUTH_FORCE_ACCOUNT=<index\|email\|id>` | Force one account for a single forwarded `codex-multi-auth-codex` run (equivalent to the `--account` flag, which wins when both are set). Ephemeral and fail-hard; requires the runtime rotation proxy. See [Force an account for one invocation](reference/commands.md#force-an-account-for-one-invocation) |
| `CODEX_MULTI_AUTH_APP_ROTATION_IDLE_MS=<ms>` | Override idle shutdown for the wrapper-launched Codex app helper |
| `CODEX_MULTI_AUTH_APP_ROTATION_MAX_LIFETIME_MS=<ms>` | Absolute ceiling on a runtime helper's life regardless of activity (default 24h; `0` disables) |
| `CODEX_MULTI_AUTH_APP_ROTATION_DETACHED_IDLE_MS=<ms>` | Idle window that applies once a helper's launcher is gone, nothing is connected, and the helper has never served a request (default 15m; `0` restores the full idle timeout) |
| `CODEX_MULTI_AUTH_APP_BIND=0/1` | Alias-style opt-out for first-run packaged Codex app bind (see also `CODEX_MULTI_AUTH_APP_BIND_INSTALL`) |
| `CODEX_MULTI_AUTH_APP_BIND_INSTALL=0/1` | Opt out/in of packaged Codex app bind self-heal on first durable CLI run or rotation enable |
| `CODEX_MULTI_AUTH_APP_LAUNCHER_INSTALL=0/1` | Opt out/in of supported user-level launcher routing on first durable CLI run or rotation enable |
Expand Down
20 changes: 18 additions & 2 deletions docs/development/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,13 +197,28 @@ Because no shim means no `CODEX_MULTI_AUTH_APP_SERVER_ACCOUNT_LABEL` in the forw

A helper that cannot start is a hard failure on all of these branches — unlike the shadow path, there is no rotation-off shape left to degrade into, and quietly serving a resident server unrotated is worse than not serving it. Hard means a diagnostic on stderr and exit 1, not an unhandled rejection: `createRuntimeRotationProxyContextIfEnabled` catches the launch failure, releases the compatibility home the caller already built, and returns a `startupError` that `forwardToRealCodex` turns into an exit code before the official CLI is ever spawned.

Helper self-reaping is identity-checked and bounded. A detached helper decides "is my launcher still alive" by PID **plus the launcher's kernel start time** (passed at spawn via `CODEX_MULTI_AUTH_APP_ROTATION_OWNER_START_TIME_MS`), and reaps itself on whichever of three deadlines comes first.

| Rule | Behavior |
| --- | --- |
| Owner identity | PID plus kernel start time. A bare `kill(pid, 0)` cannot tell a launcher from a later process that recycled its PID, and because the idle deadline only ever moved forward, one false "alive" was never corrected — helpers were observed running 33 hours past a 12-hour idle timeout, hundreds deep. |
| Recheck cadence | At most once a minute; a `ps` spawn per tick would cost more than it saves. A *failed* re-read keeps the previous verdict rather than declaring a live owner dead — under the process-table pressure this exists for, `fork` itself can fail. |
| Degraded check | Where no start time is known at all, the check degrades to bare liveness. |
| Idle timeout | `CODEX_MULTI_AUTH_APP_ROTATION_IDLE_MS`, default 12h, refreshed by traffic and by a live owner. |
| Detached window | `CODEX_MULTI_AUTH_APP_ROTATION_DETACHED_IDLE_MS`, default 15m, `0` disables. Applies from the moment the owner is *confirmed* dead — a helper with no recorded owner PID is not a helper whose owner is dead, and stays on the idle timeout. The detach grace hands helpers off optimistically — any launcher exiting cleanly within it leaves its helper running — so every short forwarded command stranded a helper that then held the full idle timeout with no owner, no traffic, and nothing connected. The window only reaps a helper that has **never served a request**: every leaked helper in #663 had `totalRequests: 0`, while a live `codex app` session that is merely idle between turns holds no socket either (the proxy leaves `keepAliveTimeout` at Node's 5s default), so reaping on the socket check alone would kill a working proxy under the desktop app. A helper that served anything falls back to the idle timeout and the lifetime ceiling. |
| Connection gating | The detached window fires only while the proxy reports zero open client connections, so a consumer who really did take the handoff — `codex app` giving the desktop app its proxy — is never reaped out from under. Traffic after the owner's death pushes the deadline out by another window, so a consumer that reconnects per request survives on its own evidence. An unreadable connection count fails open into reaping: treating "unknown" as "attached" would restore the leak for any shape that stopped answering. |
| Lifetime ceiling | `CODEX_MULTI_AUTH_APP_ROTATION_MAX_LIFETIME_MS`, default 24h, `0` disables. Unconditional on activity — the backstop that turns any future accounting bug into a bounded leak instead of an unbounded one. |
| Telemetry | Per process: each helper publishes `runtime-rotation-app-helper.<pid>.json` (the un-suffixed legacy path is still read for pre-upgrade helpers), publishes only on change plus a heartbeat rather than every tick, removes its owner file on exit, and each launcher sweeps metadata files whose helper PID is dead before spawning the next one — terminal status stamps survive until that sweep, long enough to be read without accumulating forever. |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

The published `idleExpiresAt` reports whichever of these deadlines is actually enforced, so `rotation status` cannot advertise 12h to a helper minutes from being reaped. Terminal states are `idle-timeout`, `owner-gone`, `max-lifetime`, `stopped`, and `error`; only `running` means running.

Helper shutdown is bounded rather than best-effort. `stopRuntimeRotationAppHelper` sends `SIGTERM`, waits out the graceful window, escalates to `SIGKILL` if the helper is still running, and then unconditionally destroys the helper's stdio streams and unrefs the child. That last step is the load-bearing one: the helper is spawned with piped stdio, so a helper that outlives the window — or any process that inherited those pipes — keeps the wrapper's event loop referenced and the shell prompt never returns. On Windows the signals are emulated as unconditional termination, so the stream teardown is the only part that reliably frees the wrapper there.

Two interactive sessions can therefore run concurrently against the same home — the same as running the official CLI twice — and **no lock is taken over session state**: neither session copies or syncs it, so there is nothing to clobber. Regression coverage lives in `test/codex-bin-wrapper.test.ts`.

Scope that guarantee to session state only. It does **not** extend to `config.toml`: `ensureCodexCliFileAuthStore` (`lib/codex-cli/writer.ts`) still read-modify-writes the canonical file when the store is not already `"file"`, and the atomic write does not serialize cross-process writers. That is safe in practice rather than by locking — the operation is idempotent, converges on a single value, and lands via atomic rename, so concurrent invocations agree instead of interleaving. Anything added to that write path that is *not* idempotent would need a real lock.

Internal env used by these branches (not operator-facing): `CODEX_MULTI_AUTH_APP_ROTATION_USE_CANONICAL_HOME`, `CODEX_MULTI_AUTH_APP_ROTATION_INSTALL_APP_SERVER_SHIM`, `CODEX_MULTI_AUTH_APP_SERVER_CONFIG_ARGS_JSON`, `CODEX_MULTI_AUTH_APP_ROTATION_OWNER_PID`, `CODEX_MULTI_AUTH_REAL_CODEX_HOME`.
Internal env used by these branches (not operator-facing): `CODEX_MULTI_AUTH_APP_ROTATION_USE_CANONICAL_HOME`, `CODEX_MULTI_AUTH_APP_ROTATION_INSTALL_APP_SERVER_SHIM`, `CODEX_MULTI_AUTH_APP_SERVER_CONFIG_ARGS_JSON`, `CODEX_MULTI_AUTH_APP_ROTATION_OWNER_PID`, `CODEX_MULTI_AUTH_APP_ROTATION_OWNER_START_TIME_MS`, `CODEX_MULTI_AUTH_REAL_CODEX_HOME`.

* * *

Expand Down Expand Up @@ -273,7 +288,8 @@ Canonical multi-auth root: `~/.codex/multi-auth`.
| `budget-guards.json` | Local request/token/cost limits |
| `local-client-tokens.json` | Local bridge token hashes (no plaintext) |
| `usage/usage-ledger.jsonl` | Append-only local usage metadata (+ rotated archives) |
| `runtime-rotation-app-helper.json` | Wrapper-launched Codex app helper status |
| `runtime-rotation-app-helper.<pid>.json` | Wrapper-launched Codex app helper status, one per live helper (the un-suffixed name is the pre-per-PID legacy path, still read) |
| `runtime-rotation-app-helper-owner.<pid>.json` | Owner identity token for a wrapper-launched helper, one per helper; removed by the helper on exit and swept when its PID is dead |
| `app-bind/` | Packaged app bind state, backup metadata, router status/log |
| `logs/` | Diagnostics when logging is enabled |
| `cache/` | Prompt/cache artifacts |
Expand Down
3 changes: 3 additions & 0 deletions docs/development/CONFIG_FIELDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,10 @@ Cross-process refresh lease knobs: `CODEX_AUTH_REFRESH_LEASE`, `CODEX_AUTH_REFRE
| `CODEX_MODE` | Toggle Codex mode |
| `CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY` | Toggle localhost Responses proxy for forwarded Codex sessions (`1`/`true` to enable, `0`/`false` to disable) |
| `CODEX_MULTI_AUTH_APP_ROTATION_IDLE_MS` | Override idle timeout for the wrapper-launched Codex app runtime helper |
| `CODEX_MULTI_AUTH_APP_ROTATION_MAX_LIFETIME_MS` | Absolute ceiling on a runtime helper's life regardless of activity (default 24h; `0` disables). The backstop that bounds the leak if activity accounting is ever wrong again |
| `CODEX_MULTI_AUTH_APP_ROTATION_DETACHED_IDLE_MS` | Idle window applied from the moment a helper's launcher is confirmed dead, and only while no client connection is open and the helper has never served a request (default 15m; `0` keeps the full idle timeout). Bounds helpers stranded by the detach grace; a helper that served traffic, or one with no recorded owner PID, stays on the idle timeout |
| `CODEX_MULTI_AUTH_APP_ROTATION_OWNER_PID` | Internal owner PID used by the wrapper-launched app helper |
| `CODEX_MULTI_AUTH_APP_ROTATION_OWNER_START_TIME_MS` | Internal owner process start time (epoch ms) the helper uses to tell its launcher from a later process that recycled the PID |
| `CODEX_MULTI_AUTH_REAL_CODEX_HOME` | Internal original Codex home pointer used by runtime rotation helpers |
| `CODEX_MULTI_AUTH_APP_BIND_INSTALL` | Opt out/in of packaged Codex app bind self-heal on first CLI run or rotation enable |
| `CODEX_MULTI_AUTH_APP_BIND` | Legacy/manual app-bind override consumed by the first-run setup hook (`lib/runtime/first-run.ts`) |
Expand Down
7 changes: 4 additions & 3 deletions docs/privacy.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@
| Local bridge client tokens | `~/.codex/multi-auth/local-client-tokens.json` | SHA-256 token hashes plus prefixes and labels; plaintext tokens are shown only on create/rotate |
| Named backups | `~/.codex/multi-auth/backups/` | Operator-exported named account-pool backups |
| Project account pools | `~/.codex/multi-auth/projects/<project-key>/` | Per-repo account pools when project scope is enabled |
| Runtime app helper status | `~/.codex/multi-auth/runtime-rotation-app-helper.json` | Local helper status for wrapper-launched Codex app sessions |
| Runtime app helper status | `~/.codex/multi-auth/runtime-rotation-app-helper.<pid>.json` (one per helper; plus the legacy un-suffixed file from older versions) | Local helper status for wrapper-launched Codex app sessions |
| Runtime app helper owner identity | `~/.codex/multi-auth/runtime-rotation-app-helper-owner.<pid>.json` (one per helper) | Local identity token and launcher PID, so a helper can tell its own launcher from a recycled PID; removed on helper exit and swept once the PID is dead |
| Persistent app bind state/logs | `~/.codex/multi-auth/app-bind/` | Reversible packaged-app router state, backup metadata, and local router log |
| Logs | `~/.codex/multi-auth/logs/codex-plugin/` | Optional diagnostics |
| Prompt/cache files | `~/.codex/multi-auth/cache/` | Cached prompt/template metadata |
Expand Down Expand Up @@ -88,7 +89,7 @@ rm -rf ~/.codex/multi-auth/refresh-leases
rm -rf ~/.codex/multi-auth/usage
rm -rf ~/.codex/multi-auth/backups
rm -rf ~/.codex/multi-auth/projects
rm -f ~/.codex/multi-auth/runtime-rotation-app-helper.json
rm -f ~/.codex/multi-auth/runtime-rotation-app-helper.json ~/.codex/multi-auth/runtime-rotation-app-helper.*.json ~/.codex/multi-auth/runtime-rotation-app-helper-owner.*.json
rm -rf ~/.codex/multi-auth/app-bind
rm -rf ~/.codex/multi-auth/logs/codex-plugin
rm -rf ~/.codex/multi-auth/cache
Expand All @@ -115,7 +116,7 @@ Remove-Item "$HOME\.codex\multi-auth\refresh-leases" -Recurse -Force -ErrorActio
Remove-Item "$HOME\.codex\multi-auth\usage" -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item "$HOME\.codex\multi-auth\backups" -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item "$HOME\.codex\multi-auth\projects" -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item "$HOME\.codex\multi-auth\runtime-rotation-app-helper.json" -Force -ErrorAction SilentlyContinue
Remove-Item "$HOME\.codex\multi-auth\runtime-rotation-app-helper*.json","$HOME\.codex\multi-auth\runtime-rotation-app-helper-owner*.json" -Force -ErrorAction SilentlyContinue
Remove-Item "$HOME\.codex\multi-auth\app-bind" -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item "$HOME\.codex\multi-auth\logs\codex-plugin" -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item "$HOME\.codex\multi-auth\cache" -Recurse -Force -ErrorAction SilentlyContinue
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,8 @@ Common operator overrides (aligned with [../configuration.md](../configuration.m
- `CODEX_MULTI_AUTH_RUNTIME_ROTATION_PROXY`
- `CODEX_MULTI_AUTH_FORCE_ACCOUNT` — force one account for a single forwarded `codex-multi-auth-codex` run (selector: index/email/id); `--account` wins when both are set
- `CODEX_MULTI_AUTH_APP_ROTATION_IDLE_MS`
- `CODEX_MULTI_AUTH_APP_ROTATION_MAX_LIFETIME_MS` — absolute ceiling on a helper's life regardless of activity (default 24h; `0` disables)
- `CODEX_MULTI_AUTH_APP_ROTATION_DETACHED_IDLE_MS` — idle window once the helper's launcher is confirmed gone, nothing is connected, and the helper has never served a request (default 15m; `0` keeps the full idle timeout)
- `CODEX_MULTI_AUTH_APP_BIND_INSTALL`
- `CODEX_MULTI_AUTH_APP_LAUNCHER_INSTALL`
- `CODEX_TUI_V2`
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/storage-paths.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Override root:
| Budget guards | `~/.codex/multi-auth/budget-guards.json` |
| Local bridge client tokens | `~/.codex/multi-auth/local-client-tokens.json` |
| Cross-process refresh leases | `~/.codex/multi-auth/refresh-leases/` |
| Runtime app helper status | `~/.codex/multi-auth/runtime-rotation-app-helper.json` |
| Runtime app helper status | `~/.codex/multi-auth/runtime-rotation-app-helper.<pid>.json` |
| Runtime app helper owner metadata | `~/.codex/multi-auth/runtime-rotation-app-helper-owner.<pid>.json` |
| Persistent app bind directory | `~/.codex/multi-auth/app-bind/` |
| Named pool backups | `~/.codex/multi-auth/backups/` |
Expand Down Expand Up @@ -159,7 +159,7 @@ Runtime rotation adds local state only when enabled or when a helper has recentl
| Path | Purpose |
| --- | --- |
| `~/.codex/multi-auth/runtime-observability.json` | request counters, last selected runtime account metadata, and cooldown context for status/report commands |
| `~/.codex/multi-auth/runtime-rotation-app-helper.json` | wrapper-launched `codex app` helper state, idle timeout, request count, and last-account metadata |
| `~/.codex/multi-auth/runtime-rotation-app-helper.<pid>.json` | wrapper-launched `codex app` helper state, idle timeout, request count, and last-account metadata — one file per helper; the un-suffixed name is the legacy shared path from older versions, still read. A terminal stamp persists until the next helper launch sweeps files whose PID is dead; the owner file is removed on clean helper exit |
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
| `~/.codex/multi-auth/app-bind/runtime-rotation-app-bind.json` | persistent packaged-app bind state |
| `~/.codex/multi-auth/app-bind/codex-config-backup.json` | backup metadata for restoring the real Codex `config.toml` |
| `~/.codex/multi-auth/app-bind/runtime-rotation-app-bind-status.json` | persistent app router status |
Expand Down
Loading