[Bug]: Container hub repeatedly refuses startup with unverified proxy ownership #8552

Client or integration

Other

Area

Service lifecycle

Summary

In a Kubernetes hub deployment using the community image nordz0r/opencodex:2.100.0, the process failed repeatedly at recoverStartJournalAfterOwnerRecheck (src/cli/start-owner-fence.ts:14) with the following error. Persistent state included ocx.pid containing 1 and runtime-port.json recording PID 1. The exact cause of the inconclusive liveness observation has not been established; stale records and PID namespace reuse are hypotheses, not a proven mechanism.

Proxy ownership is unverified; startup refused. PID record and recovery journal were preserved. Retry when the owner can be verified.

The pod entered CrashLoopBackOff. The trigger that produced the stale state was not established, nor was the liveness observation kind recorded. The expected behavior is safe recovery when absence of the previous owner can be established, without compromising ownership fencing.

Reproduction

1. Observe a Kubernetes hub pod running the community image nordz0r/opencodex:2.100.0 with a persistent state volume. In the observed incident it repeatedly exited with the error below; the trigger is unknown.
2. While the pod was stopped, its state directory contained ocx.pid with 1 and runtime-port.json with PID 1, port 10100 and hostname 0.0.0.0. Other ownership and journal files were present.
3. The operator stopped the workload and quarantined ocx.pid, run/.c10100.lease, codex-home/opencodex-journal.json and .opencodex-owner.json; on restart the same error remained.
4. After additionally moving runtime-port.json aside and restarting, the pod became ready and healthz returned 200. This recovery observation does not establish which file or liveness branch caused the failure. A minimal docker kill reproduction has not been verified.

Version

2.100.0 (crash observed); fence code identical in 2.100.2 (src/cli/start-owner-fence.ts, src/server/proxy-liveness.ts unchanged between the two tags).

Operating system

Debian (k3s container, linux/amd64); oven/bun image, proxy as container PID 1.

Provider and model

Not provider-specific.

Logs or error output

error: Proxy ownership is unverified; startup refused. PID record and recovery journal were preserved. Retry when the owner can be verified.
      at recoverStartJournalAfterOwnerRecheck (/home/bun/app/src/cli/start-owner-fence.ts:15:15)
      at recoverStartStateUnderOwnershipLease (/home/bun/app/src/cli/start-owner-fence.ts:45:23)
      at async handleStart (/home/bun/app/src/cli/index.ts:446:41)

kubelet then reports CrashLoopBackOff for the pod on every restart.

Screenshots and supporting files

Observed stale state and recovery sequence are described above. Do not remove ownership records unless the previous owner has been independently proven absent; the exact failure mechanism requires further investigation.

Redacted configuration

{"runtimeRole":"hub","hostname":"0.0.0.0","port":10100}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.