Repository navigation
Troubleshooting
Start with the symptom that most closely matches the failure. Regardless of provider, keep TLS verification enabled, preserve the first relevant log, and do not use broad Docker/WSL resets or prune commands as a first response.
- Quick diagnostics
- Startup reports that the pool controller configuration lock is already held
- Windows no-Go startup prints an HTTP/2 named-pipe diagnostic
- A Docker workload fails with an architecture error
- Docker Sandboxes is unavailable or its preflight fails
- Docker Sandboxes daemon health passes but inventory hangs
- Docker Sandboxes rejects template, policy, or capacity
- Docker Sandboxes creation fails after a runtime-helper prompt
- Docker Sandboxes rejects a staging workspace because SSH-agent forwarding is present
- Docker Hub login succeeds but a private pull is denied in Docker Sandboxes
- Docker Sandboxes reports package-manager contention
- A candidate listener ends before readiness is observed
- An idle runner reports GitHub or Sandbox health warnings
- EPAR exits during a GitHub or registry incident
- GitHub Actions runner release resolution reports HTTP 403 or 429
- A scheduled image check or update fails
- A runner is held for diagnostics or an acknowledgement
- Docker image build runs out of space
- Storage keeps growing after updates
- Docker image build fails with TLS certificate errors
- Windows Docker Desktop WSL2 disk is smaller than expected
- Docker Container startup fails
- WSL provider image build fails early
- GitHub runner registration fails
EPAR writes logs under work/logs by default. Start with work/logs/epar-last-error.log, then inspect the matching build log in work/logs/builds/ or instance transcript in work/logs/instances/. Manager events are console-only by default; raw transcripts are file-only unless logging.transcriptSinks includes console.
Long Buildx operations show a bounded console summary with downloaded bytes, completed layers, the active BuildKit step, elapsed time, and growing direct-archive bytes when an export is in progress. The complete raw progress remains in the printed build-log path.
./start --help
./start version
docker version
docker info
docker system dfWithout local Go, use ./start --help; the wrapper selects the containerized toolchain. For Windows WSL2, a WSL-backed Docker daemon, or the WSL provider, also run:
wsl --version
wsl -l -v
docker context ls
docker run --rm ghcr.io/catthehacker/ubuntu:full-latest df -h /Container-visible free space is the relevant value for Docker builds. Windows Explorer or Finder free space does not necessarily equal the free space in a Linux VM backing the daemon.
./start exits with an error like:
ephemeral-action-runner: pool controller configuration lock is already held for "/Users/someone/epar/.local/config.yml" (owner config="/Users/someone/epar/.local/config.yml" provider="docker-sandboxes" prefix="example-pool" pid=12345 startedAt=2026-01-02T03:04:05Z)
This can appear after the startup wrapper rebuilds the native controller because its source digest changed. The rebuild prepares a new binary but does not replace an already-running controller, so the new process still stops at the ownership lock.
The lock prevents two mutating controllers from managing the same pool. Treat the reported config path, provider, prefix, PID, and start time as owner-identification evidence rather than deleting a lock file.
On macOS or Linux, inspect the reported PID, replacing 12345 with the value from the error:
ps -p 12345 -o pid=,ppid=,lstart=,command=On Windows PowerShell:
Get-Process -Id 12345 -ErrorAction SilentlyContinue | Format-List Id, StartTime, PathIf the PID is the intended live EPAR controller, do not start another controller for that pool. Return to its terminal and press Ctrl-C once, or stop it through the service manager that launched it, then wait for Cleanup complete. EPAR can now exit safely. before running ./start again. Use this sequence when activating copied source or a rebuilt controller: stop the old controller cleanly, wait for cleanup, then start the new version.
If no process owns the reported PID, rerun ./start. The operating-system lock is authoritative and is released when its owning process exits; stale descriptive metadata may remain on disk but is replaced on the next successful acquisition. Do not manually delete files under the controller-lock state directory, and do not signal a process solely because its PID matches stale metadata—PID values can be reused.
If the exact EPAR controller is unresponsive, prefer its service manager's graceful stop or a single interrupt and allow time for cleanup. After a confirmed abnormal exit, rerun ./start so normal lifecycle reconciliation can inspect and recover exact owned resources. Avoid force-killing the process, deleting runner records manually, or using broad Docker, Sandbox, WSL, or GitHub cleanup commands as a first response.
Multiple EPAR controllers may run on one machine, but every concurrently active controller must use both a distinct canonical configuration path and a distinct normalized pool.namePrefix. Reusing either the same config path or the same pool prefix intentionally conflicts, including across different project directories or providers. Give each controller its own config and pool identity instead of bypassing the lock.
The Windows no-Go bootstrap prints a line like:
http2: server: error reading preface from client //./pipe/dockerDesktopLinuxEngine: file has already been closed
Docker Desktop can emit this named-pipe transport diagnostic when a client connection closes. If the bootstrap docker build succeeds and the wizard or command continues, it is not an EPAR runner-group or GitHub API failure. The wrapper suppresses only this exact successful-build diagnostic and keeps full Docker stderr when the build fails.
If startup stops, verify the selected engine and context:
docker version
docker info
docker context showResolve a failed command or unhealthy engine; do not treat every named-pipe line as harmless when Docker returned a nonzero exit code.
Docker or Compose exits with exec format error, cannot execute binary file, a platform-mismatch warning, no matching manifest, a QEMU loader error, or exit code 139.
Inspect the runner, daemon, image manifest, and Compose platform setting:
uname -m
docker info --format '{{.OSType}}/{{.Architecture}}'
docker image inspect --format '{{.Os}}/{{.Architecture}}' IMAGE
docker buildx imagetools inspect IMAGE
docker compose configno matching manifest means the image does not publish the requested platform; emulation cannot create a missing manifest. A platform warning alone does not prove failure, and exit code 139 alone does not prove an architecture mismatch. Enabling emulation does not change normal manifest selection, so request the foreign image with docker run --platform ... or the affected Compose service's platform: property. EPAR does not inject DOCKER_DEFAULT_PLATFORM.
For Docker Sandboxes, keep provider.platform native. QEMU/binfmt is disabled by default on Linux, macOS, and Windows with dockerSandboxes.architectureEmulation: native-only, which verifies the guest and private Docker architecture without attempting foreign execution. As of sbx v0.39, QEMU is not fully supported inside Docker Sandboxes, so a passing diagnostic, template build, or image pull is not proof that a foreign-architecture workload will run. Prefer building or publishing a multi-platform image instead of relying on emulation.
If a job requires another architecture, such as X64 on ARM64 or ARM64 on X64, either run EPAR on a machine with the target architecture when using the Docker Sandboxes provider, or run a separate EPAR instance/configuration with provider.type: docker-container. Docker Container uses Docker-in-Docker for isolation and can support QEMU/binfmt inside its private daemon for trusted jobs. Select it in the setup wizard by choosing C. Show compatibility providers, then Docker Container. Keep the target-architecture pool on a distinct runner label and verify the actual workload rather than only pulling the image.
The best-effort and required modes remain available as explicit configuration choices, but they do not make Docker Sandboxes QEMU fully supported on sbx v0.39. Do not add a privileged installer to a Docker Sandboxes workflow: it cannot repair a sandbox kernel that does not expose the required filesystem.
The wizard marks Docker Sandboxes unavailable, or sbx diagnose --output json reports failures.
Check the diagnostic result before editing configuration:
sbx diagnose --output jsonEPAR requires a controller architecture with an available Linux guest template and at least one diagnostic pass with zero failures. Diagnostic warnings and skipped checks remain visible but do not disable the provider. Review the failed item and its hint in the JSON output, fix the prerequisite, then choose R to refresh availability; do not manually force a provider selection or substitute a compatibility provider for a configured Docker Sandboxes pool. Use C. Show compatibility providers only when you deliberately intend to create or maintain a compatibility configuration.
sbx daemon status --json reports running and sbx diagnose -o json passes, but sbx ls --json hangs, returns an empty response slowly, or ends with a runtime-list cancellation. EPAR may repeatedly report context canceled, quarantine instances after the host-trust or inventory deadline, and keep the controller process alive with no ready capacity.
The daemon health and diagnostic commands do not prove that the /sandbox inventory path or a managed sandbox's inner Docker API is responsive. Run sbx ls --json only with an external operating-system timeout; record whether it returns valid JSON, its elapsed time, and whether stderr reports a cancelled runtime listing. Treat a slow or cancelled inventory response as provider state unknown even when daemon status and diagnostics pass.
Preserve the controller output, daemon log, process tree, durable pool lifecycle state, and a before/after count of the daemon's managed Docker-socket descriptors. Do not repeatedly invoke unbounded sbx ls, start a second EPAR controller, use sbx reset, remove a sandbox by prefix, prune Docker state, or manually delete GitHub runners. EPAR intentionally quarantines uncertain resources and keeps them counted against pool.instances until exact ownership and cleanup can be verified.
With the default dockerSandboxes.recoveryMode: exclusive-auto, EPAR keeps the controller alive and holds one host-wide provider recovery lease from the complete authoritative census through the durable reservation, cold stop, authoritative stopped-state confirmation, configured quiescence interval, sanitized detached start, and three identical parsed safety-evidence inventory snapshots covering name, stable provider ID, status, agent/source, and exact workspaces. During that sequence it lets already-running provider commands drain and gates new provider commands through the same host lock, so another cooperating configuration cannot create a sandbox or race the daemon restart. The installed sbx ls --json API is treated as the complete snapshot because it exposes no independent count or generation; duplicate keys and durable-identity omissions fail closed, while an otherwise valid response that omits an unrelated sandbox outside the cooperating lease cannot be independently detected. It backs off after failed attempts and preserves exact capacity while recovery is unresolved. The recovery path never invokes sbx reset, sbx logout, sbx prune, or wildcard deletion. Set recoveryMode: observe only when an operator deliberately needs to prevent daemon mutation during maintenance.
If automatic recovery is disabled or the controller is not running, stop only the affected controller before a manual daemon interruption. Then run sbx daemon stop, wait until sbx daemon status --json reports stopped, and start with env -u SSH_AUTH_SOCK -u SSH_AUTH_SOCK_GATEWAY -u SSH_AGENT_PID sbx daemon start --detach. Before starting a new EPAR controller, require sbx daemon status --json to report running, sbx diagnose -o json to have no failed checks, and repeated bounded sbx ls --json calls to complete with valid JSON in a stable time. A passing diagnostic report without a responsive inventory is not a healthy Sandbox provider.
Startup reports a template identity/digest mismatch, policy-generation drift, QEMU/binfmt setup failure, or insufficient capacity.
Docker Sandboxes resolves the configured source selector, records the exact OCI identities in a local artifact receipt, verifies the host-global policy fingerprint, and binds the immutable tonistiigi/binfmt release and native-platform manifest into the reusable template evidence. If the desired source, platform, scripts, template inputs, runner inputs, trust inputs, or locked emulation artifact changes, rerun ./start; EPAR builds and imports a replacement and activates it only after exact readback succeeds.
An imported Docker Sandboxes template does not require a matching Docker image. EPAR builds directly to a verified archive, imports that archive, and then removes the transient workspace. If startup reports a missing Docker staging image, the controller is stale; rebuild the native controller and rerun ./start. If direct archive verification or sbx template load fails, use the printed Buildx transcript and archive error; EPAR does not fall back to the memory-heavy Docker load/save path.
Capacity admission accounts for phase-overlapping physical growth on each resolved capacity domain and applies the fixed storage.minimumFree reserve once per domain. The checkout, Docker Engine, Docker Desktop disk, Docker Sandboxes state/cache, WSL distribution backing, and Tart store can be on different filesystems. Docker Sandboxes root and inner-Docker sizes are independent sparse logical maxima and are not added as immediate host usage. Inspect the matching operation with ./start storage status --operation template-build --provider docker-sandboxes --config <path> --project-root <path>: a measured domain with insufficient capacity blocks, while status=unknown and available=unknown mean EPAR could not measure that local domain and will warn but continue enforcing all measurable domains. Use the exact prune preview to reclaim space, or deliberately retry only confirmed insufficiency with --allow-insufficient-storage; that flag does not bypass an unknown measurement, remote Docker-context rejection, or malformed storage topology. A documented-default-assumed warning means EPAR verified a local Docker Desktop context but could only measure the documented default/system location; it never grants cleanup authority. Avoid broad cleanup commands: they can delete stopped containers and intentionally retained resources.
On macOS or Linux, host security asks whether to allow a Docker Sandboxes helper such as mkfs.ext4, mkfs.erofs, or containerd-shim-nerdbox-v1; macOS may say that the helper “is an app downloaded from the Internet.” After a required prompt is denied or blocked, EPAR reports create docker sandbox failed, and sbx may report 500 Internal Server Error: failed to run sandbox container. The runner is neither registered nor marked ready. That exact signature also causes EPAR to append a conditional SSH-agent daemon hint while preserving the original command error; unrelated create failures are returned without that hint.
Docker Sandboxes uses mkfs.ext4 to create an ext4 filesystem inside each sandbox's private Docker disk-image file, mkfs.erofs to construct the read-only template snapshot, and containerd-shim-nerdbox-v1 to launch and manage the sandbox VM. Expected file targets are regular sandbox-owned files beneath the Docker Sandboxes runtime data directory—for example, current macOS releases may use ~/.sbx/run/d/containerd/.../images/<runner>-docker.img and ~/.sbx/run/d/containerd/.../snapshots/<id>/layer.erofs. With the current Homebrew sbx package, the runtime and shim are beneath /opt/homebrew/Caskroom/sbx/<version>/. A formatter must not target a physical device such as /dev/disk*, an EPAR checkout, a home-directory document, or another unrelated path.
If each executable belongs to the Docker Sandboxes installation you intentionally installed and any displayed target is the expected sandbox-owned file, allow the operation through the host's security or application-control prompt, then rerun the same EPAR start or verification command if creation already failed. Do not invoke a formatter or shim yourself, disable host security broadly, or approve a command with an unfamiliar target. The configured private Docker disk is sparse, so its logical maximum does not mean the formatter immediately consumes that amount of physical storage.
If no prompt appeared, or approval still produces the 500 error, preserve the failed runner evidence and inspect the Docker Sandboxes daemon/client logs and sbx diagnose --output json; the same top-level error can also represent a runtime, capacity, or host-policy failure. See Private Filesystem and VM Helper Approval for the provider contract.
sbx v0.42.1 also injects the default SSH_AUTH_SOCK=/run/ssh-agent.sock when forwarding is disabled and the socket is absent. Older EPAR templates reject that inert value too; repeated daemon restarts or --env SSH_AUTH_SOCK= do not fix it. Updated EPAR templates and runtime verification normalize only this exact absent endpoint, with no gateway or PID, while retaining actual forwarding rejection. A published prebuilt template must itself contain the fix; updating only the checkout/controller is insufficient. Do not delete sockets or bypass template verification to work around an old artifact.
Sandbox creation reaches verify dedicated docker sandbox staging workspace and fails with a message that host SSH-agent forwarding is not permitted. Diagnostics may show SSH_AUTH_SOCK=/run/ssh-agent.sock or SSH_AUTH_SOCK_GATEWAY=... inside the guest even though the imported template does not define them.
Docker Sandboxes may forward the host SSH agent through inherited environment or, in sbx v0.42+, persistent forwarding settings. EPAR rejects a sandbox exposing the socket or gateway because a workflow could use host SSH credentials. The generic failed to run sandbox container message alone does not prove that forwarding caused the failure; inspect the daemon logs and template entrypoint if the cause remains unclear. An explicit forwarding rejection is not evidence that the staging mount is missing or read-only, and deleting only /run/ssh-agent.sock is insufficient when forwarding remains enabled.
For sbx v0.42+, first inspect sbx settings get ssh.agentForwardingEnabled. If it is enabled, coordinate with every other local sbx user before running sbx settings set ssh.agentForwardingEnabled false and restarting the daemon using the sequence below. The setting is host-wide and requires a restart to apply; sanitizing process environment alone does not override it. EPAR does not change shared settings automatically. See Docker Sandboxes provider guidance for the isolation requirements. If disabling forwarding and restarting does not fix creation, investigate the original error rather than repeating daemon restarts.
In the default recoveryMode: exclusive-auto, EPAR treats a provider-owned create deadline or the known immediate create-stage signature as a bounded admission incident and may perform one stop-wait-start recovery using the existing host-global gates, then retry lifecycle reconciliation. A timed-out create gets one bounded read-only identity readback; if the immutable identity is still unavailable, EPAR retains a durable quarantine and capacity fence instead of treating an empty inventory as proof of absence. The host-scoped recovery ledger persists the admission latch and crash-safety cooldown across EPAR controller restarts; if that ledger is corrupt or unavailable, EPAR fails closed and does not intervene in the daemon. The recovery is limited to one daemon restart per incident, including controller reconciliation retries; recoveryMode: observe never probes or mutates the daemon. Coordinate the interruption with every process using the shared Docker Sandboxes daemon. If the automatic attempt has already been used or manual recovery is required, stop it and restart it with all forwarding variables removed before retrying EPAR:
sbx daemon stop
env -u SSH_AUTH_SOCK -u SSH_AUTH_SOCK_GATEWAY -u SSH_AGENT_PID sbx daemon start --detachEPAR strips these variables from Docker Sandboxes commands it launches, so a stopped daemon auto-started through those commands is sanitized, but an already-running daemon retains the environment with which another shell or tool started it. Do not disable this admission check, use sbx reset or sbx logout, or forward an agent into a reusable runner template. The installed sbx CLI does not expose a durable create-operation identity, so an identityless timed-out create cannot be adopted after a controller restart from name, workspace, or template metadata alone; it remains report-only and capacity-fenced until an operator performs an exact, evidence-backed resolution. If the failed creation predates the immutable-receipt fix, preserve its reported sandbox UUID and use exact provider cleanup; never delete a same-name resource by prefix alone.
Report-only resources discovered by prefix are retained as capacity fences but are excluded from runner liveness probes until provider-specific exact ownership evidence is available. Docker Sandboxes may recover a matching orphan when its stable sandbox ID, configured staging workspace, and staging receipt can be reconstructed; otherwise normal reconciliation continues to report it without authorizing deletion, adoption, or capacity reuse.
A workflow's Docker login step reports Login Succeeded, but a later pull of a private Docker Hub image fails with insufficient_scope: authorization failed, pull access denied, or an equivalent authorization response. The same workflow and credentials may succeed with Docker Container or a GitHub-hosted runner. Using docker --config /home/agent/.docker pull ... produces the same denial.
First verify only metadata, never credential contents: the listener should run as agent with HOME=/home/agent and DOCKER_CONFIG=/home/agent/.docker, its control-plane environment should explicitly use http://gateway.docker.internal:3128, and a post-login config should be owned by agent with restrictive permissions. The unconditional job-start hook never propagates that gateway proxy to workflow steps. On a Windows overlay runner it sets only HTTPS_PROXY/https_proxy to the raw-tunnel workflow listener at http://127.0.0.1:3130, after proving the authenticated EPAR relay is live; plain HTTP and other protocols remain direct. The private Docker daemon separately uses the TLS-terminating listener at http://127.0.0.1:3129. Otherwise the hook clears all proxy variables. If the GITHUB_ENV boundary or relay validation fails, the job stops before user steps. If Docker reaches the registry and returns an authorization response, do not investigate CA copying unless an x509 or TLS error is also present.
Inspect the host Docker Sandboxes daemon log for a message that the proxy is overriding a client-supplied registry credential with a host credential. When that message is present, the workflow credential was written correctly but dockerd used the credential-injecting forward path. An explicit guest config path cannot bypass that path.
On a Windows controller with image.hostTrustMode: overlay, /run/epar/egress-relay-active must be a root-owned regular marker, docker info must report http://127.0.0.1:3129 as its HTTPS proxy and an empty HTTP proxy, and sbx policy log <sandbox-name> must contain a fresh allowed transparent record for the controller relay port, normally rendered as localhost:<port>. A fresh Docker Hub forward record during activation is an admission failure. NoProxy=* is only the pre-activation bootstrap contract or the runtime contract for configurations that do not require the Windows relay; a Windows overlay job never falls back to that route. In either mode /etc/docker/daemon.json must be root-owned and non-symlinked. Stop the controller, let exact cleanup finish, rebuild, and use a newly created runner after any contract mismatch.
If host-trust relay activation fails, start with work/logs/epar-last-error.log. The error includes a fixed stage such as private-dockerd-contract, registry-tls-proof, or guest-bridge-health; use that stage to select the matching guest and Docker Sandboxes diagnostics. These stage messages are intentionally redacted and do not include the relay token or configuration payload.
If a transparent Docker Sandboxes connection presents Norton Web/Mail Shield Untrusted Root or another antivirus-generated untrusted issuer, do not add that issuer to EPAR's trust overlay. It is a synthetic error certificate indicating that the inspector rejected Docker Sandboxes' upstream path, not a missing ordinary inspection root. Current Windows overlay runners avoid that path by making the native controller host establish the public TCP connection. Workflow clients retain end-to-end TLS through the raw listener and validate the normal host-approved inspection chain with /opt/epar/trust/ca-bundle.pem. The Docker listener terminates daemon TLS with a root-only per-sandbox ephemeral authority and independently verifies the upstream host-approved chain, working around Docker Engine's stalled HelloRetryRequest without exposing registry credentials to the host relay. Failure of either listener, daemon restart/readback, Registry TLS proof, or fresh relay policy evidence blocks registration.
Do not set DOCKER_SANDBOXES_NO_PROXY expecting it to disable credential injection. That host variable only excludes destinations from an optional upstream proxy used after traffic reaches the mandatory Sandbox proxy. Replacing docker/login-action with docker login, combining login and pull in one shell step, or changing DOCKER_CONFIG also leaves an old daemon's forward route unchanged.
Keep the host sbx login identity intentionally different from the workflow identity when proving this fix. A successful private pull together with a fresh transparent relay-port entry and no fresh Docker Hub forward entry proves that the guest credential is authoritative. Changing the host login to match the workflow can diagnose the old interception behavior, but it is a shared-identity workaround rather than the fix.
EPAR does not query or mutate host-global sbx secrets. After creating its exact sandbox, provider admission rejects any nonempty authentication capability actually attached to that sandbox without exposing credential metadata, then passes the Docker Sandboxes gateway proxy only to registration and the Actions listener. The hook's GITHUB_ENV validation and local relay substitution are scoped to the current runner's _runner_file_commands directory; they do not mutate unrelated sandboxes or sbx login. A root-capable workflow can still deliberately reconnect a client to Docker Sandboxes' gateway proxy, and v0.37.1 has no documented per-sandbox switch that disables the interceptor. Arbitrary nested images also need their own CA installation because the host bundle path does not exist inside them. Use a least-privilege host sbx account and choose Docker Container if that residual capability is outside the trust boundary. See Docker Hub Credentials and Transparent Egress.
Docker Sandboxes starts a best-effort apt metadata refresh during guest boot. EPAR's bundled helper recognizes the complete legacy and v0.43.0 command identities, verifies the exact shell and apt-get executables and argument vectors, and terminates only that bootstrap refresh before runner registration. It never kills an arbitrary apt or dpkg process by name alone.
If another package-manager process remains after the bounded wait, the guest log reports EPAR_CANDIDATE_RECOVERY=package-manager-contention before the process diagnostic. The normal registered, replacement-enabled supervisor keeps healthy peers running, quarantines this candidate, preserves the strict physical-capacity fence until exact cleanup succeeds, and retries allocation using the candidate-recovery backoff. This path works with --external-outage-retry=off and continuous; the external-outage option does not control it. One-shot verification and replacement-disabled runs still return the candidate failure.
If the human timeout diagnostic appears without the recovery marker, treat it as an older or modified guest helper and rebuild or refresh the configured Docker Sandboxes template from this checkout. EPAR intentionally does not infer recoverability from diagnostic prose. Preserve the bounded process listing in the guest transcript when a marked candidate repeats; raw workflow logs and credentials are not collected automatically.
A short-lived ephemeral runner can register, accept work, and end between EPAR readiness observations. The controller may report that the listener ended before readiness was observed or that readiness remained uncertain through its deadline. This message describes the observed lifecycle only; it does not prove that a job ran or completed. Check the GitHub Actions run for the authoritative job result.
For the normal registered, replacement-enabled supervisor, this is a recoverable candidate event during initial provisioning and steady-state replacement. EPAR keeps running, preserves healthy peers, retains the candidate in quarantine and inside the strict pool.instances cap, and uses exact provider and GitHub reconciliation to decide whether the candidate can be adopted or safely removed. It does not create another candidate while uncertain or cleanup-pending capacity still occupies the slot.
Allocation backs off using pool.replacementRetryInitialSeconds, pool.replacementRetryMultiplier, pool.replacementRetryMaxSeconds, and pool.replacementRetryJitterPercent. Reconciliation, safe cleanup, peer health monitoring, and host-trust lease maintenance continue while allocation waits. Repeated failures can therefore leave the pool visibly below desired ready capacity while the controller remains available and retries at the capped delay. A successful provision or verified healthy adoption resets the candidate-recovery backoff.
This recovery does not require --external-outage-retry and does not turn authentication, trust, admission, configuration, durable-state, or unrelated local failures into retryable events. One-shot verification and operation with replacement explicitly disabled keep their existing behavior. If recovery does not progress, preserve the candidate's instance log and controller output, then inspect its exact local and GitHub identity; do not manually remove it by prefix or clear lifecycle state.
An unowned-runner warning means GitHub returned a name under the configured pool prefix but the current durable lifecycle state does not contain the same nonzero runner ID in a live record. Prefix similarity alone cannot authorize deletion because another controller, configuration, or historical attempt may have created the registration. EPAR therefore leaves it report-only. The remote registration does not consume local pool.instances capacity and does not block healthy runner provisioning.
The first observation of an exact runner name and ID is a warning. Repeated observations and status or busy-state flapping are suppressed and summarized at most once every 30 minutes with the latest state, observation count, and episode duration. A different runner ID is a new identity and is reported immediately. Successful reconciliation reports once when an announced identity is no longer classified as prefix-only report-only in that reconciliation; this transition can reflect remote absence or a change in exact reconciliation state and does not assert that GitHub deleted the registration. The cooldown survives disappearance and reappearance so flapping cannot restart the flood. GitHub API errors do not clear the episode or produce a false resolution message. The interval is a code constant and does not change reconciliation frequency.
One possible source is an interrupted registration attempt. A provider command session can disconnect while the guest-side Actions configuration briefly continues; a single immediate GitHub absence result can then precede the registration becoming visible. Preserve the candidate guest transcript, lifecycle record, provider audit log, exact sandbox identity, and surrounding timestamps. Do not delete by prefix or clear lifecycle state. Resolve an existing offline registration manually only after verifying that no live lifecycle record, local provider instance, job, or other controller owns that exact name and runner ID.
When a Docker Sandboxes guest-session command path stops responding, EPAR can still read global sandbox inventory while sbx exec, stop, or delete calls for an exact sandbox time out. A failed host-trust lease is fail-closed: EPAR quarantines the local instance and deletes only its immutable name-and-ID GitHub registration so it cannot accept new work after the lease expires. The resulting cleanup-pending instance remains inside the strict physical-capacity cap. Seeing no GitHub runner alongside a live EPAR process therefore means the safety fence is working; it does not prove the sandbox was removed or that a job completed.
The supervisor no longer probes cleanup-pending instances as runnable peers. It retries one exact cleanup target per maintenance allowance and carries the target identity forward when the caller budget ends. One such deadline cannot restart shared infrastructure. On a later observation, default dockerSandboxes.recoveryMode: exclusive-auto performs a fresh exact identity readback and harmless command. A healthy or authoritatively absent result resumes reconciliation; only a provider-owned failure from that fresh probe authorizes the existing host-wide coordinated recovery. An identity mismatch, cancellation, or unavailable proof retains capacity and backs off. Healthy peers continue lease maintenance during the probe.
Check for the explicit host-trust safety fence deleted the exact GitHub runner registration and independent provider command-path verification messages. Preserve the controller and instance logs, and use ./start status --no-github to inspect local lifecycle state. Do not delete lifecycle state, reset Docker Sandboxes, or create a manual replacement while exact cleanup is unresolved. Docker Desktop and OrbStack are host Docker-compatible engines used by template construction; this incident path belongs to Docker Sandboxes' separate sandboxd daemon and guest sessions.
A GitHub 429/5xx response or an sbx command timeout makes runner health temporarily unknown; it does not prove that the Actions listener stopped. EPAR preserves uncertain local capacity, but a runner whose host-trust transport or lease cannot be maintained is quarantined and its exact GitHub registration is fenced after immutable name-and-ID verification so it cannot accept new work with an expired lease. Cleanup for an inactive listener requires two consecutive guest probes that successfully execute and explicitly report the process stopped. Review the instance guest transcript when warnings repeat; do not delete the runner merely because one API or Sandbox inspection failed.
Health monitoring shares time with host-trust maintenance. With host trust enabled, each monitoring pass has a bounded budget so health checks cannot indefinitely delay lease refresh. A monitoring-budget deadline is different from a provider command exceeding its own deadline: neither proves a stopped listener, and one controller-owned deadline cannot trigger a shared-daemon restart. A repeated exact-instance deadline may prompt a fresh independently bounded harmless probe; only a typed provider failure from that fresh probe can authorize coordinated recovery. Increasing timeouts.commandSeconds does not extend the shorter monitoring budget. EPAR resumes health work fairly rather than checking further runners with an already-expired context.
Timing estimates for instance checks belong to each exact runner, rise immediately when checks slow down, and decay gradually after faster successful checks. An idle runner therefore cannot overwrite a busy runner's timing estimate. These estimates only schedule work; they do not extend the monitoring budget or suppress genuine failures. Initial and replacement provisioning keep survivor leases maintained independently, preserving one-time busy-job handoffs and fail-closed expiry. Ordinary Docker Sandboxes commands can overlap, but an exclusive recovery admission blocks new commands until existing commands drain. A timeout containing command not started was spent waiting for that admission rather than executing a guest or inventory command. Debug logs distinguish command admission wait from execution duration; failed lease writes label any subsequent revocation failure separately.
A template inventory parse failure means EPAR could not establish the cache contents, not that the template is missing or corrupt. EPAR retries the read once and, if it still fails, reports a fixed parsing category and stdout byte count without printing the provider payload. It retains uncertain catalog resources rather than treating the result as an empty cache. If the warning persists, capture bounded sbx template ls --json output for local inspection; do not delete templates based on this warning alone.
The first unknown-health warning is immediate; repeated warnings for the same exact runner are limited to one every five minutes. The summaries retain the latest failure, failure count, episode duration, check stage, deadline category, and time since the last successful health verification. A return to a known health state ends an announced episode with a recovery message. The warning cooldown survives intermittent recovery so a flapping check cannot restart the flood. A quiet console between summaries does not mean the check succeeded; logging.level: debug includes phase-level results and durations when investigating progress. Do not disable host-trust protection or suppress all warnings to work around health-check timeouts; persistent failures still need investigation.
networkBaseline: open is a sandbox-scoped public-egress compatibility rule with EPAR host-alias deny guardrails. It does not alter the host-global policy. If a required service is blocked, use a narrow additionalAllow hostname rule; do not allow host.docker.internal, gateway.docker.internal, kubernetes.docker.internal, or host.containers.internal through the Open-policy guardrails.
Plain ./start intentionally keeps fail-fast startup behavior for typed external dependency incidents during a human-attended invocation. Candidate listener exit and readiness uncertainty use the separate default recovery described above. For an unattended host, start the configured controller with --external-outage-retry=continuous or a bounded duration such as --external-outage-retry=4h. Use ./start status --no-github to inspect the local incident stage, attempt, next retry, and deadline even while GitHub is unavailable.
Only typed transient failures such as HTTP 408, 429, 5xx, rate-limited 403 responses, DNS failures, connection resets/refusals, and timeouts are retried. Correct authentication, ordinary authorization, certificate trust, missing image manifests, storage, local Docker or Sandbox readiness, ownership, configuration, platform, and custom-script failures instead of expecting outage supervision to mask them. EPAR never queries GitHub Status and never silently changes a provider, source image, registry, credential, or trust policy.
A GitHub App installation token can be rejected with HTTP 401 before EPAR's cached expiry. For an installation-authenticated GitHub request, EPAR invalidates that rejected cached token, obtains the current installation token, and repeats the logical request once. A second 401 remains terminal because it indicates a persistent authentication or installation problem; EPAR does not place repeated 401 responses into the external-outage retry loop. Verify github.appId, the private key, the organization installation, and Actions runner permissions when a 401 persists.
The public actions/runner Releases API has a separate unauthenticated rate limit. An EPAR GitHub App installation token is not used for that public repository because it may not have permission to read it. On a 403, 429, timeout, or transient server response, EPAR prints the HTTP status and any safe rate-limit, retry, and request-ID headers, then uses the reviewed exact release in third_party/actions-runner-release.lock.json. This is an explicit metadata-only fallback: the selected version must match image.runnerVersion when that value is pinned, the expected asset name and canonical GitHub URL are checked, and the downloaded package must still match the lock's SHA-256 before it is installed.
If EPAR says the checked-in fallback is unusable, restore that file from the EPAR release or update to a release with a refreshed lock; do not replace the digest with an unverified value. Authentication, malformed successful API responses, selector mismatches, missing assets, and other non-transient API failures remain errors rather than using the fallback.
Docker Desktop can supply a stored ghcr.io credential to Buildx even when the requested Catthehacker image is public. A stale, revoked, wrong-account, or insufficiently scoped credential can then fail authorization instead of using anonymous public access. When the failed operation is authorization-shaped, EPAR performs one separate read-only anonymous OCI descriptor probe using the configured build-trust roots. If that exact built-in source is anonymously readable, the error explains the credential conflict. EPAR does not retry the failed pull anonymously, run docker logout or tart logout, or alter Docker Desktop's credential store.
For public-only GHCR use, remove or refresh the stored credential yourself. For private GHCR use, sign in again with a valid credential that has package pull access. Removing a host credential may also remove access to private packages, so EPAR never performs that action automatically. If the anonymous diagnostic probe cannot prove public access, EPAR preserves the original failure without speculating about the credential.
Run ./start status to see the last successful remote check, next check or retry, pending immutable identity, deferred reason, and last error. A failed scheduled check or build keeps the previous exactly verified generation available and retries with bounded backoff; a missing artifact or changed local configuration still fails closed. Use ./start image update to retry an immediate remote check, or correct local input errors and rerun ./start.
An instance is retained, quarantined, or shown as requiring an acknowledgement after a provisioning, policy, or runtime failure.
Preserve the instance and inspect work/logs/instances/<instance>.guest.log, the matching runner diagnostics, and controller output before acknowledging or removing it. EPAR deliberately keeps uncertain ownership, failed cleanup, and unverified remote state inside the strict pool.instances cap instead of creating a replacement storm.
If an incident requires stopping new work immediately, stop the controller with Ctrl-C or the service manager that launched it. This prevents replacement; it does not erase retained evidence. Use the configured EPAR cleanup command only after identifying the exact affected pool. Do not use a broad docker system prune, WSL unregister, or reset as an incident-disable switch.
Set EPAR_DISABLE_DOCKER_SANDBOXES=1 before starting EPAR when Docker Sandboxes admission must remain disabled during an incident or compatibility investigation. This fails the provider closed without changing configuration or deleting evidence.
After reviewing retained Docker Sandboxes diagnostics, acknowledge that review only for the exact configured pool:
./start cleanup --acknowledge-failed-diagnosticsstart or image build reports No space left on device or E: You don't have enough free space in /var/cache/apt/archives/..
The temporary guest or Docker writable layer is full; this does not necessarily mean the host OS drive is full. Inspect the active Docker daemon:
docker run --rm ghcr.io/catthehacker/ubuntu:full-latest df -h /
docker system df
docker system df -vIncrease the relevant Docker/VM data-disk allocation or deliberately remove unneeded data after reviewing it. Docker prune commands can remove stopped containers, unused images, build cache, networks, and volumes; they are not a safe generic fix.
Old EPAR images, Docker Sandboxes templates, staging archives, or no-Go controller files remain after an update or an interrupted start.
Run ./start once more. Startup reconciles incomplete exact-owned work and retires an unreferenced superseded generation after its replacement passes readback. It does not delete shared images, prefix-only historical resources, active containers, active sandboxes, or resources referenced by another configuration.
Inspect the exact classification before removing anything manually:
./start storage status
./start storage prune
./start storage prune --legacyUse normal storage prune --execute only for exact catalog-owned resources. Legacy prefix-era entries require the plan hash printed by storage prune --legacy; they are not removed automatically. Do not use broad Docker prune/reset commands or VHDX compaction as a substitute for this review.
After .local/state, .local/cache, a Docker image, or a Docker Sandboxes template was removed manually, startup reports a missing project-local receipt/evidence path or discovers stale external ownership.
Current EPAR treats a genuinely missing cataloged receipt or compact evidence file as unavailable cache rather than trusted ownership evidence. It skips reuse of that external template and reacquires or rebuilds the configured immutable artifact; malformed JSON, digest disagreement, redirected paths, and non-regular evidence still fail closed as integrity errors.
First stop other controllers for the same configuration and start once more. If recovery remains blocked, preview an exact reset:
./start storage status --config .local/config.yml
./start storage reset --config .local/config.ymlReview every exact target and shared-resource line. Execute only with the new preview hash printed by the command. The config, GitHub App key, certificates, and custom scripts are preserved, while the next start performs a cold acquisition. See Generated files and recovery before deleting project or host data manually.
After switching between Docker Desktop, OrbStack, or another local Docker daemon, Buildx output names the previous context or socket, reports a node-level connection error despite a successful docker buildx inspect exit status, or reports that the expected buildx_buildkit_epar-... control container does not exist on the current daemon.
Current EPAR controllers record the Docker Engine identity with each config-scoped Buildx receipt. When valid ownership metadata proves that the stale definition belongs to the current project and configuration, EPAR automatically removes only that exact client-side definition while preserving daemon-local BuildKit state, recreates it against the active backend, bootstraps it, and verifies its image, configuration, registry trust, and CA bundle before continuing. Missing control containers and node errors embedded in otherwise successful Buildx output receive the same bounded recovery. No manual docker rm, docker buildx rm, or broad prune is normally required.
EPAR never adopts or removes a same-name builder without valid exact ownership metadata. If automatic recovery reports that the definition remained present, or the recreated builder fails verification twice, preserve the error report and inspect .local/storage/buildx/<config-id>/metadata.json together with docker context show, docker info, and docker buildx inspect <builder>; repair the real daemon or ownership problem rather than deleting shared Docker state. A context that changes repeatedly during one reconciliation is detected and retried once, but the Docker target must remain stable for the image build itself. An unavailable daemon, failed registry access, or failure to produce a required verified artifact can still block startup.
HTTPS access fails with curl: (60), certificate verification failed, or an unknown issuer during an EPAR build or job.
Do not disable certificate verification. First identify which trust boundary failed.
For a no-Go native-controller build, ./start automatically reads host system roots, excludes explicitly distrusted certificates, validates the short-lived feed in an offline container, and mounts only the resulting CA bundle into the Go compiler container. Runner CA inheritance remains independent. If the build still reports an unknown issuer, inspect work/logs/epar-native-controller-build.log: the wrapper prints the requested host, presented certificate subject and issuer, SHA-256 fingerprint, validity, and verification result, and on Windows it lists matching roots from LocalMachine\Root and CurrentUser\Root. A remaining failure means the expected issuer was absent, distrusted, malformed, expired, or not the certificate actually presented; EPAR never disables TLS verification or retries insecurely.
For an EPAR Buildx failure, leave image.hostTrustMode unchanged. EPAR automatically supplies host system roots to its config-owned builder and prints the full build transcript path before docker buildx build. The console and error report include a bounded redacted tail. Inspect the underlying x509 line together with the registry host, builder identity, and active trust generation:
docker buildx ls
Get-ChildItem .local/storage/buildx -Recurse -Filter metadata.json | Get-Content
Get-ChildItem .local/storage/buildkit -Recurse -Filter buildkitd.toml | Get-ContentThe owned metadata records the Docker backend identity, exact registry set, configuration digest, certificate bundle, and trust generation. Rerunning the same command reconciles that exact builder and preserves its daemon-local BuildKit state; EPAR never changes Docker's shared/default builder. If the source-image docker pull itself fails before Buildx starts, configure the authorized CA in the host daemon because builder trust cannot repair host-daemon trust.
Configure runner overlay only when jobs inside an ephemeral runner must inherit host roots:
image:
hostTrustMode: overlay
hostTrustScopes: [system, user]Use [system] on Linux. Overlay mode collects the current host roots, validates them before registration, and combines them with Ubuntu roots and any image.trustedCaCertificatePaths; it is root-anchor inheritance rather than exact Windows/macOS TLS-policy emulation. It requires runner.ephemeral: true. Omitted or disabled mode remains valid for Docker Sandboxes; the unconditional job-start preparation hook still runs and accepts the explicit disabled marker while clearing workflow proxy variables.
Use the normal host entry point so EPAR can inspect the real Windows certificate stores or macOS Keychain:
./start
./start image build --replaceUse ./start image build --replace on every supported shell, including native Windows PowerShell. The wrapper uses a native-host trust feed while compiling the native controller; the resulting native controller reads host trust directly for start, image build, pool up, and pool verify, even when runner overlay is disabled. Direct scripts/run-with-docker.* calls are wrapper-development diagnostics. A bare Linux toolchain container is not a replacement for the project-local native-controller path.
If Windows wrapper startup reports that a trust watcher did not become ready, inspect the expected PID, observed lock owner, ready marker, and feed rejection reason in that message. Timestamp diagnostics distinguish an expired or future-dated snapshot from a read/parse failure; filesystem failures include the exception type and HRESULT without certificate contents. Readiness polls read one snapshot at a time and tolerate transient publication overlap within the existing startup deadline. A persistent invalid feed still blocks startup: do not extend the 30-second freshness window or reuse an older snapshot to bypass it.
docker run --rm ghcr.io/catthehacker/ubuntu:full-latest df -h / shows much less capacity than Windows Explorer.
Docker Desktop stores Linux container data in a WSL-backed virtual disk. Older WSL2 installations can have a smaller default VHD maximum than newer ones, but the reported container filesystem is the evidence that matters for image pulls and builds. Inspect Docker usage first, then change Docker Desktop/WSL storage using the product's supported settings. See Microsoft WSL disk-space guidance and Docker Desktop WSL guidance.
Docker Container requires a host Docker runtime that permits privileged Linux containers:
docker run --rm --privileged alpine:3.20 trueIf nested Docker operations fail with overlay-mount errors, retain the default inner storage driver:
EPAR_DOCKERD_STORAGE_DRIVER=vfs
Use overlay2 or auto only in a derived image after proving it works on the exact host runtime.
The WSL image build fails before import, during import with 0xffffffff, or before systemd is ready.
The default WSL build obtains a Docker source image before importing it into WSL. Verify Docker and WSL first:
docker version
docker pull ghcr.io/catthehacker/ubuntu:full-latest
wsl -l -vFor Wsl/Service/CreateInstance/E_UNEXPECTED, Catastrophic failure, or import exit 0xffffffff, stop EPAR, save work in other distros, then run wsl --shutdown. This stops every running WSL distro, including any Docker backend using WSL. Restart the affected Docker host runtime, verify a normal distro command returns 0, then rerun ./start; a matching cached source rootfs is reused. If it persists, update WSL, shut it down again, reboot, and consult Microsoft's WSL troubleshooting guidance.
If a guest exists but systemd does not become ready, inspect work/logs/builds/<image>.wsl-build.log and work/logs/builds/<temporary-distro>.guest.log. Do not unregister a distro until you have identified the exact EPAR-owned target and accepted that unregistration is irreversible.
EPAR cannot request a registration token, add a runner to a group, or observe the runner online.
Verify GitHub App organization self-hosted-runner read/write permission and a readable private key:
github:
appId: 123456
organization: your-org
privateKeyPath: .local/github-app.pemThen inspect runner-group policy and the first registration error. A strict policy can intentionally block a group that is default, overly broad, or public-repository enabled. See Runner Group Security.
For a confirmed stale EPAR resource, run the configured cleanup command:
./start cleanupCleanup is bounded by the configured pool and exact lifecycle/provider evidence; a configured prefix can authorize only a provider-specific orphan recovery after exact identity, workspace, receipt, and absence checks, never a broad prefix deletion, wildcard, Docker prune, or removal of unknown/shared resources. Keep pool.namePrefix unique per controller and organization.
Generated from the main repository docs at 8526d9b. Edit README.md and docs/; the wiki copy is overwritten by automation.