Skip to content

fix(provider-proxy): stop one slow status probe from restarting the only replica - #594

Merged
baktun14 merged 1 commit into
mainfrom
fix/provider-proxy-tolerant-probes-and-replicas
Sep 25, 2026
Merged

baktun14 merged 1 commit into
mainfrom
fix/provider-proxy-tolerant-probes-and-replicas

Conversation

@baktun14

@baktun14 baktun14 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Why

provider-proxy-mainnet runs a single replica, and both of its probes call /status with a 5s timeout and failureThreshold: 1. One answer slower than 5 seconds is enough for the kubelet to restart the container, and with one replica that takes the whole service down until the new container is ready.

That happened on 2026-09-23 at 15:21 UTC and again on 2026-09-25 at 10:26 UTC. Both times the container got SIGTERM in place: the pod was not replaced, nothing else in the cluster restarted, and the process shut down cleanly with exit code 0, which is why the restart shows up as Completed. While it was down, console-api could not reach provider-proxy at all. Every lease created in that window failed its provider reachability check with a 500 and paged the 5xx alert, and every open log, event and shell stream dropped.

The /status handler also gets slower the longer the process runs, because it walks the stats of every websocket opened since startup (2 ms on average right after a restart, 137 ms after two and a half days). That part is fixed in akash-network/console#4055, and akash-network/console#4056 runs two replicas in prod mainnet. A single slow answer still shouldn't cost the only replica.

What

  • Read replicas from values ({{ .Values.replicas | default 1 }}, the same pattern console-api and provider-inventory use) and add replicas: 1 to values.yaml, so sandbox and staging keep one replica. Prod mainnet will set 2 from the console repo's .helm values.
  • The liveness probe restarts the container after 3 failed checks in a row (30 seconds) instead of 1.
  • The readiness probe takes a pod out of the Service after 2 failed checks instead of 1.
  • Bump the chart to 0.7.7 and move the nodejs-app pin from 0.4.1 to 0.4.2, refreshing Chart.lock. With the old pin helm dependency build fails with can't get a valid version for dependency nodejs-app, so the bumped chart would not be published. 0.4.2 only adds the optional extraNodeOptions, which renders nothing when it isn't set.

Verified with helm lint and helm template. With default values the rendered manifests differ only in the two failureThreshold values, and --set replicas=2 changes only spec.replicas.

…nly replica

Both probes called /status with a 5s timeout and failureThreshold 1, and
the deployment ran exactly one replica, so a single answer slower than 5s
restarted the container and took provider-proxy down until the new one was
ready. That happened on 2026-09-23 at 15:21 UTC and on 2026-09-25 at
10:26 UTC.

Liveness now restarts after 3 failed checks in a row and readiness drops
the pod after 2. The replica count is read from values, defaulting to 1.
The chart moves to 0.7.7 and the nodejs-app pin to 0.4.2, because with the
0.4.1 pin helm dependency build fails and the chart would not publish.
@baktun14
baktun14 merged commit be1494e into main Sep 25, 2026
2 checks passed
@baktun14
baktun14 deleted the fix/provider-proxy-tolerant-probes-and-replicas branch September 25, 2026 17:53
baktun14 added a commit that referenced this pull request Oct 2, 2026
…om values (#597)

stats-web runs one replica with only a liveness probe on /api/healthz
(1s timeout, no
readiness probe). A short CPU burst that delays one healthz answer past
a second gets the
container killed, and with one replica stats.akash.network is down until
it boots again.
That happened on 2026-10-02 at 02:22 UTC.

- Startup, readiness and liveness probes match console-web (#593): 3s
timeouts, a 120s
boot budget, readiness drops a pod after 2 failures, liveness restarts
after 3.
- Read replicas from values ({{ .Values.replicas | default 1 }}, as in
#594) and default
to 1, so staging keeps one replica. Prod sets 2 from the console repo's
.helm values.
- Bump the chart to 0.2.5 and move the nodejs-app pin from 0.4.1 to
0.4.2, refreshing
Chart.lock. With the old pin helm dependency build fails, so the chart
would not publish.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants