Repository navigation
fix(provider-proxy): stop one slow status probe from restarting the only replica - #594
Merged
Merged
Conversation
…nly replica Both probes called /status with a 5s timeout and failureThreshold 1, and the deployment ran exactly one replica, so a single answer slower than 5s restarted the container and took provider-proxy down until the new one was ready. That happened on 2026-09-23 at 15:21 UTC and on 2026-09-25 at 10:26 UTC. Liveness now restarts after 3 failed checks in a row and readiness drops the pod after 2. The replica count is read from values, defaulting to 1. The chart moves to 0.7.7 and the nodejs-app pin to 0.4.2, because with the 0.4.1 pin helm dependency build fails and the chart would not publish.
This was referenced Sep 25, 2026
chalabi2
approved these changes
Sep 25, 2026
baktun14
deleted the
fix/provider-proxy-tolerant-probes-and-replicas
branch
September 25, 2026 17:53
baktun14
added a commit
that referenced
this pull request
Oct 2, 2026
…om values (#597) stats-web runs one replica with only a liveness probe on /api/healthz (1s timeout, no readiness probe). A short CPU burst that delays one healthz answer past a second gets the container killed, and with one replica stats.akash.network is down until it boots again. That happened on 2026-10-02 at 02:22 UTC. - Startup, readiness and liveness probes match console-web (#593): 3s timeouts, a 120s boot budget, readiness drops a pod after 2 failures, liveness restarts after 3. - Read replicas from values ({{ .Values.replicas | default 1 }}, as in #594) and default to 1, so staging keeps one replica. Prod sets 2 from the console repo's .helm values. - Bump the chart to 0.2.5 and move the nodejs-app pin from 0.4.1 to 0.4.2, refreshing Chart.lock. With the old pin helm dependency build fails, so the chart would not publish.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
provider-proxy-mainnet runs a single replica, and both of its probes call
/statuswith a 5s timeout andfailureThreshold: 1. One answer slower than 5 seconds is enough for the kubelet to restart the container, and with one replica that takes the whole service down until the new container is ready.That happened on 2026-09-23 at 15:21 UTC and again on 2026-09-25 at 10:26 UTC. Both times the container got SIGTERM in place: the pod was not replaced, nothing else in the cluster restarted, and the process shut down cleanly with exit code 0, which is why the restart shows up as
Completed. While it was down, console-api could not reach provider-proxy at all. Every lease created in that window failed its provider reachability check with a 500 and paged the 5xx alert, and every open log, event and shell stream dropped.The
/statushandler also gets slower the longer the process runs, because it walks the stats of every websocket opened since startup (2 ms on average right after a restart, 137 ms after two and a half days). That part is fixed in akash-network/console#4055, and akash-network/console#4056 runs two replicas in prod mainnet. A single slow answer still shouldn't cost the only replica.What
replicasfrom values ({{ .Values.replicas | default 1 }}, the same patternconsole-apiandprovider-inventoryuse) and addreplicas: 1tovalues.yaml, so sandbox and staging keep one replica. Prod mainnet will set 2 from the console repo's.helmvalues.nodejs-apppin from 0.4.1 to 0.4.2, refreshingChart.lock. With the old pinhelm dependency buildfails withcan't get a valid version for dependency nodejs-app, so the bumped chart would not be published. 0.4.2 only adds the optionalextraNodeOptions, which renders nothing when it isn't set.Verified with
helm lintandhelm template. With default values the rendered manifests differ only in the twofailureThresholdvalues, and--set replicas=2changes onlyspec.replicas.