Load-balancer-aware rolling deploys: take each node out of rotation, drain, deploy, rejoin when healthy #2975

Summary

Roll a replicated deploy_component through the cluster one node at a time behind the load balancer, the way upgrades already roll:

1. Take the node out of rotation.
2. Drain its traffic. This is the default; a setting skips it for faster deploys.
3. Deploy and restart.
4. Wait until the node is healthy.
5. Put it back into rotation, then move on to the next node.

Every node is out of rotation the whole time it changes, so no client ever reaches a node mid-swap, and a release that fails its health gate never takes traffic on any node.

Why

• restart: 'rolling' today restarts nodes one at a time, but each node stays in rotation the whole time. Its workers are replaced while it keeps taking traffic, and the release is live on its disk before any of them has loaded it.
• Staged component deploys: land build-aside-then-swap as a sequence of small PRs #2315 step 2 (canary rollout certification, Certify a restarting deploy in a canary worker before it rolls out #2981, shipping in v5.4) narrows that, but only per worker. It holds one replacement worker out of traffic until the component loads, and restores the previous release if it does not. The node stays in rotation, though, and its other workers keep serving while the canary decides.
• Step 2's planning review asked for certification before the swap, so that an uncertified release can never become a node's boot target or take traffic. Taking the node out of rotation for the whole change gives that guarantee at the cluster level. It does so without making the per-node activation transaction (.deploy-staging swap, journal, recovery) any more complicated.
• Upgrades already roll behind the traffic director. Deploys do not (harper-pro#899).

What already exists

• Out of rotation: the availability status (set_status { id: 'availability', status: 'Unavailable' }, server/status/definitions.ts). The traffic director reads it. Optionally keep a node out of rotation (or gate component load) until secondary indexes are built #1358 proposes driving it from index-build state.
• One node at a time: the restart: 'rolling' job (restart_service with replicated: true, bin/restart.ts). It restarts the origin, then each peer in sequence.
• Build everywhere, switch later: staged deploys. activate: false builds the release on every node without loading it, and deployment_id then activates it at that node's turn without rebuilding, certifying it there in a canary when that activation restarts (Staged component deploys: land build-aside-then-swap as a sequence of small PRs #2315 steps 2, 5 and 6).
• A health signal: component status from get_status and the step 2 canary verdict.

Sketch, open for design

• Settings: per deploy, with a config default.
  • Rotation mode: drain (the default), immediate (no drain), or off, which is today's behavior.
  • A drain deadline.
  • A health-gate deadline.
  • What a failure does: stop the rollout, or continue and report.
• Order:
  • Stage the release on every node first.
  • Then, for each node in turn: mark it Unavailable, drain (bounded), activate and restart, run the health gate, mark it Available, move on.
• Quorum guard: never take the last node, or a majority, out of rotation at once. A single-node cluster deploys in place.
• Failure: stop at the failing node, restore its previous release (step 5), return it to rotation, and report which nodes run which release.

Open questions

• Who owns rotation in each environment: the GTM, the Symphony load balancer, or only the availability status?
• What counts as drained: in-flight HTTP requests, idle keep-alive connections, WebSocket/MQTT sessions, replication traffic?
• How long-lived connections (WebSocket, MQTT, SSE) are handed off, and whether drain waits for them or closes them.
• How it relates to restart: true and 'rolling': a new mode, or what 'rolling' becomes?
• How it reports partial rollouts and mixed-version clusters (cross-node ordering is Two-phase deploy: concurrent activations from different origins can leave the cluster on different versions with both reporting success #2294).

Related: #2315, #641, #1358, #2294, harper-pro#899.