[Feat][Router]: Add automatic retry with exponential backoff and jitter - #939

Overview

Adds opt-in automatic retry for transient backend failures using exponential backoff with jitter, and folds the existing instance-failover reroute into the same mechanism.

Motivation: Mitigating the thundering herd problem

A forwarded request fails in one of two ways, which call for opposite responses:

Both draw on one budget: --max-retries. Retrying is disabled by default, so a request is attempted exactly once unless --enable-retries is passed.

CLI

Without --enable-retries the other flags are ignored, so a shared config template carrying them cannot change behaviour or block startup.

vllm-router --port 8000 \
    --service-discovery static \
    --static-backends "http://localhost:9001,http://localhost:9002" \
    --static-models "facebook/opt-125m,facebook/opt-125m" \
    --routing-logic roundrobin \
    --enable-retries \
    --max-retries 5 \
    --initial-backoff-ms 100 \
    --max-backoff-ms 60000 \
    --backoff-multiplier 2.0 \
    --jitter-factor 0.1

Backoff

delay  = min(initial_backoff_ms * multiplier ^ retry, max_backoff_ms)
delay' = delay * (1 + U[-jitter_factor, +jitter_factor])

With the defaults, retries land at 50ms, 75ms, 112.5ms, 168.8ms, each spread across a +/-20% window. Without the jitter, routers that backed off from the same incident retry in lockstep and re-create the overload they backed off from.

Changes

Validation lives in RetryConfig.__post_init__, so an invalid config cannot be constructed.

Behaviour

• A retryable status does not blacklist the engine, so it stays a candidate after the backoff. This is what makes retrying useful on a single-engine deployment.
• Failover between healthy engines is never delayed; the backoff applies only once every engine has been excluded and the pool is given another chance.
• Once the budget is spent the backend response passes through, so the client sees the engine's own status and body rather than a synthesised router error.
• Non-retryable statuses return on the first attempt.
• Retries apply only before streaming begins; once the first chunk is forwarded the headers are already with the client, so a mid-stream error passes through untouched.

Breaking change

--max-instance-failover-reroute-attempts is removed. It rerouted a failed request to another engine and is now subsumed by --max-retries, which covers the same case and adds backoff.

Its default was 0 (a single attempt), which is also the default here, so deployments that never set it are unaffected. Deployments that did set it should migrate:

- --max-instance-failover-reroute-attempts 2
+ --enable-retries --max-retries 3

Note for reviewers

process_request reports a backend error status by yielding it (yield backend_response.headers, backend_response.status), not by raising, and there is no raise_for_status on that call. A retry implementation that only catches HTTPException never fires for an engine 5xx. This PR inspects the status after the first anext, and aclose()s a discarded response so the upstream connection is released.

Tests

56 new tests, 281 passing suite-wide.

• test_retry_config.py — status classification, backoff growth, cap, jitter bounds, validation, from_args mapping.
• test_request_retry.py — end to end through route_general_request: default single attempt, transport reroute, no-backoff-on-failover, single-engine retry, budget exhaustion passthrough, non-retryable passthrough, generator close on discard, exact sleep sequence.

Every backend-status stand-in yields the status, matching process_request; one that raises HTTPException would pass whether or not retrying works.