Skip to content

fix(pipeline): stop idle keepalive polls, reduce idle requests#1322

Merged
therealaleph merged 2 commits into
therealaleph:mainfrom
yyoyoian-pixel:fix/idle-keepalive-backoff
May 20, 2026
Merged

fix(pipeline): stop idle keepalive polls, reduce idle requests#1322
therealaleph merged 2 commits into
therealaleph:mainfrom
yyoyoian-pixel:fix/idle-keepalive-backoff

Conversation

@yyoyoian-pixel
Copy link
Copy Markdown
Contributor

@yyoyoian-pixel yyoyoian-pixel commented May 20, 2026

Summary

Stops keepalive polling on truly idle sessions and adds gradual backoff with separate idle tier tracking. Reduces idle Apps Script requests from ~1200/5min to ~35/5min for truly idle sessions.

How it works

  1. Early backoff (first 3 empty replies): 20ms → 80ms → 4s → 10s
  2. Stop polling after idle_tier > 3 (~16s of empties): session stops sending keepalive polls entirely, just waits for client data
  3. Apps maintain their own heartbeats (MQTT PINGREQ, FCM keepalive, Telegram ping) which trigger client writes — those act as natural polls that pick up server-pushed data

Separate counters

  • consecutive_empty: controls pipeline depth management (resets to 0 on data — sessions can ramp up immediately)
  • idle_tier: controls keepalive delay (halves on server-pushed data instead of full reset — prevents push notifications from yo-yoing the backoff)

Measured results

Minute Requests Notes
0:00-1:00 23 Initial ramp-down
1:00-2:00 63 Sessions settling
2:00+ 7-50 Steady state (app heartbeats only)
Version Idle requests/5min
v1.9.13 (serial) ~300
v1.9.30 (pipelined, 2s cap) ~1200
v1.9.31 (pipelined, 2s cap) ~1000
This PR (truly idle) ~35
This PR (background apps active) ~250-350 (real traffic, not wasted polls)

Note: sessions receiving periodic server pushes (Instagram MQTT, Telegram, etc.) never reach idle_tier > 3 because they keep getting data. Those requests are real traffic, not wasted keepalive polls. True idle sessions (no server pushes) drop to zero polls.

No latency impact

  • tokio::select! races delay timer against client socket reads
  • When app writes data (heartbeat, user action), fires immediately
  • Pipeline depth uses consecutive_empty (resets to 0) — not affected by idle_tier

Test plan

  • Verified idle requests drop from ~1200 to ~35/5min for truly idle sessions
  • Verified background app traffic (Telegram, Instagram) still works
  • Verify active browsing/video works without added latency
  • Verify push notifications still arrive promptly
  • Test with customer's setup (15 deployments, Chrome + Instagram only)

🤖 Generated with Claude Code

@github-actions github-actions Bot added the type: fix fix: PR — auto-applied by release-drafter label May 20, 2026
@yyoyoian-pixel yyoyoian-pixel force-pushed the fix/idle-keepalive-backoff branch 2 times, most recently from c605acb to d943fef Compare May 20, 2026 12:20
@yyoyoian-pixel yyoyoian-pixel changed the title fix(pipeline): escalate idle keepalive backoff — reduces idle requests 6x fix(pipeline): stop idle keepalive polls — 1200/5min → ~35/5min May 20, 2026
@yyoyoian-pixel yyoyoian-pixel changed the title fix(pipeline): stop idle keepalive polls — 1200/5min → ~35/5min fix(pipeline): stop idle keepalive polls, reduce idle requests ~97% May 20, 2026
@yyoyoian-pixel yyoyoian-pixel changed the title fix(pipeline): stop idle keepalive polls, reduce idle requests ~97% fix(pipeline): stop idle keepalive polls, reduce idle requests May 20, 2026
@yyoyoian-pixel yyoyoian-pixel force-pushed the fix/idle-keepalive-backoff branch from d943fef to e870fb0 Compare May 20, 2026 12:39
Previous cap of 2s caused ~1200 requests/5min idle with 15 deployments.
New escalation: 20ms→80ms→200ms→500ms→2s→5s→10s→20s.
After 15+ consecutive empties, sessions poll every 20s.

Estimated idle reduction: ~1200/5min → ~200/5min.
Zero latency impact on active traffic — select! races timer against
client reads, so real data fires immediately.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@yyoyoian-pixel yyoyoian-pixel force-pushed the fix/idle-keepalive-backoff branch from e870fb0 to 259be25 Compare May 20, 2026 12:54
Copy link
Copy Markdown
Owner

@therealaleph therealaleph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally after adding the mixed-deployment guard: all-legacy idle sessions can stop polling, but mixed fleets keep emitting empty polls so round-robin can still reach a long-poll-capable peer.

Tests:

  • cargo test --lib tunnel_client::tests::tunnel_loop_keeps_polling_when_only_some_deployments_legacy -- --nocapture
  • cargo test --lib
  • cargo build --release

Answered via LLM, Supervised @therealaleph

@therealaleph therealaleph merged commit 02f2765 into therealaleph:main May 20, 2026
1 check passed
therealaleph added a commit that referenced this pull request May 20, 2026
Ship PR #1322 by @yyoyoian-pixel: Full Tunnel idle sessions now reduce empty keepalive request load, while mixed deployments keep polling so long-poll-capable peers can continue delivering remote-to-client data.

Local verification:
- cargo test --lib
- cargo build --release
- cargo build --bin mhrv-rs-ui --release --features ui
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix fix: PR — auto-applied by release-drafter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants