fix: forced health check and re-check storm on failure bursts#2984
fix: forced health check and re-check storm on failure bursts#2984IsoLeyN wants to merge 2 commits into
Conversation
|
Problem 1 Today I noticed different behavior in the program. It turns out that the GUI I'm using doesn't automatically populate the URL for url-test groups. I manually set it, and tests started running automatically when I launch the core. However, this doesn't solve the issue of having to wait for the fallback interval. In the video, I demonstrated the core's operation both without my code and with it. In the second part of the video, you can see how fallback immediately selects a group that has at least some content. Demonstration video: https://youtu.be/MRBBIPpImy8. Problem 2 Demonstration video: https://youtu.be/Vkm76fPFLiE. I placed a stopwatch nearby to track the time elapsed between tests. In my config, the interval is set to 900 seconds. In the first example, you can see that the servers start being tested in less than 2 minutes, which is incorrect. In the second example, I show the corrected code where the proper 900-second wait time is enforced. |
This PR fixes two related bugs in outboundgroup (fallback / url-test group logic) that affect setups using empty-fallback: REJECT.
Problem 1: fallback/url-test never recovers from an empty group
When a url-test or fallback group has no alive members, it resolves to its empty-fallback proxy (in this config, REJECT). The issue is that dialing through REJECT is not treated as a failure by the group: REJECT's DialContext returns a no-op connection with a nil error (it never actually connects anywhere), so onDialFailed is never called, the group's failure counter never increments, and the forced health check that would normally recover the group is never triggered.
In practice, this meant that if a url-test group's underlying provider was temporarily empty or unreachable at startup, the group would get stuck silently serving REJECT forever, only recovering at the next scheduled interval (up to 15 minutes in this config), instead of noticing the problem and re-checking its members.
Fix: added forcedHealthCheckNeeded(), which detects when the proxy actually being used to dial is REJECT/REJECT-DROP or is marked dead. When that's the case, DialContext/ListenPacketContext in both fallback.go and urltest.go now proactively kick off an async health check, so the group re-evaluates its members instead of waiting for the next scheduled interval.
Video Demonstration: https://youtu.be/watch?v=LoVrG-ICByg.
Problem 2: failure-triggered health checks can storm the network
GroupBase.onDialFailed already had logic to trigger a forced health check after N consecutive failed dials (maxFailedTimes). However, there was no cooldown between these forced checks. With a group containing ~200 proxies, a burst of failed dials (e.g. during a flaky connection) could re-trigger a full health check of every proxy in the group every few seconds, causing repeated storms of health-check traffic against all providers - visible in logs as "because failed multiple times, activate health check" firing every 6-30 seconds.
You might have noticed that in
url-testgroups, the ping fluctuates SEVERAL times right before your eyes. This is precisely the issue I'm referring to.Fix: added a 30-second cooldown (forcedHealthCheckCooldown) around GroupBasegered path, tracked via lastForcedCheck (atomic.TypedValue[time.Time]).Scheduled health checks (via interval config) are unaffected - only the failure-triggered path is throttled. Also reset the failure counter after a forced check fires, so it doesn't keep re-triggering warnings on every subsequent failed dial within the same window.
Testing
Added forcedcheck_test.go with three tests (written test-first, confirmed fafore the fix):