Retry the xapi connection instead of latching it broken forever#90
Open
olivierlambert wants to merge 1 commit into
Open
Retry the xapi connection instead of latching it broken forever#90olivierlambert wants to merge 1 commit into
olivierlambert wants to merge 1 commit into
Conversation
Auth sets masterConnectionBroken when a XenAPI call times out, and OpenSession never attempts another connection once the flag is set. A long-lived xsconsole (e.g. the one agetty spawns on the physical console) that experiences a single 15-second xapi hang therefore never talks to xapi again: the status screen shows 'Pool master is unreachable.' forever, while newly launched xsconsole instances on the same host work fine. This is easy to hit during an HA pool master failover, where xapi on the surviving host can be unresponsive for a while, and was reported after exactly that scenario. Keep the flag as a cooldown rather than a latch: after a timeout, skip connection attempts for CONNECTION_RETRY_SECONDS (so an unresponsive xapi doesn't block the UI for the socket timeout on every update cycle), then try again, and clear the flag on the next successful login. Also log the transitions, since previously the process went quiet with no trace of why. Reproduced and verified on XCP-ng 8.3: freeze xapi (SIGSTOP) for 60s while pressing F5 in xsconsole; before this change the console never recovers, with it the connection recovers within a minute of xapi returning.
Contributor
|
There are several |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #89
AuthsetsmasterConnectionBrokenwhen a XenAPI call times out, andOpenSession()never attempts another connection once the flag is set — the flag is never reset anywhere. A long-lived xsconsole (e.g. the one on the physical console) that experiences a single 15-second xapi hang therefore never talks to xapi again and showsPool master is unreachable.forever, while newly launched instances work fine. Easy to hit during an HA pool-master failover, which is how it was reported: https://xcp-ng.org/forum/post/106689This keeps the flag but as a cooldown rather than a latch:
socket.timeout, connection attempts are skipped forCONNECTION_RETRY_SECONDS(60s), so an unresponsive xapi doesn't block the UI for the socket timeout on every update cycle — the behaviour the latch presumably existed to prevent;OpenSession()call tries again;XenAPI connection timed out - retrying in 60 seconds at most/XenAPI connection recovered), where previously the process went silent with no trace.The timeout handling is factored into
Auth.ConnectionTimedOut(), also used by the test-modeTCPAuthenticate()path so the retry timestamp is always set alongside the flag.Validation
On XCP-ng 8.3:
kill -STOP $(pgrep -x xapi), press F5 twice on the console xsconsole,kill -CONTafter 60s.Before (current master): the console is stuck on
Pool master is unreachable.forever; only restarting xsconsole recovers it.After (this branch, from the host's syslog):
…and the console repopulates with live data on the next refresh, no restart needed.