Skip to content

Retry the xapi connection instead of latching it broken forever#90

Open
olivierlambert wants to merge 1 commit into
xapi-project:masterfrom
olivierlambert:fix-xapi-connection-latch
Open

Retry the xapi connection instead of latching it broken forever#90
olivierlambert wants to merge 1 commit into
xapi-project:masterfrom
olivierlambert:fix-xapi-connection-latch

Conversation

@olivierlambert

Copy link
Copy Markdown

Fixes #89

Auth sets masterConnectionBroken when a XenAPI call times out, and OpenSession() never attempts another connection once the flag is set — the flag is never reset anywhere. A long-lived xsconsole (e.g. the one on the physical console) that experiences a single 15-second xapi hang therefore never talks to xapi again and shows Pool master is unreachable. forever, while newly launched instances work fine. Easy to hit during an HA pool-master failover, which is how it was reported: https://xcp-ng.org/forum/post/106689

This keeps the flag but as a cooldown rather than a latch:

  • after a socket.timeout, connection attempts are skipped for CONNECTION_RETRY_SECONDS (60s), so an unresponsive xapi doesn't block the UI for the socket timeout on every update cycle — the behaviour the latch presumably existed to prevent;
  • once the cooldown expires the next OpenSession() call tries again;
  • a successful login clears the flag;
  • the transitions are logged (XenAPI connection timed out - retrying in 60 seconds at most / XenAPI connection recovered), where previously the process went silent with no trace.

The timeout handling is factored into Auth.ConnectionTimedOut(), also used by the test-mode TCPAuthenticate() path so the retry timestamp is always set alongside the flag.

Validation

On XCP-ng 8.3: kill -STOP $(pgrep -x xapi), press F5 twice on the console xsconsole, kill -CONT after 60s.

Before (current master): the console is stuck on Pool master is unreachable. forever; only restarting xsconsole recovers it.

After (this branch, from the host's syslog):

17:44:58 xsconsole: XenAPI connection timed out - retrying in 60 seconds at most
17:45:24 bugtest: xapi unfrozen
17:45:59 xsconsole: XenAPI connection recovered

…and the console repopulates with live data on the next refresh, no restart needed.

Auth sets masterConnectionBroken when a XenAPI call times out, and
OpenSession never attempts another connection once the flag is set.  A
long-lived xsconsole (e.g. the one agetty spawns on the physical
console) that experiences a single 15-second xapi hang therefore never
talks to xapi again: the status screen shows 'Pool master is
unreachable.' forever, while newly launched xsconsole instances on the
same host work fine.  This is easy to hit during an HA pool master
failover, where xapi on the surviving host can be unresponsive for a
while, and was reported after exactly that scenario.

Keep the flag as a cooldown rather than a latch: after a timeout, skip
connection attempts for CONNECTION_RETRY_SECONDS (so an unresponsive
xapi doesn't block the UI for the socket timeout on every update
cycle), then try again, and clear the flag on the next successful
login.  Also log the transitions, since previously the process went
quiet with no trace of why.

Reproduced and verified on XCP-ng 8.3: freeze xapi (SIGSTOP) for 60s
while pressing F5 in xsconsole; before this change the console never
recovers, with it the connection recovers within a minute of xapi
returning.
@last-genius

Copy link
Copy Markdown
Contributor

There are several pylint CI failures:

XSConsoleAuth.py:207:20: R0204: Redefinition of self.error type from Exception to XenAPI.XenAPI.Failure (redefined-variable-type)
XSConsoleAuth.py:207:20: R0204: Redefinition of self.error type from Exception to XenAPI.XenAPI.Failure (redefined-variable-type)
XSConsoleAuth.py:196:0: I0021: Useless suppression of 'redefined-variable-type' (useless-suppression)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

xsconsole never reconnects to xapi after a single connection timeout (console stuck on "Pool master is unreachable.")

2 participants