fix(graph): resolve startup deadlock on liveliness query with many live tokens - #226
Open
YuanYuYuan wants to merge 1 commit into
Open
fix(graph): resolve startup deadlock on liveliness query with many live tokens#226YuanYuYuan wants to merge 1 commit into
YuanYuYuan wants to merge 1 commit into
Conversation
YuanYuYuan
force-pushed
the
dev/liveliness-deadlock
branch
from
July 15, 2026 10:14
9a379de to
084f2d8
Compare
6 tasks
zenoh's liveliness_query() synchronously replays every currently-live matching token onto the query's reply channel, on the calling thread, before the call returns. The default handler is a bounded 256-slot flume channel, so once more than 256 tokens match the pattern, the synchronous replay blocks forever waiting for a receiver that cannot run until the call itself returns, deadlocking the caller.
YuanYuYuan
force-pushed
the
dev/liveliness-deadlock
branch
from
August 14, 2026 18:23
084f2d8 to
1008e4d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A node that joins a system with more than 256 already-live liveliness tokens hangs forever inside
ZContextBuilder::build(). This PR gives the startup liveliness query an explicitFifoChannelof capacity 65536, so the replay burst can no longer fill the channel.Root cause
The replay of live tokens completes before the caller's drain loop starts.
Graph::new_with_patternget(..).wait()returnsSession::liveliness_query(zenoh 1.9.0)DeclareTokenhandling pushes oneReplyper matching tokensession.rsAPI_DATA_RECEPTION_CHANNEL_SIZENo receiver runs during the replay. At token 257 the sender blocks, and startup never completes.
sequenceDiagram autonumber participant T as Startup thread participant Z as Zenoh session participant C as Reply channel (cap 256) T->>Z: liveliness().get(pattern).wait() activate Z Z->>Z: send_interest — replay live tokens loop each matching live token Z->>C: push Reply end Note over C: token 257 — channel full Z--xC: push blocks Note over T,C: the drain loop starts only after get() returns, so no receiver exists yet deactivate Z T-->>T: never reaches replies.recv()The defect sits in the zenoh reply-delivery path, not in how hiroz uses it. eclipse-zenoh/zenoh#2678 proposes the root-cause fix upstream. This PR removes hiroz's exposure without waiting for that change.
Fix
Graph::new_with_patternpasses an explicit handler to the liveliness query:65536 is a ceiling above any live-token count a realistic ROS graph reaches. A comment at the call site states the ordering, so a reader does not take the constant as arbitrary.
What fails without this
No failing test in CI. The reproduction needs more than 256 matching live tokens in a running system, which the test suite does not build.
ZContextBuilder::build()blocks forever on a node that joins a large graph.084f2d81e2ab9dbe27370fad1e16bab4163fa815, 0 failed, 0 pending.Note
This change makes the deadlock unreachable at any practical token count. It does not make it impossible. A graph with more than 65536 matching live tokens hits the same ordering. eclipse-zenoh/zenoh#2678 is what removes the class.
Breaking Changes
None.