Skip to content

fix(graph): resolve startup deadlock on liveliness query with many live tokens - #226

Open
YuanYuYuan wants to merge 1 commit into
mainfrom
dev/liveliness-deadlock
Open

fix(graph): resolve startup deadlock on liveliness query with many live tokens#226
YuanYuYuan wants to merge 1 commit into
mainfrom
dev/liveliness-deadlock

Conversation

@YuanYuYuan

@YuanYuYuan YuanYuYuan commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

A node that joins a system with more than 256 already-live liveliness tokens hangs forever inside ZContextBuilder::build(). This PR gives the startup liveliness query an explicit FifoChannel of capacity 65536, so the replay burst can no longer fill the channel.

Root cause

The replay of live tokens completes before the caller's drain loop starts.

Fact Where
Startup issues a blocking liveliness query Graph::new_with_pattern
The caller drains replies only after get(..).wait() returns drain loop
The query registers its callback, then sends the interest on the calling thread Session::liveliness_query (zenoh 1.9.0)
A callback inside zenoh's DeclareToken handling pushes one Reply per matching token session.rs
The default handler capacity is 256 API_DATA_RECEPTION_CHANNEL_SIZE

No receiver runs during the replay. At token 257 the sender blocks, and startup never completes.

sequenceDiagram
    autonumber
    participant T as Startup thread
    participant Z as Zenoh session
    participant C as Reply channel (cap 256)
    T->>Z: liveliness().get(pattern).wait()
    activate Z
    Z->>Z: send_interest — replay live tokens
    loop each matching live token
        Z->>C: push Reply
    end
    Note over C: token 257 — channel full
    Z--xC: push blocks
    Note over T,C: the drain loop starts only after get() returns, so no receiver exists yet
    deactivate Z
    T-->>T: never reaches replies.recv()
Loading
  • Above the capacity the hang is deterministic, not a race.
  • In the field it looks intermittent, because the live-token count varies between runs around the 256 threshold.
  • That matches the reported hang rate of roughly 30% of startups on the affected system.

The defect sits in the zenoh reply-delivery path, not in how hiroz uses it. eclipse-zenoh/zenoh#2678 proposes the root-cause fix upstream. This PR removes hiroz's exposure without waiting for that change.

Fix

Graph::new_with_pattern passes an explicit handler to the liveliness query:

let replies = session
    .liveliness()
    .get(&c_liveliness_pattern)
    .with(zenoh::handlers::FifoChannel::new(65536))
    .timeout(std::time::Duration::from_secs(3))
    .wait()?;

65536 is a ceiling above any live-token count a realistic ROS graph reaches. A comment at the call site states the ordering, so a reader does not take the constant as arbitrary.

What fails without this

No failing test in CI. The reproduction needs more than 256 matching live tokens in a running system, which the test suite does not build.

  • Symptom without the fix: ZContextBuilder::build() blocks forever on a node that joins a large graph.
  • Evidence for the threshold: the constant and the delivery path linked in Root cause, both read at zenoh 1.9.0.
  • 25 checks green on 084f2d81e2ab9dbe27370fad1e16bab4163fa815, 0 failed, 0 pending.

Note

This change makes the deadlock unreachable at any practical token count. It does not make it impossible. A graph with more than 65536 matching live tokens hits the same ordering. eclipse-zenoh/zenoh#2678 is what removes the class.

Breaking Changes

None.

zenoh's liveliness_query() synchronously replays every currently-live
matching token onto the query's reply channel, on the calling thread,
before the call returns. The default handler is a bounded 256-slot
flume channel, so once more than 256 tokens match the pattern, the
synchronous replay blocks forever waiting for a receiver that cannot
run until the call itself returns, deadlocking the caller.
@YuanYuYuan
YuanYuYuan force-pushed the dev/liveliness-deadlock branch from 084f2d8 to 1008e4d Compare August 14, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant