deps: use innate builds of rmw_zenoh - #642
Conversation
Switches the Zenoh RMW to innate-built packages, matching how we already consume rws (ros-humble-innate-rws). The innate build carries a fix for a lost-wakeup race in rmw_wait that upstream 0.1.9 still ships, which intermittently deadlocks any long-lived multi-topic subscriber -- rws_server in particular -- leaving the process alive but delivering nothing. Also names zenoh-cpp-vendor explicitly in both files. rmw-zenoh-cpp depends on it without a version constraint, so apt is free to upgrade one and not the other; that skew shipped an rmw built against a newer libzenohc than the installed vendor and took a robot down with an undefined-symbol error at dynamic-link time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Greptile SummarySwitches the common ROS 2 dependency manifest from upstream Zenoh RMW packages to Innate-built variants carrying the lost-wakeup fix.
Confidence Score: 5/5No unacknowledged defect was identified, but the PR must remain unmerged until both Innate packages are published as explicitly required by its description. The dependency rename consistently covers the RMW and its vendor package; the known installation failure before package publication is already fully documented and intentionally blocks merging. Important Files Changed
Reviews (1): Last reviewed commit: "deps: use innate builds of rmw_zenoh" | Re-trigger Greptile |
|
Reviewed the diff and verified it against the published apt repo. Re-checked after innate-packages#13 landed — the two blockers in the description are now cleared, one nit in the diff remains. Verified against the repo (as of the 20:42Z
|
ros-humble-innate-rmw-zenoh-cppandros-humble-innate-zenoh-cpp-vendorare not yet published to the innate apt repo. That repo currently carries 26 packages (thenav2family plusros-humble-innate-rws) and no zenoh builds.post_update.shinstalls this file withxargs apt-get install -y, so a single unresolvable name aborts the entire dependency install on every robot. Merging this before publishing bricks updates fleet-wide. Opened as a draft for that reason.What this does
Switches the Zenoh RMW to innate-built packages, matching how we already consume
rws(ros-humble-innate-rws) and nav2 (ros-humble-innate-nav2-*).Why
Upstream
rmw_zenoh_cpp0.1.9 has a lost-wakeup race inrmw_waitthat intermittently deadlocks any long-lived multi-topic subscriber.check_and_attach_condition()clearswait_set_data->triggeredwithout holdingcondition_mutex— a comment there still claims the caller holds it, but upstream #1005/#1015 removed that lock to fix a different deadlock (#998) and left the write behind. It races withadd_new_message()setting the flag under the mutex, so a wakeup gets swallowed andrmw_waitblocks forever with messages already queued.Fix: innate-inc/rmw_zenoh#1.
Symptoms this explains on the robot —
rws_serverstops delivering topics while its websocket stays alive (ping/pong fine, new connections accepted), affecting every client at once, with the process alive at ~0 % CPU. Restarting the webapp appeared to "fix" it because a fresh subscribe re-triggers the wait set.Measured on mars-the-44th, same binary and workload, sole delta being the one-line fix:
Note this bug has been in every humble release since 0.1.2 (2025-06-19) — it long predates the recent 0.1.8 → 0.1.9 upgrade, and is unrelated to the vendor-skew incident. Rolling back is not an option: 0.1.9 is the only humble release that has ever contained the #1015 deadlock fix, so every earlier version has strictly more deadlocks.
Before merging
innate-inc/innate-packagesfrom thehumblebranch ofinnate-inc/rmw_zenoh(after Bringup dev #1 lands there).rmw-zenoh-cppdepends on the vendor package unversioned, and that skew already took a robot down withundefined symbol: z_reply_keyexpr_defaultat dynamic-link time.Notes
zenoh-cpp-vendorline and its comment came from main; this only renames it.sim/apt-dependencies.txtwas removed on main in favour ofros2_ws/apt-dependencies.sim.txt; the sim and hardware overlays carry no zenoh entries, socommon.txtis the only file needing the change.Closes INN-786