Skip to content

deps: use innate builds of rmw_zenoh - #642

Merged
karmanyaahm merged 1 commit into
mainfrom
fix/innate-zenoh-apt-packages
Aug 11, 2026
Merged

deps: use innate builds of rmw_zenoh#642
karmanyaahm merged 1 commit into
mainfrom
fix/innate-zenoh-apt-packages

Conversation

@karmanyaahm

@karmanyaahm karmanyaahm commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

⚠️ Blocked — do not merge until the packages exist

ros-humble-innate-rmw-zenoh-cpp and ros-humble-innate-zenoh-cpp-vendor are not yet published to the innate apt repo. That repo currently carries 26 packages (the nav2 family plus ros-humble-innate-rws) and no zenoh builds.

post_update.sh installs this file with xargs apt-get install -y, so a single unresolvable name aborts the entire dependency install on every robot. Merging this before publishing bricks updates fleet-wide. Opened as a draft for that reason.

What this does

Switches the Zenoh RMW to innate-built packages, matching how we already consume rws (ros-humble-innate-rws) and nav2 (ros-humble-innate-nav2-*).

Why

Upstream rmw_zenoh_cpp 0.1.9 has a lost-wakeup race in rmw_wait that intermittently deadlocks any long-lived multi-topic subscriber. check_and_attach_condition() clears wait_set_data->triggered without holding condition_mutex — a comment there still claims the caller holds it, but upstream #1005/#1015 removed that lock to fix a different deadlock (#998) and left the write behind. It races with add_new_message() setting the flag under the mutex, so a wakeup gets swallowed and rmw_wait blocks forever with messages already queued.

Fix: innate-inc/rmw_zenoh#1.

Symptoms this explains on the robot — rws_server stops delivering topics while its websocket stays alive (ping/pong fine, new connections accepted), affecting every client at once, with the process alive at ~0 % CPU. Restarting the webapp appeared to "fix" it because a fresh subscribe re-triggers the wait set.

Measured on mars-the-44th, same binary and workload, sole delta being the one-line fix:

time to wedge messages delivered
upstream 0.1.9 13 s, 22 s, 85 s, 108 s (4 for 4) 512 / 1 951 / 10 566 / 12 363
0.1.9 + fix no wedge > 100 000, 17+ min continuous at 99 msg/s

Note this bug has been in every humble release since 0.1.2 (2025-06-19) — it long predates the recent 0.1.8 → 0.1.9 upgrade, and is unrelated to the vendor-skew incident. Rolling back is not an option: 0.1.9 is the only humble release that has ever contained the #1015 deadlock fix, so every earlier version has strictly more deadlocks.

Before merging

  1. Build and publish both packages to innate-inc/innate-packages from the humble branch of innate-inc/rmw_zenoh (after Bringup dev #1 lands there).
  2. Publish both togetherrmw-zenoh-cpp depends on the vendor package unversioned, and that skew already took a robot down with undefined symbol: z_reply_keyexpr_default at dynamic-link time.
  3. Verify a robot resolves to the innate build, then merge this.

Notes

  • The zenoh-cpp-vendor line and its comment came from main; this only renames it.
  • sim/apt-dependencies.txt was removed on main in favour of ros2_ws/apt-dependencies.sim.txt; the sim and hardware overlays carry no zenoh entries, so common.txt is the only file needing the change.

Closes INN-786

Switches the Zenoh RMW to innate-built packages, matching how we already
consume rws (ros-humble-innate-rws). The innate build carries a fix for a
lost-wakeup race in rmw_wait that upstream 0.1.9 still ships, which
intermittently deadlocks any long-lived multi-topic subscriber -- rws_server
in particular -- leaving the process alive but delivering nothing.

Also names zenoh-cpp-vendor explicitly in both files. rmw-zenoh-cpp depends on
it without a version constraint, so apt is free to upgrade one and not the
other; that skew shipped an rmw built against a newer libzenohc than the
installed vendor and took a robot down with an undefined-symbol error at
dynamic-link time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@karmanyaahm
karmanyaahm marked this pull request as ready for review August 11, 2026 06:15

@theo-michel theo-michel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Switches the common ROS 2 dependency manifest from upstream Zenoh RMW packages to Innate-built variants carrying the lost-wakeup fix.

  • Renames both rmw-zenoh-cpp and its unversioned vendor dependency together.
  • Documents why the Innate RMW build is required.
  • The PR is intentionally blocked until both renamed packages are published.

Confidence Score: 5/5

No unacknowledged defect was identified, but the PR must remain unmerged until both Innate packages are published as explicitly required by its description.

The dependency rename consistently covers the RMW and its vendor package; the known installation failure before package publication is already fully documented and intentionally blocks merging.

Important Files Changed

Filename Overview
ros2_ws/apt-dependencies.common.txt Replaces both upstream Zenoh packages with coordinated Innate package names; the acknowledged package-publication prerequisite remains the only identified merge blocker.

Reviews (1): Last reviewed commit: "deps: use innate builds of rmw_zenoh" | Re-trigger Greptile

@theo-michel

Copy link
Copy Markdown
Contributor

Reviewed the diff and verified it against the published apt repo. Re-checked after innate-packages#13 landed — the two blockers in the description are now cleared, one nit in the diff remains.

Verified against the repo (as of the 20:42Z Release)

Both packages exist on amd64 and arm64, with the right metadata:

  • ros-humble-innate-rmw-zenoh-cpp 0.1.10-1jammyConflicts/Provides: ros-humble-rmw-zenoh-cpp
  • ros-humble-innate-zenoh-cpp-vendor 0.1.9-1jammyConflicts/Provides: ros-humble-zenoh-cpp-vendor

The Conflicts/Provides pair means apt cleanly evicts the stock debs on install, same as the nav2 rename, so no removal step is needed in post_update.sh.

The fix is genuinely in the shipped binary: the submodule pins b788982c, which is the #1 merge (b9094556) plus the 0.1.10 bump, and check_and_attach_condition() there no longer writes wait_set_data->triggered = false.

Worth recording that this was briefly not the case — the first publish (0.1.9-1jammy, innate-packages#12) pinned the submodule at 03076986, two commits behind the fix; the packaging commit landed at 06:07:23Z and #1 merged at 06:09:00Z, and CI checks out with git submodule update --init --recursive (no --remote), so it built a fix-less tree. innate-packages#13 fixed that.

Bumping rmw_zenoh_cpp/package.xml to 0.1.10 rather than bumping DEB_INC is the right call — it makes the rebuild an apt-visible upgrade, so any robot that already pulled the fix-less 0.1.9-1jammy actually moves. A rebuild republished at an unchanged version would have been a silent no-op for exactly the fleet that needs it most.

One thing to fix in the diff

ros2_ws/apt-dependencies.common.txt:56 — the retained comment is now false for the pair we ship:

# unversioned Depends of the rmw — unnamed, apt leaves libzenohc.so behind and every node dies at load

The innate rmw declares Depends: ros-humble-innate-zenoh-cpp-vendor (= 0.1.9-1jammy) — an exact pin, not an unversioned dep. The skew the comment warns about can't happen for these two; the actual invariant is the opposite, and stronger: they can never resolve apart, and a vendor rebuild without a matching rmw rebuild makes the pair uninstallable rather than silently skewed. Since the previous incident came from misreading this exact relationship, worth getting the comment right.

(The vendor correctly stayed at 0.1.9-1jammy — the 0.1.10 commit touches only rmw_zenoh_cpp's CHANGELOG and package.xml, so the vendor slot restored from cache and the rmw was linked against that build.)

Two notes, no action needed in the diff

  • The description still says the packages are not yet published and the PR is blocked. Both are stale now — worth updating so nobody blocks on a condition that's already met.
  • 0.1.10 is fork-local; upstream's humble branch tops out at 0.1.9. No apt-level problem (our package name differs), but if upstream later ships its own humble 0.1.10, the two carry the same version with different content and dpkg -l alone won't tell them apart.

🤖 Reviewed with Claude Code

@karmanyaahm
karmanyaahm merged commit 8753998 into main Aug 11, 2026
10 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants