Skip to content

Add PubSub and BigTable support for Central hosted network controllers - #2526

Merged
glimberg merged 251 commits into
devfrom
gl/ctl-pubsub
Jun 29, 2026
Merged

glimberg merged 251 commits into
devfrom
gl/ctl-pubsub

Conversation

@glimberg

@glimberg glimberg commented Oct 6, 2025 •

Copy link
Copy Markdown
Contributor

This is a big one that goes along with other internal upcoming changes.

  • Refactors existing Redis & Postgres NOTIFY message passing systems into the NotificationListener interface, and adds a 3rd method for message passing via GCP PubSub. Which one to use is configurable by local.conf controller settings
  • Adds GCP BigTable as a 3rd option for writing member status updates along side Postgres and Redis. Also configurable via local.conf
  • Set up CMake for building Central controllers and extended for the regular agent build as well (optional use threre. existing build methods still exist).
  • scripts/bootstrap-deps.sh for bootstrapping external dependencies (scripts/bootstrap-deps.ps1 on Windows)
  • CMake for external dependency management instead of dumping everything in ext/. This is likely more useful for Central controller builds than end user builds for the time being.
  • Controller log formatting standardization

glimberg added 30 commits August 5, 2025 16:05
…stance for the whole library, add init/shutdown functions for it exposed to C
match other things already written
Tests currently need to be run with --test-threads=1.  Seems like the instances of the pubsub emulator stomp on each other without that
…s in new CentralDB class

This allows us to interchangeably use different listeners (pgsql, redis, pubsub) depending on configuration values passed into the constructor.
PubSub allows us to do schema validation, however it only allows one top level message at a time.  Move other sub-message declarations under the main message declaration so that we can enable schema validation in the pubsub stream directly
Muuuuch easier to use external dependencies now

Also tried out conan and vcpkg.   Ran into dependency issues when solving for packages to install with conan.

vcpkg is just obtuse as all hell to install and not easy to integrate
glimberg added 22 commits June 15, 2026 17:41
removing openssl dependency (except on Linux where it's still necessary)
upgrade cpp-httplib
new cmake-presets
lock -> notification, and double notify on erase
More use of scoped RAII guard for connections where appropriate.
Also add presets for the BSDs
Template was looking for {{networkId}} but that hadn't been populated
yet in the case where the IdP returned an error. This caused the
template parsing to crash and return its own error, disguising the error
from the IdP
Only enabled in the central controller builds.  Trying to find more info
about some mystery controller restarts in prod
@joseph-henry

Copy link
Copy Markdown
Contributor

Working notes:

  • It looks like the Smeeclient was removed.
  • The non-CMake build process for ZT_CONTROLLER=1 is now broken. I propose we remove it entirely.

@glimberg

Copy link
Copy Markdown
Contributor Author

yes, smeeclient was removed as it's no longer needed for notifications of new members joining. Thanks to pubsub, that is now handled in CV1/CV2 instead of the controller.

Will remove all the ZT_CONTROLLER=1 stuff from the makefile

Comment thread nonfree/controller/ConnectionPool.hpp
@joseph-henry

Copy link
Copy Markdown
Contributor

Did an end to end test (root, controller, two nodes) of this branch using artifacts produced by the CMake build process and the two nodes were able to ping each other so it looks like things still work.

Test output:

[2026-06-29 18:48:27] === ZeroTier Infrastructure Starting ===
[2026-06-29 18:48:27] === Regenerating root identity, controller identity, and signed planet ===
[2026-06-29 18:48:27] Generated root=74bea25db9 controller=5946b73234
wrote 0000000008eac90a.moon (signed world with timestamp 1782758907554)
[2026-06-29 18:48:27] Regen complete: root=74bea25db9 controller=5946b73234 planet=257B
[2026-06-29 18:48:27] Setting up network namespaces and bridge
[2026-06-29 18:48:27] Setting up netns mount point...
[2026-06-29 18:48:27] Cleaning up existing network resources...
[2026-06-29 18:48:27] Creating network namespaces...
[2026-06-29 18:48:27] Network namespaces created successfully
[2026-06-29 18:48:27] Creating bridge br0...
[2026-06-29 18:48:27] Configuring ns-root...
[2026-06-29 18:48:27] Configuring ns-controller...
[2026-06-29 18:48:27] Configuring ns-node1...
[2026-06-29 18:48:27] Configuring ns-node2...
[2026-06-29 18:48:27] Network namespaces configured:
[2026-06-29 18:48:27]   ns-root:       172.20.0.10:9993
[2026-06-29 18:48:27]   ns-controller: 172.20.0.11:9993
[2026-06-29 18:48:27]   ns-node1:      172.20.0.12:9993
[2026-06-29 18:48:27]   ns-node2:      172.20.0.13:9993
[2026-06-29 18:48:27]   Bridge:        172.20.0.1
[2026-06-29 18:48:27] Starting zerotier-one root on port 9993 in ns-root
[2026-06-29 18:48:27] zerotier-one root started with PID 104
[2026-06-29 18:48:27] Starting zerotier-one controller on port 9993 in ns-controller
[2026-06-29 18:48:27] zerotier-one controller started with PID 107
[2026-06-29 18:48:27] Starting zerotier-one node on port 9993 in ns-node1
[2026-06-29 18:48:27] zerotier-one node1 started with PID 111
[2026-06-29 18:48:27] Starting zerotier-one node on port 9993 in ns-node2
[2026-06-29 18:48:27] zerotier-one node2 started with PID 115
[2026-06-29 18:48:27] Waiting for zerotier-one-root on 172.20.0.10:9993...
[2026-06-29 18:48:28] zerotier-one-root is ready
[2026-06-29 18:48:28] Waiting for controller on 172.20.0.11:9993...
[2026-06-29 18:48:28] controller is ready
[2026-06-29 18:48:28] Waiting for node1 on 172.20.0.12:9993...
[2026-06-29 18:48:28] node1 is ready
[2026-06-29 18:48:28] Waiting for node2 on 172.20.0.13:9993...
[2026-06-29 18:48:28] node2 is ready
[2026-06-29 18:48:28] All services started successfully
[2026-06-29 18:48:28] Waiting for controller to fully initialize...
[2026-06-29 18:48:33] Waiting for authtoken.secret file to be created...
[2026-06-29 18:48:33] Creating network on controller
[2026-06-29 18:48:33] API Request: POST http://127.0.0.1:9993/controller/network (from ns-controller)
[2026-06-29 18:48:33] API Response Code: 200
[2026-06-29 18:48:33] API Response Body: {"authTokens":[null],"authorizationEndpoint":"","capabilities":[],"clientId":"","creationTime":1782758913775,"dns":[],"enableBroadcast":true,"id":"5946b73234c7bbe7","ipAssignmentPools":[],"mtu":2800,"multicastLimit":32,"name":"","nwid":"5946b73234c7bbe7","objtype":"network","private":true,"remoteTraceLevel":0,"remoteTraceTarget":null,"revision":1,"routes":[],"rules":[{"not":false,"or":false,"type":"ACTION_ACCEPT"}],"rulesSource":"","ssoConfig":{"ssoAuthorizationEndpoint":null,"ssoClientId":"","ssoIssuer":"","ssoLinkedId":"","ssoProvider":""},"ssoEnabled":false,"tags":[],"v4AssignMode":{"zt":false},"v6AssignMode":{"6plane":false,"rfc4193":false,"zt":false}}
[2026-06-29 18:48:33] Network created: 5946b73234c7bbe7
[2026-06-29 18:48:33] Configuring network settings
[2026-06-29 18:48:33] API Request: POST http://127.0.0.1:9993/controller/network/5946b73234c7bbe7 (from ns-controller)
[2026-06-29 18:48:33] API Response Code: 200
[2026-06-29 18:48:33] Network configured with IP pool 10.147.17.0/24
[2026-06-29 18:48:33] Getting node identities
[2026-06-29 18:48:33] Node 1 address: 3f033c9539
[2026-06-29 18:48:33] Node 2 address: 150817588f
[2026-06-29 18:48:33] Joining node 3f033c9539 to network 5946b73234c7bbe7
[2026-06-29 18:48:33] API Request: POST http://127.0.0.1:9993/network/5946b73234c7bbe7 (from ns-node1)
[2026-06-29 18:48:33] API Response Code: 200
[2026-06-29 18:48:33] Joining node 150817588f to network 5946b73234c7bbe7
[2026-06-29 18:48:33] API Request: POST http://127.0.0.1:9993/network/5946b73234c7bbe7 (from ns-node2)
[2026-06-29 18:48:33] API Response Code: 200
[2026-06-29 18:48:35] Authorizing nodes on controller
[2026-06-29 18:48:35] API Request: POST http://127.0.0.1:9993/controller/network/5946b73234c7bbe7/member/3f033c9539 (from ns-controller)
[2026-06-29 18:48:35] API Response Code: 200
[2026-06-29 18:48:35] API Request: POST http://127.0.0.1:9993/controller/network/5946b73234c7bbe7/member/150817588f (from ns-controller)
[2026-06-29 18:48:35] API Response Code: 200
[2026-06-29 18:48:35] Nodes authorized on network
[2026-06-29 18:48:37] Verifying network membership
[2026-06-29 18:48:37] Network 5946b73234c7bbe7 has 0 members
[2026-06-29 18:48:37] Dumping network configuration to network.json
[2026-06-29 18:48:37] Network configuration saved to /test-results/network.json
[2026-06-29 18:48:37] === ZeroTier Infrastructure Ready ===
[2026-06-29 18:48:37] 
[2026-06-29 18:48:37] Network Namespaces (bridge: 172.20.0.1/24, all on port 9993):
[2026-06-29 18:48:37]   ns-root:       172.20.0.10:9993 (74bea25db9)
[2026-06-29 18:48:37]   ns-controller: 172.20.0.11:9993 (5946b73234)
[2026-06-29 18:48:37]   ns-node1:      172.20.0.12:9993 (3f033c9539)
[2026-06-29 18:48:37]   ns-node2:      172.20.0.13:9993 (150817588f)
[2026-06-29 18:48:37] 
[2026-06-29 18:48:37] Network:        5946b73234c7bbe7
[2026-06-29 18:48:37] Logs:           /test-results/*.log
[2026-06-29 18:48:37] Network Config: /test-results/network.json
[2026-06-29 18:48:37] 
[2026-06-29 18:48:37] === ZeroTier-to-ZeroTier Connectivity Test ===
[2026-06-29 18:48:37] 
[2026-06-29 18:48:37] Waiting up to 60s for ZeroTier IP assignment...
[2026-06-29 18:48:48] Both nodes have an overlay IP after 10s
[2026-06-29 18:48:48] Node1 ZeroTier IP: 10.147.17.114
[2026-06-29 18:48:48] Node2 ZeroTier IP: 10.147.17.72
[2026-06-29 18:48:48] 
[2026-06-29 18:48:48] Testing ZeroTier connectivity (5 pings each direction)...
[2026-06-29 18:48:48] 
[2026-06-29 18:48:48] Node1 -> Node2 (10.147.17.114 -> 10.147.17.72):
[2026-06-29 18:48:52]   node1 -> node2 (10.147.17.72): OK
[2026-06-29 18:48:52] 
[2026-06-29 18:48:52] Node2 -> Node1 (10.147.17.72 -> 10.147.17.114):
[2026-06-29 18:48:56]   node2 -> node1 (10.147.17.114): OK
[2026-06-29 18:48:56] 
[2026-06-29 18:48:56] ZeroTier connectivity test: SUCCESS
[2026-06-29 18:48:56] 
[2026-06-29 18:48:56] === ZeroTier Overlay Throughput Test (iperf3, 10s) ===
[2026-06-29 18:48:56] 
[2026-06-29 18:48:56] Starting iperf3 server in ns-node2 (10.147.17.72:5201)
[2026-06-29 18:48:57] Running iperf3 node1 -> node2 (10.147.17.114 -> 10.147.17.72) for 10s...
Connecting to host 10.147.17.72, port 5201
[  5] local 10.147.17.114 port 48746 connected to 10.147.17.72 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   504 MBytes  4.22 Gbits/sec  162   1.05 MBytes       
[  5]   1.00-2.00   sec   519 MBytes  4.35 Gbits/sec    0   1.60 MBytes       
[  5]   2.00-3.00   sec   522 MBytes  4.38 Gbits/sec    0   2.01 MBytes       
[  5]   3.00-4.00   sec   522 MBytes  4.38 Gbits/sec    0   2.35 MBytes       
[  5]   4.00-5.00   sec   521 MBytes  4.37 Gbits/sec    1   1.86 MBytes       
[  5]   5.00-6.00   sec   524 MBytes  4.40 Gbits/sec    0   2.22 MBytes       
[  5]   6.00-7.00   sec   522 MBytes  4.38 Gbits/sec    0   2.52 MBytes       
[  5]   7.00-8.00   sec   521 MBytes  4.37 Gbits/sec   16   1.48 MBytes       
[  5]   8.00-9.00   sec   521 MBytes  4.37 Gbits/sec    0   1.91 MBytes       
[  5]   9.00-10.00  sec   521 MBytes  4.37 Gbits/sec  172   1.31 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.08 GBytes  4.36 Gbits/sec  351             sender
[  5]   0.00-10.00  sec  5.07 GBytes  4.36 Gbits/sec                  receiver

iperf Done.
[2026-06-29 18:49:07] iperf3 overlay throughput test: SUCCESS
[2026-06-29 18:49:07] 
[2026-06-29 18:49:07] Writing result to /test-results/result.txt
[2026-06-29 18:49:07] Result: 1 (1=success, 0=failure)
[2026-06-29 18:49:07] 
[2026-06-29 18:49:07] === Node info ===
[2026-06-29 18:49:07] ns-root:
200 info 74bea25db9 1.16.2 ONLINE
[2026-06-29 18:49:07] ns-controller:
200 info 5946b73234 1.16.2 ONLINE
[2026-06-29 18:49:07] ns-node1:
200 info 3f033c9539 1.16.2 ONLINE
[2026-06-29 18:49:07] ns-node2:
200 info 150817588f 1.16.2 ONLINE
[2026-06-29 18:49:07] 
[2026-06-29 18:49:07] Snapshotting daemon home directories to /test-results/daemon-homes
[2026-06-29 18:49:07] Daemon home directories written to /test-results/daemon-homes
[2026-06-29 18:49:07] 
[2026-06-29 18:49:07] Complete

@glimberg
glimberg merged commit 5951981 into dev Jun 29, 2026
12 checks passed
@glimberg
glimberg deleted the gl/ctl-pubsub branch June 29, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants