Repository navigation

Add PubSub and BigTable support for Central hosted network controllers - #2526

Merged

glimberg merged 251 commits into

devfrom

Jun 29, 2026

This pull request is big! We're only showing the most recent 250 commits

Commits on Aug 5, 2025

fix dependencies when temporal isn't needed
glimberg
committed
f9500ca
gcloud-pubsub only if ztcontroller is flagged
glimberg
committed
44d0e81
fix macos build
glimberg
committed
a552163
fix windows bui ld
glimberg
committed
4b3b847

Commits on Aug 7, 2025

tokio is needed by both temporal & gcloud pubsub, so make just one instance for the whole library, add init/shutdown functions for it exposed to C
glimberg
committed
837f15d

Commits on Aug 11, 2025

cleanup warnings
glimberg
committed
f079f8b

Commits on Aug 12, 2025

WIP: pubsub wrapper in Rust
glimberg
committed
06b2ce9

Commits on Aug 13, 2025

plumb through callbacks
glimberg
committed
0e94891
function naming cleanup.
glimberg
committed
bf8c9d0

Commits on Aug 14, 2025

updates & tests.
glimberg
committed
ccb0fa7
expose change_handler to C via FFI
glimberg
committed
3a209e2
remove ztcontroller from default feature list
glimberg
committed
a842ad8
feature cleanup
glimberg
committed
d4ee95e
fix build script
glimberg
committed
e4147f7
fix calling into async functions from non-async via the FFI
glimberg
committed
4fab227
revert default feature list once again
glimberg
committed
4237830

Commits on Aug 15, 2025

more calling async from non-async changes
glimberg
committed
b90ad51
Add C++ wrapper around pubsub listeners
glimberg
committed
650fc0c

Commits on Aug 20, 2025

rename classes
glimberg
committed
ebe8fdb

Commits on Aug 21, 2025

Commits on Aug 25, 2025

rework protobuf messages
glimberg
committed
f8a4a5d

Commits on Aug 27, 2025

Update central controller build to use CMake + conda
glimberg
committed
7f3b150
spacing
glimberg
committed
9bdd564

Commits on Aug 29, 2025

Commits on Sep 1, 2025

wire up status writers
glimberg
committed
d119547
add sleep at end of online notificaiton loop
glimberg
committed
e347f89
Merge branch 'dev' into gl/ctl-pubsub
glimberg
committed
17b48d7
PubSub to gcloud-cpp library for simplicity's sake
glimberg
committed
b114896
rework the pubsub listener again so the subscription is set up on its own thread
glimberg
committed
db6e698
remove rust-based pubsub
glimberg
committed
0f63783
add def for new style controllers
glimberg
committed
496e69e
Merge branch 'adam/1.16' into gl/ctl-pubsub
glimberg
committed
195d5b4

Commits on Sep 2, 2025

Commits on Sep 3, 2025

setup github action for building
glimberg
committed
7c1bfc9
runs-on matrix
glimberg
committed
0f13363
skip caching until this all works
glimberg
committed
09f1ee7
trying to get this working
glimberg
committed
97681b5
test
glimberg
committed
e1db40d
no linux/arm64 build available for google-cloud-cpp 2.39 :/. Downgrade to 2.38
glimberg
committed
cab33df
dependency juggling
glimberg
committed
22484c7
fix cmake options
glimberg
committed
3ad36c7
more conda
glimberg
committed
f43bc3d
more conda
glimberg
committed
21e20f2
skip cmake action
glimberg
committed
648f091
cmake --build args
glimberg
committed
3470ceb
fix arm64 build flag
glimberg
committed
d3f1ec7
fix deps for selftest
glimberg
committed
0f3a514
set ZT_AES_NEON for arm64
glimberg
committed
d31b3a0
cache conda environment
glimberg
committed
1dae4c4
cache conda environment
glimberg
committed
4395574
fix run arg
glimberg
committed
b09956a
bump
glimberg
committed
3ff4d67
put git branch ref into cache key name
glimberg
committed
a841e96
only update conda env if cache was not hit
glimberg
committed
3f3e684
let's build a docker image
glimberg
committed
000f3d5
fix short hash
glimberg
committed
7fd112e
fix multi-arch manifest creation
glimberg
committed
594d227
update ld.so.conf for paths to libraries
glimberg
committed
4abf902
new modified startup script for central controller docker images with support for configuring bigtable, pubsub, redis
glimberg
committed
9d73c34
no trailing comma on bigtable block
glimberg
committed
3ad352c
text alignment
glimberg
committed
6ceb8ea
add jemalloc
glimberg
committed
4f07599

Commits on Sep 4, 2025

change tag to match existing
glimberg
committed
2416c42
need a conda init here first apparently
glimberg
committed
4516e6a
stray backtick
glimberg
committed
96aa893
also have to add postgres to the container for pg_isready
glimberg
committed
158d245
no absolute path for pg_isready
glimberg
committed
1da9e24
ok back to full path
glimberg
committed
092fc26
run process in conda env directly from docker
glimberg
committed
d216a37
debug output of main script
glimberg
committed
c9c8e95
.
glimberg
committed
b94fa2d
fix case for redis: null
glimberg
committed
7e4e7ef
and instead of or in postgres check
glimberg
committed
ce84de6
debug printf
glimberg
committed
6866535
get a running controller in docker
glimberg
committed
f33e706
debugging runtime link issues on arm64
glimberg
committed
73e3287
try to make this work on both x64 & arm64
glimberg
committed
2264452
will this make arm64 containers work?
glimberg
committed
7d5a60b
add conda builder dockerfile
glimberg
committed
40f6508
trying to build in a docker container
glimberg
committed
70d24fc
build in containers
glimberg
committed
8bd96d3
move up docker auth
glimberg
committed
67526d4
move up the gcp auth step
glimberg
committed
3616c40
cleanup controller github action
glimberg
committed
dcd583f
enable otel exporter
glimberg
committed
f208007
fix otel exporting
glimberg
committed
b16d937
query fix
glimberg
committed
20746b2
another query fix
glimberg
committed
3b7958a
another query fix
glimberg
committed
e9565cf
another query fix
glimberg
committed
a18e370

Commits on Sep 5, 2025

fix
glimberg
committed
a061c86
fix
glimberg
committed
48ff61b
another fix
glimberg
committed
282d7c7
add a ChangeSource enum to the change protobuf messages
glimberg
committed
ba6200f
Add `frontend` column to networks & network_memberships tables so the controller knows which system owns the networks & members
glimberg
committed
74f0525
wire up pubsub outgoing status changes from controller -> CV2
glimberg
committed
024824c
whoops. mesed up the migrate naming convention
glimberg
committed
7ad264d

Commits on Sep 6, 2025

Commits on Sep 8, 2025

fix stack corruption in DB::get
glimberg
committed
71d68e9
fix typo in cmake file
glimberg
committed
ebf66b7
more fixes
glimberg
committed
006ced2
alloc meatadata with new so pubsub can delete it
glimberg
committed
190d836
delete pubsub subscription on shutdown
glimberg
committed
f51bfdb
append 8 random hex chars to the subscription id
glimberg
committed
88f08fa
forgot a ;
glimberg
committed
934aa93
pass version info to status writer
glimberg
committed
1ba4ccd
back to bulk apply for status updates in BigTable
glimberg
committed
1d65c1e
only update os, architecture and client version if they've changed (or don't exist)
glimberg
committed
facca36
removing some log output
glimberg
committed
ee87eb3
wire up pubsub notifications from controller to frontend
glimberg
committed
012443a

Commits on Sep 10, 2025

clean up deprecation warnings in pqxx
glimberg
committed
7aa451b
fix version string
glimberg
committed
493cc4f

Commits on Sep 11, 2025

add `use_for_new_networks` column in controller table
glimberg
committed
1ce37e7

Commits on Sep 13, 2025

add tables for the controller load log and sso nonces & expiry
glimberg
committed
ad1fc5f
fix migration
glimberg
committed
cb2de5a

Commits on Sep 18, 2025

make pubsub topics configurable
glimberg
committed
6113bad

Commits on Sep 22, 2025

Commits on Sep 25, 2025

set objtype on messages received via PubSub
glimberg
committed
069d7c1
fix subscription creation
glimberg
committed
e51e516
query fix for new networks
glimberg
committed
c6b7c26
json field fix
glimberg
committed
315bf24

Commits on Sep 30, 2025

more logging for errors on pubsub listener
glimberg
committed
97d20a1
stray ` got in there
glimberg
committed
02908e5

Commits on Oct 1, 2025

fix attempted null -> string conversion
glimberg
committed
20a67ab
debug output from parsing pubsub messages
glimberg
committed
b4f018b
debug output
glimberg
committed
f028213
more logging
glimberg
committed
dad1a1e
more logging
glimberg
committed
aecd7fd
more debug output
glimberg
committed
1f557b5
make the default an empty array here
glimberg
committed
7faf705
fix tags/capabilities?
glimberg
committed
80f8213
think this is where things are crashing, but I'm not sure why yet. Comment it out to make sure
glimberg
committed
a8b5c89
throwing stuff at the wall cuz I can't actually debug this right now
glimberg
committed
fb75369
pubsub writer should work now
glimberg
committed
3bbf194
one last json parsing fix
glimberg
committed
0d21250

Commits on Oct 2, 2025

Commits on Oct 6, 2025

Commits on Oct 9, 2025

dont NACK failed messages. That gets us into a loop of failing the same messages over and over again
glimberg
committed
946d96f
robustify the handling of tags/capabilities/rules which are strings of encoded json
glimberg
committed
46a8cd7
more robustifying incoming messages
glimberg
committed
4736767
compiler fix
glimberg
committed
e3f65ff
more variations on checking json
glimberg
committed
19ccc98
these should be empty json arrays, not just "[]".
glimberg
committed
bf0fe2c
missed one
glimberg
committed
77aa8c7
ensure change source is controller if otherwise unset
glimberg
committed
76ba890

Commits on Oct 27, 2025

Added some more logging to the ctl around deauths
glimberg
committed
c21ff23
Set ordering key on pubsub message publishing.
glimberg
committed
30c4484

Commits on Oct 30, 2025

Enable cross-service propagation of otel trace metadata
glimberg
committed
5c27068

Commits on Nov 6, 2025

Merge branch 'dev' into gl/ctl-pubsub
glimberg
authored
00dca6a

Commits on Nov 12, 2025

controller build readme update
glimberg
committed
50daf95

Commits on Nov 13, 2025

set --provenance false on docker build to try and fix docker image creation issue
glimberg
committed
2ba50f4

Commits on Jan 23, 2026

Merge branch 'dev' into gl/ctl-pusub
glimberg
committed
68a9634
Remove extra verbose logging from controller
glimberg
committed
ccb9a45
disable peer metrics in Central controller
glimberg
committed
35f7bf2

Commits on Feb 24, 2026

WIP: Update sso info retrieval method
glimberg
committed
c653e76

Commits on Feb 25, 2026

add a linked_id column to the oidc_config table.
glimberg
committed
0f0e6b3
fix db migrations
glimberg
committed
ae7ee51
drop index
glimberg
committed
0ad6b19
woops. out of order here
glimberg
committed
38f4d12

Commits on Mar 12, 2026

set network member frontend based on the network its a member of
glimberg
committed
03aa33b

Commits on Mar 17, 2026

add SSO nonce/auth PubSub messaging with frontend routing
glimberg
committed
a840b94
add assigned_central_version column to controllers_ctl
glimberg
committed
2c57f85

Commits on Mar 18, 2026

Undo change to old migration that shouldn't have been made
glimberg
committed
d9507dd
another fix
glimberg
committed
7faf30d

Commits on Mar 19, 2026

plumb through config changes for sso pubsub
glimberg
committed
e49b347
configure assigned central version in startup script
glimberg
committed
7ec4246

Commits on Mar 31, 2026

print a message when the SSO PSK is configured
glimberg
committed
b047038
temporary logging
glimberg
committed
895b060
update settings to enable SSO networks
glimberg
committed
78b25f4
sso query fix in controller
glimberg
committed
dd6e69f

Commits on Apr 1, 2026

Skip redundant nonce sending with an expiry time of 0
glimberg
committed
20f7311

Commits on Apr 2, 2026

Remove smee from CentralDB.
glimberg
committed
ea5c91b

Commits on Apr 3, 2026

Commits on Apr 11, 2026

fix poison pill blocking proccessing
glimberg
committed
4ca5c9b
Fix silent PubSub message loss in controller subscription loop
glimberg
committed
910334c
logging
glimberg
committed
d2361a9
make sure subscribe pulls stay running
glimberg
committed
0e8ec66

Commits on Apr 30, 2026

change sso topic names for clarity
glimberg
committed
23d395d

Commits on May 4, 2026

addressing pr comments
glimberg
committed
8c3a777

Commits on May 8, 2026

fix a typo in some json handling that could be affecting an issue I'm tracking down
glimberg
committed
f0522a7

Commits on May 13, 2026

Bugfix pass
glimberg
committed
b79592c
Fix json key inconsistency in pubsub writer
glimberg
committed
7b44e0a
align JSON keys between CentralDB and PubSub listener/writer
glimberg
committed
1c75584

Commits on May 14, 2026

update network.proto in line with CV1 and CV2
glimberg
committed
ac20c32
fix for missing field in json
glimberg
committed
9a9a8c0

Commits on May 19, 2026

ensure controller doesn't change assigned frontend
glimberg
committed
80f5e81

Commits on May 27, 2026

handle nulls for remote trace target
glimberg
committed
b142715

Commits on May 28, 2026

remove noisy log
glimberg
committed
346dc90
Merge branch 'dev' into gl/ctl-pubsub
glimberg
committed
29f5464
cargo fix
glimberg
committed
3dffacc
This workflow does need to be removed from this branch.
glimberg
committed
f76e6b9

Commits on Jun 2, 2026

More sso error logging & removing a few logs as well
glimberg
committed
174130c
Add handling sso networks for both CV1 & CV2 simultaneously
glimberg
committed
eec53a5

Commits on Jun 10, 2026

perf(controller): drop per-row BigTable reads in status writer
glimberg
committed
af7c882
better closed connection handling to postgres
glimberg
committed
a001eba
Better data handling if a db write fails
glimberg
committed
a3a45a7
better logging of commit thread ticks
glimberg
committed
fede9b9
remove db check from onlineNotificationThread
glimberg
committed
85eec45
fix pubsub flood on ctl startup
glimberg
committed
407bfdf

Commits on Jun 12, 2026

Repo cleanup.
glimberg
committed
a4904c9
main controller cmake file cleanup
glimberg
committed
378dd27

Commits on Jun 13, 2026

bring CMake in line with make-linux/mac/bsd
glimberg
committed
052ea55
Fix PIC/PIE flags
glimberg
committed
e09e7b5

Commits on Jun 15, 2026

More linux/mac cmake cleanup
glimberg
committed
0e637a1
build fix
glimberg
committed
f1c6a64
Set thinkgs up for Windows in CMake
glimberg
committed
8ae153b
update to use makefiles by default on *ix
glimberg
committed
14faaaa
fix linking via CMake on Windows
glimberg
committed
4988d53
fix incompatible options on crypto files in windows
glimberg
committed
404fa9b

Commits on Jun 16, 2026

Commits on Jun 17, 2026

Addressing comments
glimberg
committed
fd98e0b
bugfixes
glimberg
committed
1112281
Pubsub devlivery reliability updates
glimberg
committed
8271fca
Better connection handling for postgres status writer
glimberg
committed
6f57a77
status update batch delivery reliablilty
glimberg
committed
4a517e2
Concurrency hygene updates
glimberg
committed
7be64c6
Correcting a few things
glimberg
committed
7192b65
fix printf, connection pool smart pointer semantics, wait for ready spin
glimberg
committed
afd8a82
Conneciton pool update to a timed wait on connection pool exhaustion
glimberg
committed
bf42649
enable metrics by default for the central controller
glimberg
committed
4706ed9
use the checkin timestamps for the cell timestamps in BigTable
glimberg
committed
ab9efc7
Update build.sh with build instructions
glimberg
committed
5640b82
Ensure everything in nonfree/controller has the proper header comment
glimberg
committed
75b51ec
standardize log formatting across the controller
glimberg
committed
fcb016b
update documentation for ConnectionPool
glimberg
committed
8304ec7
bump actions to get rid of node warnings
glimberg
committed
64cc25b

Commits on Jun 22, 2026

Fix the sso template for errors returned by IdP
glimberg
committed
4bcff9d

Commits on Jun 26, 2026

Commits on Jun 29, 2026

Remove ZT_CONTROLLER=1 from makefile. No longer used to build central
glimberg
committed
1f6a1fd