Experimental: changelog table for crdb by ecordell · Pull Request #3211 · authzed/spicedb

ecordell · 2026-07-02T22:21:02Z

Note: for reference, hasn't been validated

Description

CockroachDB changefeeds stop emitting resolved timestamps during bulk loads, which stalls SpiceDB's CRDB Watch (it relies on changefeed resolved checkpoints to know a revision is complete). This adds an experimental, opt-in, CRDB-only Watch path that doesn't depend on the changefeed for correctness, so Watch keeps advancing under bulk load.

Behind --datastore-experimental-crdb-changelog-watch (default off — when off, behavior is unchanged):

Creates an append-only relationship_changelog table (hash-sharded PK on the HLC commit timestamp, row-level TTL) at init.
Relationship, schema, and bulk-load writes dual-write into it in the same transaction.
Watch polls the table at present time instead of consuming a changefeed. The table is append-only, so a present read sees every committed row; completeness comes from the cluster's max clock offset (checkpoint/cursor advances only to clusterNow − maxOffset), not changefeed resolved, so no write is skipped. A lightweight changefeed on the table is used only as a low-latency nudge, with the poll interval as the correctness backstop.

Notes:

Checkpoint floor ≈ maxOffset (--datastore-experimental-crdb-changelog-watch-max-offset, default 250ms). Must be ≥ the cluster's --max-offset (CRDB default 500ms) or Watch can miss commits.
Prototype scope: no cutover backfill, no integrity metadata in the changelog.

Testing

Extensive unit and integration tests, but the impact to write throughput has not been measured, that could make this a dealbreaker.

TODO: run with --datastore-engine cockroachdb --datastore-experimental-crdb-changelog-watch and exercise Watch during a bulk load.

…isions

…atch

appendRelationshipChangelogBatch built one multi-row INSERT with 14 columns per relationship. Postgres/pgx cap bound parameters at 65535, so a BulkLoad of more than ~4600 relationships would overflow the limit and fail the whole transaction, defeating the point of the feature. Split the insert into chunks of changelogInsertChunkSize (default 4000) executed within the same transaction, keeping the per-transaction ordinal counter continuous across chunks so (change_ts, ordinal) uniqueness is preserved. Adds a regression test that lowers the chunk size and bulk-loads across multiple chunks, asserting every row is observed over the changelog watch.

AddChangedDefinition's *core.CaveatDefinition branch looked up a prior entry under nsPrefix+name to decrement its byte size, but caveats are stored under caveatPrefix+name. The mismatch meant re-changing the same caveat within one buffering window never found the prior entry, over-counting currentByteSize on every repeat change. Shared by all datastores' watch path (Postgres, MySQL, and the CRDB changelog path); not data loss, but a real accounting bug. Adds a regression test mirroring the existing namespace-replacement test, which fails against the prior code.

…watch watchViaChangelog always buffers and dedups changes per poll interval via common.NewChanges and does not honor opts.EmissionStrategy, even though the CRDB datastore advertises WatchEmitsImmediately support. A caller requesting EmitImmediatelyStrategy silently gets buffered, checkpoint-grouped delivery instead, and Creates are reported as Touch. Document this as an accepted limitation of the experimental path rather than re-architecting it; immediate emission is moot given the follower-read latency floor anyway.

…ion_metadata table

…set floor

…angelog watch

ecordell · 2026-07-02T22:21:26Z

 		delete(record.caveatsDeleted, t.Name)

-		if existing, ok := record.definitionsChanged[nsPrefix+t.Name]; ok {
+		if existing, ok := record.definitionsChanged[caveatPrefix+t.Name]; ok {


this looks like a latent bug to me

😱 yes, please send a different PR

codecov · 2026-07-02T22:26:36Z

Codecov Report

❌ Patch coverage is 65.28804% with 235 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/datastore/crdb/watch_changelog.go	61.67%	108 Missing and 30 partials ⚠️
internal/datastore/crdb/changelog.go	74.61%	34 Missing and 14 partials ⚠️
internal/datastore/crdb/readwrite.go	50.00%	15 Missing and 6 partials ⚠️
internal/datastore/crdb/caveat.go	0.00%	14 Missing and 2 partials ⚠️
internal/datastore/crdb/crdb.go	77.36%	8 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

miparnisari · 2026-07-17T18:48:24Z

      --datastore-disable-watch-support                                       disable watch support (only enable if you absolutely do not need watch)
      --datastore-engine string                                               type of datastore to initialize ("cockroachdb", "mysql", "postgres", "spanner") (default "memory")
      --datastore-experimental-column-optimization                            enable experimental column optimization (default true)
+      --datastore-experimental-crdb-changelog-watch                           EXPERIMENTAL: use a dual-written changelog table polled at a closed timestamp for Watch instead of a changefeed (CockroachDB driver only). Keeps Watch advancing during bulk loads at the cost of write latency.


at the cost of write latency.

we should verify this before merge

miparnisari · 2026-07-17T18:49:08Z

+      --datastore-experimental-crdb-changelog-watch                                     EXPERIMENTAL: use a dual-written changelog table polled at a closed timestamp for Watch instead of a changefeed (CockroachDB driver only). Keeps Watch advancing during bulk loads at the cost of write latency.
+      --datastore-experimental-crdb-changelog-watch-max-offset duration                 EXPERIMENTAL: safety margin subtracted from the cluster clock to derive the changelog Watch completeness cursor/checkpoint; MUST be >= the cluster's --max-offset (CockroachDB default 500ms) or Watch can miss commits (CockroachDB driver only) (default 250ms)


nit: these are the only flags that we added (to serve); why do they also appear under gc and repair?

miparnisari · 2026-07-17T18:49:46Z

 		delete(record.caveatsDeleted, t.Name)

-		if existing, ok := record.definitionsChanged[nsPrefix+t.Name]; ok {
+		if existing, ok := record.definitionsChanged[caveatPrefix+t.Name]; ok {


😱 yes, please send a different PR

miparnisari · 2026-07-17T19:00:05Z

+	case tuple.UpdateOperationCreate:
+		return "create", nil
+	case tuple.UpdateOperationTouch:
+		return "touch", nil


if i send a touch but the relationship already existed and nothing got written, will this writes an event to the changelog? if the answer is yes, this seems useless 🤔 what consumer would care about this?

miparnisari · 2026-07-17T19:13:03Z

+	TableCaveat                = "caveat"
+	TableRelationshipCounter   = "relationship_counter"
+	TableTransactionMetadata   = "transaction_metadata"
+	TableRelationshipChangelog = "relationship_changelog"


Suggested change

TableRelationshipChangelog = "relationship_changelog"

TableChangelog = "changelog"

miparnisari · 2026-07-17T19:24:14Z

+// changelogRowFields holds the reconstructed non-key columns of a changelog
+// row, shared by the buffered and immediate scan paths.
+type changelogRowFields struct {
+	kind                         string


miparnisari · 2026-07-17T19:31:35Z

+// distinguishes Touch vs Delete — at the row level a physical insert is
+// indistinguishable from a touch of a previously nonexistent row — so those
+// backends report inserts as Touch. The immediate path emits through
+// streamingChangeProvider, which does represent Create distinctly, so it passes


The immediate path emits through streamingChangeProvider, which does represent Create distinctly

what's the reason for this and do we want to preserve it? seems weird that each "mode" returns different things

ecordell added 16 commits July 1, 2026 11:20

feat(crdb): add experimental changelog-watch flag plumbing

67cef25

feat(crdb): gated creation of experimental relationship_changelog table

daedbc9

feat(crdb): dual-write relationships into changelog table

12c08cd

fix(crdb): make changelog ordinal transaction-scoped to avoid PK coll…

deef3b6

…isions

feat(crdb): dual-write schema changes into changelog table

d9310f4

feat(crdb): poll-based changelog Watch at closed timestamp

8fb5224

fix(crdb): emit changelog rows at exactly the poll target revision

d3d4b3b

feat(crdb): dual-write bulk-loaded relationships into changelog

2bf3343

feat(crdb): changefeed nudge for low-latency changelog watch

98c6a4b

feat(crdb): stale-cursor handling + docs for experimental changelog w…

8ca0b10

…atch

feat(crdb): carry transaction metadata in changelog and skip transact…

ca15ca6

…ion_metadata table

feat(crdb): emit changelog changes immediately, checkpoint at max-off…

c227002

…set floor

test(crdb): deterministic immediate-mode cursor/dedup coverage for ch…

5f485b9

…angelog watch

ecordell commented Jul 2, 2026

View reviewed changes

github-actions Bot added area/cli Affects the command line area/datastore Affects the storage system area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) labels Jul 2, 2026

miparnisari reviewed Jul 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Experimental: changelog table for crdb - #3211

Experimental: changelog table for crdb#3211
ecordell wants to merge 16 commits into
authzed:mainfrom
ecordell:evan/crdbchangetbl

ecordell commented Jul 2, 2026

Uh oh!

ecordell Jul 2, 2026

Uh oh!

miparnisari Jul 17, 2026

Uh oh!

codecov Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

miparnisari Jul 17, 2026

Uh oh!

miparnisari Jul 17, 2026 •

edited

Loading

Uh oh!

miparnisari Jul 17, 2026

Uh oh!

miparnisari Jul 17, 2026

Uh oh!

miparnisari Jul 17, 2026

Uh oh!

miparnisari Jul 17, 2026

Uh oh!

miparnisari Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		--datastore-experimental-crdb-changelog-watch EXPERIMENTAL: use a dual-written changelog table polled at a closed timestamp for Watch instead of a changefeed (CockroachDB driver only). Keeps Watch advancing during bulk loads at the cost of write latency.
		--datastore-experimental-crdb-changelog-watch-max-offset duration EXPERIMENTAL: safety margin subtracted from the cluster clock to derive the changelog Watch completeness cursor/checkpoint; MUST be >= the cluster's --max-offset (CockroachDB default 500ms) or Watch can miss commits (CockroachDB driver only) (default 250ms)

	TableRelationshipChangelog = "relationship_changelog"
	TableChangelog = "changelog"

Uh oh!

Conversation

ecordell commented Jul 2, 2026

Description

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miparnisari Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jul 2, 2026 •

edited

Loading

miparnisari Jul 17, 2026 •

edited

Loading