fix(postgres): guard strict reads against replica revision gaps

[josephschorr]

Problem

Fixes #2525.

With Postgres read replicas enabled, SpiceDB intermittently returned object definition <ns> not found (surfaced as Unknown/FailedPrecondition) even when the configured follower delay exceeded the replication latency. It reproduced only under load and went away when replicas were removed.

Root cause

Reads filter rows with pg_visible_in_snapshot(created_xid, revision.snapshot) — a pure function of the revision's snapshot. A snapshot makes visible any committed transaction with an xid < xmax that is not in its in-progress (xip) list, so visible data can come from transactions with an xid at or above xmin.

The strict read-replica guard, however, only asserted that transaction xmin - 1 had resolved on the replica. When a concurrent, still-open transaction holds xmin below a freshly committed write, a lagging replica that has applied xmin-1 but not that write passes the guard while the row is absent → zero rows, no error, no fallback → not-found. At production write QPS there is almost always an in-progress write with a lower xid than the most recently committed ones, which is why it only showed up under load.

Fix

Replace the single-transaction check with an inline snapshot-domination check, evaluated on the replica's own connection so it stays valid behind a load balancer (which is why the check must be inline rather than a separate CheckRevision round-trip):

• the replica's current snapshot xmax must have reached the revision's xmax; and
• the replica must not still consider in-progress any transaction below the revision's xmax that the revision treats as committed.

This is the inline equivalent of the comparison CheckRevision performs. A single visible xid is not sufficient, because commit order does not follow xid order: a lower-xid transaction the revision sees as committed can still be replaying on the replica after higher xids have been applied.

Testing

New datastore tests (//go:build datastore && postgres) stand up a real primary + streaming hot-standby replica in containers and drive replication lag deterministically with pg_wal_replay_pause():

• TestStrictReadGuardDetectsVisibleDataGapOnReplica — drives the production guard SQL directly; it raises replica missing revision while behind and succeeds after catch-up.
• TestReplicaLagStrictReadFallsBackForNamespace — end-to-end through the strict replicated datastore; the read now falls back to the primary and finds the namespace instead of reporting it not-found.
• TestReplicaServesCaughtUpRevisionWithoutFallback — verifies a caught-up (min-latency) revision is still served by the replica with no fallback, asserted via the fallback metric.

Verified against Postgres 14, 17, and 18; mage testds:postgres is green.

Routing FullyConsistent reads to the primary is a separate optimization and is intentionally out of scope here.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[AlexeyRaga]

Hey, great work, can it be merged now?