Give restored and copied databases a new generation, and end subscriptions from before the copy instead of resuming them

[dawsontoth]

restore_backup (online and offline), branch databases and the LMDB→RocksDB migration now stamp a new database generation into the copy before it can be read. Generations are RocksDB-only; an LMDB database carries none. A live subscription from before the copy is ended with DatabaseGenerationChangedError instead of silently receiving the copy's events. This is the core of #2451. A resumable position is valid only if it names the current generation and sits at or above a per-generation resume floor that every prune raises (isResumablePosition). #2448 wires that check into Table.subscribe. The design comment on #2451 covers three planning-review rounds.

For the human reviewer

1. Requirement: as specified, with one scope call.
   • restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451 asks for three things: a generation stamped at every state-copy path, a resume check against it, and each of the three state carriers either consulting it or documented as out of scope.
   • It is warranted: Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 (P1, v5.3) is blocked on it, and [Epic] Restore a get_backup archive into a running Harper (incl. Fabric) #2632's items 5 and 6 depend on it.
   • This PR delivers the generation, every stamp, the resume predicate and live-subscription teardown. It does not wire the predicate into Table.subscribe; that is Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448. Replication's per-peer state and base copy are harper-pro's: Replication should treat a peer whose database generation changed as a new incarnation, so a restored node does not silently diverge harper-pro#944.
   • Whether this closes restore_backup rolls a database back in time with no epoch, so audit cursors, record versions, and replication sequence state all read as valid afterwards #2451 now or once Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 lands is your call, so the body says "Implements", not "Closes".
2. The planning framing did not clear on codex.
   • Codex returned better-alternative-exists in all three planning rounds. Each alternative refined the one before and was adopted; the design comment's resolution table lists every one. Gemini cleared rounds 2 and 3.
   • Two round-3 points were overruled as pre-existing limitations owned elsewhere. A third, about an incomplete harper copydb target, no longer applies now that LMDB copies stamp nothing:
     • A prune's floor raise is not power-loss durable before the purge deletes files. Now The audit retention floor is raised before a purge but not made durable before the purge deletes log files #2912, P3.
     • The stamp is only as durable as the restore-marker protocol it runs inside. [Epic] Restore a get_backup archive into a running Harper (incl. Fabric) #2632's staged publication (restore_backup: shared publication protocol and online restore into a new database (create-if-absent) #2637) closes that.
3. No scalar mode, a contract Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 inherits.
   • A position without a generation id is never resumable, so every existing timestamp-only consumer resyncs once when Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 adopts the check.
   • The alternative was to accept cursor >= epoch without an id. That certifies the 900/970/1000 legacy-bootstrap cursor, because no timestamp is immune to clock rollback. Adding it later reopens that window.
4. A second floor.
   • audit-resume-floor sits beside the audit floor. Every prune raises it in the same transaction, and it is read as never below a finite audit floor.
   • The audit floor itself is never lowered, and is raised only where a copy carried no log. The reason is that Table.commit's out-of-order reconciliation walks history on Infinity and skips below a finite floor, so rewriting the floor changes merge results.
   • Since the resume floor is read as a max, its own record only matters when the audit floor is unknown. That is exactly the case that previously blocked resume forever.
   • Known gap: a binary from before this change that prunes while the audit floor is unknown records nothing either record can show. This is documented rather than fixed. Failing closed instead, by refusing every resume while the audit floor is unknown, matters once Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 calls the predicate, and adopting it later would refuse positions that resume today.
5. Stamping inside the restore.
   • Both restore paths open the restored directory privately with Harper's openRocksDatabase options, stamp it, flush, and close it, all before completeRestore.
   • This runs under the restore's lock. When harper#2649's open exclusion lands, it must let the restore's own open through.
6. Live subscriptions end at a generation change.
   • After an online restore, a subscriber from before it now receives DatabaseGenerationChangedError (a ClientError, 409, code DATABASE_GENERATION_CHANGED, with a message that names no path) and closes.
   • Before this change it either stalled silently on the closed handle, or received the restored database's writes once any new subscriber attached.
   • A reopen under the same generation now ends each subscription with the retryable DatabaseClosingError (503, DATABASE_CLOSING), from kriszyp's review. Before, it stayed open with its commit listener on the closed audit store and silently received nothing. Rebinding it would not work: live delivery reads each record through the subscribing table's primary store (eventFromAudit), which belongs to the closed handle too. In production a same-generation reopen follows only an online restore that failed before changing anything; a schema reload never closes the stores.
   • A registration through any handle other than the one the path was last opened with, or through a closed one, is refused with the same pair of errors: DatabaseGenerationChangedError if the generation differs, else DatabaseClosingError. That covers a resubscribe a listener makes while its old subscription is being ended (from cb1kenobi's review), and a retry after DatabaseClosingError through the handle it already held. Static subscribe was already refused by getResource's closed-root check; a TableResource loaded before the close was not. The alternative was to rebind such a registration to the current database. A refusal can be loosened later without breaking a caller, but a rebind, once callers rely on it, cannot be taken back.
   • The terminal signals are two classes: the new DatabaseGenerationChangedError (409, resynchronize) and the existing DatabaseClosingError (503, retry), so clients branch on both. A single dedicated class was the alternative; the existing one already means "this database handle closed, retry".
   • A legacy LMDB auditPath layout keeps main's behavior, with no teardown and no handle check. That root is reopened on every metadata read, so a sweep there would end its subscriptions each time. resources/DESIGN.md lists it as not covered.
7. Smaller calls.
   • Generations and the resume floor are RocksDB-only (from kriszyp's review). An LMDB database carries neither, harper copydb and compaction (LMDB paths) stamp nothing, and isResumablePosition is always false there, so Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 applies the check to RocksDB alone and LMDB resumes stay as they are today. An LMDB reopen still ends its subscriptions, with the retryable DatabaseClosingError, since nothing on LMDB can report a replaced database. The LMDB branch of the metadata write stays, because the audit floor has always used it.
   • An unreadable generation record is never repaired by an open. Every resume against it is refused until a copy stamps a new one.
   • Creating a branch in read-only mode now fails, because the fork's generation cannot be written.
   • Ids are per database per node — the origin-log incarnation that harper-pro#922 binds — and never cluster-wide.
   • Keep delivering each record to a key's other subscribers when one subscription ends mid-delivery #2913 (fan-out splice fix) also edits resources/transactionBroadcast.ts, in Subscription.end() and the delivery loops. The overlap is textual only, and whichever PR lands second rebases.

Framing-Verdict: better-alternative-exists (5cef2e51e4bd)

Changes

• resources/auditStore.ts
  • Records: the generation record (a 16-byte id plus an epoch) and the resume-floor record.
  • Open: openAuditStore now establishes the generation, then ends the subscriptions registered through an earlier handle.
  • Metadata writes: updateAuditFloor is rebuilt on commitAuditMetadata. It writes all records or none: every write is read back, and a mismatch throws inside the transaction.
  • Pruning: raiseAuditFloor skips only when the floor, and on RocksDB the resume record, cover the cutoff, and raises the resume floor on RocksDB in the same transaction as the audit floor.
  • New functions:
    • getDatabaseGeneration and getAuditResumeFloor, which is never below a finite audit floor and reads the audit floor first, so a prune committing between its two reads is seen (from cb1kenobi's review).
    • isResumablePosition.
    • establishDatabaseGeneration, which skips an LMDB store. Genesis is a compare-and-set, and starts at the audit floor. An unreadable record is never repaired.
    • stampDatabaseGeneration and stampDatabaseDirectory, which does a private open, stamps, and flushes.
  • JSDoc: the floor docs now point resume decisions at the generation: the bootstrap mark, getAuditFloor and copies.
• resources/transactionBroadcast.ts
  • A registry entry now records the generation it registered under.
  • endSubscriptionsFromEarlierHandles detaches that entry, then ends each subscription: DatabaseGenerationChangedError for another or an unknown generation, the retryable DatabaseClosingError for the same one. The close is forced even when a listener throws.
  • Before any listener runs, the sweep records the reopened store as the path's current handle, as a token so a retired database's stores are not retained, and addSubscription refuses a registration through any other handle or a closed one. A full-database registration (replication's whenNextTransaction) is skipped rather than refused, which leaves replication as it was.
• utility/errors/hdbError.ts: DatabaseGenerationChangedError, a 409 ClientError.
• Stamp call sites
  • dataLayer/rocksdbBackup.ts: the online and offline restores, each before completeRestore.
  • resources/branchDatabase.ts: the branch staging checkpoint, before the rename.
  • bin/copyDb.ts:
    • the migration's staging store is stamped and flushed before the rename;
• Docs
  • resources/DESIGN.md: a new section. The floor section is condensed where the generation supersedes it (see the intro and copies).
  • dataLayer/DESIGN.md: the restore stamp step.
  • DESIGN.md: the index.
  • User docs: the companion Document that a restore ends subscriptions opened before it documentation#699 documents in the restore_backup reference and the 5.3 release notes that an online restore ends each pre-restore subscription with DATABASE_GENERATION_CHANGED (409). Nothing else here is user-facing yet: isResumablePosition has no public caller until Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448.

Verification

• New unit suites

  • unitTests/resources/databaseGeneration.test.js: 22 tests on RocksDB, and 2 on LMDB: an LMDB database gets neither record, and a reopen ends its subscriptions with DatabaseClosingError. It covers genesis, the resume floor (including a prune committing between its two reads), stamping, resumable positions and live subscriptions, including a resubscribe from inside the teardown, a resource loaded before the replacement, a same-generation reopen and a retry through the handle that reopen closed.
  • unitTests/dataLayer/restoreGeneration.test.js (RocksDB) drives the production restoreBackupOffline and a real reopen: the old subscriber is ended, and old positions are refused while new ones are accepted. It keeps analytics off while it runs: an analytics flush is followed by a schema rescan that opens every database under STORAGE_PATH, which could be one the suite is about to restore or, once the suite has ended, the next suite's. Linux CI caught the second case in rocksdbBackup.test.js.

• Fails on base: the first restore test fails on af42ef2a4 because the pre-restore subscription stays open, and it passes here. The other two restore tests fail on base only because getDatabaseGeneration does not exist. The two registration tests fail on 2c2b2c3f6, the head before the fix. There the reentrant resubscribe is accepted, and its entry's old generation tag makes the next same-generation reopen end the copy's subscribers. A resource loaded before the replacement fails with a TypeError instead of the 409. The same-generation test times out on 99923de8f: the first listener gets neither the write nor a terminal signal, which is kriszyp's check. The retry test fails on 8f0ebdf83 with a missing rejection.

• Extended suites

  • unitTests/resources/branchDatabase.test.js: a fork mints a new generation, and adopting it again keeps it.
  • unitTests/bin/migrationStagingRecovery.test.js: a migrated store is not genesis.
  • unitTests/resources/auditPurge.test.js: the fake now keys records separately.

• Gates. Failures are attributed by name against the same files on af42ef2a4. The test:unit:resources rows are on f72bcec47, the subscription integration row on b59c9c5cb, the Linux row on 2c2b2c3f6, and the rest on a4c517ba4.

  Gate	Result	Failures
  test:unit:resources (RocksDB)	3654 passing, 3 failing	the 3 sourceApplyConflictRetry failures, which fail on base
  test:unit:resources (LMDB)	2797 passing, 0 failing	14 fewer than before: the LMDB generation tests are gone
  test:unit:main	6244 passing, 19 failing	Run without applicationSpawn.test.js, which hangs locally. The same 19 fail on base.
  test:unit:main on Linux (a Node 24 container)	6249 passing, 3 failing	The same 3 component suites fail on base in the same container.
  test:integration:all	2230 of 2258 pass, 10 failing	isolated-application, shutdown-drain-e2e, record-lock-concurrency and log-rotation-write-path all fail on base too
  Integration, subscription subset: 19 files that subscribe over MQTT, WebSocket, SSE, REST or GraphQL, plus the drop and row-permission suites	188 of 188 pass	none

  lint:required, prettier --check, tsc and check-design-docs are clean.

• End-to-end route. A refused persisted cursor is not observable through a public surface until Consume the audit staleness floor in Table.subscribe: an opt-in stale-start check so MQTT durable resume signals truncation instead of replaying short #2448 calls isResumablePosition from Table.subscribe. The restore test drives the production restore and a real reopen.

🤖 Generated with Claude Code

Complexity: complicated

Origin — the dispatch brief this PR was written from

Give restored and copied databases a new generation, and end subscriptions from before the copy instead of resuming them

LIVE CONVERSATION about #2914.

You are answering a person, in a thread, one turn at a time. Every turn:

1. Read the whole thread in this dispatch file's # Log — it is the conversation so far, and each of your previous turns is in it. Read the PR/issue and the code as needed.
2. Answer the LAST message. Append your answer to # Log as your turn. Prose, not a report: they are talking to you, and a status template is not an answer.
3. Set status: needs-input and stop. The thread stays open; their next message resumes it.

Each turn arrives as ASK (answer it, change nothing) or PERFORM (do it, then say what you did) — the person chose which when they sent it, and the run's own prompt tells you which one this is. Never infer it from the wording: an unrequested commit in the middle of a discussion and a polite description of work that was supposed to happen are the two failures this exists to prevent.

Never mark a PR ready and never merge from this conversation.

_{Dispatch: task chat-pr-harper-2914-kriszyp · queued by unknown · ran by codex/gpt-6-astra/low · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=gemini,codex,cursor-composer; adjudicated=domain; blocked=cursor-grok(failed); declined=cursor-kimi,cursor-muse; rounds=6; full=1 @ f72bcec}

_{Human-Review-Need: 3 (decisions: rocksdb-only-generations, unknown-floor-exclusion-documented, handle-token-gate, every-reopen-ends-streams, legacy-layout-excluded, primitive-before-consumer, unreadable-generation-is-permanent, generation-error-contract, per-node-generation) @ f72bcec}

[gemini-code-assist]

Code Review

This pull request introduces database generations and resumable positions to prevent database copies, restores, and branches from incorrectly inheriting or resuming subscriptions or positions from their source states. It implements tracking for a database generation ID and a resume floor in the audit store metadata, stamps a new generation during copy, restore, and branch materialization, and terminates live subscriptions with a DatabaseGenerationChangedError when a generation change is detected. As there are no review comments provided, I have no feedback to provide on the review.

[claude]

Reviewed; no blockers found.

[cb1kenobi]

Generation-change notification is reentrant and can leave a newly registered stale subscription alive. Guard registration with the authoritative current generation while closing old subscriptions.

—
Reviewed 2c2b2c3

[cb1kenobi]

No confirmed blocking defects remain on the changed lines. The previously raised reentrant subscription issue is addressed at this commit.

—
Reviewed 99923de

[cb1kenobi]

No confirmed blocking defects remain on the changed lines. The previously raised reentrant subscription issue is fixed at this commit.

—
Reviewed 99923de

[kriszyp]

This looks good, although I don't think the overhead/complexity of supporting LMDB here is worth keeping.

_{🤖 Reviewed with Codex}

[cb1kenobi]

No confirmed blocking defects remain on the changed lines. Copy, restore, branch, and migration stamp a new generation before the copy is readable, and a replaced-database subscriber is ended and cannot rejoin through a stale handle. Same-generation reopen still leaves existing subscriptions registered without reattaching the commit listener; that is the pre-existing behavior this PR kept and is already on the thread.

—
Reviewed 99923de

[cb1kenobi]

The resume-floor lookup can observe inconsistent metadata during a concurrent prune and accept a cursor whose events were deleted. Read the records from one snapshot, or read the audit floor before the monotonically raised resume floor.

—
Reviewed 0d7830e

[cb1kenobi]

No confirmed blocking defects remain on the changed lines. Previously reported subscription and resume-floor races are fixed at this commit.

—
Reviewed 3953775

[cb1kenobi]

No confirmed blocking defects remain on the changed lines. Previously reported subscription and resume-floor races are fixed at this commit.

—
Reviewed 3953775

[cb1kenobi]

No confirmed blocking defects remain on the changed lines. The RocksDB-only generation scope and LMDB subscription behavior are internally consistent.

—
Reviewed f72bcec

[kriszyp]

Great!

_{🤖 Reviewed with Codex}

[kriszyp]

Each openAuditStore adds or replaces a currentHandles entry, but nothing removes one when its database is closed or dropped. A long-lived worker that creates and drops databases under distinct paths retains every path and generation ID indefinitely. Could close/drop remove the entry when its token still matches? The closed-store guard at resources/transactionBroadcast.ts:53 can continue rejecting registrations before the next open.