feat(reporting-storage): fence the orphan collector against a cloned or restored ledger database

.changeset/orphan-collector-clone-fence.md

@@ -0,0 +1,21 @@
+---
+'@adcp/reporting-storage': minor
+---
+
+Reporting storage (experimental): the orphaned-object collector is fenced against a cloned or restored ledger database (#3292).
+
+A clone or point-in-time restore of the ledger database shares the live database's installation identity, namespace key and configuration, so it also shares `acknowledgeExclusiveNamespace: true`. Its collector saw an older set of references and would have deleted the live database's newer row objects. A real `collectOrphanedRowObjects` pass now claims the namespace first and refuses to delete when another database holds a newer claim.
+
+- **Fence.** A chain of create-only objects, one per epoch, under `{prefix}/{namespace_key}/_collector-fence/`, each holding a random token. The database records its last claim in `adcp_reporting_orphan_collector_fences`. A pass deletes only while the highest epoch in the bucket is exactly the claim the database recorded; it claims the next epoch before it deletes anything and re-reads the fence before every page that deletes and every 25 deletes within one. A copy taken before the original's latest claim sees a higher epoch and refuses.
+- **Reporting.** A refused binding has `status: 'fenced'` and `fence: { cause, detail, fenceEpoch?, recordedEpoch? }`; the result has a new required `fenced: string[]`, and the maintenance task raises a warning. Causes: `fence_advanced`, `claimed_elsewhere`, `token_mismatch`, `fence_missing`, `fence_regressed`, `fence_unreadable`, `claim_lost`, `migration_required`. `dryRun` reads the fence without claiming it and reports fenced bindings too.
+- **Takeover.** New `store.takeOverOrphanCollectorFence({ acknowledgePreviousOwnerGone: true, bindings?, includeReadOnly? })` for a database restored in place or whose predecessor is gone. It claims the next epoch and fences every other copy out; stop the previous owner's collector first, because a pass already running notices the new epoch at its next fence check (at most 25 deletes later).
+- **Limit.** Nothing in the database tells a copy from the original until one of them claims, so the first claim wins. Run orphan collection only on the live database, never on a copy of the deployment, and claim from the live database first when you roll this out. A fenced binding is checked again within the hour.
+
+Migration note (minor, breaking only for orphan collection):
+
+1. Run `npx adcp-reporting-storage migrate`: new version `0009`, `reporting_orphan_collector_fence` (additive; the SQL is exported as `REPORTING_ORPHAN_FENCE_MIGRATION`). Until it is applied, a pass reports every binding as `fenced` with cause `migration_required` and deletes nothing, and `postgresStorage().checkSchema()` names `0009` as pending.
+2. Run one real pass on the live database before any staging copy runs a collector.
+3. A custom store that implements `collectOrphanedRowObjects` for the maintenance task must return `fenced: []`. `ReportingOrphanBindingResultV1.status` gains `'fenced'`.
+4. Fence objects live in your object-store prefix. The provider contract is unchanged (create-only `putIfAbsent`, `get`, `list`, `delete`), except that `list` must now reflect keys `putIfAbsent` has already returned (read-after-write), as the bundled providers do. Run `runReportingRowObjectProviderConformanceV1` on a custom provider.
+
+See `docs/guides/REPORTING-OPERATIONS.md` and `docs/migration-reporting-storage-package.md`.

docs/guides/REPORTING-OPERATIONS.md

@@ -198,12 +198,65 @@ service runs for you. See
    of them. The age threshold does not protect against that, which is why a real
    (not dry) pass needs `acknowledgeExclusiveNamespace: true`: it is your statement
    that this database is the only owner. A clone made by copying the deployment
-   copies the flag too, so the flag does not protect it: remove it, or the whole
-   option, from every copy. Disable the option on clones, and after
-   restoring a database in place expect the collector to delete the objects of the
-   row sets the restore lost, which is its purpose.
+   copies the flag too, so the flag alone does not protect it, and a persisted
+   **ownership fence** backs it. It needs migration `0009`
+   (`npx adcp-reporting-storage migrate`); until then every real pass and dry run
+   reports each binding as fenced with cause `migration_required` and deletes
+   nothing.
+
+   - **Claim.** The first real pass claims the namespace: it records
+     `(epoch, token)` in `adcp_reporting_orphan_collector_fences` and writes a
+     create-only object, `{prefix}/{namespace_key}/_collector-fence/{epoch}.json`,
+     for that epoch. Every later real pass claims the next epoch before it
+     deletes anything, and prunes the claims older than the previous one.
+   - **Check.** A pass deletes only while the highest epoch in the bucket is
+     exactly the claim this database recorded. It re-reads the fence before every
+     page that deletes (and every 25 deletes within one). Anything else refuses, deletes nothing more and reports the
+     binding as `fenced` with a `cause` (`fence_advanced`, `claimed_elsewhere`,
+     `token_mismatch`, `fence_missing`, `fence_regressed`, `fence_unreadable`,
+     `claim_lost`, `migration_required`), in the result's `fenced` list and as a
+     maintenance warning. Bindings that share a bucket, prefix and installation
+     share one fence.
+   - **Clones and restores.** A copy that still holds an older claim sees the
+     live database's newer epoch and refuses. So does a database restored to an
+     earlier state. A `dryRun` reads the fence without claiming it, so it
+     reports a fenced binding too and writes nothing.
+   - **First claim wins.** Nothing in the database distinguishes a copy from the
+     original until one of them claims. When you roll the fence out, claim from the
+     live database first: one real pass, or `takeOverOrphanCollectorFence` (on a
+     namespace nobody has claimed it is a claim and nothing more). If a clone collects before the original
+     has run a pass since the copy was taken, the clone claims first and the
+     original is fenced, and the warning is the signal. The age threshold still
+     holds: the clone cannot delete an object the original wrote after the copy
+     until that object is `minAgeSeconds` old, so turn the clone's collector off
+     within that window. Run orphan collection only on the live database, and
+     never copy the deployment with the option on.
+   - **Takeover.** After restoring a database in place, or when the original is
+     gone, call `store.takeOverOrphanCollectorFence({ acknowledgePreviousOwnerGone: true })`
+     once. It claims the next epoch for this database and fences every other
+     copy out. It refuses without the acknowledgement. Stop the previous owner's
+     collector first: a pass already running notices the new epoch at its next
+     fence check, up to 25 deletes later, and the provider cannot refuse a delete
+     on the fence's behalf. Do not call it from a
+     clone while the original is running: that fences the original and lets the
+     clone delete its objects.
+
+   A fenced binding is checked again within the hour (never a whole `intervalSeconds`
+   later), so applying `0009` or taking the namespace over resumes collection
+   without a restart. A pass writes one small fence object per pass (and prunes
+   the older ones), so the binding's credentials need write access, including a
+   `readOnly` binding collected with `includeReadOnly`. Keep `_collector-fence/` out
+   of bucket lifecycle rules and manual cleanup: deleting the newest fence object
+   makes an older copy look current again. The claim holds a database connection
+   and an advisory lock across the object-store listing and read.
+
+   After a restore in place, expect the collector to delete the objects of the
+   row sets the restore lost once you have taken the namespace over, which is its
+   purpose.
    The cost grows with bucket size: one listing request per page, plus one
-   database query per page and one transaction per deleted object.
+   database query per page and one transaction per deleted object. The fence
+   adds a small listing and a read per pass, and one more before each deleting
+   page.
 
 Alert on these row-storage signals:
 
@@ -216,6 +269,10 @@ Alert on these row-storage signals:
   host does not configure, and `overdue` (expired more than 24 hours ago, still
   unswept) means no host is handling them and their objects are leaking. The
   maintenance helper reports both through `onError`.
+- The `orphan object collection` task reporting `fenced` bindings: another
+  database holds the namespace's collector fence, or this one cannot prove it
+  does. The `fence.cause` says why; see the ownership fence above before you
+  take the namespace over.
 - The `orphan object collection` task reporting `failed` or `unsupported`
   bindings (their provider cannot list, so nothing there is collected). A listing
   that fails is retried from the page that failed (or from the start, if it was

docs/migration-reporting-storage-package.md

@@ -166,7 +166,19 @@ Reporting storage is now one decision. `postgresStorage({ pool })` (this package
 | `subscriptions: { principals: createPrincipalLifecycle({ store, notifications, agentUrl, resolvePrincipal }), stateStore }` | `subscriptions: { principal: { notifications, agentUrl, resolvePrincipal } }` with `storage`; serve `protocol: reporting.principals.protocol` |
 | `createReliableReportingConsumerV1({ persistence: createPostgresReportingConsumerRuntimeV1({ db, namespace }) })` | `createReliableReportingConsumerV1({ storage })` |
 
-The migration constants (`REPORTING_LEDGER_MIGRATION`, `REPORTING_ROW_STORAGE_MIGRATION`, `REPORTING_LEDGER_CHANGES_MIGRATION`, `REPORTING_MANAGED_DELIVERY_MIGRATION`, the object-write and finality-fence constants) are deprecated re-exports for one beta. A schema they set up migrates cleanly: the first `migrate` re-runs them as no-ops and records versions `0001`–`0008`. See the [package README](https://github.com/adcontextprotocol/adcp-client/blob/main/packages/reporting-storage/README.md#apply-migrations-npx-adcp-reporting-storage-migrate) for the version table.
+The migration constants (`REPORTING_LEDGER_MIGRATION`, `REPORTING_ROW_STORAGE_MIGRATION`, `REPORTING_LEDGER_CHANGES_MIGRATION`, `REPORTING_MANAGED_DELIVERY_MIGRATION`, the object-write and finality-fence constants) are deprecated re-exports for one beta. A schema they set up migrates cleanly: the first `migrate` re-runs them as no-ops and records versions `0001`–`0009`. See the [package README](https://github.com/adcontextprotocol/adcp-client/blob/main/packages/reporting-storage/README.md#apply-migrations-npx-adcp-reporting-storage-migrate) for the version table.
+
+### Orphan collector ownership fence (storage `0.3`, migration `0009`)
+
+Only deployments that enable `orphanCollection` (or call `collectOrphanedRowObjects`) are affected. A clone or restore of the ledger database shares the live database's namespace key and configuration, including `acknowledgeExclusiveNamespace: true`, and would delete the live database's newer row objects. A real pass now claims the namespace first and refuses to delete when another database holds a newer claim.
+
+1. Run `npx adcp-reporting-storage migrate` (version `0009`, `reporting_orphan_collector_fence`, additive). Until it is applied, every pass, `dryRun` included, reports each binding as `fenced` with cause `migration_required` and deletes nothing. `postgresStorage().checkSchema()` names `0009` as pending.
+2. Roll `0009` out and claim the namespace from the **live** database before any staging copy of it runs a collector: run one real pass there, or call `store.takeOverOrphanCollectorFence({ acknowledgePreviousOwnerGone: true })`, which on a namespace nobody has claimed is a claim and nothing more. The first claim wins, and a copy has no way to tell it is a copy; an already-running copy with `acknowledgeExclusiveNamespace: true` that claims first fences the live database out.
+3. Handle the new result shape: `ReportingOrphanBindingResultV1.status` can be `'fenced'` (with `fence: { cause, detail, fenceEpoch?, recordedEpoch? }`), and `ReportingOrphanCollectionResultV1` has a required `fenced: string[]`. A custom store that implements `collectOrphanedRowObjects` for the maintenance task must return `fenced` (an empty array is fine). The maintenance task raises a warning for each fenced binding.
+4. After restoring a database in place, or when the database that held the namespace is gone, call `store.takeOverOrphanCollectorFence({ acknowledgePreviousOwnerGone: true })` once, then collection resumes. It refuses without the acknowledgement.
+5. The fence chain lives under `{prefix}/{namespace_key}/_collector-fence/` in each object binding's bucket. Your provider needs nothing new (create-only `putIfAbsent`, `get`, `list` and `delete`, which the conformance suite already covers), and a bucket lifecycle rule must not expire objects under the owned prefix (the probe already refuses one that does).
+
+See [Orphaned object collection](guides/REPORTING-OPERATIONS.md#adopting-row-storage-retention-and-the-change-feed) for the claim, check and first-claim-wins rules.
 
 ### `service.setup.migrations` comes from the store
 

packages/reporting-storage/README.md

@@ -83,16 +83,17 @@ SDK's `python -m adcp migrate` uses), so concurrent runners serialize. `migrate(
 `pendingMigrations(pool)` and `migrations()` are the same operations in code. Startup never changes the schema:
 `storage.checkSchema()` names the pending versions and this command.
 
-| Version | Name                              | Contents (the deprecated constant that holds the same SQL)                                                 |
-| ------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
-| 0001    | `adcp_state`                      | principal records and reporting-subscription engine state (`ADCP_STATE_MIGRATION` from `@adcp/sdk/server`) |
-| 0002    | `reporting_ledger`                | the ledger (`REPORTING_LEDGER_MIGRATION`)                                                                  |
-| 0003    | `reporting_ledger_finality_fence` | **stopped-worker:** fences pre-SDK-14 writers (`REPORTING_LEDGER_FINALITY_WRITER_FENCE_MIGRATION`)         |
-| 0004    | `reporting_row_storage`           | row storage, PostgreSQL 13+ (`REPORTING_ROW_STORAGE_MIGRATION`)                                            |
-| 0005    | `reporting_ledger_changes`        | the host change feed, PostgreSQL 13+ (`REPORTING_LEDGER_CHANGES_MIGRATION`)                                |
-| 0006    | `reporting_managed_delivery`      | managed delivery (`REPORTING_MANAGED_DELIVERY_MIGRATION`)                                                  |
-| 0007    | `reporting_object_writes`         | the GCS write fence (`REPORTING_OBJECT_WRITE_MIGRATION`, `REPORTING_OBJECT_WRITE_AUTHORITY_MIGRATION`)     |
-| 0008    | `reporting_consumer`              | the buyer consumer (`REPORTING_CONSUMER_POSTGRES_MIGRATION` from `@adcp/sdk/reporting/consumer`)           |
+| Version | Name                               | Contents (the deprecated constant that holds the same SQL)                                                              |
+| ------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
+| 0001    | `adcp_state`                       | principal records and reporting-subscription engine state (`ADCP_STATE_MIGRATION` from `@adcp/sdk/server`)              |
+| 0002    | `reporting_ledger`                 | the ledger (`REPORTING_LEDGER_MIGRATION`)                                                                               |
+| 0003    | `reporting_ledger_finality_fence`  | **stopped-worker:** fences pre-SDK-14 writers (`REPORTING_LEDGER_FINALITY_WRITER_FENCE_MIGRATION`)                      |
+| 0004    | `reporting_row_storage`            | row storage, PostgreSQL 13+ (`REPORTING_ROW_STORAGE_MIGRATION`)                                                         |
+| 0005    | `reporting_ledger_changes`         | the host change feed, PostgreSQL 13+ (`REPORTING_LEDGER_CHANGES_MIGRATION`)                                             |
+| 0006    | `reporting_managed_delivery`       | managed delivery (`REPORTING_MANAGED_DELIVERY_MIGRATION`)                                                               |
+| 0007    | `reporting_object_writes`          | the GCS write fence (`REPORTING_OBJECT_WRITE_MIGRATION`, `REPORTING_OBJECT_WRITE_AUTHORITY_MIGRATION`)                  |
+| 0008    | `reporting_consumer`               | the buyer consumer (`REPORTING_CONSUMER_POSTGRES_MIGRATION` from `@adcp/sdk/reporting/consumer`)                        |
+| 0009    | `reporting_orphan_collector_fence` | the orphan collector's ownership fence (`REPORTING_ORPHAN_FENCE_MIGRATION`); needed before orphan collection can delete |
 
 A schema set up by applying those constants by hand migrates cleanly: the statements are idempotent, re-run as no-ops,
 and the versions are recorded. The constants stay exported as deprecated re-exports for one beta.

packages/reporting-storage/scripts/run-tests.mjs

@@ -40,6 +40,7 @@ export const SERIAL_TESTS = [
   'reporting-gcp-validation-pg.test.js',
   'reporting-head-of-line-pg.test.js',
   'reporting-host-reads-pg.test.js',
+  'reporting-orphan-collector-fence-pg.test.js',
   'reporting-row-orphan-collector-pg.test.js',
   'reporting-row-retention-pg.test.js',
 ];

packages/reporting-storage/src/index.ts

@@ -15,6 +15,7 @@ export * from './postgres';
 export * from './row-storage-migration';
 export * from './row-storage';
 export * from './row-storage-object';
+export type { ReportingOrphanFenceCauseV1, ReportingOrphanFenceRefusalV1 } from './orphan-fence';
 export * from './object-writes';
 export * from './change-feed';
 export * from './maintenance';