perf(reporting-storage): drop the per-object advisory lock from deletes and self-recorded adoption

.changeset/remove-object-lock.md

@@ -0,0 +1,11 @@
+---
+'@adcp/reporting-storage': patch
+---
+
+Reporting storage (experimental): retirement, compaction and the write-intent sweep no longer take the per-object advisory lock, and a commit only takes it for an adopted object its own intent did not record (#3281).
+
+The lock (`adcp-reporting-object:{binding}:{objectRef}`) made every reference-checked delete a transaction that held a pooled connection and an advisory lock across the provider delete, and made every commit that adopted an existing object pay one lock round trip per object. Since every writable key template must include `{revision_key}`, only one row set can ever name a key, so the lock no longer ordered anything on those paths. Safety there comes from the reference check plus the fences that already existed: the account lock with the `FOR SHARE` on a header-only revision's predecessor, the `pruning` state each retirement and compaction commits before it deletes, the intent's `open` to `sweeping` transition against the commit that closes it, and version-pinned deletes. The full argument, case by case, is in `docs/guides/REPORTING-LEDGER.md` and on `deleteUnreferencedObjects`.
+
+Two places keep the lock, because nothing else orders them: the orphan collector's delete (its object is named by no record) and a commit that adopts an object its own write intent did not record (an earlier attempt's `PUT` landed before it was recorded). The collector now takes every binding's lock for the object in one transaction, as before. A retry that adopts an object its own intent recorded takes no lock and still proves it with the version-pinned `stat`.
+
+Why patch and not minor: no exported API, no schema and no wire behavior changes, and the lock key text is unchanged, so a rolling upgrade between a version that locks everywhere and this one is safe (every pair that still needs mutual exclusion keeps the lock on both sides). Nothing to migrate. The Python SDK carries the same item (adcp-client-python#1492); the exact final contract is under "Cross-SDK contract" in `docs/guides/REPORTING-LEDGER.md`.

docs/guides/REPORTING-LEDGER.md

@@ -744,16 +744,71 @@ await store.readyRowStorage();
   several row sets. Every delete path (period retirement, compaction and the
   intent sweep) therefore skips an object while a live row set outside the
   operation still references the same binding, key and native version, or an
-  open write intent created it. (With unique keys, a create-only write only
-  finds an existing object when a writer retries the same row set.) The check
-  and delete run under the per-object lock described below. The reference check and the delete run in one
-  transaction under a per-object advisory lock, and a commit that adopts an
-  existing object takes the same lock, proves the object is still there and
-  holds the lock until its chunk rows commit. A delete that wins the lock makes the adopting
-  commit fail with `STATE_UNAVAILABLE` (the writer re-uploads once its intent
-  lapses); a commit that wins is seen by the delete, which skips. A skipped
-  object is deleted by the retirement, compaction or sweep of the last row set
-  that referenced it, so it is not leaked.
+  open write intent created it. With unique keys, a create-only write only
+  finds an existing object when a writer retries the same row set.
+- **Object deletes and commits take no per-object lock, with one exception.**
+  Retirement, compaction and the intent sweep run the reference check as a
+  plain query, then delete the exact native version, holding no transaction or
+  pooled connection across the provider call. They are safe without a lock
+  because of five fences, none of which is the lock:
+  1. Every writable key includes `{revision_key}` (a SHA-256 of the row set ID)
+     and `{content_sha256}`, so one row set alone can ever name a key through a
+     writable binding. A committed row set is replayed: a late writer that
+     passed the replay check can adopt its just-committed object, but its
+     commit inserts nothing (the row set exists), so it cannot reference an
+     object a retirement or compaction then deletes. A `readOnly` binding's old
+     template can map several row sets to one object, but nothing adopts through
+     it; there the reference check and fence 4 are the whole argument. Bindings
+     that share one bucket and prefix must route a row set to one binding across
+     attempts, because the reference check is per binding (the orphan collector
+     checks every binding).
+  2. A header-only revision's commit holds the account lock and `FOR SHARE`-locks
+     a predecessor that must still be `live`. Retirement and compaction mark a
+     row set `pruning` under the same account lock before they delete anything,
+     and a `pruning` row set neither counts as a reference nor can be shared. So
+     a sharer commits first (its chunk rows then make the delete skip) or fails
+     with `STATE_UNAVAILABLE` and re-uploads under its own key.
+  3. The intent sweep moves an expired `open` intent to `sweeping` under the
+     account lock; a writer closes that intent, still `open`, in the commit
+     transaction that inserts its chunk rows. Exactly one wins, so a writer past
+     its lease cannot commit objects the sweep deletes, and a retry cannot take
+     over an intent being swept. A takeover renews the expiry, so the sweep
+     cannot claim it.
+  4. Concurrent retirements and compactions count only row sets that are not
+     `pruning`, each after committing its own `pruning` state, so at least one
+     of any two deletes a shared object and none can both skip it.
+  5. Deletes name an exact native version, so one cannot remove an object
+     re-created at the same key. This needs write-unique native versions, a
+     provider requirement the conformance check `write-unique-version` tests.
+
+  The advisory lock `adcp-reporting-object:{binding_id}:{object_ref}` stays in
+  two places that pair with each other, because nothing else orders them. A
+  commit that adopts an object its own intent did not record (an earlier
+  attempt's `PUT` landed before the intent recorded it, or a writer created it
+  after a takeover) takes the lock, in key order, before it proves the object with
+  a version-pinned `stat`, and holds it until its chunk rows commit. An object its own intent recorded takes
+  no lock: until the transaction commits, the open intent names it. The orphan
+  collector deletes only objects no record names, so each of its deletes takes
+  the lock of every binding (they can share a bucket and prefix), repeats the
+  reference check under it, and then deletes. A collector that wins makes the
+  adopting commit fail its version-pinned `stat` with `STATE_UNAVAILABLE` (the
+  writer re-uploads once its intent lapses); a commit that wins is seen by the
+  collector, which skips. The collector's clone fence is a separate protection:
+  an advisory lock is per database and cannot order a clone against the
+  original. A skipped object is deleted by the retirement, compaction or sweep
+  of the last row set that referenced it, so it is not leaked.
+- **Cross-SDK contract.** A ledger database has one writer SDK (the design record's
+  §5), so the Python and JavaScript SDKs need not agree on a lock to coexist; this
+  is the intended contract for both, with the same key text. The Python side
+  (adcp-client-python#1492) mirrors it: (1) the delete paths above take no object lock; (2) the key text
+  is `adcp-reporting-object:` + binding ID + `:` + object ref, hashed with
+  `hashtextextended(text, 0)` for `pg_advisory_xact_lock`; (3) an adopting commit
+  locks only adopted objects absent from its own intent's `created_objects`
+  (matching on object ref and native version), in object-ref order, after the
+  account lock and after closing its intent, then proves each adopted object with
+  a version-pinned `stat`; (4) the orphan collector locks every binding's key for
+  the object, sorted by binding ID, in the transaction that repeats the
+  reference check and deletes. Lock order is account lock, then object locks.
 - An upload runs only while the obligation lease still covers
   `writeDeadlineMilliseconds` plus five seconds. A fenced write intent records
   every object the upload created, and the ledger commit closes the intent in
@@ -780,7 +835,7 @@ await store.readyRowStorage();
   absent, without reading the body. The commit transaction uses it to prove an
   adopted object is unchanged (native versions are write-unique, so the same
   version at the recorded size is the same bytes) instead of downloading it
-  while holding the account and object locks. A provider without `stat` falls
+  while holding the account lock. A provider without `stat` falls
   back to reading the whole object there. All bundled providers implement it.
   Also implement the optional `list({ prefix, pageToken, pageSize })`, which
   returns `{ objects: [{ objectRef, nativeVersionRef, sizeBytes, createdAt }],

docs/guides/REPORTING-OPERATIONS.md

@@ -180,10 +180,13 @@ service runs for you. See
    `minAgeSeconds` ago (default 7 days, never less than 1 day, which must stay
    well above the write deadline plus intent expiry); no row set that is not yet
    `pruned` has a chunk at that key and no write intent lists it, at any version;
-   and, under the per-object advisory lock every other delete takes, no `live` or
-   `unavailable` row set references that exact version and no open intent
-   created it, checked under every binding because two bindings can share a
-   bucket and prefix (a retired binding kept `readOnly` next to its successor).
+   and, under the per-object advisory lock (the only delete that takes it: a
+   commit that adopts an object its own intent did not record takes the same
+   lock, so the two cannot interleave), no `live` or `unavailable` row set
+   references that exact version and no open intent created it, checked under
+   every binding because two bindings can share a bucket and prefix (a retired
+   binding kept `readOnly` next to its successor). Retirement, compaction and the
+   intent sweep take no lock and hold no connection across their provider deletes.
    `readOnly` bindings are skipped unless `includeReadOnly: true`. A binding
    whose provider has no `list` is reported as `unsupported` (and as a
    maintenance warning), never silently skipped. Everything under the owned prefix

docs/migration-reporting-row-storage-alignment.md

@@ -62,8 +62,10 @@ read-only binding is exempt from the requirement, is never chosen for new
 writes, and is still used for reads and retirement (`readyRowStorage` still
 probes it, so keep its location reachable). Compaction ignores read-only
 bindings, because nothing can adopt through them. The
-reference-checked deletes and the per-object lock are unchanged, because
-header-only revisions still share chunks within an obligation.
+reference-checked deletes are unchanged, because header-only revisions still
+share chunks within an obligation. (The per-object lock this note once said was
+unchanged has since been removed from every path but the orphan collector; see
+`docs/guides/REPORTING-LEDGER.md`.)
 
 ## Renamed fields