Fix the TimescaleDB deadlock between first writes of new series and chunk creation

[fungiboletus]

What

On TimescaleDB, concurrent writers could deadlock (deadlock detected, HTTP 500) when one registered a new series and another created a chunk. Reproduced with eight writers of the same new series, and with writers of different series.

Stacked on #46 (dedup-at-ingestion, not merged yet): the reproducing test lives there, so the diff against main includes its commits until it merges. The commits of this PR are the last five.

Cause

The value hypertables are partitioned by 7-day range and by by_hash('sensor_id', 2). The first sample in a chunk that does not exist yet creates it: ShareUpdateExclusiveLock on the hypertable, then ShareRowExclusiveLock on sensors (the foreign key is copied onto the chunk). That waits for every transaction holding an uncommitted sensor insert. A batch that registered a new sensor and then needs a chunk takes the two locks in the opposite order. Not limited to the same series; confirmed with pg_locks, the server log and two hand-driven sessions.

Fix

1. Lock order: a batch with sensors to register first runs LOCK TABLE ONLY <its hypertables> IN SHARE UPDATE EXCLUSIVE MODE (alphabetical), then inserts the sensors. Still one transaction, so a failed batch leaves no sensor behind. Batches of known series take no lock. The PostgreSQL backend passes an empty list.
2. Retry: the inserts meet the types in code order, not alphabetical, so one inversion remains possible. publish retries up to 5 times on deadlock_detected (SQLSTATE 40P01) with a 10-60 ms random pause. The foreign-key retry shares the loop.
3. The TimescaleDB exception in the concurrent dedup test is removed.

Rejected: advisory lock per uuid (does not cover different series), a first committed transaction (breaks the no-leftover-sensor guarantee), retry alone (54 deadlocks and 26 s in the extreme case, still failing).

Tests

• Two concurrent first-write tests on a new time range: fail 3/3 without the lock, 0 deadlocks in 30 runs with it.
• A deterministic test for the retry: fails with one attempt, passes with five.
• The dedup concurrent test alone: 4/4 failures before, 10/10 passes after.
• cargo test --no-default-features --features timescaledb (243 unit, 292 integration) passes; cargo clippy --all-targets --features all-storage -- -D warnings and cargo fmt --check are clean. The PostgreSQL suite was not run (no container).

Cost (release binaries before/after, tests/perf/scale.sh 3000, 3 runs each)

• Write of 3000 new series: 3.05/2.96/2.93 s before, 3.04/3.01/2.91 s after. Known series: 1.17/1.15/1.14 s vs 1.17/1.17/1.11 s. No change.
• Regression to know about: registering batches now take turns per value type, so 4 parallel loaders of 3 000 new series each take 11.5-11.9 s instead of 4.5-5.3 s. One-time per series. Options in ideas/timescaledb-parallel-first-writes.md.

Not covered

Deduplication on plus concurrent registration under load (a cycle through the series advisory lock looks possible by reasoning; the retry should recover it), and large batches holding the lock while others wait.

Details: done/timescaledb-first-write-deadlock.md.

🤖 Generated with Claude Code