Opt-in deduplication of samples at ingestion

[fungiboletus]

What

SENSAPP_DEDUPLICATE_ON_INGEST=true (or deduplicate_on_ingest = true), off by default, makes the write itself leave out the samples that are stored already, and write the repeated samples of one request once. The rule is the vacuum's: same series, timestamp and value (coordinates for a location). Two different values at one timestamp are both kept, nothing is rejected, the request succeeds. It covers independent, repeated and overlapping requests, not only retries. The vacuum stays for duplicates written before it is turned on, and for ClickHouse.

Backend	Support	Several writers / instances
PostgreSQL, TimescaleDB	yes	exact: advisory lock per series, hashed into 1 024 buckets, held until the transaction ends
SQLite, DuckDB	yes	exact: single writer / single process
ClickHouse, BigQuery, RRDCached	no	the server refuses to start with the flag set

How

• PostgreSQL / TimescaleDB (src/storage/pg_samples.rs, shared): the unnest insert becomes INSERT .. SELECT DISTINCT .. WHERE NOT EXISTS against the stored rows inside the time window of the batch (a per-row probe of the BRIN index cost 0.36 ms a row), with a transaction-local planner guard (no nested loop / seq scan: new samples are newer than the statistics, which gave 3 to 5 s plans for 20 000 samples).
• SQLite: WITH u AS (VALUES ..) INSERT .. WHERE NOT EXISTS. DuckDB: the appenders write a temp table, one statement copies the new rows.
• PostgreSQL migration: autosummarize on the BRIN indexes.
• Config entry, settings.toml, Helm value, docs/CONFIGURATION.md, docs/DATA_LIFECYCLE.md.

For reviewers

• A read before the insert is not enough with several instances. A concurrent-writers test first stored 401 rows where 51 were expected; the advisory locks fix it. A stress test (10 concurrent requests of 8 000 series) also showed one lock per series exhausting PostgreSQL's shared lock table (out of shared memory, HTTP 500); hence the fixed 1 024 buckets. Consequence: concurrent large batches take turns.
• PostgreSQL condition. The probe goes through BRIN, which returns every range not summarized yet in full. Right after a bulk load a small write took 64 ms (3.5 to 4 ms once autovacuum caught up; it did by itself in 90 s with default settings). A large import is slower with the flag on (10.5 s vs 6 s for 1 M samples). Documented, with the advice to turn it off for a clean one-time import.
• Cost (release build, 1 M rows, tests/perf/dedup.sh, one run, indicative): no visible difference for a 20 000 sample request on SQLite, DuckDB, TimescaleDB; PostgreSQL 0.14 s vs 0.19 s once summarized. Small writes: SQLite +0, DuckDB +1.5 ms, TimescaleDB +0.5 to 1.5 ms, PostgreSQL +1 to 3 ms. Re-sending known samples is as fast or faster. All numbers and the baseline are in done/ingestion-deduplication.md.
• Tests (backend-generic, tests/integration/deduplication.rs): every value type, repeats inside a request, exact duplicates only, partial overlap, switching off, concurrent writers, repeated/overlapping requests through the HTTP API, compressed TimescaleDB chunks, which backends accept the switch, the migration. Full suites on the five backends and cargo clippy --all-targets --features all-storage -- -D warnings pass (the DuckDB doctest only fails locally for the known libduckdb.dylib reason).
• Found on the way, not caused by this PR: on TimescaleDB, concurrent first writes of the same new series deadlock (8 of 8 runs with the flag off). The concurrent test skips that half of its rounds on TimescaleDB; written up in ideas/timescaledb-concurrent-first-write-deadlock.md, to fix separately.
• Not done: ClickHouse (no exact mechanism at insert time), throughput of many concurrent writers of overlapping series, DuckDB staging refinements. See the end of done/ingestion-deduplication.md.

🤖 Generated with Claude Code