Repository navigation

Opt-in deduplication of samples at ingestion - #46

Merged

fungiboletus merged 12 commits into

mainfrom

dedup-at-ingestion

Oct 3, 2026

Commits on Oct 3, 2026

docs: 📝 plan the ingestion deduplication experiment, with the baseline numbers

fungiboletus
and
claude
committed
2dab1ba
feat: ✨ drop stored samples at ingestion on PostgreSQL and TimescaleDB (experiment)

fungiboletus
and
claude
committed
9367c2e
perf: ⚡️ keep the deduplicating insert on a hash join of the batch window

fungiboletus
and
claude
committed
db285d0
style: 🎨 fix the doc list of prefer_hash_join

fungiboletus
and
claude
committed
0e7e75f
feat: ✨ drop stored samples at ingestion on SQLite and DuckDB (experiment)

fungiboletus
and
claude
committed
72b5e9e
docs: 📝 results and verdict of the ingestion deduplication experiment

fungiboletus
and
claude
committed
8a83c02
docs: 📝 tidy the verdict of the deduplication experiment

fungiboletus
and
claude
committed
f95d102
fix: 🐛 hash the series locks of the deduplication into 1024 buckets

fungiboletus
and
claude
committed
06fdee0
test: ✅ keep the concurrent deduplication test off a pre-existing TimescaleDB deadlock

fungiboletus
and
claude
committed
794aecf
feat: ✨ configure the deduplication at ingestion, refuse to start where it is unsupported

fungiboletus
and
claude
committed
24ec43f
docs: 📝 close the ingestion deduplication task, with the final numbers

fungiboletus
and
claude
committed
62ff7a8
docs: 📝 point the deadlock idea at the closed task

fungiboletus
and
claude
committed
d9109b0