Repository navigation
Default DuckDB to Scan V2 and optimize sparse and wide scans - #10964
joseph-isaacs wants to merge 8 commits into
Conversation
e503266 to
f0752c6
Compare
Merging this PR will degrade performance by 5.77%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
47bbb02 to
e1d38d8
Compare
e1d38d8 to
d3f067e
Compare
Polar Signals Profiling ResultsLatest Run
Previous Runs (56)
Powered by Polar Signals Cloud |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.978x ➖, 2↑ 2↓)
datafusion / parquet / ns (1.043x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.035x ➖, 1↑ 3↓)
duckdb / parquet / ns (1.017x ➖, 0↑ 2↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.983x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.964x ➖, 4↑ 0↓)
duckdb / parquet / ns (1.018x ➖, 0↑ 1↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 3↑ 0↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.975x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.965x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.057x ➖, 0↑ 3↓)
|
6cc6bb6 to
92b8101
Compare
e0bc723 to
e6afd4d
Compare
e6afd4d to
1f41995
Compare
Emit small startup chunks per DuckDB worker, refresh dynamic pruning and row filters when bounds change, and adapt predicate order using observed cost and selectivity with occasional exploration. Preserve exact file and row pairs during DuckDB V2 late materialization. Retain the file-index hint for pre-open file pruning and gather sparse dictionary selections before conversion when encoded take kernels apply. Add benchmark controls, diagnostics, and regression coverage for startup, dynamic thresholds, adaptive ordering, paired row lookup, and export. Validation before develop integration: 301 DuckDB tests passed (2 S3 skips), workspace all-target/all-feature Clippy, Rust/C++ formatting, and diff checks. Q23 improved 37-41% across repeated 40-warm-sample local comparisons; payload rows 70 -> 10 and payload arrays 49 -> 7. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Select sparse row lookup from positive input selections, schedule only selected regions, and preserve encoded take kernels at segment leaves and dictionary values. Keep zero-copy native slices and the dense scan path. Expose an access-mode override and diagnostics for mode selection and gather work. Validation: 1,864 affected library tests passed (2 S3 skips), workspace all-target/all-feature Clippy, pinned formatting, layout doctests, and matched DuckDB/DataFusion benchmarks. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Preserve selective predicate demand at scan leaves while retaining vectorized evaluation for dense scan masks. Memoize referenced string dictionary predicates and reuse packed whole results after bulk evaluation; keep numeric comparisons on their vectorized path. Read dictionary codes before referenced value chunks for sparse random access, preserve duplicate/null code order, and keep partial reads out of whole-value caches. Fetch flat value segments alongside codes. Validation: 1,890 affected library tests passed (2 S3 skips), workspace all-target/all-feature Clippy with warnings denied, pinned formatting, layout doctests, and five complete Q23 row comparisons against V1. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
1f41995 to
7dcaf56
Compare
DuckDB extension scans default to Scan V2;
VORTEX_SCAN_V2=0restores V1. An explicit benchmark--scan-v2 true/falseoverrides the default and environment. Develop's per-suite orchestrator selection is preserved.The scan keeps exact
(file, row)pairs during late payload retrieval, schedules selected regions for sparse input, and gathers referenced dictionary values before conversion. Dense inputs retain vectorized execution. Mutable TopK bounds are excluded from dictionary predicate memoization; queued pruning and row filters refresh when the bound changes before projection.Integration with develop
Rebased onto
7baa376169, including develop's layout-aware DuckDB scheduler, allocator configuration, and Spark nested projection changes. The test-helper conflict is resolved using the shared counting source, retaining the sparse/dynamic regression tests. Layout boundaries stay on one worker so dictionary expressions are not duplicated across workers. Each worker reserves actual queued work under DuckDB's global lock and executes it outside the lock. This replaces the earlier estimated file/worker budgets and shared root-planner lock. Root chunked results retain their individual dictionary encodings before export.Sparse random-access retrieval bypasses broad layout parts. The optional 8K startup path and diagnostics remain available; startup defaults to zero. Wide full-row projections retain byte-based chunk sizing. File scans also preserve query column order when files store columns in a different order.
Native local performance
Measurements and completed benchmark comparisons below are pinned to
1f41995f5eagainst19dcd2f2. The final rebase to7dcaf5612cchanges Java/JNI code and Rust test helpers; the measured DuckDB and production shared scan paths are unchanged.release_debugwithRUSTFLAGS='-C target-cpu=native -C force-frame-pointers=yes'. Xeon 6975P-C, 16 physical / 32 logical cores. ClickBench has 99,997,497 rows across 100 files. Modes run sequentially on pinned cores, without connection reuse; foreign benchmark/compiler overlap is excluded.The focused comparison uses 48 warm samples per mode in reversed orders, excluding the first three iterations of each 27-iteration run:
Q23 p95 is 50.94 ms V1 / 39.02 ms V2; CV is 2.01% / 2.14%. First-query medians are 80.86 / 62.80 ms with the filesystem cache left intact. The earlier high-variance Q5/Q23 samples are superseded by these longer repeats.
Matching CI's files-per-worker ratio locally gives Q5 37.41 ms V1 / 38.35 ms V2 with 34 files / 32 workers (16 warm samples, CV 2.50% / 2.89%). The 17-file / 16-physical-core control is also nearly flat: 28.90 / 29.17 ms.
Full native screens use four warm samples per query:
Q1/Q7 remain slower. TPC-DS Q62 is 11.09 vs 9.18 ms in the short screen; Q43 is 14.26 vs 12.47 ms. The earlier large Q39 gaps do not appear in this screen: ClickBench 26.60 vs 27.07 ms; TPC-DS 24.66 vs 24.74 ms. This is not a universal-speedup claim. CI uses 94 physical ARM cores, so absolute local times are not directly comparable.
Q23 diagnostics
Separate debug-enabled native runs: first DuckDB batch about 2.02 ms; first observed TopK bound 2.94 ms. Around 221 dynamic pruning rechecks and 44 row-filter rechecks occur as the bound tightens. Payload retrieval plans seven chunks covering about twelve row coordinates and returns exactly ten selected rows from seven files.
Discovery takes about 20 ms and payload retrieval/export about 19 ms in those instrumented runs. Predicate, pruning, compute, and I/O-wait counters sum work across workers and must not be added as query wall time. Counts vary with scheduling. Logs confirm both learned orders: most chunks use EventTime first, while 7–21 chunks per warm query use URL first as their preferred order, plus 7–15 exploration chunks.
Validation and CI
EXCEPT ALLcomparisons and the exact Q5 count match V1 on the native build.The fresh main CI run and all seven jobs in the SQL benchmark run pass on
1f41995f5e.CI compares the PR against develop
19dcd2f2, rather than the local V1-mode baseline:The original Codspeed run lost its self-hosted runner during shard 7, confirmed by the check annotation. Its logs are unavailable, and retries later failed before creating jobs. No code failure was identified.
Final head
7dcaf5612cis pushed and mergeable against7baa376169. Fresh main CI and Codspeed runs have started, along with SQL benchmarks. These current-head checks are still in progress or queued at the last check. The PR remains a draft.