Repository navigation
Speed up DuckDB V2 scans of wide and layout-chunked files - #11060
Open
joseph-isaacs wants to merge 1 commit into
Open
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
joseph-isaacs
added
changelog/performance
This was referenced Oct 30, 2026
joseph-isaacs
added
action/bench-sql
- ExpressionPackRule expanded the struct root into a pack of every field before partitioning, on every application. A field access of the root is already in expanded form, so the pack is now built only for an expression that uses the bare root. On ClickBench's ~105-column files this was most of a scan's planning time. - ScanBuilder::with_max_batch_rows caps the rows of a projection batch, 65,536 by default as before. DuckDB reads batches in its own vectors, so its driver lifts the cap and exports each layout split as one batch, as V1 does, instead of re-slicing and decoding the split's segments in two. - ChunkPlanner::single plans an engine's own unit of work as one chunk. The DuckDB driver uses it per part, so a part is not cut again on the chunk grid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs
removed
the
action/bench-sql
joseph-isaacs
force-pushed
the
ji/duckdb-scan-v2-clickbench-perf
branch
from
October 30, 2026 19:00
69e3829 to
7d9bfda
Compare
joseph-isaacs
added
the
action/bench-sql
github-actions
Bot
removed
the
action/bench-sql
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.967x ➖, 3↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
|
Merging this PR will degrade performance by 23.27%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
|
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.012x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.015x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.985x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.970x ➖, 2↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.073x ➖, 0↑ 3↓)
datafusion / parquet / ns (0.982x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.056x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.084x ➖, 0↑ 0↓)
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #11047. These changes bring ClickBench on the DuckDB V2 scan to a geomean of 0.979× V1 locally, down from 0.999× on develop.
Changes
ExpressionPackRule. The rule rewrote the struct root into a pack of every field each time it was applied, even when the expression only reads$.field, which is already in expanded form. On ClickBench's files of about 105 columns, this was most of each file's planning time. The pack is now built only for expressions that use the bare root. This also benefits DataFusion's V2 scans.ScanBuilder::with_max_batch_rows.ProjectionPlannercut dense selections into 65,536-row batches to keep Arrow conversion copies small. DuckDB reads each batch in its own 2,048-row vectors, so its driver lifts the cap. A layout split (87,382 rows on ClickBench) is then exported as one batch, as in V1, instead of being split into 65,536 + 21,846 rows that each slice and decode the same segments. The default stays at 65,536.ChunkPlanner::single. Plans an engine's own unit of work as one chunk. The DuckDB driver uses it for each part, so parts are no longer cut again on the chunk grid.Results
Results are from a local 32-core x86 machine, using CI's flags (
-C target-cpu=native -C force-frame-pointers=yes) and therelease_debugprofile. They're the best of 2 rounds of 5 iterations each, with V2 timings divided by V1 timings.In ClickBench Q02, single-threaded, V2 now hands DuckDB 1,100 batches, as V1 does, instead of 2,200, and runs in 103ms instead of 130ms (V1: 88ms).
Correctness was compared row by row with Parquet and V1:
LIMIT, which also differ between V1 runs.Still slower than V1
ClickBench Q07 (1.14×) and Q01 (1.09×) both filter on
AdvEngineID <> 0. Their remaining cost is spread across per-part executor work rather than concentrated in one hotspot.Checks
cargo nextest run -p vortex-layout -p vortex-duckdb: 1,008 passed, including the newsingle_chunk_batches_follow_max_batch_rowscases.cargo clippy -p vortex-layout -p vortex-duckdb --all-targets --all-features -- -D warnings: clean.fmtwith the pinned nightly.vortex-datafusiontests locally. The rule change preserves behaviour and is covered by thevortex-layoutplan tests.🤖 Generated with Claude Code
https://claude.ai/code/session_01MZ3d6HFyGGeUGy8brEr6BM