Repository navigation
DuckDB on Vortex fails the TPC-DS nightly since the tables were split into files #10933
Description
Since #10519 split the TPC-H and TPC-DS tables into files of about 200 MB, the nightly TPC-DS benchmark fails for DuckDB on Vortex. bench.vortex.dev has no TPC-DS SF=10 results after 2026-10-21, and no SF=100 results for DuckDB on Vortex.
| Nightly job | 2026-10-22, one file per table | Every night since 2026-10-23 |
|---|---|---|
tpcds-nvme-10, tpcds-s3-10 |
Pass (13 min, 43 min) | Fail: query 64 on DuckDB with Vortex runs out of memory |
tpcds-nvme-100-duckdb-vortex |
Pass (12 min) | Cancelled at the 2-hour timeout |
tpcds-s3-100-duckdb-vortex |
Pass (64 min) | Cancelled at the 2-hour timeout |
DataFusion on Vortex and DuckDB on Parquet still pass at SF=100. The DuckDB views now read web_sales_[0-9][0-9][0-9][0-9].vortex, not web_sales.vortex. The error in the SF=10 jobs:
query 64 failed: Other error: Failed to execute query: Out of Memory Error: could not allocate block of size 256.0 KiB (150.6 GiB/150.6 GiB used)
The benchmark stops at the first failed query, so the SF=10 jobs record no results for any engine.
Last good run: https://github.com/vortex-data/vortex/actions/runs/39700968343
First failed run: https://github.com/vortex-data/vortex/actions/runs/39879486102
Possible fixes: keep one file per table for the DuckDB with Vortex runs, to restore the data quickly; find why the multi-file Vortex scan in DuckDB uses this much memory and time; or let one failed query skip, so that the job still records the other results.
🤖 Generated with Claude Code
Activity
robert3005 commentedon Oct 28, 2026
The root cause is
Two Vortex behaviours seem to cause this:
So DuckDB thinks the store_sales side is about 25× larger than it is. It then pre-aggregates both halves and joins them late, which forces the cross product. With one file per table, DuckDB also had the Vortex min/max statistics, and they kept the estimates low (1.15 M on that side). DuckDB drops those statistics when a scan reads more than one file. SF=3 passes (54 ms), probably because the estimates stay below the point where the plan flips; I haven't checked its plan.
myrrc commentedon Oct 29, 2026
This is an upstream bug. We need to merge it and then report cardinality without any 0.2 multiplication.