Skip to content

More Polars plan optimizations for TPC-DS#22395

Draft
Matt711 wants to merge 2 commits intorapidsai:mainfrom
Matt711:imp/pdsds/more-pds-optimizations
Draft

More Polars plan optimizations for TPC-DS#22395
Matt711 wants to merge 2 commits intorapidsai:mainfrom
Matt711:imp/pdsds/more-pds-optimizations

Conversation

@Matt711
Copy link
Copy Markdown
Contributor

@Matt711 Matt711 commented May 6, 2026

Description

Hand-tune polars_impl for 19 TPC-DS benchmark queries in python/cudf_polars/cudf_polars/experimental/benchmarks/pdsds_queries/. Each rewrite preserves query semantics and only changes how the polars LazyFrame is constructed; duckdb_impl is unchanged.

The optimizations apply a small set of recurring patterns that the polars optimizer does not (yet) perform automatically:

  • Predicate pushdown on dimension tables — pre-filter date_dim, item, store, etc. by literal predicates (year, quarter, month window, category/class/brand) before any join, so the join builds smaller hash tables.
  • Semi-join fact-table pre-filtering — use selective dimension keys (and in some cases store_returns (customer, item) pairs) as semi-join probes against the fact tables, shrinking them before the expensive joins.
  • Projection pushdownselect(...) only the columns each table contributes before joining, instead of relying on the planner to prune them later.
  • Condition-join → equi-join — replace cross-join + filter and CONDITIONALJOIN-style patterns with constant-key equi-joins where the predicate is equivalent.
  • Single-pass bucket aggregation — collapse multiple independent global-sum group-bys over the same fact table into one pass that emits the values in a single aggregation, replacing N scans with 1.
  • Join reordering — defer non-selective joins (e.g. customer) until after the selective filter chain so the row count entering the deferred join is much smaller.

Test plan

  • Run TPC-DS validation against DuckDB on the 19 modified queries
  • Run benchmark sweep and confirm no regressions vs. main on unmodified queries
  • Confirm result equality (sorted output) matches DuckDB reference

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@Matt711 Matt711 added Performance Performance related issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels May 6, 2026
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 6, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added Python Affects Python cuDF API. cudf-polars Issues specific to cudf-polars labels May 6, 2026
@GPUtester GPUtester moved this to In Progress in cuDF Python May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cudf-polars Issues specific to cudf-polars improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Performance Performance related issue Python Affects Python cuDF API.

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants