Skip to content

bug: incorrect ORC predicate pushdown with OR #2369

Description

@Flyangz

Describe the bug
ORC predicate pushdown produces wrong results for a WHERE clause of the form A OR B when one disjunct cannot be converted to an ORC Predicate.

In orc_exec.rs, collect_or_predicates silently drops any OR disjunct that fails to convert (convert_expr_to_orc returns None), then builds Predicate::or(...) from only the surviving disjuncts. This narrows the
pushed predicate into a subset of the real filter. Since the pushed predicate is used to skip row groups whose statistics prove no match, the narrowed predicate skips row groups that actually contain matching
rows → rows are lost, results are wrong.

A common trigger is comparing a string column with a numeric literal: b = 900000 is coerced to cast(b as double) = 900000.0 (a CastExpr, not convertible), while a sibling disjunct like id = 5 (Column = Literal) is
convertible. The mixed OR id = 5 OR b = 900000 then pushes down only id = 5, and every row group whose id statistics exclude 5 is skipped — dropping the row where b = '900000'.

To Reproduce

test("test OR pushdown with an unconvertible disjunct for orc table") {
    withTable("orc_or") {
      sql("create table orc_or(id int, b string) using orc")
      // enough rows so `id` statistics differ across row groups and pruning kicks in
      sql("insert into orc_or select cast(id as int), cast(id as string) from range(0, 1000000)")
      // `b = 900000` (string col vs int literal) -> cast(b as double)=2.0 -> not convertible
      // `id = 5`                            -> convertible
      // OR drops the b-branch -> pushes only `id = 5` -> row groups without id=5 are skipped,
      // losing the row where b='900000'
      checkSparkAnswerAndOperator("select * from orc_or where id = 5 or b = 900000")
    }
  }

Expected behavior
Auron returns the same rows as vanilla Spark (both the id = 5 row and the b = '900000' row). An OR predicate must be pushed down all-or-nothing: if any disjunct cannot be converted, the whole OR must not be pushed,
while the convertible AND conjuncts around it still push down safely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions