Skip to content

Native arithmetic does not honor ANSI overflow semantics (e.g. negate of Int.MinValue) #2368

Description

@robreeves

Describe the bug

Under ANSI mode (spark.sql.ansi.enabled=true, the default in Spark 4.x), the native engine does not honor Spark's arithmetic-overflow semantics. Integer/long negation that overflows wraps around in the native engine, whereas vanilla Spark throws SparkArithmeticException: [ARITHMETIC_OVERFLOW].

How to reproduce

On Spark 4.x (ANSI on by default):

CREATE TABLE t1(col1 int) USING parquet;
INSERT INTO t1 VALUES (-2147483648);
SELECT negative(col1), -(col1) FROM t1;
  • With Auron disabled: throws [ARITHMETIC_OVERFLOW].
  • With Auron enabled: returns -2147483648 (no error).

This surfaced via the AuronExpressionSuite."UnaryMinus" test once the config alt-key fix (#2360) made checkSparkAnswerAndOperator's spark.auron.enable=false baseline actually run vanilla Spark. As a stopgap, that test now runs with spark.sql.ansi.enabled=false so both engines wrap consistently (see PR #2361); this issue tracks the underlying native ANSI-overflow gap.

Suggested fix

Make the native engine honor ANSI overflow semantics when spark.sql.ansi.enabled=true

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions