Describe the bug
Under ANSI mode (spark.sql.ansi.enabled=true, the default in Spark 4.x), the native engine does not honor Spark's arithmetic-overflow semantics. Integer/long negation that overflows wraps around in the native engine, whereas vanilla Spark throws SparkArithmeticException: [ARITHMETIC_OVERFLOW].
How to reproduce
On Spark 4.x (ANSI on by default):
CREATE TABLE t1(col1 int) USING parquet;
INSERT INTO t1 VALUES (-2147483648);
SELECT negative(col1), -(col1) FROM t1;
- With Auron disabled: throws
[ARITHMETIC_OVERFLOW].
- With Auron enabled: returns
-2147483648 (no error).
This surfaced via the AuronExpressionSuite."UnaryMinus" test once the config alt-key fix (#2360) made checkSparkAnswerAndOperator's spark.auron.enable=false baseline actually run vanilla Spark. As a stopgap, that test now runs with spark.sql.ansi.enabled=false so both engines wrap consistently (see PR #2361); this issue tracks the underlying native ANSI-overflow gap.
Suggested fix
Make the native engine honor ANSI overflow semantics when spark.sql.ansi.enabled=true
Describe the bug
Under ANSI mode (
spark.sql.ansi.enabled=true, the default in Spark 4.x), the native engine does not honor Spark's arithmetic-overflow semantics. Integer/long negation that overflows wraps around in the native engine, whereas vanilla Spark throwsSparkArithmeticException: [ARITHMETIC_OVERFLOW].How to reproduce
On Spark 4.x (ANSI on by default):
[ARITHMETIC_OVERFLOW].-2147483648(no error).This surfaced via the
AuronExpressionSuite."UnaryMinus"test once the config alt-key fix (#2360) madecheckSparkAnswerAndOperator'sspark.auron.enable=falsebaseline actually run vanilla Spark. As a stopgap, that test now runs withspark.sql.ansi.enabled=falseso both engines wrap consistently (see PR #2361); this issue tracks the underlying native ANSI-overflow gap.Suggested fix
Make the native engine honor ANSI overflow semantics when
spark.sql.ansi.enabled=true