Background
Auron currently accelerates first / first(ignoreNulls) aggregates natively (AggFirst / AggFirstIgnoresNull), but last / last(ignoreNulls) are not implemented natively. As a result they fall back to the generic UDAF path (a JNI call back into the JVM), losing vectorized acceleration.
Proposal
Add native last aggregate support, mirroring the existing first implementation:
- Implement
AggLast and AggLastIgnoresNull in datafusion-ext-plans with the same columnar accumulator layout as first, but with "later value wins" semantics (every visited row overwrites; on merge the later partial state wins).
- Wire the two new functions through the
AggFunction enum, create_agg, the protobuf contract (LAST / LAST_IGNORES_NULL), the protobuf::AggFunction -> AggFunction conversion, and the window-aggregate mapping.
- Add the
Last expression conversion in NativeConverters.convertAggregateExpr (dispatching to LAST / LAST_IGNORES_NULL by ignoreNulls).
- Declare the native aggregate buffer schema for
Last in NativeAggBase.computeNativeAggBufferDataTypes ([dataType] for ignoreNulls, [dataType, Boolean] otherwise) so the partial -> shuffle -> final buffer schema matches the native side.
Scope / Non-goals
- Covers
last(col) and last(col, ignoreNulls = true) over primitive / boolean / string / binary / complex (scalar-value) types, consistent with first.
- Window aggregate (
last over a window) reuses the same AggFunction.
- Semantics follow Spark:
last is order-sensitive and non-deterministic without an explicit ordering (same as first).
Tests
- Rust unit test
agg_exec::test::test_agg_last: partial -> final two-phase aggregation over a nullable column, verifying last (keeps the last visited row including null) and last(ignoreNulls) (keeps the last non-null value).
- Scala end-to-end test in
AuronDataFrameAggregateSuite ("native last / last(ignoreNulls) aggregate", spark34 + spark35): a grouped aggregate exercising the full partial -> shuffle -> final native path, asserting correct values and that the plan offloads to NativeAggBase.
Background
Auron currently accelerates
first/first(ignoreNulls)aggregates natively (AggFirst/AggFirstIgnoresNull), butlast/last(ignoreNulls)are not implemented natively. As a result they fall back to the generic UDAF path (a JNI call back into the JVM), losing vectorized acceleration.Proposal
Add native
lastaggregate support, mirroring the existingfirstimplementation:AggLastandAggLastIgnoresNullindatafusion-ext-planswith the same columnar accumulator layout asfirst, but with "later value wins" semantics (every visited row overwrites; on merge the later partial state wins).AggFunctionenum,create_agg, the protobuf contract (LAST/LAST_IGNORES_NULL), theprotobuf::AggFunction -> AggFunctionconversion, and the window-aggregate mapping.Lastexpression conversion inNativeConverters.convertAggregateExpr(dispatching toLAST/LAST_IGNORES_NULLbyignoreNulls).LastinNativeAggBase.computeNativeAggBufferDataTypes([dataType]for ignoreNulls,[dataType, Boolean]otherwise) so the partial -> shuffle -> final buffer schema matches the native side.Scope / Non-goals
last(col)andlast(col, ignoreNulls = true)over primitive / boolean / string / binary / complex (scalar-value) types, consistent withfirst.lastover a window) reuses the sameAggFunction.lastis order-sensitive and non-deterministic without an explicit ordering (same asfirst).Tests
agg_exec::test::test_agg_last: partial -> final two-phase aggregation over a nullable column, verifyinglast(keeps the last visited row including null) andlast(ignoreNulls)(keeps the last non-null value).AuronDataFrameAggregateSuite("native last / last(ignoreNulls) aggregate", spark34 + spark35): a grouped aggregate exercising the full partial -> shuffle -> final native path, asserting correct values and that the plan offloads toNativeAggBase.