Skip to content

Support native last / last(ignoreNulls) aggregate #2358

Description

@zhuxiangyi

Background

Auron currently accelerates first / first(ignoreNulls) aggregates natively (AggFirst / AggFirstIgnoresNull), but last / last(ignoreNulls) are not implemented natively. As a result they fall back to the generic UDAF path (a JNI call back into the JVM), losing vectorized acceleration.

Proposal

Add native last aggregate support, mirroring the existing first implementation:

  1. Implement AggLast and AggLastIgnoresNull in datafusion-ext-plans with the same columnar accumulator layout as first, but with "later value wins" semantics (every visited row overwrites; on merge the later partial state wins).
  2. Wire the two new functions through the AggFunction enum, create_agg, the protobuf contract (LAST / LAST_IGNORES_NULL), the protobuf::AggFunction -> AggFunction conversion, and the window-aggregate mapping.
  3. Add the Last expression conversion in NativeConverters.convertAggregateExpr (dispatching to LAST / LAST_IGNORES_NULL by ignoreNulls).
  4. Declare the native aggregate buffer schema for Last in NativeAggBase.computeNativeAggBufferDataTypes ([dataType] for ignoreNulls, [dataType, Boolean] otherwise) so the partial -> shuffle -> final buffer schema matches the native side.

Scope / Non-goals

  • Covers last(col) and last(col, ignoreNulls = true) over primitive / boolean / string / binary / complex (scalar-value) types, consistent with first.
  • Window aggregate (last over a window) reuses the same AggFunction.
  • Semantics follow Spark: last is order-sensitive and non-deterministic without an explicit ordering (same as first).

Tests

  • Rust unit test agg_exec::test::test_agg_last: partial -> final two-phase aggregation over a nullable column, verifying last (keeps the last visited row including null) and last(ignoreNulls) (keeps the last non-null value).
  • Scala end-to-end test in AuronDataFrameAggregateSuite ("native last / last(ignoreNulls) aggregate", spark34 + spark35): a grouped aggregate exercising the full partial -> shuffle -> final native path, asserting correct values and that the plan offloads to NativeAggBase.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions