Skip to content

Native window aggregation: support Flink split-distinct physical plans #166

Description

@liuyongvs

Follow-up to the completed base window COUNT(DISTINCT) support in #97. On main aea6056352c402e0c7008f6f223c7574324d796a, enable table.optimizer.distinct-agg.split.enabled=true and execute this shape over a runtime source with event-time watermarks:

SELECT k, window_start, window_end, COUNT(DISTINCT s), SUM(v), COUNT(*)
FROM TABLE(TUMBLE(TABLE src, DESCRIPTOR(rt), INTERVAL '10' SECOND))
GROUP BY k, window_start, window_end;

TUMBLE, HOP and CUMULATE comparisons all match released Flink 2.2.1 through full fallback. Even with TWO_PHASE requested, every generated plan records:

Calc: unsupported function/operator: HASH_CODE
window aggregate: attached-window aggregation requires two-phase execution
global window aggregate: requires SUM/MIN/MAX/COUNT or paired AVG partials

The corresponding splitDistinct=false paths run native. The unchanged upstream WindowDistinctAggregateITCase contracts currently require fallback for all three splitDistinct=true methods.

Acceptance:

  • Support the actual optimized split-distinct stages and their attached/global accumulator layouts, in addition to the separately tracked HASH_CODE scalar.
  • Preserve NULLs, duplicates across buckets/partials, mixed ordinary aggregates, temporal bounds, watermarks and late rows.
  • Verify state cleanup and memory/RocksDB recovery for each admitted layout.
  • Require native work in every necessary stage of the three unchanged upstream methods and their backend variants.
  • Retain precise fallback for any unverified form and update the window coverage page.

Retracting DISTINCT remains tracked by #99; this issue does not duplicate that changelog extension.

HASH_CODE scalar tracking: #165.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions