Close Flink 1.18 legacy UDF, lookup and INT96 coverage gaps - #258
Merged
Merged
Conversation
added 4 commits
September 26, 2026 11:55
Resolve legacy and modern scalar definitions through the host-version adapter so deprecated registrations retain the existing columnar evaluator, constructor state, lifecycle and job parameters. Preserve specialization and shared mutable-call guards instead of creating a separate legacy evaluator. Add legacy registration parity and fallback tests plus native execution contracts for the unchanged upstream rich-function cases. Focused UDF regressions pass on Flink 1.18 and 2.2. This extends coverage within native islands; no standalone throughput improvement is claimed.
Use the released host provider resolution and generated runners for legacy lookup sources, keeping the same Arrow probe and result boundaries as modern sources. Preserve synchronous and asynchronous lifecycle, constant keys, residual filtering and retry semantics without introducing a second lookup runtime. Require native row-processing evidence for legacy upstream lookup contracts. New sync/async fixtures cover duplicate matches, misses, nulls, dimension filters and lifecycle; existing modern lookup regressions pass on both supported host lines. This is coverage parity, without a new throughput claim.
Keep standard Arrow column encoders for ordinary leaves and use typed INT96 encoders for timestamp leaves within the same row group. Preserve nested definition and repetition levels and account for buffered timestamp values in row-group sizing. Flink continues to own filesystem writes and checkpoint commits. UTC encoding preserves the host millisecond/fraction arithmetic without narrowing to epoch nanoseconds. Local INT96 uses one JVM callback per timestamp column to preserve the host calendar, default timezone and DST behavior. Local-time INT64 remains a planner fallback. Validation: 26 native Parquet unit tests; eight physical-file timestamp comparisons on each host line covering extreme dates, local zones, nested null/empty collections, slices and multiple row groups; partitioned SQL sink parity and admission tests. Combined focused Java regressions: 129 passed and 2 skipped on 1.18, all 124 passed on a clean 2.2 build. Coverage extension only; no release benchmark or throughput claim.
The unchanged 1.18 timestamp fixture supplies an unused time-unit value in INT96 mode. Flink ignores it, but native admission still validated it, leaving a silent fallback after INT96 encoding was added. Validate units only for INT64 and pin ignored-unit behavior through host/native SQL sink parity. Validation: all 30 focused planner and SQL sink tests pass on 2.2. All eight unchanged upstream 1.18 Parquet cases pass; all four timestamp sink plans are admitted without fallback and the timestamp class creates a native writer. Also record the successful 262-case upstream UDF/lookup run and its 26 native execution contracts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flink 1.18 now routes deprecated scalar registrations and legacy sync/async lookup sources through the existing native operators, and Parquet sinks support the host’s default INT96 timestamps. This closes the three selected 1.18 parity gaps while preserving the existing UDF specialization, changelog and local-time INT64 fallbacks.
Stacked on #257 (
fix/flink118-stock-rocksdb); the diff excludes its backend substitution work.Validation:
These are scoped validations, not a refreshed full-suite acceleration count. No release benchmark or throughput improvement is claimed. Shared coverage gaps and Delta are outside this change.