Share SQL/JSON across Flink lines and preserve legacy ENCODE bytes - #259
Merged
Merged
Conversation
added 2 commits
September 26, 2026 13:12
Flink 1.18 SQL declares BINARY(1) for ENCODE while returning variable-length byte arrays. Reuse the shared batch evaluator and carry affected scalar results as Arrow Binary internally, keeping the original expressions and public schema intact. Keep explicit fallbacks where fixed-width metadata would cross another operator or a native sink. A cast to BYTES establishes the ordinary variable-width boundary. Document the admitted forms and retained limits. Validated 31 ENCODE/DECODE/charset and boundary cases on Flink 1.18.1, the 22 shared charset cases on 2.2.1, and 45 shared binary-UDF/lifecycle regressions on each line. This adds coverage through the existing JVM batch bridge; no throughput improvement is claimed.
Flink 1.18 uses Jackson 2.14.2 with Double JSON values and no newer parsing limits, while 2.2 uses Jackson 2.18.2 with BigDecimal values. Select those semantics through small verified runtime adapters while sharing parsing, SIMD selection, path handling, and buffer ownership. Reuse the existing JDK 17 double formatter and grow the legacy parser stack for deep nesting. Keep unverified runtimes on the batch host evaluator. Move recycler-history and double-bit parity fixtures into the common suite and document the supported profiles. Validation: 735 distinct targeted cases on Flink 1.18.1 (714 passed, 21 version-specific skips), 727 passed on Flink 2.2.1, and 106 native function tests passed. Direct JNI checks cover randomized doubles, overflow, signed zero, extreme exponents, buffer history, and 4096-level nesting. These are correctness and coverage checks with a debug native build, not throughput measurements.
jordepic
changed the base branch from
feat/flink118-udf-lookup-int96
to
main
September 26, 2026 20:26
jordepic
marked this pull request as ready for review
September 26, 2026 20:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flink 1.18 SQL declares
ENCODEasBINARY(1)while returning variable-length bytes. The existing batch evaluator now preserves the full values using Arrow Binary internally, without changing the public schema or Flink expressions. Final projections and fused consumers are admitted; incompatible operator/native-sink boundaries retain explicit fallback. Casting toBYTESestablishes an ordinary variable-width boundary.SQL/JSON now uses one native reader across 1.18 and 2.2.1. Small compatibility adapters select the released Jackson recycler contract, floating-number behavior and parsing limits. The 1.18 profile requires Jackson 2.14.2/JDK 17 and preserves Double formatting, signed zero, overflow and unrestricted nesting; 2.2 retains its Jackson 2.18.2/BigDecimal behavior. Buffer-history and double-bit JNI tests are shared between profiles. Unsupported runtimes retain the host evaluator.
Validation uses released Flink 1.18.1 and 2.2.1 on JDK 17 with the debug native build:
Coverage and fallback documentation is updated. No throughput claim or full upstream inventory refresh is included. This PR is stacked on #258; Delta remains outside its scope.