[FLINK-39142] Add Glue Schema Registry Protobuf SQL format factory (stacked on Avro #236) - #2
Merged
fmorillo7694 merged 5 commits intoAug 10, 2026
Conversation
fmorillo7694
force-pushed
the
gsr-protobuf-sql-format-stacked
branch
5 times, most recently
from
August 8, 2026 20:30
ed5a903 to
dda7aab
Compare
fmorillo7694
force-pushed
the
gsr-avro-sql-format
branch
from
August 9, 2026 10:14
df61bf9 to
5a4edd5
Compare
fmorillo7694
force-pushed
the
gsr-protobuf-sql-format-stacked
branch
4 times, most recently
from
August 10, 2026 10:12
225d444 to
fc0d637
Compare
Add the `protobuf-glue` Flink SQL format that serializes/deserializes RowData through AWS Glue Schema Registry, stacked on the Avro SQL format (PR apache#236). Mirrors the Avro module layout: format factory + options, RowType<->Protobuf converters, schema converter, RowData ser/de schemas, SQL uber-jar module, docs, and E2E test module. Applies the following review findings on top of the candidate implementation: - B4: encode/decode TIMESTAMP and TIMESTAMP_LTZ as epoch-millis int64, DATE/TIME as int32, and DECIMAL as its lossless BigDecimal text form, using the correct RowData accessors instead of getString() (which previously threw ClassCastException for these very common types). - C3: fail fast for genuinely unsupported complex types (ARRAY, MAP, MULTISET, ROW, RAW) instead of silently coercing them to `string`. - C4: sanitize column names to valid proto field identifiers and carry the original SQL name as the field's json_name, avoiding DescriptorValidationException at open() for names with spaces, hyphens, or a leading digit. - V2: pin protobuf-java and protobuf-java-util to a single <protobuf.version> property (3.25.x) to remove the version-skew risk between the two artifacts. Adds ProtobufTypeCoverageTest covering the temporal/decimal round-trips, field-name sanitization + json_name preservation, and the fail-fast path. Module builds green: 15 tests pass (mvn test, JDK 17), spotless clean. Known follow-ups (tracked in the GSR review, deferred): the SQL decode path still strips the fixed GSR header and rebuilds the reader descriptor from the local RowType rather than resolving the writer schema through the GSR deserialization facade (finding B1), so compression is not yet symmetric on read (C1) and proto3 implicit presence keeps null<->default round-trips lossy (C2).
fmorillo7694
force-pushed
the
gsr-protobuf-sql-format-stacked
branch
from
August 10, 2026 10:26
fc0d637 to
afaa483
Compare
fmorillo7694
added a commit
that referenced
this pull request
Aug 10, 2026
feat(glue-schema-registry): Add JSON SQL format factory (stacked on Protobuf #2)
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[FLINK-39142] Add Glue Schema Registry Protobuf SQL format factory
Stacked PR — this builds on the Avro GSR SQL format work in
apache#236 (apache#236).
gsr-avro-sql-format(the Avro PR [FLINK-24379] Add Glue Schema Registry Avro SQL format factory apache/flink-connector-aws#236 head branch)gsr-protobuf-sql-format-stackedReviewing only the top commit (
[FLINK-39142]) gives a clean, Protobuf-only diff(21 files). The Avro commit below it is the stacking base and lands via apache#236.
What this adds
A Glue Schema Registry Protobuf SQL format for Flink SQL, mirroring the Avro
format's structure:
flink-formats-aws/flink-protobuf-glue-schema-registry/flink-formats-aws/flink-sql-protobuf-glue-schema-registry/flink-connector-aws-e2e-tests/flink-formats-protobuf-glue-schema-registry-e2e-tests/docs/content/docs/connectors/table/formats/protobuf-glue.md,docs/data/protobuf-glue.ymlflink-formats-aws/pom.xml(+2 modules),flink-connector-aws-e2e-tests/pom.xml(+1)Review findings addressed (from the Schema Registry deep-dive review)
int64, DATE/TIME →int32,DECIMAL → lossless
BigDecimaltext; encode/decode via the correctRowDataaccessors (fixes the
getString()ClassCastException).(
UnsupportedOperationException) instead of silently coercing tostring.is preserved as the field's
json_name.protobuf-java+protobuf-java-utilpinned to a single<protobuf.version>(3.25.5).Review blockers resolved in this revision (2026-08-09)
RowDataby field name (FacadeGsrProtobufReader/GsrProtobufReader). Unknown fields -> null for nullable columns, error for NOT NULL.optional) for NULLABLE columns; null <-> default round-trip regression test added.glue.schema.registry.versionbumped 1.1.18 -> 1.1.25 (in the base Avro commit).Verification
mvn -pl flink-formats-aws/flink-protobuf-glue-schema-registry -am test(JDK 17) →BUILD SUCCESS, 15 tests, 0 failures, 0 errors
(
GlueSchemaRegistryProtobufFormatFactoryTest,ProtobufTypeCoverageTest,GsrProtobufRoundTripPropertyTest,ProtobufRoundTripIntegrationTest).mvn spotless:checkon the module → clean.package -DskipTests→ BUILD SUCCESS.End-to-end SQL test (real Glue Schema Registry)
New
GlueSchemaRegistryProtobufSqlKinesisITCaseconverts the manual driver to the repo's credential-gated e2e convention (Localstack Kinesis + real AWS GSR,@Tag("requires-aws-credentials"), skips withoutIT_CASE_GLUE_SCHEMA_*creds). Scenarios: multi-row round-trip with auto-registration, nullable columns with real NULLs (exercises the proto3 explicit-presence fix), and ZLIB compression round-trip (exercises the read-path decompression fix).