[FLINK-39810][Connectors/kinesis] Support upsert changelog streams in Kinesis SQL connector - #249
Open
fmorillo7694 wants to merge 2 commits into
Open
Conversation
gguptp
approved these changes
Aug 8, 2026
fmorillo7694
force-pushed
the
feature/kinesis-upsert-support
branch
from
August 10, 2026 09:14
b214425 to
abeb22a
Compare
… Kinesis SQL connector When a PRIMARY KEY is defined on the table, the Kinesis sink accepts upsert changelog streams (INSERT and UPDATE_AFTER), uses the primary key fields as the Kinesis partition key, and restricts the writer to a single in-flight request to preserve per-key ordering. DELETE events are intentionally not supported: Kinesis records have no key/value separation, so a delete cannot carry the deleted key in a format-agnostic way. Delete-producing queries (e.g. CDC sources, Top-N) are rejected at planning time via the declared ChangelogMode. Options incompatible with upsert mode fail validation: sink.partitioner, sink.requests.max-inflight != 1, and PARTITIONED BY clauses. Includes a SQL-path integration test against Localstack Kinesis verifying the end-to-end upsert flow (GROUP BY changelog written with primary-key-derived partition keys, latest value per key materializable from the stream) and the plan-time rejection of delete-producing queries.
fmorillo7694
force-pushed
the
feature/kinesis-upsert-support
branch
from
August 10, 2026 11:52
abeb22a to
b1898c8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of the change
Adds upsert changelog stream support to the Kinesis SQL connector. When a
PRIMARY KEYis defined on the table, the sink accepts the results of aggregations (GROUP BY), deduplication, and streaming joins over append-only inputs, writing the latest row per key to Kinesis Data Streams.Design decisions
kinesisconnector, not a separateupsert-kinesisconnector.upsert-kafkais a separate connector because compacted topics give it distinct semantics on both sides: the source reads a compacted topic as a changelog (null value = DELETE tombstone), and the sink writes tombstones that log compaction actually applies. Kinesis has neither compaction nor key-based retention, so there is no materialized latest-per-key view for an upsert source to read — the capability is inherently sink-only. A separate connector identifier would duplicate every source/sink option for the sake of one sink behavior; instead, the presence of aPRIMARY KEYactivates upsert mode on the existing factory (the same signalupsert-kafkarequires), and everything incompatible with it fails validation explicitly.{INSERT, UPDATE_AFTER}— DELETE is intentionally not supported. Kinesis records have no key/value separation, so a delete event cannot carry the deleted key in a format-agnostic way (an empty-payload tombstone is unusable by consumers and breaks JSON deserialization). Delete-producing queries (CDC sources, Top-N) are rejected at planning time with a descriptive planner error instead. Asink.delete-strategyoption can be added later if a sound tombstone contract is found.sink.requests.max-inflight = 1so request batches cannot overtake each other; an explicit conflicting value fails validation. Residual limitation (documented): the KinesisPutRecordsAPI does not guarantee ordering within one request, so two updates for the same key in a single batch may be sequenced out of order. A key-deduplicating flush buffer (analogous to upsert-kafka'ssink.buffer-flush.*) is a natural follow-up that would close this and restore throughput.sink.partitioner, aPARTITIONED BYclause, orsink.requests.max-inflight != 1all fail with aValidationExceptionrather than being silently overridden.Changes
upsertModeflag; restrictedChangelogMode;UpsertSerializationSchemaWrappernormalizesUPDATE_AFTERtoINSERTfor insert-only formats (RowKind restored via try/finally; DELETE/UPDATE_BEFORE rejected defensively); constructor precondition onmaxInFlightRequests.Verifying this change
This change added tests and can be verified as follows:
UpsertSerializationSchemaWrapper(INSERT pass-through, UPDATE_AFTER normalization + RowKind restore, restore-on-exception, DELETE/UPDATE_BEFORE rejection)sink.partitioner/sink.requests.max-inflight != 1/PARTITIONED BYeach fail validationKinesisUpsertTableSinkITCase, Localstack Kinesis): aGROUP BYupsert changelog is written end-to-end with primary-key-derived partition keys and the latest value per key is materializable from the stream; a delete-producing Top-N query fails at planning time withdoesn't support consuming delete changes(also empirically confirming thatGROUP BYover insert-only input is not inferred to produce DELETE)flink-connector-aws-kinesis-streamsmodule suite passes, spotless, checkstyle and dependency-convergence cleanSignificant changes
@Public(Evolving))