Skip to content

feat: support input changelog for primary key writes#318

Open
QuakeWang wants to merge 3 commits into
apache:mainfrom
QuakeWang:feat/input-changelog
Open

feat: support input changelog for primary key writes#318
QuakeWang wants to merge 3 commits into
apache:mainfrom
QuakeWang:feat/input-changelog

Conversation

@QuakeWang
Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #255

This PR supports changelog-producer=input for primary-key table writes in the scoped Rust write path.

The implementation double-writes primary-key input rows into changelog files while keeping normal table data files deduplicated, then commits changelog files through separate changelog manifest metadata.

Brief change log

  • Add typed parsing for changelog-producer and changelog file options in CoreOptions.
  • Add PreparedFiles and propagate changelog files separately from normal data files.
  • Teach KeyValueFileWriter to write input changelog files from the full sorted input rows, while normal data files still use merge-engine selected rows.
  • Compute changelog DataFileMeta from changelog rows, including row count, key stats, sequence range, and retract row count.
  • Add CommitMessage::new_changelog_files and commit changelog entries into a separate changelog manifest list.
  • Populate snapshot changelogManifestList and changelogRecordCount.
  • Reject unsupported combinations early, including non-PK tables, non-deduplicate merge engines, cross-partition dynamic bucket, bucket=-2, rowkind.field, unsupported changelog producers, and overwrite with changelog files.
  • Preserve existing normal data manifest, index manifest, and table record count behavior.

Tests

  • cargo fmt --all -- --check
  • cargo clippy -p paimon --all-targets -- -D warnings
  • cargo test -p paimon changelog
  • cargo test -p paimon

API and Format

Documentation

QuakeWang added 3 commits May 14, 2026 13:14
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

# Conflicts:
#	crates/paimon/src/table/kv_file_writer.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support changelog-producer is input

1 participant