Released Paimon 2.0.0 has a bundle write entry for append tables only; its merge-tree writer takes one row at a time and creates level-0 files privately. The native primary-key sink therefore writes level-0 files itself and hands them to Paimon's writer through the entry the dedicated compaction operator uses for files written elsewhere (divergences/32-paimon-pk-l0-through-the-compactor-hook.md). Two costs follow:
- Paimon's writer sees files, not rows, so it can look idle between checkpoints; Paimon then closes it and the next hand-off recreates it with a scan of the bucket's committed files (the same scan its dedicated compactor pays).
- The level-0
DataFileMeta (key bounds, statistics, sequence range, delete count) is assembled on our side from Paimon's public factories rather than by Paimon's own key-value file writer.
Propose upstream a bundle entry on the merge-tree writer that accepts a sorted, key-merged batch for one bucket in Paimon's key-value layout with the caller's sequence range, and creates the level-0 file and its metadata itself. With that entry the compaction hand-off, the idle-writer rescan, and the metadata assembly disappear from StreamFusion; the native sort/merge and Parquet encoding stay. Track the upstream discussion and the Paimon release that ships it here, then switch the sink over and retire divergence 32.
Split from #33.
Released Paimon 2.0.0 has a bundle write entry for append tables only; its merge-tree writer takes one row at a time and creates level-0 files privately. The native primary-key sink therefore writes level-0 files itself and hands them to Paimon's writer through the entry the dedicated compaction operator uses for files written elsewhere (
divergences/32-paimon-pk-l0-through-the-compactor-hook.md). Two costs follow:DataFileMeta(key bounds, statistics, sequence range, delete count) is assembled on our side from Paimon's public factories rather than by Paimon's own key-value file writer.Propose upstream a bundle entry on the merge-tree writer that accepts a sorted, key-merged batch for one bucket in Paimon's key-value layout with the caller's sequence range, and creates the level-0 file and its metadata itself. With that entry the compaction hand-off, the idle-writer rescan, and the metadata assembly disappear from StreamFusion; the native sort/merge and Parquet encoding stay. Track the upstream discussion and the Paimon release that ships it here, then switch the sink over and retire divergence 32.
Split from #33.