Skip to content

[core] fix stats evolution for data evolution tables#7797

Merged
JingsongLi merged 3 commits into
apache:masterfrom
steFaiz:fix_files_system_table
May 11, 2026
Merged

[core] fix stats evolution for data evolution tables#7797
JingsongLi merged 3 commits into
apache:masterfrom
steFaiz:fix_files_system_table

Conversation

@steFaiz
Copy link
Copy Markdown
Contributor

@steFaiz steFaiz commented May 9, 2026

Purpose

Currently stats evolution in Files System table do not consider data evolution, which may cause:

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at org.apache.paimon.memory.MemorySegmentUtils.getLongMultiSegments(MemorySegmentUtils.java:779) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
	at org.apache.paimon.memory.MemorySegmentUtils.getLong(MemorySegmentUtils.java:769) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
	at org.apache.paimon.data.BinaryArray.getLong(BinaryArray.java:160) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
	at org.apache.paimon.stats.SimpleStatsEvolution$NullCountsEvoArray.getLong(SimpleStatsEvolution.java:215) ~[paimon-flink-1.18-1.4-beta-20260415.042901-14.jar:1.4-beta-SNAPSHOT]
...

This PR fixes this by passing the writeCols on creation.

Tests

See org.apache.paimon.table.system.FilesTableTest

leaves12138
leaves12138 approved these changes May 11, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@leaves12138 leaves12138 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The change correctly scopes SimpleStatsEvolution by both schema id and writeCols, so the stats row/array is first interpreted in the data-evolution file's written-column layout and then evolved back to the current table schema. This should avoid the out-of-bounds/null-count mapping issue in the $files table, and the new regression test covers partial-column data-evolution files across schema evolution.

@JingsongLi
Copy link
Copy Markdown
Contributor

JingsongLi commented May 11, 2026

+1

@JingsongLi JingsongLi merged commit d121dd3 into apache:master May 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leaves12138 leaves12138 leaves12138 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@steFaiz @JingsongLi @leaves12138