null_count is omitted for large columns in parquet files

Describe the bug, including details regarding any error messages, version, and platform.

Currently in ParquetMetadataConverter.java, there is a guard that prevents the writing of statistics such as min/max AND null_count when the stats are larger than the max allowed size under truncation. The rationale for this makes sense for omitting min/max, however null_count can be written on the file despite the size of its content. See the code below:

parquet-java/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java

Lines 800 to 807 in 7be05b4

    
           public static Statistics toParquetStatistics( 
        
               org.apache.parquet.column.statistics.Statistics stats, int truncateLength) { 
        
             Statistics formatStats = new Statistics(); 
        
             // Don't write stats larger than the max size rather than truncating. The 
        
             // rationale is that some engines may use the minimum value in the page as 
        
             // the true minimum for aggregations and there is no way to mark that a 
        
             // value has been truncated and is a lower bound and not in the page. 
        
             if (!stats.isEmpty() && withinLimit(stats, truncateLength)) {

The missing null_count metadata sometimes causes downstream consumers of the parquet files to error. For example in Snowflake we are seeing the following kind of error:

non-nullable column without default has null values according to file statistics

Component(s)

Core

GH-3574: Statistics.toParquetStatistics always set null_count(#3575)

Revert "apacheGH-3574: Statistics.toParquetStatistics always set null…

Revert "GH-3574: Statistics.toParquetStatistics always set null_count(#…

@Fokko

@Fokko Hi! I was wondering if we have anyone tackling the contract change on iceberg side (if there's an issue or PR already)? I agree that emitting the null counts is a good change and should not be skipped when the the min/max exceed the cap.

Also wondering if anyone has context on why the field is in the Statistics object. I know ColumnChunkMetaData holds values like total size and total values, so I was curious why null count (which is relatively small) is kept separate

Navigation Menu

Repository navigation

null_count is omitted for large columns in parquet files #3574

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	public static Statistics toParquetStatistics(
	org.apache.parquet.column.statistics.Statistics stats, int truncateLength) {
	Statistics formatStats = new Statistics();
	// Don't write stats larger than the max size rather than truncating. The
	// rationale is that some engines may use the minimum value in the page as
	// the true minimum for aggregations and there is no way to mark that a
	// value has been truncated and is a lower bound and not in the page.
	if (!stats.isEmpty() && withinLimit(stats, truncateLength)) {

null_count is omitted for large columns in parquet files #3574

Description

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)

Activity

gtrettenero commented on Aug 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions