Add Java-written ALP fixtures at multiple vector sizes + #105 corner cases#1
Open
vinooganesh wants to merge 1 commit into
Open
Conversation
…orner cases Six fixtures from apache/parquet-java#3397 to broaden the cross-language test surface: alp_java_arade_v1_vs1024.parquet alp_java_arade_v2_vs4096.parquet alp_java_float_arade_v2_vs1024.parquet alp_java_float_spotify1_v1_vs4096.parquet These cover both page versions (V1/V2) and both ALP vector sizes (1024/4096) across float and double — the axes from Prateek's original ask. alp_java_cornercases.parquet 59 KB alp_java_cornercases_expect.csv 130 KB Small fixture (14 cols × 2048 rows) targeting parquet-testing#105: no exceptions, one exception, all-NaN, NaN+Inf+(-0.0), constant (bit_width=0), differing exponents per vector, optional with nulls. The sidecar _expect.csv is independent ground truth emitted from the construction recipe, not from reading the parquet back. Already verified bit-exact: Arrow C++ ALP decoder (apache/arrow#48345) reads all six files; 1,587,306 values vs _expect.csv truth, 0 mismatches. ~1.1 MB total. Have 11 more variants locally — kept this PR small given Andrew's size concerns in apache#105.
This was referenced May 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks onto apache#100. Adds six fixtures from apache/parquet-java#3397 to broaden the cross-language test surface:
Dataset-derived (4 files, ~960 KB) — covers both page versions (V1/V2) × both ALP vector sizes (1024/4096) × float/double from the existing PR apache#100 source data:
alp_java_arade_v1_vs1024.parquetalp_java_arade_v2_vs4096.parquetalp_java_float_arade_v2_vs1024.parquetalp_java_float_spotify1_v1_vs4096.parquetCorner-case (2 files, ~190 KB) — targets apache#105:
alp_java_cornercases.parquet— 14 columns × 2048 rows, each engineered to hit a specific case: no exceptions, one exception per vector, all-NaN, NaN+Inf+(-0.0), constant (bit_width=0), differing exponents per vector, optional with nulls — both f32 and f64alp_java_cornercases_expect.csv— sidecar ground truth emitted directly from the construction recipe (not from reading the parquet back), so it's independent of any encoderVerification already done
Arrow C++ ALP decoder (apache/arrow#48345) reads all six files; 342,306 values bit-compared against the
_expect.csvtruth tables, 0 mismatches. (The full 17-file local matrix totals 1,587,306 values verified across all V1/V2 × 1024/4096 combinations, also 0 mismatches.)parquet-java reads the same fixtures bit-exact via the round-trip tests in apache/parquet-java#3397.
What's held back
11 additional fixtures from the full {v1,v2} × {1024,4096} × {arade,spotify1,float_arade,float_spotify1} matrix (~2.5 MB combined). Kept this PR to ~1.1 MB given the size discussion in apache#105 — happy to add any subset on request.