Add integration tests for all `parquet-testing` files

[kevinjqliu]

Feature request

Add integration tests that run against every Parquet file in the apache/parquet-testing repository.

Problem

parquet-java currently tests only selected files from parquet-testing. As a result, new interoperability fixtures are not automatically tested, and compatibility regressions may go undetected.

There are already several parquet-java issues involving files from parquet-testing.

Proposed solution

Add integration tests that classify files into two groups:

• Valid files: must open successfully and allow all row groups and columns to be read.
• Invalid files: must fail during footer parsing or record decoding with an expected exception.

InterOpTester already supports downloading individual files from parquet-testing. This feature would extend that approach to provide repository-wide coverage.

Acceptance criteria

• Every Parquet file in the pinned revision is classified.
• All valid files are read successfully.
• All invalid files fail as expected.
• Each file is reported as an independent test.
• Unclassified files cause the test to fail.
• Exclusions require an explicit entry linked to a tracking issue.

Related issues

• #3336: Hadoop parquet reader and parquet CLI fail on some files from parquet-testing
• #1686: Add integration test for BYTE_STREAM_SPLIT
• #3633: Older parquet-java readers fail on files with unprojected VARIANT columns

[wgtmac]

I would prefer adding parquet-testing as a git submodule before doing this. We've found occasional network error when downloading test files.

[kevinjqliu]

Thanks @wgtmac

Ive had bad experiences with git submodule BUT im happy to give it another try 😄

Looks like there was a previous attempt at this and decided against doing so:

• PARQUET-1992: Move submodules update to ci-test profile and encryption interop test to maven integration-test phase #877
• PARQUET-1992: Manually download interop files inside the test and move encryption interop test to maven integration-test phase #878

Im not too familiar with the codebase, do you think adding parquet-testing as a git submodule is doable?

[wgtmac]

Arrow C++ is using git submodule exactly: https://github.com/apache/arrow/tree/main/cpp/submodules. I've no idea on previous attempt in this repo. @gszadovszky Do you have any comment on this?

[gszadovszky]

I don't recall the exact problem we've had, but the tarball was mentioned. I'm not against using git submodules, but we need to test the entire workflow, including release, that they won't be affected in a bad way. For release, every test should be executed automatically. But we shall not include the test files to the tarball. There might be other issues as well. Also, if we go with git submodules, we shall update the previous solution we used at #878 as well.