refactor(iceberg): adopt Apache datafusion-iceberg - #742
Merged
alexanderbianchi merged 1 commit intoSep 24, 2026
Merged
alexanderbianchi merged 1 commit into
alexanderbianchi merged 1 commit into
Conversation
alexanderbianchi
force-pushed
the
deps/apache-datafusion-iceberg
branch
from
September 22, 2026 20:41
c525a95 to
21f28f9
Compare
alexanderbianchi
changed the title
alexanderbianchi
marked this pull request as ready for review
September 22, 2026 21:27
gabotechs
approved these changes
Sep 23, 2026
gabotechs
left a comment
gabotechs
left a comment
Collaborator
There was a problem hiding this comment.
Nice, thanks @alexanderbianchi!
alexanderbianchi
merged commit Sep 24, 2026
2e6e1e3
into
datafusion-contrib:main
68 of 70 checks passed
alexanderbianchi
added a commit
that referenced
this pull request
Sep 24, 2026
## Summary Revert #742 until Apache `datafusion-iceberg` uses an `iceberg-rust` revision containing [FileScanTask serialization support](apache/iceberg-rust#3091). #742 aligned our Iceberg dependencies with upstream's older `665c64e` revision, dropping that support. Distributed scans panic in `FileScanTaskWorkUnit::serialized()` with `Serialization not implemented for this field`, and `executes_with_estimated_scan_tasks` then hangs. Reproduced on the merge commit. This is an exact revert: restore the previous `4d83bc7` dependency pins and local predicate conversion implementation, without changing task-count or transport behavior. Fixes #748. ## Validation - `cargo test --locked -p datafusion-distributed-iceberg --features integration` — all 84 unit/integration tests and 1 doctest pass, including the previously hanging test. - `cargo fmt --all -- --check` - `git diff --check HEAD^ HEAD`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
datafusion-icebergcrate as a pinned dependency of the DFD Iceberg integrationconvert_filters_to_predicateimplementation and remove DFD's copied implementation and testsicebergandiceberg-storage-opendalrevisions with the revision used bydatafusion-icebergDFD still depends directly on iceberg-rust for core Iceberg model, runtime, scan-task, and storage types because
datafusion-icebergdoes not re-export them. DFD's table providers and data source also remain custom: they build anIcebergDataSourcebacked by distributed work-unit feeds, whereas the upstream providers build a localIcebergTableScan.Ideally we see datafusion-iceberg and DFD converge such that DFD has it's own data source but it's swapped out during planning. This is because DFD relies on work-unit feeds which are not a general datafusion concept.
Validation
cargo check -p datafusion-distributed-icebergcargo test -p datafusion-distributed-iceberg --libcargo fmt --all -- --checkgit diff --check