feat!: upgrade DataFusion to 55

[discord9]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

• DataFusion 55.0.0: apache/datafusion@d5552342
• Global LIMIT across multiple partitions: apache/datafusion#23800
• Safe cast predicate preimages: apache/datafusion#22906
• Chained projection regression: apache/datafusion#24686
• DataFusion PostgreSQL integration: GreptimeTeam/datafusion-postgres@3c77e6c3
• Greptime SQL parser 0.62 range-query port: GreptimeTeam/sqlparser-rs@9e9019bb

What's changed and what's your intention?

This PR upgrades GreptimeDB from the DataFusion 53.1 dependency line to DataFusion 55.0.0, Arrow 59.2, and Parquet 59.2. It rebases the work on 307fe0a692848422f2eb0d1c51990160437ebd55; the GreptimeDB head is 37830649eadce7d8028550c483fd45614913f42c.

The upgrade adapts GreptimeDB planning and execution integrations to DataFusion 55 while retaining the behaviors that are material to GreptimeDB: distributed ordering via EnforceSorting, custom physical plans and expression traversal, recursive MergeScan fallback handling, global Top-K through MergeSort, table-function planning, PostgreSQL compatibility, generated range-query validation, count_hash compatibility, JSON field ordering, and topology-specific SQLness plan expectations.

Dependency versions and pins

Dependency	Revision / version	Purpose
DataFusion	exact 55.0.0; fork 7ac19906c191e51433c0c3d8810d4fd47ab1529a	DataFusion 55 plus the retained GreptimeDB correctness and repair patches below
Arrow / Parquet	exact 59.2.0	DataFusion 55 dependency line
datafusion-orc	released 0.10.0	DataFusion 55 ORC integration
orc-rust	released 0.9.0	Arrow 59-compatible ORC reader
datafusion-pg-catalog / arrow-pg	3c77e6c32b8d	DataFusion 55 PostgreSQL catalog and Arrow integration
sqlparser	0.62.0 at 9e9019bb	Preserves GreptimeDB range-query extensions

All 15 DataFusion entries in [patch.crates-io] use the same immutable fork revision, 7ac19906c191e51433c0c3d8810d4fd47ab1529a; the corresponding 35 Cargo.lock source blocks use that same revision. The released ORC packages replace the temporary ORC git patches, and datafusion-pg-catalog and arrow-pg remain pinned together at their DataFusion 55-compatible commit.

Thin-fork history

The fork starts at official DataFusion 55.0.0 commit d555234201. The six original retained commits remain for these purposes:

Topic	Thin-fork commit	Purpose
Global multi-partition fetch / LIMIT	96625f97aa	Preserve global fetch semantics for multi-partition input; ports apache/datafusion#23800
Duplicate Substrait root names	d1dd7b9ad6	Preserve duplicate root names with deterministic internal qualifiers
Safe cast predicate preimages	da6529d542	Rewrite safe-cast predicates through exact or range preimages
Ordered timestamp widening	e47063d1e4	Preserve correct ordered timestamp-widening predicate rewrites
Chained logical projections	6922ddce3	Restore one-layer-per-pass projection merging and prevent incorrect nested-projection results
Custom float hash representations	dec6af133d	Preserve floating-point hash representations required by count_hash state compatibility

The scalar-planner hook restoration aab310aac36 is retained for JSON2. Four additional fork commits are now pushed: 72872aedb (lint-only timestamp conversion cleanup), a17899ec6 (recursive-CTE schema metadata), 85fc18b51 (Date32 predicate widening), and 7ac19906c (cross-timezone regression correction).

Targeted repairs

JSON2 scalar-function planning

• Why: DataFusion 55 no longer called the scalar ExprPlanner hook GreptimeDB uses to supply JSON2 extraction types. Removing the obsolete GreptimeDB method would compile but makes expressions such as abs(j.metrics.value) fail during planning before analyzer rules can repair them.
• What: The retained scalar-planner patch restores the existing RawScalarExpr / plan_scalar extension point after scalar argument resolution. Modified Original arguments continue through later planners and normal function construction. GreptimeDB keeps its production JSON2 type injection; this does not introduce a test-only path or a new rewrite framework.

Recursive CTE schema metadata

• Why: The stream-backed-scan reopening fix in GreptimeDB fix: re-scan stream-backed tables in recursive CTEs #9039 addresses execution-time reuse across recursive iterations. Its query over information_schema.tables also exposes a distinct DataFusion 55 physical-planning failure: Cannot project plan to expected schema: schema metadata differ. A literal seed and table-backed recursive branch may have different top-level metadata even though the declared recursive output takes the seed metadata; the query must continue to return depths 0, 1, 2.
• What: Commit a17899ec6 makes the existing project_plan_to_schema helper construct ProjectionExec with the declared output schema metadata rather than reject only a top-level metadata difference. The exact-schema fast path remains. Column-count, data-type, nullability-narrowing, and field-metadata mismatches remain rejected. It neither strips scan metadata nor changes fix: re-scan stream-backed tables in recursive CTEs #9039's stream-reopening behavior.

Date32 predicate widening

• Why: The safe-cast preimage allowlist omitted lossless, order-preserving Date32 → Date64, so existing simplify_expr.slt EXPLAIN cases could not eliminate those casts. Every i32 day value scaled by 86,400,000 fits in i64.
• What: Commit 85fc18b51 allows only Date32 → Date64. Inverse-literal whole-day alignment and Date32-domain checks remain, and Date64 → Date32 remains rejected. Coverage includes boundaries, ordered comparisons, nonaligned and out-of-range literals, and reverse-direction rejection.

Cross-timezone equality regression

• Why: A prior SLT positive case treated Brussels 2024-02-01 00:00:01 +01 and Los Angeles 2024-01-31 16:00:01 -08 as equal, although they differ by one hour. Because the view explicitly casts the input to timestamp before assigning Brussels, its matching Los Angeles wall clock is 15:00:01.
• What: Commit 7ac19906c changes the positive literal to 15:00, retains the expected positive row, and preserves 16:00 as a no-match regression. It changes neither timezone implementation nor the positive result semantics.

The lint-only commit 72872aedb removes four redundant i128::from(timestamp_unit_scale(...)) conversions: the helper already returns i128, so strict clippy rejects these identity conversions. Timestamp arithmetic and guards are unchanged.

EnforceSorting remains intentionally preserved: distributed plan-shape regression coverage depends on it to retain ordering behavior.

Validation and PR state

Current head 37830649eadce7d8028550c483fd45614913f42c: locked Cargo metadata, cargo check --locked -p cmd --bin greptime --features vector_index, builds of greptime and sqlness-runner, Rust formatting, and manifest formatting pass. The dependency update changes only 15 manifest pins and 35 lock-file sources, with no unrelated version changes. All 32 standalone/distributed SQLness executions pass, including the recursive CTE over information_schema.tables returning depths 0, 1, 2, all 11 JSON2 cases in each deployment, and four related controls per deployment. Existing SQLness expected files are unchanged.

Fork revision 7ac19906c: all-target/all-feature strict clippy passes; schema-alignment and recursive-plan focused tests pass; the complete datafusion-expr-common crate passes (175 unit tests and 11 doctests). simplify_expr.slt and datetime/timestamps.slt pass. No completed full extended-suite result is claimed.

Earlier migration validation (c488399dc82): 692 query tests passed / 2 skipped; 2519 affected-crate tests passed (one index-build synchronization test passed on its third attempt); seven-crate strict clippy and 30 standalone/distributed SQLness executions passed. These broader results predate the final rebase and pin update and are not a current-head full-suite result.

Current-head CI: builds, Rust tests, Clippy, formatting, and dependency checks passed. Integration CI is not green: all five SQLness variants fail the delta_temporality PromQL regression with an array-type comparison error; four Fuzz jobs also failed and are being diagnosed. These failures remain unresolved; the 32 targeted local SQLness passes above do not cover the failing temporality case.

Cross-version compatibility CI is pending; a normal push does not trigger that workflow. This PR remains a Draft. Transitioning it to Ready for review requires separate approval; no compatibility or current CI-green result is claimed.

PR Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible. (Public GreptimeDB interfaces; internal DataFusion Rust APIs are migrated.)
• Schema or data changes are backward compatible. (Compatibility CI pending.)
• This PR needs to be backported to release branches.

[gemini-code-assist]

Code Review

This pull request upgrades the DataFusion dependency from version 53.1.0 to 54.0.0, which involves extensive refactoring to adapt to API changes such as the removal of explicit as_any() downcasting, updates to the Cast expression structure, and changes to PruningStatistics and partition_statistics signatures. It also introduces a frozen hash_v1 module to preserve compatibility with legacy count_hash states. The review identified critical compilation errors: a missing import for as_largestring_array in the new hash_v1.rs file, and incorrect iterations over the return value of hash_utils::create_hashes in union_distinct_on.rs since it now returns Result<()> instead of the hashes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[sunng87]

orc and pg-catalog are our managed libraries, make sure it merges to upstream before merging this.

[sunng87]

I'm working on the pg-catalog upgrade for datafusion 54. will commit to this branch.

[discord9]

Thanks — I see 193726d60a updates pg-catalog to 0.18 and adds the related planner/session compatibility changes. The PR DCO check is currently failing because this commit has no Signed-off-by trailer. Could you please add the sign-off using the DCO app’s accepted flow? I’ll retain and validate this commit while reconciling the latest main; I won’t overwrite the branch update.

[sunng87]

DCO is a non-blocking check, just set it passed.

[discord9]

Thanks, DCO is green now. One remaining managed-dependency setup: GreptimeTeam/datafusion still has main on 53.1, and my direct attempt to create a 54 branch returned HTTP 403 (push=false). Could you create an organization target branch such as greptimedb-54.0.0 from official DataFusion 54.0.0@45d943dfb8699dc9cb9ef2320e955b73e3e6c03b? The validated thin-fork head is discord9/datafusion:thin-fork/2026-07-16-datafusion-54@027358559b74632d3231eea46a34fcc7f7c99fe9; once the target exists I can open the PR and move #8555 to the organization-controlled SHA.

[sunng87]

datafusion-orc is now on 0.9

[discord9]

The pg-catalog 0.18 integration uncovered one remaining upstream compatibility gap during real PostgreSQL-protocol SQLness validation: OID aliases are represented as Int32 + pg.oid_alias metadata, but reverse casts and wire mapping currently expose numeric values (25::regtype::text -> 25, typinput -> 46) instead of PostgreSQL names/types. I filed datafusion-postgres#384 with library-level reproductions. I am fixing the Greptime-owned wire metadata mapping, pg_attribute.attnum ordering, and unstable OID assertions locally; I will not accept the numeric snapshot as the baseline. The symbolic alias-to-text fix belongs upstream and remains a Draft gate.

[discord9]

Updated locally to the managed datafusion-orc 0.9.0 crates.io release. The temporary personal git source is removed; the locked common-datasource and mito2 checks pass. This will be included in the pending latest-main merge commit.

[discord9]

Could you confirm the intended ownership for the DataFusion thin-fork pin itself? The existing dependency used GreptimeTeam/datafusion, but the validated DF54 head is currently discord9/datafusion@027358559b...; I cannot create the requested GreptimeTeam 54 target branch (push=false, direct push returns 403). Is moving this SHA into GreptimeTeam/datafusion a hard pre-merge requirement, or is the immutable personal-fork SHA acceptable for this upgrade? I am treating it as a conservative dependency-ownership gate until clarified.

[discord9]

The managed DataFusion 54 path is now available. I created GreptimeTeam/datafusion:greptimedb-54.0.0 directly from the exact official 54.0.0@45d943dfb... release commit and opened GreptimeTeam/datafusion#20 from the validated thin-fork head 027358559b.... PR #20 is CLEAN and review has been requested from @sunng87. After it merges, #8555 can replace all personal DataFusion URLs with the organization-controlled immutable SHA.

[sunng87]

datafusion 54.1 is out.
I also have a new release for datafusion-pg-catalog 0.18.2 with some bugfix. Consider sending the quote patch to upstream.

[discord9]

Addressed: eaf98f1a53 extracts the duplicated coordinate×resolution signature construction, coordinate/resolution argument parsing, and per-row resolution conversion into shared helpers (signature_of_coordinates_and_resolution, extract_coordinate_resolution_args, resolution_at) and drops the redundant RESOLUTION_TYPES/DISTANCE_TYPES/POSITION_TYPES alias statics in src/common/function/src/scalars/geo/h3.rs. Behavior, signature order, error messages, and visibility are unchanged; common-function h3 tests pass (1/1) and clippy/fmt are clean.

[sunng87]

We can use this commit for arrow-pg and datafusion-pg-catalog: 3c77e6c32b8db80635a0d2f4b318a36b31170bc3

[fengjiachun]

LGTM

[discord9]

Marking this PR as draft for now. We will merge it after the 1.2 release.

[MichaelScofield]

hold the merge for easier 1.2 backport plz

[killme2008]

This upgrade also fixes #9038 (projection pushdown into a recursive CTE prunes columns the recursive term still reads, causing project index N out of bounds or silently missing rows). Fork rev dec6af1 contains apache/datafusion#22476, which landed in DataFusion 55.0.0. Worth adding a sqlness case in tests/cases/standalone/common/cte/cte.sql here — see #9038 for repros.