Fold json_get casts written as TRY_CAST, arrow_cast or arrow_try_cast - #127
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #127 +/- ##
==========================================
- Coverage 94.00% 93.94% -0.06%
==========================================
Files 18 18
Lines 1902 1950 +48
Branches 1902 1950 +48
==========================================
+ Hits 1788 1832 +44
- Misses 64 66 +2
- Partials 50 52 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
adriangb
force-pushed
the
fix-try-cast-json-get
branch
from
September 10, 2026 19:07
5ff1fd6 to
86ee661
Compare
adriangb
changed the base branch from
main
to
keep-cast-when-folding-json-get
September 10, 2026 19:08
adriangb
added this pull request to stack #131
September 10, 2026 19:09
adriangb
force-pushed
the
fix-try-cast-json-get
branch
from
September 11, 2026 15:23
86ee661 to
536486c
Compare
adriangb
force-pushed
the
fix-try-cast-json-get
branch
from
September 29, 2026 00:40
536486c to
02c967d
Compare
Only `Expr::Cast` was folded, so three other spellings of the same cast fell back to materializing the JSON union and casting it away. `TRY_CAST` builds an `Expr::TryCast`, which the rewriter's `match` never looked at. `arrow_cast` and `arrow_try_cast` are still scalar function calls at this point: DataFusion lowers them to `Cast` / `TryCast` in `SimplifyExpressions`, an optimizer rule, whereas function rewrites are applied by `ApplyFunctionRewrites` at the start of the analyzer, which never runs again. So the `Cast` node does not exist yet at the only moment the rewriter can see it, and this holds for the non-`try` `arrow_cast` too. All four spellings now share one type table. Because the fold keeps the cast whenever the accessor returns a wider type, `arrow_cast(x, 'Int32')` folds to `CAST(json_get_int(x) AS Int32)` and still comes out as `Int32`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adriangb
force-pushed
the
fix-try-cast-json-get
branch
from
September 29, 2026 00:51
02c967d to
9044fe2
Compare
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JsonFunctionRewriterfoldsCAST(json_get(a, 'b') AS BIGINT)intojson_get_int(a, 'b')so the JSON union never has to be materialized. Three other ways of spelling the same cast are missed, and each falls back to building the union and casting it away:All four return the correct value, so this is a missed optimization rather than a correctness bug. It matters most for a table provider that pushes the typed accessors down to a column or an index: it sees
json_getand its union return type instead ofjson_get_int, so the pushdown and any statistics-based pruning keyed on the concrete type are lost as well as the union round-trip.Two separate causes:
TRY_CASTbuilds anExpr::TryCast, which the rewriter'smatchnever looked at.arrow_castandarrow_try_castare stillExpr::ScalarFunctioncalls when the rewriter runs. DataFusion lowers them toExpr::Cast/Expr::TryCastinSimplifyExpressions, an optimizer rule, whereas function rewrites are applied byApplyFunctionRewritesat the start of the analyzer, which never runs again. So theCastnode does not exist yet at the only moment the rewriter can see it, and this holds for the non-tryarrow_casttoo.Changes
All four spellings now share one type table.
TRY_CASTgoes through the same path asCAST. The typed accessors already yield NULL for a value of another type rather than failing, and because #129 keeps the cast where the accessor returns a wider type, a value outside the target type's range still becomes NULL rather than being handed back as the accessor's wider type.arrow_castandarrow_try_castare matched by function name, which is all that is visible at analyzer time, and only their first argument is replaced, so the named type is still what comes out. The type argument is parsed the way DataFusion parses it itself inArrowCastFunc::return_field_from_args, andArrowCastFunc::simplifythen drops the cast by itself when the accessor already returns the named type.Because #129 keeps the cast, this needs no restriction to the types an accessor returns exactly:
arrow_cast(x, 'Int32')folds toCAST(json_get_int(x) AS Int32)and keeps itsInt32, so it stops materializing the union for no reason.Tests
tests/main.rsgains plan and value tests for each new spelling, includingtest_plan_arrow_cast_fn_narrowing_type_keeps_cast.The property matrix in
tests/rewrite_differential.rswidens from the two SQL cast spellings to all five (CAST,::,TRY_CAST,arrow_cast,arrow_try_cast) × 8 target types, soprop_fold_is_invisibleandevery_cast_spelling_folds_as_documentedcover every folding pair rather than only the SQL ones.narrowing_cast_preserves_type_and_narrowsgains itsTRY_CASTcolumn: on{"a": 3000000000},TRY_CAST(json_get(x,'a') AS INT)isNULL(Int32), asTRY_CASTpromises.Verification
cargo testgreen (192 tests),cargo fmt --checkandcargo clippy --all-targets -- -D warningsclean.🤖 Generated with Claude Code