Skip to content

fix: convert Arrow JSON when updating columns via merge/update path#7472

Open
xloya wants to merge 1 commit into
lance-format:mainfrom
xloya:upstream-pr/json-fragment-update
Open

fix: convert Arrow JSON when updating columns via merge/update path#7472
xloya wants to merge 1 commit into
lance-format:mainfrom
xloya:upstream-pr/json-fragment-update

Conversation

@xloya

@xloya xloya commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Problem

Updating an Arrow JSON column through fragment.update_columns / the update merge path fails with a type-mismatch error.

Root cause

The HashJoiner right-side stream did not convert Arrow JSON (Utf8) to Lance JSON (LargeBinary/JSONB) before joining.

Fix

Wrap the right-side stream in a JsonConvertingReader that converts Arrow JSON fields to Lance JSON before the join.

Test

Python test_fragment_update_columns_with_json_column plus dataset update tests.

@github-actions github-actions Bot added A-python Python bindings bug Something isn't working labels Jun 25, 2026
@xloya xloya force-pushed the upstream-pr/json-fragment-update branch from 7de1961 to 9939817 Compare June 25, 2026 08:06
@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.50000% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/dataset/write/merge_insert.rs 97.47% 2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant