Skip to content

perf(spark): read Utf8View values in place instead of copying each one - #11117

Closed
gitRasheed wants to merge 1 commit into
vortex-data:developfrom
gitRasheed:spark-utf8view-in-place
Closed

gitRasheed wants to merge 1 commit into
vortex-data:developfrom
gitRasheed:spark-utf8view-in-place

Conversation

@gitRasheed

Copy link
Copy Markdown
Contributor

Summary

Vortex exports string columns to Arrow as Utf8View, so StringViewAccessor serves every string column Spark reads, and it copied each value into a new byte[]. StringAccessor already returns Utf8 values in place.

Changes

StringViewAccessor.getUTF8String reads each value into a reused NullableViewVarCharHolder and returns UTF8String.fromAddress over the holder's buffer, the same way StringAccessor and LargeStringAccessor do. Arrow's holder overload points at the view itself for values of up to 12 bytes and at the data buffer and offset the view names for longer ones. A value is valid until its batch is released, like the values of Spark's own column vectors.

Tests

VortexArrowColumnVectorTest.stringViewValues reads inline, out-of-line, empty and multi-byte values spread over two data buffers, one at a nonzero offset, and checks each value is off-heap. It fails on develop, where every value is a copied byte[].

Checked locally: :vortex-spark-3.5_2.12:test, :vortex-spark-3.5_2.13:test, :vortex-spark-4.0_2.13:test, spark41CompatTest, spark42CompatTest and spotlessCheck.

I used AI tools to help write and review this change.

Vortex exports every string column to Arrow as Utf8View, so
StringViewAccessor serves every string column Spark reads, and it copied
each value into a new byte[]. Read each value through Arrow's view holder
and return UTF8String.fromAddress over vector memory, as StringAccessor
already does for Utf8.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
@gitRasheed

Copy link
Copy Markdown
Contributor Author

I just saw #11109 does this too: it reads string views in place, so I'm closing this one. The difference is that this PR goes through Arrow's NullableViewVarCharHolder, while #11109 reads the view buffers directly. #11109 has no string-view test. This PR's stringViewValues passes on it unchanged (inline values at the 12/13-byte boundary, out-of-line values at a nonzero offset and in a second data buffer, and an off-heap check), so I'm happy to move it over if it's useful.

@gitRasheed gitRasheed closed this Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant