Repository navigation
perf(spark): read Utf8View values in place instead of copying each one - #11117
Closed
gitRasheed wants to merge 1 commit into
Closed
gitRasheed wants to merge 1 commit into
gitRasheed wants to merge 1 commit into
Conversation
Vortex exports every string column to Arrow as Utf8View, so StringViewAccessor serves every string column Spark reads, and it copied each value into a new byte[]. Read each value through Arrow's view holder and return UTF8String.fromAddress over vector memory, as StringAccessor already does for Utf8. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Contributor
Author
|
I just saw #11109 does this too: it reads string views in place, so I'm closing this one. The difference is that this PR goes through Arrow's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vortex exports string columns to Arrow as
Utf8View, soStringViewAccessorserves every string column Spark reads, and it copied each value into a newbyte[].StringAccessoralready returnsUtf8values in place.Changes
StringViewAccessor.getUTF8Stringreads each value into a reusedNullableViewVarCharHolderand returnsUTF8String.fromAddressover the holder's buffer, the same wayStringAccessorandLargeStringAccessordo. Arrow's holder overload points at the view itself for values of up to 12 bytes and at the data buffer and offset the view names for longer ones. A value is valid until its batch is released, like the values of Spark's own column vectors.Tests
VortexArrowColumnVectorTest.stringViewValuesreads inline, out-of-line, empty and multi-byte values spread over two data buffers, one at a nonzero offset, and checks each value is off-heap. It fails on develop, where every value is a copiedbyte[].Checked locally:
:vortex-spark-3.5_2.12:test,:vortex-spark-3.5_2.13:test,:vortex-spark-4.0_2.13:test,spark41CompatTest,spark42CompatTestandspotlessCheck.I used AI tools to help write and review this change.