Skip to content

Commit dacf961

Browse files
committed
Export decimal Widen arrays to Arrow with one copy
Arrow exports every precision up to 38 as Decimal128, and the decimal exporter executed its input to the canonical DecimalArray first. A decimal Widen at precision 18 or less therefore copied its child twice: once into the canonical width, and again into the Arrow width. Stop at a Widen and export its child with the precision and scale of the logical dtype, so each value is copied once. The converters take that dtype apart from the values, so no DecimalArray ever pairs it with the narrower child width. Struct, list, dictionary and run-end children reach this exporter too, so they get the single copy as well. The converters narrowed every row, so a null row whose value does not fit in the Arrow width failed the export. Executing the Widen hid this, because the canonical copy skips null rows. Narrow only the valid rows and write zero into the null rows, like the canonical copy does. This also fixes the same failure for a plain DecimalArray stored wider than its canonical width. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
1 parent 351f8d1 commit dacf961

1 file changed

Lines changed: 342 additions & 133 deletions

File tree

0 commit comments

Comments
 (0)