Repository navigation
Commit dacf961
committed
Export decimal Widen arrays to Arrow with one copy
Arrow exports every precision up to 38 as Decimal128, and the decimal
exporter executed its input to the canonical DecimalArray first. A
decimal Widen at precision 18 or less therefore copied its child twice:
once into the canonical width, and again into the Arrow width.
Stop at a Widen and export its child with the precision and scale of
the logical dtype, so each value is copied once. The converters take
that dtype apart from the values, so no DecimalArray ever pairs it with
the narrower child width. Struct, list, dictionary and run-end children
reach this exporter too, so they get the single copy as well.
The converters narrowed every row, so a null row whose value does not
fit in the Arrow width failed the export. Executing the Widen hid this,
because the canonical copy skips null rows. Narrow only the valid rows
and write zero into the null rows, like the canonical copy does. This
also fixes the same failure for a plain DecimalArray stored wider than
its canonical width.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>1 parent 351f8d1 commit dacf961
1 file changed
Lines changed: 342 additions & 133 deletions
0 commit comments