[connortsui20]

Tracking Issue: #10701

Stacked on #11005. #10992 builds on it. It does not need #11006.

Summary

A narrow DecimalByteParts MSP should remain narrow after decoding, while every decimal leaf must have the canonical width of its own dtype. Returning a Widen preserves both rules and avoids a widening copy on the common read path.

Merge gates:

• Wait for 0.91.0 to publish the compatibility fixtures, then run uv run vortex-test/compat-gen/scripts/compat.py check --mode last on this branch.
• Merge Build DecimalByteParts from the canonical decimal array #10992 immediately afterward to replace storage-based splitting.
• The CUDA changes need a CUDA reviewer. They were not built locally.

Changes

Decoding shares from_stored_decimal with the other decimal reader. A lone MSP supplies exact cached extrema or one integer MinMax pass, and the selected child caches those extrema for later aggregates. Lower parts must be u64, with their count bounded by precision, so only a lone MSP can exceed the canonical width.

CUDA has no Widen kernel, so it widens a narrow MSP on the device. Device narrowing remains an error because overflow detection would require a host synchronization. Decode benchmarks stop at the decoded form, with a separate canonicalization benchmark for the widening cost.

Historical local benchmark results (Apple M4, medians)

These retained measurements predate this style pass and rebase. They do not measure the current head.

cargo bench -p vortex-decimal-byte-parts --bench dbp_compute, 8,192 rows. Before is the tip of #10946 before it was split into #11002 to #11006, which decoded DecimalByteParts as develop did at the time. After is the earlier implementation that produced these measurements. The results isolate this PR. msp_i64_p18_canonical is the control: its MSP is already at the canonical width, so this PR does not change its decode.

bench	case	before	after
decode	gap_msp_i16_p10	82.75 ns	1.29 µs
decode	msp_i32_p18	73.97 ns	1.03 µs
decode	msp_i32_p18_nullable	80.49 ns	5.332 µs
decode	msp_i64_p18_canonical	72.02 ns	89.28 ns
canonicalize (new)	gap_msp_i16_p10		2.519 µs
canonicalize (new)	msp_i32_p18		2.041 µs
canonicalize (new)	msp_i32_p18_nullable		6.957 µs
filter	gap_msp_i16_p10	749.7 ns	1.791 µs
filter	msp_i32_p18	1.041 µs	2.04 µs
filter	msp_i32_p18_nullable	2.582 µs	6.291 µs
filter	msp_i64_p18_canonical	1.999 µs	2.249 µs
min_max	gap_msp_i16_p10	5.124 µs	1.958 µs
min_max	msp_i32_p18	4.083 µs	1.739 µs
min_max	msp_i32_p18_nullable	6.165 µs	5.957 µs

Benches that this PR does not change, such as compare_constant and the control, ran about 20% slower in the after run than in the before run, so part of each after number is run-to-run noise. canonicalize includes the widening cost that the narrow MSP decode avoids.

API Changes

DecimalByteParts decoding returns a canonical decimal array or a decimal Widen. Constructors and the v2 reader require u64 lower parts and reject counts beyond the precision's width. CUDA decoding returns the canonical width and rejects a wider MSP.

[codspeed]

Merging this PR will degrade performance by 39.09%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 3 improved benchmarks
❌ 22 regressed benchmarks
✅ 2335 untouched benchmarks
🆕 5 new benchmarks
🗄️ 24 archived benchmarks run¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	decode[msp_i32_p18_nullable]	18.7 µs	218.8 µs	-91.45%
❌	CPU Simulation	decode[msp_i64_p38]	17.4 µs	129.2 µs	-86.49%
❌	CPU Simulation	decode[gap_msp_i16_p10]	17.4 µs	117.7 µs	-85.22%
❌	CPU Simulation	decode[msp_i32_p18]	17.5 µs	96.3 µs	-81.82%
❌	CPU Simulation	sum[gap_msp_i16_p10]	80 µs	168.2 µs	-52.43%
❌	CPU Simulation	sum[msp_i32_p18_nullable]	155.7 µs	319.6 µs	-51.28%
❌	CPU Simulation	filter[msp_i32_p18_nullable]	141.7 µs	256.7 µs	-44.8%
❌	CPU Simulation	filter[gap_msp_i16_p10]	87.6 µs	155.1 µs	-43.48%
❌	CPU Simulation	sum[msp_i32_p18]	94.5 µs	145.7 µs	-35.15%
❌	CPU Simulation	filter[msp_i32_p18]	109.2 µs	159.7 µs	-31.63%
❌	CPU Simulation	sum[msp_i64_p38]	142.3 µs	203.2 µs	-30%
❌	CPU Simulation	min_max[gap_msp_i16_p10]	106 µs	148.1 µs	-28.4%
❌	CPU Simulation	filter[msp_i64_p38]	152.3 µs	208.5 µs	-26.96%
❌	Walltime	i128_to_i256_neon	10.7 µs	14.5 µs	-26.56%
❌	CPU Simulation	decompress[dbp_bp_i32]	55 µs	73.6 µs	-25.2%
❌	Walltime	i64_to_i128_neon	10.3 µs	13.4 µs	-23.47%
❌	CPU Simulation	bench_compare_varbin[(50000, 2)]	132 µs	162.2 µs	-18.61%
❌	CPU Simulation	bench_compare_varbin[(50000, 64)]	133.8 µs	164 µs	-18.45%
❌	CPU Simulation	compare[1]	54.5 µs	63.4 µs	-14.08%
❌	CPU Simulation	bench_compare_sliced_dict_varbinview[(1000, 10000)]	77.9 µs	88.9 µs	-12.42%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/dbp-canonical-decode (a69e7aa) with ct/decimal-widen-stored (b3b006c)}

Footnotes

1. 24 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

[connortsui20]

So the regressions are definitely real, but without narrow decimal encoding there's not much we can do.

I will note though that I think it is good we regress these benchmarks because now it actually shows how expensive a very large decimal is in its fully decoded representation.

[robert3005]

But we have Widen now so shouldn't this decode to Decimal(Widen) to preserve behaviour?