[connortsui20]

Summary

take and mask can make an extension array nullable, while min_max builds non-null result dtypes. These operations need to change nullability without failing. Previously, ExtDType::with_nullability called the plugin validator again and panicked if it returned an error.

An extension type must accept either top-level nullability, so an already valid dtype can be reused with that flag changed. with_nullability now relies on this contract instead of validating again.

Changes

Keeps metadata and storage types intact when changing nullability. Also fixes erased dtype Debug output, which wrote metadata before the struct and displayed metadata: Ok(()) in its place.

Regression tests cover both nullabilities, validation call counts, and compact and pretty Debug output. Checks for the style cleanup are left to CI.

API Changes

Documents that ExtVTable::validate_dtype must be independent of the storage dtype's top-level nullability. Validators can still constrain nullability inside nested storage types.

[codspeed]

Merging this PR will regress 1 benchmark

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 3 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

⚡ 4 improved benchmarks
❌ 1 regressed benchmark
✅ 2313 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	CPU Simulation	encode_primitives[u8, (2000, 4)]	135.5 µs	163.3 µs	-17.06%
⚡	CPU Simulation	filter_powerlaw_by_random[10000]	684 ns	539.5 ns	+26.77%
⚡	CPU Simulation	test_repeated_scan_filter[int]	2.4 ms	2.1 ms	+12.9%
⚡	CPU Simulation	test_scan_filter_to_arrow_array[int]	2.8 ms	2.5 ms	+10.95%
⚡	CPU Simulation	test_scan_filter[int]	2.9 ms	2.6 ms	+10.51%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/ext-types/dtype-invariants (bf78865) with develop (96fc43d)}

[connortsui20]

So this is definitely wrong, and it seems like we need to do a bit more design work around storage dtypes and nullability