[robert3005]

Struct fields are stored unmasked, so aggregates over them saw values under null struct rows. Each chunk of a sequential write stream now carries a demand mask, false in the rows an ancestor nulls out, and DynAccumulator gains accumulate_demanded to aggregate the logical column without masking first.

The struct writer intersects the demand with its validity for each field, the list writer drops the elements of null and undemanded lists so every element written is demanded, and re-chunking writers carry the mask with the rows. Zone maps and file statistics are now written for children of nullable structs, and the fuzzer generates nullable structs.

fixes #9205

[codspeed]

Merging this PR will degrade performance by 10.27%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 3 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

⚡ 5 improved benchmarks
❌ 9 regressed benchmarks
✅ 2304 untouched benchmarks
🆕 6 new benchmarks
🗄️ 47 archived benchmarks run¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Walltime	mul_u64_nonnull_neon	20.3 µs	39.2 µs	-48.3%
❌	CPU Simulation	filter_powerlaw_by_random[10000]	684 ns	934.9 ns	-26.84%
❌	CPU Simulation	bench_compare_varbin[(50000, 2)]	132.1 µs	162.4 µs	-18.68%
❌	CPU Simulation	bench_compare_varbin[(50000, 64)]	133.9 µs	164.2 µs	-18.42%
❌	CPU Simulation	encode_primitives[u8, (2000, 4)]	135.5 µs	163.8 µs	-17.31%
❌	CPU Simulation	bench_compare_sliced_dict_varbinview[(1000, 10000)]	76.9 µs	90.1 µs	-14.65%
❌	CPU Simulation	bench_compare_sliced_dict_varbinview[(7500, 10000)]	198.6 µs	228.3 µs	-13%
❌	CPU Simulation	bench_compare_sliced_dict_varbinview[(10000, 10000)]	204.8 µs	234.5 µs	-12.64%
❌	CPU Simulation	bench_compare_sliced_dict_primitive[(2000, 10000)]	66.7 µs	74.2 µs	-10.11%
⚡	CPU Simulation	take_filter_primitive_small_uncached_random_mask_random_indices[4096, 10]	24 µs	21 µs	+14.58%
⚡	Walltime	starts_with_avx2[64, VarBin]	61 µs	53.7 µs	+13.68%
⚡	CPU Simulation	test_repeated_scan_filter[int]	2.4 ms	2.1 ms	+13.1%
⚡	CPU Simulation	test_scan_filter_to_arrow_array[int]	2.8 ms	2.5 ms	+11.09%
⚡	CPU Simulation	test_scan_filter[int]	2.9 ms	2.6 ms	+10.83%
🆕	CPU Simulation	list_elements_demand_by_validity[0.5]	N/A	841.2 µs	N/A
🆕	CPU Simulation	list_elements_demand_by_validity[0.99]	N/A	90.5 µs	N/A
🆕	CPU Simulation	demanded[null_count]	N/A	82.3 µs	N/A
🆕	CPU Simulation	demanded[sum]	N/A	603.8 µs	N/A
🆕	CPU Simulation	nullable[null_count]	N/A	40 µs	N/A
🆕	CPU Simulation	nullable[sum]	N/A	565 µs	N/A

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing rk/aggregate-demand-mask (55180f2) with develop (96fc43d)}

Footnotes

1. 47 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩