Shard-aware IterableDataset for event-related epoching

[neuromechanist]

Depends on: #401, and nemarOrg/nemar-cli#1060 for the events file.

Summary

Naive random-access epoch sampling over a chunked remote store wastes most of what it reads. The fix is on the client, not in the chunk size.

Analysis

Inner chunk is 4.0 s. For a 1.0 s ERP epoch at 63 channels and 250 Hz:

Inner chunk	Chunks per epoch	Read	Useful	Amplification
4 s (current)	1.25	158 kB	32 kB	5.0x
2 s	1.50	94 kB	32 kB	3.0x
1 s	2.00	63 kB	32 kB	2.0x

So re-chunking would buy 2.5x. Reading the whole 300 s shard and yielding every epoch in it buys 5x, because all 300 seconds get used. The 300 s shard is already a WebDataset-style shard.

Crossover is clean: N epochs read individually cost N x 1.25 x 4 seconds-equivalent against 300 for the shard, so they cross at ISI = 5 s.

ISI	Epochs per shard	Strategy
1-4 s	300-75	read the shard
6 s and up	50 and fewer	range-read chunks

Most ERP paradigms sit at ISI 1 to 3 s, so shard reads win. Sparse designs (long-ISI oddball, rare-target, spindle marking) fall the other way.

Proposal

• IterableDataset, not map-style. Map-style random access defeats shard locality by construction and this is the single biggest throughput determinant.
• Shard-level shuffling with an in-memory buffer: shuffle shard order, read a shard, shuffle epochs inside it, yield.
• Adaptive strategy switch keyed on events per shard, computed from events.parquet (Publish <id>/zarr/events.parquet with converter-computed sample indices nemarOrg/nemar-cli#1060).
• Assign shards to DataLoader workers rather than epochs, so each worker holds one decompressed shard.
• Open the store lazily inside each worker. zarr plus boto3 or s3fs clients do not survive fork, which is the default worker start method on Linux, and this is the same discipline the existing boto3 client already needs.