Implement a direct reader for HDF5 files

[JamesWrigley]

For specific kinds of datasets, this retrieves the chunk information from HDF5 and then reads it manually using os.preadv(). Reading it ourselves releases the GIL so reads can be multithreaded and be much faster.

This originated from a multiprocessing streamer that I wrote ages ago, and when cleaning it up recently I figured it would be nice to reuse that machinery to make to make .xarray() and .ndarray() faster as well. It didn't go very well though because multiprocessing requires shared memory and that's limited on most systems, and since reading from KeyData objects is so common I didn't want to require shared memory for that. Long story short, I ended up ✨ vibe coding ✨ a Python native HDF5 reader that can be used with threads instead of processes.

Example performance improvement for compressed AGIPD data:

And uncompressed JF data:

Times are with warm data. With cold data there's also an improvement but it's like 2-5x instead of 10x.

Design notes:

• Reading data is done through direct_read.read() which can transparently fall back to h5py for unsupported datasets (strings, etc).
• The new DatasetReader class is created at read time with things like the ROI and a file descriptor of the file. The original files are followed through the VDS if one is being used.
• The new DatasetInfo class holds chunk info about a dataset and is lazily cached in FileAccess objects. It's reused across reads by DatasetReader.
• Supporting ROIs added a lot of complexity... The direct reader will read contiguous 'bands' of rows of the ROI into a scratch buffer and can then slice down the columns to copy into the output buffer.
• The Planner class takes read operations (ReadOp), creates DatasetReader objects, and creates jobs that can be run on a thread pool.
• The tests did not create data on disk, which means that the direct reader would always fall back to h5py in the tests. I refactored the fixtures in 3ad0113 to both reduce the size on disk and fill them with data. There's also some explicit tests for the direct reader.
• KeyData and the detector components should now be fast by default ⏩
• The EXTRA_DATA_DIRECT_READ_DEBUG environment variable can be used to figure out whether doing direct reads failed and for what reason.

Performance notes:

• For contiguous unchunked datasets we can parallelize reading as much as we like, but for some reason GPFS doesn't like multithreaded reads larger than 48MiB. Hence there's a SPLIT_BYTES size to specify the maximum size of a chunk (file, not HDF5) that we read.
• Read times of compressed datasets seem to be bottlenecked by decompression and GIL overhead.
• gpfs_fcntl() exists 🐙 The header file on maxwell is at /usr/lpp/mmfs/include/gpfs_fcntl.h. I tried a few flags in there but none made any performance difference. GPFS_CLEAR_FILE_CACHE is amazeballs though, that will clear the file cache so you can benchmark cold reads easily.

For reviewers, I tried to keep the commits atomic so I'd recommend reviewing them one-by-one. direct_read.py in particular is mostly vibe-coded, with various tweaks from me (not that I'm an HDF5 expert).

I believe we could also reuse this in the DAMNIT API. It should already work for regular ndarray's. For DataArray's I think we can use open_dataarray() to lazily create the DataArray object with the metadata, load the main dataset with the direct reader, and then copy the lazily-opened object and swap in the main array.

I'm not quite satisfied with the streamer yet so I'll leave that for a later PR.

[takluyver]

Nice, I will take a proper look after a long weekend.

I also coincidentally did something similar with writing recently - set up an empty dataset in advance, then have parallel processes writing into it directly with pwrite():

https://git.xfel.eu/kluyvert/extra-azint/-/blob/main/extra_azint/output.py?ref_type=heads