zarr virtual/kerchunked datasets

[dnelson]

Using kerchunk we can wrap a HDF5 file as a "virtual" zarr dataset. All the original data remains in the HDF5 file(s), the zarr wrapper file simply contains the needed metadata for zarr (or xarray) to directly access the data, bypassing the HDF5 library entirely.

An example for GAIA is here:

/virgotng/mpia/obs/GAIA/gaia_dr3.zarr

(it is actually a json file). It already concatenates the "gaia_dr3.hdf5" and "gaia_dr3_aux.hdf5" files.

It can be read by zarr as:

import fsspec
import zarr

m = fsspec.filesystem("reference", fo="gaia_dr3.zarr").get_mapper("")
ds = zarr.open(m, mode='r'))

We could support this by default, detect if a requested load is in fact such a kerchunked json, in which case, pass to fsspec and then treat m as a zarr file.

[dnelson]

One advantage (possible use case) are remote (HDF5) datasets. When wrapped in this manner, fsspec can transparently handle the remote request (e.g. S3, HTTPS, ...).

The file

/virgotng/mpia/obs/GAIA/gaia_dr3_remote.zarr

is such a test. In theory, it reads the data directly from www.tng-project.org via HTTPS, and with only this file a user can load and analyze the full GAIA data (on any machine).

(Not really tested yet).

[dnelson]

Unfortunately, we cannot use this approach for general TNG-like data yet:

fsspec/kerchunk#333

[dnelson]

Noticed https://github.com/zarr-developers/VirtualiZarr but the limitation seems the same, still waiting for support for "variable-length chunks to the zarr data model".