Proposed Solution
https://zarr.dev is emerging as the standard for scientific array data on-disk. It also conveniently provides a way to overlay interpretations of collections of arrays via https://github.com/zarr-conventions. From the spec repo:
A Zarr Convention is a set of attributes on a Zarr Array or Group which confer special meaning about the data contained within.
Conventions are defined via Array or Group attributes.
The key feature of Zarr Conventions is that they are safely ignorable by low-level Zarr implementations.
Conventions therefore may not change how data are encoded or stored; only how they are interpreted by the end user.
I thus believe these are a good fit for binsparse because the constituent arrays of a given sparse array fit this description - zarr can read them independently and correctly at a low level, and a convention indicates their usefulness in summation as a sparse array for readers.
I know there has been some discussion but it has never seemingly landed on a structured and principled solution for where to put the binsparse metadata. I think that a zarr convention provides this. For example,
|
Supported Binary Containers {#supported_containers} |
|
--------------------------------------------------- |
|
|
|
Currently supported binary containers include HDF5 and NetCDF (but should include more). |
states that hdf5 is a compliant container, but where does the JSON live? How do readers know where to find it? I see
#62 talking about this but it hasn't been merged.
Another advantage of conventions is that they are opt-in, and not necessary. Formalizing this as a convention is purely additive. I'd be happy to join a meeting if there were interest in this.
zarr also has a zulip where you would be able to discuss with the wider community: https://ossci.zulipchat.com/#narrow/channel/423692-Zarr-Python/topic/binsparse.20convention/with/625287021
Altenrate solution
binsparse could independently formalize this same thing in zarr (and hdf5, I guess) - where to find the json needed to interpret the constituent arrays.
cc @keewis
Proposed Solution
https://zarr.dev is emerging as the standard for scientific array data on-disk. It also conveniently provides a way to overlay interpretations of collections of arrays via https://github.com/zarr-conventions. From the spec repo:
I thus believe these are a good fit for binsparse because the constituent arrays of a given sparse array fit this description - zarr can read them independently and correctly at a low level, and a convention indicates their usefulness in summation as a sparse array for readers.
I know there has been some discussion but it has never seemingly landed on a structured and principled solution for where to put the binsparse metadata. I think that a zarr convention provides this. For example,
binsparse-specification/spec/draft/index.bs
Lines 876 to 879 in 28577ec
Another advantage of conventions is that they are opt-in, and not necessary. Formalizing this as a convention is purely additive. I'd be happy to join a meeting if there were interest in this.
zarr also has a zulip where you would be able to discuss with the wider community: https://ossci.zulipchat.com/#narrow/channel/423692-Zarr-Python/topic/binsparse.20convention/with/625287021
Altenrate solution
binsparse could independently formalize this same thing in zarr (and hdf5, I guess) - where to find the json needed to interpret the constituent arrays.
cc @keewis