Tracking Issue: New unified ZstdArray #10618

Right now there exists two Zstd encodings - Zstd and ZstdBuffers.

ZstdArray

The first one inlines sizes of buffers into single stream and compresses everything as a contiguous memory region. This is better for compression but worse for almost any other operation, since we store frames we can perform some degree of pruning before decompression.

ZstdBuffersArray

Focuses on efficient buffer access, we have prototyped it for on gpu decompression where we can hand off individual buffers to the nvcomp library and parallelise decompression and access. Since buffers are not concatenated together it will usually be slightly bigger than regular ZstdArray.

The goal is to come up with one unified ZstdArray that tries to maintain good compression behaviour and frame pruning while also being friendly to gpu operations.