[diskann-garnet] Fix Q8 quantizer state loading

[metajack]

The previous work on saving/restoring quantizer state in diskann-garnet did not save/restore state for the Q8 quantizer assuming that constructing it was sufficient. However, the DoubleHadamard transform generates a random transform on construction, so a fresh quantizer will not work to decode quant vectors.

This PR saves and restores the Q8 quantizer state, which is essentially just the DoubleHadamard matrix.

It adds similar serialization/deserialization to MinMaxQuantizer as the spherical quantizer which is then used by diskann-garnet.

[Copilot]

Pull request overview

This PR fixes restart correctness for the Q8 (MinMax 8-bit) quantizer in diskann-garnet by persisting and restoring the quantizer’s randomized DoubleHadamard transform state via FlatBuffers, instead of assuming reconstruction yields an identical transform.

Changes:

• Added FlatBuffers serialization/deserialization support for MinMaxQuantizer (including transform state) in diskann-quantization, plus round-trip tests.
• Updated diskann-garnet Q8 initialization to load persisted MinMax quantizer state when present and persist it when first created; added a restart test for Q8.
• Added a MinMax FlatBuffers schema and build integration; bumped diskann-garnet package versions to 4.0.2.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
diskann-quantization/src/minmax/quantizer.rs	Adds MinMax quantizer FlatBuffers serialize/deserialize + tests to ensure transform state round-trips.
diskann-quantization/src/flatbuffers/minmax/quantizer_generated.rs	New generated FlatBuffers bindings for MinMax quantizer state.
diskann-quantization/src/flatbuffers.rs	Wires MinMax FlatBuffers module into the crate import tree.
diskann-quantization/schemas/minmax_v001.fbs	Introduces the MinMax quantizer schema with identifier mq00.
diskann-quantization/build.rs	Includes MinMax schema in FlatBuffers build generation.
diskann-garnet/src/quantization.rs	Uses MinMaxQuantizer FlatBuffers serialization and adds construction-from-bytes for restart.
diskann-garnet/src/provider.rs	Persists/loads Q8 quantizer state in metadata; adds a Q8 restart test.
diskann-garnet/diskann-garnet.nuspec	Version bump to 4.0.2.
diskann-garnet/Cargo.toml	Version bump to 4.0.2.
Cargo.lock	Updates locked version for diskann-garnet.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.18467% with 54 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.44%. Comparing base (1552e51) to head (096f80e).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...tion/src/flatbuffers/minmax/quantizer_generated.rs	60.33%	48 Missing ⚠️
diskann-garnet/src/provider.rs	95.08%	3 Missing ⚠️
diskann-quantization/src/minmax/quantizer.rs	96.80%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1264      +/-   ##
==========================================
+ Coverage   90.16%   90.44%   +0.27%     
==========================================
  Files         509      498      -11     
  Lines       97238    95832    -1406     
==========================================
- Hits        87678    86676    -1002     
+ Misses       9560     9156     -404     

Flag	Coverage Δ
miri	90.44% <81.18%> (+0.27%)	⬆️
unittests	90.11% <81.18%> (+0.27%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-garnet/src/quantization.rs	96.86% <100.00%> (+1.35%)	⬆️
diskann-quantization/src/flatbuffers.rs	100.00% <ø> (ø)
diskann-garnet/src/provider.rs	81.01% <95.08%> (+0.65%)	⬆️
diskann-quantization/src/minmax/quantizer.rs	92.82% <96.80%> (+1.26%)	⬆️
...tion/src/flatbuffers/minmax/quantizer_generated.rs	60.33% <60.33%> (ø)

... and 64 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hildebrandmw]

I looked over the diskann-quantization code. One small suggestion for testing, otherwise looks fine.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

diskann-quantization/src/minmax/quantizer.rs:821

• The serialization roundtrip test only checks PartialEq on MinMaxQuantizer, so it can pass even if the deserialized quantizer produces different codes (e.g., if Transform’s PartialEq misses some internal state). Since the goal here is to ensure persisted state restores the exact quantization behavior, add a check that compress_into produces identical canonical bytes for a fixed input vector (and this will also make the NBITS-specific test variants meaningfully exercise per-bitrate codegen).

            let serialized = quantizer.serialize(GlobalAllocator).unwrap();
            let deserialized = MinMaxQuantizer::try_deserialize(&serialized).unwrap();

            assert_eq!(deserialized, quantizer);
        }