Skip to content

fix: bound index I/O and validate release artifacts - #123

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/bound-index-io-release-gates
Nov 16, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/bound-index-io-release-gates

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Large IVF-RQ partitions could bypass the 64 MiB batch limit, malformed headers could drive large allocations, and JNI output was limited by a single Java byte array. Release jobs also lacked a shared source-version check and runtime validation of their final wheels.

Changes

  • Stream oversized IVF-RQ lists in aligned scan blocks, including a bounded delta-ID window. Preserve top-K, range, filtered, batch, and continued-probe results while accounting for the ID window in the allocation budget.
  • Validate header-derived PQ/RQ memory before construction, use fallible allocations, defer rotation creation, and commit loaded metadata only after complete success so retries cannot retain partial arrays.
  • Write JNI output in reusable 1 MiB chunks, update position after each successful chunk, and preserve the original Java exception.
  • Gate the release orchestrator and independent language workflows on tag, Cargo workspace/path-dependency, Python, and Maven version consistency. RC tags use the existing clean source-version policy.
  • Install each final repaired wheel in an isolated environment and exercise native loading, training, writing, reading, and searching before artifact upload and publication.

Validation

  • cargo test --workspace --no-fail-fast: 737 passed, 2 existing ignored.
  • python3 -m unittest discover -s tools/tests -p 'test_*.py' -q: 26 passed, including final/RC tags, mismatches, and non-tag inputs.
  • cargo clippy --all-targets --workspace -- -D warnings, formatting, license-header, and binary legal-inventory checks passed.
  • Java API tests and native chunk/boundary tests passed, including -Xcheck:jni, buffer reuse, byte order, and original exception identity.
  • A locally built and delocate-repaired macOS arm64 wheel passed isolated installation and the end-to-end smoke test.

Linux x86-64/aarch64 and Windows final-wheel runtime validation is configured as a required step in the release workflows; it still needs the corresponding CI runners.

@github-actions

Copy link
Copy Markdown

Benchmark run · success · attempt 1

Vector index benchmark

🟢 No alerts (5/5 indexes)

Index Recall@10 (Δ) Single QPS Δ Batch QPS Δ Status
IVF_FLAT 100.00% (+0.00 pp) +1.0% +1.0% 🟢 OK
IVF_SQ 99.26% (+0.00 pp) +1.0% +1.0% 🟢 OK
IVF_PQ 46.60% (+0.00 pp) -0.5% +0.8% 🟢 OK
IVF_RQ 90.91% (+0.00 pp) +1.7% +6.3% 🟢 OK
DISKANN 77.18% (+0.00 pp) -1.1% -1.7% 🟢 OK

🟡 QPS ↓ >10% or Recall ↓ >1pp · 🔴 QPS ↓ >20% or Recall ↓ >3pp. Advisory only; QPS is measured after warmup.

Absolute metrics & environment

Medians, base → PR. QPS is timed after warmup; P95 uses the first warm sweep.

Index Single QPS Batch QPS P95 (µs) Build (ms) Process RSS (MiB) Size (MiB)
IVF_FLAT 77,941.50 → 78,689.80 89,287.92 → 90,190.18 15.84 → 16.05 25.00 → 25.00 19.83 → 19.83 2.47 → 2.47
IVF_SQ 62,291.24 → 62,891.55 116,480.91 → 117,594.61 18.10 → 17.96 18.00 → 18.00 19.83 → 19.83 0.67 → 0.67
IVF_PQ 16,301.41 → 16,227.28 133,571.29 → 134,632.05 75.99 → 76.23 190.00 → 191.00 19.83 → 19.83 0.17 → 0.17
IVF_RQ 11,110.03 → 11,294.31 45,654.93 → 48,548.95 112.30 → 108.39 33.00 → 33.00 19.83 → 19.83 0.52 → 0.52
DISKANN 10,147.59 → 10,039.75 19,991.65 → 19,653.76 120.59 → 120.01 1,110.00 → 1,102.00 19.87 → 19.92 3.22 → 3.22

Base f6a5558c495fb797aaece78eb177019fd46e215c · PR merge 410252eb5fdbc479c067836526133ff70f3098c0
rustc 1.100.0 (0ef1bc386 2026-11-09) · Linux-7.0.0-1017-azure-x86_64-with-glibc2.39 · AMD EPYC 7763 64-Core Processor
6 samples/version/index, alternating execution order; 2 threads; 10k × 64D vectors; 2,048 queries; ≥1s per timed mode.
RSS is the process lifetime peak up to build completion, including dataset/ground truth.
Raw CSVs, sample ranges, workload settings and logs are in the workflow artifact.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed oversized IVF-RQ list streaming and block alignment, retained delta-ID window accounting, metadata allocation validation and failed-load cleanup, JNI chunked output and Java exception preservation, and the shared release-version/wheel validation gates. No blocking issues found in this head.

Local validation in an isolated source tree:

  • cargo test --locked --workspace --no-fail-fast: 737 passed, 2 ignored.
  • Release-tool Python unit tests: 26 passed.
  • JDK 8 Java API execution and the native output/panic-boundary regression completed successfully, including a run with -Xcheck:jni.
  • Built a Linux x86-64 wheel, verified its legal/native contents, and passed the new isolated-install smoke test covering native loading, training, writing, reading, and searching.
  • cargo fmt --all -- --check and workspace/all-target Clippy with -D warnings passed.

All 10 current GitHub check runs are successful. The local wheel was a Linux platform wheel, not a repaired manylinux artifact; I did not execute the macOS/Windows release workflows or publish artifacts locally.

@JingsongLi
JingsongLi merged commit c2758d9 into apache:main Nov 16, 2026
10 checks passed
JingsongLi added a commit that referenced this pull request Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants