Skip to content

perf(flat): accelerate scans and cache IVF partitions - #122

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/ivfflat-optimizations
Nov 13, 2026
Merged

JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/ivfflat-optimizations

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

IVF-FLAT repeatedly decoded partition payloads across searches, evaluated shared batch filters for each query, and rescanned all Top-K slots after accepted candidates. Reuse aligned, unfiltered partitions within the reader memory budget, compact filters shared by enough queries, batch short-vector L2 scans, and use an indirect max heap for K >= 32 while preserving tie ordering.

The unified reader now passes its existing memory-budget options to IVF-FLAT. Direct IVFFlatIndexReader::open remains uncached; the new open_with_options enables caching. Oversized lists bypass the cache. Regression tests cover cache bounds, filter isolation, I/O retries, metadata invalidation, heap ties, and range-search semantics. The v1 storage format is unchanged.

Local project ann_bench measurements (M4 Pro, 8 threads, Top-10, nprobe=64, two runs per version) showed warm batch QPS gains of 43% on SIFT1M, 4% on GIST1M, and 53% on GloVe100, with unchanged recall. Warm single-query throughput improved 5.15–5.82x. GIST's first batch regressed approximately 16%, so high-dimensional batch gains are limited and cache-fill overhead remains a tradeoff.

Validation on an isolated checkout of this commit:

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --workspace -- -D warnings
  • cargo test --workspace --quiet: 723 passed, 2 ignored.

Benchmark harnesses, scripts, and documentation changes are excluded from this PR.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit f8ffdbb. No blocking issues found.

Checked the partition-cache accounting and eviction, unfiltered cache reuse with query-local filtering, shared batch-filter compaction, the short-vector L2 Top-K path, and indirect-heap replacement/tie ordering. The follow-up test changes address the initial Rust/JNI failures without changing range-search kernel-consumption semantics or relying on payload reads after cache warmup.

Validation on this commit in an isolated checkout:

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --workspace -- -D warnings
  • cargo test --workspace --no-fail-fast using a fresh target directory: 723 passed, 2 ignored.
  • Java Maven test phase and VectorIndexNativeValidationTest, including the callback-error checks and 168 core range-search oracle cases.

Some GitHub checks are still queued/running at review time; the approval is based on code inspection and the local verification above. The reported large-dataset performance measurements were not independently rerun.

@github-actions

Copy link
Copy Markdown

Benchmark run · success · attempt 1

Vector index benchmark

🟢 No alerts (5/5 indexes)

Index Recall@10 (Δ) Single QPS Δ Batch QPS Δ Status
IVF_FLAT 100.00% (+0.00 pp) +602.6% +29.3% 🟢 OK
IVF_SQ 99.26% (+0.00 pp) +0.1% +0.0% 🟢 OK
IVF_PQ 46.60% (+0.00 pp) -0.2% +1.0% 🟢 OK
IVF_RQ 90.91% (+0.00 pp) -1.4% -3.1% 🟢 OK
DISKANN 77.18% (+0.00 pp) +0.6% +0.2% 🟢 OK

🟡 QPS ↓ >10% or Recall ↓ >1pp · 🔴 QPS ↓ >20% or Recall ↓ >3pp. Advisory only; QPS is measured after warmup.

Absolute metrics & environment

Medians, base → PR. QPS is timed after warmup; P95 uses the first warm sweep.

Index Single QPS Batch QPS P95 (µs) Build (ms) Process RSS (MiB) Size (MiB)
IVF_FLAT 14,492.22 → 101,829.24 84,251.41 → 108,961.70 101.17 → 12.26 26.50 → 27.00 19.71 → 19.71 2.47 → 2.47
IVF_SQ 75,209.08 → 75,256.24 141,015.02 → 141,075.91 15.12 → 15.12 15.00 → 16.00 19.71 → 19.71 0.67 → 0.67
IVF_PQ 20,325.85 → 20,294.13 169,611.74 → 171,301.05 61.01 → 54.04 156.50 → 155.00 19.71 → 19.71 0.17 → 0.17
IVF_RQ 13,714.15 → 13,521.24 60,876.53 → 58,978.71 114.55 → 104.66 28.00 → 28.00 19.75 → 19.80 0.52 → 0.52
DISKANN 12,104.93 → 12,180.75 24,063.14 → 24,118.67 99.65 → 99.81 858.50 → 855.00 19.92 → 19.80 3.22 → 3.22

Base 72e86aa87b6f426cfe6bb89895da5b40cd1c2082 · PR merge a722d617ad1594cab629af817c4ed3555317a45c
rustc 1.100.0 (0ef1bc386 2026-11-09) · Linux-7.0.0-1017-azure-x86_64-with-glibc2.39 · AMD EPYC 9V74 80-Core Processor
6 samples/version/index, alternating execution order; 2 threads; 10k × 64D vectors; 2,048 queries; ≥1s per timed mode.
RSS is the process lifetime peak up to build completion, including dataset/ground truth.
Raw CSVs, sample ranges, workload settings and logs are in the workflow artifact.

@JingsongLi
JingsongLi merged commit eb12c77 into apache:main Nov 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants