Skip to content

Remove automatic probe sizing from the centroid router - #133

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/remove-native-router-probes
Nov 19, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/remove-native-router-probes

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Paimon chooses the outer shard count for each row-ID range using the filter, top-K and, for the joint IVF plan, shard headers. The native router receives that count and ranks the centers. Maintaining the small-router defaults here as well would give the two projects separate copies of the policy. Paimon integration

Remove Java CentroidRouter.resolveNprobe, its JNI declaration and export, and Rust CentroidRouter::resolve_nprobe. The router still assigns vectors and returns the nearest centers for a caller-supplied width. Each native IVF reader continues to size its own local search. The router payload and shard files retain their existing formats.

The removed method was introduced in #131 and tuned in #132. Searches of the current Paimon integration, this repository and indexed public GitHub code found only the bindings, tests and documentation. This removes a public snapshot API. Any downstream snapshot caller using it should pass its outer width to findNearestCentroids(query, width) or Rust find_nearest_centroids(query, width) when upgrading.

Verification

  • Rust workspace tests in release mode, rustfmt and Clippy with warnings as errors passed.
  • Java 8 API and JNI validation passed with the rebuilt library, including caller-specified routing width, all five shard types and 168 range-oracle cases. Checked the packaged Java class and exported JNI symbols for removal of the old method.
  • Paimon at 2482d2aa74 passed 72 focused Spark 3 / JDK 8 tests using this Java JAR and JNI library, including nine centroid SQL integration tests.
  • Replayed 864 paired queries on pinned SIFT snapshots across all five index families, with filters, explicit widths and top-K up to 5,000. Every pair matched the original native-policy implementation (Paimon 22f2f74744 with native perf: tune automatic probing for small centroid routers #132) on the row-ID/score fingerprint, recall, file count and routing bytes. Total logical bytes also matched for each IVF query.

@github-actions

Copy link
Copy Markdown

Benchmark run · success · attempt 1

Vector index benchmark

🟢 No alerts (5/5 indexes)

Index Recall@10 (Δ) Single QPS Δ Batch QPS Δ Status
IVF_FLAT 100.00% (+0.00 pp) -0.0% -0.0% 🟢 OK
IVF_SQ 99.26% (+0.00 pp) -0.3% +0.2% 🟢 OK
IVF_PQ 46.60% (+0.00 pp) +0.3% +0.2% 🟢 OK
IVF_RQ 90.91% (+0.00 pp) +1.4% -0.8% 🟢 OK
DISKANN 78.32% (+0.00 pp) +0.6% +1.4% 🟢 OK

🟡 QPS ↓ >10% or Recall ↓ >1pp · 🔴 QPS ↓ >20% or Recall ↓ >3pp. Advisory only; QPS is measured after warmup.

Absolute metrics & environment

Medians, base → PR. QPS is timed after warmup; P95 uses the first warm sweep.

Index Single QPS Batch QPS P95 (µs) Build (ms) Process RSS (MiB) Size (MiB)
IVF_FLAT 101,149.27 → 101,129.71 107,761.28 → 107,727.08 12.13 → 12.38 28.50 → 29.50 19.89 → 19.89 2.47 → 2.47
IVF_SQ 74,957.92 → 74,705.11 139,491.52 → 139,766.30 15.15 → 15.18 15.00 → 15.50 19.89 → 19.89 0.67 → 0.67
IVF_PQ 21,213.79 → 21,278.51 167,236.93 → 167,593.04 53.48 → 52.71 156.00 → 156.50 19.89 → 19.89 0.17 → 0.17
IVF_RQ 14,700.87 → 14,913.07 60,540.85 → 60,044.01 77.74 → 77.64 28.00 → 28.50 19.89 → 19.89 0.52 → 0.52
DISKANN 12,165.04 → 12,243.33 23,730.82 → 24,068.26 98.84 → 99.57 861.50 → 885.50 19.89 → 19.89 3.22 → 3.22

Base 4e45fbad34f627e5be09b13cdc934445ad6812c9 · PR merge 59667a31abdebb038342be098ff199906cb9967e
rustc 1.100.0 (0ef1bc386 2026-11-09) · Linux-7.0.0-1017-azure-x86_64-with-glibc2.39 · AMD EPYC 9V74 80-Core Processor
6 samples/version/index, alternating execution order; 2 threads; 10k × 64D vectors; 2,048 queries; ≥1s per timed mode.
RSS is the process lifetime peak up to build completion, including dataset/ground truth.
Raw CSVs, sample ranges, workload settings and logs are in the workflow artifact.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JingsongLi
JingsongLi merged commit ac16418 into apache:main Nov 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants