Repository navigation
Remove automatic probe sizing from the centroid router - #133
Merged
JingsongLi merged 1 commit intoNov 19, 2026
Merged
Conversation
|
Benchmark run · success · attempt 1 Vector index benchmark🟢 No alerts (5/5 indexes)
🟡 QPS ↓ >10% or Recall ↓ >1pp · 🔴 QPS ↓ >20% or Recall ↓ >3pp. Advisory only; QPS is measured after warmup. Absolute metrics & environmentMedians, base → PR. QPS is timed after warmup; P95 uses the first warm sweep.
Base |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Paimon chooses the outer shard count for each row-ID range using the filter, top-K and, for the joint IVF plan, shard headers. The native router receives that count and ranks the centers. Maintaining the small-router defaults here as well would give the two projects separate copies of the policy. Paimon integration
Remove Java
CentroidRouter.resolveNprobe, its JNI declaration and export, and RustCentroidRouter::resolve_nprobe. The router still assigns vectors and returns the nearest centers for a caller-supplied width. Each native IVF reader continues to size its own local search. The router payload and shard files retain their existing formats.The removed method was introduced in #131 and tuned in #132. Searches of the current Paimon integration, this repository and indexed public GitHub code found only the bindings, tests and documentation. This removes a public snapshot API. Any downstream snapshot caller using it should pass its outer width to
findNearestCentroids(query, width)or Rustfind_nearest_centroids(query, width)when upgrading.Verification
2482d2aa74passed 72 focused Spark 3 / JDK 8 tests using this Java JAR and JNI library, including nine centroid SQL integration tests.22f2f74744with native perf: tune automatic probing for small centroid routers #132) on the row-ID/score fingerprint, recall, file count and routing bytes. Total logical bytes also matched for each IVF query.