[Router][Feat] --tokenizer: local token-id computation for served-name deployments - #1
Open
tyler2cr wants to merge 1 commit into
Open
Conversation
…utation Engines that serve under a vLLM --served-model-name advertise an alias that is not a resolvable tokenizer id, so the router can never load a tokenizer locally and pays a remote /tokenize round trip (plus engine CPU) on every kv-aware/load-aware routing decision. --tokenizer supplies the real tokenizer id or a local path; _ensure_tokenizer prefers it over the served name, so routing tokenizes in-process after a one-time load. Works air-gapped (point it at a mounted directory of tokenizer files). Must name the same tokenizer the engines run, or router-side ids drift from engine-side KV hashes and kv-aware routing silently degrades - the help text carries that warning. Signed-off-by: tyler <tcr@enfuse.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For the PVAMU K8s/platform team to consider, stacked on the kvaware chat-tokenization branch (upstream PR vllm-project#1045) so this diff is ONLY the override.
What it does
A router flag supplying the tokenizer the router should load for kv-aware/load-aware token-id computation, instead of the name the engines advertise:
Why you'd want it
Your engines serve under
--served-model-name gemma4_31b_it— an alias, not a resolvable tokenizer id — so the router can never tokenize locally and pays one HTTP round trip to an engine's/tokenize(plus engine CPU) on every routing decision. With the override, the router loads the real tokenizer once and computes token ids in-process: zero network on the routing hot path. Also works air-gapped: the flag accepts a local directory of tokenizer files baked into the image or mounted.The one correctness rule
The override must be the same tokenizer your engines run (for you: the exact repo the engines load,
RedHatAI/gemma-4-31B-it-FP8-block). A different tokenizer means router-side ids drift from engine-side KV hashes and kv-aware silently degrades to QPS fallback — the flag's help text carries this warning.Scope / sequencing
c0c4a42+ this commit) instead ofkvaware-chat-minimal, trading a larger carry-diff for zero per-request tokenize round trips.2 tests added (override preferred over served name — the endpoint stub has no
model_namesat all, proving the endpoints aren't consulted; no-override path unchanged); suite 238/238.🤖 Generated with Claude Code