[Router][Bugfix] Keep the KV-aware /tokenize fallback off the event loop

[David-Wu1119]

KvawareRouter.route_request tokenizes the prompt locally and falls back to the engine's /tokenize endpoint when the local tokenizer can't be loaded. That fallback was a synchronous requests.post(..., timeout=10) called directly inside the coroutine. So whenever the fallback is taken, the router's event loop is blocked for the whole HTTP round trip, up to the 10 s timeout, and every other in-flight request on the router stalls with it.

LoadAwareRouter, which subclasses KvawareRouter, already solved this in #1035. It has an async tokenize_prompt() that does the same local-first tokenization but runs the remote call in an executor. This PR moves tokenize_prompt() up to KvawareRouter and has KvawareRouter.route_request use it. Both routers now share one implementation, and LoadAwareRouter's behavior is unchanged because it inherits the method.

Test: test_kvaware_remote_tokenize_fallback_does_not_block_event_loop makes local tokenizer loading fail and replaces requests.post with a 0.3 s blocking call. It then runs route_request next to a coroutine that ticks every 10 ms. On main the ticker never runs during the fallback (ticks == 0); with this change it keeps ticking, and routing still uses the tokens returned by /tokenize.

• src/tests/test_kvaware_router.py + src/tests/test_loadaware_router.py: 38 passed.
• black 25.1.0, isort 6.0.0 and ruff 0.12.0 (the pre-commit revs) pass on the changed files.

---

• Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

Found with ruff's ASYNC210 (blocking HTTP call in an async function); written with AI assistance.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gemini-code-assist]

Code Review

This pull request refactors the KV-cache-aware routing logic by moving the tokenize_prompt method and calling it within route_request to handle local-first tokenization with an asynchronous remote fallback. It also adds a unit test to verify that the remote fallback does not block the event loop. Feedback is provided regarding a potential IndexError if endpoints is empty, suggesting a defensive check to raise a 503 HTTPException instead.

[gemini-code-assist]

If endpoints is empty (e.g., when no backends are currently discovered or healthy), calling self.tokenize_prompt(endpoints, request_json) will raise an unhandled IndexError when attempting to access endpoints[0]. To prevent a 500 Internal Server Error, we should defensively check if endpoints is empty and raise a 503 HTTPException, consistent with how LoadAwareRouter handles this scenario.

Suggested change

	# TODO (Yuhan): Handle chat completions
	try:
	tokenizer = self._get_tokenizer(endpoints)
	token_ids = tokenizer.encode(request_json.get("prompt", ""))
	except Exception:
	# Remote /tokenize fallback (let errors bubble up to keep behavior simple)
	remote_url = endpoints[0].url + "/tokenize"
	headers = {"Content-Type": "application/json"}
	data = {
	"model": endpoints[0].model_names[0],
	"prompt": request_json.get("prompt", ""),
	}
	body = requests.post(
	remote_url, headers=headers, json=data, timeout=10
	).json()
	token_ids = body["tokens"]
	token_ids = await self.tokenize_prompt(endpoints, request_json)
	if not endpoints:
	raise HTTPException(
	status_code=503, detail="No backend endpoints available"
	)
	# TODO (Yuhan): Handle chat completions
	token_ids = await self.tokenize_prompt(endpoints, request_json)

[ruizhang0101]

Could you fix the gemini issue? otherwise LGTM

[David-Wu1119]

@ruizhang0101 thanks for the review. Fixed the gemini issue in 53d1045: with no discovered endpoints, the KV-aware router now raises the same 503 ("No backend endpoints available") the other routers use, instead of an IndexError from endpoints[0] in the /tokenize fallback. I added test_kvaware_without_endpoints_returns_503, which fails with that IndexError without the check. src/tests passes locally (242 tests) and pre-commit is clean.