Skip to content

[Doc][Router] Clarify RequestStatsMonitor TTFT timing #1053

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dsxyy wants to merge 2 commits into
base: main
Choose a base branch
from
+11 −9
Open

[Doc][Router] Clarify RequestStatsMonitor TTFT timing #1053

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3fad59c
docs: clarify RequestStatsMonitor TTFT timing
dsxyy Aug 25, 2026
31605b2
docs: qualify TTFT observation point
dsxyy Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions src/vllm_router/stats/request_stats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,9 @@ class RequestStatsMonitor(metaclass=SingletonMeta):
Monitors the request statistics of all serving engines.
"""

# NOTE (ApostaC): Currently, QPS is calculated based on the number of
# arrived requests in the sliding window, but the inter_token_latency and
# ttft are calculated based on the number of completed requests in the
# sliding window.
# NOTE (ApostaC): Currently, QPS is calculated based on requests arriving
# in the sliding window, while TTFT is calculated when the first response
# token is received for requests in the sliding window.
def __init__(self, sliding_window_size: float = None):
if hasattr(self, "_initialized"):
return
Expand Down Expand Up @@ -173,12 +172,12 @@ def on_new_request(self, engine_url: str, request_id: str, timestamp: float):

def on_request_response(self, engine_url: str, request_id: str, timestamp: float):
"""
Tell the monitor that a response token has been received for a request.
Tell the monitor that the first response token has been received.
Comment thread
dsxyy marked this conversation as resolved.
Outdated
Comment thread
dsxyy marked this conversation as resolved.
Outdated

Args:
engine_url: The URL of the serving engine
request_id: The global request ID
timestamp: The timestamp when the response token was received
timestamp: The timestamp when the first response token was received
"""
if (engine_url, request_id) not in self.request_start_time:
return
Expand Down Expand Up @@ -245,8 +244,8 @@ def get_request_stats(self, current_time: float) -> Dict[str, RequestStats]:
Returns:
A dictionary where the key is the serving engine URL and the value
is the request statistics for that engine.
The TTFT and inter token latency will be -1 if there is no requests
finished in the sliding window.
TTFT will be -1 if no request received its first response token in
the sliding window.
"""
ret = {}
urls = set(self.in_prefill_requests.keys()).union(
Expand Down