refactor: split api server endpoints

[lvhan028]

Motivation

api_server.py had grown into a large module containing endpoint handlers, request validation, shared runtime state, lifecycle management, middleware, and server startup logic. This made individual APIs difficult to locate, test, and maintain.

The existing VariableInterface name also did not clearly describe its purpose as the shared runtime context for serving endpoints. The class-level state made endpoint dependencies less explicit and complicated isolated testing.

Modification

• Split API handlers into a new domain-oriented lmdeploy.serve.openai.endpoints package:

  • Chat completions
  • Completions
  • Generate
  • Models
  • Auxiliary APIs
  • Management APIs
  • DistServe APIs

• Added create_openai_router(server_context) to assemble and register all endpoints.

• Replaced VariableInterface with an instance-based ServerContext.

• Passed ServerContext explicitly to endpoint routers instead of accessing class-level global state.

• Moved chat, completion, and generate request validators alongside their corresponding endpoints.

• Moved shared model-list and OpenAI error-response helpers into openai/utils.py.

• Updated Anthropic endpoints, proxy imports, and tests for the new context and module structure.

• Added tests covering the complete route manifest and context isolation.

• Preserved all existing HTTP paths, request/response schemas, validation dependencies, streaming behavior, and generated OpenAPI components.

[Copilot]

Pull request overview

This PR refactors the OpenAI-compatible API server by splitting the monolithic api_server.py endpoint implementations into dedicated endpoint modules and introducing an instance-based ServerContext to hold runtime dependencies/state.

Changes:

• Replaces the static VariableInterface pattern with an injectable ServerContext, and assembles routes via create_openai_router.
• Moves OpenAI endpoint handlers into lmdeploy/serve/openai/endpoints/* and centralizes shared helpers in openai/utils.py + endpoints/common.py.
• Updates tests and call sites (including proxy + Anthropics endpoints) to use the new router/context wiring.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_lmdeploy/serve/test_session_cleanup.py	Removes API-server-wrapper cleanup test; remaining cleanup coverage now targets with_request_cleanup.
tests/test_lmdeploy/serve/test_health_monitor.py	Updates health endpoint tests to resolve handlers from the new OpenAI router using ServerContext.
tests/test_lmdeploy/serve/test_generation_config.py	Updates generate request validation import and adapts fake context to new async_engine.* access pattern.
tests/test_lmdeploy/serve/openai/test_openai_endpoints.py	Adds router-manifest tests to validate route assembly and context injection behavior.
tests/test_lmdeploy/serve/anthropic/test_endpoints.py	Adapts fake server context to new session manager wiring (async_engine.session_mgr).
lmdeploy/serve/proxy/proxy.py	Switches create_error_response import to the new OpenAI utils module.
lmdeploy/serve/openai/utils.py	Adds get_model_list and create_error_response helpers; keeps tool-call filtering utilities.
lmdeploy/serve/openai/serving_generate.py	Removed (request validation moved under endpoint modules).
lmdeploy/serve/openai/serving_completion.py	Removed (request validation moved under endpoint modules).
lmdeploy/serve/openai/serving_chat_completion.py	Removed (request validation moved under endpoint modules).
lmdeploy/serve/openai/endpoints/init.py	Exposes create_openai_router for API-server assembly.
lmdeploy/serve/openai/endpoints/router.py	New router assembly module registering endpoint groups against a shared server_context.
lmdeploy/serve/openai/endpoints/models.py	Implements /v1/models endpoint using get_model_list(server_context).
lmdeploy/serve/openai/endpoints/management.py	Implements management endpoints (/health, /terminate, sleep/wakeup, abort, weight update).
lmdeploy/serve/openai/endpoints/common.py	Shared validation + generation-config merge helpers for endpoint modules.
lmdeploy/serve/openai/endpoints/chat_completions.py	Chat completions handler migrated from api_server.py into its own module.
lmdeploy/serve/openai/endpoints/completions.py	Completions handler migrated from api_server.py into its own module.
lmdeploy/serve/openai/endpoints/generate.py	/generate handler migrated from api_server.py into its own module.
lmdeploy/serve/openai/endpoints/auxiliary.py	Migrates auxiliary endpoints (encode/pooling/ppl/embeddings).
lmdeploy/serve/openai/endpoints/distserve.py	Migrates DistServe endpoints under /distserve/*.
lmdeploy/serve/openai/api_server.py	Simplifies API server to create ServerContext, mount the new router, and wire startup/lifespan logic.
lmdeploy/serve/anthropic/endpoints/messages.py	Updates Anthropics endpoints to access engine config/session manager via server_context.async_engine.*.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

tests/test_lmdeploy/serve/test_session_cleanup.py:85

• This change removes the only test that exercised with_request_cleanup behavior when the consumer task is cancelled mid-stream. Since the serving endpoints rely on with_request_cleanup to close engine generators and remove sessions, it would be good to keep a cancellation-focused test to prevent regressions in that cleanup path.

def test_idle_async_close_removes_session_and_allows_reuse():
    asyncio.run(_run_idle_async_close_removes_session_and_allows_reuse())


async def _run_request_cleanup_removes_unstarted_generator_session():

lmdeploy/serve/openai/endpoints/chat_completions.py:640

• The PR description says all existing HTTP paths are preserved, but the deprecated /v1/chat/interactive route that existed in api_server.py is no longer registered anywhere (no matches in the new endpoints package). If clients still call it, this becomes a breaking change; consider re-adding it (still hidden from schema) with the same deprecation error response as before.

            response['cache_block_ids'] = cache_block_ids
            response['remote_token_ids'] = remote_token_ids

        return response

lmdeploy/serve/openai/endpoints/completions.py:146

• with_cache is read from the raw JSON, but this endpoint still allows prompt to be a list (multiple sessions/generators). When with_cache is true, _inner_call mutates the shared cache_block_ids/remote_token_ids nonlocals (later in the function), which will race / break if there is more than one generator. Either support per-prompt cache metadata, or (simplest) reject with_cache unless exactly one prompt is provided.

        sessions = []
        if isinstance(request.prompt, str):
            request.prompt = [request.prompt]
            sessions.append(server_context.create_session(request.session_id))
        elif isinstance(request.prompt, list):
            for _ in request.prompt:
                sessions.append(server_context.create_session())
        if isinstance(request.stop, str):
            request.stop = [request.stop]

lmdeploy/serve/openai/endpoints/distserve.py:33

• engine_info returns response.model_dump_json() (a JSON string). FastAPI will treat this as a plain string response (typically text/plain) instead of a JSON object, which can break clients expecting JSON. Return a dict (or a JSONResponse) instead.

            num_gpu_blocks=engine_config.num_gpu_blocks)

        return response.model_dump_json()