Skip to content

[BUG] Model Switching Fails with Concurrent Requests #304

Open
Open
[BUG] Model Switching Fails with Concurrent Requests#304
Labels
bugSomething isn't working

Description

@bold84
bold84
opened on Mar 14, 2025

OS

Linux

GPU Library

CUDA 12.x

Python version

3.12

Describe the bug

When attempting to switch models implicitly through API requests, TabbyAPI fails with multiple errors:

  1. rich.errors.LiveError: Only one live display may be active at once - This occurs when multiple model loading operations are attempted simultaneously
  2. torch.OutOfMemoryError: CUDA out of memory - The system runs out of GPU memory during model loading
  3. AttributeError: 'NoneType' object has no attribute 'bos_token' - The tokenizer is not properly initialized when trying to use a model

This does not always happen.

Reproduction steps

  1. Start TabbyAPI with a model loaded
  2. Send a chat completion request specifying a different model than the currently loaded one
  3. Send additional requests in quick succession that also specify different models
  4. Observe the errors in the logs

Expected behavior

TabbyAPI should:

  1. Properly unload the current model before loading a new one
  2. Handle concurrent model loading requests by either queuing them or rejecting additional requests with a clear error while a model is being loaded
  3. Ensure tokenizers are properly initialized before attempting to use them
  4. Provide clear error messages to the client when model switching fails

Logs

tabbyapi-1  | 2025-03-14 11:05:06.977 INFO:     Unloading existing model.
tabbyapi-1  | 2025-03-14 11:05:08.884 INFO:     Model unloaded.
tabbyapi-1  | 2025-03-14 11:05:08.898 INFO:     Attempting to load a prompt template if
tabbyapi-1  | present.
tabbyapi-1  | 2025-03-14 11:05:08.900 INFO:     Attempting to load a prompt template if
tabbyapi-1  | present.
tabbyapi-1  | 2025-03-14 11:05:08.907 INFO:     Using template "qwq" for chat completions.
tabbyapi-1  | 2025-03-14 11:05:09.064 INFO:     Using template "qwq" for chat completions.
tabbyapi-1  | 2025-03-14 11:05:09.064 INFO:     172.19.0.1:33056 - "POST /v1/chat/completions
tabbyapi-1  | HTTP/1.1" 500
tabbyapi-1  | 2025-03-14 11:05:09.125 INFO:     Loading draft model:
tabbyapi-1  | /app/models/Qwen2.5-0.5B-Instruct_exl2_6.0bpw
tabbyapi-1  | 2025-03-14 11:05:09.126 INFO:     Loading with a manual GPU split (or a one GPU
tabbyapi-1  | setup)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:    Exception in ASGI application
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:    Traceback (most recent call last):
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.p
tabbyapi-1  | y", line 409, in run_asgi
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        result = await app(  # type:
tabbyapi-1  | ignore[func-returns-value]
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py",
tabbyapi-1  | line 60, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        return await self.app(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await super().__call__(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 187, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 165, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await self.app(scope, receive, _send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py",
tabbyapi-1  | line 62, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await
tabbyapi-1  | wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await route.handle(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in
tabbyapi-1  | handle
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await wrap_app_handling_exceptions(app,
tabbyapi-1  | request)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        response = await f(request)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        raw_response = await
tabbyapi-1  | run_endpoint_function(
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in
tabbyapi-1  | run_endpoint_function
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        return await dependant.call(**values)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File "/app/endpoints/OAI/router.py", line
tabbyapi-1  | 112, in chat_completion_request
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await load_inline_model(data.model,
tabbyapi-1  | request)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/app/endpoints/OAI/utils/completion.py", line 172, in load_inline_model
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        await model.load_model(
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File "/app/common/model.py", line 112, in
tabbyapi-1  | load_model
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        async for _in load_model_gen(model_path,
tabbyapi-1  | **kwargs):
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File "/app/common/model.py", line 86, in
tabbyapi-1  | load_model_gen
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        progress.start()
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/progress.py", line 1173, in start
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        self.live.start(refresh=True)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/live.py", line 113, in start
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        self.console.set_live(self)
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 835, in set_live
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:        raise errors.LiveError("Only one live
tabbyapi-1  | display may be active at once")
tabbyapi-1  | 2025-03-14 11:05:09.132 ERROR:    rich.errors.LiveError: Only one live display
tabbyapi-1  | may be active at once
tabbyapi-1  | 2025-03-14 11:05:09.468 INFO:     Loading model:
tabbyapi-1  | /app/models/QwQ-32B-8.0bpw-h8-exl2
tabbyapi-1  | 2025-03-14 11:05:09.468 INFO:     Loading with autosplit
tabbyapi-1  | 2025-03-14 11:05:09.606 INFO:     Attempting to load a prompt template if
tabbyapi-1  | present.
tabbyapi-1  | 2025-03-14 11:05:09.613 INFO:     Using template "qwq" for chat completions.
tabbyapi-1  | 2025-03-14 11:05:09.613 INFO:     172.19.0.1:33062 - "POST /v1/chat/completions
tabbyapi-1  | HTTP/1.1" 500
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:    Exception in ASGI application
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:    Traceback (most recent call last):
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.p
tabbyapi-1  | y", line 409, in run_asgi
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        result = await app(  # type:
tabbyapi-1  | ignore[func-returns-value]
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py",
tabbyapi-1  | line 60, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        return await self.app(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await super().__call__(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 187, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 165, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await self.app(scope, receive, _send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py",
tabbyapi-1  | line 62, in **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await
tabbyapi-1  | wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await route.handle(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in
tabbyapi-1  | handle
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await wrap_app_handling_exceptions(app,
tabbyapi-1  | request)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        response = await f(request)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        raw_response = await
tabbyapi-1  | run_endpoint_function(
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in
tabbyapi-1  | run_endpoint_function
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        return await dependant.call(**values)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File "/app/endpoints/OAI/router.py", line
tabbyapi-1  | 112, in chat_completion_request
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await load_inline_model(data.model,
tabbyapi-1  | request)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/app/endpoints/OAI/utils/completion.py", line 172, in load_inline_model
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        await model.load_model(
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File "/app/common/model.py", line 112, in
tabbyapi-1  | load_model
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        async for _in load_model_gen(model_path,
tabbyapi-1  | **kwargs):
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File "/app/common/model.py", line 86, in
tabbyapi-1  | load_model_gen
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        progress.start()
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/progress.py", line 1173, in start
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        self.live.start(refresh=True)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/live.py", line 113, in start
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        self.console.set_live(self)
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 835, in set_live
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:        raise errors.LiveError("Only one live
tabbyapi-1  | display may be active at once")
tabbyapi-1  | 2025-03-14 11:05:09.614 ERROR:    rich.errors.LiveError: Only one live display
tabbyapi-1  | may be active at once
tabbyapi-1  | 2025-03-14 11:05:10.023 INFO:     Attempting to load a prompt template if
tabbyapi-1  | present.
tabbyapi-1  | 2025-03-14 11:05:10.031 INFO:     Using template "qwq" for chat completions.
tabbyapi-1  | 2025-03-14 11:05:10.032 INFO:     172.19.0.1:33070 - "POST /v1/chat/completions
tabbyapi-1  | HTTP/1.1" 500
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:    Exception in ASGI application
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:    Traceback (most recent call last):
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.p
tabbyapi-1  | y", line 409, in run_asgi
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        result = await app(  # type:
tabbyapi-1  | ignore[func-returns-value]
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py",
tabbyapi-1  | line 60, in **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        return await self.app(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await super().__call__(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 187, in **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 165, in **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await self.app(scope, receive, _send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py",
tabbyapi-1  | line 62, in **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await
tabbyapi-1  | wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await route.handle(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in
tabbyapi-1  | handle
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await wrap_app_handling_exceptions(app,
tabbyapi-1  | request)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        response = await f(request)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        raw_response = await
tabbyapi-1  | run_endpoint_function(
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in
tabbyapi-1  | run_endpoint_function
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        return await dependant.call(**values)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File "/app/endpoints/OAI/router.py", line
tabbyapi-1  | 112, in chat_completion_request
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await load_inline_model(data.model,
tabbyapi-1  | request)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/app/endpoints/OAI/utils/completion.py", line 172, in load_inline_model
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        await model.load_model(
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File "/app/common/model.py", line 112, in
tabbyapi-1  | load_model
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        async for _in load_model_gen(model_path,
tabbyapi-1  | **kwargs):
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File "/app/common/model.py", line 86, in
tabbyapi-1  | load_model_gen
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        progress.start()
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/progress.py", line 1173, in start
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        self.live.start(refresh=True)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/live.py", line 113, in start
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        self.console.set_live(self)
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/rich/console.py", line 835, in set_live
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:        raise errors.LiveError("Only one live
tabbyapi-1  | display may be active at once")
tabbyapi-1  | 2025-03-14 11:05:10.032 ERROR:    rich.errors.LiveError: Only one live display
tabbyapi-1  | may be active at once
tabbyapi-1  | 2025-03-14 11:05:18.453 INFO:     127.0.0.1:48666 - "GET /health HTTP/1.1" 200
tabbyapi-1  | 2025-03-14 11:05:28.805 INFO:     Model successfully loaded.
tabbyapi-1  | Loading draft modules ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 51/51   0:00:00
tabbyapi-1  | Loading model modules ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 131/131 0:00:00
tabbyapi-1  | 2025-03-14 11:05:28.806 INFO:     172.19.0.1:33050 - "POST /v1/chat/completions
tabbyapi-1  | HTTP/1.1" 500
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:    Exception in ASGI application
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:    Traceback (most recent call last):
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.p
tabbyapi-1  | y", line 409, in run_asgi
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        result = await app(  # type:
tabbyapi-1  | ignore[func-returns-value]
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py",
tabbyapi-1  | line 60, in **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        return await self.app(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await super().__call__(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 187, in **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 165, in **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await self.app(scope, receive, _send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py",
tabbyapi-1  | line 62, in **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await
tabbyapi-1  | wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await route.handle(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in
tabbyapi-1  | handle
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await wrap_app_handling_exceptions(app,
tabbyapi-1  | request)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        response = await f(request)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        raw_response = await
tabbyapi-1  | run_endpoint_function(
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in
tabbyapi-1  | run_endpoint_function
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        return await dependant.call(**values)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File "/app/endpoints/OAI/router.py", line
tabbyapi-1  | 126, in chat_completion_request
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        prompt, embeddings = await
tabbyapi-1  | apply_chat_template(data)
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/app/endpoints/OAI/utils/chat_completion.py", line 303, in apply_chat_template
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        prompt, mm_embeddings, template_vars =
tabbyapi-1  | await format_messages_with_template(
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File
tabbyapi-1  | "/app/endpoints/OAI/utils/chat_completion.py", line 275, in
tabbyapi-1  | format_messages_with_template
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        special_tokens_dict =
tabbyapi-1  | model.container.get_special_tokens(
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:      File "/app/backends/exllamav2/model.py",
tabbyapi-1  | line 964, in get_special_tokens
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:        "bos_token": self.tokenizer.bos_token if
tabbyapi-1  | add_bos_token else "",
tabbyapi-1  | 2025-03-14 11:05:28.814 ERROR:    AttributeError: 'NoneType' object has no
tabbyapi-1  | attribute 'bos_token'
tabbyapi-1  | 2025-03-14 11:05:29.274 INFO:     Attempting to load a prompt template if
tabbyapi-1  | present.
tabbyapi-1  | 2025-03-14 11:05:29.287 INFO:     Using template "qwq" for chat completions.
tabbyapi-1  | 2025-03-14 11:05:29.493 INFO:     Loading draft model:
tabbyapi-1  | /app/models/Qwen2.5-0.5B-Instruct_exl2_6.0bpw
tabbyapi-1  | 2025-03-14 11:05:29.493 INFO:     Loading with a manual GPU split (or a one GPU
tabbyapi-1  | setup)
tabbyapi-1  | Loading draft modules ╸                                         2%  1/51 -:--:--
tabbyapi-1  | 2025-03-14 11:05:29.632 INFO:     172.19.0.1:49426 - "POST /v1/chat/completions
tabbyapi-1  | HTTP/1.1" 500
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:    Exception in ASGI application
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:    Traceback (most recent call last):
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.p
tabbyapi-1  | y", line 409, in run_asgi
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        result = await app(  # type:
tabbyapi-1  | ignore[func-returns-value]
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py",
tabbyapi-1  | line 60, in **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        return await self.app(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await super().__call__(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 187, in **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 165, in **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await self.app(scope, receive, _send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py",
tabbyapi-1  | line 62, in **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await
tabbyapi-1  | wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await route.handle(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in
tabbyapi-1  | handle
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await wrap_app_handling_exceptions(app,
tabbyapi-1  | request)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        response = await f(request)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        raw_response = await
tabbyapi-1  | run_endpoint_function(
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in
tabbyapi-1  | run_endpoint_function
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        return await dependant.call(**values)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File "/app/endpoints/OAI/router.py", line
tabbyapi-1  | 112, in chat_completion_request
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await load_inline_model(data.model,
tabbyapi-1  | request)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/app/endpoints/OAI/utils/completion.py", line 172, in load_inline_model
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        await model.load_model(
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File "/app/common/model.py", line 112, in
tabbyapi-1  | load_model
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        async for _in load_model_gen(model_path,
tabbyapi-1  | **kwargs):
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File "/app/common/model.py", line 90, in
tabbyapi-1  | load_model_gen
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        async for module, modules in load_status:
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File "/app/backends/exllamav2/model.py",
tabbyapi-1  | line 576, in load_gen
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        async for value in
tabbyapi-1  | iterate_in_threadpool(model_load_generator):
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File "/app/common/concurrency.py", line 30,
tabbyapi-1  | in iterate_in_threadpool
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        yield await asyncio.to_thread(gen_next,
tabbyapi-1  | generator)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/lib/python3.10/asyncio/threads.py", line 25, in to_thread
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        return await loop.run_in_executor(None,
tabbyapi-1  | func_call)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        result = self.fn(*self.args,
tabbyapi-1  | **self.kwargs)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File "/app/common/concurrency.py", line 20,
tabbyapi-1  | in gen_next
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        return next(generator)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 57,
tabbyapi-1  | in generator_context
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        response = gen.send(request)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File "/app/backends/exllamav2/model.py",
tabbyapi-1  | line 638, in load_model_sync
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        for value in self.draft_model.load_gen(
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/model.py", line 335, in
tabbyapi-1  | load_gen
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        module.load()
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116,
tabbyapi-1  | in decorate_context
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        return func(*args, **kwargs)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/attn.py", line 246, in load
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        self.q_proj.load(device_context =
tabbyapi-1  | device_context)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116,
tabbyapi-1  | in decorate_context
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        return func(*args, **kwargs)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/linear.py", line 138, in load
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        device_context =
tabbyapi-1  | self.model.get_device_context(self.device_idx)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/model.py", line 732, in
tabbyapi-1  | get_device_context
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        if not context.ready:
tabbyapi-1  | context.prepare(scratch)
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/device.py", line 81, in
tabbyapi-1  | prepare
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:        self.scratch =
tabbyapi-1  | torch.empty((self.scratch_bytes // 2,), dtype = torch.half, device =
tabbyapi-1  | _torch_device(self.device_idx))
tabbyapi-1  | 2025-03-14 11:05:29.634 ERROR:    torch.OutOfMemoryError: CUDA out of memory.
tabbyapi-1  | Tried to allocate 260.00 MiB. GPU 1 has a total capacity of 23.53 GiB of which
tabbyapi-1  | 190.50 MiB is free. Including non-PyTorch memory, this process has 23.33 GiB
tabbyapi-1  | memory in use. Of the allocated memory 22.70 GiB is allocated by PyTorch, and
tabbyapi-1  | 87.85 MiB is reserved by PyTorch but unallocated. If reserved but unallocated
tabbyapi-1  | memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to
tabbyapi-1  | avoid fragmentation.  See documentation for Memory Management
tabbyapi-1  | (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
tabbyapi-1  | 2025-03-14 11:05:30.058 INFO:     Attempting to load a prompt template if
tabbyapi-1  | present.
tabbyapi-1  | 2025-03-14 11:05:30.070 INFO:     Using template "qwq" for chat completions.
tabbyapi-1  | 2025-03-14 11:05:30.283 INFO:     Loading draft model:
tabbyapi-1  | /app/models/Qwen2.5-0.5B-Instruct_exl2_6.0bpw
tabbyapi-1  | 2025-03-14 11:05:30.283 INFO:     Loading with a manual GPU split (or a one GPU
tabbyapi-1  | setup)
tabbyapi-1  | Loading draft modules ╸                                         2%  1/51 -:--:--
tabbyapi-1  | 2025-03-14 11:05:30.413 INFO:     172.19.0.1:49432 - "POST /v1/chat/completions
tabbyapi-1  | HTTP/1.1" 500
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:    Exception in ASGI application
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:    Traceback (most recent call last):
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.p
tabbyapi-1  | y", line 409, in run_asgi
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        result = await app(  # type:
tabbyapi-1  | ignore[func-returns-value]
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py",
tabbyapi-1  | line 60, in **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        return await self.app(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await super().__call__(scope, receive,
tabbyapi-1  | send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 112,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 187, in **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line
tabbyapi-1  | 165, in **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await self.app(scope, receive, _send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85,
tabbyapi-1  | in **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py",
tabbyapi-1  | line 62, in **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await
tabbyapi-1  | wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 715, in
tabbyapi-1  | **call**
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await self.middleware_stack(scope,
tabbyapi-1  | receive, send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 735, in app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await route.handle(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in
tabbyapi-1  | handle
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await self.app(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 76, in app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await wrap_app_handling_exceptions(app,
tabbyapi-1  | request)(scope, receive, send)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 53, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        raise exc
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line
tabbyapi-1  | 42, in wrapped_app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await app(scope, receive, sender)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        response = await f(request)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 301, in app
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        raw_response = await
tabbyapi-1  | run_endpoint_function(
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 212, in
tabbyapi-1  | run_endpoint_function
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        return await dependant.call(**values)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File "/app/endpoints/OAI/router.py", line
tabbyapi-1  | 112, in chat_completion_request
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await load_inline_model(data.model,
tabbyapi-1  | request)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/app/endpoints/OAI/utils/completion.py", line 172, in load_inline_model
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        await model.load_model(
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File "/app/common/model.py", line 112, in
tabbyapi-1  | load_model
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        async for _in load_model_gen(model_path,
tabbyapi-1  | **kwargs):
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File "/app/common/model.py", line 90, in
tabbyapi-1  | load_model_gen
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        async for module, modules in load_status:
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File "/app/backends/exllamav2/model.py",
tabbyapi-1  | line 576, in load_gen
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        async for value in
tabbyapi-1  | iterate_in_threadpool(model_load_generator):
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File "/app/common/concurrency.py", line 30,
tabbyapi-1  | in iterate_in_threadpool
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        yield await asyncio.to_thread(gen_next,
tabbyapi-1  | generator)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/lib/python3.10/asyncio/threads.py", line 25, in to_thread
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        return await loop.run_in_executor(None,
tabbyapi-1  | func_call)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        result = self.fn(*self.args,
tabbyapi-1  | **self.kwargs)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File "/app/common/concurrency.py", line 20,
tabbyapi-1  | in gen_next
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        return next(generator)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 57,
tabbyapi-1  | in generator_context
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        response = gen.send(request)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File "/app/backends/exllamav2/model.py",
tabbyapi-1  | line 638, in load_model_sync
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        for value in self.draft_model.load_gen(
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/model.py", line 335, in
tabbyapi-1  | load_gen
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        module.load()
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116,
tabbyapi-1  | in decorate_context
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        return func(*args, **kwargs)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/attn.py", line 246, in load
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        self.q_proj.load(device_context =
tabbyapi-1  | device_context)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116,
tabbyapi-1  | in decorate_context
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        return func(*args, **kwargs)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/linear.py", line 138, in load
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        device_context =
tabbyapi-1  | self.model.get_device_context(self.device_idx)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/model.py", line 732, in
tabbyapi-1  | get_device_context
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        if not context.ready:
tabbyapi-1  | context.prepare(scratch)
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:      File
tabbyapi-1  | "/usr/local/lib/python3.10/dist-packages/exllamav2/device.py", line 81, in
tabbyapi-1  | prepare
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:        self.scratch =
tabbyapi-1  | torch.empty((self.scratch_bytes // 2,), dtype = torch.half, device =
tabbyapi-1  | _torch_device(self.device_idx))
tabbyapi-1  | 2025-03-14 11:05:30.414 ERROR:    torch.OutOfMemoryError: CUDA out of memory.
tabbyapi-1  | Tried to allocate 260.00 MiB. GPU 1 has a total capacity of 23.53 GiB of which
tabbyapi-1  | 170.50 MiB is free. Including non-PyTorch memory, this process has 23.35 GiB
tabbyapi-1  | memory in use. Of the allocated memory 22.73 GiB is allocated by PyTorch, and
tabbyapi-1  | 80.82 MiB is reserved by PyTorch but unallocated. If reserved but unallocated
tabbyapi-1  | memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to
tabbyapi-1  | avoid fragmentation.  See documentation for Memory Management
tabbyapi-1  | (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
tabbyapi-1  | 2025-03-14 11:05:48.478 INFO:     127.0.0.1:47330 - "GET /health HTTP/1.1" 200

Additional context

No response

Acknowledgements

  • I have looked for similar issues before submitting this one.
  • I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
  • I understand that the developers have lives and my issue will be answered when possible.
  • I understand the developers of this program are human, and I will ask my questions politely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions