Skip to content

Hosting the MCP server, ingest endpoint and scheduler in one process #4

Closed
Closed
Hosting the MCP server, ingest endpoint and scheduler in one process#4
Assignees
ryansull
Labels
wayfinder:researchAFK: surface a fact a decision waits on

Description

@ryansull
ryansull
opened on Oct 17, 2026

Question

SyncShelf is one container that must serve three things: the MCP endpoint (Streamable
HTTP), a plain HTTP POST /ingest/goodreads-csv for n8n, and in-process scheduled syncs.
What is the cleanest current way to compose those in Python, and do the originating
prompt's tech choices hold up?

Establish:

  • Current MCP Python SDK state: FastMCP (the SDK's bundled one vs the separate fastmcp
    package — which is the "official" path now?), Streamable HTTP transport, mounting it as an
    ASGI app inside Starlette/FastAPI alongside ordinary routes, lifespan handling, and how
    per-request auth context (the caller's identity) reaches a tool function.
  • Stateless vs stateful Streamable HTTP sessions — which suits a single-replica homelab
    service, and what breaks across container restarts.
  • Scheduler options inside an asyncio ASGI app: APScheduler 3.x vs 4.x (4.x maturity?),
    a plain asyncio loop, or something else. Concurrency guarantees (no overlapping runs of
    the same sync, sync_now racing a scheduled run).
  • SQLite from async code: aiosqlite vs sync sqlite3 in a thread, WAL settings, single
    writer implications.
  • Anything in the originating stack (Python 3.12, httpx, SQLite WAL, APScheduler, pytest)
    with a clearly better option for this size of service — or confirmation there isn't one.

Output: findings note in .wayfinder/research/003-*.md with sources and SDK versions.

Activity

  1. ryansull
    added
    wayfinder:researchAFK: surface a fact a decision waits on
    on Oct 17, 2026
  2. ryansull
    added a parent issue on Oct 17, 2026
  3. ryansull
    self-assigned this
    on Oct 17, 2026
  4. ryansull
    marked this as blocking 2 issues on Oct 17, 2026

  5. ryansull commented on Oct 17, 2026

    @ryansull
    ryansull
    on Oct 17, 2026
    OwnerAuthor

    Resolution

    One process works cleanly: the official MCP Python SDK v2, mounted in a plain Starlette app and served by uvicorn with one worker. The ingest route sits before the MCP mount. A single lifespan opens SQLite, starts the sync loops and runs mcp.session_manager.run(). MCP runs stateless.

    Full findings, wiring sketch and sources: the findings comment on this ticket (moved there from a research branch, which has been deleted). I checked the version claims against PyPI and the mcp 2.3.0 wheel.

    Changes to the originating stack

    • FastMCP → MCPServer (mcp 2.3.0). SDK v2 (July 2026) renamed it, and 1.x is maintenance-only. The standalone fastmcp 4.x package is only worth adding if The front door: auth architecture needs its OAuth proxy.
    • Python 3.12 → 3.14. 3.12 is security-fix-only.
    • APScheduler → plain asyncio tasks. One loop per source, each with an asyncio.Lock that sync_now also takes, so a manual run never overlaps a scheduled one. APScheduler 4 is still alpha (4.0.0a6), and the note records an unreleased fix in 3.11.3 for a scheduler-loop failure that stops all jobs.
    • httpx: keep, but consider httpx2. It's the same API, and mcp 2.3.0 already depends on it.
    • SQLite WAL and pytest: keep. For async tests, use anyio's pytest plugin rather than pytest-asyncio.

    Facts later tickets build on

    • Identity in tools. A tool reads the caller via get_access_token() (subject, claims); Context carries no identity. v2 also runs handlers with the HTTP request's context, so verified-header middleware works too. → The front door: auth architecture
    • Why stateless. Claude can't be assumed to speak the newest protocol revision. In stateful mode a restart, or the new 30-minute idle expiry, gives legacy clients "404 Session not found". SyncShelf's tools don't need the server-to-client channel that stateless mode gives up.
    • SQLite access. stdlib sqlite3, one writer connection on a single-thread executor, BEGIN IMMEDIATE, WAL, synchronous=NORMAL, busy_timeout=5000, foreign_keys=ON. aiosqlite only with a write lock, since coroutines sharing one connection can interleave transactions. → SQLite schema and repo structure
    • Gotchas. A public hostname must be in transport_security or every request gets 421. MCP requests are capped at 4 MiB (relevant if a large Goodreads CSV ever went through MCP rather than /ingest). Mount("/") must be the last route. MCP auth doesn't cover /ingest, which needs its own secret check. Set validate_token_resource=True explicitly.
    • Unverified. Which SQLite version Debian base images ship, relative to the WAL-reset bug (3.7.0 to 3.51.2). The single-writer design avoids the trigger anyway. → Deployment topology: Traefik, Cloudflare Tunnel, reaching Chaptarr
  6. ryansull
    closed this as completedon Oct 17, 2026

  7. ryansull commented on Oct 17, 2026

    @ryansull
    ryansull
    on Oct 17, 2026
    OwnerAuthor
    Research findings: One-process findings (full note, moved here from a deleted research branch)

    003 — Hosting the MCP server, ingest endpoint and scheduler in one process

    Research findings for ticket 003. Researched 2026-10-17. Facts and recommendations only;
    the decisions are made in later tickets (010 auth, 018 schema/repo, 019 deployment, 020 spec).

    1. Short answer

    Recommended composition: official MCP Python SDK v2 (mcp 2.3.x, MCPServer,
    formerly FastMCP) mounted at / in a plain Starlette app, served by uvicorn with one
    worker    . The POST /ingest/goodreads-csv route sits before the MCP mount. One top-level
    lifespan opens the DB, starts the sync loops and enters mcp.session_manager.run(). Streamable
    HTTP runs with stateless_http=True. Syncs run as plain asyncio tasks, one per source,
    each with its own asyncio.Lock    , and sync_now takes the same lock. SQLite uses
    stdlib sqlite3: one writer connection on a single-thread executor plus read
    connections, in WAL mode with busy_timeout, and BEGIN IMMEDIATE for writes.

    How the originating choices hold up:

    Originating choice Verdict One-line reason
    Python 3.12 Change → 3.14 (3.13 also fine) 3.12 has been security-fix-only since April 2025. 3.14 is a year old and every dependency here supports it. 3.15.0 shipped on 2026-10-09 and is too new.
    Official MCP Python SDK with FastMCP Holds, but it is v2 now: MCPServer, not FastMCP SDK 2.0.0 (2026-07-28) renamed FastMCP to MCPServer, and v1.x is maintenance-only. v2 fixed a stale-identity bug in stateful sessions (§6).
    Standalone fastmcp package instead? Not by default. Revisit only if 010 needs an OAuth proxy fastmcp 4.x is built on SDK v2 and adds an OAuth proxy for IdPs without DCR, which SyncShelf may never need. It also says "breaking changes may occur in minor versions".
    Streamable HTTP Holds The only non-deprecated remote transport. One streamable_http_app() serves both legacy (≤2025-11-25) and 2026-07-28 clients.
    (Implicit) FastAPI Use plain Starlette The SDK already depends on Starlette, and one ingest route doesn't justify FastAPI.
    httpx Holds. Consider httpx2 SDK v2 itself depends on httpx2 (the Pydantic-maintained continuation of httpx, same API). httpx 0.28.1 hasn't had a release since 2024-12. Using httpx2 means one HTTP stack instead of two.
    SQLite WAL Holds Right size for two users. Needs single-writer discipline and a directory volume on local disk (§5).
    APScheduler in-process Change → plain asyncio loops + per-source lock (APScheduler 3.11.x is the fallback) sync_now needs a lock anyway, which makes max_instances redundant. 4.x is still alpha, and 3.11.3 has an unreleased asyncio fix (§6). Fixed N-minute polling doesn't need cron or misfire handling.
    pytest Holds Use the anyio pytest plugin, which ships with anyio and is what the SDK docs use, rather than pytest-asyncio.

    2. Minimal wiring sketch (illustrative only)

    # app.py: SDK v2 (mcp 2.3.x), Starlette 1.x, uvicorn --workers 1
from contextlib import asynccontextmanager
from starlette.applications import Starlette
from starlette.routing import Mount, Route
from mcp.server import MCPServer
from mcp.server.auth.middleware.auth_context import get_access_token
from mcp.server.transport_security import TransportSecuritySettings

mcp = MCPServer("syncshelf", token_verifier=Verifier(), auth=AuthSettings(...,
          resource_server_url=PUBLIC_MCP_URL, validate_token_resource=True))  # shape depends on 010

@mcp.tool()
async def sync_now(source: str) -> str:              # async def: sync tools run in a worker thread
    user = household_user(get_access_token())       # AccessToken.subject / .claims -> users row
    return await syncs[source].run_now(triggered_by=user)  # same lock as the scheduled loop

async def ingest_goodreads(request):                # n8n; NOT covered by MCP auth, so check a shared secret
    require_ingest_secret(request.headers)
    form = await request.form(); csv = await form["file"].read()
    await db.write(store_goodreads_snapshot, csv)    # then nudge the goodreads sync (§4)
    return JSONResponse({"accepted": True}, status_code=202)

mcp_app = mcp.streamable_http_app(stateless_http=True,
    transport_security=TransportSecuritySettings(allowed_hosts=[PUBLIC_HOST, "localhost:*"]))

@asynccontextmanager
async def lifespan(app):
    await db.open()                                  # PRAGMAs, writer executor
    async with mcp.session_manager.run():           # REQUIRED: a mounted app's own lifespan never runs
        tasks = [s.start() for s in syncs.values()]  # one asyncio loop per source
        try: yield
        finally: await stop_all(tasks)
    await db.close()

app = Starlette(lifespan=lifespan, routes=[
    Route("/healthz", healthz),
    Route("/ingest/goodreads-csv", ingest_goodreads, methods=["POST"]),
    Mount("/", app=mcp_app),                        # LAST: Mount("/") swallows everything after it
])

    Ways the caller's identity reaches a tool:

    • The SDK's own auth (token_verifier= + auth=). get_access_token() returns the
      AccessToken your verifier built, with fields client_id, scopes, subject,
      expires_at, resource and claims. It returns None outside an authenticated HTTP
      request, for example in in-memory tests. Context carries no identity:
      Context.client_id was removed in v2 because it was never an authenticated identity.
    • A front door that injects a verified header, if 010 lands there (for example a
      Cloudflare Access JWT). A pure-ASGI middleware on the outer Starlette app verifies the
      header and sets an app-owned ContextVar. In v2 each handler is dispatched under the
      sending HTTP request's contextvars snapshot (runner._sender_context,
      jsonrpc_dispatcher last_context), so that var is visible inside the tool. ctx.headers
      also exposes raw headers, but the docs warn they are "never an identity" unless something
      verified them.
    • The SDK has no built-in JWKS/JWT verifier. TokenVerifier is a one-method protocol, so
      a JWT verifier is about 20 lines with pyjwt[crypto], which is already an SDK dependency.
      fastmcp ships a JWTVerifier.

    Where things live:

    • /.well-known/oauth-protected-resource/mcp is served inside the MCP sub-app. Mounting
      that app at / keeps it at the root where discovery expects it.
    • Plain routes (@mcp.custom_route or your own Starlette routes) are never authenticated
      by MCP auth.

    3. Stateless vs stateful (single replica)

    Recommendation: stateless_http=True, and leave json_response at its default (False).

    • What the flag affects. It only matters for legacy clients (protocol ≤2025-11-25).
      Connections on 2026-07-28 are sessionless by construction ("Mcp-Session-Id is never set")
      whatever the flag says. The SDK routes each request by its MCP-Protocol-Version header.
    • Which protocol Claude connectors use is unverified. I found no Anthropic doc saying
      Claude's custom connectors speak 2026-07-28. The help center confirms Streamable HTTP and
      OAuth with DCR. Plan for legacy clients.
    • What restarts break in stateful mode. Legacy session state is a "plain in-process
      dict". A container restart, or the v2.2.0+ default idle expiry of 30 minutes
      (session_idle_timeout=1800), drops every session. The next request carrying the old
      Mcp-Session-Id gets 404 Session not found and the client must re-initialize.
      Recovery depends on the client handling that correctly.
    • What restarts break in stateless mode. Only requests in flight at the moment of the
      restart. There's nothing to expire, so session_idle_timeout, max_sessions and
      event_store are unused.
    • The cost of stateless. On legacy connections it removes the server-to-client
      back-channel: ctx.elicit(), sampling and roots raise NoBackChannelError, and
      request-tied notifications are silently dropped. SyncShelf's tools (search, rate, sync,
      queue review) don't need any of these. The SDK's own advice: "tools don't call back into
      the client → use stateless_http=True."
    • json_response=True also removes the per-request stream, so progress notifications
      are dropped. There's no benefit for a single replica.
    • Multi-round-trip requestState (2026-07-28 elicitation via Resolve): the keys are
      process-generated and "gone on restart". This only matters if a tool ever asks the user
      for input.
    • Run uvicorn with exactly one worker. Stateless mode would tolerate more, but the
      in-process sync loops and the single SQLite writer assume exactly one process.

    4. Scheduler concurrency (no overlap; sync_now vs a scheduled run)

    Recommendation: one supervised asyncio task per source, each owning an asyncio.Lock.

    • Scheduled loop. while True: await run_once(); await sleep(interval + jitter). A loop
      can't overlap itself by construction. Wrap each run in try/except Exception with logging
      so one failure never kills the loop, and cancel the task in lifespan shutdown.
    • run_once and sync_now share the lock. sync_now checks lock.locked():
      • If a run is going, it returns "already running since T, run id R" rather than queueing a
        duplicate. Awaiting completion is also possible, but connector tool calls have client
        timeouts, so it's better to return quickly and let sync_status report the result.
      • Otherwise it starts the run as a task held by the sync object, not tied to the HTTP
        request, so a dropped connection doesn't cancel a half-done sync, and it returns the
        run id.
    • Scheduled run fires while sync_now holds the lock. The scheduled tick either skips
      when locked() (preferred, logged) or waits. Either way only one run per source executes
      at a time.
    • Ingest endpoint. It stores the CSV snapshot (a short write transaction), then calls the
      same run_now on the Goodreads sync, or just sets an event the loop waits on. It returns
      202 straight away. n8n never waits on a sync.
    • Cross-source exclusion, if two syncs write the same Hardcover rows: add one global
      "hardcover-writer" lock that sources take around their write phase. The DB writer is
      already serialized (§5).
    • Tools that touch these locks must be async def. In v2, plain def tools run in a
      worker thread (anyio.to_thread.run_sync), where an asyncio.Lock can't be used.
    • No auth context in scheduled runs. get_access_token() returns None there. Syncs act
      per household user from stored per-user configuration, never from a request token.

    If APScheduler is kept instead (pin APScheduler>=3.11.3,<4):

    • Use AsyncIOScheduler, created and .start()ed inside the lifespan, so it attaches to
      uvicorn's running loop, and .shutdown(wait=...) it on exit.
    • Use coroutine jobs with explicit id= and max_instances=1, coalesce=True, plus a
      misfire_grace_time.
    • Still put the same asyncio.Lock around the job body. max_instances only stops a job
      overlapping itself. Triggering sync_now through the scheduler
      (job.modify(next_run_time=now)) makes a busy job skip with only a log warning
      ("maximum number of running instances reached"), which gives the tool no useful answer.
    • Never run 4.x: it is still at 4.0.0a6 (2025-04-27) with breaking changes in every alpha.

    5. SQLite access

    Recommendation: stdlib sqlite3, one dedicated writer connection on a
    ThreadPoolExecutor(max_workers=1), and units of work as plain sync functions.

    • Writes: await loop.run_in_executor(writer, fn, *args), where fn is one function and
      one BEGIN IMMEDIATE … COMMIT.
    • Reads: asyncio.to_thread with a per-thread read connection (or one more small
      executor). WAL lets them run alongside the writer.
    • Why this shape:
      • Single-writer is structural, not a convention.
      • Transaction boundaries are function boundaries.
      • The DB layer is ordinary sync code that pytest tests directly.
      • No extra dependency.

    Settings:

    PRAGMA journal_mode = WAL;        -- persistent in the file; set once at open anyway
PRAGMA synchronous = NORMAL;      -- safe with WAL (durability of the last txn on power loss is the only trade)
PRAGMA busy_timeout = 5000;       -- per connection; covers checkpoint/backup contention
PRAGMA foreign_keys = ON;         -- per connection, off by default
-- writes: BEGIN IMMEDIATE (take the write lock up front)

    Notes:

    • Why BEGIN IMMEDIATE. A deferred transaction that reads and then writes can fail
      straight away with SQLITE_BUSY (snapshot conflict), and busy_timeout doesn't help in
      that case. Open connections with sqlite3.connect(path, autocommit=True) (new in 3.12) or
      isolation_level=None, and issue BEGIN IMMEDIATE yourself.
    • Python 3.12+ deprecated the default datetime adapters and converters. Store
      timestamps as explicit ISO-8601 UTC text or epoch ints.
    • aiosqlite (0.22.1, 2025-12-23) is acceptable but has a trap. It runs one thread per
      connection and serializes statements, not transactions. If several coroutines share
      one aiosqlite connection, coroutine B's statements can land inside coroutine A's open
      transaction across an await. That makes an asyncio.Lock around every write transaction
      mandatory. The executor pattern above gets the same result with less ceremony.
    • SQLAlchemy async is overkill (it rides on aiosqlite anyway). If ticket 018 picks
      Alembic, SQLAlchemy Core (sync) on the same writer executor still fits.
    • Docker volume. Mount a directory volume (db + -wal + -shm together) on local
      disk      . WAL "does not work over a network filesystem", so no NFS or SMB.
    • Backups. Use the SQLite backup API or VACUUM INTO, never a bare copy of the .db
      file. A file separated from its -wal can lose committed transactions.
    • WAL-reset corruption bug. It affects SQLite 3.7.0–3.51.2 and needs two connections
      writing or checkpointing at the same instant      . It is fixed in 3.51.3 (2026-03-13), with
      backports to 3.44.6 and 3.50.7. A single writer connection keeps SyncShelf out of the
      trigger condition. Still, log sqlite3.sqlite_version at startup and prefer a base image
      whose libsqlite3 is ≥3.51.3 or a patched backport. Debian's packaged versions were not
      verified here.

    6. Version pins observed and gotchas

    Versions observed on 2026-10-17:

    Package Latest Notes
    mcp 2.3.0 (2026-10-02) v2.0.0 went stable 2026-07-28. 1.x is maintenance-only (1.30.0, 2026-09-07). The 2.3.0 tag requires Python ≥3.10, main now ≥3.11. Dependencies: httpx2>=2.10, starlette>=0.27 (≥0.48 on 3.14), anyio>=4.9 (≥4.10 on 3.14), uvicorn>=0.31.1, pyjwt[crypto], python-multipart, sse-starlette.
    fastmcp 4.1.0 (2026-10-08) 3.4.9 (2026-10-16) backports security fixes to 3.x. 4.x runs on SDK v2. Python ≥3.10.
    starlette 1.8.0 (2026-10-13)
    uvicorn 0.54.0 (2026-09-25)
    httpx2 2.13.1 (2026-09-23) Owned by Pydantic.
    httpx 0.28.1 (2024-12-06) 1.0 is only at dev pre-releases (1.0.dev6, 2026-08-31).
    APScheduler 3.11.3 (2026-06-28) 4.x is alpha only (4.0.0a6, 2025-04-27).
    aiosqlite 0.22.1 (2025-12-23)
    pytest 9.1.1 (2026-06-19)
    Python 3.15.0 (2026-10-09) 3.14 is the mature current line. 3.12 is security-only.

    Suggested constraints: mcp>=2.3,<3, starlette>=1.8,<2, uvicorn>=0.54, httpx2>=2.13
    (or httpx==0.28.*), APScheduler>=3.11.3,<4 only if kept, pytest>=9.1. Lock exact
    versions with uv.

    Gotchas and known bugs:

    • Mount and lifespan:
      • A mounted sub-app's lifespan never runs. Without async with mcp.session_manager.run()
        in the top-level lifespan, the server starts but the first /mcp request fails with
        RuntimeError: Task group is not initialized.
      • mcp.session_manager only exists after streamable_http_app() has been called. Build
        the app at module level and touch the manager only inside the lifespan.
    • Mount("/") must be the last route. Routes after it are unreachable. Mounting under a
      prefix instead moves both the endpoint (/prefix/mcp) and the well-known metadata path.
    • 421 Misdirected Request. With the default host="127.0.0.1", DNS-rebinding protection
      auto-enables with a localhost-only allowlist. Behind a real hostname (a tunnel or reverse
      proxy), every request gets 421 until transport_security=TransportSecuritySettings(allowed_hosts=[...])
      includes the public host. Also set uvicorn's --proxy-headers / --forwarded-allow-ips
      for the proxy.
    • Stale identity in 1.x stateful sessions (issue #2208, closed 2026-08-10, fixed in v2
      via #2298/#2710). get_access_token() could return the token of the request that
      created the session, not the current caller. That's another reason to use v2 and
      stateless mode, and a reason never to stay on 1.x with stateful sessions in a multi-user
      setup.
    • v2 renames:
      • from mcp.server import MCPServer (also mcp.server.mcpserver).
      • mcp.server.fastmcp raises ModuleNotFoundError.
      • get_context() is removed; declare ctx: Context instead. ctx.fastmcp → ctx.mcp_server.
      • stateless_http, json_response, streamable_http_path, event_store, host and
        port moved from the constructor to run() / streamable_http_app().
      • MCP_* env vars and .env files are no longer read.
      • The default server name changed to mcp-server, so pass a name.
    • v2 server lifespan= is entered once when the session manager starts and is shared by
      all requests. In v1 it ran per session or per request.
    • 4 MiB request-body cap on MCP endpoints (HTTP 413). Tune it with
      max_request_body_size. This doesn't apply to the Starlette ingest route. Starlette's
      max_part_size (1 MB) limits non-file form fields only, so n8n must send the CSV as a
      file part.
    • validate_token_resource. Leaving it unset with resource_server_url set emits
      MCPDeprecationWarning, and SDK 3.0 will default it to True. Set it to True
      explicitly.
    • v2.1.0 behaviour changes:
      • Unexpected handler exceptions return a generic error to the client. Log server-side, and
        use Client(..., raise_exceptions=True) in tests.
      • Content-block return types no longer advertise outputSchema unless
        structured_output=True is passed.
    • Testing auth. The in-memory Client(mcp) skips HTTP, so get_access_token() is None
      there. Identity-dependent tool tests need to either go through the ASGI app with an
      httpx(2) ASGITransport, which also exercises the lifespan if you use a lifespan manager,
      or route identity through a small current_user() seam that tests can override.
    • APScheduler 3.11.3. An unreleased fix in its changelog says AsyncIOScheduler
      "no longer stops processing jobs permanently if _process_jobs() raises". In 3.11.3 a
      single failure in the scheduler's own loop can silently halt all scheduling. This is part
      of why §4 prefers plain loops.
    • fastmcp says "breaking changes may occur in minor versions" and its docs track main.
      Pin exactly if it's adopted. Its 4.1.0 also added a 30-minute HTTP session idle timeout,
      mirroring the SDK.
    • Starlette 1.x removed the old on_startup/on_shutdown hooks, so lifespan is the only
      way. The sketch already uses it.

    7. Sources

    MCP Python SDK:

    fastmcp:

    Claude connectors:

    HTTP clients:

    Scheduling:

    SQLite:

    Other packages:

    Python:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

wayfinder:researchAFK: surface a fact a decision waits on

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions