Repository navigation
feat(runtime): engine-owned session cache; Flash-Next keeps many prompts warm (A/B/A 16.3 s → 4.9 s) - #825
Open
fivetide wants to merge 2 commits into
Open
feat(runtime): engine-owned session cache; Flash-Next keeps many prompts warm (A/B/A 16.3 s → 4.9 s)#825fivetide wants to merge 2 commits into
fivetide wants to merge 2 commits into
Conversation
…pts warm Add hipfire_runtime::session_cache: one cache per loaded model that owns keying (CacheDomain scoped per route/chunk/state format + prefix fingerprint), planning, LRU eviction (CheckpointPool, renamed from QwenCheckpointPool, + pop_lru), a memory guard (UMA: MemAvailable with 8 GiB headroom; discrete: free VRAM), placement (SnapshotLocation, device only for now) and every copy. Architectures implement SessionState and only describe their state parts; ArchModel gains session_cache_attached, session_plan and session_commit. Qwen4 (Flash-Next) moves onto it and drops its single whole-chunk prefix checkpoint. Snapshots are self-contained (GDN, PLE, hyper state plus QSA K/V, raw and pooled rows below the boundary; with MTP the head and draft policy), captured at every prefill-chunk boundary and published only on client commit, so several sessions/subagents stay warm and a reset no longer drops them. New config memory.session_cache_bytes / HIPFIRE_SESSION_CACHE_BYTES (default 8 GiB, 0 disables). Verification (Strix Halo gfx1151, qwen3.8-flash-next-gptq3.mq4 8b15b6fe): - session_cache_hw: two-chunk restore after an unrelated prompt, final logits byte-equal to cold, 16 greedy ids equal. - serve A(21550 tok)/B(8142)/A, greedy MTP: A#2 cached_tokens 16384, 16.3 s -> 4.9 s, identical content; cache off: cached 0, same content. - qwen4_mtp_fill warm (300,4096 after 1 chunk): plan == 8192. - serve_harness chain greedy, MTP on and off: 5/5 turns, identical text.
5 of 6 tasks
…; refresh crate maps and env inventory The formatter rewrote all of hipfire-daemon/src/main.rs (5392 -> 5477 lines, over the 5377 daemon_lines ceiling). Keep only the 3-line session-cache change. Regenerate the stale crate maps and add HIPFIRE_SESSION_CACHE_MODEL (session_cache_hw) to docs/env-vars.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flash-Next (Qwen4) now keeps many prompt prefixes warm at once. Before this change it had a single whole-chunk prefix checkpoint: any request with a different prefix, such as another session or a subagent, overwrote it. A new engine-owned cache in
hipfire-runtimeholds every reuse decision (which snapshot, eviction, memory guard, storage placement, copies). An architecture only describes which device byte ranges make up its state.hipfire-runtime + session_cache.rs # SessionCache, SessionState trait, SnapshotLocation (tier enum) checkpoint_pool.rs # QwenCheckpointPool → CheckpointPool, + pop_lru serve_contract.rs # + CacheDomain::for_model / scoped arch_model.rs # + session_cache_attached / session_plan / session_commit (defaulted) hipfire-arch-qwen4 state.rs, mtp_gpu.rs # prefix arenas deleted → session_parts / prepare_session_restore codecs bundle.rs # prefix cache deleted → impl SessionState for Qwen4Bundle mtp_spec.rs # reports boundaries; reset_after_unrepaired_terminal override deleted hipfire-config # + memory.session_cache_bytes (HIPFIRE_SESSION_CACHE_BYTES), default 8 GiB, 0 = offbegin.MemAvailable≥ snapshot + state growth + 8 GiB. Discrete GPUs require free VRAM ≥ snapshot + growth + 256 MiB. When the guard fails, the cache evicts LRU entries and then skips the capture.hipMalloc/hipFreedirectly, not the pool free list, so evictions return memory and the guard sees it.HIPFIRE_QWEN_PROMPT_CACHEno longer affects Qwen4. Qwen3.5 rings and the other arches keep their own code.ponytail:): every snapshot is a full copy from row 0. One 8192-token MTP snapshot is 513 MiB. Delta snapshots would cut that if budgets bind.Which surface(s) does this touch?
crates/hipfire-loader(Qwen4 carrier),crates/hipfire-daemon(cache-capable arm),hipfire-configcrates/hipfire-generate(Qwen4 AR + MTP plan/commit)hipfire-arch-qwen4,hipfire-arch-qwen35(pool rename only)Evidence
Strix Halo gfx1151, HIP 7.2,
qwen3.8-flash-next-gptq3.mq4sha2568b15b6fede7d7c5bfed0db4720a8295bedda51bc93e545fa242bd50d0f200972, daemon md502516ae6a393d82f1486e81c88468b0c.After: serve, greedy, MTP. A =
benchmarks/prompts/qwen38_issue693_longcode_20676.txt(21,550 tokens), B =deepseek4_mq2r_prose_ctx8192.txt(8,142 tokens):crates/hipfire-arch-qwen4/tests/session_cache_hw.rs(AR, F32 QSA, Q8 GDN, VMM). It prefills A cold, then an unrelated B, then restores A's two-chunk snapshot and prefills A again. The final logits are byte-equal to the cold run's, and 16 greedy ids are equal.qwen4_mtp_fill … 32768 300,4096 1. Both lengths restore the one-chunk snapshot, and the plan matchesstart_pos= 8192.serve_harness(greedy, thinking off, max_tokens 256):cargo build --releaseis clean.hipfire-runtimesession_cache(GPU toy state: exact restore, commit-gated publish, LRU eviction, over-budget skip) andcheckpoint_pool, plushipfire-configall pass.Test plan
./scripts/no-gpu-ci.shpasses, or the CI jobs are green. Locally it stops at pytest collection: the.venvnumpy can't loadlibstdc++.so.6, an environment problem. Its cargo stages before that passed, and its later steps pass when run by hand.cargo build --releasecleancargo test --lib --workspace: I ran only the changed crates.hipfire-arch-qwen4has one failure,compact_runner_reports_device_backed_state_parity_scenarios. It expects the substring "cache-hit suffix", which the unchanged message invalidate_native_mtp_prefill_requestdoesn't contain.serve_harness battery, MTP on
[ { "request_id": "chatcmpl-3206048-1", "ctx": 84, "cached": 0, "gen": 182, "finish": "stop", "think_words": 46, "ans_words": 46, "prefill_ms": 461.0, "prefill_tok_s": 182.2, "decode_tok_s": 47.0, "decode_estimated": false, "tau": 1.68, "cycles": 68, "dflash": null, "mtp": true, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.465, "wall_s": 4.337, "attractor": false, "empty": false, "runaway": false, "ans_preview": "```python\ndef merge_sorted(a, b):\n \"\"\"Merge two already-sorted lists into one sorted li", "assistant_content": "```python\ndef merge_sorted(a, b):\n \"\"\"Merge two already-sorted lists into one sorted list.\"\"\"\n result = []\n i = j = 0\n\n while i < len(a) and j < len(b):\n if a[i] <= b[j]:\n result.append(a[i])\n i += 1\n else:\n result.append(b[j])\n j += 1\n\n result.extend(a[i:])\n result.extend(b[j:])\n\n return result\n```", "content": "```python\ndef merge_sorted(a, b):\n \"\"\"Merge two already-sorted lists into one sorted list.\"\"\"\n result = []\n i = j = 0\n\n while i < len(a) and j < len(b):\n if a[i] <= b[j]:\n result.append(a[i])\n i += 1\n else:\n result.append(b[j])\n j += 1\n\n result.extend(a[i:])\n result.extend(b[j:])\n\n return result\n```", "reasoning_content": "We need answer user: \"Write a Python function `merge_sorted(a, b)` that merges two already-sorted lists into one sorted list without using sorted(). Include a short docstring.\"\n\nNeed provide code only likely. Need final concise. Ensure no sorted(). Could implement two-pointer. Maybe mention assumes ascending. Include docstring.\n", "tool_calls": [], "request_md5": "d132ee5dc813b137af73ebbd809fb5fb", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "43ca0d15712d3dfb777b51ae76d8fd5f", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3206048-2", "ctx": 95, "cached": 0, "gen": 171, "finish": "stop", "think_words": 20, "ans_words": 47, "prefill_ms": 452.7, "prefill_tok_s": 209.9, "decode_tok_s": 61.1, "decode_estimated": false, "tau": 2.21, "cycles": 53, "dflash": null, "mtp": true, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.455, "wall_s": 3.252, "attractor": false, "empty": false, "runaway": false, "ans_preview": "Distance = speed \u00d7 time.\n\n1. First part: \n \\(60 \\text{ mph} \\times 2.5 \\text{ hours} = ", "assistant_content": "Distance = speed \u00d7 time.\n\n1. First part: \n \\(60 \\text{ mph} \\times 2.5 \\text{ hours} = 150 \\text{ miles}\\)\n\n2. Second part: \n \\(40 \\text{ mph} \\times 1.5 \\text{ hours} = 60 \\text{ miles}\\)\n\n3. Total distance: \n \\(150 + 60 = 210 \\text{ miles}\\)\n\n**Final answer: 210 miles**", "content": "Distance = speed \u00d7 time.\n\n1. First part: \n \\(60 \\text{ mph} \\times 2.5 \\text{ hours} = 150 \\text{ miles}\\)\n\n2. Second part: \n \\(40 \\text{ mph} \\times 1.5 \\text{ hours} = 60 \\text{ miles}\\)\n\n3. Total distance: \n \\(150 + 60 = 210 \\text{ miles}\\)\n\n**Final answer: 210 miles**", "reasoning_content": "We need answer simple arithmetic. Need final with steps and final number. Compute: 60*2.5=150 miles. 40*1.5=60 miles. total 210 miles.\n", "tool_calls": [], "request_md5": "139df51d4b153d8f094d8c816de14f54", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "640e0fd4f55996cb175a422f0a12cef5", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3206048-3", "ctx": 65, "cached": 0, "gen": 144, "finish": "stop", "think_words": 73, "ans_words": 44, "prefill_ms": 359.1, "prefill_tok_s": 181.0, "decode_tok_s": 51.3, "decode_estimated": false, "tau": 1.57, "cycles": 56, "dflash": null, "mtp": true, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.364, "wall_s": 3.168, "attractor": false, "empty": false, "runaway": false, "ans_preview": "Earth\u2019s axis is tilted relative to its orbit around the Sun. This tilt causes different pa", "assistant_content": "Earth\u2019s axis is tilted relative to its orbit around the Sun. This tilt causes different parts of Earth to receive varying amounts of sunlight during the year. When a hemisphere is tilted toward the Sun, it experiences summer; when tilted away, it experiences winter.", "content": "Earth\u2019s axis is tilted relative to its orbit around the Sun. This tilt causes different parts of Earth to receive varying amounts of sunlight during the year. When a hemisphere is tilted toward the Sun, it experiences summer; when tilted away, it experiences winter.", "reasoning_content": "We need to answer user: \"What causes the seasons on Earth? Answer in exactly three sentences.\" Need exactly three sentences. Simple. Ensure no extra sentences. Could say: Earth's axis is tilted relative to its orbit around the Sun. This tilt causes different parts of Earth to receive varying amounts of sunlight during the year. When a hemisphere is tilted toward the Sun, it experiences summer; when tilted away, it experiences winter. Three sentences.\n", "tool_calls": [], "request_md5": "40cbc2ba3f86735cef065887cf37ddda", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "8f66b4c97988825bd8e7840aaf44357e", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3206048-4", "ctx": 73, "cached": 0, "gen": 134, "finish": "stop", "think_words": 36, "ans_words": 69, "prefill_ms": 375.8, "prefill_tok_s": 194.3, "decode_tok_s": 34.1, "decode_estimated": false, "tau": 1.0, "cycles": 67, "dflash": null, "mtp": true, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.38, "wall_s": 4.305, "attractor": false, "empty": false, "runaway": false, "ans_preview": "Every morning, Elias climbed the lighthouse stairs and swept his gaze across the gray sea.", "assistant_content": "Every morning, Elias climbed the lighthouse stairs and swept his gaze across the gray sea. One dawn, he spotted a glass bottle wedged between the rocks, its cork sealed with red wax. Inside was not a message, but a tiny brass key that glowed faintly in the mist. When he turned it in the lighthouse\u2019s oldest door, the lamp above him lit itself for the first time in years.", "content": "Every morning, Elias climbed the lighthouse stairs and swept his gaze across the gray sea. One dawn, he spotted a glass bottle wedged between the rocks, its cork sealed with red wax. Inside was not a message, but a tiny brass key that glowed faintly in the mist. When he turned it in the lighthouse\u2019s oldest door, the lamp above him lit itself for the first time in years.", "reasoning_content": "We need to respond to user: \"Write a four-sentence story about a lighthouse keeper who finds something unexpected washed up on the rocks.\" Simple. Need final only four sentences. Ensure exactly four sentences. Could be concise.\n", "tool_calls": [], "request_md5": "653264a80e34fe3a32c4a09076c87d98", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "8fe0ad36f61bcf4992cc9df81cdf3817", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3206048-5", "ctx": 71, "cached": 0, "gen": 121, "finish": "stop", "think_words": 34, "ans_words": 55, "prefill_ms": 372.9, "prefill_tok_s": 190.4, "decode_tok_s": 39.7, "decode_estimated": false, "tau": 1.2, "cycles": 55, "dflash": null, "mtp": true, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.377, "wall_s": 3.424, "attractor": false, "empty": false, "runaway": false, "ans_preview": "1. Use clear, descriptive names for variables, functions, and classes.\n2. Keep functions s", "assistant_content": "1. Use clear, descriptive names for variables, functions, and classes.\n2. Keep functions small and focused on a single responsibility.\n3. Write meaningful comments only where they add value, not to explain obvious code.\n4. Follow consistent formatting and style conventions across the codebase.\n5. Add tests to protect behavior and make future changes safer.", "content": "1. Use clear, descriptive names for variables, functions, and classes.\n2. Keep functions small and focused on a single responsibility.\n3. Write meaningful comments only where they add value, not to explain obvious code.\n4. Follow consistent formatting and style conventions across the codebase.\n5. Add tests to protect behavior and make future changes safer.", "reasoning_content": "We need to respond to user: \"List exactly five tips for writing maintainable code, as a numbered list, one line each.\" Need final exactly five numbered lines. Ensure one line each. Could be concise.\n", "tool_calls": [], "request_md5": "734a4c6d0632eed48dbe7364c78af176", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "8bed8e2d056dc1d47dccae9d32dbecf4", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null } ]serve_harness battery, MTP off
[ { "request_id": "chatcmpl-3196254-1", "ctx": 84, "cached": 0, "gen": 182, "finish": "stop", "think_words": 46, "ans_words": 46, "prefill_ms": 421.6, "prefill_tok_s": 199.2, "decode_tok_s": 33.5, "decode_estimated": false, "tau": null, "cycles": null, "dflash": null, "mtp": null, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.44, "wall_s": 5.863, "attractor": false, "empty": false, "runaway": false, "ans_preview": "```python\ndef merge_sorted(a, b):\n \"\"\"Merge two already-sorted lists into one sorted li", "assistant_content": "```python\ndef merge_sorted(a, b):\n \"\"\"Merge two already-sorted lists into one sorted list.\"\"\"\n result = []\n i = j = 0\n\n while i < len(a) and j < len(b):\n if a[i] <= b[j]:\n result.append(a[i])\n i += 1\n else:\n result.append(b[j])\n j += 1\n\n result.extend(a[i:])\n result.extend(b[j:])\n\n return result\n```", "content": "```python\ndef merge_sorted(a, b):\n \"\"\"Merge two already-sorted lists into one sorted list.\"\"\"\n result = []\n i = j = 0\n\n while i < len(a) and j < len(b):\n if a[i] <= b[j]:\n result.append(a[i])\n i += 1\n else:\n result.append(b[j])\n j += 1\n\n result.extend(a[i:])\n result.extend(b[j:])\n\n return result\n```", "reasoning_content": "We need answer user: \"Write a Python function `merge_sorted(a, b)` that merges two already-sorted lists into one sorted list without using sorted(). Include a short docstring.\"\n\nNeed provide code only likely. Need final concise. Ensure no sorted(). Could implement two-pointer. Maybe mention assumes ascending. Include docstring.\n", "tool_calls": [], "request_md5": "d132ee5dc813b137af73ebbd809fb5fb", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "43ca0d15712d3dfb777b51ae76d8fd5f", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3196254-2", "ctx": 95, "cached": 0, "gen": 171, "finish": "stop", "think_words": 20, "ans_words": 47, "prefill_ms": 458.5, "prefill_tok_s": 207.2, "decode_tok_s": 33.9, "decode_estimated": false, "tau": null, "cycles": null, "dflash": null, "mtp": null, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.466, "wall_s": 5.507, "attractor": false, "empty": false, "runaway": false, "ans_preview": "Distance = speed \u00d7 time.\n\n1. First part: \n \\(60 \\text{ mph} \\times 2.5 \\text{ hours} = ", "assistant_content": "Distance = speed \u00d7 time.\n\n1. First part: \n \\(60 \\text{ mph} \\times 2.5 \\text{ hours} = 150 \\text{ miles}\\)\n\n2. Second part: \n \\(40 \\text{ mph} \\times 1.5 \\text{ hours} = 60 \\text{ miles}\\)\n\n3. Total distance: \n \\(150 + 60 = 210 \\text{ miles}\\)\n\n**Final answer: 210 miles**", "content": "Distance = speed \u00d7 time.\n\n1. First part: \n \\(60 \\text{ mph} \\times 2.5 \\text{ hours} = 150 \\text{ miles}\\)\n\n2. Second part: \n \\(40 \\text{ mph} \\times 1.5 \\text{ hours} = 60 \\text{ miles}\\)\n\n3. Total distance: \n \\(150 + 60 = 210 \\text{ miles}\\)\n\n**Final answer: 210 miles**", "reasoning_content": "We need answer simple arithmetic. Need final with steps and final number. Compute: 60*2.5=150 miles. 40*1.5=60 miles. total 210 miles.\n", "tool_calls": [], "request_md5": "139df51d4b153d8f094d8c816de14f54", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "640e0fd4f55996cb175a422f0a12cef5", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3196254-3", "ctx": 65, "cached": 0, "gen": 144, "finish": "stop", "think_words": 73, "ans_words": 44, "prefill_ms": 376.0, "prefill_tok_s": 172.9, "decode_tok_s": 33.3, "decode_estimated": false, "tau": null, "cycles": null, "dflash": null, "mtp": null, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.388, "wall_s": 4.708, "attractor": false, "empty": false, "runaway": false, "ans_preview": "Earth\u2019s axis is tilted relative to its orbit around the Sun. This tilt causes different pa", "assistant_content": "Earth\u2019s axis is tilted relative to its orbit around the Sun. This tilt causes different parts of Earth to receive varying amounts of sunlight during the year. When a hemisphere is tilted toward the Sun, it experiences summer; when tilted away, it experiences winter.", "content": "Earth\u2019s axis is tilted relative to its orbit around the Sun. This tilt causes different parts of Earth to receive varying amounts of sunlight during the year. When a hemisphere is tilted toward the Sun, it experiences summer; when tilted away, it experiences winter.", "reasoning_content": "We need to answer user: \"What causes the seasons on Earth? Answer in exactly three sentences.\" Need exactly three sentences. Simple. Ensure no extra sentences. Could say: Earth's axis is tilted relative to its orbit around the Sun. This tilt causes different parts of Earth to receive varying amounts of sunlight during the year. When a hemisphere is tilted toward the Sun, it experiences summer; when tilted away, it experiences winter. Three sentences.\n", "tool_calls": [], "request_md5": "40cbc2ba3f86735cef065887cf37ddda", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "8f66b4c97988825bd8e7840aaf44357e", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3196254-4", "ctx": 73, "cached": 0, "gen": 134, "finish": "stop", "think_words": 36, "ans_words": 69, "prefill_ms": 392.5, "prefill_tok_s": 186.0, "decode_tok_s": 33.4, "decode_estimated": false, "tau": null, "cycles": null, "dflash": null, "mtp": null, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.404, "wall_s": 4.415, "attractor": false, "empty": false, "runaway": false, "ans_preview": "Every morning, Elias climbed the lighthouse stairs and swept his gaze across the gray sea.", "assistant_content": "Every morning, Elias climbed the lighthouse stairs and swept his gaze across the gray sea. One dawn, he spotted a glass bottle wedged between the rocks, its cork sealed with red wax. Inside was not a message, but a tiny brass key that glowed faintly in the mist. When he turned it in the lighthouse\u2019s oldest door, the lamp above him lit itself for the first time in years.", "content": "Every morning, Elias climbed the lighthouse stairs and swept his gaze across the gray sea. One dawn, he spotted a glass bottle wedged between the rocks, its cork sealed with red wax. Inside was not a message, but a tiny brass key that glowed faintly in the mist. When he turned it in the lighthouse\u2019s oldest door, the lamp above him lit itself for the first time in years.", "reasoning_content": "We need to respond to user: \"Write a four-sentence story about a lighthouse keeper who finds something unexpected washed up on the rocks.\" Simple. Need final only four sentences. Ensure exactly four sentences. Could be concise.\n", "tool_calls": [], "request_md5": "653264a80e34fe3a32c4a09076c87d98", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "8fe0ad36f61bcf4992cc9df81cdf3817", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null }, { "request_id": "chatcmpl-3196254-5", "ctx": 71, "cached": 0, "gen": 121, "finish": "stop", "think_words": 34, "ans_words": 55, "prefill_ms": 385.5, "prefill_tok_s": 184.2, "decode_tok_s": 33.4, "decode_estimated": false, "tau": null, "cycles": null, "dflash": null, "mtp": null, "mtp_ngram": null, "ngram_mod_windows": null, "ngram_mod_drafts": null, "ngram_mod_accepted": null, "ngram_mod_accept_rate": null, "mtp_windows": null, "ar_windows": null, "mtp_retired": null, "mtp_window_timings": null, "ttft_s": 0.395, "wall_s": 4.009, "attractor": false, "empty": false, "runaway": false, "ans_preview": "1. Use clear, descriptive names for variables, functions, and classes.\n2. Keep functions s", "assistant_content": "1. Use clear, descriptive names for variables, functions, and classes.\n2. Keep functions small and focused on a single responsibility.\n3. Write meaningful comments only where they add value, not to explain obvious code.\n4. Follow consistent formatting and style conventions across the codebase.\n5. Add tests to protect behavior and make future changes safer.", "content": "1. Use clear, descriptive names for variables, functions, and classes.\n2. Keep functions small and focused on a single responsibility.\n3. Write meaningful comments only where they add value, not to explain obvious code.\n4. Follow consistent formatting and style conventions across the codebase.\n5. Add tests to protect behavior and make future changes safer.", "reasoning_content": "We need to respond to user: \"List exactly five tips for writing maintainable code, as a numbered list, one line each.\" Need final exactly five numbered lines. Ensure one line each. Could be concise.\n", "tool_calls": [], "request_md5": "734a4c6d0632eed48dbe7364c78af176", "atem_leak": false, "terminal_count": 1, "terminal_reasons": [ "stop" ], "post_terminal_bytes": 0, "saw_done": true, "stream_error": null, "prompt_md5": "8bed8e2d056dc1d47dccae9d32dbecf4", "expected_substrings": [], "retrieval_missing": [], "kv_backend": null, "kv_backend_legacy": false, "kv_backend_warning": null, "kv_backend_reason": null, "kv_mode": null } ]Merge Danger
Door: two-way. Setting
memory.session_cache_bytes = 0turns the cache off at runtime. A revert restores the old single-checkpoint code. No on-disk format changes.Blast Radius: Flash-Next serving.
MemAvailableheadroom per capture, or it skips the capture. Load still reserves room for one snapshot, as before.ArchModelgains three defaulted methods, so other arches are unchanged. TheCheckpointPoolrename touches Qwen3.5serve_engine/checkpoint(name only).scripts/fmt-changed.shreformatted whole touched files, so the diff includes earlier formatting debt (mostly inserve_engine.rs,daemon/main.rs,bundle.rs).Architecture-trait change?
No
Architecturetrait change inarch.rs.ArchModel(arch_model.rs) gainssession_cache_attached,session_planandsession_commit, all defaulted.