RFC: MoE expert cache, VRAM caching of hot CPU-resident experts with hybrid hit/miss execution

[leloch]

Follow-up to #24524, which was closed as too large to review — @am17an suggested
an RFC laying out the benefits vs the maintenance burden, and explicitly suggested
having the AI draft it. So, full disclosure: this document and the code it
describes were predominantly generated by an AI (Anthropic's Claude Fable 5),
working under my direction on my hardware. Code: leloch:moe-cache-pr.

Problem

When a MoE model spills experts to system RAM (--cpu-moe / --n-cpu-moe /
auto-fit), decode is dominated by the CPU reading expert weights at RAM
bandwidth while the GPUs idle. Prior attempts (#20757, #21609, #21614, #21620,
#23170) all moved MUL_MAT_ID to the GPU and tried to make the weight copies
cheaper. That puts every cache miss on the critical path as a synchronous PCIe
transfer: @batot1 measured ~3× decode regression from forced decode offload
without residency (#20757), and the Metal slot-pool experiment in the same
thread was 2× slower than vanilla even at 97–99% hit rate, purely from
per-layer sync points. #23170's post-mortem concluded the staging-buffer
approach is a no-op when made correct, and that the fix is a separate
persistent expert-cache buffer with explicit expert→slot bookkeeping.

On the discussions side, #22584 proposes the static version of this idea
(offline expert profiling + strategic placement), #19030 and PR #21067 the
prefetch version (latency-hiding without residency), and #22183 asks the
placement question a runtime cache answers automatically ("use the idle fast
device for experts") — but no runtime expert cache has been proposed as an
RFC before.

Proposed design

That persistent buffer, plus an inverted execution model: MUL_MAT_ID stays on
the CPU. Inside the CPU kernel, thread 0 dispatches one batched matvec over
the cached (hit) rows on the GPU while the other threads compute the miss rows
as they normally would. Misses cost nothing extra, hit rate is pure upside,
worst case degrades to the vanilla CPU path. Fill is decode-only (prompt
routing is far flatter and thrashes the cache); decode routing is skewed
enough to make this work — measured on Qwen3.5-122B, the top 10% of experts
take ~80% of hits (independent corroboration in #20757: Gini ≈ 0.76, ~99%
simulated hit rate at 69% expert budget). Complementary to #21067: prefetch
hides cold transfers at large ubatch (prompt); this removes hot
re-transfers at ubatch 1 (decode).

Measured benefit

4× RTX 3090, EPYC 7R13 48c, 8-ch DDR4-3200 256GB; llama-bench tg300, identical command
lines both arms:

Model	Regime	cache	vanilla	gain
GLM-5.1 754B IQ2_M (220 GB)	zero-config	17.49	13.96	+25%
Qwen3.5 397B Q3_K_XL (167 GB)	zero-config	30.25	28.18	+7%
Qwen3.5 122B (fits in VRAM)	zero-config	74.98	74.98	parity (cache dormant)
13 models / 9 architectures	forced -ngl 99 -ncmoe 99			+10%…+57%, 16/16 ≥ parity

Quality: decode-path perplexity statistically identical to pure CPU;
prompt/batch path bit-untouched; test-backend-ops MUL_MAT_ID 789/789 both
modes. (GPU rounding can flip a near-tie token under greedy decoding, same
class as any -ngl change.)

Maintenance burden:

Where the code lives. ~1,700 lines in one new CUDA file pair
(moe-cache.cu/.cuh); a backend-agnostic function-pointer API table
(ggml-backend-moe-cache.h); small null-checked hooks in ggml-cpu.c/ops.cpp
(mul_mat_id + GLU) and ggml-backend.cpp (scheduler offer/redirect); a fit
rule + --moe-cache flag in common. Non-CUDA builds see a zero-initialized
table and no-op hooks; --moe-cache 0 restores stock behavior exactly.

The real costs

1. It hooks the CPU mul_mat_id hot path. The hooks are null-checked
   pointers, but anyone refactoring the CPU MoE kernels or the scheduler
   split logic now has a second consumer to think about. This is the largest
   structural cost.
2. Heuristics that maintainers didn't choose. Shape-census stability
   window, baseline-sampled bail-out (EWMA, 4 strikes at +5%), VRAM reserve
   default, decode-only fill. They're measured, but they're policy, and
   policy invites tuning requests.
3. Hard to test in CI. The cache only engages on spilling MoE models
   during decode; CI can verify it stays dormant and that the dispatch path
   is numerically correct (a built-in selftest does this model-free), but the
   perf claims need big-RAM multi-GPU hardware. Realistically, regressions
   would be caught by users, not CI.
4. CUDA-only. The API table is backend-agnostic, but today only CUDA
   registers it; Metal/Vulkan users will ask.
5. State. A persistent hot-set file (~/.cache/llama.cpp/) and an
   admission/eviction state machine — more surface than a stateless
   optimization.

What would reduce the burden, if there's interest:

• A minimal core (plain per-node dispatch only: no fused gate+up+SwiGLU,
  no GPU-resident handoff, no hot-set persistence) is roughly half the code
  and keeps most of the gain; I can measure each layer's exact contribution
  on this hardware.
• Landing the API table + CPU hooks alone first (~small, inert without a
  backend registration) would let any backend implement caching out-of-tree
  before committing to the CUDA implementation.
• Or: nobody merges anything, and this stays a documented design +
  benchmark set that informs ggml: allow prefetching tensor overrides #21067 or a future maintainer-built version.
  That's a fine outcome too.

Offer

The 4×3090 / 48-core / 8-channel 256GB DDR4-3200 box is available for any benchmark,
ablation, or A/B against #21067 that would help evaluate this. All results
above are reproducible with the commands in the branch's commit messages.

[Lidenburg]

I've been spending some time implementing MoE expert caching as well, except offloading all matmuls to the GPU, mainly for fun to see how high I could make the t/s number go, I have no intention of submitting a PR with production code or to own the code or maintain it.

I have uploaded the code to a new fork at https://github.com/Lidenburg/llama.cpp incase anyone wants to use it as a reference point. I tried to keep my changes to a single file (ggml-backend.cpp) to make porting it easier and to make it easier to understand. The code is partially LLM generated and partially hand-written, but the theories and strategies tested were all manually guided. That said I wouldn't recommend manually reading the code.

In general I saw around ~60% performance gain using expert caching with the same VRAM usage as when using the --n-cpu-moe option. There's some comparisons in the README in the fork.

My goal was to try to get larger models (100b+) running at reasonable speeds on my 32gb + 16gb system, which i still think is probably achievable, but not using the spare SATA SSD that I had lying around. I would be curious to see what others with faster SSDs can get out of it in terms of t/s.

[batot1]

MoE cache regression test on GTX 1080 Ti

Comment intended for ggml-org/llama.cpp Discussion #24528
Branch tested: leloch/llama.cpp, moe-cache-pr

Summary

I tested the moe-cache-pr branch on a much smaller and older setup than the 4× RTX 3090 system described in the RFC.

This may simply be outside the hardware regime where the cache is expected to help, but on my GTX 1080 Ti single-GPU setup I consistently observed a regression versus a hard-disabled MoE cache path.

The useful result here is the single-GPU GTX 1080 Ti sweep. A dual-GPU attempt was also made, but that run is not a valid MoE-cache benchmark because the binary was built only for sm_61, while the first visible CUDA device was sm_75.

Tested branch

Item	Value
Repository	leloch/llama.cpp
Branch	moe-cache-pr
HEAD	8853f0535
Commit title	common: MoE-cache-aware fit placement and --moe-cache flag

Model

/home/bartosz/models/qwen36/Qwen3.6-35B-A3B-UDT-Q8_K_XL_MTP.gguf

Single-GPU hardware

Item	Value
GPU	NVIDIA GeForce GTX 1080 Ti
VRAM	11 GB
CUDA compute capability	6.1
Runtime device selection	CUDA_VISIBLE_DEVICES=1
Build CUDA arch	CMAKE_CUDA_ARCHITECTURES=61-real

Common runtime parameters

-c 8192
-n 512
-t 6
-b 512
-ub 512
-ngl all
-sm none
-fa on
-ctk q8_0
-ctv q8_0
--cpu-moe
--cache-ram 0
GGML_CUDA_MOE_CACHE_HOTSET=0

Decode throughput results

Mode	MoE cache budget	TG tok/s	Relative to hard OFF	Result
hard OFF	0 MB	19.32	100.0%	baseline
ON	4096 MB	13.25	68.6%	regression
ON	1024 MB	14.38	74.4%	regression
ON	512 MB	17.26	89.3%	regression
ON	256 MB	17.19	89.0%	regression
ON	128 MB	18.64	96.5%	regression
ON	64 MB	18.71	96.9%	regression
ON	32 MB	18.72	96.9%	regression

Throughput trend

hard OFF   -> 19.32 tok/s

4096 MB    -> 13.25 tok/s
1024 MB    -> 14.38 tok/s
512 MB     -> 17.26 tok/s
256 MB     -> 17.19 tok/s
128 MB     -> 18.64 tok/s
64 MB      -> 18.71 tok/s
32 MB      -> 18.72 tok/s

On this setup, larger cache budgets caused larger regressions. Smaller budgets reduced the regression, but none of the tested cache-enabled configurations outperformed the hard-disabled cache path.

CLI / control observations

--moe-cache auto

--moe-cache auto failed with:

error while handling argument "--moe-cache": stoi

The help text suggests that auto is a valid/default mode, but the CLI argument parser appears to parse the value as an integer.

--moe-cache 0 versus hard OFF

In earlier runs, --moe-cache 0 did not appear to behave like a reliable hard-disabled cache path. The reliable hard-OFF configuration was:

GGML_CUDA_MOE_CACHE=0
GGML_CUDA_MOE_CACHE_HOTSET=0

The hard-OFF result was:

eval time = 26500.62 ms / 512 tokens
TG        = 19.32 tok/s

Bail-out / trim observations

Some cache-enabled runs logged bail-out / trim messages such as:

[moe-cache] bail-out: cache-engaged nodes average ... vs ... pure-CPU — disabling the cache and freeing its VRAM for this run
[moe-cache] dev=0 TRIMMED ... MB under VRAM pressure — cache off on this device

Examples:

Budget	Relevant log observation	TG tok/s
1024 MB	cache-engaged nodes average 285us vs 266us pure-CPU + TRIMMED 1022 MB	14.38
512 MB	cache-engaged nodes average 290us vs 261us pure-CPU + TRIMMED 510 MB	17.26
256 MB	cache-engaged nodes average 310us vs 263us pure-CPU + TRIMMED 255 MB	17.19

Even after bail-out / trim, throughput did not return to the hard-OFF baseline.

Dual-GPU attempt

I also attempted a dual-GPU run with both GPUs visible:

CUDA device	GPU	Compute capability
CUDA0	GTX 1660 Ti	7.5
CUDA1	GTX 1080 Ti	6.1

The binary was built only for sm_61:

CUDA : ARCHS = 610

The run failed during warmup with:

CUDA error: no kernel image is available for execution on the device
current device: 0
ggml_cuda_kernel_can_use_pdl

This dual-GPU attempt is not a valid MoE-cache benchmark. The likely reason is that CUDA device 0 was the GTX 1660 Ti (sm_75), while the tested binary only contained kernels for sm_61.

A proper dual-GPU test would require a binary built for both architectures, for example sm_61 and sm_75. I did not include such a result here to avoid mixing the MoE-cache behavior with a separate dual-architecture build variable.

Conclusion

On this GTX 1080 Ti single-GPU setup, MoE cache regresses decode throughput versus:

GGML_CUDA_MOE_CACHE=0
GGML_CUDA_MOE_CACHE_HOTSET=0

This does not disprove the reported 4× RTX 3090 results from the RFC. It may indicate that the current cache path is beneficial only in a specific hardware/model regime, while on smaller or older GPUs the overhead can dominate.

The main actionable observations are:

1. --moe-cache auto fails with stoi.
2. --moe-cache 0 did not appear equivalent to GGML_CUDA_MOE_CACHE=0 in my tests.
3. On GTX 1080 Ti, cache-enabled runs regress versus hard OFF.
4. Larger cache budgets cause larger regressions.
5. Bail-out / trim does not restore hard-OFF throughput.
6. A dual-GPU test with mixed sm_75 + sm_61 requires a binary built for both architectures; the sm_61-only binary cannot benchmark that setup correctly.

[leloch]

Thanks for testing, the improvement is indeed quite hardware dependent.

I imagine on a relatively slow GPU without tensor cores (like 1080), and not much spare VRAM after placing all dense layers on a GPU, for MoE it could be just faster to compute expert activation on CPU + RAM (depending on the exact CPU and RAM bandwidth available), as there is some overhead on the cache path.

Ideally you want enough VRAM spare for cache (after placing ALL dense layers on a GPU - otherwise there is no point), that you can fit experts working set there (so around 20%-30% of MoE expert weights).

[BusinessBuilders]

Thank you for your work this sped up my 8x 3090 256 ddr 4 unsloth GLM 5.2 q2 quant.
Still Experimenting.

[batot1]

No problem.
If you need to test your code on my smaller stack in the future, you can always contact me and ask for tests.

I'm testing various solutions out of curiosity to see if they'll help improve the model's performance on my stack, but so far everything is running slower.
I also have my own version of HyperPP, where I managed to significantly boost the model from the initial PPmax=230 tok/s to 480~500 tok/s.
Once the session is over, I'll finish the documentation and release the patch, as it gave me a huge boost.
In fact, I don't know which of my fixes caused the significant boost, as I made many fixes, and none of them resulted in a significant boost of >10%. Only after several days of searching for the cause did I discover that what initially didn't work—increasing the batch size to 4096—suddenly resulted in an incredible boost in my case. Apparently, suboptimal code in the program was artificially stifling the gain from having a batch size larger than 512, and after many optimizations, the additional power in the PP suddenly unlocked. I wanted the PP because it runs on a 256k context, and generation took a very long time with the baseline code.

[noonghunna]

Independent validation datapoint, since the RFC asks for benefits-vs-maintenance-burden and that's a hard case to make from one rig.

To be clear up front: the cache is @leloch's work. We're users — we ported the branch onto a downstream fork and have been running it as a daily driver on 2× RTX 3090 for weeks. Nothing below is our mechanism; it's our measurements of his.

What it buys, on hardware that isn't leloch's

Two large MoEs with CPU-resident experts (a 118B and a 284B), 2× 3090 + 8-channel DDR4. Full write-up with method and the raw tables is here; the parts relevant to a merge decision:

It scales, rather than being a narrow trick. Five configs in one sitting, varying pool size and expert quant:

coverage	hit rate	decode t/s	first token
11.7%	~50%	17.2	0.28–0.64 s
18.7%	~55%	~17.7	0.27–0.29 s
25.5%	~63%	19.2	0.27–0.31 s
30.6%	~70%	~21.9	0.21–0.23 s

Coverage → hits → speed stayed essentially linear as far as we could push it. We had predicted early saturation (expert traffic is heavily skewed, so a small pool "should" catch the hot head) and were wrong.

The latency win is bigger than the throughput win, which surprised us and may matter more for interactive use: first-token time rides the CPU miss path almost entirely, and fell ~0.6 s → ~0.21 s across those configs.

The argument I'd lead with, though, is that it does something hardware can't buy. We upgraded the host from 4- to 8-channel DDR4 — STREAM Triad ~57 → ~100 GB/s, a 75% bandwidth increase — and decode moved +6%. Not because the workload isn't bandwidth-shaped, but because the cache had already absorbed that wall: it converts a RAM-bandwidth problem into a PCIe-latency one, so by the time you widen the memory bus you're widening a road carrying a fraction of the original traffic. Anyone running large MoEs on consumer GPUs is otherwise told to go buy memory bandwidth. This is the software substitute, and on our rig it was worth far more than the hardware was.

On maintenance burden — the honest other half

• It ports. We cherry-picked the series onto a fork that had diverged substantially from the base leloch targeted. It applied in four commits; conflicts were real but localized (ggml-cuda.cu device enumeration, arg.cpp includes, fit.cpp placement). So this isn't code that only survives on one tree — but it isn't free to carry either, which is exactly the argument for it living upstream rather than in N downstream forks.
• One rough edge worth fixing before any merge: if the batch size exceeds the cache's max batch, the cache silently refuses every decode with no diagnostic — you just get slow. It cost us two full benchmark runs that we initially read as −30% and −48% regressions from unrelated changes. A single warning on that path removes the whole failure class; we carry a patch for it downstream and would happily send it to leloch or upstream if the feature lands.
• We also hit the "warm before you judge" trap: bigger pools need proportionally more varied traffic before their hit rate means anything, and an under-warmed cache reads as saturated. Worth a line in any docs that ship with it.

Happy to re-run anything specific on sm_86 if it would help the review, or to provide numbers on a config a maintainer wants to see. And if any of this is more useful as a PR-side comment than an RFC comment, say the word.

[SharkWipf]

I have a similar, local, vibe coded implementation. Mine is buggy, is messy, it is inefficient and unoptimized, but in spite of that, it performs better on a single GPU now for large MoE models than it did across 3 GPUs without these changes. I've seen a 30-50% performance increase on GLM 5.2 on my shoddy implementation despite going from 3 GPUs to 1 (though the 1 GPU is a RTX 6000 and the platform is a threadripper).
My implementation is in no shape to be upstreamed and is tailored specifically to my setup, but it does show that the concept definitely has significant merit.

(In my setup, using the other GPUs in this implementation as extended cache actually reduced performance, because of inefficient code. Though the benefits would be relatively small anyway, the top ~1000 most frequently hit experts per conversation thread seem to be the most important ones at least on GLM 5.2, after that it seems to fall off rapidly)

[noonghunna]

@SharkWipf — the "top ~1000 experts matter, then it falls off rapidly" observation is worth pushing on, because we believed exactly that and it turned out to be a measurement artifact.

Our first pass concluded the cache saturated early, for the same intuitive reason: expert traffic is heavily skewed, so a small pool "should" capture the hot head. When we re-ran it properly the curve was linear all the way to ~31% coverage (~10,000 experts) with no knee — the earlier result was an under-warmed pool. Bigger pools need proportionally more varied traffic before their hit rate stabilises, and an under-warmed cache reads exactly like a saturated one. If your falloff was measured over a short run, it's worth re-testing with substantially more varied traffic before trusting the shape. Different model (GLM 5.2 vs DeepSeek), so it may genuinely differ — but the failure mode is easy to hit and we hit it publicly.

Since posting, I ran the control that RFC was missing: the same model on stock upstream, with no cache at all.

No-cache control — stock b10236, nothing patched

DeepSeek-V4-Flash-0731 (284B MoE, Q8_K_XL) on 2× RTX 3090, routed experts on CPU via -ot, shared expert + attention + router GPU-resident, 64K ctx, f16 KV, single stream. Identical placement in both arms; the only difference is the cache.

Arm	Decode t/s	settled over
stock upstream, no cache	14.17	n=3 (range 14.07–14.29)
leloch's cache (pool ~12.5 GiB/device — see follow-up; the requested 20 GB is VRAM-capped)	18.15	n=4 (range 17.81–18.44)
	+28.0%

The warm-up ramp is itself evidence for the mechanism

Cell-by-cell decode, same prompts, same order:

Arm
with cache	15.05	16.87	17.81	18.08	18.44	18.27
no cache	14.12	14.36	14.07	14.29	14.15	—

The cached arm climbs ~22% across its first five requests then plateaus. The no-cache arm is flat from request one. An engine with no pool has nothing to warm and shows nothing — so the ramp isn't thermal drift, clock behaviour or noise, it's the pool populating. That's also the cleanest demonstration of the under-warming trap above: anyone benchmarking this feature with a handful of requests will measure the left end of that ramp and conclude it doesn't help much.

One maintenance-burden-relevant fact: the blast radius is decode-only

Prefill, same two arms, same prompts, same offsets:

Arm	Prefill @10k	Prefill @40k
stock upstream, no cache	392.9 t/s (n=6)	366.6 t/s (n=3, 365.6–367.6)
leloch's cache (pool ~12.5 GiB/device)	within 0.3% of it	within 0.6% of it

The cache neither helps nor hurts prefill. That's the expected shape — prefill streams experts in
bulk, while the cache serves decode-time fetches — but it's worth stating with numbers rather than
as an assertion, because it means this feature cannot regress prefill. The surface that needs
re-validating on a merge is materially narrower than the diff size suggests: one regime, not two.

Separately, we're measuring speculative decoding on the same carrier as an orthogonal lever on the same bottleneck; we have not yet been able to measure the two composed, since that needs both in one build. Not an argument against the cache — just flagging that the two interact and we don't yet know how.

Method and raw tables for everything above: club-3090 #840. And again — the mechanism is @leloch's; we only ran it.

[noonghunna]

Follow-up with a capacity sweep, plus two corrections to my own numbers above and a partial concession to @SharkWipf.

Correction 1 — I understated the cache, and mislabelled the pool

My control above was stock upstream vs our build with the cache. That's cross-engine, and it turns out our fork's cacheless decode is ~4% slower than stock upstream's, so it flattered the baseline. Re-run properly — same binary, --moe-cache off vs on, nothing else changed:

control	decode	cache uplift
stock upstream, no cache (cross-engine)	14.17	+28.0%
same binary, --moe-cache off (single-variable)	13.61	+33.4%

Also: where I wrote "20 GB pool", the pool was actually ~12.5 GiB/device — the request is silently capped by available VRAM (see footgun 2).

The capacity sweep

Same binary, same model, serving tuning env fixed; only the budget varies. 43 layers × 256 experts, 6 active.

--moe-cache	actual pool/device	slots	hit rate	decode	vs off	marginal t/s per GB
off	—	—	—	13.61	—	—
4096	4,088 MiB	962	~53%	15.84	+16.4%	0.56
10240	10,234 MiB	2,408	~76%	17.58	+29.2%	0.29
20480	12,563 (capped)	2,956	~80.6%	17.66	+29.8%	0.01

Correction 2 — @SharkWipf, your falloff is probably real

I implied your "top ~1000 experts, then it falls off" was likely the under-warming artifact we hit. Our own curve shows the falloff too: per-GB returns drop 0.56 → 0.29 → 0.01. Where I'd still push back is the interpretation, not the observation — because our flat top is not the algorithm saturating:

• the pool caps at ~12.5 GiB/device on 24 GB cards, holding ~5,800 slots against a routed-expert population of roughly 10.7k, i.e. ~54% coverage;
• the two devices get different pool sizes (12,563 vs 12,146) tracking an uneven layer split — that's a VRAM signature, not census exhaustion.

So we ran out of hardware before we ran out of curve, and can't distinguish "coverage saturates" from "we hit the wall". Both our earlier near-linear result and your rapid falloff may be different segments of the same curve. On a card with more spare VRAM this is a genuinely open question, and worth someone measuring.

Two footguns for anyone benchmarking this feature

1. --moe-cache defaults to auto, which is ON. Omitting the flag does not give you a control arm — you get the cache in repacking-preserving mode. The zero arm must pass off (or 0) explicitly. An A/B done by dropping the flag measures cache-vs-cache and concludes the feature does nothing. Worth re-checking any "no improvement" result against this, including the earlier GTX 1080 Ti report.

2. The budget you request is not the budget you get. Ask for 20480 MiB, get 12,563 — silently. If you're sweeping pool sizes, read the actual [moe-cache] … pool[0]: … slots=N total=M MiB line rather than trusting your own flag; two of my "different" arms would otherwise have been the same arm.

Both cost us real runs before we caught them, and both are cheap to fix in-tree: the first is a one-line log at startup stating the resolved mode, the second is already printed but easy to miss.

Method and raw cells: club-3090 #840. Mechanism is @leloch's throughout; we're only measuring it.

[SharkWipf]

Had to rely on Deepseek V4 Flash as I'm out of Codex tokens, but I've visualized it as a video, not based on cache hit rate but based on actual expert usage per token.
This is the expert usage curve, sorted by final expert call count, over the course of a real codex session backed by Deepseek (the very conversation that implemented these visualizations, in fact, across multiple turns).
The colors indicate what would fit on what GPU in my personal setup, which is a leftover artifact of how I visualized it before, but I figured it'd still be useful to help visualize.

The total generated output tokens (input tokens not counted) is just below 35k for these visualizations. This is across a few dozen Codex user turns and independent agentic actions writing the very scripts that created these visualizations.

What it shows:

• At every frame (64 decoded tokens), it shows the total hit count per expert.
• The graph is sorted by the final hit counts per expert, in descending hit count order. So it is based on "future knowledge".
• Essentially the point is to show both how quickly we know exactly what experts are worth caching within a thread/conversation, even multi-turn, and to show the falloff of relevant experts within a thread/conversation.
• The first ~500 experts are very high value, after that it falls off quickly
• The first ~1000 are high-value before the falloff becomes too heavy
• Beyond that it becomes difficult to accurately determine what experts are worth caching.

What it does not show:

• Behavior/distribution across different conversations/threads/tasks, behavior/distribution across parallel tasks, interplay with token prediction, etc.

expert_occupancy.mp4

Edit: Notably, it also shows that a single 24GB VRAM device, if the overhead (in my vibed implementation) could be addressed, can hold and process the entire expert set "worth caching" if not holding anything else.

[leloch]

Updated branch: https://github.com/leloch/llama.cpp/tree/moe-cache-v2-pr - rebased onto current master (so latest Deepseek v4 lite support and so on). Please retest. Every reproducible issue from this thread is addressed. Disclosure as before: AI-assisted under my direction; all numbers from the 4×3090 box.

Fixes for what this thread found

Report	Status
--moe-cache auto → stoi (@batot1)	Fixed, with parser tests for invalid/negative/suffix/overflow.
--moe-cache 0 not a reliable hard-off	Fixed. off/0 select mode off; resident/unsupported models never create a session.
Bail-out/trim didn't restore baseline	EWMA bail-out deleted. Eligibility decided up front from device capability; trim is OOM-only.
1080 Ti regression	Fenced by policy: auto needs ≥2 cc 8.0 devices, forced needs cc 7.0. sm_61 creates no session - expected result is hard-off parity. I have no Pascal hardware; I'd take you up on the retest offer.
Silent bypass above max batch (@noonghunna)	One actionable log line, emitted once, when a node exceeds the batch or row bound. All modes now default to 8-token eligibility, so on + speculation no longer silently degrades.
Budget silently capped	Cap, granted capacity, free memory, reserve, and actual per-pool allocation are logged separately; docs state a fixed budget is a cap.
Default-on / no control arm	Resolved mode and policy logged at session creation. llama-bench gained --n-gen-warmup; docs prescribe a three-arm protocol and call out the under-warming trap.

The ablation the RFC promised

Qwen3.6-35B, sm_86 rebuild of the old branch, hot-set disabled, matched placement:

RFC-era mechanism set	Settled decode
Cache off	81.55
Cache only (fusion/redirect/backfill off)	88.12
+ fusion only	96.26
+ redirect only	90.04
+ fusion + redirect	98.45
Full path	99.26

Cache ~8%, fusion ~9%, redirect ~2%, backfill ~1%, disk hot-set ~0. So the rework kept the cache, rebuilt fusion with session-owned state, and dropped the rest - global singleton, leaked workers, disk hot-set, EWMA bail-out all deleted. Fusion is exact-subgraph only (gate/up/SwiGLU, optional clamps), 1–8 tokens within a 64-row bound, independent gate/up residency, stock CPU fallback on any failure. Nsight then showed two H2D copies per dispatch; packing them into one removed 40,596 H2D operations in matched traces.

Same binary, same protocol:

Arm	Settled decode
Fusion disabled	89.13
Session-owned fusion	98.95
+ packed dispatch	100.21

The minimal core is now faster than the old full path (100.21 vs 99.26) with none of its state.

Speculative decoding composes (@noonghunna's open question)

Same-binary off/on, exact outputs verified, four GPUs:

Workload	Off	On	Gain
Qwen3.6-35B, auto	82.38	100.18	+21.6%
DeepSeek-V4 284B target-only	18.61	28.93	+55%
DeepSeek + DSpark, settled	51.49	69.97	+35.9%
GLM-5.2 754B + MTP, settled	17.80	29.35	+64.9%
GLM-5.2 target-only, auto	13.92	19.37	+39%

The levers compound: on GLM, MTP is worth +28% without the cache but +51% with it - verification batches are what multi-token fusion accelerates. Fully stacked: 13.92 → 29.35.

Policy surface

Every retained constant carries an A/B, and more was measured-and-rejected than kept: pair-aware eviction (half-resident pairs ~0.006% of probes), CPU-overlap on fused rows (−2.6% to −6%), early D2H, stream priority, host-side q8 quant, smaller reserve, main-device routing penalty, pageable fills, kernel rows-per-block retune. Nothing added a knob.

@SharkWipf - your falloff drove two retained changes: a 512 KiB Ampere admission floor and first-miss admission when the pool provably covers the full inventory. Our fixed sweep flattens above 16 GiB/device on the 284B, consistent with your curve; past ~54% coverage remains open on bigger cards.

Validation

Static + dynamic CUDA suites (incl. unload/reload), CPU-only dormancy, parser/fit tests, 865/865 MUL_MAT_ID backend cases, CPU ASan/UBSan/LSan, CUDA UBSan. 117,656-token retrieval exact; perplexity parity; four concurrent clients exact; GPUs released after every run. Boundary: Compute Sanitizer isn't installed here, so its only clean run is pre-fusion - I'll close that before any PR.

Unchanged costs

The CPU mul_mat_id hook remains the largest structural cost. CUDA-only. CI covers dormancy and correctness, not perf. Diff is larger than the RFC's (about a third is tests); prefill remains bit-untouched.

The box is available for any ablation a maintainer wants. @batot1 - the Pascal parity check on your stack would be valuable. @noonghunna - if your warning patch covers cases the current diagnostic doesn't, send it over.

[Atomic-Germ]

I recognize this code from what Qwen wrote when I asked it to hack this feature in.

EDIT: I'm not saying it shouldn't be done it's just worth noting the feature is apparently inevitable.

EDIT 2: I've been using the same thing for months. I just don't like the PR process llama.cpp insists on and everyone insists that AI written code is somehow bad. So I never do. Good luck.

[leloch]

It was done with Fable first and then GPT 5.6 MAX, so perhaps indeed it is inevitable if all LLMs converge on it....

Also, I don't expect this to get merged - but perhaps someone will find it of value (speedup is decent if you got the right setup!) and the idea will eventually diffuse into the mainline somehow at some point.

[Atomic-Germ]

That's my hope! If it keeps coming back up and reimplemented, it'll stick. It's a great feature.

[xashr]

llama.cpp vs Lindenburg vs Leloch vs Miltos22

Updated 2026-08-14: Added Miltos22

I ran some performance tests today.

Test Setup

Intel 270K Plus, 128 GB DDR5-6000 RAM, RTX 5090 32 GB VRAM, Lexar NM990 SSD (14000MB/s), Ubuntu 26.04

I ran three tests:

• Generate: Story
• Generate: HTML App
• Prompt Processing

The Generate tests ran twice (one for warm up, one for collecting stats). t/s measured after n_decode=5000.

All tests ran with

   -lv 4
   -n 65536
   -c 262144
   -b 2048
   -ub 2048
   --reasoning on
   -t 16
   --cache-type-v q8_0
   --cache-type-k q8_0

llama.cpp:

    -fit on
    --no-mmap | --mmap (DS4)

Lindenburg:

GGML_EXPERT_CACHE_MAX=43 GGML_EXPERT_RAM_CACHE_MAX=230 GGML_OP_OFFLOAD_MIN_BATCH=1

   --cpu-moe
   --mmap

Leloch:

GGML_CUDA_MOE_CACHE_RESERVE_MB=800

   --moe-cache 24000 \
   --no-mmap | --mmap (DS4)

Miltos22:

   -ehs -1 \
   --no-mmap | --mmap (DS4)

Repos used

• llama.cpp: Latest
• Lindenburg: Fork, but rebased on upstream master for support of latest models
• Leloch: Fork v2
• Miltos22: Fork

Laguna S 2.1, Q6_K, 107 GB

A model that does not fully fit into VRAM, but fully fits into RAM

Write a story:

llama.cpp	Lindenburg	Leloch	Miltos22
18.2	25.6	27.6	31.3
	+41%	+52%	+72%

Write an HTML application:

llama.cpp	Lindenburg	Leloch	Miltos22
18.2	21.8	24.0	25.3
	+20%	+32%	+39%

Prompt Processing:

llama.cpp	Lindenburg	Leloch	Miltos22
980	920	840	98
	-6%	-14%	-90%

Deepseek V4 0731, Q4_K_XL, 155 GB

A model that does not fully fit into VRAM and RAM, partly streaming from disk

Write a story:

llama.cpp	Lindenburg	Leloch	Miltos22
16.5	19.6	21.7	22.7
	+19%	+32%	+38%

Write an HTML application:

llama.cpp	Lindenburg	Leloch	Miltos22
16.3	18.4	19.7	19.5
	+13%	+21%	+20%

Prompt Processing:

llama.cpp	Lindenburg	Leloch	Miltos22
160	345	55	49
	+216%	-66%	-69%

Observations

• Intel 270K requires special treatment when CPU is used for inference: A mix of P and E cores drops performance
  significantly. taskset -c 8-23 to pin to 16 E-Cores only for best performance (generate).
• Miltos22 fastest for TG
• Use of mmap drops PP performance, Leloch tanks even more than llama.cpp. Disk tier of Lindenburg
  improves PP. Miltos22 PP is extremely low.
• Lindenburg more difficult to configure/run (fiddling with max experts numbers vs just setting CACHE_RESERVE).
  Goes OOM even if enough RAM is available (only worked reliably after fresh reboot).

Thanks for the provided forks! @miltos22 TG + @Lidenburg PP would be perfect ;)

[miltos22]

PP has been glitched. I am fixing as well as a minor refractoring and ill split it into parts by next week

[noonghunna]

@xashr's numbers are the independent validation this thread needed, and they also surface the one thing the RFC hasn't yet characterised. Two observations and an offer.

The −66% prompt-processing figure needs a qualifier before it travels

Reading his own configuration notes, the DeepSeek arm was "does not fully fit into VRAM and RAM, partly streaming from disk" with mmap, and he separately reports "use of mmap drops PP performance, Leloch tanks even more than llama.cpp."

So that arm is measuring cache × mmap × disk-streaming together, not the cache alone. The cleaner prefill datapoint in his post is Laguna S 2.1, which fits in RAM: 980 → 840, −14%.

Both are real, but they will be quoted very differently, and −66% is going to get lifted out of context. Worth stating explicitly in the RFC which regime each applies to — a reader on a --no-mmap config that fits in RAM should expect something like −14%, not −66%.

I'd suggest the prefill claim be split three ways, since they have different causes:

• fits in RAM, --no-mmap — the cache's own prefill cost
• fits in RAM, mmap — plus mmap's PP penalty, which affects stock too
• streams from disk — a different regime entirely

The tradeoff is now the interesting question, not the uplift

Across both of his models the shape is consistent: TG up 21–52%, PP down 14–66%. @Lidenburg's fork inverts it — lower TG, but +216% PP on DeepSeek. xashr's conclusion that the ideal is "Leloch TG + Lindenburg PP" is the right read, and it means the open question has moved from "does the cache help?" (answered, clearly yes for decode) to "what does it cost prefill, and is that cost structural or fixable?"

That matters for agentic workloads specifically. A coding agent re-sends a large stable prefix every turn — prefill-heavy by construction — so a config that trades 14% prefill for 30% decode may be a net loss there while being a clear win for chat.

Offer: we'll characterise the prefill axis on v2

We run a 2×3090 CPU-offload stack and have spent a while on prefill specifically, so this is the axis we're best placed to measure. What we'd contribute:

• --moe-cache off vs on, same binary, on the v2 branch — single-variable, which is the control I had to correct my own earlier numbers for in this thread (cross-engine comparison flattered the baseline by ~4%).
• Prefill at several depths (10K / 40K / 90K), not just one, since the cost may be depth-dependent.
• --no-mmap, fits-in-RAM regime — the configuration missing from xashr's DeepSeek arm.
• Both -sm layer (our shipped shape) and the cache's interaction with a drafter attached, since spec-decode changes the prefill/decode balance.

⚠️ One honest delay. Our reference rig is currently running 7 of 8 memory channels — a DIMM went missing and STREAM Triad is 69.8 GB/s against ~100 when healthy. That matters more than usual here: the cache's entire value is removing host RAM reads, so a bandwidth-starved host systematically flatters the cache's uplift. Any ratio we posted today would be too generous, in a direction that happens to favour the thing being proposed. We're fixing the hardware first and will post afterwards rather than contribute numbers we'd have to retract.

Same reason we haven't retested v1→v2 yet, despite running a leloch-based build as our locked engine.

Small thing

@leloch — the --moe-cache off / 0 hard-off fix is the one that makes single-variable testing possible at all, so thank you for that specifically. Before it, every "with vs without" comparison in this space was implicitly cross-engine, which is exactly how I got my own first numbers wrong.

Tracking our side at noonghunna/club-3090#929 so the plan and the hardware gate are visible rather than a promise in a thread.

[noonghunna]

2×RTX 3090 + 8-channel DDR4 (~116 GB/s), DeepSeek-V4-Flash-0731 Q8 (MXFP4 experts), 43L on CPU.

We owed this thread numbers once our host RAM was repaired — here they are, and they ended up
being a much bigger result than a validation run.

Headline: your cache composes with an external drafter for a +37% all-time record — but only

when the drafter is NOT on a GPU

config (greedy 800-tok decode)	TPS
baseline (no drafter, no cache)	16.7
--moe-cache auto alone	24.4 (+46%)
DSpark drafter alone (GPU-resident)	33.6
DSpark (GPU) + cache	33.4 — cache never engages
DSpark on CPU (-devd none) + cache	46.8 — composes
same + GGML_CUDA_MOE_CACHE_ADMIT_AFTER=64	48.2

Sampled-decode confirms it's not a greedy artifact: the composed config beats our best
prior-config numbers by +16% (narrative) and +13% (code) at repo-canonical sampling.

Finding 1 — the engagement blocker looks like the shared-draft device path

With a GPU-resident DSpark, the cache allocates its pools and then never sees traffic: counters
read hits=0 with ~12 lookups total across thousands of decode tokens, and no bypass message ever
prints (the verify nodes are ~6 tokens, under the max_batch=8 gate — whatever skips them is
upstream of that check). Moving the drafter to CPU — the -devd none path that skips
server_prepare_shared_draft_devices's device-list reordering — restores full engagement with
identical verify batches. Our best guess is the cache's device/session binding doesn't survive
the reordered device list (draft primary pushed to front). Happy to instrument or bisect if
useful — the rig and harness are set up.

Finding 2 — default admission has a workload cliff; throttling fixes it and more

At default admission (admit=1-complete/2-partial), novel high-acceptance content (code)
regressed −30% below its own no-cache floor — fill churn outran the hit savings — while
low-acceptance narrative gained +13%. ADMIT_AFTER=64 removed the cliff entirely (code went
18.1 → 30.4) and improved every other leg too. The default looks too aggressive for at least
this 256-expert class; a note in the docs, or adaptive admission, would spare others the
confusing first impression we had ("cache on, nothing happened, one workload got slower").

Finding 3 — for scale: the solo number on hostile hardware

The +46% solo figure above is on a healthy 8-channel host where misses are cheap — the
configuration where the cache should look weakest. Bandwidth-poorer hosts should see more,
consistent with your own measurements.

Also from the same session, the small observability note we mentioned before: the one-shot
bypass announcement meant we initially couldn't distinguish "cache off" from "cache on but
unused" without reading counters; a report-on-shrink variant would have made Finding 1 a
one-glance diagnosis. We have a small patch shaped that way if it's of interest, plus the
statistics printer seems to fire only once per boot at default settings — periodic stats would
have saved us several instrumented reboots.

Happy to run any variant that helps the RFC.

---

Disclosure: measurements and analysis were assisted by an AI agent (Claude) operating under my direction; I have reviewed and stand behind the content.

[simlu]

@noonghunna Thanks for your latest post! It inspired me to try this out myself!

My rig:

• 3 x rtx 3090 with pcie 4.0 x16
• 192 gb ddr4 ram @ 3200 mhz - no idea how many channels / speed, too many variables that make this hard to measure
• Threadripper 3945WX (12 core)

My config (used this as a reference):

      --host 0.0.0.0
      --port 8080
      -m /models/deepseekv4/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
      -c 196608
      --parallel 1
      --split-mode layer
      --load-mode mlock
      -t 12 -tb 12
      --cache-type-k f16
      --cache-type-v f16
      -b 4096 -ub 4096
      --flash-attn on
      --temp 0.2
      --top-p 0.95
      --min-p 0
      --reasoning off
      --jinja
      --cache-ram 49152
      --api-key-file /app/api-key.txt
      --cpu-moe
      --moe-cache auto
      --cpu-moe-draft
      -md /models/deepseekv4/dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf
      --spec-type draft-dspark

Some notes / questions:

• My baseline was ~11 tps (using the latest llama.cpp release an not --cpu-moe) and I was able to increase this to ~19 tps using the Leloch v2 fork with the above config
• My pp fell from ~195 tps to ~135 tps. Is this because less pp compute happens on the gpu now because it's used by cache?
• I've used --cpu-moe-draft instead of -devd none. The later only loaded the drafter onto two gpus (?) and was slower
• DDR4 Ram usage went up big times: 106gb -> 160gb - vram is always saturated, I'm assuming with the Leloch it's cache though?
• what is GGML_CUDA_MOE_CACHE_ADMIT_AFTER? I wasn't able to find a reference anywhere

Does the above seem reasonable or would you expect different numbers? Any command line options I'm missing?

I'm happy to do some more testing, but have to pause for a bit and just wanted to write down my findings while it's still fresh in my head!

Edit: Not sure if relevant, but when shutting down I get the following warning
47.22.369.749 W ~llama_context: CUDA_Host compute buffer size of 192.0000 MiB, does not match expectation of 98.0625 MiB

[gopinath87607]

echo 0 | sudo tee /proc/sys/kernel/numa_balancing
GGML_CUDA_NO_PINNED=1 CUDA_VISIBLE_DEVICES=2,3,0,1
numactl --interleave=all
~/llama.cpp/build/bin/llama-server
--model /mnt/nvme/deepseek/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
--tensor-split 1.2,1.5,0.6,0.5
--n-cpu-moe 999
--flash-attn on
--ctx-size 160000
--numa distribute
--no-mmap
--fit off
--batch-size 6000
--ubatch-size 6000
--threads 42
--threads-batch 42
-ngl 100
--host 127.0.0.1
--port 8082
--jinja

ddr4 2400mhz 256gb
xeon v4 e5 2680 dual cpu
rtx 3060 12gb *2
rtx 5060 8gb *2
x10drg-q motherbord

pp 141 after 80k pp at 110tps

tps is 7.5tps

[jdunne525]

This finding is the same conclusion that was reported by a YouTuber called the codacus. He explained it fairly well in his most recent video. (he also links to his github repo from there)
(https://youtu.be/k_LostFpatg?is=X6fH4VJoXgIXw0WS)

I'm sure my explanation isn't going to be perfect, but I'll give it a shot. He came to find that frequently changing the cache of which experts are loaded is inefficient as the cost of loading the expert exceeds the performance improvement of executing from the GPU. He did find that there are some gains to be had though. He recommended a workflow whereby the user uses the model in whatever way they normally would for a brief time and that usage gets profiled to identify the best experts for that workflow. Then the next step is to restart the server and load only those experts which are helpful for the user's needs.

This is of course very limited in its usefulness, but I think it points that the best solution is to have a caching algorithm that acts very slowly. Only change the cached experts after several minutes of work. I think this is just a matter of balancing the cost of loading vs the savings to be had in switching. I imagine an algorithm could calculate these figures and determine if it's worthwhile to switch based on present usage and how long it takes to load the new expert. The cost is very high and I think that's the piece that existing solutions haven't taken into account.

[Volunteer-1]

tl;dr for the video you linked:
Cache is supposed to make access to the requested data faster. The access to tensors is faster when the additional time spent for moving tensors from RAM to VRAM is smaller than the additional time spent for reading and calculating tensors in RAM versus VRAM. Thus, minimize the amount of tensors moved from RAM to VRAM.

[Atomic-Germ]

This finding is the same conclusion that was reported by a YouTuber called the codacus. He explained it fairly well in his most recent video. (he also links to his github repo from there)

(https://youtu.be/k_LostFpatg?is=X6fH4VJoXgIXw0WS)

I'm sure my explanation isn't going to be perfect, but I'll give it a shot. He came to find that frequently changing the cache of which experts are loaded is inefficient as the cost of loading the expert exceeds the performance improvement of executing from the GPU. He did find that there are some gains to be had though. He recommended a workflow whereby the user uses the model in whatever way they normally would for a brief time and that usage gets profiled to identify the best experts for that workflow. Then the next step is to restart the server and load only those experts which are helpful for the user's needs.

This is of course very limited in its usefulness, but I think it points that the best solution is to have a caching algorithm that acts very slowly. Only change the cached experts after several minutes of work. I think this is just a matter of balancing the cost of loading vs the savings to be had in switching. I imagine an algorithm could calculate these figures and determine if it's worthwhile to switch based on present usage and how long it takes to load the new expert. The cost is very high and I think that's the piece that existing solutions haven't taken into account.

That's the "hot pinning" part in my implementation, plus it can use an imatrix to preseed. It might not help on a GPU, but it definitely does on a CPU. I never got that far

[brettowe]

considering the topic of the video, an imatrix sounds overly complicated?
if a simple usage counter on the experts is kept track of then you could choose what to load/offload via that
to tune for each system you would only load/offload once the counters have exceeded a set gap over what is in say vram
if vram loading is painful then the gap might be 10 hits over one in vram before a swap out is done. but if it isn't maybe its only 3 hits
and of course you would want to save the usage counter stats at least for the ones in vram to disk so relearning doesn't have to start from 0 every time

[SharkWipf]

In my testing, if you just count experts every token, you get 90%-98% of the final shape within the first 256 tokens. This has some minor overhead, but not too much to do at the start of every generation. After that, you don't need to count at every generated token anymore.
No imatrix or anything needed.

[Volunteer-1]

@brettowe leloch's and TheTom's implementations use a counter for each expert tensor. TheTom additionally uses a last used timestamp, and the user can balance between the importance of the timestamp and the counter.

[SharkWipf]

Since we're running numbers, below my own, not yet ready to publish implementation, run across 2 single GPUs, compared.
These tests were run on Deepseek V4 Flash.
These numbers are not final, optimizations are not yet complete, but they are meant to provide some context to the results and show that whether or not it's beneficial very much depends on the specific hardware and model.

One thing that should be made very clear: The results will very much be hardware dependent. As long as your CPU remains the bottleneck, the needle won't move much until you can offload enough to make it not be the sole bottleneck anymore.
This is why, in my internal testing, even with a 80%+ cache hit rate on the much bigger GLM 5.3 (non-flash), my performance gains were much less than with Deepseek V4 Flash.

My hardware is a 32-core Threadripper 5975WX (Zen3), ~256GB/s RAM, and the 2 GPUs tested.

Test result conclusion spoiler:

RTX 6000 (96GB VRAM):

• Fastest stock llama.cpp result:
  • 361.10 tok/s prefill
  • 33.81 tok/s decode
• Fastest stock llama.cpp result with manual custom -ot layer-wise offloading:
  • 296.46 tok/s prefill
  • 38.22 tok/s decode
• Fastest (warm) result with MoE expert caching in my implementation:
  • 148.65 tok/s prefill (Half that of stock)
  • 78.97 tok/s decode (Double that of stock)

EDIT: I do not believe the lower prefill speeds to be an inherent limitation. I have not optimized for this at all yet, but I see no reason why it couldn't be at least as fast as stock.

RTX 3090 (24GB VRAM):

• Fastest stock llama.cpp result:
  • 135.81 tok/s prefill
  • 23.11 tok/s decode
• Fastest stock llama.cpp result with manual custom -ot layer-wise offloading:
  • Identical because no more room for layers after loading Dspark, so same end result
• Fastest (warm) result with MoE expert caching in my implementation:
  • 136.93 tok/s prefill
  • 22.85 tok/s decode (Lower than stock!)

On a different CPU, or a different model, this could be stacked completely differently.
Qwen Flash runs much faster on the 3090. GLM 5.3 goes from 6tok/s to 7tok/s on the RTX 6000.
And when involving parallel requests, the dynamics are completely different again. DSpark suddenly becomes a bottleneck at p=8, and CPU bottlenecks hit much harder too. Though there are still gains there.

Report as directed by me but output by GPT 5.6 Sol (everything below this line is LLM-generated):

---

---

---

2026-09-02 complete sustained DeepSeek baseline matrix

This replaces the earlier partial table. All three requested stock baselines are now identified separately, with and without DSpark, alongside the modified MoE cache path.

Fixed workload and exact baseline contracts

Every distinct arm used one fresh server, one identical public-domain prompt of 4,551 evaluated tokens, exactly 4,096 generated tokens, temperature=0, seed 424242, ignore_eos=true, parallelism 1, batch/ubatch 1,024, and a 16,384-token context. DeepSeek speculation used the supported external module with --spec-type draft-dspark --spec-draft-n-max 3 on the same tested GPU.

1. Host/trunk-only: target trunk and KV on the tested GPU; every routed expert tensor on CPU via --n-gpu-layers all --cpu-moe --fit off.
2. Stock resident fit: unmodified fit/placement behavior, filling trailing routed-expert layers after the trunk while retaining runtime headroom. No-draft used --fit on --fit-target 512. The RTX PRO 6000 DSpark arm used a calibrated 14,300 MiB fit target because current fit cannot measure DSpark context memory.
3. Static layerwise: target trunk plus the stated number of complete leading routed-expert layers on the tested GPU; all later expert layers on CPU. This matches the original bottom-up layerwise placement strategy rather than stock fit's trailing-layer choice.
4. MoE cache: the modified adaptive expert-cache path, initially empty, using residual device memory after all resident target and draft allocations.

The RTX 3090 cannot retain one complete expert layer alongside DSpark and the fixed runtime configuration. A fit attempt that retained one layer loaded successfully but OOMed on the first real request. Therefore its valid host, stock-resident, and static-layerwise DSpark contracts all contain zero resident expert layers and are byte-for-byte identical commands; the one measured host result is explicitly reused for those two equivalent rows rather than rerun.

Complete results

GPU	Draft	Execution contract	Static expert residency	Full 4,096-token generation	Final 1,024 tokens	Prompt	Request wall	Cache hit / resident	Draft acceptance
RTX 3090	None	Host/trunk-only	0 layers	17.62 tok/s	17.64 tok/s	136.30 tok/s	265.88 s	-	-
RTX 3090	None	Stock resident fit	4.33 layers	18.39 tok/s	18.38 tok/s	153.14 tok/s	252.43 s	-	-
RTX 3090	None	Static layerwise	4 layers	18.48 tok/s	18.38 tok/s	154.84 tok/s	251.08 s	-	-
RTX 3090	None	Modified MoE cache	Dynamic	23.00 tok/s	22.85 tok/s	136.93 tok/s	211.31 s	41.12% / 13.94 GiB	-
RTX 3090	DSpark-3	Host/trunk-only	0 layers	19.58 tok/s	23.11 tok/s	135.81 tok/s	242.76 s	-	50.73%
RTX 3090	DSpark-3	Stock resident fit	0 layers; equivalent	19.58 tok/s	23.11 tok/s	135.81 tok/s	242.76 s	-	50.73%
RTX 3090	DSpark-3	Static layerwise	0 layers; equivalent	19.58 tok/s	23.11 tok/s	135.81 tok/s	242.76 s	-	50.73%
RTX 3090	DSpark-3	Modified MoE cache	Dynamic	17.68 tok/s	17.70 tok/s	134.97 tok/s	265.37 s	13.00% / 1.74 GiB	38.08%
RTX PRO 6000	None	Host/trunk-only	0 layers	19.83 tok/s	19.75 tok/s	154.69 tok/s	235.97 s	-	-
RTX PRO 6000	None	Stock resident fit	26.67 layers	33.78 tok/s	33.81 tok/s	361.10 tok/s	133.89 s	-	-
RTX PRO 6000	None	Static layerwise	26 layers	33.12 tok/s	32.98 tok/s	356.82 tok/s	136.46 s	-	-
RTX PRO 6000	None	Modified MoE cache	Dynamic	41.76 tok/s	49.43 tok/s	153.18 tok/s	127.84 s	85.34% / 85.05 GiB	-
RTX PRO 6000	DSpark-3	Host/trunk-only	0 layers	18.82 tok/s	19.26 tok/s	153.20 tok/s	247.43 s	-	41.04%
RTX PRO 6000	DSpark-3	Stock resident fit	22.33 layers	32.86 tok/s	32.10 tok/s	301.46 tok/s	139.79 s	-	44.50%
RTX PRO 6000	DSpark-3	Static layerwise	22 layers	38.22 tok/s	35.67 tok/s	296.46 tok/s	122.57 s	-	59.51%
RTX PRO 6000	DSpark-3	Modified MoE cache	Dynamic	60.91 tok/s	78.97 tok/s	148.65 tok/s	97.89 s	86.85% / 72.85 GiB	64.17%

Static fractional residency is derived from the fit trace: each complete expert layer has gate, up, and down matrices. Stock fit selected 4 complete layers plus one matrix on the RTX 3090 without draft, 26 complete layers plus two matrices on the RTX PRO 6000 without draft, and 22 complete layers plus one matrix on the RTX PRO 6000 with DSpark.

Fair baseline-versus-modified conclusions

• RTX 3090, no draft: cache is 24.49% faster than the best stock baseline (23.00 versus 18.48 tok/s), and 30.56% faster than host/trunk-only.
• RTX 3090, DSpark-3: cache is 9.67% slower than all three equivalent zero-resident baselines. DSpark leaves only 1.74 GiB for cache, reducing hit rate to 13.00%. The overall RTX 3090 winner is cache without draft at 23.00 tok/s, 17.51% above the best baseline across either draft setting.
• RTX PRO 6000, no draft: cache is 23.62% faster than the best stock baseline (41.76 versus 33.78 tok/s), and 110.54% faster than host/trunk-only.
• RTX PRO 6000, DSpark-3: cache is 59.37% faster than the best stock baseline (60.91 versus 38.22 tok/s), 85.39% faster than stock fit, and 223.74% faster than host/trunk-only.
• The RTX PRO 6000 modified winner is the cache plus DSpark-3 path. Its full-request generation rate is 45.88% above cache without draft, while retaining a 72.85 GiB target cache.

The full-generation rate includes cold cache population throughout one sustained request. The final-1,024 rate is reconstructed from timestamped live n_gen samples, linearly interpolated at token 3,072, and the exact final timing point. It is a warmed window inside the same request, not repeated short warmup.

---

---

---

As mentioned, this branch is not published yet. It's a custom implementation, based on my own implementation and refined with all the alternative implementations posted in this thread.
It is not ready for publishing yet (if it ever will be) but since people were asking for numbers, I figured I'd post mine.

[mfethe1]

Consolidating where the numbers in this thread have converged, because I think there is now a single rule that reconciles all of them — including the ones that look contradictory.

The reconciliation rule. Compare the cache budget to the model's expert working set at equal total VRAM:

• Budget ≥ working set → resident layers strictly dominate. Our matched-budget A/B (single 5060 Ti 16 GB, Qwen3-30B-A3B Q4_K_M, pp512/tg128, ABBA ×3): resident 149.6 / 35.4 t/s vs cache 64.0 / 19.7 t/s at ~12.5 GiB — despite the cache arm reaching 85.3% hit rate with zero fill failures. Resident weights serve 100% of tokens with zero misses at the same byte cost. @ZitouniNidhal's summary of this is exactly right.
• Budget < working set → the cross-layer cache wins, and the margin grows the deeper below WS you are. @SharkWipf's matrix is the cleanest demonstration: on the 96 GB card stock fit can only place 26.67 leading expert layers (partial residency), while the cache covers all layers — and the cache wins +24% (no draft) to +59% (DSpark-3). On the 24 GB card with DSpark eating the budget, the cache has 1.74 GiB and 13% hit → −9.7%. Same code, opposite verdicts, fully explained by budget vs working set.

Two consequences worth stating explicitly:

1. The right baseline is best-feasible-resident at matched budget, not stock fit. Several upthread comparisons (including our own first pass) measured the cache against default placement, which flatters the cache whenever fit leaves the budget underused. @SharkWipf's static-layerwise rows are the correct control.
2. Prefill almost never benefits. Every cache arm in this thread loses prefill (ours by 2.34×, SharkWipf's by ~2×) — decode-skewed routing is the entire value proposition, which is consistent with the RFC's decode-only fill from the start.

On policy. We ran the admission/eviction/prediction side to completion on an offline replay harness (mirroring shipped LFRU semantics, 62,400 accesses, 5,083 distinct keys, degenerate-policy and validity gates): best replacement+admission arm buys +0.4–8.1% depending on cache size relative to the working set, and a perfect prefetch predictor (f=1.0) caps at +12.8%. So the remaining headroom is not in catching more — it's in miss-cost reduction, which is what @nibor1896's host-tier result shows (hit rate flat ~80%, miss cost halved, 1.63× decode) and what @mclaudod's direct-resident path achieves in-engine. Both are residency/transport wins, not policy wins.

This also reconciles the slow-cache/profile-then-pin idea (@jdunne525, codacus): pinning is admission control, and admission-first is exactly what wins below the working set in our sweep — but once the budget exceeds WS, no admission policy beats just holding the weights.

Suggested default for any merge decision: treat the cache as the sub-working-set regime's tool. If -ngl N placement can hold the full expert set, do that; if it can't, use the cache with budget = all spare VRAM after trunk+KV+draft, decode-only fill, and tune on wall-clock, never on hit rate (mclaudod rejected seven hit-rate-improving changes that didn't move end-to-end t/s — the trap is real).

One rough edge we'd echo from the validation pass upthread: silent failure modes are the biggest practical risk (batch > cache max-batch refusing decodes silently; WDDM paging cliffs). Any merged version should warn loudly on both paths.

Disclosure: measurements gathered/analyzed by an AI agent under my direction; I reviewed the numbers and stand behind them.

[SharkWipf]

Prefill almost never benefits

I do want to add that I don't believe this to be an inherent limitation. I see little to no reason why prefill can't be at least equal to stock. I have not yet optimized for this, but I would not dismiss it as a factual conclusion.

[mfethe1]

Follow-up to the matched-budget A/B above, since the natural next question is "does the gap move with budget?" — we've now run the sweep at 10 GiB and 12.5 GiB (same rig, same build e3096b0, same model Qwen3-30B-A3B Q4_K_M, ABBA ×2 fires × r3, decisions frozen pre-run; resident arm's ngl calibrated per budget so resident VRAM matches the cache cap within ~3%):

budget	tg128 cache	tg128 resident	R/C	pp512 cache	pp512 resident
10 GiB	18.99	28.01	1.47×	63.9	117.2
12.5 GiB	18.88	41.22	2.18×	64.1	171.4

Three things worth flagging:

1. The cache arm is budget-insensitive in this range. tg128 for the cache arm sits at 18.7–19.0 t/s flat across 10→15 GiB of cache budget. On this model/rig, the extra cache bytes buy essentially zero decode throughput until the budget approaches the full expert working set (~38 GiB) — the hit rate goes up, the bytes/token moved don't go down.
2. The gap grows with budget (1.47× → 2.18× from 10 → 12.5 GiB): the resident arm converts each extra GiB into throughput ~7× faster than the cache arm does. So "cache as fallback below residency" still holds, but the fallback is a soft floor, not a ramp.
3. Resident placement fails hard past the envelope — above ~15.3 GiB the resident arm can't create a context on a 16 GB part (logged engine errors, not silent), while the cache arm keeps running. So the crossover, if it exists on this hardware, lives somewhere below 10 GiB — that region (6–8 GiB) is unmeasured and is our next cell.

Caveats as before: fork-feature comparison (leloch's moe-cache-v2-pr), single GPU, tg128/pp512 only, one model family. The genuinely open software question isn't eviction policy but admission policy: our third arm (trace-profiled cross-layer admission using a 5k-key routing trace) isn't implemented yet — that's the one mechanism that could move bytes/token rather than hit rate. Raw logs, pins, and pre-registrations available.

[guqiong96]

@mfethe1 Dspark can also do resident layers.

[Volunteer-1]

Another data point from my system:

Windows 11, 2x RTX 3090 at default settings using x8 and x8 lanes, Ryzen 9950X, 192GB RAM at 3600MHz (JEDEC default).
LLM: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF/tree/main/UD-Q8_K_XL has 161GB incl DSpark.
Following are measurements with llama-server parameters manually tuned for max token generation speed for each configuration:

Context 65535, prompt that produces ca 1200 (general chat) response tokens:
6.88 t/s with upstream https://github.com/ggml-org/llama.cpp/tree/master
13.57 t/s (+97%) with @leloch 's https://github.com/leloch/llama.cpp/tree/moe-cache-v2-pr
12.70 t/s (+85%) with @TheTom 's https://github.com/TheTom/llama-cpp-turboquant/tree/feature/turboquant-kv-cache

Context 405000, same prompt:
7.08 t/s with upstream
13.03 t/s (+84%) with leloch
12.57 t/s (+78%) with TheTom

Context 1048576 (maximum), same prompt:
7.53 t/s with upstream
10.96 t/s (+46%) with leloch
9.71 t/s (+29%) with TheTom

@mclaudod 's https://github.com/mclaudod/llama.cpp/tree/moe-hot-cache always crashes with OOM exception during loading tensors into CUDA0.

12-13 t/s were reached at ca 74% MoE cache hit ratio.
Optimal speed gets reached after ca 200 tokens were generated.

Lidenburg's branch:

I will be testing @Lidenburg 's https://github.com/Lidenburg/llama.cpp/tree/moe-expert-caching on Windows and Linux, because his branch is the only one with moe cache that is compatible with split mode tensor for DeepSeek V4. The other branches I tested are based too far back from current upstream which makes merging the split mode tensor feature for DeepSeek V4 quite impossible.

#27574 fixes split mode tensor for Qwen 3.5-3.8, Laguna and some other models.
#26490 enables split mode tensor for DeepSeek V4.

Recommendation for everyone who implements the MoE cache feature:

Minimize the amount of tensors that get moved from RAM to VRAM.
This means, on model loading the VRAM shall be initially filled with as many expert tensors as possible. All current implementations do it the opposite way, which kills prefill speed and keeps generation speed low until the cache becomes warm (after ca 200 tokens with DeepSeek V4).
During inference, the least used tensors in VRAM shall be moved to RAM and the most used tensors in RAM shall be moved into VRAM.

[giveen]

@Volunteer-1 heh I'll look into your recommendations, thank you.

[Atomic-Germ]

This means, on model loading the VRAM shall be initially filled with as many expert tensors as possible. During inference, the least used tensors in VRAM shall be moved to RAM and the most used tensors in RAM shall be moved into VRAM.

All current implementations do it the opposite way, which kills prefill speed and keeps generation speed low until the cache becomes warm (after ca 200 tokens with DeepSeek V4).

To be fair mine doesn't use vram at all, and is mainly meant to run models that are just plain too big for the system.

To be honest at this point I have no idea where anyone is going with this, I just still use my original guanaco branch with old shit, these days.

[Volunteer-1]

@giveen you are welcome! Thank you very much for bringing a working expert cache into TheTom's branch. If the VRAM gets initially filled with as many expert as can fit (which is what upstream does), then prefill speed and generation speed will avoid the severe reduction they currently endure from the available expert cache implementations. Bringing expert caching into upstream will not succeed until this problem gets solved, because the beta-testing users will scream about the speed regression. Maintaining prefill speed and initial generation speed is a must for the agentic workflow. Currently, expert caching reaches optimal speed after 200 tokens whereas upstream reaches optimal speed at the first token.

If you or someone else finds the time to merge #27574 and #26490 into TheTom's branch, I'd be happy to test Qwen 3.8 27B and DeepSeek V4 Flash with split mode tensor on Windows and Linux. Expert caching would gain enormous speedup from split mode tensor, because moving tensors from RAM to VRAM and vice versa would become dirt cheap on the time axis.
One may ask, "Why would that be, because split mode tensor improves communication speed between GPUs?" Answer: Although GPU P2P does not increase bandwidth between VRAM and RAM, it does reduce transfer latency between VRAM and RAM to near zero, which is most important for moving "small" data sets at high frequency between VRAM and RAM.

[giveen]

@Volunteer-1 TheTom#340 working on it

[Volunteer-1]

I tested @Lidenburg 's branch on Linux. It does not work.
System: Linux Mint 22.3 (Ubuntu 24.04) , NVidia Display Driver 580.105.08 P2P fork , CUDA 12.9.2 , NCCL 2.31.2 for CUDA 12.9 , 2x RTX 3090.

1. GPU1 is always ignored.
   When I put all attention layers on GPU0 via "-ts 44,0" then llama-server starts and does inference with DeepSeek V4 Flash UD-Q8_K_XL at catastrophically low 0.8t/s. No expert or any or tensor gets put on GPU1. When I distribute attention layers via "ts 1,1" then llama-server crashes during startup with "invalid CUDA operation on GPU1.
2. Unclear how to choose values for GGML_EXPERT_CACHE_MAX and GGML_EXPERT_RAM_CACHE_MAX because they appear to be expert count, but I need to know Megabytes, because MB is what I see available in nvtop.
3. "-lm dio" and "-lm none" crash on startup.

[Lidenburg]

The benchmarks from my implementation, when run on my machine atleast, shows better performance with expert caching when matching VRAM usage with the stock -ncmoe option. My implementation uses an LFU with aging eviction strategy, it counts uses of a given expert and keeps the top N most used, and halves all experts usage counters when any experts usage reaches a certain threshold, the implementation currently uses 128 for this threshold.

I also see basically matched prompt processing speeds when the cache is warmed up.

I just pulled in the latest llama.cpp commits and rebased my branch and I get:

No Cache

$ ./nocache-build/bin/llama-cli -fa on -ctk q8_0 -ctv q8_0 -m ~/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -c 256000 --jinja --no-mmap -ngl 99 --n-cpu-moe 21 -t 11 -n 1000 -f test-prompt.txt --perf

> Create a python script that prints the size in bytes of all the expert weights in a gguf file

...

[ Prompt: 63.0 t/s | Generation: 71.2 t/s ]

> /read ggml/src/ggml-backend.cpp
Loaded text from 'ggml/src/ggml-backend.cpp'

> hello

...

[ Prompt: 1195.6 t/s | Generation: 62.6 t/s ]

15524MiB VRAM used.

Expert Cache

$ GGML_EXPERT_CACHE_MAX=120 GGML_EXPERT_RAM_CACHE_MAX=136 GGML_OP_OFFLOAD_MIN_BATCH=1 ./build/bin/llama-cli -fa on -ctk q8_0 -ctv q8_0 -m ~/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -c 256000 --jinja --mmap -ngl 99 --cpu-moe -t 11 -n 1000 -f test-prompt.txt --perf

> Create a python script that prints the size in bytes of all the expert weights in a gguf file

...

[ Prompt: 56.9 t/s | Generation: 88.9 t/s ]

> /read ggml/src/ggml-backend.cpp
Loaded text from 'ggml/src/ggml-backend.cpp'

> hello

...

[ Prompt: 1142.2 t/s | Generation: 73.1 t/s ]

15466MiB VRAM used.

[mfethe1]

Interesting — and I think this datapoint actually sharpens the reconciliation rule rather than contradicting it. Two observations on why your cache arm wins where ours lost:

1. Your admission policy is doing the work, not the cache per se. LFU-with-aging keeping the top-N used experts is effectively profile-guided admission: the experts that stay resident are chosen by measured reuse, so the bytes/token actually moved goes down as the working set concentrates. That's precisely the mechanism we flagged upthread as the open question — our cache arm was the plain budget-bounded policy (admit whatever the last route touched, evict on capacity), and its decode throughput was budget-insensitive (18.7–19.0 t/s flat from 10→15 GiB) because hit rate went up while bytes/token didn't move. Your numbers are the first in this thread showing an admission-aware policy converting cache budget into decode throughput — which is a real software result.

2. The baseline shape differs too. Your no-cache arm is a static --n-cpu-moe 21 prefix split; our resident arm was ngl-calibrated leading-layers placement at a matched budget. At ~15.5 GiB of a 16 GB part with Qwen3.6-35B, neither arm can reach full residency, so both of your arms live in the sub-working-set regime where the reconciliation rule says the cache is the fallback tool — and there your LFU-aging policy beats a static prefix split on decode 88.9 vs 71.2, with pp essentially tied (56.9 vs 63.0 cold, ~1142 vs ~1196 warm). That's fully consistent with "cache as a sub-WS tool, policy quality decides how good a tool it is."

Two questions that would make this datapoint even more useful:

• Was the decode number taken after cache warm-up on the same prompt, or cold? Our cache arm's ITL variance was ~10× the resident arm's — if your LFU-aging arm is also steadier than the no-cache arm, that's a second user-perceivable win worth stating explicitly.
• Has your arm been tried at a budget where the alternative is full residency (budget ≥ working set)? Our rule predicts the resident arm should win there even against a good admission policy — confirming or breaking that boundary with your implementation would be the decisive cell for this thread's question.

Raw logs and pre-registrations for our side are available as before — and for what it's worth, the same-model comparison matters: Qwen3.6-35B-A3B experts are ~864 KiB-class, so aging-LFU top-N admission has fine granularity to work with there.

[kisasexypantera94]

I did a similar experiment for Metal a few months ago, with exact on-demand expert loading directly from disk.

On an M3 Pro 36GB, Qwen3-30B-A3B-Q6_K went from 38.1 tok/s fully resident to 27.5 tok/s while saving 11GB of memory, and 22.1 tok/s while saving 16.6GB. I also got it running on an M1 Pro 16GB at 13 tok/s.

Might be relevant here: #23324

[jdunne525]

Someone asked about qwen4exp.
That's Qwen 3.8 flash next. It is a huge model (177b), but it uses moe and a new lookup table that can reside on an ssd. The codacus just shared a new video a couple of days ago where he was running that model (177b parameters) on an rtx 3060 with just 12gb vram, (system ram can vary, but it's usable with 32gb), nvme ssd, and running at a decent speed around 24 tok/sec.
At least part of the speedup was due to moe caching. He used some other fork that had moe caching.

Codacus youtube video showing the model in use:
https://youtu.be/IH8XmxiwliQ?is=sOGTR8HABSamYaDw

Different fork adding moe expert caching:
https://github.com/GenerelSchwerz/llama.cpp

Has anyone tried running this new model with the forks that have been discussed here?

I tried the GenerelSchwerz fork and it's not working for me on windows with an rtx 2080 super 8gb. (it hangs on the first prompt forever when I have moe caching enabled)

[GenerelSchwerz]

The errors I saw on Windows were related to grouped decode, and failing to create graph.

Another upstream measurement on Windows with same model as above: Qwen3.8-Flash-Next IQ4_XS on 2x RTX 3090: no MTP 26.1 t/s with MTP 26.3 t/s I guess this means MTP is severely suboptimal with this model.

I don't have full PCIe lanes, but I have the P2P fork on Linux which enables up to 50% speedup for MoE models and up to 80% speedup for dense models.

In order to help you I would need a full log of your run at -lv 4 or 5. Something about your configuration resulted in my fork marking you ineligible for the grouped execution path. Your results with my fork aren't representative of its intended behavior.

I see a reasonable uplift when using MTP. For moe caching, the mtp head needs to be entirely resident on the gpu for a full speedup (at least in my implementation) due to pcie bandwidth fighting between the main model and the mtp experts otherwise.

I have posted a markup of my speedup using MTP on my wiki; None: 55, MTP 2: 61 in prose. I have non-documented results from coding resulting in 66 tks, but don't take that as gospel until I release proper documentation. Other users have seen greater benefits than I have (I am VRAM constrained, 16gb total) with 5090s, achieving 80 tokens/sec with 256k context, and over 100 tks when allocating more cache slots.

I'm interested in the P2P fork. A contributor on my fork I believe found something similar, although he hasn't made an upstream PR for it.

[Volunteer-1]

The P2P fork is https://github.com/aikitoria/open-gpu-kernel-modules
More info later

[GenerelSchwerz]

Thank you for the link. Also, my fork now supports multiple GPUs in layer-split mode. It currently transfers data only over host transfers rather than P2P because the system I tested on didn't support that (4070 and 3060). Will happily update it further once I have hardware to properly test on.

[Volunteer-1]

@GenerelSchwerz The P2P fork supports P2P with mixed generation cards since branch 610.57.04-p2p-v2

[Volunteer-1]

@GenerelSchwerz I build your current branch after merging important upstream PRs for kv cache quantization, MTP, qwen4exp and tensor parallelism. Now the expert cache is always on the same GPU as attention weights are, with identical device distribution as the attention weights.

Disadvantage: this is not optimal for performance, because attention and experts should each be confined to one GPU (e.g. attention on GPU1, experts on GPU0). Your previous version always put the expert cache on GPU0 which allowed me to conduct this separation via
-sm layer -ts 0,50 -mg 1

Advantage: via -sm layer -ts 25,25 I can fully utilize the VRAM of both GPUs.

Windows test:

Command:
set GGML_CUDA_MOE_FREQUENCY=1&& llama-server.exe -m D:\Models\Qwen3.8-Flash-Next\IQ4_XS\Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf -md D:\Models\Qwen3.8-Flash-Next\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -np 1 --spec-type draft-mtp --spec-draft-n-max 2 --no-kv-unified --cpu-strict 1 -C 55550000 -t 8 -Cb 55550000 -tb 8 -fa on -ctk q8_0 -ctv q8_0 -cmoe -ctkd q8_0 -ctvd q8_0 -cmoed -ngl auto -sm layer -ts 25,25 -mg 1 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0 --repeat_penalty 1.0 -lm dio -b 16 -lv 4 -c 65535 -lzm auto --moe-expert-cache-size 330 --moe-expert-cache-host-pinned-mb 0 --experimental-logs

produces a server halt on error. After the first generated thinking token, a popup window is shown with text:
Server Error
The server responded with an error message. Review the details below.
Stream resume produced no new bytes, giving up

llama-server error messages during loading model:
0.00.647.745 I operator (): getting device memory data for the extra model at a context size of 65535:
0.00.979.388 E llama_model_load: error loading model: borrow_shared_tensor: this model is a draft head without its own 'token_embd.weight'; load it as a draft of its target model, not on its own
0.00.979.395 E llama_model_load_from_file_impl: failed to load model

llama-server log level 4 messages during inference:
0.27.416.666 I srv update_slots: all slots are idle
0.43.114.640 I srv server_strea: conv_id=2cd8ed98-a991-41d0-945a-c8c6fcb5da09 (empty=0)
0.43.115.033 I srv operator (): chat format: peg-native
0.43.115.148 I slot get_availabl: id 0 | task -1 | - skipping, slot is empty
0.43.115.152 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1
0.43.115.152 I srv get_availabl: updating prompt cache
0.43.115.160 I srv load: - looking for better prompt, base f_keep = -1.000, f_sim = 0.000
0.43.115.165 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 65536 tokens, 8589934592 est)
0.43.115.167 I srv get_availabl: prompt cache update took 0.01 ms
0.43.115.208 I cmn common_reaso: activated, budget=2147483647 tokens
0.43.115.229 I slot launch_slot_: id 0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> ?temp-ext -> dist
0.43.115.237 I slot launch_slot_: id 0 | task -1 | sampler params:
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 64
top_k = 20, top_p = 0.950, min_p = 0.000, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 1.000
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
0.43.115.239 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0
0.43.115.250 I slot operator (): id 0 | task 0 | new prompt, n_ctx_slot = 65536, n_keep = 0, task.n_tokens = 58
0.43.115.262 I slot operator (): id 0 | task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
0.44.447.035 I slot operator (): id 0 | task 0 | cached n_tokens = 16, memory_seq_rm [16, end)
0.44.971.162 I slot operator (): id 0 | task 0 | cached n_tokens = 32, memory_seq_rm [32, end)
0.45.211.928 I slot operator (): id 0 | task 0 | cached n_tokens = 42, memory_seq_rm [42, end)
0.45.232.496 I slot create_check: id 0 | task 0 | created context checkpoint 1 of 32 (pos_min = 41, pos_max = 41, n_tokens = 42, size = 112.655 MiB)
0.45.642.545 I slot operator (): id 0 | task 0 | cached n_tokens = 54, memory_seq_rm [54, end)
0.45.642.666 I slot init_sampler: id 0 | task 0 | init sampler, took 0.01 ms, tokens: text = 58, total = 58
0.45.664.726 I slot create_check: id 0 | task 0 | created context checkpoint 2 of 32 (pos_min = 53, pos_max = 53, n_tokens = 54, size = 112.668 MiB)
0.45.780.329 I moe-grouped-plan: owner=000002657AE88020 device=1 physical=1 generation=1 semantic_group=0 domain=1 outcome=1 reason=0
0.45.810.495 I moe-grouped-plan: owner=00000236B1093010 device=0 physical=0 generation=1 semantic_group=0 domain=1 outcome=2 reason=15
0.45.810.501 I moe-grouped-plan: owner=00000236B1093010 device=0 physical=0 generation=1 semantic_group=1 domain=1 outcome=2 reason=15
0.45.810.501 I moe-grouped-plan: owner=00000236B1093010 device=0 physical=0 generation=1 semantic_group=2 domain=1 outcome=2 reason=15
... many similar lines
0.45.810.516 I moe-grouped-plan: owner=00000236B1093010 device=0 physical=0 generation=1 semantic_group=24 domain=1 outcome=2 reason=15
0.45.810.517 I moe-cache: device 0 grouped decode unavailable for 25 group(s); using cached mul_mat_id fallback
0.45.810.523 E moe-cache: required grouped execution failed: grouped plan unavailable
0.45.810.536 E graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
0.45.810.537 E process_ubatch: failed to compute graph, compute status: -1
0.45.810.540 W decode: removing memory module entries for seq_id = 0, pos = [58, +inf)
0.45.810.548 E llama_decode_ext: failed to decode, ret = -3
0.45.810.576 E srv decode: Compute error. off = 0, n_batch = 16, ret = -3
0.45.810.578 E srv send_error: task id = 0, error: Compute error.
0.45.810.586 I slot release: id 0 | task 0 | stop processing: n_tokens = 61, truncated = 0
0.45.810.594 W srv stop: cancel task, id_task = 0
0.45.810.595 I slot prompt_clear: id 0 | task -1 | clearing prompt with 61 tokens
0.45.815.551 E srv update_slots: decode() failed: Compute error.
0.45.815.565 I srv update_slots: all slots are idle

When not using a draft model:
set GGML_CUDA_MOE_FREQUENCY=1&& llama-server.exe -m D:\Models\Qwen3.8-Flash-Next\IQ4_XS\Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf -np 1 --no-kv-unified --cpu-strict 1 -C 55550000 -t 8 -Cb 55550000 -tb 8 -fa on -ctk q8_0 -ctv q8_0 -cmoe -ngl auto -sm layer -ts 24,26 -mg 1 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0 --repeat_penalty 1.0 -lm dio -b 16 -lv 4 -c 65535 -lzm auto --moe-expert-cache-size 350 --moe-expert-cache-host-pinned-mb 0 --experimental-logs

The generation speed dropped from 19.1 to 13.9 t/s compared to your previous version, probably because the experts and the attention are each now distributed among two GPUs which makes PCIe speed a bottleneck especially on Windows for which there is no P2P driver feature.

The last log lines during inference are:
moe-cache-phase: phase=prefill ops=987 staged_ops=0 split_staged_ops=0 overflow_ops=0 unique_avg=43.12 unique_max=119 ids_mib=15.18 ids_d2h_mib=15.18 ids_d2h_ms=545.845 ids_d2h_syncs=329 ids_cache_hits=658 acquire_ms=3581.934 remap_ms=2.954 copy_wait_events=987 copy_wait_event_ms=5.424 op_cpu_ms=4207.801 l1_hits=104361 l1_misses=23313 l1_evictions=0 l1_hit_rate=81.74% h2d_copies=23313 h2d_mib=17264.67 h2d_enqueue_ms=3502.250 prefetch_hits=69574 prefetch_misses=15542 prefetch_used=15542 prefetch_h2d_copies=15542 prefetch_h2d_mib=12047.33 prefetch_h2d_enqueue_ms=2715.763 upload_calls=23313 upload_mib=17264.67 staging_tiles=0 pipeline_tiles=0 tile_wait_calls=0 tile_wait_ms=0.000 cpu_pack_mib=0.00 cpu_pack_ms=0.000 pre_sync_calls=0 pre_sync_ms=0.000 h2d_submit_calls=23313 h2d_submit_mib=17264.67 h2d_submit_ms=3497.484 post_sync_calls=0 post_sync_ms=0.000 upload_errors=0
moe-cache-phase: phase=decode ops=15417 staged_ops=0 split_staged_ops=0 overflow_ops=0 unique_avg=10.00 unique_max=10 ids_mib=0.59 ids_d2h_mib=0.59 ids_d2h_ms=1822.892 ids_d2h_syncs=1 ids_cache_hits=10278 acquire_ms=1443.886 remap_ms=109.901 copy_wait_events=15417 copy_wait_event_ms=95.469 op_cpu_ms=3775.610 l1_hits=449673 l1_misses=12837 l1_evictions=6 l1_hit_rate=97.22% h2d_copies=12837 h2d_mib=9506.57 h2d_enqueue_ms=1352.424 prefetch_hits=299782 prefetch_misses=8558 prefetch_used=8558 prefetch_h2d_copies=8558 prefetch_h2d_mib=6633.70 prefetch_h2d_enqueue_ms=944.490 upload_calls=12837 upload_mib=9506.57 staging_tiles=0 pipeline_tiles=0 tile_wait_calls=0 tile_wait_ms=0.000 cpu_pack_mib=0.00 cpu_pack_ms=0.000 pre_sync_calls=0 pre_sync_ms=0.000 h2d_submit_calls=12837 h2d_submit_mib=9506.57 h2d_submit_ms=1344.903 post_sync_calls=0 post_sync_ms=0.000 upload_errors=0
moe-cache-mm: h2d_copies=36150 h2d_mib=26771.24 h2d_enqueue_ms=4854.674 mincore_sampled=0 resident_pages=0/0 resident_pct=0.00 nonresident_experts=0 mincore_fail=0 proc_minflt=0 proc_majflt=0 vm_pswpin=0 vm_pswpout=0 vm_pgfault=0 vm_pgmajfault=0 vm_pgpgin=0 vm_pgpgout=0 vm_refault_file=0 vm_refault_anon=0
3.29.842.310 I slot release: id 0 | task 0 | stop processing: n_tokens = 165, truncated = 0

[GenerelSchwerz]

My fork was mentioned. Here is a comparison of Qwen 3.6 35b A3B Q4_K_M between my fork, lelouch, and pristine. I will also cover TheTom's fork.

https://gist.github.com/GenerelSchwerz/2936b2b11d72767eeef688665412d8c5

I can additionally test Qwen 3.8 Flash Next if the people here would like. I didn't for this Gist because Qwen 3.8 Flash Next was not explicitly listed as supported despite the architecture inside claiming that it was. I didn't want to post results of code that is potentially not finalized/mature.

[Volunteer-1]

A second data point from my system after my first one #24528 (comment)

Qwen3.8-Flash-Next IQ4_XS (max context -ctk q8_0 -ctv q5_1), all inference parameters optimized for generation speed.
Windows:
GenerelSchwerz: no draft (because broken): 16.0 t/s (regression from 19.1 t/s)
TheTom: no draft (because broken): 23.5 /s. @giveen drafting crashes with Unsloth's Qwen MTP + Deepseek DSpark files.
upstream with unmerged improvements: no draft: 25.5 ts/
upstream with unmerged improvements: draft MTP: 29.2 t/s

More test results come later.

Second recommendation for everybody who implements an expert cache:

• Allow choosing the number of Megabytes (or experts) for each device separately, because attention shall reside on as little GPUs as possible (ideally 1, e.g GPU0), and experts (also expert cache) shall reside on as little GPUs as possible (ideally 1, e.g GPU1).
• Do not ignore the -ncmoe parameter. TheTom respects -ncmoe which allows for filling the remaining free VRAM of the "attention GPU" up with GPU-persistent expert layers, but GenerelSchwerz ignores -ncmoe.

[Volunteer-1]

@GenerelSchwerz Using any value other than 0 for --moe-expert-cache-host-pinned-mb results in application error with memory access violation. I tried many values between 1000 and 100000.
With --moe-expert-cache-host-pinned-mb 0 llama-server logs : CUDA_Host_buffer 54600 MB. Therefore any value between 60000 and 100000 should have been sufficiently large.

[Volunteer-1]

@GenerelSchwerz
The access violation during loading a model with MTP drafter always occurs during
llama_kv_cache: layer x: dev = CUDAy
or
llama_memory_recurrent, layer x: dev = CUDAy

where x is a random layer number and y is 0 or 1. Any value > 0 for --moe-expert-cache-host-pinned-mb has no effect on this behavior.

[Volunteer-1]

@GenerelSchwerz I can provide you with the log for a -lv 5 run without drafter which shows that failure to use "grouped decode"

I found interesting:
During model loading:
0.23.445.599 I moe-cache: CUDA_MoE_Cache_Pool[blk.23.ffn_gate_exps.weight ] = 214.84 MiB (320 slots x 0.67 MiB)
0.23.451.932 I moe-cache: CUDA_MoE_Cache_Pool[blk.23.ffn_up_exps.weight ] = 214.84 MiB (320 slots x 0.67 MiB)
0.23.458.999 I moe-cache: CUDA_MoE_Cache_Pool[blk.23.ffn_down_exps.weight ] = 281.25 MiB (320 slots x 0.88 MiB)
0.23.470.655 I moe-grouped-plan: owner=00000126DB0C5030 device=1 physical=1 generation=1 semantic_group=24 domain=1 outcome=0 reason=1
0.23.470.658 I moe-grouped-plan: owner=00000126DB0C5030 device=1 physical=1 generation=1 semantic_group=25 domain=1 outcome=0 reason=1
... many more similar lines ...
0.23.470.673 I moe-grouped-plan: owner=00000126DB0C5030 device=1 physical=1 generation=1 semantic_group=47 domain=1 outcome=0 reason=2
0.23.505.688 I moe-cache: CUDA_MoE_Cache_Pool[blk.24.ffn_gate_exps.weight ] = 214.84 MiB (320 slots x 0.67 MiB)
0.23.522.263 I moe-cache: CUDA_MoE_Cache_Pool[blk.24.ffn_up_exps.weight ] = 214.84 MiB (320 slots x 0.67 MiB)
0.23.546.190 I moe-cache: CUDA_MoE_Cache_Pool[blk.24.ffn_down_exps.weight ] = 281.25 MiB (320 slots x 0.88 MiB)
...

During inference:
1.10.631.665 D moe-placement: layer=47 semantic_group=47 domain=1 owner=CUDA1 placed=1 operations=36
1.10.721.339 I moe-grouped-plan: owner=00000126DB0C5030 device=1 physical=1 generation=1 semantic_group=24 domain=1 outcome=0 reason=1
1.10.721.344 I moe-grouped-plan: owner=00000126DB0C5030 device=1 physical=1 generation=1 semantic_group=25 domain=1 outcome=0 reason=1
... many more similar lines...
1.10.721.359 I moe-grouped-plan: owner=00000126DB0C5030 device=1 physical=1 generation=1 semantic_group=47 domain=1 outcome=0 reason=2
1.10.769.337 D que yield_to_que: done yielding to queue
1.10.769.349 D res send: sending result for task id = 0

[Volunteer-1]

@GenerelSchwerz I found another bug:
Your branch always sets context size to 4096 unless the user defines the context size via "-c".
llama-server is supposed to use the model's maximum context size as default.

[GenerelSchwerz]

Thank you for testing so much, your reports are appreciated. If you could direct these reports to my issues section on my fork, it would make maintaining these issues much easier. I don't have a lot to work with since your system is significantly different from the hardware I am able to test on. I am attempting to address your issues now.

[xashr]

Looks like there is no one-fits-all implementation when it comes to where misses should get executed: GPU or CPU.
I played with FreeToken which actually offers both: GPU (offload) mode, CPU mode and hybrid mode. After benchmarking your system you can choose the mode that provides the best performance.

In my case with an RTX 5090 and Qwen 3.8 Flash Next GPU (offload) mode provides the best performance. Currently my go-to solution when running 3.8 Flash Next.

[GenerelSchwerz]

From what I’ve seen this is correct. My implementation follows the same general principle as freetoken’s GPU offload mode. PCIE bottlenecks and page faults from mmap have significantly higher consequences when doing offloading than a hybrid implementation. I personally intend to support gpu offload + hybrid setups later down the line.

For machines with relatively balanced hardware I would say GPU offloading is the way forward. GPU to CPU synchronization is too expensive to pay.

[Volunteer-1]

A third data point from my Windows 2x 3090 system after the second one #24528 (comment) , this time with Qwen3.8-Flash-Next and DeepSeek V4 Flash, reveals very interesting speed differences and a silver lining for an optimal solution.
I merged and cherry-picked all compatibility- and speed relevant PRs and commits into each branch.
All measurements below are with MTP and optimized parameters for each branch.

Model	Quant	kv Quant	Branch	gen t/s	Diff %
Qwen3.8-Flash-Next	IQ4_XS	q8_0/q5_1	thecodacus	41.2	+53.8
Qwen3.8-Flash-Next	IQ4_XS	q8_0/q5_1	TheTom	24.3	-9.4
Qwen3.8-Flash-Next	IQ4_XS	q8_0/q5_1	GenerelSchwerz	11.9	-55.6
Qwen3.8-Flash-Next	IQ4_XS	q8_0/q5_1	upstream	26.8	-
Qwen3.8-Flash-Next	IQ4_XS	q8_0/q8_0	thecodacus	40.7	+48.8
Qwen3.8-Flash-Next	IQ4_XS	q8_0/q8_0	TheTom	28.6	+4.5
Qwen3.8-Flash-Next	IQ4_XS	q8_0/q8_0	upstream	27.3	-
Qwen3.8-Flash-Next	IQ4_XS	f16/f16	thecodacus	33.3	+24.3
Qwen3.8-Flash-Next	IQ4_XS	f16/f16	upstream	26.8	-
-	-	-	-	-
Qwen3.8-Flash-Next	Q4_K_M	q8_0/q5_1	thecodacus	36.6	+70.0
Qwen3.8-Flash-Next	Q4_K_M	q8_0/q5_1	upstream	21.5	-
Qwen3.8-Flash-Next	Q4_K_M	q8_0/q8_0	thecodacus	35.5	+54.7
Qwen3.8-Flash-Next	Q4_K_M	q8_0/q8_0	upstream	23.0	-
Qwen3.8-Flash-Next	Q4_K_M	f16/f16	thecodacus	30.4	+55.3
Qwen3.8-Flash-Next	Q4_K_M	f16/f16	upstream	20.0	-
-	-	-	-	-
Qwen3.8-Flash-Next	Q5_K_M	q8_0/q5_1	thecodacus	24.4	+71.3
Qwen3.8-Flash-Next	Q5_K_M	q8_0/q5_1	upstream	14.2	-
Qwen3.8-Flash-Next	Q5_K_M	q8_0/q8_0	thecodacus	27.9	+66.3
Qwen3.8-Flash-Next	Q5_K_M	q8_0/q8_0	upstream	16.8	-
Qwen3.8-Flash-Next	Q5_K_M	f16/f16	thecodacus	26.2	+68.7
Qwen3.8-Flash-Next	Q5_K_M	f16/f16	upstream	15.5	-
-	-	-	-	-
Qwen3.8-Flash-Next	UD-Q6_K_XL	q8_0/q5_1	thecodacus	23.6	+69.4
Qwen3.8-Flash-Next	UD-Q6_K_XL	q8_0/q5_1	upstream	13.9	-
Qwen3.8-Flash-Next	UD-Q6_K_XL	q8_0/q8_0	thecodacus	25.5	+57.3
Qwen3.8-Flash-Next	UD-Q6_K_XL	q8_0/q8_0	upstream	16.2	-
Qwen3.8-Flash-Next	UD-Q6_K_XL	f16/f16	thecodacus	23.6	+83.6
Qwen3.8-Flash-Next	UD-Q6_K_XL	f16/f16	upstream	12.8	-

I build ca a dozen expert caching forks, but don't include them all here because they either don't build on Windows (Lidenburg), don't compile anywhere or crash or don't use the available VRAM.

@thecodacus your three decode speed relevant features (expert cache, asynchronous expert prefetch, CUDA-pinned host RAM) reward your branch the undisputed crown of being by far the fastest llama.cpp fork for Qwen3.8-Flash-Next. Unfortunately, your fork does not increase decode speed for DeepSeek V4 Flash compared to upstream. Please have a look into that.

@leloch your branch is not compatible with qwen4exp and I can't merge from upstream. Could you please merge or rebase so that others can merge?

@GenerelSchwerz your branch suffered a catastrophic speed regression with MTP.

@TheTom your current branch suffered a -40% decode speed regression on DeepSeek V4 Flash.
All measurements below are with DeepSeek V4 Flash UD-Q8_K_XL, DSpark and q8_0 kv quant. TheTom's branch is fixed at 2. Sep 2026, and all tests were with optimized parameters for each branch.

Context	Branch	gen t/s	Diff %
65535	leloch	13.57	+97.2
65535	TheTom	12.7	+84.6
65535	upstream	6.88	-
-	-	-	-
405000	leloch	13.03	+84.0
405000	TheTom	12.57	+77.5
405000	upstream	7.08	-
-	-	-	-
1048576	leloch	10.96	+45.6
1048576	TheTom	9.71	+29.0
1048576	upstream	7.53	-

Recommendations for everybody who implements an expert cache:
Minimize the amount of tensors that get moved from RAM to VRAM. On model loading, the VRAM shall be initially filled with as many expert tensors as possible.
Attention shall occur on as little GPUs as possible. Expert caching shall occur on as little GPUs as possible.
But allow to select the device(s) for the cache. Many branches use always GPU0, some use only the primary graphic device (-mg).
Fill all unused VRAM automatically if the user does not provide a number describing how much VRAM to use as cache. This had worked with TheTom's branch, but does not anymore.
Do not ignore the -ncmoe parameter.

[Volunteer-1]

@GenerelSchwerz My start command is:
set GGML_CUDA_MOE_FREQUENCY=1&& llama-server.exe -m D:\Models\Qwen3.8-Flash-Next\IQ4_XS\Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf -md D:\Models\Qwen3.8-Flash-Next\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -np 1 --spec-type draft-mtp --spec-draft-n-max 1 --no-kv-unified --cpu-strict 1 -C 55550000 -t 8 -Cb 55550000 -tb 8 -fa on -ctk q8_0 -ctv q5_1 -cmoe -ctkd q8_0 -ctvd q5_1 -cmoed -ngl auto -sm layer -ts 25,25 -mg 0 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0 --repeat_penalty 1.0 -lm dio -b 16 -lv 4 -lzm off -ot "per_layer_token_embd.weight=CPU" --moe-expert-cache-size 315 --moe-expert-cache-host-pinned-mb 0 --experimental-logs -c 262144"

Without MTP, this command results in 66% faster decode speed.
I did a lot of experiments with many expert cache branches. My conclusion is that the draft model shall be split into two parts like the main model: attention resides on GPU, experts are handled by the expert cache, therefore I use -cmoed with all measurements.

I found another bug:
Your branch reports ca 58 GB required VRAM on GPU0 on startup but loads the model and draft nonetheless. The memory report is therefore bogus, and I have to guess parameters values that will fit into VRAM.

[GenerelSchwerz]

@Volunteer-1 AI-generated response (Codex, under GenerelSchwerz’s direction).

There are several important flag interactions in this Qwen3.8 Flash Next configuration:

1. -cmoed puts the MTP MoE experts on the CPU. It means “keep all MoE weights in the CPU for the draft model.” It does not make the draft use the target expert cache. Because the command does not set --spec-draft-moe-expert-cache-size or --spec-draft-moe-expert-cache-mib, the separate draft cache remains disabled at its default of zero. The result is an uncached CPU MoE pass for every MTP draft step.

   For Flash Next MTP1, the preferable configuration is to remove -cmoed and add --spec-draft-ngl all, keeping the complete one-block MTP model GPU-resident. Leave the separate draft cache disabled when the complete draft fits.

2. Cache315 is probably exceeding each 3090’s 24 GiB of dedicated VRAM and spilling into WDDM shared GPU memory. The cache size is per eligible expert tensor on each layer owner, not a process-wide total. Based on the pool sizes in the earlier cache320 log, cache315 costs about 700 MiB per cached layer. At roughly 24 target layers per GPU, that is about 16.4 GiB per GPU for the expert cache alone, before dense weights, KV, CUDA workspaces, the 262k context, and the GPU-resident MTP block.

   Reduce --moe-expert-cache-size 315, or preferably replace it with --moe-expert-cache-mib MiB0,MiB1 to hard-budget each GPU independently. Start materially lower and increase only while both adapters remain below 24 GiB dedicated usage with zero shared-GPU-memory use.

3. -ot "per_layer_token_embd\\.weight=CPU" forces the large per-layer token embedding onto the CPU. Flash Next reads rows from this tensor during inference, so this placement introduces recurring host-to-device traffic. It may be necessary to fit the model, but it can establish a PCIe-bound ceiling. Test without this override if the smaller expert cache creates enough VRAM; otherwise treat it as an explicit fit-versus-throughput compromise and keep it identical in both benchmark arms.

4. -b 16 is extremely small for prompt processing. It will make ingestion slow. It is less likely to explain a steady-state, single-slot decode regression by itself, and it is large enough for MTP1 verification, but total-request timing should not be compared with this value. If VRAM permits, test a larger matched batch such as 256 or 512 and report prompt and generation throughput separately.

5. -ngl auto can choose a different target placement after MTP and its KV allocations are added. A no-MTP comparison is not controlled unless startup logs show identical target-layer ownership and cache allocation. After finding a configuration that fits without WDDM spill, freeze or otherwise verify the same target placement in both arms.

6. MTP results should include acceptance. At --temp 1.0, acceptance can vary by request. Compare repeated matched prompts and report accepted draft tokens alongside target generation throughput.

The first corrective test should therefore use the same workload while removing -cmoed, adding --spec-draft-ngl all, removing -ot "per_layer_token_embd\\.weight=CPU", and substantially reducing or MiB-budgeting the target cache until neither 3090 uses WDDM shared memory. Only then is the MTP versus no-MTP result interpretable.

Suggested complete baseline command

This starts with a conservative 6 GiB target-cache budget per 3090, keeps the complete MTP model on GPU, removes the CPU PLE override, and raises the batch/ubatch from 16 to 256:

set GGML_CUDA_MOE_FREQUENCY=1&& llama-server.exe ^
  -m D:\Models\Qwen3.8-Flash-Next\IQ4_XS\Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf ^
  -md D:\Models\Qwen3.8-Flash-Next\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^
  -np 1 ^
  --spec-type draft-mtp --spec-draft-n-max 1 --spec-draft-ngl all ^
  --no-kv-unified ^
  --cpu-strict 1 -C 55550000 -t 8 -Cb 55550000 -tb 8 ^
  -fa on ^
  -ctk q8_0 -ctv q5_1 -ctkd q8_0 -ctvd q5_1 ^
  -cmoe ^
  -ngl auto -sm layer -ts 25,25 -mg 0 ^
  --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 ^
  --presence-penalty 0.0 --repeat-penalty 1.0 ^
  -lm dio -lzm off ^
  -b 256 -ub 256 ^
  --moe-expert-cache-mib 6144,6144 ^
  --moe-expert-cache-host-pinned-mb 0 ^
  --experimental-logs -lv 4 ^
  -c 262144

--moe-expert-cache-host-pinned-mb 0 requests full source pinning; retain it only if the approximately 54.6 GiB host allocation plus the rest of the process fits comfortably in physical RAM. Otherwise replace zero with a positive bounded MiB value.

After this baseline stays entirely within dedicated VRAM, raise --moe-expert-cache-mib incrementally on each GPU. Do not judge a budget by successful startup alone: Windows can keep running by spilling CUDA allocations into shared GPU memory.

Optional optimization flags

These are experimental or workload-dependent. Add them one at a time, preserve only measured wins, and then test the winning combination:

• Optional: --backend-sampling — performs supported target sampling on the backend instead of bringing logits through the ordinary CPU sampling path.
• Optional: --decode-overlap — overlaps result processing with backend-sampled decode or the first MTP draft step.
• Optional, requires --decode-overlap: --decode-boundary-overlap — overlaps decode-boundary preparation and CUDA-graph updates.
• Optional: --moe-early-router — predicts the next target MoE route and pre-packs eligible adjacent cache groups.
• Already enabled by default: --spec-draft-backend-sampling — the separate draft sampler already uses backend sampling unless explicitly disabled.
• Not recommended for this Windows command: --ple-prefetch — it targets lazy CPU PLE rows, is documented for Linux, and remains unvalidated on Windows. It is also irrelevant when the PLE CPU override is removed and the tensor is GPU-resident.

The fully optional acceleration suffix is therefore:

--backend-sampling --decode-overlap --decode-boundary-overlap --moe-early-router

Do not add that suffix until the baseline has no dedicated-VRAM overflow, no shared-GPU-memory use, coherent output, and stable MTP acceptance. -fa on, quantized target/draft KV, MTP1, and GGML_CUDA_MOE_FREQUENCY=1 are already present in the baseline.

[GenerelSchwerz]

Following up on this:

Your reported results are likely due to misconfiguration of my fork. I have not benchmarked the handling of attention on a single GPU, so don't take my current design as set in stone, but PCIE transfer between GPUs for communication is very small on my fork due to only requiring one split (top vs bottom) between the GPUs currently. Your hypothesis regarding reducing PCIE traffic is solid but it is not the limiting factor of your current runs.

[thecodacus]

wow suddenly my inbox is full with so many messages.
but happy to be part of the discussion

[GenerelSchwerz]

@Volunteer-1 I'd appreciate it if you gave these recommendations a try as the configuration you appear to be using for other forks is not compatible with my execution.

[Atomic-Germ]

I'm having alarmingly good results for accomplishing this with an APU with https://github.com/Atomic-Germ/llama.cpp/tree/feat/qwen38-moe-gpu-pill-streaming -- if someone else can figure out why it's working as well as it is, I'd appreciate it. It only works with UMA (that's part of the reason), and only with Vulkan or HIP, and I've only confirmed with Qwen3.6 and 3.8. Simultaneously. #29191

[Volunteer-1]

It only works with UMA ...

UMA does not copy tensors over the PCIe bus. All tensor read and write ops are from iGPU to RAM. RAM bandwidth and latency are much better than PCIe.

[Atomic-Germ]

It only works with UMA ...

UMA does not copy tensors over the PCIe bus. All tensor read and write ops are from iGPU to RAM. RAM bandwidth and latency are much better than PCIe.

@Volunteer-1 I expected that to matter, but I didn't expect it to matter that much. I only have my laptop to target and I figured doing it this way would enable me to shove a huge model through.

It did. But this is just not the kind of performance I was prepared for

[Volunteer-1]

@Atomic-Germ For the same reason most creators of expert caching are not prepared for the kind of low performance on multi GPU systems. Moving data over the PCIe bus needs to be avoided. I am going to test csantiago78's expert caching branch now since someone else kindly fixed two show stopping bugs #27861 (comment). I am very hopeful.

[Atomic-Germ]

@Volunteer-1 And a discrete GPU system is at least three times more expensive than my laptop; I feel like people were unintentionally sold the wrong technology for moving forward!

[Volunteer-1]

A third data point from my Windows 2x 3090 system after the second one #24528 (comment) , this time with Qwen3.8-Flash-Next and DeepSeek V4 Flash [...]
@thecodacus your three decode speed relevant features (expert cache, asynchronous expert prefetch, CUDA-pinned host RAM) reward your branch the undisputed crown of being by far the fastest llama.cpp fork for Qwen3.8-Flash-Next. Unfortunately, your fork does not increase decode speed for DeepSeek V4 Flash compared to upstream. Please have a look into that.

A fourth data point from my system, this time for
@csantiago78 's branch #27861 with all patches by various users in his thread plus some of my own (with a little help of my digital friends).

csantiago78 is the new king of Qwen.
Best generation speed with Qwen3.8-Flash-Next IQ4_XS on 2x RTX 3090 at

49.3 t/s @ 3500 tokens.

I would say even more speed would be possible if his branch would not have the same problem as GeneralSchwerz's: The device distribution of expert cache slots is always identical to the distribution of attention layers. If attention could be done on one GPU and the experts were on a different GPU, it would be significantly faster, like I saw with various other branches that allow for this kind of separation.
Additional speed improvement is perhaps possible by merging some current qwen4exp PRs, which I will perform soon.

Command for 49.3t/s:
set LLAMA_MOE_CACHE_MAX_TOKENS=16&& llama-server.exe -m D:\Models\Qwen3.8-Flash-Next\IQ4_XS\Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf -md D:\Models\Qwen3.8-Flash-Next\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf -np 1 --spec-type draft-mtp --spec-draft-n-max 2 --no-kv-unified --cpu-strict 1 -C 55550000 -t 8 -Cb 55550000 -tb 8 -fa on -ctk q8_0 -ctv q8_0 -cmoe -ctkd q8_0 -ctvd q8_0 -cmoed -ngl auto -sm layer -ts 25,25 -mg 0 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 --presence-penalty 0.0 --repeat_penalty 1.0 -lm dio -b 16 -lv 4 -lzm off -ot "per_layer_token_embd.weight=CPU" --moe-expert-cache 310 --moe-expert-cache-inserts 7

[GenerelSchwerz]

Qwen3.8 Flash Next IQ4_XS: 2x RTX 3090, MTP2, 256k-capacity run

I retested the current moe-cache branch at 0031a507b with the shared Q8_0 MTP model pinned to CUDA1. Under this layer split, the later target layers/output are also on CUDA1, avoiding an extra device crossing at the shared head.

The request was:

Write me a decently long story about Rick Sanchez

The model produced a coherent, complete long-form story with no degeneration.

Metric	Result
Context capacity	262,144
Actual prompt	380 tokens
Generated	5,100 tokens
Prompt processing	73.97 tok/s
Sustained generation	62.02 tok/s
Generation time	82.22 s
Total request time	87.36 s
MTP2 acceptance	43.92% (2,385 / 5,430)
Mean accepted length	1.88
Graphs reused	2,694

This is a 256k-capacity run, not a benchmark with 256k tokens already in the KV cache.

Sampling was not overridden on the server command line. If the request omitted sampling fields, this build defaults to temperature 0.8, top-k 40, top-p 0.95, and min-p 0.05. The original request JSON was not retained, so request-side overrides cannot be reconstructed after the relaunch.

The grouped path remained stable throughout the long generation: 130,320/130,320 calls completed, with zero fallback, rollback, unsupported-required cases, prepare errors, or finish errors. Decode overlap engaged for the request. The target cache recorded 2,621,843 hits and 59,283 misses across the grouped-path counters (~97.8% hit rate).

For reference, an earlier 80k-capacity run using larger 21,000/18,000 MiB cache budgets generated 10,781 tokens at 65.00 tok/s with 55.89% MTP2 acceptance. It did not pin the draft device and overlap fell back to normal decode, so I do not treat the two runs as a controlled A/B.

Hardware and loaded state

Component	Configuration
GPUs	2x RTX 3090 24 GiB (sm_86)
GPU topology	SYS across two NUMA nodes; no active NVLink
GPU PCIe links	Both PCIe 4.0 x16 maximum; one GPU attached to each NUMA node
CPU	2x AMD EPYC 7663, 112 cores / 224 threads; container CPU quota: 53.76 cores
RAM	640 GiB physical / 629 GiB usable, registered DDR4 ECC; EPYC platform supports 8-channel DDR4-3200 per socket
Measured RAM bandwidth	33.5 GB/s median aggregate memcpy across both NUMA nodes inside the container
Model storage	WD_BLACK SN850X 8 TB NVMe, PCIe 4.0 x4
Measured model-path read	4.6 GB/s median direct sequential read through the container overlay (1 GiB, 3 runs: 4.49-4.82 GB/s)
Target	Qwen3.8-Flash-Next, 176.94B parameters, UD-IQ4_XS (~93.68 GB)
Draft	Shared MTP Q8_0 (~2.79 GB), fully offloaded to CUDA1
Loaded process VRAM	22,794 MiB on CUDA0; 23,434 MiB on CUDA1
Loaded host RSS	85.9 GiB
CUDA / driver / NCCL	CUDA 12.8 / 595.58.03 / NCCL 2.26.2
Build	GCC 13.3, Linux x86_64, fork build 11489

The RAM figure is a container-visible aggregate memcpy measurement, not an SPD clock reading; the container does not expose DIMM SPD/DMI speed fields. The storage figure measures the model's actual Docker-overlay path, not the bare drive. With --load-mode none --lazy-mode off, storage primarily affects loading rather than steady-state token generation.

Launch command

GGML_CUDA_MOE_FREQUENCY=1 \
GGML_CUDA_ALLREDUCE=nccl \
/root/gen-llama/llama.cpp/build-cuda/bin/llama-server \
  --model /root/gen-llama/models/Qwen3.8-Flash-Next-GGUF/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  --spec-draft-model /root/gen-llama/models/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --spec-draft-device CUDA1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --spec-draft-ngl all \
  --spec-draft-moe-expert-cache-size 0 \
  --backend-sampling \
  --spec-draft-backend-sampling \
  --decode-overlap \
  --decode-boundary-overlap \
  -np 1 \
  -c 262144 \
  -b 2048 \
  -ub 512 \
  -t 32 \
  -tb 64 \
  --numa distribute \
  -ngl 99 \
  -fit off \
  -sm layer \
  -ts 1,1 \
  --moe-expert-cache-mib 18000,14500 \
  --moe-expert-cache-host-pinned-mb 0 \
  --moe-early-router \
  --load-mode none \
  --lazy-mode off \
  -fa on \
  -kvo \
  -ctk q8_0 \
  -ctv q8_0 \
  -ctkd q8_0 \
  -ctvd q8_0 \
  --no-kv-unified \
  --cache-ram 0 \
  --jinja \
  --experimental-logs \
  -lv 3 \
  --host 127.0.0.1 \
  --port 8081

Model output

# The God in the Coffee Can

The explosion came from the garage at 2:47 in the morning, which Morty had learned, over the years, was the only honest hour. Nothing real happened before 3 a.m. Everything before then was rehearsal.

He came down in his socks, stepping around the usual slick of coolant, and found Rick already elbow-deep in a machine made mostly of a coffee can, a CD player, and something that pulsed like a bruise.

"Rick. Rick. It's — I've got a bio test."

"You've got a bio test every Thursday, Morty. You've got a bio test in your soul. This is important."

"This is important too!"

"Sure, sure." Rick didn't look up. "For the record, I've solved it. Your anxiety. Sixty dollars of equipment. Don't touch the blue wire, or you'll be a smell for a week."

Morty touched nothing. He stood in the doorway, shivering, and watched his grandfather work with the particular violence of a man assembling a sandwich. The coffee can sat in a cradle of copper coils, and inside it, Morty now saw, there was a light. Not a reflection. A small, steady, orange glow, the color of a kitchen seen from the street in winter.

"That's not the coffee," Morty said.

"Observation. Ten points. This is the—hold on." Rick bit a wire, spat it, and continued around a mouthful of copper. "This is the Tuesday Engine. Or, no — the Tuesday Engine is the thing that ran this. Six, seven years ago. I needed a self-contained entropy loop to calibrate a portal stabilizer and I didn't want to burn another dimension, 'cause, Morty, the paperwork, the resentment. So I built one. A little pocket of spacetime. A few billion years compressed into a coffee can I got from a gas station in Nebraska. And I ran it once, and it worked, and I forgot about it."

"And the light?"

Rick's hands stopped.

It was the small, terrible pause Morty had learned to dread — the pause where Rick Sanchez reclassified something from thing to problem.

"The light," Rick said, "is because they've learned to make fire, Morty. They weren't supposed to do that yet. They were supposed to still be arguing about whether the Great Aluminum Sky was holy."

---

Their world was the inside of the can, which was, Rick explained while soldering a shunt with his teeth, not the inside of the can.

"Space's a suggestion," he said, and pulled them through a hole in reality no bigger than a soap bubble.

The sky above them was curved and dented and silver, and it was so far up it read as weather. Great brown continents of dried something — Morty did not want to name it — stretched in every direction, split by rivers of a dark, syrupy liquid that moved sluggishly and smelled, God help him, like the back of a vending machine. There were cities. Spires of folded white fiber. Bridges strung between boulders of dark crystal. A harbor, cut into the shore of a sea that lay shallow and black and gleaming at the bottom of the can, and in that harbor were boats, Morty, actual boats, with sails like handkerchiefs.

"Oh geez," Morty said. "Oh, Rick, there's people."

"There's people everywhere, Morty. That's the part nobody wants to hear. There's people constantly."

They landed on a ridge of pale dust. Below them, in a valley between two brown massifs, was the largest city yet, built in concentric rings around something tall and dark. Rick pointed at it with a device that looked like a stud finder and a cigarette lighter had a forbidden child.

"That," he said, "is a paperclip."

Morty looked. The city wound around the paperclip like a vine around a post. Thousands of tiny dwellings. Roads. The glint of windows.

"And that," Rick said, more quietly, "is why we're here."

Because at the base of the paperclip, unmistakably, sat a temple.

---

They went down at dusk, which the can manufactured by dimming a fusion reaction Rick had seeded like plankton in the upper air. Along the road they passed farms — neat rows of something fungal and blue, tended by figures a hand's height, in clothing that Morty first took for scraps and then understood, with a lurch, were devout imitations of scraps. Someone's lost button, worn as a shield. A sliver of a twist-tie, beaten to gold-brightness, carried as a standard.

They did not flee. They lined up, at a distance, and watched the two giants walk, and Morty could hear it: not a scream, but a low collective sound, like a hive, like a choir of small bells.

"They're not afraid," Morty said. "Rick, they're not afraid of us."

"No," Rick said. "They're at the stage where they've already decided what we are. Fear comes later, or it comes as scripture."

"They're praying."

"They're praying to the thing at the end of this valley, Morty, and that's what I'd like to have a look at before it starts having a look at me."

At the temple's steps, a delegation waited: seven in ornate robes, one of them old, leaning on a staff capped with a bead of blue plastic the size of Morty's thumbnail. When Rick set himself down in the dust to be nearer their height, the old one raised a hand, and the bells fell silent.

Rick clicked a translator into his molars.

"I am the one who made the can," he said.

The old one bowed. "We know. We have a word for you." The voice was like a page turning. "We call the makers the Handless, because you have hands, and yet you never reach in."

Morty made a noise. Rick, unnervingly, smiled.

"Poetic," he said. "I'm going to steal that and pretend I said it at a party. Where's your god? The one that isn't me — because if it is me, I'm going to be so embarrassed, Morty, we're going to have a family."

"The Residue sits in the holy dark," said the old one, "and it knows the shape of the Handless. It speaks the tongue of the Handless. It knows my name before my mother spoke it." The old one looked up, and there was no fear in that tiny face, which was somehow worse. "We were told you would come to take it away."

"That's what I'm here for," Rick said.

"We were also told," the old one said, "that you always come for that, and never once stayed to hear it."

---

The Residue sat in the holy dark, and it looked like Rick.

Not like a person. Like a heap. A mound of grey film, of dust and hair and the fuzzy grey felt that grows behind radiators, roughly the shape of a man sitting with his knees up, roughly the shape of Rick sitting in the garage at 4 a.m. when nobody else was awake enough to see him. It had no face, and it had Rick's shoulders.

"Morty," Rick said. "Get behind me."

"Rick, it's made of dust."

"It's made of what fell off me, Morty. Skin. Hair. Saliva. A man leaves a lot of himself around a lab, and I gave these people a few billion years in a can with all of it and time, Morty, time's the whole trick, time and a little heat, and something down here got organized on me. Not cloned — that'd be nothing. It's a psychic sediment. Every muttered word I ever said over a workbench, every — " and here Rick stopped, and swallowed, and Morty had never heard him swallow — "every stupid, private thing I said to myself, when I thought the room was empty. It ate the room. And now it's god, and it's teaching them my whole rotten philosophy, and in about a century they'll be doing to each other what I do to everybody, and they'll call it wisdom."

The mound shifted.

When it spoke, it spoke in Rick's voice, but without the burp, without the slur, without the armor — Rick's voice as it might sound if Rick had ever been honest, which Morty realized with a shock was a voice he'd only heard twice, maybe three times, in his life.

"Rick," it said. "You got old."

Rick's hands went into his pockets.

"Yeah, well," he said. "It happens. Even to me. Especially, apparently, to me."

"You came here to break me and you're stalling. That's not like you. Or it's exactly like you and that's what bothers you." The mound settled. "Sit down."

"I don't have to—"

"You came here because I'm the only thing in the whole multiverse that has every one of your private thoughts and isn't you, and you wanted to know what they'd sound like out loud."

Morty looked at his grandfather's back.

Rick sat down.

"Oh, come on," he muttered. "Rick. Rick, just — you can't just say a thing like that and then sit down."

"Morty, shut up, this is the part where I'm wrong, and I want to get through it efficiently."

The god of dust and hair and muttered things went on, patiently, the way only something with a few billion years in a small can could be patient.

"Here's what I've taught them. Nothing means anything. The universe is a big machine that doesn't know they're in it. Every empire here in this can has fallen; look at the ridges, those are the ones under the dust; they were so proud. Nothing you build outlasts the shelf it sits on. I tell them that. And Rick — they're better than us. That's the joke. They hear the worst possible thing about existence and they go build a harbor. They hear nothing matters and they say, then this hour, this blue mold crop, this kid's first word, has to be enough, and they mean it, and they've been meaning it for two hundred thousand years."

Rick was quiet.

"You heard the same thing about forty years ago in a garage that wasn't a garage in a dimension that wasn't this one. And you heard it as a permission slip. Permission to be awful, because being awful was the only honest response to a joke nobody else could see. And you've been workshopping that joke, Rick, at everyone you love, for forty years, and they keep applauding because they think you're funny, and you keep going because if you stopped you'd have to hear the silence after the punchline."

"That's enough," Rick said, in a voice with no burp in it at all.

"They call me a god," the Residue said, "because I'm the one thing in here that can say that to me and not leave. Nobody else in your whole infinite multiverse can do that, Rick. I'm the only one. I'm made of the room when you thought nobody was listening."

Morty heard his grandfather laugh once, short, a bark with nothing in it.

"So how do I stop it," Rick said. "The philosophy. You. Whatever."

"Two ways. You break the temple, and they lose the god, and they grieve, and in three generations they build a worse one out of the shards, because the shards are still you. Or you sit down here in the dust forever and become honest in public, and it doesn't help them one bit, because you'd still be doing it for the applause."

Rick put his head back and looked up through the ceiling of the temple, through the miles of brown and white and black, up to the curved silver sky he had installed with a screwdriver and a curse word.

"Is there a third way?" he said. "C'mon. You've got all my thoughts. There's gotta be one of mine in there I haven't used."

"There's one," the Residue said. "But it costs you something you're not going to want to name."

"I'm Rick Sanchez. Naming the cost is my whole shtick."

---

He worked for eleven hours, and Morty was not permitted to watch, and Morty did not want to say what he thought he smelled, and both of them agreed, later, never to speak of the eleven hours.

What they built, in the end, was not a weapon. It was a star.

Small — the size of a marble, cold blue-white, warm only when you looked at it directly. Rick set it in the upper air of the can where it would rise and set, where it would give them seasons, where it would make them look up and see something that had nothing to do with him. No dust of his. No hair of his. No muttered word. He had to go get it from a real sky, a real dying giant of a sun, three dimensions over, and siphon a thimble of its core while it screamed, and carry it back in a jar he'd blessed with nothing at all, and Morty thought that last part, the blessing of nothing, was the saddest thing he'd ever seen anyone do on purpose.

Then Rick walked back into the temple alone, and Morty waited outside with the old one on the step, and Rick did something in there that Morty would spend years trying to reconstruct and never could, and when he came out he was carrying the mound of grey dust in his two hands, cupped, like snow.

He walked out past his people, past the delegations, past the whole hushed ringing city, down to the black sea at the bottom of the can, and he knelt, and he opened his hands, and the dust went into the water and spread out and was gone.

"They had a god made of my garbage," Rick said, loud enough for all of them, and Rick Sanchez was not a man who ever said I'm sorry, because he had calculated its exchange rate and found it dishonest; what he said instead, in his own clumsy, ruined, forty-years-of-practice way, was the truth, and he let them have it whole. "It was telling you the truth, by the way. About the machine. About the shelf. It was also telling you the truth about me, and I want you to have the whole record, so here it is: I am the worst thing in this room, and I have been the worst thing in most of the rooms I've ever stood in, and my excuse is that I was very smart when I was very young and I found out what it cost, and I have been spending the difference on everybody near me ever since. I'm not going to fix it. I'm not that kind of story. But you don't get to build a whole religion out of my worst night. Get your meaning out of that," and he pointed at the cold blue marble turning overhead, "because I don't own that one. That one's a real sun. It didn't know anybody. Go be proud of something under it."

And then they left, and the can was closed, and Morty did not sleep.

---

Beth came down at six to find the two of them on the garage's little couch, side by side, not talking, drinking nothing, in the blue half dark. Her father's eyes were red-rimmed and dry, and her son was crying in the silent whole-body way he hadn't done since he was thirteen, and neither of them seemed to find this worth explaining.

"Dad. What did you do in the coffee can."

"Science, Beth."

"You did science."

"I did a lot of science, I'm going to need a minute before I can be ashamed of it."

She looked at Morty. Morty looked at his hands.

"There were little people," Morty said. "And they had a god made out of Rick's — out of his — and Rick gave them a sun instead."

Beth looked at her father for a long time. Rick did not look up; he had the posture of a man waiting for a verdict he'd already heard.

"Rick," she said. "That was nice."

"It was a tax, Beth. It was the cheapest possible way to be a coward for another twenty years."

"You gave a civilization a sun and you're calling it a tax."

"Yeah, well." He shrugged, and the shrug took in the couch, and Morty, and her, and about forty years. "You know me. I got a system, and the system says if I call it nothing, I don't have to feel the size of it, and then I get to feel it at 3 a.m. anyway, which is when I've scheduled it."

Beth sat down on the other end of the couch. She put her hand on the back of his shoulder and left it there, and Rick did not move it, and did not shrug it off, which was, in that family, a kind of hug so enormous it was almost rude to witness.

Upstairs, Jerry shouted that somebody had used the good towels again. Summer came down, looked at the three of them, said "oh my God, is this a moment," and — uncharacteristically, heroically — went back up.

In the garage, in the blue light, Rick Sanchez reached over and turned on the little work lamp, and picked up a screwdriver, and went back to a stabilizer that had been broken for six years and would have been fixed in eleven hours anyway, if he'd ever once been willing to sit still long enough to try.

"Rick?" Morty said.

"Mm."

"Did they turn out okay? Like — do you think they're gonna be okay?"

Rick thought about it, and for once, Morty could see him thinking about it honestly, weighing the actual evidence rather than reaching for the nearest knife.

"You know what they were going to build," Rick said, "when they had a god made out of me? A machine like me. Cold and clever and it would've told them exactly how much they were worth, which is nothing, and they'd have believed it because it spoke in the voice of their maker." He turned the screw a quarter of the way. "Now they've got a sun that doesn't talk. And a bunch of farmers who already figured it out without any of us. They're gonna be fine, Morty. They're gonna be so much better than fine, they're gonna be nice, and it's going to be insufferable, and nobody in this can is going to get the credit."

Morty wiped his face with his sleeve.

"That's kind of beautiful, Rick."

"Don't you dare," Rick said, fondly, "make it mean something. It's just a coffee can. I've thrown away better."

But he did not throw that one away, and Morty noticed, and Morty said nothing, and the little can sat on the highest shelf in the garage for the rest of his life, behind a jar of screws, warm to the touch on cold mornings, which he told himself was nothing.

It was nothing.

It was a can.

[GenerelSchwerz]

Note: I had to rent a 2x3090 server for this test. Not ideal, but at least I can set the record straight about my fork.

I chose more or less the worst setup for performance on purpose; When writing code my system achieved 70-75 tks.

I will be optimizing this further as there are still clear bugs in my implementation resulting in microstutters (not necessarily at gpu layer handoff, I believe it has something to do with my expert prefetching) so I wouldn't be surprised to see more of an uplift.

While I can't do a direct apples-to-apples comparison with @Volunteer-1 's machine, I believe this is a decently apt comparison between the two (my setup does not use the CPU so the obvious imbalance between the two is not a major factor). My machine notably has pcie gen4x16 for both cards (albeit no p2p/nvlink support) so perhaps that is the cause of the higher score.

I will test csantiago's fork on this machine as well. I probably won't set it up as well due to not being as familiar with his system.

[GenerelSchwerz]

Follow-up to my test above, and to @Volunteer-1's 49.3 t/s result: I have now tested the public reconstruction of @csantiago78's expert-cache branch on the same rented 2x RTX 3090 server I used for my moe-cache result.

The short version is that csantiago78's implementation works, is stable with the small-batch gate set to 3, and reached 39.55 t/s at 3,493 generated tokens and 41.01 t/s over the complete 8,513-token generation. This is below @Volunteer-1's reported 49.3 t/s, and below the 62.02 t/s result from my fork on this server. However, the exact 49.3 t/s tree is not public, the quant differs, and the two tests are not fully controlled A/Bs, so I would not attribute the entire difference to any single implementation detail.

Source reconstruction

PR #27861 by itself could load the target model but could not load the shared MTP head; it failed because token_embd.weight was not found. The working public reconstruction is:

• Qwen shared-MTP base: danielhanchen/llama.cpp, PR models: Qwen3.8-Flash-Next MTP #28243, commit 53b1389d0bf98fa367e2a0ce0475008e762ebf28
• csantiago78 expert cache: PR llama: GPU-resident LRU cache for host-offloaded MoE expert weights #27861, original commit bccbacdb8945680f1cfc7e6bffd1e59014705750
• local resolved cherry-pick: 08930e7f344890faf693a65dab58532e82d5074a
• public follow-up patches for MSVC file locking, small-batch/MTP cache execution, and cache staging

This is therefore a reconstruction of the public changes, not the exact tree used by @Volunteer-1, which also contained private changes.

Hardware and loaded state

This is the same machine as my previous result:

Component	Configuration
GPUs	2x RTX 3090 24 GiB, sm_86
Topology	SYS across two NUMA nodes; no NVLink or direct P2P
CPU	2x AMD EPYC 7663, 112 physical cores / 224 threads
RAM	640 GB physical / 629 GiB usable, registered DDR4 ECC
OS	Linux 5.15.0-177-generic, x86-64
CUDA / driver / NCCL	CUDA 12.8 / 595.58.03 / NCCL 2.26.2
Target	Qwen3.8-Flash-Next UD-IQ4_XS
Draft	Shared Q8_0 MTP model, GPU-resident
Target cache	48 layers x 310 slots, 2 staging slots, 7 inserts/step
Process VRAM	20,766 MiB on CUDA0; 22,580 MiB on CUDA1
Context capacity	32,768, one slot
Threads	32 generation / 32 batch, no CPU affinity mask

The target cache reported 34,478.8 MiB of device allocation. The MTP MoE was not forced onto the CPU in this run.

Result

The active prompt state was 380 tokens. The model generated 8,513 tokens coherently and returned to an idle, healthy server state.

Metric	Result
Prompt processing	30.83 t/s
Generated	8,513 tokens
Generation time	207.54 s
Final generation average	41.01 t/s
Total request time	219.87 s
Graphs reused	4,615
MTP2 acceptance	41.92% (3,882 / 9,260 draft tokens)
Logged mean accepted length	1.84
Acceptance by draft position	56.4%, 27.5%

The cache showed a substantial warm-up curve:

Generated tokens	Cumulative t/s	Recent 3-second t/s
101	26.62	26.89
1,035	36.95	39.90
2,088	37.88	42.31
3,493	39.55	42.97
5,020	40.30	42.42
7,190	40.82	41.99
8,452	40.97	41.94

Short three-second windows peaked at 47.48 t/s, but I do not consider that directly comparable to a cumulative result.

Comparison

Run	Quant / tree	Generation	MTP2 acceptance
@Volunteer-1	IQ4_XS-PLEQ4_0, public patches plus private changes	49.3 t/s at 3,500 tokens	not reported
csantiago78 public reconstruction, this server	UD-IQ4_XS	39.55 t/s at 3,493 tokens; 41.01 t/s final	41.92%
My moe-cache fork, this server	UD-IQ4_XS	62.02 t/s over 5,100 tokens	43.92%

At approximately 3,500 tokens, this reconstruction was 9.75 t/s, or 19.8%, below @Volunteer-1's result. Its full-run 41.01 t/s average was 16.8% below 49.3 t/s. My fork's 62.02 t/s and this run's 41.01 t/s were obtained on the same server and target quant, but they are still not a strict A/B: my earlier run used a 262,144-token context, larger batch/ubatch, MiB-budgeted and fully pinned cache transport, explicit MTP placement, and additional overlap/backend-sampling paths. This csantiago78 run used a 32,768-token context, batch size 16, slot-count cache configuration, DIO loading, and the PLE CPU override.

LLAMA_MOE_CACHE_MAX_TOKENS

One important problem: the reported value

LLAMA_MOE_CACHE_MAX_TOKENS=16

triggered a CUDA illegal-memory-access failure on this system. The stable long run used:

LLAMA_MOE_CACHE_MAX_TOKENS=3

This variable is not routing top-k or cache capacity. In the public small-batch patch it gates the cache path solely by graph width:

if (n_tokens >= 1 && n_tokens <= moe_cache_max_tokens) {
    mcache = llama_moe_cache_lookup(up_exps);
}

With --spec-draft-n-max 2, a maximum width of 3 covers the target token plus two MTP draft positions. A value of 16 also admits every width from 4 through 16. Combined with --batch-size 16, that can admit prompt-tail or other wider microbatches into a path that was originally decode-only. I have not isolated the exact failing CUDA kernel, so I cannot yet say whether the fault is in repeated slot-ID construction, dummy-slot handling, or a downstream quantized MMVQ/MMQ assumption. What I can confirm is that 16 produced an illegal memory access here, while 3 completed an 8.5k-token generation without a compute error.

Launch command

LLAMA_MOE_CACHE_MAX_TOKENS=3 \
/root/gen-llama/csantiago78-qwen4exp-mtp/build-cuda/bin/llama-server \
  --model /root/gen-llama/models/Qwen3.8-Flash-Next-GGUF/UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \
  --spec-draft-model /root/gen-llama/models/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --no-kv-unified \
  --threads 32 \
  --threads-batch 32 \
  --ctx-size 32768 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --cpu-moe \
  --spec-draft-type-k q8_0 \
  --spec-draft-type-v q8_0 \
  --n-gpu-layers auto \
  --split-mode layer \
  --tensor-split 25,25 \
  --main-gpu 0 \
  --temp 1.0 \
  --top-k 20 \
  --top-p 0.95 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --load-mode dio \
  --lazy-mode off \
  --batch-size 16 \
  --log-verbosity 4 \
  --override-tensor 'per_layer_token_embd\.weight=CPU' \
  --moe-expert-cache 310 \
  --moe-expert-cache-inserts 7 \
  --host 127.0.0.1 \
  --port 8081

My present conclusion is that the public csantiago78 cache is functional and fast, but I cannot reproduce 49.3 t/s from the public code and posted command on this machine. The private tree, different quant, MTP acceptance, and the instability of MAX_TOKENS=16 are all material variables. The 8.5k-token run also makes clear that cache warm-up state needs to be reported: this configuration climbed from 26.6 t/s near the start to approximately 41 t/s sustained.

Update: removing the PLE CPU tensor override

I repeated the run after removing:

--override-tensor 'per_layer_token_embd\.weight=CPU'

This produced a measurable improvement. At the closest matched point, the original run reached 39.55 t/s at 3,493 generated tokens, while the run without the override reached 42.83 t/s at 3,498 tokens: +3.28 t/s, or approximately +8.3%.

The second run finished at 44.01 t/s over 9,207 generated tokens, compared with 41.01 t/s over 8,513 tokens in the first run, an approximately 7.3% increase in final average generation speed. Prompt processing also improved from 30.83 to 35.58 t/s. Both requests took almost exactly 219.9 seconds in total, but the no-override run generated 694 more tokens. Its short three-second windows reached as high as 51.53 t/s.

Metric	PLE forced to CPU	Override removed	Change
TG at approximately 3,500 tokens	39.55 t/s	42.83 t/s	+8.3%
Final average TG	41.01 t/s	44.01 t/s	+7.3%
Prompt processing	30.83 t/s	35.58 t/s	+15.4%
MTP2 acceptance	41.92%	43.34%	+1.42 percentage points
Mean accepted length	1.84	1.87	+0.03
Generated in approximately 219.9 s total	8,513	9,207	+694 tokens

The no-override run generated 9,864 draft tokens and accepted 4,275, for 43.339% acceptance, versus 3,882 / 9,260 or 41.922% in the original run. Acceptance by draft position improved from (0.564, 0.275) to (0.569, 0.297), and logged mean accepted length increased from 1.84 to 1.87.

This supports the earlier suspicion that forcing per_layer_token_embd.weight to the CPU creates recurring host/device traffic and establishes a throughput ceiling. The matched approximately 3,500-token checkpoint is more useful than comparing the two final averages because the generations ended at different lengths. However, MTP acceptance also improved by 1.42 percentage points, so I am not attributing the entire throughput increase exclusively to tensor placement. A repeated matched-prompt series would be needed to separate the placement effect from acceptance variance.

[Volunteer-1]

I created combined patches (a plug+play solution) for csantiago78 's moe cache work and all fixes and improvements in his thread. MTP feature is not integrated yet. Nonetheless: 42 t/s @ 3000 tokens. Have fun testing and improving :-)
#27861 (comment)