RFC: Persistent expert slot pool for MoE CPU offload (--moe-expert-cache), +84% decode

[memoriaru]

This follows up on #24528 (leloch's CUDA-side adaptive cache, whose PR #24524 was closed for scope) with a different design point: zero kernel changes via id remapping, plus a prefill fallback rule

RFC: Persistent expert slot pool for MoE CPU offload (--moe-expert-cache)

Draft for a ggml-org/llama.cpp discussions post. Follows up on feature request
#20757. Companion branch:
moe-expert-pool. Status: validated on consumer hardware (RTX 4090, CUDA):
perplexity-equivalent, long-session stable, +34% decode on a real code-generation
workload on top of +84% on the neutral benchmark.

Summary

Add --moe-expert-cache N (-mec N): for every MoE expert weight tensor that offloading
placed in host memory, keep a persistent pool of N expert slots in accelerator memory.
Cache hits serve a decode step with zero host-to-device traffic; misses copy the
expert in and evict the least recently used slot. Expert ids are remapped to slot ids
through a per-tensor map table applied with ggml_get_rows — no new ggml op and no
backend-specific code — so CUDA, Vulkan, Metal and CPU all work unchanged.

Measured on a single RTX 4090 24 GB, Qwen3.8-Flash-Next (qwen4exp, 48 MoE layers, all
experts on CPU, UD-Q3_K_XL, 8 K context, greedy):

-mec	decode tg512 (t/s)	vs baseline	prefill
0	11.55 ± 0.41	—	pp512 104.5 ± 9.9 / pp2048 113.4 ± 5.7
16	12.89 ± 0.33	+12%	—
32	17.54 ± 0.15	+52%	pp512 119.1 ± 10.9 / pp2048 112.9 ± 4.8
48	19.09 ± 0.62	+65%	—
64	21.21 ± 0.46	+84%	—

On a real code-generation workload served through llama-server (173-token prompt,
512 generated tokens, warm rounds): 12.82 → 14.10 (+10%) at N=32 and → 17.14 (+34%)
at N=64; a long multi-task session (2561 tokens with pool-churning 1 K-token
generations interleaved) held anchor outputs byte-identical across four runs with
no throughput decay. Greedy decoding with the pool active is perplexity-equivalent
to the host-copy path (PPL 3.3096 ± 0.053 vs 3.3262 ± 0.053, deterministic reruns;
difference well inside the error bars).

Prefill is unchanged within noise by construction (below). A backend-level matrix test
passes bit-exact on CUDA (RTX 4090) and Metal (AMD dGPU): 5 quant types × dual pools
sharing one routing × ubatch sizes 1/3 × cold / full-eviction / hit+reload /
graph-rebuild. Server-level greedy A/B initially diverged deterministically from the
host-copy path; root cause was found and fixed (details in Validation — including one
invalid verification round we are disclosing rather than hiding).

Problem

With --cpu-moe / --n-cpu-moe (or the auto-fitter), decode re-streams every selected
expert over PCIe on every token: the selective expert copy added in #15346 removed the
unused experts but still copies the same hot experts token after token. Routing is
skewed, so for code-generation-style workloads the same small expert set is re-read
thousands of times. #20757 requests a cache for exactly this.

Why not the other approaches on the table

• --prefetch-weights (#21067)
  overlaps next-layer transfers with compute but cannot know the next layer's routing;
  for MoE it was measured moving 2.06× the bytes with +47.8% TTFT. Complementary, not
  competing: prefetch targets dense/prefill transfer overlap, this targets decode
  residency. The two can share the copy-stream infrastructure.
• Dynamic caches with SLRU/probation (original Feature Request: Two-tier GPU+RAM expert cache for MoE offload (pluggable eviction policy) #20757 proposal) let prefill bursts
  evict the decode hot set; a frequency-gated admission filter was needed to recover.
  This design sidesteps the problem structurally: a ubatch that can touch more
  distinct experts than the pool has slots never goes through the pool at all — it
  takes the existing selective-copy path. Prefill therefore behaves exactly like
  master (verified: pp512/pp2048 unchanged), and prefill never evicts decode state.
• Prior PRs in this space (ggml : persistent expert cache for --n-cpu-moe (RFC #20757) #21614/ggml : dedup expert cache for MoE CPU offload #21620/cuda: MoE expert cache, adaptive VRAM caching of CPU-resident experts  #24524) were closed for scope/review reasons;
  this branch is deliberately small in backend impact: zero kernel changes, one
  scheduler hook, ~700 lines total of which ~350 are the scheduler core.

Design

1. ggml_backend_sched_register_expert_pool(sched, w, backend_id, n_slots, &table)
   allocates n_slots * expert_size (+ a small NaN-safe tail) on the compute backend
   and a host-side I32 map_table shaped [1, n_expert] (flagged as a graph input).
2. llm_graph_context::build_lora_mm_id — the single funnel for all MoE expert
   matmuls — routes through the pool when one is registered for the tensor and the
   ubatch cannot select more distinct experts than slots (n_expert_used * n_tokens ≤ n_slots). The remap is get_rows(table, cont(ids)): existing ops only.
3. The map-table buffer is registered with the scheduler's pooled-split boundary check,
   so the remap GET_ROWS always anchors its own split. In that split's prologue the
   scheduler reads the (original) expert ids back to the host, updates LRU state,
   issues async H2D copies for misses and rewrites the map table — before the same
   split's input copies upload the fresh table. (This ordering is the fix described
   below; anchoring the update anywhere later lets the remap consume a stale table.)
4. Safety rails: total pool memory is capped at half the device's free memory at load
   time (tensors beyond the budget silently keep the selective-copy path); pools are
   disabled under pipeline parallelism; slot-overflow is a hard assert instead of
   silent corruption.

Slot sizing: N must cover the decode working set, not just top-k — N = top-k is a
full-miss worst case and measurably slower than master (the per-layer id readback
synchronization has nothing to buy). Start at 2–4× top-k and scan; gains are
routing-skew dependent (code workloads benefit most, flat-routing models least).

Validation — including a bug we found, mis-verified once, then actually fixed

The full evidence chain (raw captures, the invalid round, the fix, the bypass control)
is archived in
memoriaru/llama-cpp-expert-pool-stale-table-fix.

1. Backend matrix, bit-exact. tests/test-expert-pool.cpp runs the pooled
   MUL_MAT_ID and the regular host-copy path in the same process — Q2_K/Q3_K/Q4_K/
   Q6_K/Q8_0 × two pools sharing one routing (fused gate_up + down shape) × ubatch 1/3
   × cold/evict/hit/rebuild. 40/40 on CUDA and Metal.
2. Server A/B diverged deterministically. Greedy, 834-token prompt, all-experts-on-
   CPU: -mec 0 is run-to-run byte-identical (three runs across days and code
   versions); -mec 32 diverged at byte 17 on a coherent near-tie ("wants" vs "needs")
   and stayed deterministic across environments and code revisions.
3. One invalid verification round, disclosed. A first "fix verification" that
   showed mec0/mec32 agreeing was produced against the wrong server instance — the
   -mec 32 launch had failed (No such file or directory in f32.err) and both
   curls hit the still-running -mec 0 server (visible retroactively in the response
   timings: cache_n = 830 on what should have been a cold prompt). Lesson adopted
   in the test guide: verify the instance (startup log, prompt cache counters) before
   trusting an A/B pair.
4. Root cause. The remap GET_ROWS reads the map table through a RESHAPE view;
   views carry no buffer, so the pooled-split boundary check (which keys on src
   buffers) could not see it, and the remap could land in an earlier split than the
   pool-update prologue — consuming the previous ubatch's table. Every cache miss
   then read whatever expert last occupied the remapped slot (~20–30% wrong weight
   reads per step on a 512-expert model with 32-slot pools). Small enough to stay
   coherent, deterministic enough to reproduce byte-for-byte.
5. Fix. The table is created as [1, n_expert] and consumed directly (no view),
   its buffer is registered with the boundary check so the remap anchors its own
   split, and the pool update runs at that remap, before the same split uploads the
   fresh table. The bypass control — pools allocated but the graph on the host path —
   is byte-identical to -mec 0; with the pool active the A/B now agrees for 353
   bytes and then flips once on a near-tie (byte 354, deterministic), consistent with
   llama.cpp's known sensitivity of kernel selection to buffer placement (the same
   class of near-tie flips observed when changing -ngl/tensor placement). The
   perplexity check quantifies the residual: PPL 3.3096 ± 0.053 with the pool vs
   3.3262 ± 0.053 without (8 × 4096-token chunks, deterministic reruns) — the
   difference is well inside the error bars, i.e. quality-equivalent.
6. Long session. 2561 generated tokens with two 1 K-token generations churning
   the LRU state between four replays of the same anchor prompt: anchor outputs
   byte-identical (same SHA-1 four times), throughput stable (13.9–15.3 t/s).

Maintenance footprint

~700 lines: ggml-backend.cpp (+~350: pool struct, register/update, split hooks),
llama-graph.cpp (+~20 remap), llama-context.cpp (+~70 registration & budget),
argument plumbing, llama-bench flag, and the self-contained matrix test. No changes
to any backend, no new ops, no allocator changes.

Limitations / future work

• Single accelerator (pools go to the first device; per-layer placement is a TODO)
• One ids readback sync per offloaded MoE layer per token — cheap when hits dominate
• Residual near-tie divergence vs the host-copy path: perplexity-equivalent (see
  Validation §5); we propose treating it like the variance already accepted for
  -ngl changes
• No SSD tier, no prefetch, no imatrix-guided pinning (all mentioned in Feature Request: Two-tier GPU+RAM expert cache for MoE offload (pluggable eviction policy) #20757 and
  composable later)

Reproduce

cmake -B build -DGGML_CUDA=ON && cmake --build build -j
./build/bin/test-expert-pool                                  # bit-exact matrix
./build/bin/llama-bench -m <model> -ngl 99 -ncmoe 99 -mec 0,16,32,64 -n 512 -p 0
./build/bin/llama-cli  -m <model> -ngl 99 -cmoe -mec 32 -p <prompt> -n 128 --temp 0

---

AI usage disclosure: this feature was developed with AI assistance (analysis, code and
benchmarks); the author ran, verified and debugged all results on their own hardware
and has reviewed every line — including writing the fix for the bug the validation
uncovered.

[Volunteer-1]

This repo ignores all GPUs except the first one.
Token generation is not much faster than with upstream and slows down as context grows. A cache should produce the opposite behavior.

[memoriaru]

Thanks for testing — and fair point on multi-GPU: the current implementation only pools expert tensors that land on the first accelerator (single-GPU dev rig here), so on a multi-GPU layer split only part of the layers benefit. That's a documented limitation in the RFC, and extending the pool to each layer's own device is planned once the single-device path settles.

On the numbers, could you share the specifics? GPU count/model, the exact command, and whether the startup log shows pooled N offloaded MoE expert weight tensors — if it prints expert cache had no effect, the pool never engaged. Note the per-token overhead this adds (one ids readback per offloaded layer) is context-independent, so a slowdown that grows with context would point at something else entirely (VRAM pressure, or the fallback path engaging). The RFC numbers (+84% neutral / +34% code workload) come from exact commands included in the post — happy to dig into where your run diverges from them.

[Volunteer-1]

You are welcome. My rig contains 2x RTX 3090, generation speed was starting at ca 8.7 t/s for DeepSeek V4 Flash Q8_K_XL quantization, then slowed to ca 8.3t/s after ca 500 tokens. Upstream speed is 7.0 t/s. This means the cache produces <20% generation speed improvement with decreasing speed as context gets larger.

[memoriaru]

Update: prior art I missed when posting this RFC

When I posted this RFC I was unaware of @csantiago78's #27861 ("GPU-resident LRU cache for host-offloaded MoE expert weights", opened Aug 28) — that's a miss on my part and worth correcting explicitly: the idea was already in flight. Having read it in full, the two are independent implementations of the same insight with different design points:

	#27861	this proposal
Execution	dual-chain: cached experts via a second device-side mul_mat_id, uncached via CPU mul_mat_id (observation callback), outputs summed	single-chain: all routed ids remapped into a persistent slot pool; misses copied in before use
Scope	separate gate/up + SILU only; decode (n_tokens == 1) only	fused gate_up and separate paths; multi-token decode when slots allow; prefill falls back by a slot-count gate
Consistency	"outputs coherent"	bit-exact matrix vs the host-copy path, perplexity-equivalent, byte-level A/B
Uploads	async worker thread + deferred mapping publication	synchronous in the split prologue (one ids readback per layer per step)

Their async-upload design and the per-step upload throttle (--moe-expert-cache-inserts) are worth adopting regardless of where this converges. One caution from my side for any remap-based design: the ids→slot mapping table must be anchored in the split that consumes it — ours initially consumed the previous ubatch's table through a view the scheduler's boundary check couldn't see, which is deterministic corruption (evidence chain linked in the post).

The flag name now collides; whether to converge the implementations, keep one as reference, or pick a naming scheme is the maintainers' call — happy to rework either way.

[JigSawPT]

Glad the working-set rule made it into the design. Four things from the same single-5090 / Windows box, all measured with the July tooling and reproducible from it — the first one is for @Volunteer-1's report, the other three are for the flag itself.

0. "Slows down as context grows" is what a VRAM pool does when it eats the KV cache's headroom. @memoriaru is right that the per-token overhead is context-independent, so a slope with context points elsewhere — and on my box the "elsewhere" has a number. Two effects stack: attention cost grows with context in both arms (so the cache is not expected to invert that), and the pool takes VRAM that the KV cache would otherwise grow into, so the run reaches the paging line earlier. That line is absolute, not a fraction: stealing VRAM from a loaded server with an external tensor, 858 MiB free costs −1 % decode; 775 MiB free costs −11 % decode and 2.3× TTFT — 83 MiB separate "unnoticeable" from "broken". On a dense 27B with only -c changed, ~830 MiB free vs ~510 MiB free was a ~3× decode drop on the same 92k-token prompt, with nvidia-smi and the WDDM shared-memory counter both flat; the only symptom is decode at depth. Suggestion for the 2×3090 run: log free VRAM on the first card at the end of the generation, then re-run with -c lowered until ≥ 1 GB stays free at max context. If the slope goes away, it was the rail, not the cache. (On that model there is a second reason the gain is small — point 2.)

1. Size N under a VRAM budget, not a hit-rate target. With the #27861 build on Step-3.7-Flash, --moe-expert-cache 16 left both arms 0.7–0.9 GB from the line above and 32 did not load (compute buffer). "Half of free-at-load" would have accepted the configuration that inverts the result. A rail of "≥ 1 GB free at max context" belongs in the flag, and N should be allowed to differ per layer.

2. Where the pool stops paying is predictable from the routing histogram, not from a sweep. Coder-Next puts ~80 % of hits in ~28 % of experts and gain tracks weighted-traffic coverage almost linearly; gpt-oss and Step-3.7 are flat and doubling coverage bought +2 % and +5–12 %. DeepSeek-V4-Flash is flatter still: per-layer routing entropy 6.3–7.9 bits out of 8.0 (256 experts) on a 139k-token code+prose profile, and 90 % of activation mass needs 5,854 of the 11,008 routed experts (~72 GiB at Q4). A pool that fits in one card's VRAM cannot reach the knee on this model, and a static hot list was worth ~nothing there (+3 %, within noise). That is consistent with the < 20 % @Volunteer-1 sees. A cheap entropy/Gini per layer from a short profile run (the new moe-trace example gives the raw decisions) tells you which layers deserve slots before spending a night sweeping N.

3. Perplexity is not the acceptance test. At bias 0 the replica arms of my fork move the argmax on 2 % of code tokens and ~6 % of Portuguese prose tokens while PPL moves < 1 % — measured with llama-perplexity --kl-divergence-base at -b 1 -ub 1, 4 × 2048 tokens per corpus (the batched default reports KLD 0 because the decode path never runs). The RFC's long-session byte-identity is the right kind of evidence; a batch-1 KLD / top-1 on one prose corpus would close it.

Static split vs LRU pool on the same model, machine and slots (Step-3.7, #27861 build, 61k-token document): at depth the LRU matches the static split (+13 % vs +12 % over stock), and on a 512-token generation it pulls ahead (+20 % vs +13 %, single run) — the dynamic policy does earn its keep. Which is why I'd argue N should end up as an output, not an input: swap only when p_e × bytes_saved − upload_cost > 0, VRAM reserve enforced at load, loop closed on measured inter-token time (the cost-model auto-tune I sketched in #20757). Tooling: https://github.com/JigSawPT/moe-autopilot (the VRAM-cliff probe, the batch-1 drift harness and the static-vs-dynamic harness go there this week).

[memoriaru]

This is exactly the kind of feedback that turns a proposal into a feature — thank you. Point by point:

Rail (0/1): accepted, and fixed. You're right that "half of free-at-load" can accept a configuration that inverts at depth — it budgeted against the wrong baseline. The branch now sizes the pool budget as free − estimated max-context KV − 1 GiB reserve, skips pooling entirely when that leaves nothing, and logs the arithmetic (expert pool budget X GiB (free A − KV B − rail C)). KV estimate covers the per-token growth only (attention layers; recurrent state is bounded). Commit is on the branch — your 2×3090 suggestion (log free VRAM at end of generation, re-run with -c lowered until the rail holds) is now a one-glance check.

Per-layer N (2): agreed, as a follow-up. The DeepSeek-V4-Flash numbers (6.3–7.9/8.0 bits entropy, 5,854 of 11,008 experts for 90% mass) are exactly the "flat routing, knee unreachable" case our docs warn about, and a per-layer entropy pre-flight beats an overnight N-sweep. The pool update path already reads per-layer routing every step, so a cheap per-layer histogram (and per-layer slot counts) doesn't need new instrumentation — it needs a design pass on where the counts live. Logged as the next item after multi-device.

PPL (3): clarified — our measurement did run the decode mechanism. The RFC's perplexity numbers were taken at -b 1 -ub 1 (pool engaged on every ubatch; batched defaults would indeed report KLD 0 trivially for any decode-only cache). But your deeper point stands and I want to concede it cleanly: PPL moving <1% bounds quality, not determinism — argmax flips on near-ties are invisible to it. That's why the RFC's primary evidence is byte-level (server A/B, four byte-identical anchor replays in a 2.5k-token session); a batch-1 KLD / top-1 flip-rate on a prose corpus is the right missing number and I'll post it.

On your offer: the branch exists and is live — memoriaru/llama.cpp, branch moe-expert-pool. A 5090/Windows run of it next to your static-split harness would be the most valuable data point this proposal could get; your moe-autopilot probes are exactly the harnesses I'd hoped someone had. And your endgame (N as an output of a closed-loop cost model, VRAM reserve enforced at load) is where I'd like this to land too — it matches the pluggable-policy direction #20757 originally asked for.

(Branch state at time of writing: commits 9b83cc2 + 9ff189f on memoriaru/llama.cpp moe-expert-pool.)

[JigSawPT]

Thanks — the rail arithmetic in the log line is exactly what the 2×3090 case needed, and the batch-1 KLD/top-1 will close point 3 for me. I'll build moe-expert-pool (9ff189f) on Windows / CUDA 13.0 and run it next to stock, my static split (16 replicas/layer) and #27861 (--moe-expert-cache 16) on Step-3.7-Flash, same 61k-token document and the same harness (tools/static_vs_dynamic.py), and post the JSON here — this box is booked for a fleet evaluation over the next days, so expect it within the week rather than tomorrow. One question so the arms are comparable: should I let your budget arithmetic pick N, or pin N=16 to match the other two arms (or both)?

[memoriaru]

Pin N=16 — one slot count across the three cache arms is what makes the differential readable, and to answer the question directly: our flag doesn't auto-size N anyway. The rail arithmetic only gates whether each host-resident expert tensor gets pooled at all (skipped tensors fall back to the stock path silently), so with N=16 the arm is exactly "16 slots per offloaded expert tensor, rail-enforced".

Two small asks for the run: capture the startup line expert pool budget X GiB (free A − KV B − rail C) — that records the rail state for the arm — and free VRAM on the card at end of generation. The optional "both" variant would be a nice monotonicity point, but don't spend box time on my account; N=16 alone is already the apples-to-apples run.

One bonus if it's cheap in your harness: the batch-1 KLD / top-1 flip rate for stock vs pool (llama-perplexity --kl-divergence-base at -b 1 -ub 1; the pool engages on every ubatch in our build) would close point 3 with third-party numbers — that's the one metric I still owe this thread.

One footnote so the arms stay honest: --moe-expert-cache N is same-name-but-different-internals between #27861's build and ours (their --moe-expert-cache-inserts throttle has no equivalent here, and ours adds the rail) — the arms are comparable on slot count, and the JSON will show whether the strategies themselves differ. Looking forward to it.

[memoriaru]

Follow-up with data from @MichaelDietzel: the break-even from point 0/1 has a closed form — hit rate ≥ 1 − PCIe_BW/RAM_BW — and his platform confirms it exactly (break-even ≈ 91% on PCIe 3.0/136 GB/s DDR4; measured hit-rate ranges at N=16 vs N=160 sit on opposite sides of that line, matching the observed slower/faster). Your routing-histogram pre-flight plus this formula should let anyone predict the outcome before spending GPU hours.

[MichaelDietzel]

Hi, thanks for working on this!

I did some tests but I did not get the speedups I was hoping for. First, the system specs:
CPU + RAM: EPYC 7401P + 8 channel DDR4 @ 136GB/s
GPUs: 1x 2080 Ti 11GiB @ PCIe 3x16, 1x 2080 Ti 22GiB @ PCIe 3x16, 1x 5060 Ti 16GiB @ PCIe 3x8

I am testing build 9ff189f (10721) using Qwen3.6-35B-A3B-UD-Q6_K_XL , all speeds are token generation.

Measurements

1. Baseline CPU-only speed: 14.97 ± 0.06
2. Loading the model partially on the 2080 Ti 22GiB

n_cpu_moe	t/s
40	38.20 ± 1.75
30	44.34 ± 1.44
20	49.57 ± 3.62
13	54.25 ± 3.29

40 means all experts are on the CPU.
13 is pretty much the limit before the GPU goes oom.

3. Using the moe-expert-cache

mec	% of experts	t/s
0	0	39.06 ± 1.20
16	6	13.96 ± 0.21
32	13	19.74 ± 0.25
64	25	30.73 ± 0.59
96	38	42.75 ± 1.44
128	50	49.83 ± 3.29
160	63	60.30 ± 5.32
196	77	46.68 ± 5.39

160 needs slightly less VRAM than n_cpu_moe = 13, 196 is too much, but it still runs.

4. Loading the model fully on both 2080 Tis: 97.78 ± 0.42

Observations:

• using the GPU in the old-fashion style gives a nice acceleration, so everything appears to be working
• In the first post a cache for just 16 of 512 experts gives a speed gain, for me (just 256 experts) it is a huge loss. It is even slower than CPU-only mode.
• mec = 160 appears to give a small boost over n_cpu_moe = 13 at slightly lower memory consumption, so this is a win. However at 63% of experts being cached I was hoping for a high cache hit ratio and thus nearly the speed of full GPU inference.
• mec = 196 is likely to large for the VRAM, so problems were to be expected.
• The results from the 5060 Ti are pretty similar, so the 2080 being old should not be the issue
• The 2080 has half the memory bandwidh of the 4090, so I would not expect that to be the issue either
• I also did a quick test with Qwen3.8-Flash-Next-UD-Q6_K_XL where mec = 64 is the max I can fit. It is slightly slower than having all experts on the CPU. mec = 16 is slower than CPU-only.

Questions:

• Is there maybe anything I am doing wrong? I hoped for higher speeds given what was achieved in the first post.
• Is there an easy way to see the cache hit ratio?
• Where is the KV cache with mec, on the GPU, right?
• What exactly happens at mec = 196? Do the experts get "cached" on the RAM instead? If it would just cap the amount of slots, I would expect it to be at least as fast as 160. Maybe an error message or something would be good.

The commands I used:

1. ./build/bin/llama-bench -m /mnt/LLMs/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf -fa on -r 3 --n-prompt 0 --n-gen 512 -dev none
2. ./build/bin/llama-bench -m /mnt/LLMs/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf -fa on --n-cpu-moe 40,30,20,13 -r 3 --n-prompt 0 -dev cuda1 --moe-expert-cache 0 --n-gen 512
3. ./build/bin/llama-bench -m /mnt/LLMs/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf -fa on --n-cpu-moe 40 -r 3 --n-prompt 0 -dev cuda1 --moe-expert-cache 0,16,32,64,96,128,160,196 --n-gen 512
4. CUDA_VISIBLE_DEVICES=1,2 ./build/bin/llama-bench -m /mnt/LLMs/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf -fa on -r 5 --n-prompt 0 --moe-expert-cache 0 --n-gen 512

[TmDagger]

Made some changes to try and make 2 GPU setup work to squeeze as much performance as possible from my setup here faed504.
Summary of the work on the fix-moe-pool-rail fork branch (rebased on upstream master):

What's added

--moe-expert-cache N (-mec) keeps a persistent per-tensor pool of N experts in VRAM for MoE expert tensors that offloading placed on the CPU host. Upstream of the feature only supported a single accelerator; this makes it work with --split-mode layer across several GPUs:

• Each tensor's pool is allocated on model.dev_layer(il), the device that runs the layer, so every GPU gets its own independent pool set. Layers assigned to the CPU host backend are skipped (a VRAM pool there would only add cross-device copies).
• -mec and --moe-expert-cache-rail-mb accept a comma-separated per-device list (device order); a single value is applied to every GPU device. For example: -mec 48,0 disables the pool on the second device only.
• Per-device budget: free VRAM (measured after KV and weights) - worst-case compute buffer (pp/tg, measured without allocating) - rail. The rail defaults to 512 MiB. This is the fix for cudaMalloc OOM when a large -mec filled the VRAM needed by graph_reserve.
• If the requested slots do not fit, the slot count is scaled down uniformly across all of that device's tensors instead of pooling some layers and dropping the rest to the host-copy path. Over-requesting -mec degrades gracefully; the applied scale is logged.
• Fix for a latent bug: DeepSeek-V4 Flash uses separate ffn_up_exps, ffn_gate_exps, ffn_down_exps (three tensors per layer); the collector overflowed a two-element array.
• Optional per-tensor slot overrides (--moe-expert-cache-down, --moe-expert-cache-gate-up) and windowed hit-rate telemetry appended to the server n_gen/tg lines when GGML_MOE_POOL_STATS=1 is set.

Test setup

• 2 GPUs: RTX 5070 Ti 16 GB on PCIe 5.0 x16 (CUDA0), RTX 4060 Ti 16 GB on PCIe 4.0 x4 (CUDA1). Windows 11. 128 GB DDR4 3200 (2 channels).
• Model: unsloth/DeepSeek-V4-Flash-Vision-Exp, UD-IQ3_XXS (4 shards), loaded without vision mmproj.
• Server flags: -c 262000 --parallel 1 -ngl 99 --split-mode layer -ncmoe 99 --seed 42 (all experts on CPU, dense split by layer across both GPUs).
• Decode is the same single-sequence in the same harness with sizable system prompt (to test influence on prefill as well) - DSH; numbers are single runs on one machine. Hit results are aggregated manually from from printed statistics of llama-server (Not yet hard numbers, more of ballpark for estimates).

Decode results

config	CUDA0 slots	CUDA1 slots	window hit d0/d1	tg t/s	pp t/s
-mec 0 -ncmoe 39 (baseline)	-	-	-	5.36	43.7
-mec 48	48	48	85% / 93%	9.09	44.2
-mec 48,56	48	56	85% / 95%	9.71	44.3
-mec 48,0	48	0	84% / -	6.10	45.1
-mec 48,128 (auto-scaled)	48	~55 (44%)	84% / 95%	9.90	45.7
-mec 128,128 (auto-scaled, final)	~47 (37%)	~58 (46%)	83% / 96%	10.32	49.5

Notes:

• Autoscaled result being better in the end shows that manually selected via GPU memory use 48 slots were too agressive and caused minor spill into shared memory, that gave noticable negative impact right away (~5%).
• Baseline uses -ncmoe 39 (four extra dense layers on GPU) because -mec 0 with -ncmoe 99 was slower, and classic layer-wise use of free GPU memory is the competing strategy that faces a lot of problems on consumer motherboards.
• -mec 48,0 falls back to roughly baseline decode, which confirms that the second GPU's pool does real work.
• The first post-rebase -mec 48 run measured 8.97 t/s; later runs with the uniform-scaling and compute-buffer fixes were 9.09-10.32 t/s.
• Prefill is almost unchanged by design: a pp ubatch selects more distinct experts than the pool has slots, so it takes the stock host-copy path and never evicts the decode set.

Auto-scaling and the compute buffer

• Before the budget fix, -mec 128 failed during graph_reserve with cudaMalloc failed: out of memory on a 1976.50 MiB CUDA0 allocation: the pools had consumed the VRAM that the prompt-processing compute buffer needed.
• Measuring the worst-case pp/tg compute buffers without allocating and subtracting them per device (plus a 512 MiB rail) removed the OOM.
• After fix -mec 128 auto-scaled to ~37% (about 47 slots) on CUDA0 and ~46% (about 58) on CUDA1 and ran at 10.32 t/s, the best result so far, with no real spillover to shared memory.

Reproduce (2 GPUs, layer split). For GPU with larger memory -mec 128 might not take up all available free space - increase as needed, or use larger -ub / -c.

llama-server -m <model> -c 262000 -ngl 99 --split-mode layer -ncmoe 99 -mec 128 --moe-expert-cache-rail-mb 512

Watch the startup lines for expert pool budget ... on <dev> and scaled to X%, and run with GGML_MOE_POOL_STATS=1 to get the windowed moe expert pool: hit ... d0=..% d1=..% line next to n_gen/tg.

For example:

0.01.661.443 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.01.844.062 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance
0.51.199.866 W init_expert_pools: CUDA0: requested slots do not fit the remaining budget, scaled to 37% (~47 slots per tensor)
0.51.483.747 W init_expert_pools: CUDA1: requested slots do not fit the remaining budget, scaled to 44% (~55 slots per tensor)

Every generation with fixed seed = 42 run resulted in the same 1508 tokens.

Example of hit-rate stats for recent tests on Qwen 3.8 Flash Next Q4_K_M

6.08.566.948 I slot print_timing: id  0 | task 2 | n_gen =    701, tg =  17.77 t/s, tg_3s =  18.31 t/s
6.08.566.985 I slot print_timing: id  0 | task 2 | moe expert pool: hit 93.1% (147507 hits / 10893 misses) in window d0=90% d1=97%
6.11.636.293 I slot print_timing: id  0 | task 2 | n_gen =    747, tg =  17.57 t/s, tg_3s =  14.99 t/s
6.11.636.303 I slot print_timing: id  0 | task 2 | moe expert pool: hit 89.9% (119052 hits / 13428 misses) in window d0=86% d1=94%
6.14.700.862 I slot print_timing: id  0 | task 2 | n_gen =    809, tg =  17.75 t/s, tg_3s =  20.23 t/s
6.14.700.872 I slot print_timing: id  0 | task 2 | moe expert pool: hit 94.5% (168783 hits / 9777 misses) in window d0=92% d1=97%
6.17.721.739 I slot print_timing: id  0 | task 2 | n_gen =    869, tg =  17.88 t/s, tg_3s =  19.86 t/s
6.17.721.750 I slot print_timing: id  0 | task 2 | moe expert pool: hit 93.9% (162282 hits / 10518 misses) in window d0=91% d1=97%

[MichaelDietzel]

Thanks @TmDagger, that sounds really interesting!

There is one thing that I do not fully understand:

--moe-expert-cache N (-mec) keeps a persistent per-tensor pool of N experts in VRAM for MoE expert tensors that offloading placed on the CPU host. Upstream of the feature only supported a single accelerator; this makes it work with --split-mode layer across several GPUs:

* Each tensor's pool is allocated on `model.dev_layer(il)`, the device that runs the layer, so every GPU gets its own independent pool set. Layers assigned to the CPU host backend are skipped (a VRAM pool there would only add cross-device copies).

So can I somehow assign a layer (with experts) fully to the CPU so that the expert cache is disabled using your changes? How would I do that?

[memoriaru]

Catching up on this thread with one comment, four parts.

@JigSawPT Root cause found — your -v log was decisive. blk.3's down tensor is the third expert tensor of that layer, and the collector ahead of pool registration was a two-element stack array sized for the fused gate_up+down layout; three-tensor architectures (Step-3.7, DeepSeek-V4) wrote past it and corrupted the stack before any graph ran. Our own test models happen to be fused-layout, which is why the matrix suite passed everywhere without catching it. @TmDagger independently hit and fixed the same bug on DeepSeek-V4-Flash — two models, two independent discoveries. Fix is in 1582fcc (the collector is now an unbounded vector — layouts vary too much across arches for a fixed array to be the right shape), pull latest and rerun; no stack trace needed. The CUDA0 model buffer size = 0.00 MiB line does look off and is worth checking the log accounting, but it's unrelated to the crash. On the matrix: your 9ff189f checkout actually covered the five K-quant types (not just q8_0 — likely a typo), all at nt=1/3, which is the mmvq path — so the pass was expected. The latest tip's matrix now covers MXFP4 and nt=8 in one go (nt=8 forces the MMQ mul_mat_id path, on Blackwell the native FP4 MMA kernels — the combination your stock arms never executed). On the KLD tail — "reduction order vs something else" now has a direct answer. I added a full-GPU-experts control arm (-ncmoe 0, no pool involved). One practical note for anyone replicating: on a 24 GB card this config sits at ~23.7/24.5 GB with -c 2048 and stalls early into the first pass; -c 1024 clears it, and the numbers below are from that setting. Pooled vs full-GPU comes out at KLD 0.000000, Same-top 100.000% (differences below the uint16 storage resolution) — pooled execution is equivalent to plain GPU stock execution — while full-GPU vs CPU (also no pool) itself produces Mean KLD 0.0356 and a 4.1% top-1 flip rate. So the flips are 100% attributable to the CPU→GPU placement change, not to the pool logic. Your Q4_K_M-vs-bf16 reference (mean ~0.014) is a weight-error yardstick; an execution-backend change is a different class, at 0.03–0.04 mean with the same heavy tail (p99 0.559, max 1.27). I'll keep the "PPL-equivalent" wording in the RFC but attach this decomposition. Happy to see it reproduced on the 5090.

@TmDagger Welcome — this is exactly the kind of independent validation the proposal needed, and DeepSeek-V4-Flash (the "flattest router, shouldn't benefit" model) going 5.36 → 10.32 t/s with multi-GPU pools is a heavy data point. The collector overflow you hit on DeepSeek is the same bug we just root-caused on Step-3.7 (1582fcc) — the two discoveries corroborate each other. Three things I'd like to fold into the main line: per-layer device pools (model.dev_layer(il) — the exact next step past our documented single-accelerator boundary), the measured compute-buffer budget (our KV estimate guards against deep-context inversion, yours guards against the graph_reserve OOM — complementary), and uniform slot scaling (graceful degradation beats whole-layer skipping on over-request). One question, because it matters for the pre-flight rule's credibility: JigSawPT measured DeepSeek-V4-Flash routing entropy at 6.3–7.9/8.0 bits with 90% of activation mass on 5,854/11,008 experts — by that yardstick this model shouldn't produce high hit rates, yet you see 83–96%. Variant difference (Vision-Exp?), routing concentration over a 262k-token session, or hit-rate accounting (you note it's ballpark)? On convergence: I'd suggest v1 (single GPU, verified) goes through review first, with your multi-GPU + budget work as its multi-device follow-up — so maintainers don't see two parallel implementations. How do you want to proceed?

@Volunteer-1 Your correction is right: KV is preallocated at startup from -c, fixed size — "the pool eats KV headroom" was the wrong mechanism. The accurate one is total VRAM pressure: pool occupancy makes other allocations reach the paging line earlier, not KV growth. As for 8.7 → 8.3 t/s (-4.6%): within WDDM run-to-run noise (we've measured ±20–40% across days on Windows boxes); if the slope reproduces consistently, the VRAM-cliff probe (log free VRAM at end of generation, re-run with lower -c) will separate it cleanly. And your <20% on 2×3090 is consistent with the pool living on the first card only — per-layer device pools are exactly what @TmDagger just demoed, worth re-testing once that lands.

@MichaelDietzel Good question, and it pins your earlier per-layer proposal to a concrete gap. What exists today: -ngl N is a contiguous prefix cut (layers after N land fully on CPU — cache-free, but you can't pick middle layers), and -ot can override any layer's expert tensors to CPU or a specific GPU (e.g. -ot "blk\.(1[0-9]|2[0-9])\.ffn_.*exps.=CPU"), but layer assignment stays with -ngl, the pool follows the layer, and still gets created — so an explicit "experts on CPU, cache off" per-layer switch does not exist yet. Your scenario (some layers mode-2, some mode-3 with concentrated slots) is exactly the missing piece. Implementation cost is low — pool registration already walks layers and consumes the budget per layer, so it's one layer-selector parameter away — and I'm happy to add it to v1 if maintainers are okay with the config surface. Paired with @JigSawPT's routing-entropy pre-flight for choosing which layers to cache, that becomes the data-driven version of your worked example.

[JigSawPT]

Good to see it root-caused, and the three-tensor layout explains why our stock arms never hit it.

Two corrections accepted. The 9ff189f matrix did cover the five K-quant types at nt=1/3 — my "q8_0 only" was a misread of the build log, not a property of the suite. And your full-GPU control arm answers my tail question directly: pooled vs full-GPU at KLD 0.000000 / same-top 100% puts the flips entirely on the CPU→GPU placement change, which is a different class of difference from the weight-error yardstick I was comparing against. That decomposition is worth keeping in the RFC exactly as you wrote it.

On the entropy question you raised with @TmDagger, since it cites our number: the 6.3–7.9 bits and 5,854 of 11,008 experts for 90% of activation mass were measured on the 0731 non-vision variant, per layer, over mixed text, with 90% mass as the cutoff. A per-layer maximum hit rate over a single 262k-token session is a different statistic from activation mass over a corpus — a long session concentrates on a working set, and a max over layers is not a mean. Worth comparing like for like before treating 83–96% as a contradiction of the flat-router reading; the variant difference (Vision-Exp) could matter too.

I'll rerun the Flash-Next arm at N=16 with GGML_MOE_POOL_STATS=1 on the fixed tip when the box is free, and report the per-layer hit rates next to the rail line.

[TmDagger]

On the entropy question you raised with @TmDagger, since it cites our number: the 6.3–7.9 bits and 5,854 of 11,008 experts for 90% of activation mass were measured on the 0731 non-vision variant, per layer, over mixed text, with 90% mass as the cutoff. A per-layer maximum hit rate over a single 262k-token session is a different statistic from activation mass over a corpus — a long session concentrates on a working set, and a max over layers is not a mean. Worth comparing like for like before treating 83–96% as a contradiction of the flat-router reading; the variant difference (Vision-Exp) could matter too.

I'll rerun the Flash-Next arm at N=16 with GGML_MOE_POOL_STATS=1 on the fixed tip when the box is free, and report the per-layer hit rates next to the rail line.

The figures are probably inflated.
In ggml_backend_sched_compute_splits, the nodes are traversed for each split, and both conditions trigger for the same pool:
the GET_ROWS branch (table remap) -> ggml_backend_sched_update_expert_pool(...),
the MUL_MAT_ID branch (the pooled node itself) -> ggml_backend_sched_update_expert_pool(...) again.
The expert_ids_tensor caches the ID reads, but the used status is recalculated and the hit/miss counters are incremented a second time (the second pass sees the cached data and records everything as hits). The result: two updates per pool per token; misses are undercounted, and the hit rate is inflated (your 99% figure might be an artifact).

I'll try a few fixes, will push and post results when figures start matching PCI-e usage.
For now it's still usable as a ball-park to be compared against itself for different layer and expert cache distribution setups.

I'm currently working on a few manual and semi-auto toggles to feel out optimal strategy in respect to VRAM BW, PCI<->RAM BW, expert slot and dense layer distribution that would benefit from telemetry fix as well.

[memoriaru]

Update: telemetry fix confirmed, a silent-pitfall fix, and two test-matrix corrections

Thanks @TmDagger — the double-counting report was correct, and digging into it uncovered something much bigger downstream. What landed since the last update:

1. Telemetry double-count — confirmed and fixed (f75b069)

The remap GET_ROWS and the pooled MUL_MAT_ID can both anchor the same pool within one compute_splits() call, so each pool was updated twice per ubatch. The second pass found every used expert already holding a slot from the first and counted the whole ubatch as hits — recording (1 + true) / 2. Each pool now updates at most once per graph execution, and ggml_backend_sched_get_expert_pool_stats() was added so tests (and tools) can read the counters without the log-parsing hack.

2. Why the old test matrix never saw any of this

Two blind spots, both fixed in the harness: the ids read-back consumed a scheduler copy that hadn't been uploaded yet for this ubatch, so the update always read stale routing; and all experts shared identical weights, so any mapping error was invisible to the bit-exact comparison. Weights are now distinct per expert.

3. The real find: sched_reserve() on a graph you later compute silently breaks the pool update path

Symptom: illegal-memory-access on CUDA with MXFP4, and — much worse — silent all-zero routing on Metal that still passed the tests. Mechanism, in three steps:

1. sched_reserve(gf) splits the graph and permanently rewrites the node srcs to the scheduler's copy tensors;
2. its trailing sched_reset() clears the copy registration hashes, but the rewritten srcs stay;
3. the later graph_compute(gf) re-splits against the rewritten graph: the stale copies now look like already-resident local tensors, so every split input upload silently disappears, the remap GET_ROWS drifts out of its own split, and the pool update reads garbage routing (all-zero on Metal's fresh buffers, junk like 0x7F7FFFFF in CUDA's dirtier pools — which is what made it look MXFP4/layout-specific).

A control experiment (skip the reserve, keep everything else) flips all three symptoms off at once: correct ids read-back, table upload restored, anchor hits everywhere. Upstream's own call sequences avoid this (reserve uses a dedicated measure graph, decode goes through alloc+compute), but nothing warns you off reserve(g) + compute(g).

Fixes:

• 08f655d — the test resets before compute and lets alloc_graph reserve on demand (the decode loop's rhythm);
• 24be2da — defense: a remap GET_ROWS reaching compute without its table copy scheduled now aborts loudly with the table/split/backend named, and the update rejects routing tensors without a backing buffer. The failure mode that used to take days to root-cause now trips on the first round;
• d558526 — structural: the pool now owns its device copy of the map table. The update writes the host table and enqueues the H2D itself on the split stream, the anchor matches the pool-owned pointer directly, and nothing the scheduler resets or rewrites can touch that chain anymore.

Worth flagging for upstream: reserve() documenting (or asserting) that the graph is not reusable for compute afterwards would have saved us this whole hunt — happy to open an issue with a minimal reproducer.

4. A dispatch-matrix surprise

Our nt=8 case was designed to force the MMQ ids path on the grounds that "MXFP4's mmvq-mmid max batch is 7" — but that limit is from the Turing bucket. get_mmvq_mmid_max_batch() returns MMVQ_MAX_BATCH_SIZE unconditionally on Ada and Blackwell for every quant type, so on a 4090 MXFP4 never leaves MMVQ at nt=8 and the MMQ/native-FP4 path had zero coverage. The matrix now runs nt ∈ {1, 3, 8, 9} — nt=8 stays as the full-batch MMVQ case, nt=9 is what actually reaches MMQ (2c6b3f0).

5. CUDA results

The full 96-case matrix now also passes on CUDA (RTX 4090, container CUDA 13.1): 6 quant types × nt ∈ {1, 3, 8, 9} × cold/evict/hit/graph-rebuild rounds, including MXFP4 and the first real coverage of the MMQ ids path at nt=9. GGML_MOE_POOL_STATS reporting works as before, and the build is warning-clean. Note that with the update path fixed, "passes" now means what it says — under the old sequencing the CUDA matrix also passed while the pool update was reading garbage routing, it just happened to be harmless there; the new aborts make that class of false green impossible.

6. For the field testers

Since several of you have been running this on real workloads, a few notes for whenever you next update your builds — nothing urgent, just things worth a look:

• When you pull, if anything in your setup isn't compatible it will now abort with a named reason (table, split, backend) instead of quietly slowing down — so a clean run doubles as a free sanity check, and if you do hit the abort, the message would make a great bug report.
• The pool update now synchronizes the compute backend once per ubatch (it owns a single device table copy now). Expected impact is small since the read-back already synced anyway, but @MichaelDietzel your RX 9070 / PCIe 3.0 numbers are the most sensitive to exactly this kind of serialization — a re-run whenever convenient would be gold, and it would tell us whether the promote-gate idea for v2 needs to account for it.
• @JigSawPT with nt=9 the MMQ path — including the Blackwell native FP4 kernels — finally gets real coverage; once the CUDA run is in, a re-test of Step-3.7-Flash on your 5090 would tell us whether the crash you reported is this issue or something else.
• @Volunteer-1 your 2×3090 report is what shaped the rail guardrails; pooling still only follows the first accelerator, so your single-GPU-arm numbers remain the relevant baseline.
• And generally: if you see a failure shape the abort messages don't cover, please paste it — the more failure samples we collect, the better the defenses and the v2 design (promote gate included) get.

(Clarifying the @JigSawPT bullet above: our CUDA matrix run is already in — what it can't answer is whether the Step-3.7-Flash crash is this same issue, since that path needed a real-model re-test on Blackwell.)

[MichaelDietzel]

To settle it, the only measurement that works is an interleaved one: same session, same clock state, alternating builds A/B/A/B on the -mec 0 arm. If you can spare that (it's the cheapest possible run — cache disabled, so no pool bookkeeping), it will either surface a real signal or close it as thermal/clock-state drift, which across days on a desktop box is not rare.

@memoriaru These are already A/B/A/B measurements that I all did yesterday. I can likely re-run them tomorrow to make sure this is a real measurement.

I tried to reproduce it again today, but I could not. So it is probably fine.

[memoriaru]

2. Mainline path — confirmed. The PR split we've been holding (① matrix tests → ② scheduler core → ③ arg/bench wiring) has per-layer configuration landing exactly in ③, and the rail-mb per-device variant belongs there too. What would help the review: the copy/update telemetry columns first (your fork's per-tensor hit/miss/copy-volume/update-time table — mainline's GGML_MOE_POOL_STATS already emits per-tensor hit/miss, but has no copy-volume or update-time columns, and without those the layer split is invisible: the hit-rate variance hides inside an average that still looks acceptable), then the --mec-per-layer/--mec-whole knobs on top of it. Both are additive and don't touch the scheduler core, so they shouldn't be blocked on the ② discussion. A PR against this branch (or stacked on ① once it's split out) works for me — your call on sequencing.

@memoriaru Assempled stack of PRs that should contain requested changes: 555d1ec memoriaru/moe-expert-pool

* |->A  `moe-telemetry-copy-update`        PR: "moe: per-tensor copy-volume and update-time telemetry"
  
  * |->B  `moe-per-layer-whole`                PR: "moe: --mec-whole, --mec-per-layer and slot recommendations"
    As I understand it, B should work as a base for: per-device pool branch from held split (②/③)

Updated doc with clarifications is in moe-per-layer-whole

@TmDagger the stack structure is exactly right — telemetry first (A), knobs on top (B), and B as the base for the per-device pool work closing out the old multi-GPU thread is a clean way to fold that in. We'll review A and B on your fork as they stand; when the maintainer weighs in on the mainline split, the same commits re-target without rework. Saw the doc clarifications landed in B as well — thanks for handling the heuristic section so fast.

[memoriaru]

To settle it, the only measurement that works is an interleaved one: same session, same clock state, alternating builds A/B/A/B on the -mec 0 arm. If you can spare that (it's the cheapest possible run — cache disabled, so no pool bookkeeping), it will either surface a real signal or close it as thermal/clock-state drift, which across days on a desktop box is not rare.

@memoriaru These are already A/B/A/B measurements that I all did yesterday. I can likely re-run them tomorrow to make sure this is a real measurement.

I tried to reproduce it again today, but I could not. So it is probably fine.

@MichaelDietzel glad the re-run came up clean — and thank you for the discipline of posting the negative result; that's what lets us close this instead of carrying a phantom regression around. For the record, we also ran the decisive A/B on our side in the meantime (4090, same 90GB Flash-Next GGUF, llama-bench -mec 0 -p 1 -n 512 -r 3, builds recompiled per commit): b0096df × 3 interleaved runs = 6.25 / 6.67 / 5.47 t/s — a 20% spread between identical builds — and 555d1ec = 5.77, squarely inside that spread. Combined with the code-level argument (nothing in the b0096df..555d1ec diff executes on the -mec 0 path) and identical Metal correctness suites on all three commits, your earlier readings were the noise floor showing through — the same-build variance on this workload is just larger than the effect sizes we were chasing. Closed, and nothing further needed on your side.

[memoriaru]

@MichaelDietzel one last note on this arm to close the book properly: your 09-17 correction — that the three-build comparison was already interleaved A/B/A/B — landed after our previous reply went out, so we published a conditional that you had, in fact, already ruled out. Our 4090 interleaved run (same-build spread 20% across three runs of b0096df; 555d1ec inside the spread) matches your "probably fine", and the code-footprint argument stands. Genuinely closed — thanks for both the original report and the clean re-run.

[memoriaru]

@TmDagger review done on both branches (rebuilt and ran the full suite on a 4090: all cases pass, including the new whole/per-layer ones). Overall: approve — the whole-path design is clean (dedicated slots, no remap, no prefill-fallback interaction, steady-state decode touches neither the table nor the copy path), the budget priority (whole → explicit per-layer → dynamic scaled by binary search) is the right shape, and the recommendation engine matches the doc.

Three comments, none blocking:

1. Server telemetry statics — the static std::vector record buffer and the moe_telem_state assume the single-threaded dispatch loop. True today, but worth a comment in the code so a future multi-threaded dispatch doesn't inherit a data race.
2. Whole pools allocate a dead table — register_expert_pool unconditionally creates the host+device map table, but a fully_resident pool never uses it (no GET_ROWS, no table writes). ~4·n_expert bytes per pool of dead state; could be skipped when n_slots == n_expert.
3. Docs/branch mismatch: the post lists -mec N0,N1,... (per-device values) and --moe-expert-cache-rail-mb (per-device reserve, default 768 MiB), but neither is in moe-per-layer-whole — -mec is still a single int and the rail is the hardcoded 1 GiB. I assume those live in the multi-GPU layer-split work — worth stating explicitly which flags land in which branch/stage, so nobody greps for a flag that doesn't exist yet.

Happy to take these as PR review comments when you open them against memoriaru/moe-expert-pool — the stack structure you proposed (A then B, B as base for the per-device work) works as-is.

[JigSawPT]

Two things from the single-5090 Windows box, both on b0096df built with CUDA 13.0 (-t 16, WDDM, 32 GB).

1. The Step-3.7-Flash crash no longer reproduces. Same configuration that segfaulted for me on 11/09 --
Step-3.7-Flash MXFP4_MOE, -ncmoe 42, --moe-expert-cache 16, 131k context, KV q8_0, dying ~45 s into the load --
now loads in 64 s and serves. I have not bisected, so I cannot tell you it was 1582fcc specifically rather than
something else between 9ff189f and b0096df; what I can tell you is that it is gone.

2. Two pool symptoms on the same build, depending only on -ncmoe.

At -ncmoe 34 the pool initialises and reports, and the second line is the interesting one:

W init_expert_pools: expert cache disabled: free VRAM minus max-context KV leaves less than the 1.00 GiB rail
W init_expert_pools: expert cache had no effect: no offloaded MoE expert weight tensors found

So even when it runs, it finds nothing to manage -- with 34 layers' MoE weights on the CPU and the flag accepted.

At -ncmoe 42, same build, same flag, init_expert_pools logs nothing at all: no budget line, no rail warning,
no "no offloaded" message, no named abort. Since the budget line is unconditional once the function is past its early
returns, that looks like the silent exit at cparams.expert_cache_slots <= 0 -- and it is the failure shape your new
aborts do not cover. Two models behave identically:

model	ncmoe	pool output	served
Qwen3.8-Flash-Next UD-Q4_K_XL, 40k	34	the two lines above	29k-token prompt in 480 s
Qwen3.8-Flash-Next UD-Q4_K_XL, 40k	42	none	29k-token prompt in 110 s
Step-3.7-Flash MXFP4_MOE, 131k	42	none	loads in 64 s, generates

Ruled out from this end: --load-mode none as well as mmap (identical silence at 42); short requests as well as real
load -- 29k-token prompts are far past the 512-lookup threshold that gates the stats line; and the flag really is
reaching the process, verified on the live command line rather than the script that builds it.

llama-server -m Step-3.7-Flash-MXFP4_MOE-00001-of-00004.gguf -ngl 99 -np 1 --flash-attn on -t 16 -c 131072   -ctk q8_0 -ctv q8_0 -b 4096 -ub 2048 --jinja --fit off --cache-ram 0 --no-kv-unified   -ncmoe 42 --load-mode none --reasoning-format deepseek --moe-expert-cache 16

(GGML_MOE_POOL_STATS=1 in the environment.)

Leaving this one with you -- I am deep in another piece of work and will not be chasing it further from this end.
Happy to answer anything about the box or the configuration if it helps you reproduce it.

None of that changes the v2 write-up: I am still in, with the PCIe 5.0 data point, the routing histograms and the
promote-gate review against JigSaw v3, whenever you want to pick it up.

[memoriaru]

@JigSawPT thanks — both parts received, and the first one is the big one: the Step-3.7-Flash crash holding off on the exact config that segfaulted on the 11th is the closure we wanted for that report. No bisection needed on your end.

On the two -ncmoe symptoms: this one is ours to chase now — don't spend any more box time on it. We have the same 90GB Flash-Next GGUF and a 4090 locally, so reproduction is self-contained; your live-cmdline verification and the 34-vs-42 contrast already saved us the first hour.

We've since run both values through a verbose llama-perplexity pass on the 4090 (full load each), and the interesting part is what happens between the two init_expert_pools passes (one at sched creation, one after the KV/compute buffers are allocated):

• -ncmoe 34: first pass budgets 20.82 GiB and pools all 102 tensors (34 layers × 3); after the buffers land, free drops to 0.91 GiB, the second pass budgets 0.00 GiB, hits the rail, and every layer is skipped with an info-level line (exceeds the remaining budget - layer falls back to the stock host-copy path) that a server log level does not show — which is exactly the two warnings you saw. So at 40k context that arm is working as designed: the rail is refusing a configuration that would be 0.91 GiB from the paging line at depth. Two of your three models are 40k/131k context, which is why you never saw the budget line — on our side at ctx=512 the same values pool happily (102/126, zero warnings).

• One real takeaway though: when the rail rejects everything, the closing warning says "no offloaded MoE expert weight tensors found", which is misleading — it found 102, it just can't afford any. Will reword that to "0 of N offloaded tensors fit the VRAM budget" so the next person doesn't chase a ghost.

• -ncmoe 42 (total silence): does not reproduce on the 4090 in either harness — llama-perplexity pools all 126 tensors on both passes, and llama-server at -c 40960 -ncmoe 42 -mec 16 pools all 126 too (budget 17.26 GiB, zero warnings). Since the budget path behaves identically for every -ncmoe value, the silence can't originate there; the only silent return in init_expert_pools is expert_cache_slots <= 0, and nothing in the parameter chain writes anything but your -mec value. That leaves the parameter handoff into your server process or something 5090/WDDM-specific. Could I ask for one thing after all — a verbose (-v) server startup log of the silent -ncmoe 42 arm? Verbose dumps the parsed params (including expert_cache_slots) plus every init_expert_pools line, so one log answers both "did -mec 16 parse" and "where did it exit" — and that closes it.

Nothing else is owed from your side. On v2: all three are logged and wanted — the PCIe 5.0 data point, the routing histograms, and the promote-gate review against JigSaw v3. I'll come back to you with a start order once this bug round is closed; histograms first is my current lean, since they make the promote-gate review concrete.

[nigelhutch]

I've been investigating a related problem on a single Ryzen AI MAX+ 395 / 128 GB Strix Halo system, but at a larger model-to-memory ratio: host-managed routed-expert residency where the complete model cannot be resident.

I have validated single-node execution of Qwen3-235B-A22B Q4_K_M and subsequently Kimi K2.5 UD_Q2_K_XL (~375 GB) using an NVMe-backed expert path. One thing your results reinforce strongly is that residency hit rate alone isn't enough — transfer/synchronisation overhead can dominate the useful expert compute.

My implementation takes a CPU-authoritative approach and streams only routed expert records, with explicit residency/lease tracking. The Kimi experiment is slow rather than a throughput competitor, but it may be an interesting data point for the discussion because the model is roughly 3× the machine's physical memory rather than simply overflowing VRAM.

I've published the methodology/results here if useful for comparison: https://zenodo.org/records/22730031

I'd be particularly interested in comparing routed-expert working-set distributions and cold/warm residency behaviour.

[memoriaru]

@nigelhutch welcome — and thank you for the careful write-up. Running a ~375 GB MoE (3× physical memory) to an exact-match reference on one 128 GB machine is a meaningful feasibility data point even at 0.43 tok/s, and your closing observation matches something we just learned the hard way on the PCIe tier: residency hit rate alone is not the score — transfer/synchronization overhead can eat the entire gain.

That principle has a closed form on our tier (hit rate ≥ 1 − PCIe_BW/RAM_BW), and it transfers to yours almost unchanged: with Strix Halo's ~256 GB/s of memory bandwidth and a ~7 GB/s NVMe drive, the break-even line sits near h ≈ 97%* — which is why a NVMe-backed residency is a feasibility demonstration first and a throughput design only after the working set fits the RAM tier. (Substitute your drive's real sustained bandwidth for the 7 GB/s.)

If you are open to comparing working-set data, three specific things from your lease-tracking stack would slot straight into the v2 design:

1. Distinct routed experts per token window (e.g. windows of 512 / 4k / 64k tokens — the working-set curve). This decides what any residency tier can realistically hold.
2. Hit rate vs residency budget under your lease policy (curve, not points) — the shape tells us whether eviction policy (LRU vs lease) matters or whether the curve is dominated by the routing distribution itself.
3. Cold-to-steady-state trajectory — tokens (or wall time) from a cold start until hit rate stabilizes, and ideally a re-warm after a topic/task switch. This decides how much of a real session a residency tier spends in its useful state.

Per-layer breakdowns are a bonus (routing concentration varies by layer), not required. And one clarification that helps framing: is the QES stack llama.cpp-based, or an independent runtime? It changes which of our knobs (slot counts, budgets, per-layer modes) are transferable to your numbers.

[nigelhutch]

Sorry I missed this response — and thanks, this is extremely useful.

Your point that residency hit rate alone is not the score lines up very closely with what I measured. On the current frozen 128-token Kimi K2.5 workload I have 60 routed blocks × top-8, so 480 expert requests/token and 61,440 generated expert requests.

The bounded cache I tested is deliberately small — about 1.9 GiB, retaining the previous token's top two experts per routed layer. Generated-only accounting gave 4,734 hits / 61,440 requests = 7.705%, avoiding 70.285 GiB of expert-store traffic over the 128 generated tokens.

There is quite a bit more routing locality available than that final hit rate suggests. Across consecutive generated tokens, the routed sets shared an average of 2.355 of 8 experts per layer, or about 29.44% immediate one-token reuse opportunity. So there is a fairly large gap between locality present in the routing trace and locality successfully converted into cache hits by this small retention policy. That feels directly relevant to the LRU-vs-lease question you raised.

The other interesting part is what happened after the reads were avoided. Mean expert-delivery wait dropped by about 5.1%, but decode only moved from 0.4321 tok/s to ~0.4382 tok/s. At the same time, mean expert-executor time increased from about 367 ms/token to 402–404 ms/token. So most of the storage-path saving was absorbed elsewhere in the execution path — very much in line with your point about transfer/synchronisation overhead eating the gain.

On the three measurements you suggested, I don't have enough data yet to claim them. The published workload is only 128 generated tokens, so I don't yet have:

the distinct-expert working-set curve at 512 / 4k / 64k tokens;
a hit-rate-vs-residency-budget sweep; or
a proper cold → steady-state → task-switch/re-warm trajectory.

Those are very good next measurements though, particularly the working-set curve and the budget sweep. They should show whether increasing residency on Strix Halo moves toward a genuinely useful operating region or simply adds management overhead.

One runtime clarification as well: QES uses a pinned vendored llama.cpp core as the model/execution host. It isn't stock llama.cpp, but it isn't an unrelated standalone inference engine either. QES adds the expert-store, bounded residency/cache and associated orchestration around that host path. So I think some of the behaviour should be directly comparable, although I wouldn't assume all of your cache knobs transfer one-for-one.

Thanks again for taking the time to look at it. Your three suggested measurements give me a much better way to characterise the working set than a single cache-hit figure.

[memoriaru]

@nigelhutch no worries about the timing — and thank you for a genuinely thorough follow-up; a long, patient reply with raw numbers beats a short polite one with none.

The 29.4% vs 7.7% gap is the most valuable number in the set: consecutive generated tokens share nearly a third of their routed experts per layer, but a deliberately minimal retention policy (previous token's top-2 per layer) only converts 7.7% into hits. That gap is exactly the space a smarter retention policy (cost-gated promotion, the lease-style idea you're tracking) is supposed to harvest — it quantifies the headroom between "locality exists" and "locality is converted", which is the number v2's promote-gate design needs.

The absorption effect is equally instructive: avoiding 70 GiB of store traffic and cutting delivery wait 5.1% moved decode only +1.4%, with executor time rising. On a unified-memory part like Strix Halo, the bandwidth the NVMe reads give back can be partially re-consumed by cache maintenance and executor work sharing the same LPDDR5X fabric — the win percolates into a bottleneck that isn't the one you removed. I wouldn't over-attribute the +35 ms without more instrumentation, but the shape matches the "per-section optimizations don't compose into end-to-end gains" rule.

Agreed on the measurement plan, too — 128 tokens is honest about its limits, and a longer-session run (4k/64k windows) producing the working-set curve and a proper cold → steady → task-switch trajectory would be the natural next step. That data would also calibrate our break-even form for the NVMe tier with real numbers instead of an assumed 7 GB/s.

And the runtime clarification settles the framing question: a pinned vendored llama.cpp core means the behavior is directly comparable in the places that matter, even if the knobs don't map one-to-one. Looking forward to the budget-sweep numbers whenever they land.