feat(kimi-k3): Kimi K3 support: MXFP4 to BF16 conversion and inference graph - #58

…ues for Adreno 850 GPU (ggml-org#25745)

* opencl: workaround for A850 compiler compat

* opencl: fix DX compiler version parsing and cleanup

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>

Assisted-by: pi:llama.cpp/Qwen3.6-27B

* CUDA: dedup MoE gate/up activation quantization (fp4)

For MoE gate/up projections the src1 activation is broadcast across the
routed experts (ne11 == 1), so ids_src1 maps every one of a token's
n_expert_used slots to the same physical row. The MMQ path therefore
re-quantized each token's activation n_expert_used times.

For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of
once per expert. For NVFP4 a single quantize+scatter kernel
(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the
resulting block_fp4_mmq straight to all n_expert_used slots, using an
inverse token->compact-row map (build_tok2c). MXFP4, and
GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique
rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).
Both are bit-identical to the previous gather-then-quantize path (identical
source data, deterministic per-block quantization), verified by
test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the
default, gather, and per-expert paths) and by coherent end-to-end
generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original
per-expert path.

Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192
(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):
activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused
quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The
fused path avoids materializing and re-reading the 8x compact buffer,
writing the expert copies directly from registers.

* CUDA: bounds-check token ids in build_tok2c_kernel

Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or
t >= n_tokens) and drop entries beyond n_expert_used per token instead of
writing past the token's tok2c region. No behavior change for valid MoE
routing data; test-backend-ops MUL_MAT_ID 790/790.

* Refactor the code based on review comments

- Removed previously added kernels that were not necessary anymore\
- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.

* Adding q8_1 support for dedup and addressing review comments

* Add pragma unrolls

* Remove redundant cudaMemsetAsync call

* Removing follow up redundancies

---------

Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>

PR ggml-org#16308 set info.devices[id].integrated = false unconditionally for all
CUDA/HIP devices as a workaround for corrupted output on Jetson Orin
(ggml-org#15034). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag is
needed: with the cached field forced to false, supports_buft() refuses
CUDA host buffers on AMD APU/UMA parts, while get_type() already reads
prop.integrated (ggml-org#23007) — an inconsistency that breaks integrated-GPU
host-buffer use on ROCm.

Guard the workaround so it only applies to non-HIP (CUDA) builds and
restore prop.integrated for HIP, keeping the Jetson workaround intact for
CUDA.

Fixes ggml-org#23977

Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>

Otherwise this gives lots of unnecessary warnings:

  W srv    operator(): (CORS) skip non-localhost origin:

* support cuda virtual devices

* disable NCCL path when virtual devices are used

* label virtual devices in description; add GPUx2 server CI jobs

* code refactor

…l-org#25733)

* spec: fix dflash target tokenizer mismatch during conversion

* fix ci ty check

)

Microsoft BitNet Hugging Face configs use BitNetForCausalLM while the
converter only registered BitnetForCausalLM, causing conversion to fail
with "Model BitNetForCausalLM is not supported".

Register both spellings in TEXT_MODEL_MAP and the Bitnet model class.

Fixes ggml-org#25629

…t to avoid race (ggml-org#25229)

)

The current integration treats SME as a single capability (CPU_FEATURE_SME)
with no distinction between SME(v1) and SME2. The kernels dispatched under
CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect
on SME(v1)-only hardware.

We introduce build-time and runtime distinction between SME and SME2, and
wire SME(v1) and SME2 kernels based on actual hardware support.

…ing) and more MUL_MAT updates (ggml-org#25762)

* hex-mm: fix artificial limit in the solver that restricted number of act-prep threads

* hex-mm: fix warning

* hex-prof: do not apply --top to the timeline report

* hmx-mm: add suport for tiled act-processing to better distribute hvx work

* hex-l2: add tracing for l2flush events

* workqueue: redo the legacy workpool api to match hmx-queue and dma-queue

* hmx-mm: fix f32 activation buffer alignmnet for nhvx=5,6,7

* hex-work: minor cleanup for work-queue apis

* hex-work: further cleanup of the work-queue api

* hex-l2: optimize l2flushes at the opbatch level

* hex-work: remove unused mask

* hex-work: no need to drop hvx ctx in the work-queue

* hex-work: add explicit wakeup/suspend and make threads spin

* hex-bufs: mark any non-weight tensor as compute

* hex-dma: dma-queue support for alias queues and cached dma

* hex-l2: track tensor aliases and delay or skip flushes as much as possible

* hex-l2: simplify tensor alias handling

* hex-l2: handle overlapping views as a circular list of aliases

* hex-tens: add flags helper

* hex-l2: add helper for marking tensors clearn/dirty

* hex-l2: mark binary and rope outputs as l2-clean and keep the rest as is for now

* hex-l2: proper support for handling all tensor overlap scenarios

* hex-trace: instrument matmul init code and cleanup trace checks

* hex-thread: introduce dedicated main thread with explicit stack and priority

* hex-l2: track dirty state as bitmap and introduce threaded flush

* hex-trace: remove redundant checks for ctx != null

* hex-l2: allocate entire context as one buffer and l2fetch it after big flushes

* hex-l2: disable tensor clearing in binary and rope for now seems to cause issues with fusion

* hmx-mm: update act proc to use fastdivs and fix DMA overflow

* hmx-mm: make MUL_MAT_ID kernels robust to multi-chunk cases (start_row>0)

* hex-queue: remove obsolete queue interfaces and flush hmx-queue at the end of the op-batch

* hex-queue: dont use early wakeup for small op-batches

* hex-tensors: properly cap max_tensors in op-batches and dirty_map

* hex-l2: make sure threaded l2flush does proper rounding

* hex-l2: factor out htp_tensor_flush for reuse (if needed)

* hex-l2: optimize tensor flushes by coalescing flush-all

* hex-l2: optimize multi-threaded flush

* hex-drv: futureproof version checks

* hexagon: fix errors and warnings on windows

* hex-main: update main thread to only use dspqueue_read, dspqueue_peek is not available on some platforms

* hex-main: add fallback mode for dspqueue with callbacks

* hex-main: introduce fallback mode for using dspqueue callbacks for full op processing

* hex-main: remove early wakeup, not helping and seems to cause some errors with certain batch sizes

* hex-l2: make sure to use invalidate version of flushall

* hex-l2: dont try to trace early l2flush at the start of op-batch

* hex-main: remove offset_ctx that must be zero anyway

* hex-hmx: fix hmx_queue_depth to use idx_write - idx_read

* hex-hmx: use atomic_load for idx_read/write

* hex-main: add static assert to make sure n_threads are aligned

* dsv4 hc-ops

* add missing files;

* add cparams

* update rpc version

* address review comments

* address review comments

…for Adreno A7x GPUs) (ggml-org#25780)

* opencl: load quant as uint in mv_q4_k_f32_flat

helps older compilers (e.g., E031.41, boosts 2x),
no impact on newer compilers (e.g., E031.45 or newer)

* opencl: load quant as uint in mv_q5_K_f32_flat

helps older compilers (e.g., E031.41)

* opencl: format

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>

…25690)

* sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel

This is the only Q5_K kernel that was not using KQPI.

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: add missing second half processing to reordered q5_k

Error found while running

  GGML_SYCL_PRIORITIZE_DMMV=1 \
  build/bin/test-backend-ops test -o MUL_MAT

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix potential off-by-one error

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* sycl: fix missing row > nrows check

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

---------

Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>

* vulkan: Support Q2_0

The backend perf tests for mat-vec-mul weren't very good at first (worse than
q2_k), doubling the rows per workgroup made a big difference.

* reorder

* resolve merge conflict, adjust err threshold for f16->q2_0 set_rows

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* common: auto-download dflash- and eagle3- HF sidecars

Mirror the existing mtp- sidecar logic to support auto-discovery and
download of DFlash (dflash-) and Eagle3 (eagle3-) speculative decoding
sidecars from Hugging Face repos.

Changes:
- Add --dflash and --eagle3 CLI flags to trigger sidecar download
- Add find_best_dflash() and find_best_eagle3() using find_best_sibling
- Exclude dflash- and eagle3- filenames from primary model selection
- Filter dflash- and eagle3- from cached model listings
- Wire download tasks that set speculative.draft.mparams as fallback

Assisted-by: pi:llama.cpp/Qwen3.6-27B

* docs : regen

…nel tensors (ggml-org#25822)

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>

…#25805)

* add a direct size condition for `large` weights; the original
  dimension condition is insufficient -- q6_K lm_head for gemma-4 E2B
  has [1536, 262144], which is big enough to slowdown gemv_noshuffle but
  does not satisfy the dimension condition (ne0 >= 2048)

* ggml: use dynamic allocation for split graph inputs

Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically
allocated buffers in the backend scheduler. This fixes crashes when
loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on
multi-backend setups where graph splits exceed 30 input tensors.

- split->inputs: dynamic array with grow-on-demand
- sched->graph_inputs: dynamic array with grow-on-demand
- graph_size calculation now uses actual input count instead of fixed constant

* cont : clean-up

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* server: add get_info tool

* fix --rpc in docs

* server: harden get_info probe result handling

Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.

* server: simplify get_info result handling

Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.

---------

Co-authored-by: Pascal <admin@serveurperso.com>

…#26172)

* tests: add model resolution test on synthetic repo listings

Include download.cpp and arg.cpp inside a namespace with hf_cache
monkey patched to serve hardcoded listings, so the resolution and the
model handler assembly are tested end-to-end through the real CLI
parsing, without modifying the tested code and without network access.

Covers the primary, shard, mmproj, sidecar and preset resolution on
layouts mimicking real vendor conventions, replays every case on
permutations of the listing to assert determinism, and asserts the
final wired paths for the spec type auto-selection, the -md precedence
and the fallback suppression.

* tests: keep model resolution checks active and let the handler resolve

Replace assert with a REQUIRE macro alive in Release builds, key the
fake hf_cache by repo id so the real handler init resolves every plan
itself, check the exact shard sets, restrict the permutation exception
to the order dependent picks, and cover dflash and eagle3.

* tests: fix model resolution build on fatal warnings CI and Windows

The namespaced copy of the sources leaves many static functions unused
in this TU, exempt it from the unused warnings. Pre-include the
windows headers so arg.cpp does not pull them inside the namespace.
Declare the renamed copies of the download.h functions, verbatim from
the header and renamed in sync by the macros, so missing declaration
and missing prototype warnings are satisfied on every toolchain.

* tests: fix winsock inclusion order for the model resolution test

WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does
not redefine the socket types afterwards.

* tests: link cpp-httplib to the model resolution test

The test compiles its own copy of download.cpp, which calls httplib
directly, and the private link of llama-common does not propagate the
symbols under lld-link.

* common_http_client

* common: finish the http client wrapper

Add the virtual Head, Get and Post methods and the passthrough
setters to the common_http_client skeleton, move follow_location
into the constructor, expose the underlying client for the ranged
pull path, and rename the missed common_http_client_init call sites.

* tests: rewrite model resolution on the http client stub

Replace the namespace inclusion of the sources by a plain TU: the
common_http_client factory returns a stub serving hardcoded HF API
responses, so the real hf_cache parsing, resolution and CLI handler
run against synthetic listings in an isolated cache directory.

Failures print the named case, the reordering and the actual versus
expected values, the assembly cases use the full command line as
context, and the empty result cases are checked once to keep the
logs short.

* tests: fix the model resolution on Windows and the builds without TLS

Assert the exact expected paths composed like the cache does instead of
suffix matching on forward slashes, set the environment portably, and
serve the stub through an http endpoint so the builds rejecting the
https scheme still reach it. Pause the log so the negative cases can be
replayed on every reordering.

* tests: make the model resolution failures self explanatory

Resume the paused log before the failure report so the CI shows why
the tested code bailed, and format the stub oids portably.

* common: hold the http client factory behind exported functions

The factory was an inline variable, and the Windows shared builds
export functions but not data, so the executable and the DLL each had
their own instance: the stub installed by the test was invisible to
the library, which reached for the real endpoint and resolved nothing.
Route the creation through functions compiled into the library and
format the stub oids portably.

* common: add the http client factory source missed in the previous commit

* common: typedef the http client factory callback

Address review from @ngxson

* tests: serve the model resolution repos over the loopback

Replace the client stub by a real httplib server bound to the
loopback, so no C++ object crosses the module boundary anymore and
the library exercises its own client and transport end to end. The
debug shared build on Windows crashed inside the stubbed path.

* common: add portable common_get_env and common_set_env helpers

Address review from @ngxson

* common: drop the http client factory left without a caller

The loopback server made the stub substitution unnecessary, the client
init builds the real client directly again.

* common: read the model endpoint through the env helpers

* nit: drop the stub leftovers from the model resolution test

* common: align common_set_env and isolate the test cache per run

The POSIX branch now behaves like _putenv_s, so the helper has a single
contract on every platform, and common_get_env already reads an unset and
an empty variable alike.

The model resolution test keys its cache directory on the loopback port,
where two concurrent runs on the same machine used to share one directory
and the initial cleanup of either wiped the other.

* tests: move the model resolution server into main

* tests: support the DSpark sidecar resolution

* common: revert the http client to the plain httplib client

address review from @ngxson

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

* common/chat: update DeepSeek V4 templates

Align the DeepSeek V4 templates with the official encoders while keeping parser behavior out of this change.

- Default drop_thinking for DeepSeek V4 history so prior thinking is omitted unless preserve_reasoning is requested or tools are present.
- Add structured output response-format instructions to the V4 templates and pass the schema into template rendering.
- Add a separate Flash 0731 template for the updated high and max reasoning effort mapping.
- Cover reasoning effort, drop_thinking, structured output prompts, preserved reasoning, continuations, and empty tool arguments in template rendering tests.

Official references:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.py
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py

Assisted-by: Codex

* Fix deepseek v4 0731 template selection

* remove unneeded lower normalization

* Fix DSML parser to consume the tool call separator

* address aldehir requests

* address aldehir comment

…gml-org#25874)

* sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches

Extends the oneDNN SDPA path (PR ggml-org#25222) to handle non-F16 KV caches by
dequantizing or converting K/V to dense FP16 on-device before feeding
them into the SDPA graph. The fused systolic kernel then runs identically
to the native FP16 path.

Supported KV types:
  - Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl
  - F32: cont_to_f16_sycl<float>
  - BF16 and IQ types are excluded (no conversion kernel available)

Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only).
F16 KV runs at any length (existing behavior).

Also includes the stream sync fix (stream->wait_and_throw() unconditional,
PR ggml-org#25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are
always dequantized to separate buffers).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR)
Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

* sycl: parallelize the non-contiguous concat kernel

Launch geometry only: the non-contiguous concat kernel launched a single-lane
work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one.

SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`.

llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV),
on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)

* sycl: cap non-contiguous concat block at ne0

* sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)

* Revert "sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)"

This reverts commit 2709909.

…-org#26520)

This matches how it is done for logit_bias and mirostat samplers, see
ggml-org#25262 (comment)

…rg#26217)

bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.

Applies the same fix to chatglm.py.

* validate plamo2 byte tokens

* --typo

* vulkan : add GATED_LINEAR_ATTN op

* docs : update Vulkan ops

* vulkan : remove unused GLA spec constant

* Updated ops.md

* ops.md update

…gml-org#25401)

The Python GGUF reader lacked two guards the C++ loader has:
- n_dims read as uint32 with no GGML_MAX_DIMS bound -> crafted file with
  huge n_dims triggers oversized memmap read / OOM.
- np.prod(dims) on uint64 wraps silently -> a crafted dims triple can
  overflow to a tiny element count, passing an undersized read through.

Add a GGML_MAX_DIMS check and compute the element count with Python ints.

Fixes ggml-org#25378

…org#26510)

This commit contains a suggestion to reduce some code duplication in
common_speculative_init when adding the enabled speculative decoding
configurations.

No tests were added but the existing server tests still passes with this
change:
```console
$ ./tests.sh unit/test_speculative.py -v -x
```

…gml-org#26375)

* ci: fix pre-built binaries no longer working on macOS 15 and below

* ci: add macOS deployment target to disabled KleidiAI build

Merges 251 upstream commits on top of the fork's 392. Base was 22b208b
(2026-07-15).

What this brings in for DeepSeek V4:

- CUDA kernels for the hyper-connection ops and the lightning indexer
  (dsv4-hc.cu, lightning-indexer.cu, upstream ggml-org#25585 and ggml-org#25545). These
  landed upstream after our base, so the graph no longer needs a CPU
  fallback for those ops.
- MTP and DSpark support (ggml-org#25784), the wo_a reshape fix on load, and the
  same-K/V-cache-type enforcement (ggml-org#25871).
- Exclusion of the i32 ffn_gate_tid2eid routing table from quantization,
  which the fork did not carry.

Conflict resolution kept both architectures everywhere the two sides
touched the same code:

- llama-kv-cache: kept the fork's default-off attention-rotation policy
  and its env overrides, took upstream's GLM_DSA addition to the DSA
  indexer arch list.
- llama-context: moved the TurboQuant flash-attention auto-enable above
  upstream's generic quantized-V check, which would otherwise reject
  turbo cache types under -fa off, and dropped the fork's older V-cache
  check in favour of upstream's.
- mmq.cuh: kept the fork's int64 offsets in all three of upstream's new
  NVFP4 branches.
- fattn.cu: dropped the WMMA block, since upstream removed that kernel
  and its helpers entirely; kept the RDNA4 turbo path.
- ggml-cuda.cu: kept the host-staged cross-device copy and routed its
  peer copy through upstream's new virtual-to-physical device mapping.
- chat.cpp: rebuilt on upstream's file with the fork's Inkling and
  Laguna parsers and the leading-whitespace tolerance reapplied;
  thinking_end_tag became thinking_end_tags upstream.
- laguna.cpp/laguna.py and mtmd-image.cpp: took upstream, which already
  carries the fork's own upstreamed review fixes plus later refinements.
- Removed the inherited upstream workflows again, per 0c9a069.

GGML_OP_COUNT is 103: upstream's 101 plus the fork's TURBO_WHT and
FLASH_ATTN_EXT_BANDED.

Also drops a duplicate LLM_ARCH_LAGUNA case in test-llama-archs that the
merge would otherwise have left in moe_mandatory.

Merges feat/sync-upstream-deepseek4 into the Kimi K3 branch so K3 (text
and MoonViT-V2 vision) sits on the same tree as DeepSeek V4.

Five conflicts, all additive collisions where both sides appended to the
same list:

- llama-arch.cpp and llama-model.cpp: kept both KIMI_K3 and INKLING cases.
- mtmd/CMakeLists.txt: kept both models/kimik3.cpp and models/inkling.cpp.
- mtmd/models/models.h: kept both clip_graph_kimik3 and clip_graph_parakeet.
- gguf-py constants: same HEAD_DIM key, took the sync branch's comment.

The K3 loader requires kda, situ, attn_res and moe_latent hparams that
the synthetic fixture never set, so the arch aborted while building the
model and took the whole test binary down. Skip it the same way
deepseek4 and inkling already are, with the missing params recorded.

Predates the upstream sync: the K3 branch never registered the arch with
the fixture generator.