Prebuilt: carry Kimi-K3 on top of TML Inkling for the nightly PR set - #46
Prebuilt: carry Kimi-K3 on top of TML Inkling for the nightly PR set#46danielhanchen wants to merge 12 commits into
Conversation
Hybrid KDA (linear) + MLA (full) attention as in Kimi-Linear-48B, plus five things that architecture does not have: 1. cross-layer residual attention (attn_res_block_size) 2. latent MoE (routed experts run at n_expert_latent) 3. situ activation (replaces SwiGLU everywhere) 4. MLA output gate (sigmoid gate before o_proj) 5. full-rank KDA gate (single ssm_g instead of ssm_g_a/ssm_g_b) K3's text_config reports KimiLinearForCausalLM - the older 48B architecture - so get_model_architecture routes on the top-level name instead. The KDA decay gate has two forms, selected by linear_attn_config's gate_lower_bound. It is not a clamp: when set it swaps the activation entirely (fla/ops/kda/gate.py), from -exp(A_log)*softplus(x) to lower_bound*sigmoid(exp(A_log)*x). K3 sets it to -5.0; kimi-linear leaves it unset, so that path is unchanged. Cross-layer residuals reuse ggml_dsv4_hc_pre for the weighted sum. That op is CPU + CUDA only, so Metal/Vulkan will fall back per-node until those kernels exist. The routed experts ship as compressed-tensors "mxfp4-pack-quantized". That is bit-compatible with ggml's MXFP4 - same E2M1 code assignment, same E8M0 scale byte, only the nibble positions within a block differ - so they are repacked rather than dequantized, losslessly and without a ~5.5 TB bf16 round-trip. The repack is built lazily because gguf_writer holds every added tensor until the final write. DeepSeek-V4 was already doing the identical bit-shuffling, so it now shares the helper. Verified against Moonshot's own code path (transformers + fla's Triton KDA kernels) on a tiny model exercising every K3-specific feature. Final-position logits vs the fp32 reference: 6.7e-05 rel / corr 1.00000000 for both the chunked and the recurrent delta-net path. MXFP4 blocks dequantize to the source weights with 0.0e+00 error. Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- `_res_parts` buffers (kind, tensor) pairs, not bare tensors - `get_tensors` must return an Iterator, matching ModelBase - LazyBase's `func` takes one argument, so pass the expert loaders through `args` instead of the closure - borrowing KimiLinearModel.set_vocab from an unrelated TextModel is deliberate and safe, but not expressible in the signature No behaviour change: the MXFP4 repack still dequantizes to the source weights with 0.0e+00 error and end-to-end logits are unchanged (8.386e-03 rel, corr 0.99996630). Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Boris Dvorkin <b_dvorkin@niuitmo.ru>
K3's assistant output is an XTML-ish tagged format built by the template's
open_tag/close_tag macros. Two properties break generic parsing:
1. The generation prompt ends with open_tag('think'), so the completion
starts inside the think section with no opening marker in the output
(thinking_forced_open).
2. Only <|open|>/<|close|>/<|sep|>/<|end_of_msg|> are special tokens; tag
names ("think", "response", "message") are ordinary text tokens.
Adds common_chat_params_init_kimi_k3 (PEG_NATIVE) with detection on the
marker trio, reasoning extraction, response unwrapping, and tool-call
parsing of the tools/call/argument tag structure with argument types
taken from the tool schema. Includes the K3 chat template fixture and 9
test-chat cases derived from real generations of the full 2.8T model.
Verified end-to-end against Kimi-K3-Q2_K (GrEarl/Kimi-K3-GGUF) on 8x B200:
content, reasoning_content, streaming deltas, and tool_calls all correct;
finish_reason stop/tool_calls as appropriate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-role message-start markers for token-level span splitting. User and assistant messages carry only the role attribute, so their full opener (through <|sep|>) is used; system and tool messages continue with more attributes (type=/tool=/index=), so those delimiters stop after the role's closing quote. Verified against the K3 tiktoken vocabulary that the closing quote is always a standalone token across all attribute variants, so the token-level prefix match stays exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chat : add Kimi K3 chat format (reasoning, content, typed tool calls)
Graph node budget, LLM_TYPE for the 93-layer config, four hparams promoted to required, and an E8M0 NaN check during the MXFP4 repack. Details in the PR description. Assisted-by: Claude Code
New kimik3 projector type, its graph builder and the mmproj converter. Also adds an optional clip.%s.attention.head_dim so build_vit stops deriving d_head from n_embd, which is wrong whenever a tower's qkv width differs from n_embd. Assisted-by: Claude Code
…uild kimi-k3-unsloth was rebased onto ggml-org#26185's head, which was cut before the Parakeet clip graph and the WebGPU arch-skip rework landed upstream. Two add/add conflicts: - tools/mtmd/models/models.h: keep clip_graph_parakeet and clip_graph_kimik3, both registered - tests/test-llama-archs.cpp: upstream replaced the WebGPU skip list with DEEPSEEK32/GLM_DSA after fixing the binding alias issues; keep that list and append KIMI_K3, which upstream never evaluated Assisted-by: Claude Code
Inkling and Kimi-K3 both register a new architecture, so they collide in the arch/model/mtmd registries. Both sides are kept: - src/llama-arch.cpp: KIMI_K3 and INKLING in llm_arch_is_hybrid and llm_arch_supports_sm_tensor - src/llama-model.cpp: both llama_model_* constructors and both LLAMA_ROPE_TYPE_NONE cases - tools/mtmd/CMakeLists.txt: build models/kimik3.cpp and models/inkling.cpp Assisted-by: Claude Code
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2b4c6d0675
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| {%- if thinking is undefined -%} | ||
| {%- set thinking = true -%} |
There was a problem hiding this comment.
When callers set enable_thinking=false (including through reasoning_effort: "none"), the rendering context provides enable_thinking, not thinking, but this template defaults the unrelated thinking variable to true. Kimi-K3 therefore still inserts the thinking-effort message and opens a think section, so the standard API cannot disable reasoning for this model.
Useful? React with 👍 / 👎.
| auto value = type == "string" ? p.tool_arg_string_value(p.until(ARG_END)) : | ||
| p.tool_arg_value(p.until(ARG_END)); |
There was a problem hiding this comment.
With tools enabled, both parser branches accept arbitrary text through until(ARG_END) rather than attaching prop.value() as a schema constraint. Consequently the generated grammar permits invalid values such as text for a numeric property, and the mapper then emits malformed JSON such as {"arg":abc}; nullable-string and other union schemas can also be serialized incorrectly because only a literal string type is recognized.
Useful? React with 👍 / 👎.
| { COMMON_CHAT_ROLE_ASSISTANT, "<|open|>message role=\"assistant\"<|sep|>" }, | ||
| { COMMON_CHAT_ROLE_USER, "<|open|>message role=\"user\"<|sep|>" }, |
There was a problem hiding this comment.
When a user or assistant message has a name, the new template inserts name="..." before <|sep|>, so these full delimiters never occur in the rendered prompt. Since server message-span checkpointing uses exact token matches, named user messages are not recognized as user boundaries; use the same role-only prefix form already used for system and tool messages.
Useful? React with 👍 / 👎.
| {%- endmacro -%} | ||
|
|
||
| {%- macro json_sorted(value) -%} | ||
| {#- Minja は tojson(sort_keys=true) を実装していない。K3 参照実装は |
There was a problem hiding this comment.
This newly added comment contains Japanese Unicode, while the repository explicitly requires avoiding any Unicode characters. Rewrite it using ASCII text so the template follows the documented contribution standard.
AGENTS.md reference: AGENTS.md:L73-L73
Useful? React with 👍 / 👎.
| case LLM_ARCH_PADDLEOCR: | ||
| case LLM_ARCH_MIMO2: | ||
| case LLM_ARCH_KIMI_LINEAR: | ||
| case LLM_ARCH_KIMI_K3: |
There was a problem hiding this comment.
Adding K3 as a mandatory MoE architecture makes the CPU architecture test instantiate this fixture, but get_gguf_ctx() never writes the newly required kimi-k3.expert_latent_length, attention-residual, or SITU keys. Running test-llama-archs --arch kimi-k3 therefore exits during model loading with key not found in model: kimi-k3.expert_latent_length, before any backend comparison runs.
Useful? React with 👍 / 👎.
| uint32_t n_expert_latent = 0; // routed_expert_hidden_size (0 = experts run at n_embd) | ||
| uint32_t attn_res_block_size = 0; // 0 = no cross-layer attention residuals | ||
| float kda_gate_lower_bound = -INFINITY; | ||
| float situ_beta = 1.0f; | ||
| float situ_linear_beta = 0.0f; // 0 = no linear-beta transform on the up branch |
There was a problem hiding this comment.
These new K3 values are required again by load_arch_hparams, but llama_model_saver::add_kv_from_model() does not serialize any of them and K3 is still reported as a saver-supported architecture. Calling llama_model_save_to_file on a loaded K3 model consequently produces a file that cannot be reloaded because expert_latent_length and the other mandatory K3 metadata are absent.
Useful? React with 👍 / 👎.
| @ModelBase.register("KimiK3ForConditionalGeneration") | ||
| class KimiK3VisionModel(MmprojModel): |
There was a problem hiding this comment.
This new vision converter is unreachable during normal mmproj conversion: get_model_class(..., mmproj=true) rejects architectures absent from MMPROJ_MODEL_MAP before importing conversion.kimivl, and the commit adds K3 only to TEXT_MODEL_MAP. Every K3 mmproj conversion therefore raises Architecture 'KimiK3ForConditionalGeneration' not supported instead of constructing this class.
Useful? React with 👍 / 👎.
5 participants
Companion to #45. Kimi-K3 for the nightly prebuild, rebuilt so it merges onto the release tag.
kimi-k3-unsloth(#44) sits on ggml-org#26185's head, which was cut before the Parakeet clip graph and the WebGPU arch-skip rework landed upstream, so neither ggml-org#26185 nor #44 merges onto b10173 on its own. Inkling and Kimi-K3 also collide with each other, since both register a new architecture in the same registry files, so they have to be combined once rather than pinned as two independent heads.Two steps, both merges:
tools/mtmd/models/models.h- keepclip_graph_parakeetandclip_graph_kimik3.tests/test-llama-archs.cpp- upstream replaced the WebGPU skip list with DEEPSEEK32/GLM_DSA after fixing the binding alias issues (ggml-webgpu: Fix some binding alias issues to support all archs, fix recurrent-state-rollback test ggml-org/llama.cpp#25931); keep that list and append KIMI_K3, which upstream never evaluated.src/llama-arch.cpp- KIMI_K3 and INKLING inllm_arch_is_hybridandllm_arch_supports_sm_tensor.src/llama-model.cpp- bothllama_model_*constructors and bothLLAMA_ROPE_TYPE_NONEcases.tools/mtmd/CMakeLists.txt- buildmodels/kimik3.cppandmodels/inkling.cpp.The base branch is #45's head, so the diff here is the Kimi-K3 change plus the two-way registry resolution.