Prebuilt: carry TML Inkling on b10173 for the nightly PR set - #45
Prebuilt: carry TML Inkling on b10173 for the nightly PR set#45danielhanchen wants to merge 5 commits into
Conversation
Hybrid attention model: 55 sliding-window plus 11 global layers, banded content-dependent relative position bias instead of RoPE, per-layer short convolution state, fine-grained MoE (256 experts top-6 plus 2 shared), attention log-scaling past 128K, 1M context. Includes the GGML_OP_FLASH_ATTN_EXT_BANDED operator (CPU and CUDA, fused into the MMA flash attention kernel with an fp16 accumulator overflow guard), HF to GGUF conversion, chat template with typed content block parsing (interleaved thinking, narration and tool calls), mmproj vision and audio support, and backend op tests at production shapes.
# Conflicts: # ggml/src/ggml-cuda/mmq.cuh # src/llama-model-saver.cpp # src/llama-vocab.h
Upstream 910196f renamed common_chat_params::thinking_end_tag to thinking_end_tags and changed it from std::string to a vector, so data.thinking_end_tag = END_MESSAGE no longer compiles.
The add-inkling branch was cut before MiniMax-M3, Nanbeige4.2 and the
WebGPU arch-skip rework landed upstream, so it no longer merges onto the
release tag. Conflicts were all add/add collisions at the same insertion
points; both sides are kept:
- common/chat.cpp: keep common_chat_params_init_minimax_m3 and add
common_chat_params_init_inkling after it; both dispatch entries were
already clean
- gguf-py/gguf/constants.py, src/llama-arch.{h,cpp}: NANBEIGE and INKLING
both registered
- src/llama-quant.cpp: keep the MiniMax indexer skips and the arch-gated
Inkling shortconv/rel-proj skips
- tests/test-backend-ops.cpp: keep the large-KV F16 flash-attn cases and
the banded score-bias cases
Assisted-by: Claude Code
|
Superseded by #47. The conflict fix went onto ggml-org#25731's own branch instead, so the set pins the real PR rather than a fork-side mirror. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3af940f019
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (k->type != GGML_TYPE_F32 && v->type != GGML_TYPE_F32 && | ||
| rel->type == GGML_TYPE_F32 && ggml_is_contiguous(rel) && | ||
| // MMA ABI indexes rel by Q's batch: a singleton rel batch must take the stride-aware fallback | ||
| rel->ne[3] == q->ne[3] && rel->ne[0] <= (1 << 20)) { | ||
| ggml_cuda_flash_attn_ext(ctx, dst); |
There was a problem hiding this comment.
When K/V are F16 or BF16, this handoff also runs on HIP, Volta, and pre-Turing devices, where ggml_cuda_get_best_fattn_kernel() can select the tile or vector kernel. Only the MMA path consumes the banded rel_logits from src[5]; for example, AMD decode with one query token selects BEST_FATTN_KERNEL_TILE, silently omits the relative-position bias, and produces incorrect attention outputs. Gate this handoff on MMA selection or retain the dedicated banded kernel for those devices.
AGENTS.md reference: AGENTS.md:L15-L16
Useful? React with 👍 / 👎.
2 participants
ggml-org#25731 (add-inkling) was cut before MiniMax-M3, Nanbeige4.2 and the WebGPU arch-skip rework landed upstream, so it no longer merges onto the release tag. The nightly's resolve job dies on it, and every build job is then skipped.
This branch is the b10173 release tag with ggml-org#25731 merged in and the conflicts resolved. Pin it in scripts/unsloth/pr-set.json in place of the raw ggml-org#25731 head.
All conflicts were add/add collisions at the same insertion point, so both sides are kept:
common/chat.cpp- keepcommon_chat_params_init_minimax_m3and addcommon_chat_params_init_inklingafter it. Both dispatch entries incommon_chat_templates_apply_jinjawere already clean.gguf-py/gguf/constants.py,src/llama-arch.h,src/llama-arch.cpp- register both NANBEIGE and INKLING (enum entry plus arch name).src/llama-quant.cpp- keep the MiniMax indexer projection skips and the arch-gated Inkling shortconv / rel-proj skips.tests/test-backend-ops.cpp- keep both the large-KV F16 flash-attn cases and the banded score-bias cases.The base branch is the upstream b10173 tag, so the diff here is exactly the Inkling change.