Important
Experimental AMD gfx1151 fork. This repository adds context lookup fill
to llama.cpp's embedded MTP speculative decoder. An MTP-generated prefix is
matched against tokens already present in the request context; when the match
is exact, the remaining draft positions are filled from that known
continuation and verified by the target model in one batch. The fast path is
opt-in through --spec-lookup-fill and falls back safely when no matching
continuation exists.
在 AMD Ryzen AI Max+ 395 / Radeon 8060S（gfx1151）上，本分支使用 Vulkan
运行带内嵌 MTP/NextN head 的 Ornith-1.5-35B-A3B。实测常规生成约
73 tokens/s，在高度可预测的上下文复述任务中峰值达到 141 tokens/s。
速度取决于提示词、草稿接受率、量化、驱动及系统负载；141 tokens/s 是特定
重复文本测试的峰值，并非所有任务的通用速度。
- Fork repository: https://github.com/atfa/llama.cpp-MTP-Context-Lookup-Fill
- Upstream llama.cpp base: commit
81191affa - Tested model: Ornith-1.5-35B-A3B, model commit
1239361, Q6_K GGUF - Tested system: CachyOS, AMD Ryzen AI Max+ 395 / Radeon 8060S (
gfx1151) - Backend: Vulkan, full GPU offload, Flash Attention enabled
- ▶ Watch the 36-second 141 t/s test recording / 查看 141 t/s 实测录屏
- Benchmark helper:
scripts/bench-ornith-mtp-lookup-fill.sh
Development disclosure: the fork was explored and edited with AI assistance, then compiled, tested, parameter-swept, and reproduced on the hardware above.
LLM inference in C/C++
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
普通 MTP 投机解码会逐个生成候选 token。候选越长，潜在加速越高，但生成草稿 本身也有成本，而且低置信候选最终会被目标模型拒绝。对于“重复上一段”、文档 复述、结构化模板、代码片段续写等任务，请求上下文中往往已经存在正确答案的 大段 token。
本分支把 --spec-draft-n-max 解释为一次目标验证的总草稿预算，并用
--spec-lookup-fill 为上下文续写预留槽位。例如：
--spec-draft-n-max 7 --spec-lookup-fill 6
表示先让 MTP 产生最多 1 个高置信 token，再用这个 MTP 前缀在上下文中寻找 完全一致的 continuation；匹配成功时追加最多 6 个 token，组成最多 7-token 草稿交给目标模型一次验证。若前缀不一致、上下文太短或没有候选，则不会拼接， 从而避免把相似但错误的文本盲目注入输出。
target samples token
│
▼
MTP drafts a short, high-confidence prefix
│
▼
search request context for the same prefix
│
├── exact match ──► append known continuation ──► target verifies batch
│
└── no match ────► keep the MTP-only draft
该模型包含 MTP/NextN 权重，因此使用 --spec-type draft-mtp。-fa on 表示启用
Flash Attention 推理路径；Flash Attention 是后端优化，并不代表这是所谓的
“Flash 模型”。
- 将
--spec-lookup-fill从 DFlash2 专用路径扩展到内嵌 MTP 解码器。 - 将总草稿预算拆分为 MTP 前缀和 lookup fill，避免为了上下文中已经存在的 token 继续执行昂贵的 MTP 步骤。
- 新增严格前缀校验：只有 lookup continuation 完整包含已有 MTP 前缀时才追加。
- 修正多 NextN head 模型的限制方式：只限制 MTP 自身步数，不再错误截断 lookup 可使用的总草稿长度。
- 保留并升级 DFlash2 模型转换、tensor mapping、selector lattice 和运行时支持。
- 增加 n-gram continuation 单元测试、参数解析测试以及基于 OpenAI chat API 的 自动参数扫描脚本。
- 同步到官方 llama.cpp commit
81191affa（llama.cpp0.2.0-dev, build 55）。
| Item | Tested configuration |
|---|---|
| Operating system | CachyOS |
| Processor / APU | AMD Ryzen AI Max+ 395 |
| GPU | AMD Radeon 8060S, gfx1151 |
| Backend | Vulkan (Vulkan0) |
| Model | Ornith-1.5-35B-A3B, commit 1239361 |
| GGUF | Ornith-1.5-35B-Q6_K.gguf |
| Offload | -ngl 999 |
| Attention | Flash Attention, -fa on |
| Context | 262144 tokens, one parallel slot |
| Batch | -b 2048 -ub 512 |
| Sampling | greedy, --temp 0 |
| KV cache | target and MTP draft K/V in F16 |
Representative local results:
| Workload / profile | Throughput | Notes |
|---|---|---|
| Normal generation | 73 t/s | ordinary content; lookup opportunities are limited |
Balanced n-max=7, fill=4, p-min=0.55 |
about 107 t/s on document repetition | keeps three MTP prefix slots |
Repetition-focused n-max=7, fill=6, p-min=0.55 |
141 t/s peak | highly predictable context-copy workload |
The peak run is preserved in the 36-second screen recording included with this repository.
n-max=7 was a hardware-specific sweet spot in local Vulkan testing. Increasing
the total draft to 8 or more could be slower even with a high acceptance rate,
likely because the target verification batch crosses into a less favorable
Vulkan kernel or scheduling shape. Always benchmark your own prompt mix.
Tip
推荐直接克隆本仓库，然后要求你的 AI 编程代理阅读本 README、检查你的显卡 与本地模型路径，并为你编译、运行和扫描最佳参数。 不同 AMD GPU、驱动、 内存配置和模型需要不同参数；让 AI 在你的机器上实际构建和测试，通常比直接 复制别人机器的命令更可靠。
git clone https://github.com/atfa/llama.cpp-MTP-Context-Lookup-Fill.git
cd llama.cpp-MTP-Context-Lookup-FillYou can then give your coding AI a request such as:
Read README.md, inspect my AMD GPU and Vulkan environment, build this fork,
locate my MTP GGUF model, run the quick benchmark, and recommend the fastest
stable parameters. Do not delete or overwrite my models.
Manual CachyOS/Vulkan build:
sudo pacman -S --needed base-devel cmake ninja shaderc vulkan-headers vulkan-icd-loader curl jq
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build build -j "$(nproc)"Recommended balanced server profile:
MODEL=/path/to/Ornith-1.5-35B-Q6_K.gguf
TEMPLATE=/path/to/chat_template.jinja
build/bin/llama-server \
-m "$MODEL" -dev Vulkan0 -ngl 999 -fa on \
-c 262144 --parallel 1 -b 2048 -ub 512 \
--jinja --chat-template-file "$TEMPLATE" \
--spec-type draft-mtp \
--spec-draft-n-max 7 --spec-lookup-fill 4 --spec-draft-p-min 0.55 \
--temp 0 \
--spec-draft-type-k f16 --spec-draft-type-v f16 \
--cache-type-k f16 --cache-type-v f16 \
--alias exec --host 0.0.0.0 --port 8080For document copying or other strongly repetitive workloads, change
--spec-lookup-fill 4 to --spec-lookup-fill 6. For mostly novel code or
creative text, pure MTP around --spec-draft-n-max 3 --spec-lookup-fill 0 can
be a better starting point.
Run the included API-based parameter sweep:
# Representative quick sweep; results and logs are written under bench-results/
REPS=3 MODE=quick scripts/bench-ornith-mtp-lookup-fill.sh
# Wider parameter sweep
REPS=3 MODE=full scripts/bench-ornith-mtp-lookup-fill.shThe benchmark owns only the server process it launches on port 18080. It does
not kill unrelated llama-server instances. Override MODEL, CHAT_TEMPLATE,
PORT, PROMPT, CONFIGS, or other environment variables when needed.
The remainder of this README is retained from upstream llama.cpp. General llama.cpp documentation, licenses, acknowledgements, and contribution guidance still apply to this experimental fork.
A few options to get llama.cpp installed on your machine:
- Visit https://llama.app and follow the instructions
- Run with Docker - see our Docker documentation
- Download pre-built binaries from the releases page
- Build from source by cloning this repository - check out our build guide
Once installed:
# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli
|
Built-in web UI against llama serve
|
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on
a wide range of hardware - locally and in the cloud.
- Plain C/C++ implementation without any dependencies
- Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
- AVX, AVX2, AVX512 and AMX support for x86 architectures
- RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
- 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
- Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
- Vulkan and SYCL backend support
- CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
The llama.cpp project is build on top of the ggml library.
| Backend | Target devices |
|---|---|
| BLAS | All |
| BLIS | All |
| CANN | Ascend NPU |
| CUDA | Nvidia GPU |
| HIP | AMD GPU |
| Hexagon [In Progress] | Snapdragon |
| IBM zDNN | IBM Z & LinuxONE |
| MUSA | Moore Threads GPU |
| Metal | Apple Silicon |
| OpenCL | Adreno GPU |
| OpenVINO [In Progress] | Intel CPUs, GPUs, and NPUs |
| RPC | All |
| SYCL | Intel GPU |
| VirtGPU | VirtGPU APIR |
| Vulkan | GPU |
| WebGPU | All |
| ZenDNN | AMD CPU |
- How to build
- Running on Docker
- Build on Android
- Multi-GPU usage
- Performance troubleshooting
- GGML tips & tricks
- XCFramework
- Completions
- Models
- Release process
- Contributors can open PRs
- Collaborators will be invited based on contributions
- Maintainers can push to branches in the
llama.cpprepo and merge PRs into themasterbranch - Any help with managing issues, PRs and projects is very appreciated!
- Read the CONTRIBUTING.md for more information
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
- sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain