[AMD] Add HiCache KV offloading for Qwen3.5 MXFP4 MI355X cookbook - #35879
Open
ChangLiu0709 wants to merge 2 commits into
Open
[AMD] Add HiCache KV offloading for Qwen3.5 MXFP4 MI355X cookbook#35879ChangLiu0709 wants to merge 2 commits into
ChangLiu0709 wants to merge 2 commits into
Conversation
…ump MI355X docker image - Add 'KV Offloading' UI toggle (Disabled/HiCache) in the deployment generator, visible only for MI355X + FP4 + 397B. When enabled, emits --enable-hierarchical-cache, --hicache-ratio 1.5, --hicache-write-policy write_through, --hicache-io-backend direct, and --hicache-mem-layout page_first_direct. - Bump MI355X docker image from v0.5.15.post1-20260715 to v0.5.17-20260818. - Add HiCache prose note in the deployment notes section. Aligned with InferenceX sgl-project#2693.
ChangLiu0709
requested review from
JustinTong0323,
sogalin,
wisclmy0611 and
zijiexia
as code owners
August 21, 2026 15:49
github-actions
Bot
added
the
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add HiCache host-DRAM KV offloading support for the Qwen3.5-397B-A17B MXFP4 MI355X deployment, and bump the MI355X Docker image.
Changes
Deployment Generator (
qwen35-deployment.jsx)Disabled(default) andHiCache.--enable-hierarchical-cache--hicache-ratio 1.5--hicache-write-policy write_through--hicache-io-backend direct--hicache-mem-layout page_first_directCookbook Page (
Qwen3.5.mdx)v0.5.15.post1-rocm720-mi35x-20260715→v0.5.17-rocm720-mi35x-20260818.Motivation
Aligned with InferenceX #2693, which adds HiCache host-DRAM KV tier arms to the Qwen3.5 MXFP4 MI355X AgentX sweep (TP4 concurrency 40–64, TP2 concurrency 20–32 with HiCache).
cc @yichiche
CI States
Latest PR Test (Base): ✅ Run #32719069137
Latest PR Test (Extra): ❌ Run #32719068970
Latest PR Test (AMD ROCm 7.2): ➖ No AMD PR run found for this commit.