Add two-section causal backward for shared K/V - #2722
Open
foraxe wants to merge 4 commits into
Open
Conversation
foraxe
marked this pull request as draft
July 22, 2026 12:39
This was referenced Jul 22, 2026
Resolve the SM100 block-sparse 2-CTA conflicts while preserving the opt-in two-section causal backward state machine. Signed-off-by: ningyunxiao.nyx <ningyunxiao.nyx@antgroup.com>
Carry the varlen score-mod backward update through the two-section reducer while preserving current SM100 block-sparse 2-CTA behavior. Signed-off-by: ningyunxiao.nyx <ningyunxiao.nyx@antgroup.com>
Author
|
GB200 validation update for current head
The first upstream block-sparse actual-kernel case found the missing-loader bug immediately on the prior head. After the fix, both a 1CTA case and a 2CTA head-dim-192 case advanced past that error but did not complete CUTLASS/Flex compilation within 15-minute and 10-minute caps, respectively. I am keeping the PR in draft until an actual-kernel block-sparse case completes with a correctness result; the completed tests above should not be read as covering that unresolved gate. |
foraxe
marked this pull request as ready for review
July 25, 2026 00:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an experimental CuTe SM100/SM110 backward path for two causal query
sections that share one physical K/V owner.
_flash_attn_bwd_two_section_causalaccepts two adjacent batch-one Q/output/dO/LSE views, two causal K/V prefix lengths, and one physical K/V tensor. It
returns dQ for both logical query sections and compact dK/dV in the physical
K/V owner's layout. This avoids materializing two independent K/V-gradient
buffers before a context-parallel owner-return stage.
The existing forward and backward APIs are unchanged. Unsupported inputs fail
closed.
Supported contract
0 < prefix1 <= prefix2Changes
flash_bwd_sm100.py;Validation
The focused test passed on one GB200 using:
The reduced wheel was also validated in an opt-in Megatron CP=4 training stack
at sequence length 262144. After eight updates, all 73 checkpoint tensors
matched the TE p2p reference with worst max-absolute difference
1.9073486328125e-6(atol=2e-6,rtol=0). Median iteration time overupdates 4-7 was 1707.45 ms for TE p2p and 1413.65 ms for the NVSHMEM candidate,
17.21% lower for that single clean comparison.
This PR does not claim a standalone kernel speedup: at the focused small shape,
one two-section launch did not beat two stock backward launches. Its intended
benefit is avoiding intermediate K/V-gradient materialization in the integrated
context-parallel path.
Stack relationship
This is a dependency of an experimental Megatron NVSHMEM context-parallel
backend:
provides stream-ordered CP owner-gradient return;
owns the opt-in backend, symmetric workspaces, and lifecycle.
Rebase status
The branch now merges cleanly with current
mainafter a semantic rebase overthe SM100 block-sparse 2-CTA changes from #2661 and the varlen score-mod
backward changes from #2547.
CPU-only CuTe fake compilation passed for:
The PR remains draft until the focused correctness test and the integrated
Megatron stack are rerun on a real GB200.