[gfx1201][Windows] Random heuristic selection of FP32 T,N solution 140220 causes ~2x text-encoder slowdown #2300

Title

[gfx1201][Windows] Random heuristic selection of FP32 T,N solution 140220 causes ~2x text-encoder slowdown

Summary

On Windows / gfx1201, hipBLASLt appears to randomly choose between two FP32 T,N solutions for the same GEMM family when a process starts.

For my workload, choosing solution 140220 makes the text-encoder stage roughly ~2x slower, while solution 140215 gives normal performance. Once a process gets one of these choices, subsequent generations in the same process remain consistently fast or consistently slow.

I was able to force each solution with HIPBLASLT_TUNING_OVERRIDE_FILE, and the behavior reproduced consistently:

• Force 140215 → normal performance, 2/2 fresh process launches
• Force 140220 → slow performance, 2/2 fresh process launches

This strongly suggests that the slowdown is caused by the heuristic / solution selection itself rather than model warmup or general GPU state.

Environment

• OS: Windows 11
• GPU: AMD Radeon RX 9070 XT
• Architecture: gfx1201
• PyTorch: 2.15.0a0+rocm10.1.0a20260822
• HIP: 7.16.26332
• ROCm nightly: 10.1, 2026-08-22 build
• GPU selected with HIP_VISIBLE_DEVICES=1
• Workload: MiniMax H3 text encoder in ComfyUI
• Affected GEMMs: FP32, transA=T, transB=N

Original random behavior

With hipBLASLt logging enabled:

set HIPBLASLT_LOG_MASK=160
set HIPBLASLT_LOG_FILE=...

a normal/fast process selected solution 140215 for GEMMs such as:

m=1024  n=1034 k=5120
m=8192  n=1034 k=5120
m=25600 n=1034 k=5120
m=5120  n=1034 k=8192
m=5120  n=1034 k=25600

a_type       = f32_r
b_type       = f32_r
c_type       = f32_r
compute_type = f32_r
transA       = T
transB       = N
batch_count  = 1
solution     = 140215

A slow process selected the same GEMM shapes with:

solution_index = 140220

The same process-local selection also appeared across additional FP32 T,N GEMMs, including:

m=3456 n=4096 k=1152
m=1152 n=4096 k=1152
m=4304 n=4096 k=1152
m=1152 n=4096 k=4304
m=4608 n=1024 k=4608
m=5120 n=1024 k=4608

Controlled reproduction

I generated two tuning override files containing the affected GEMM shapes.

Force solution 140215

set HIPBLASLT_TUNING_OVERRIDE_FILE=...\te_override_140215.txt

Results:

• Fresh launch Add TensileLite #1: normal TE performance
• Fresh launch Add tensilelite #2: normal TE performance

The hipBLASLt logs confirmed that the affected FP32 T,N GEMMs used 140215.

Force solution 140220

set HIPBLASLT_TUNING_OVERRIDE_FILE=...\te_override_140220.txt

Results:

• Fresh launch Add TensileLite #1: slow TE performance
• Fresh launch Add tensilelite #2: slow TE performance

The logs confirmed that the same GEMMs used 140220.

Other unrelated GEMM solution indices still varied between these runs, but text-encoder performance consistently followed the forced 140215 vs 140220 choice.

This makes the affected FP32 T,N solution family the likely cause rather than a global slow/fast GPU state.

Example override entry

transA,transB,batch_count,m,n,k,a_type,b_type,c_type,compute_type,solution_index
T,N,1,1024,1034,5120,f32_r,f32_r,f32_r,f32_r,140215

Using the same entry with 140220 reproduces the slow case.

Expected behavior

For identical gfx1201 GEMM problems, heuristic selection should consistently choose an appropriately performing solution and should not randomly cause a very large process-to-process performance regression.

Actual behavior

Across fresh process launches, the same FP32 T,N GEMM family can select either:

140215 -> normal performance
140220 -> approximately ~2x slower text-encoder stage

The selected behavior remains stable for subsequent generations within that process.

Workaround

Using HIPBLASLT_TUNING_OVERRIDE_FILE to force solution 140215 for the affected shapes removes the random slowdown in my tests.

Additional notes

I initially suspected model warmup, dequantization, VRAM residency, or general GPU state.

Earlier profiling showed the slowdown was concentrated in regular matrix multiplications, while NVFP4 dequantization time remained essentially unchanged.

The controlled 2×2 tuning-override experiment is the strongest evidence that the regression is specifically tied to the hipBLASLt solution choice.

I can provide the complete hipBLASLt logs and tuning override files if useful.