HIP/ROCM: Memory-usage growing until server crashes

[Te-eMster]

I am running llama-server on a dual MI50 (32GB) setup. I am using the HIP-backend because for gfx906 it seems to be more reliable for long context performance than Vulkan.

I am mostly using it to run models to work with Roo Code in VS Code. I am often running orchestrator-prompts which create several subtasks autonomously.

However, I am noticing that over time, the memory usage on the GPUs steadily increases until the server eventually runs out of memory. If I manage to shut down the server manually before the crash, in the memory breakdown there is a large amount of memory in the "unaccounted" column.

Fresh start:

Just before crash:

It is happening on MiniMax M2.5/M2.7 (running mixed with some MoE-layers on CPU) and also on Gemma 4 31B (running purely on GPU). Both models have fairly large KV-Cache requirements.
Qwen 3.5 27B doesn't seem to be affected as much by the memory growth - it also reserves very little KV-Cache for max context length.

Can anyone relate and explain what's happening? Is there any runtime argument that can prevent this from happening?

[ORippler]

Can anyone relate and explain what's happening? Is there any runtime argument that can prevent this from happening?

Try running with GGML_CUDA_DISABLE_GRAPHS=1, there is an outstanding memory leak in the CUDA backend, and I'm assuming it will affect ROCm as well since I see corresponding declarations in the vendor file (#21611 #21673 and #19639 for some more context)

[Te-eMster]

Thanks a lot.

I actually found that when keeping the VRAM usage below ~82% with weights and context, the VRAM usage does not grow. Above that, it steadily grows until OOM. You can play around with --fit-target, but I have no clue as to why this threshold would have any significance.

[Recursive-Logic-Core]

@Te-eMster - The ~7.5 GB swell in the "unaccounted" column is driver/runtime memory fragmentation inside the HIP/ROCm layer, outside of GGML's tracked allocators:

1. What "unaccounted" actually measures:
   "self" strictly tracks buffers managed by llama.cpp (model weights + allocated KV cache + compute graph). The "unaccounted" metric is (total - free - self). The leak is not happening inside llama.cpp's tensor allocations, but inside the ROCm runtime driver (HSA) and rocBLAS.

2. The ~82% threshold explained:
   The ROCm HSA memory manager and rocBLAS scratchpad pool rely on buddy allocation. When static memory (weights + KV-cache) exceeds ~80-82% of total VRAM, contiguous block coalescing fails under dynamic memory pressure.
   Because Roo Code is an agent workflow, input sequence lengths vary dramatically on every turn. With each varying batch/token dimension, rocBLAS requests new GEMM workspace buffers. In low-headroom situations (<18% free), the driver allocator cannot merge fragmented chunks, fails to reuse cached workspace pages, and continuously requests fresh virtual memory blocks without releasing them back to the free pool until the card hits hard OOM.

3. Practical mitigation for dual MI50 (gfx906):

• Cap static context: Using --fit-target 0.80 or manually sizing -c so base usage stays under ~80% ensures the HSA allocator retains enough contiguous space for dynamic GEMM workspaces.
• Disable Graph Execution: Launch with GGML_CUDA_DISABLE_GRAPHS=1 (also covers HIP), as HIP graph capture leaks node allocations on variable agentic batch shapes on older ROCm releases.
• Clamp workspace overhead: Set the environment variable ROCBLAS_INTERNAL_GEMM_CACHE_SIZE=0 (or a small integer like 32) before starting llama-server to force rocBLAS to prune scratchpad caches between turns rather than retaining unbounded allocations.