Transparent Huge Pages (THP) — recommended configuration for production deployments? #9051
Replies: 7 comments 2 replies
|
To be honest, we haven't extensively tested this. I therefore cannot say whether it will be beneficial or not. My best guess is that it would behave similarly to the other databases you've mentioned. We did run a few tests with a larger default page size (64K), which did show a performance regression. But this has been some time ago.
No, we do not. If possible, I'd recommend to test it out on your own hardware to get an idea of the performance characteristics. |
|
Thanks a lot, guys! Your answers are very appreciated. For the moment, it's enough info for me. |
|
We’ve run Qdrant in production for real-time vector search (mostly HNSW), and THP has consistently been a pain point for latency-sensitive workloads. When THP is set to always, we’ve observed unpredictable stalls—especially during compaction—similar to issues seen in Redis and MongoDB. These stalls can spike query latency from sub-50ms to several hundred ms, which is unacceptable for interactive search. Our setup:
Why disable?
Shell snippet we use at deployment: echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defragUnless Qdrant changes its internal allocations or advertises hugepage support, “never” is the safe bet for production—especially for workloads with mmap-heavy vector indices. |
|
Another prod data point, we run Qdrant behind a multi-tenant legal doc platform with sub-second retrieval SLAs. Go never, for both enabled and defrag. Since Qdrant doesn't call madvise(MADV_HUGEPAGE) (timvisee confirmed above), the madvise mode does nothing for it anyway. The only thing THP can do to you in always mode is background compaction, which for an mmap-heavy process is all downside. The stall pattern looks like what others describe, clean p50 and ugly p99 spikes under memory pressure. If you want to confirm it's THP before touching anything, watch compact_stall in /proc/vmstat while the spikes happen. Runtime change: echo never > /sys/kernel/mm/transparent_hugepage/enabled To verify it stuck: grep AnonHugePages /proc/$(pidof qdrant)/smaps_rollup should read 0 kB and compact_stall should stop moving. Caveat, if your nodes never get memory pressured you might not see stalls at all, which is probably why benchmarks struggle to reproduce this. It bites when RSS gets close to node capacity. We measured no throughput loss with THP off so it's cheap insurance. |
Uh oh!
There was an error while loading. Please reload this page.
Hi,
I'm setting up a production Qdrant deployment on Linux and want to understand the recommended OS-level configuration for Transparent Huge Pages (THP).
Specifically:
Context: Qdrant relies heavily on mmap for vector index segments (HNSW). Many databases (MongoDB, Redis, TiDB) explicitly recommend disabling THP due to compaction stalls causing latency spikes. Is the same true
for Qdrant?
Any official guidance or documented recommendation would be appreciated.
All reactions