Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU.
MemVanta is a memory-first local LLM runtime for running quantized Llama-family GGUF models on CPUs with limited RAM. It uses mmap-backed model access, paged KV cache, Q4/Q8 kernels, and bounded adaptive prefetching, with reproducible memory and throughput benchmarks against pinned llama.cpp.
Website · Getting started · Latest release · PyPI · 7B benchmark · DOI · Reproduce · Contributing · Discussions
MemVanta explores a specific systems trade-off: how much resident memory can CPU LLM inference avoid while still executing a real quantized GGUF model correctly?
It is designed for experiments where RAM pressure matters more than maximum token throughput, including constrained developer machines, edge systems, and research into memory-aware local inference.
The project is intentionally transparent about the cost of that trade-off. MemVanta is memory-first; it does not claim to be faster than llama.cpp.
| Metric | MemVanta | pinned llama.cpp |
|---|---|---|
| OpenLLaMA 7B v2 Q4_0 peak RSS | 3.80 GiB | 7.24 GiB |
| Prompt processing | 2.92 ± 0.00 tok/s | 11.95 ± 0.02 tok/s |
| Token generation | 1.92 ± 0.00 tok/s | 7.97 ± 0.03 tok/s |
| Peak-RSS reduction | 47.54% | baseline |
Source of truth: results/openllama-7b-v2-ab/summary.json. The README table is generated from that file; do not edit its numbers by hand.
The result above is a repeated same-model CPU A/B test on OpenLLaMA 7B v2 Q4_0. It applies to the tested model, workload, host, and pinned comparison runtime; it is not a universal memory-reduction claim.
A separate cgroup-v2 experiment also measured execution under tight memory limits. It is systems evidence, not a physical-RAM requirement.
Benchmark details · Raw evidence · Methodology
MemVanta is also distributed on PyPI as a native Linux x86-64 CLI package. The initial PyPI package is a distribution shim for the C++ runtime; it is not a separate Python inference implementation or Python API.
python -m pip install memvanta
memvanta-real --helpThe package exposes these console commands:
memvanta
memvanta-real
memvanta-tokenize
memvanta-gguf-inspect
PyPI releases are built as manylinux wheels with MEMVANTA_NATIVE=OFF, installed and smoke-tested before publication, and published from GitHub Actions through PyPI Trusted Publishing rather than a long-lived API token.
The v0.8.3 pre-release includes a portable Linux x86-64 build (MEMVANTA_NATIVE=OFF) plus a SHA-256 checksum. Download, verify, and extract it with:
curl -fLO https://github.com/sauravsingla/MemVanta/releases/download/v0.8.3/memvanta-v0.8.3-linux-x86_64.tar.gz
curl -fLO https://github.com/sauravsingla/MemVanta/releases/download/v0.8.3/memvanta-v0.8.3-linux-x86_64.sha256
sha256sum -c memvanta-v0.8.3-linux-x86_64.sha256
tar -xzf memvanta-v0.8.3-linux-x86_64.tar.gz
cd memvanta-v0.8.3-linux-x86_64
./bin/memvanta_real --helpThen run trained-model text generation with a supported Llama-family GGUF model that you are licensed to use:
./bin/memvanta_real \
--model /path/to/model.gguf \
--prompt "Hello from MemVanta" \
--n 64 \
--threads 4 \
--ctx 2048 \
--temperature 0Build the project:
git clone https://github.com/sauravsingla/MemVanta.git
cd MemVanta
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build --output-on-failureThen run trained-model text generation with a supported Llama-family GGUF model that you are licensed to use:
./build/memvanta_real \
--model /path/to/model.gguf \
--prompt "Hello from MemVanta" \
--n 64 \
--threads 4 \
--ctx 2048 \
--temperature 0memvanta_real is the trained-model inference CLI. The separate memvanta run <file> command exercises mapped streaming/cache behavior and reports memory telemetry; it is not the text-generation command.
MemVanta's runtime is organized around explicit memory ownership and bounded data movement:
- mmap-backed GGUF access avoids requiring an unconditional full-model copy in a separate heap buffer.
- Bounded tensor slices and caching keep model access under explicit memory policy.
- Paged KV cache manages attention state with defined bounds.
- Q4/Q8 quantized CPU kernels provide compact execution paths for supported tensors.
- Byte-bounded adaptive prefetching can change look-ahead behavior without silently expanding the memory budget.
- Runtime CPU dispatch and AVX2/FMA paths improve hot paths while portability and correctness remain independently tested.
Low-memory inference guide · Architecture
Trained-model execution currently supports GGUF models with:
general.architecture=llama
The GGUF parser also validates pinned Qwen2 files, but Qwen2 inference is not implemented. Parser/container compatibility should not be interpreted as trained-model execution support.
The project currently has trained-model evidence up to 7B and remains an active research / engineering prototype rather than a drop-in replacement for a mature general-purpose inference runtime.
MemVanta's validation stack includes:
- Release and Debug correctness checks
- AddressSanitizer / UndefinedBehaviorSanitizer and ThreadSanitizer lanes
- parser limits and fuzz smoke
- deterministic trained-model checks
- x86 portability and runtime-dispatch validation
- AVX2/FMA paths
- ARM64 cross-build and QEMU validation
- repeated same-machine A/B memory and throughput measurements
Published benchmark methodology requires the identical GGUF artifact for both runtimes, a pinned comparison-runtime revision, matched workload parameters, warm-up plus repeated measured runs, and throughput reporting beside memory results.
Independent results that confirm, narrow, or contradict the current measurements are useful. Reproduction reports should include model hashes, runtime commits, machine metadata, commands, and raw outputs.
Reproduction guide · Memory benchmarking protocol
Outside systems contributors are welcome. Good first contributions include tooling that improves reproducibility, local build/smoke workflows, platform-validation documentation, and narrowly scoped runtime fixes with deterministic tests. Start with CONTRIBUTING.md or the open good first issue tasks.
If you reproduce MemVanta on different CPUs, compilers, RAM limits or model files, please share the full environment and raw outputs even when the result differs from the current measurements. Use GitHub Discussions for benchmark reproduction, hardware results, model compatibility and systems-design questions.
The first stable release is intentionally evidence-gated rather than date-gated; see release-readiness issue #56.