Where the VRAM goes
Running a language model locally, three things compete for your GPU's memory: the model weights, the KV cache, and overhead. Getting the estimate right is the difference between a model that runs and one that crashes on load or falls back to painfully slow CPU offloading.
Weights, and why quantization is nearly free capacity
The weights are the bulk of it: a model with N billion parameters at B bytes each needs N × B gigabytes. At full FP16 precision (2 bytes) a 7-billion-parameter model is 14 GB — too big for many consumer cards. Quantize it to 4 bits (0.5 bytes) and the same model is 3.5 GB, a 4× reduction that lets a 7B model run comfortably on an 8 GB card. The trade-off is a small, usually modest loss in output quality; 4-bit and 5-bit quantization are the standard sweet spot for local inference.
The KV cache: the quiet memory eater
Every token in your context window is cached so the model does not recompute it, and that cache grows linearly with context length. Its per-token cost is 2 (for keys and values) × layers × KV-heads × head-dimension × bytes. For Llama 3 8B at FP16 that is 128 KiB per token — about 1 GB at an 8k context, but 16 GB at 128k. Long-context runs can need more memory for the cache than for the weights, which surprises people who only budgeted for the model. Quantizing the KV cache to 8-bit halves it.
Why modern models are cheaper: GQA
Older models used as many KV heads as attention heads. Modern ones use grouped-query attention (GQA), sharing a small number of KV heads across many query heads — Llama 3 8B has 32 attention heads but only 8 KV heads. That cuts the KV cache severalfold, which is why a recent 8B model fits where an older one of the same size would not. The overhead factor on top covers activations, the CUDA context, and memory fragmentation.
Mixture-of-experts (MoE): total params, not active, set the VRAM
MoE models like Qwen3-235B-A22B or Qwen3-30B-A3B route each token through only a few of their experts — the "A22B" and "A3B" are the active parameter counts. That makes them fast, because only a fraction of the weights do work per token. But it does not automatically make them small: to run everything on the GPU, every expert has to be in VRAM, because any token might route to any of them. A 235B-A22B model then needs the full 235 billion parameters resident (117 GB at 4-bit), not the 22 billion active ones. Active params drive speed; total params drive memory. The calculator uses total params for the weight figure and shows the active count for context.
Expert offload: the trick that runs 235B on a 24 GB card
Here is where MoE gets interesting for local users. Because only a handful of
experts fire per token, you can keep the always-active weights — attention,
embeddings, the router, and the KV cache — on the GPU and park the bulky expert
FFNs in ordinary system RAM, streaming in only the experts each token needs.
This is what llama.cpp's expert-offload option and tools like
ktransformers do. The split is dramatic: of Qwen3-235B-A22B's 235B parameters,
about 227B are expert weights and only ~8B are the non-expert backbone. Offload
the experts and the GPU needs roughly 7 GB of VRAM plus the KV cache,
while the ~113 GB of experts (at 4-bit) live in system RAM. A 24 GB card
with 128 GB of RAM runs it. Tick "offload MoE experts to system RAM" above and
the calculator shows both figures and whether each fits. The cost is speed —
reading experts from DRAM each token is slower than from VRAM — but it is the
difference between running the model and not.
Where the estimate is a ceiling
The presets all use standard grouped-query attention, so the KV-cache figure is exact for them. Two newer attention styles make the real cache smaller than this formula: sliding-window attention (Gemma) caps most layers at a fixed window regardless of context length, and multi-head latent attention (DeepSeek) compresses the cache into a low-rank form. For those models, treat the KV number here as an upper bound.
Presets go stale as new models ship, so the fields take any model card's numbers directly. Related tools: the data transfer time calculator for how long the model download takes, and the unit converter for GB/GiB — note this page uses decimal GB throughout.