Local LLM Hardware Guide: How Much VRAM and RAM You Really Need

The memory a local LLM needs is easy to estimate: parameters (in billions) × bits per weight ÷ 8 = gigabytes of weights, plus the KV cache for your context, plus about a gigabyte of runtime overhead. At the common Q4_K_M quantization (about 4.9 bits per weight), that is roughly 0.6 GB per billion parameters: about 5 GB for an 8B model, 16.5 GB for 27B and 43 GB for 70B, before context. Whatever hardware you buy, memory capacity decides which models you can run; memory bandwidth decides how fast they generate.

Quick answer

Step 1: memory for the weights

Each parameter is stored with a certain number of bits. Full-precision checkpoints use 16 bits (2 bytes). Quantization stores them with fewer bits, trading a little quality for a lot of memory.

NVIDIA's inference guide gives the baseline: a 7B model at 16-bit needs "roughly 7B * sizeof(FP16) ~= 14 GB". llama.cpp publishes the effective bits per weight of its GGUF formats, measured on Llama 3.1 8B:

K-quants mix precisions across tensors, which is why Q4_K_M lands at 4.89 bits rather than exactly 4.

Weights-only sizing table

Using GB = parameters (billions) × bits per weight ÷ 8:

These match published numbers closely. llama.cpp lists Llama 3.1 70B at 43.1 GB in Q4_K_M, and Google's Gemma 4 docs give 17.5 GB for the 31B model at 4-bit and 69.9 GB at BF16. Real downloads run a little larger when a model includes a vision encoder or keeps some layers at higher precision.

Some models ship in other formats. OpenAI's gpt-oss quantizes its expert weights to MXFP4 at about 4.25 bits per parameter, which is why the 20B model is a 14 GB download.

Mixture of experts: load everything, compute a little

A mixture-of-experts model such as Gemma 4 26B A4B or Qwen3.6 35B-A3B activates only a few billion parameters per token, but all experts must be in memory. Google's docs note the 26B A4B needs memory close to a dense 26B model (14.4 GB at 4-bit). Use the total parameter count for memory and the active count as a rough guide to speed.

Step 2: memory for the KV cache

During generation, the model stores a key and a value vector for every token, in every layer, so it does not recompute them. That is the KV cache, and it grows with context length.

NVIDIA's formula, per token:

KV bytes per token = 2 × layers × (KV heads × head dimension) × bytes per value

The 2 is for keys and values. Most modern models use grouped-query attention, which means fewer KV heads than attention heads, and a much smaller cache. Worked examples at 16-bit (2 bytes), using each model's published config.json:

NVIDIA's Llama 2 7B example (sequence length 4096) works out to about 2 GB; the table doubles that for 8k. The Qwen2.5 32B row is the one to remember: a 32B model at Q4_K_M (about 19.6 GB) plus a 32k context (8 GiB) does not fit on a 24 GB card, even though the weights alone do.

Newer architectures reduce this further. Qwen3.8, for example, mixes Gated DeltaNet layers with standard attention layers, and linear-attention layers like DeltaNet keep a fixed-size state rather than a growing cache. Check the model's config before assuming the worst.

Ways to shrink the KV cache

• Use a shorter context when you do not need a long one. Ollama sets 4k tokens by default on GPUs under 24 GiB, 32k from 24 to 48 GiB and 256k above that. It recommends at least 64k for agents and coding, so budget for it.
• Quantize the cache. Ollama's OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the memory of f16, and q4_0 about a quarter, when flash attention is enabled.
• Check placement with ollama ps. A CPU/GPU split means something spilled out of VRAM.

Step 3: add overhead

Leave headroom for the runtime's compute buffers and for whatever your OS, display and other apps already use. On a Mac the model shares unified memory with macOS and everything else you have open, so a 64 GB Mac cannot give all 64 GB to a model.

Full example: Qwen3.8 27B is an 18 GB download on Ollama. On a 32 GB GPU or Mac that leaves well over 10 GB for a 64k context and overhead. On a 24 GB card the margin is about 6 GB, so keep the context modest or quantize the KV cache.

Choosing hardware

NVIDIA GeForce GPUs

NVIDIA's own spec comparison lists:

For LLMs, VRAM matters more than gaming performance. A 16 GB RTX 5060 Ti holds more model than a 12 GB RTX 5070, and the older 24 GB RTX 3090 and 4090 hold more than any current 16 GB card. Ollama supports NVIDIA cards with compute capability 5.0 or higher.

Two GPUs can split a model across their combined VRAM in llama.cpp and Ollama, and llama.cpp's CPU+GPU hybrid mode can run models larger than VRAM, at a big speed cost for the part on the CPU.

Apple Silicon Macs

Macs share one pool of unified memory between CPU and GPU, so the GPU can use most of the system RAM. That is the Mac's advantage: capacity. Apple's current Mac Studio specs:

A 128 GB Mac can load models no single consumer GPU can. LM Studio requires Apple Silicon and macOS 14 or newer, and recommends 16 GB or more. MLX builds of models, available in LM Studio and Ollama's library, are optimized for these chips.

NVIDIA DGX Spark and similar boxes

NVIDIA's DGX Spark has 128 GB of unified LPDDR5x memory at 273 GB/s, and NVIDIA says it handles models up to 200 billion parameters (100 billion on the 64 GB OEM configuration). It is aimed at developers who want large-model capacity on a desk, with CUDA.

AMD GPUs

Ollama supports AMD cards through ROCm v7 on Linux and a narrower set (mainly RX 7000 and PRO W7000 series) on Windows, plus Vulkan for broader coverage. llama.cpp supports AMD through HIP and Vulkan.

Capacity vs speed: what bandwidth means

To generate each token, the hardware has to read the active weights from memory. That makes generation speed roughly proportional to memory bandwidth divided by the bytes read per token. Two consequences, without inventing benchmark numbers:

• A model that fits in fast VRAM generates much faster than one that spills into system RAM. Avoid splits.
• MoE models generate faster than dense models of the same total size, because each token reads only the active experts.

Prompt processing (reading a long input) is compute-bound instead, which is where discrete GPUs pull ahead. If you want to see real speeds, measure on your own machine: same model, same quantization, same context, several runs.

Buying advice by budget of memory

• Already own a laptop with 16 GB? Start there. gpt-oss 20B and Gemma 4 12B run fine. See how to run an LLM locally.
• Building a desktop for local AI? Prioritize VRAM: 16 GB minimum, 24 to 32 GB to run the best 24B to 31B coding models with real context. Our best local LLM for coding guide lists them.
• Want the biggest models on one quiet box? A Mac with 64 to 128 GB or a 128 GB unified-memory system.
• Serving several users? Look at vLLM on Linux with NVIDIA or AMD GPUs; it is built for high-throughput serving.

Tools differ less than people think on the same hardware. If you are deciding between apps, see Ollama vs LM Studio.