What is Quantization?

How it works

Models are usually trained with 16-bit or 32-bit floating-point weights. Quantization maps those values onto a smaller set of levels. With 4 bits there are only 16 possible values per weight, so each block of weights also stores a scale factor that maps the small integers back to roughly the original range.

Simple rounding loses accuracy, so modern methods are smarter. GPTQ quantizes layer by layer and adjusts remaining weights to compensate for rounding errors. LLM.int8() keeps a few outlier features in higher precision. Formats you will meet when running models locally include GGUF (llama.cpp and Ollama, with levels like Q4_K_M and Q8_0), GPTQ, AWQ and MLX on Apple silicon. Some models are now trained or released in low precision such as FP8 or FP4 directly.

Why it matters

• Memory. A rough rule: weights need parameters times bits divided by 8 in bytes. A 70B model needs about 140 GB at 16-bit but about 35 GB at 4-bit, before the KV cache and overhead.
• Speed. Because inference is limited by how fast weights are read from memory, smaller weights usually mean more tokens per second.
• It makes local AI possible. Most people running an open-weight model on a laptop or a single GPU use a quantized version.
• There is a quality cost. 8-bit is generally close to the original; 4-bit is the common sweet spot; at 3 bits and below quality drops more noticeably, especially for maths and code. Test on your own tasks.

Example

You want a 32B model on a GPU with 24 GB of memory. At 16-bit it needs about 64 GB and will not fit. A 4-bit GGUF is about 18 to 20 GB, which fits with room for a modest context window, and in most everyday tasks the difference from the full model is hard to notice.