What is Inference?

How it works

Training adjusts a model's weights; inference only runs them. For an LLM, inference has two phases:

1. Prefill. The whole prompt is processed in parallel, and the model stores intermediate results for every token in a KV cache. This phase sets the "time to first token".
2. Decode. The model generates one token at a time, each step reading the KV cache and all its weights. This phase sets "tokens per second".

Decoding is usually limited by memory bandwidth rather than raw compute, because the full set of active weights must be read for every token. That is why memory size and speed dominate local AI hardware choices, and why mixture-of-experts models, which read only some weights per token, can generate quickly.

Serving engines such as vLLM, llama.cpp and Ollama add batching (serving many users at once), KV-cache management (vLLM's PagedAttention), quantization and speculative decoding to raise throughput.

Why it matters

• It is where the running cost is. A model is trained once but answers millions of requests. API prices per million tokens are inference prices.
• Latency shapes the product. Voice assistants and coding tools need a fast first token and steady output speed.
• Reasoning models spend more at inference. They generate long internal reasoning before answering, trading compute at run time for better answers, sometimes called test-time compute.
• It decides where AI can run: in a cloud data centre, on a home server or on a laptop.

Example

Running an 8B-parameter model at 4-bit on a laptop, the prompt "Draft a polite late-payment reminder" is prefilled almost instantly, then the reply streams out token by token. Swap to a much larger model on the same machine and the first token comes later and the stream slows, because each token requires reading far more weights from memory.