What is Temperature?

How it works

At each step a language model produces a score (a logit) for every token in its vocabulary. These scores are turned into probabilities with a softmax, and one token is sampled. Temperature divides the scores before the softmax:

• Temperature below 1 sharpens the distribution, so the top choices become even more likely.
• Temperature of 1 leaves the model's probabilities as they are.
• Temperature above 1 flattens the distribution, so less likely tokens are picked more often.
• Temperature of 0 (or close to it) in most APIs means almost always taking the single most likely token, often called greedy decoding.

Temperature is usually combined with top-p (nucleus sampling, from the Holtzman et al. paper), which samples only from the smallest set of tokens whose probabilities add up to p, and sometimes top-k. Allowed ranges and defaults differ by provider, so check the API reference of the model you use.

Why it matters

• Extraction, classification, code and factual Q&A usually work best with low temperature, because you want the same correct answer every time.
• Brainstorming, fiction and marketing copy benefit from higher temperature for variety.
• Too high produces rambling or incoherent text.

Two misunderstandings are common. Low temperature does not make a model factual: it makes it consistent, and a model can be consistently wrong (hallucination is a knowledge problem, not a sampling one). And temperature 0 is not always perfectly deterministic, because batching and floating-point effects on servers can still change outputs slightly. Some reasoning models also fix or ignore sampling settings.

Example

You ask a model for "a name for a coffee shop" five times. At temperature 0.2 you might get almost the same name each time. At 1.0 you get five different names. For a script that tags support emails as "billing", "bug" or "other", you would set temperature near 0 so the same email always gets the same label.