Why Temperature=0 Is Not 'Most Accurate'

Temperature 0 makes generation deterministic by always picking the highest-probability token. But most probable does not mean most correct. At temp=0 there is no way to detect low-confidence through sampling variation.

The support ticket reads: "We switched to deterministic mode and the answers got worse." The team had set temperature to 0 because they wanted accuracy. Consistency, they reasoned, means reliability. Reliability means correctness. The model was now giving the same wrong answer every single time — confidently, identically, on every run.

Temperature 0 is not a correctness setting. It is a determinism setting. The distinction matters more than almost any other parameter decision in production.

What temperature actually does is scale the raw logits before softmax. At temperature T, every logit is divided by T. As T approaches zero, the largest logit dominates completely — softmax produces a distribution concentrated almost entirely on the single highest-scoring token. At exactly temperature 0, the model always picks the argmax. Every run, same prompt, same token, no exceptions.

Here is the part that the "accuracy" interpretation gets wrong: the model's logit scores are not truth rankings. They are learned probability estimates over tokens given the training distribution. A token can have the highest logit and still be factually incorrect. The model has no access to ground truth. It has compressed patterns from training data into weights, and those weights can assign peak probability to a wrong answer — especially for specific facts, recent events, or domain details that were sparsely or inconsistently represented in training data.

Question: "What is the capital of Belgium?"

Model logits (before softmax):
  "Paris"    → 4.12   ← highest logit (learned from co-occurrence patterns)
  "Brussels" → 3.91   ← correct answer, second-highest
  "Antwerp"  → 2.87
  "Bruges"   → 1.44
  "Amsterdam"→ 1.02

Temperature = 0 → argmax → output: "Paris"  (wrong, every single run)

Temperature = 0.7 → sampling probabilities:
  "Paris"    → 0.47   sampled 47% of runs
  "Brussels" → 0.31   sampled 31% of runs  ← correct
  ...

Majority vote over 5 samples at temp=0.7 → "Brussels" wins

Temperature 0 also removes one of the most useful diagnostic signals in the system: variance. When a model's outputs vary significantly across runs at moderate temperature, that variance is information — it tells you the model is uncertain about this query. At temperature 0, every response looks equally confident regardless of whether the underlying logit distribution is sharp (the model is sure) or nearly flat (the model is guessing among several plausible tokens). You lose the ability to detect low-confidence outputs before they reach the user.

The team that filed the support ticket moved to temperature 0.3 with majority voting across five samples on high-stakes factual queries. Accuracy increased. Cost increased modestly. The model's uncertainty became visible and measurable rather than hidden behind false determinism. Temperature 0 is the right choice for evals (reproducibility matters for comparisons), structured output with rigid schemas, and code generation where token-level determinism often helps. It is not a correctness mechanism.