Why Softmax and Not Just Normalize: The Sharpening That Changes Everything

Regular normalization preserves relative differences linearly. Softmax exponentiates first, turning small score gaps into large probability gaps. A 3-point score lead becomes 20× the weight. That sharpening is why the model focuses.

A team builds a custom attention implementation, replaces softmax with a simpler normalize-by-sum, and ships it. The model generates fluent output. Then they start inspecting attention weights and notice something wrong: "the" and "and" are receiving nearly as much weight as the main noun in every sentence. The retrieval system built on top of this attention is noisy. Relevant terms do not stand out from filler. The bug is not in the data or the training loop — it is in what normalization actually does to a set of scores.

Both softmax and simple normalization take a set of raw scores and produce a distribution that sums to 1. That is where the similarity ends.

Simple normalization divides each score by the sum of all scores. A set of scores like [2.8, 0.8, 0.3] produces weights proportional to those values. The ratio between the highest and lowest score is preserved linearly. A score 2.8x larger than another gets 2.8x the weight. If scores are close together, weights are close together. The operation is honest but flat.

Softmax exponentiates each score before dividing. e^2.8 = 16.4, e^0.8 = 2.2, e^0.3 = 1.35. Now the ratio between highest and lowest is 16.4 / 1.35 = 12.1, not 2.8 / 0.3 = 9.3. The exponentiation amplifies score differences. A token that scores 2.5 points higher than another does not get 2.5x the weight — it gets roughly e^2.5 = 12.2x the weight. Small score differences become large probability gaps.

Raw scores for 5 tokens:
  "surgeon"  2.8
  "the"      0.8
  "quickly"  0.3
  "a"        0.2
  "agreed"  -0.5

Simple normalize (divide by sum = 3.6, negatives clamped to 0):
  "surgeon"  0.622
  "the"      0.178
  "quickly"  0.133
  "a"        0.044
  "agreed"   0.000
  → 3-point lead = 4.7x weight

Softmax (e^x / sum of e^x):
  "surgeon"  0.891
  "the"      0.060
  "quickly"  0.037
  "a"        0.033
  "agreed"   0.016
  → 3-point lead = 14.8x weight

Same scores. Same dominant token. Completely different concentration.

The consequence for attention is not subtle. In a language model, most tokens are not relevant to a given query position. The word "agreed" should attend to "surgeon" with high weight and ignore articles almost entirely. Simple normalization spreads probability mass across the whole sequence, because it treats score differences as linear. The result is diluted attention — the model integrates information from tokens that are essentially noise for this particular computation.

Softmax sharpens the distribution precisely because the exponential function is convex. Tokens with scores above the mean get amplified; tokens with scores below the mean get suppressed. The model learns to produce score differences that, after exponentiation, create the concentration it needs. This is also why temperature works the way it does: dividing all logits by T before softmax stretches or compresses the score gaps, controlling how sharp the final distribution becomes.

Normalize-by-sum is the right operation when you want a fair weighted average. Softmax is the right operation when you want a learnable mechanism to express confident focus — which is exactly what attention is.