Convert a vector of logits into a probability distribution using the softmax function.
Softmax converts a vector of arbitrary real-valued logits into a probability distribution that sums to 1: softmax(x_i) = e^(x_i) / Σ e^(x_j). It's the standard final activation for multi-class classification and the mechanism behind attention weights in transformers. The temperature parameter scales logits before exponentiation — lower temperature sharpens the distribution toward the largest logit (more confident/deterministic), while higher temperature flattens it toward uniform (more random), which is exactly how temperature sampling works in LLM text generation.
softmax(x_i) = e^(x_i / T) / sum(e^(x_j / T))
This is a standard numerical stability trick: subtracting the max logit before exponentiating prevents overflow from very large exponentials, and it doesn't change the final result because it cancels out in the normalization.
Temperature below 1 makes the distribution more peaked (closer to a one-hot/argmax choice), while temperature above 1 flattens it toward uniform, increasing randomness — this is the same temperature parameter used in LLM sampling.
Entropy measures how uncertain or spread out the resulting probability distribution is; low entropy means the model is confident in one class, while high entropy means probabilities are more evenly spread across classes.
Softmax converts logits to probabilities, and cross-entropy loss then measures how far those predicted probabilities are from the true label distribution — together they form the standard classification loss (softmax + cross-entropy).