Cross Entropy Calculator
Calculate the cross-entropy loss between a true probability distribution and a predicted one.
Inputs
Comma-separated probabilities that sum to 1, e.g. one-hot label 1, 0, 0
Comma-separated predicted probabilities, e.g. softmax output 0.7, 0.2, 0.1
Cross Entropy Loss
0.356675nats
KL Divergence
0.356675nats
True Distribution Entropy
0.000000nats
Perplexity
1.4286
Step by step
Cross entropy: H(p,q) = -Σ p(x) × ln(q(x))
-(1 × ln(0.7)) + -(0 × ln(0.2)) + -(0 × ln(0.1))
= 0.356675
True distribution entropy: H(p) = -Σ p(x) × ln(p(x))
H(p)
= 0.000000
KL divergence: H(p,q) − H(p)
0.356675 − 0.000000
= 0.356675
How it works
Cross-entropy measures how well a predicted probability distribution q matches a true distribution p: H(p,q) = -Σ p(x) × log(q(x)). It's the standard loss function for classification tasks, penalizing confident-but-wrong predictions heavily due to the logarithm. Cross-entropy decomposes into H(p,q) = H(p) + KL(p‖q), where H(p) is the true distribution's own entropy and KL(p‖q) is the extra 'inefficiency' from using q instead of p — since H(p) is fixed for a given label, minimizing cross-entropy during training is equivalent to minimizing KL divergence.
Formula
H(p, q) = -sum(p(x) * ln(q(x)))
- p(x)
- True probability for class x
- q(x)
- Predicted probability for class x
Frequently Asked Questions
Why is cross-entropy the standard classification loss?
It directly measures the distance between predicted and true probability distributions in a way that's differentiable and penalizes confidently wrong predictions much more heavily than uncertain ones, providing strong gradient signal during training.
What does it mean if cross-entropy equals the true distribution's entropy?
That happens only when the predicted distribution q exactly matches the true distribution p, meaning KL divergence is zero and the predictions are perfect.
How is perplexity related to cross-entropy?
Perplexity is simply e raised to the cross-entropy loss (when using natural log); it's commonly used to evaluate language models because it has a more intuitive interpretation as an 'effective vocabulary size' of uncertainty.
What happens if a predicted probability is exactly 0?
log(0) is undefined (negative infinity), so in practice predictions are clipped to a small epsilon value to avoid numerical errors — this calculator applies that safeguard automatically.