Cross Entropy Calculator
Calculate the cross-entropy loss between predicted and true probability distributions.
About Cross Entropy Calculator
Cross-entropy measures how well a predicted probability distribution q matches a true distribution p: H(p,q) = -Σ p(x) × log(q(x)). It's the standard loss function for classification tasks, penalizing confident-but-wrong predictions heavily due to the logarithm. Cross-entropy decomposes into H(p,q) = H(p) + KL(p‖q), where H(p) is the true distribution's own entropy and KL(p‖q) is the extra 'inefficiency' from using q instead of p — since H(p) is fixed for a given label, minimizing cross-entropy during training is equivalent to minimizing KL divergence.
Formula
H(p, q) = -sum(p(x) * ln(q(x)))
- p(x)
- True probability for class x
- q(x)
- Predicted probability for class x
Worked example
With the default values below, Cross Entropy Calculator returns Cross Entropy Loss of 0.356675. Change any input above and every figure here is recalculated in your browser.
Inputs used
- True Distribution p(x)
- 1, 0, 0
- Predicted Distribution q(x)
- 0.7, 0.2, 0.1
Results
- Cross Entropy Loss
- 0.356675 nats
- KL Divergence
- 0.356675 nats
- True Distribution Entropy
- 0.000000 nats
- Perplexity
- 1.4286
Step by step
- 1Cross entropy: H(p,q) = -Σ p(x) × ln(q(x))= 0.356675
- 2True distribution entropy: H(p) = -Σ p(x) × ln(p(x))= 0.000000
- 3KL divergence: H(p,q) − H(p)= 0.356675
Assumptions and limits
- The result is derived only from the 2 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why is cross-entropy the standard classification loss?
It directly measures the distance between predicted and true probability distributions in a way that's differentiable and penalizes confidently wrong predictions much more heavily than uncertain ones, providing strong gradient signal during training.
What does it mean if cross-entropy equals the true distribution's entropy?
That happens only when the predicted distribution q exactly matches the true distribution p, meaning KL divergence is zero and the predictions are perfect.
How is perplexity related to cross-entropy?
Perplexity is simply e raised to the cross-entropy loss (when using natural log); it's commonly used to evaluate language models because it has a more intuitive interpretation as an 'effective vocabulary size' of uncertainty.
What happens if a predicted probability is exactly 0?
log(0) is undefined (negative infinity), so in practice predictions are clipped to a small epsilon value to avoid numerical errors — this calculator applies that safeguard automatically.