Entropy Calculator
Calculate the Shannon entropy of a discrete probability distribution.
About Entropy Calculator
Shannon entropy measures the average uncertainty (in bits) of a random variable's outcome: H(X) = −Σ p(x)·log2(p(x)). A distribution concentrated on one outcome has entropy near 0 (very predictable), while a uniform distribution over n outcomes has maximum entropy log2(n) (maximally unpredictable) — a fair coin flip, for example, has entropy exactly 1 bit. In machine learning, entropy underlies decision tree splitting criteria, cross-entropy loss functions, and information-theoretic measures of model uncertainty.
Formula
H(X) = -sum(p(x) × log2(p(x)))
- p(x)
- Probability of outcome x
- log2
- Logarithm base 2 (result in bits)
Worked example
With the default values below, Entropy Calculator returns Shannon Entropy H(X) of 1.750000. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Probabilities (comma-separated)
- 0.5, 0.25, 0.125, 0.125
Results
- Shannon Entropy H(X)
- 1.750000 bits
- Normalized EntropyEntropy as a fraction of the maximum possible entropy for this many outcomes.
- 0.88%
- Maximum Entropy (Uniform)
- 2.0000 bits
Step by step
- 1Per-outcome terms: −p(x) × log2(p(x))= 0.5000, 0.5000, 0.3750, 0.3750
- 2H(X): sum of per-outcome terms= 1.750000 bits
- 3Maximum possible entropy (uniform distribution): log2(n)= 2.000000 bits
Assumptions and limits
- The result is derived only from the single input listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why use log base 2 instead of natural log for entropy?
Log base 2 gives entropy in units of 'bits,' which has an intuitive interpretation as the average number of yes/no questions needed to determine the outcome — natural log gives units of 'nats' instead, which is common in some ML loss functions but less intuitive for measuring information content.
What does zero entropy mean?
Zero entropy means the outcome is completely certain — one outcome has probability 1 and all others have probability 0, so there is no uncertainty and no information gained by observing the outcome.
How is entropy used in decision trees?
Decision tree algorithms like ID3 and C4.5 use entropy to measure how 'mixed' the class labels are in a node, and choose splits that reduce entropy the most (maximizing information gain) — see the Information Gain Calculator for that specific application.