Entropy Calculator
Calculate the Shannon entropy of a discrete probability distribution.
Inputs
Values are normalized to sum to 1 automatically.
Shannon Entropy H(X)
1.750000bits
Normalized Entropy
0.88%
Entropy as a fraction of the maximum possible entropy for this many outcomes.
Maximum Entropy (Uniform)
2.0000bits
Step by step
Per-outcome terms: −p(x) × log2(p(x))
−p × log2(p)
= 0.5000, 0.5000, 0.3750, 0.3750
H(X): sum of per-outcome terms
Σ [0.5000, 0.5000, 0.3750, 0.3750]
= 1.750000 bits
Maximum possible entropy (uniform distribution): log2(n)
log2(4)
= 2.000000 bits
How it works
Shannon entropy measures the average uncertainty (in bits) of a random variable's outcome: H(X) = −Σ p(x)·log2(p(x)). A distribution concentrated on one outcome has entropy near 0 (very predictable), while a uniform distribution over n outcomes has maximum entropy log2(n) (maximally unpredictable) — a fair coin flip, for example, has entropy exactly 1 bit. In machine learning, entropy underlies decision tree splitting criteria, cross-entropy loss functions, and information-theoretic measures of model uncertainty.
Formula
H(X) = -sum(p(x) × log2(p(x)))
- p(x)
- Probability of outcome x
- log2
- Logarithm base 2 (result in bits)
Frequently Asked Questions
Why use log base 2 instead of natural log for entropy?
Log base 2 gives entropy in units of 'bits,' which has an intuitive interpretation as the average number of yes/no questions needed to determine the outcome — natural log gives units of 'nats' instead, which is common in some ML loss functions but less intuitive for measuring information content.
What does zero entropy mean?
Zero entropy means the outcome is completely certain — one outcome has probability 1 and all others have probability 0, so there is no uncertainty and no information gained by observing the outcome.
How is entropy used in decision trees?
Decision tree algorithms like ID3 and C4.5 use entropy to measure how 'mixed' the class labels are in a node, and choose splits that reduce entropy the most (maximizing information gain) — see the Information Gain Calculator for that specific application.