Calculate the Shannon entropy of a discrete probability distribution.
Shannon entropy measures the average uncertainty (in bits) of a random variable's outcome: H(X) = −Σ p(x)·log2(p(x)). A distribution concentrated on one outcome has entropy near 0 (very predictable), while a uniform distribution over n outcomes has maximum entropy log2(n) (maximally unpredictable) — a fair coin flip, for example, has entropy exactly 1 bit. In machine learning, entropy underlies decision tree splitting criteria, cross-entropy loss functions, and information-theoretic measures of model uncertainty.
H(X) = -sum(p(x) × log2(p(x)))
Log base 2 gives entropy in units of 'bits,' which has an intuitive interpretation as the average number of yes/no questions needed to determine the outcome — natural log gives units of 'nats' instead, which is common in some ML loss functions but less intuitive for measuring information content.
Zero entropy means the outcome is completely certain — one outcome has probability 1 and all others have probability 0, so there is no uncertainty and no information gained by observing the outcome.
Decision tree algorithms like ID3 and C4.5 use entropy to measure how 'mixed' the class labels are in a node, and choose splits that reduce entropy the most (maximizing information gain) — see the Information Gain Calculator for that specific application.