Information Gain Calculator
Calculate the information gain from a decision tree split, given parent and child node class counts.
Inputs
e.g. "30, 20" for 30 samples of class A and 20 of class B before the split.
Same class order as the parent.
Same class order as the parent.
Information Gain
0.256426bits
Parent Entropy
0.970951bits
Weighted Child Entropy
0.714525bits
Step by step
Parent entropy: H(parent)
H([30, 20])
= 0.970951 bits
Child entropies
H(child1), H(child2)
= 0.650022, 0.811278 bits
Weighted child entropy: Σ (|child_i| / |parent|) × H(child_i)
(30/50) × 0.6500 + (20/50) × 0.8113
= 0.714525 bits
Information gain: H(parent) − weighted child entropy
0.970951 − 0.714525
= 0.256426 bits
How it works
Information gain measures how much a split reduces uncertainty about the class label: IG = H(parent) − Σ (|child_i|/|parent|)·H(child_i), where H is Shannon entropy and the sum is weighted by each child node's share of the parent's samples. Decision tree algorithms like ID3 and C4.5 evaluate information gain for every candidate feature/split and choose the one that maximizes it — a higher information gain means the split produces child nodes that are more 'pure' (dominated by a single class) than the parent was.
Formula
IG = H(parent) - sum((|child_i| / |parent|) × H(child_i))
- H(parent)
- Entropy of the parent node
- H(child_i)
- Entropy of child node i
- |child_i|
- Number of samples in child i
- |parent|
- Number of samples in parent
Frequently Asked Questions
What does an information gain of 0 mean?
It means the split provided no reduction in entropy at all — the child nodes have the same class-mix uncertainty as the parent, so the feature used for that split carries no useful information for classification at that node.
Why is information gain weighted by child size?
A split that creates one large, pure child and one tiny, impure child is different from one that creates two equally impure medium-sized children — weighting by |child|/|parent| ensures that entropy reduction is measured proportionally to how many samples end up in each branch.
Is information gain biased toward features with many categories?
Yes — features with many distinct values can artificially produce high information gain by creating many small, pure child nodes; the Gain Ratio (information gain normalized by split entropy) is often used instead to correct for this bias, as done in the C4.5 algorithm.