KL Divergence Calculator
Measure the Kullback-Leibler divergence between two probability distributions.
About KL Divergence Calculator
KL divergence quantifies how much information is lost when distribution Q is used to approximate the true distribution P: KL(P‖Q) = Σ P(x)·ln(P(x)/Q(x)). It is always non-negative, and equals exactly 0 only when P and Q are identical. Critically, KL divergence is asymmetric — KL(P‖Q) ≠ KL(Q‖P) in general — because it specifically measures the extra 'surprise' from encoding samples drawn from P using a code optimized for Q, not a symmetric distance between the two distributions.
Formula
KL(P||Q) = sum(P(x) × ln(P(x) / Q(x)))
- P(x)
- Probability of outcome x under the true distribution P
- Q(x)
- Probability of outcome x under the approximating distribution Q
Worked example
With the default values below, KL Divergence Calculator returns KL Divergence KL(P‖Q) of 0.106440. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Distribution P (comma-separated)
- 0.4, 0.3, 0.2, 0.1
- Distribution Q (comma-separated)
- 0.25, 0.25, 0.25, 0.25
Results
- KL Divergence KL(P‖Q)
- 0.106440
- Distribution Length
- 4
Step by step
- 1Per-element terms: P(x) × ln(P(x) / Q(x))= 0.1880, 0.0547, -0.0446, -0.0916
- 2KL(P‖Q): sum of per-element terms= 0.106440
Assumptions and limits
- The result is derived only from the 2 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why is KL divergence asymmetric?
KL(P‖Q) measures expected extra bits needed when P is the true distribution but you use a code/model built for Q — swapping which distribution is 'true' changes which mismatches get heavily penalized, since the p × ln(p/q) weighting is applied using P's probabilities, not Q's.
What does a KL divergence of 0 mean?
KL(P‖Q) = 0 if and only if P and Q are exactly the same distribution everywhere; any difference between them produces a strictly positive value.
Where is KL divergence used in machine learning?
It's central to variational autoencoders (as a regularization term pulling the latent distribution toward a prior), reinforcement learning (trust-region policy updates), and knowledge distillation (matching a student model's output distribution to a teacher's).