KL Divergence Calculator
Calculate the Kullback-Leibler (KL) divergence between two probability distributions P and Q.
Inputs
Reference ('true') distribution. Values are normalized to sum to 1 automatically.
Approximating distribution. Same length as P.
KL Divergence KL(P‖Q)
0.106440
Distribution Length
4
Step by step
Per-element terms: P(x) × ln(P(x) / Q(x))
P(x) × ln(P(x)/Q(x))
= 0.1880, 0.0547, -0.0446, -0.0916
KL(P‖Q): sum of per-element terms
Σ [0.1880, 0.0547, -0.0446, -0.0916]
= 0.106440
How it works
KL divergence quantifies how much information is lost when distribution Q is used to approximate the true distribution P: KL(P‖Q) = Σ P(x)·ln(P(x)/Q(x)). It is always non-negative, and equals exactly 0 only when P and Q are identical. Critically, KL divergence is asymmetric — KL(P‖Q) ≠ KL(Q‖P) in general — because it specifically measures the extra 'surprise' from encoding samples drawn from P using a code optimized for Q, not a symmetric distance between the two distributions.
Formula
KL(P||Q) = sum(P(x) × ln(P(x) / Q(x)))
- P(x)
- Probability of outcome x under the true distribution P
- Q(x)
- Probability of outcome x under the approximating distribution Q
Frequently Asked Questions
Why is KL divergence asymmetric?
KL(P‖Q) measures expected extra bits needed when P is the true distribution but you use a code/model built for Q — swapping which distribution is 'true' changes which mismatches get heavily penalized, since the p × ln(p/q) weighting is applied using P's probabilities, not Q's.
What does a KL divergence of 0 mean?
KL(P‖Q) = 0 if and only if P and Q are exactly the same distribution everywhere; any difference between them produces a strictly positive value.
Where is KL divergence used in machine learning?
It's central to variational autoencoders (as a regularization term pulling the latent distribution toward a prior), reinforcement learning (trust-region policy updates), and knowledge distillation (matching a student model's output distribution to a teacher's).