Measure the Kullback-Leibler divergence between two probability distributions.
KL divergence quantifies how much information is lost when distribution Q is used to approximate the true distribution P: KL(P‖Q) = Σ P(x)·ln(P(x)/Q(x)). It is always non-negative, and equals exactly 0 only when P and Q are identical. Critically, KL divergence is asymmetric — KL(P‖Q) ≠ KL(Q‖P) in general — because it specifically measures the extra 'surprise' from encoding samples drawn from P using a code optimized for Q, not a symmetric distance between the two distributions.
KL(P||Q) = sum(P(x) × ln(P(x) / Q(x)))
KL(P‖Q) measures expected extra bits needed when P is the true distribution but you use a code/model built for Q — swapping which distribution is 'true' changes which mismatches get heavily penalized, since the p × ln(p/q) weighting is applied using P's probabilities, not Q's.
KL(P‖Q) = 0 if and only if P and Q are exactly the same distribution everywhere; any difference between them produces a strictly positive value.
It's central to variational autoencoders (as a regularization term pulling the latent distribution toward a prior), reinforcement learning (trust-region policy updates), and knowledge distillation (matching a student model's output distribution to a teacher's).