AI engineering glossary · Probability
What is KL divergence?
KL divergence (also called Kullback-Leibler divergence, Relative entropy): A measure of how one probability distribution P differs from a reference distribution Q, computed as the sum of P(x) x log(P(x) / Q(x)). It is zero only when they match, and it is not symmetric, so it is not a true distance.
KL divergence: an example
In training a variational model, KL divergence penalises the learned distribution for drifting from the prior.
Related terms
More Probability terms
- Law of large numbers
- Law of total expectation
- Law of total probability
- Likelihood
- Linearity of expectation
- Marginal distribution
- Marginal probability
- Markov chain
Where you learn it
KL divergence is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.