AI engineering glossary · Classical ML algorithms
What is entropy?
Entropy (also called Shannon entropy): A measure of uncertainty or disorder in a set of outcomes: the negative sum of p * log2(p) over class proportions. It is 0 when everything is one class and 1 bit for a 50/50 split of two classes. Decision trees use it to judge how mixed a node is.
Entropy: an example
A node with 8 yes and 2 no has entropy = -(0.8*log2 0.8 + 0.2*log2 0.2), about 0.72 bits.
Related terms
More Classical ML algorithms terms
- Euclidean distance
- Expectation-maximisation
- Explained variance ratio
- Extremely randomised trees
- Feature importance
- Gaussian mixture model
- Gaussian naive Bayes
- Gaussian process
Where you learn it
Entropy is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.