AI engineering glossary · Classical ML algorithms
What is information gain?
Information gain: The drop in disorder achieved by splitting a group on a feature, computed as the entropy before the split minus the weighted average entropy of the resulting parts. Tree algorithms such as ID3 pick the split with the largest gain.
Information gain: an example
Splitting 10 customers (5 buy, 5 don't) into two pure groups of 5 reduces entropy from 1 bit to 0, a gain of 1 bit.
Related terms
More Classical ML algorithms terms
- Isolation forest
- K-means
- k-nearest neighbours
- KD-tree
- Kernel PCA
- Kernel trick
- Laplace smoothing
- Lasso regression
Where you learn it
Information gain is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.