AI engineering glossary · Deep learning core
What is log-sum-exp trick?
Log-sum-exp trick (also called LogSumExp): A numerical technique for computing log(sum of e^x_i) safely by first subtracting the largest x from every value. It avoids overflow from huge exponentials and underflow from tiny ones, and underlies stable softmax and cross-entropy implementations.
Log-sum-exp trick: an example
Computing softmax of [1000, 1001] directly overflows; subtracting 1001 first gives [-1, 0] and works fine.
Related terms
More Deep learning core terms
- Logits
- Loss scaling
- Lottery ticket hypothesis
- LSTM
- Maxout
- Mixed precision training
- Model parallelism
- Multi-task learning
Where you learn it
Log-sum-exp trick is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.