AI engineering glossary · Deep learning core
What is softplus?
Softplus: A smooth approximation of ReLU defined as ln(1 + e^x). It is always positive and differentiable everywhere, which makes it useful when a network must output a strictly positive quantity such as a variance or a rate.
Softplus: an example
softplus(0) = ln 2, about 0.69; softplus(10) is almost exactly 10.
Related terms
More Deep learning core terms
- Sparse categorical cross-entropy
- Spectral normalisation
- Stochastic depth
- Stochastic weight averaging
- Straight-through estimator
- SwiGLU
- Symmetry breaking
- Tanh
Where you learn it
Softplus is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.