AI engineering glossary · Deep learning core
What is softmax?
Softmax (also called Normalised exponential): A function that turns a vector of raw scores into probabilities that are positive and sum to 1, by exponentiating each score and dividing by the total: p_i = e^(z_i) / sum of e^(z_j). It is the standard last step for multi-class classification.
Softmax: an example
Scores [2, 1, 0] become roughly [0.67, 0.24, 0.09].
Softmax in Python
A short example you can run (Python 3.11, NumPy). The output below is what it really prints.
import numpy as np
def softmax(z):
z = z - z.max() # subtract the max for numerical stability
e = np.exp(z)
return e / e.sum()
logits = np.array([2.0, 1.0, 0.1])
p = softmax(logits)
print(np.round(p, 3), "sum =", p.sum().round(6))Output
[0.659 0.242 0.099] sum = 1.0
Related terms
More Deep learning core terms
- Softplus
- Sparse categorical cross-entropy
- Spectral normalisation
- Stochastic depth
- Stochastic weight averaging
- Straight-through estimator
- SwiGLU
- Symmetry breaking
Where you learn it
Softmax is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.