AI engineering glossary · NLP embeddings
What is hierarchical softmax?
Hierarchical softmax: A way to compute output probabilities over a large vocabulary using a binary tree whose leaves are words, so a word's probability is a product of about log(V) binary decisions instead of a sum over V words. It was used to speed up Word2Vec training.
Hierarchical softmax: an example
With 1,000,000 words, about 20 binary decisions replace one million-way normalisation.
Related terms
More NLP embeddings terms
- In-batch negatives
- InfoNCE loss
- Intent classification
- Inverse document frequency
- Kneser-Ney smoothing
- Massive Text Embedding Benchmark
- Matryoshka representation learning
- Mean pooling
Where you learn it
Hierarchical softmax is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.