AI engineering glossary · Deep learning core
What is weight tying?
Weight tying (also called Tied embeddings): Using one shared weight matrix for two roles in a network, most often the input embedding table and the output projection of a language model. It cuts the parameter count and can improve results.
Weight tying: an example
A model with a 50,000-token vocabulary and 512-dimensional embeddings saves about 25.6 million parameters by reusing the table.
Related terms
More Deep learning core terms
- Xavier initialisation
- Activation function
- Adafactor
- Artificial neural network
- Autoencoder
- Autograd
- Backpropagation
- Backpropagation through time
Where you learn it
Weight tying is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.