AI engineering glossary · Inference serving optimisation
What is low-rank factorisation?
Low-rank factorisation: Replacing a large weight matrix with the product of two much thinner matrices that approximate it. This cuts the number of parameters and multiplications when the original matrix carries less independent information than its size suggests.
Low-rank factorisation: an example
A 4096 x 4096 matrix (16.8 million values) is approximated by 4096 x 64 and 64 x 4096 matrices, about 0.5 million values.
Related terms
More Inference serving optimisation terms
- Medusa decoding
- Memory fragmentation
- Memory-bound
- Model cascade
- Model compression
- Model serving
- NCCL
- nvidia-smi
Where you learn it
Low-rank factorisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.