AI engineering glossary · LLM training alignment
What is double quantisation?
Double quantisation: A QLoRA memory-saving step that quantises the quantisation constants themselves, so the per-block scaling factors also use fewer bits. It saves a fraction of a bit per parameter, which adds up for very large models.
Double quantisation: an example
Instead of storing a 32-bit scale for each block of 64 weights, the scales are stored in 8 bits with a second-level constant.
Related terms
More LLM training alignment terms
Where you learn it
Double quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.