AI engineering glossary · LLM training alignment
What is 8-bit optimiser?
8-bit optimiser (also called 8-bit Adam): An optimiser such as Adam that stores its running statistics in 8-bit precision instead of 32-bit, cutting the memory used for optimiser state by roughly three-quarters with little loss in final quality.
8-bit optimiser: an example
Adam keeps two extra values per parameter; storing them in 8 bits saves gigabytes on a multi-billion-parameter model.
Related terms
More LLM training alignment terms
- Adapter
- Advantage function
- Alignment
- Alignment tax
- Alpaca
- Base model
- Bradley-Terry model
- Causal language modelling
Where you learn it
8-bit optimiser is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.