AI engineering glossary · LLM training alignment
What is NF4 (normalFloat 4-bit)?
NormalFloat 4-bit (also called NF4): A 4-bit number format introduced with QLoRA whose levels are placed at quantiles of a normal distribution, matching how neural network weights are typically distributed. It represents weights more accurately than evenly spaced 4-bit integers.
NormalFloat 4-bit: an example
Sixteen NF4 levels are denser near zero, where most weights lie, and sparser at the extremes.
Related terms
More LLM training alignment terms
- Odds ratio preference optimisation
- Overtrained model
- Paged optimisers
- Parameter-efficient fine-tuning
- PEFT library
- Pipeline parallelism
- Policy (reinforcement learning)
- Policy gradient
Where you learn it
NormalFloat 4-bit is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.