AI engineering glossary · Computing foundations
What is precision?
Precision (also called Numerical precision): In computing, the number of bits or digits used to represent a number, which sets how finely values can be told apart. Lower precision saves memory and time but adds rounding error.
Precision: an example
Storing a model's weights in 16-bit instead of 32-bit precision halves their memory use.
Related terms
More Computing foundations terms
- Process
- Process ID
- Public-key cryptography
- Race condition
- Random access memory
- Reference counting
- Register
- Reverse proxy
Where you learn it
Precision is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.