AI engineering glossary · Deep learning core
What is gradient checkpointing?
Gradient checkpointing (also called Activation checkpointing): A memory-saving technique that stores only some layer activations during the forward pass and recomputes the rest when needed in the backward pass. It trades extra computation for a large reduction in memory use.
Gradient checkpointing: an example
Recomputing activations may add roughly a third to step time but lets a much deeper model fit on one GPU.
Related terms
More Deep learning core terms
Where you learn it
Gradient checkpointing is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.