AI engineering glossary · LLM training alignment
What is value function?
Value function (also called Critic, Value model): In reinforcement learning, an estimate of the total future reward expected from a given state when following the current policy. In PPO-based RLHF a separate value network, called the critic, learns this so that advantages can be computed for each token.
Value function: an example
Halfway through a generated answer, the value function might predict a final reward of 0.4, and the actual reward of 0.7 then yields a positive surprise.
Related terms
- Advantage function
- Proximal policy optimisation
- Policy (reinforcement learning)
- Generalised advantage estimation
More LLM training alignment terms
- Warmup-stable-decay schedule
- Weak-to-strong generalisation
- ZeRO
- 3D parallelism
- 8-bit optimiser
- Adapter
- Advantage function
- Alignment
Where you learn it
Value function is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.