AI engineering glossary · Reinforcement learning
What is bellman equation?
Bellman equation (also called Bellman expectation equation): A recursive identity stating that the value of a situation equals the immediate expected score plus the discounted value of the next situation, averaged over choices and outcomes. Formula: V(s) = sum over a of pi(a given s) * sum over s' of P(s' given s,a) * [r + gamma * V(s')]. Many algorithms turn it into an update rule.
Bellman equation: an example
If a step gives reward 1 and the next state has value 10 with gamma = 0.9, the current state has value 1 + 0.9*10 = 10.
Related terms
More Reinforcement learning terms
- Bellman optimality equation
- Bootstrapping
- Conservative Q-learning
- Contextual bandit
- Continuing task
- Count-based exploration
- Credit assignment
- Curiosity-driven exploration
Where you learn it
Bellman equation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.