AI engineering glossary · LLM training alignment
What is next-token prediction?
Next-token prediction (also called Autoregressive language modelling, Next-word prediction): The training task in which a model reads a sequence of tokens and outputs a probability for every possible next token, and is penalised by cross-entropy when the true next token gets a low probability. Doing this over enormous text teaches the model language structure and world knowledge without any human labels.
Next-token prediction: an example
Given the tokens 'The capital of India is', the model should put high probability on ' New' and then ' Delhi'; a low probability on 'banana' produces a large loss.
Related terms
More LLM training alignment terms
- NormalFloat 4-bit
- Odds ratio preference optimisation
- Overtrained model
- Paged optimisers
- Parameter-efficient fine-tuning
- PEFT library
- Pipeline parallelism
- Policy (reinforcement learning)
Where you learn it
Next-token prediction is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.