AI engineering glossary · Evals guardrails AI security
What is training data extraction?
Training data extraction (also called memorisation attack): An attack that prompts a language model to reproduce verbatim passages it memorised during training, such as personal details, code or copyrighted text. Larger models and repeated data are more prone to memorisation, so deduplication and filtering help.
Training data extraction: an example
By prompting a model with the start of a known email signature, an attacker gets it to output a real person's phone number.
Related terms
More Evals guardrails AI security terms
- TruthfulQA
- Unbounded consumption
- Vector and embedding weaknesses
- Verbosity bias
- Access control
- Adversarial example
- Adversarial robustness
- Adversarial suffix
Where you learn it
Training data extraction is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.