AI engineering glossary · Evals guardrails AI security
What is toxicity classification?
Toxicity classification: Automatically labelling text as rude, hateful, threatening or otherwise abusive, usually with a model that outputs a probability for each category. It powers moderation filters and safety evaluations, but can wrongly flag reclaimed language or dialects, so thresholds need care.
Toxicity classification: an example
A classifier gives a comment a 0.92 toxicity score, so it is hidden pending review, while a 0.10 comment passes.
Related terms
More Evals guardrails AI security terms
- Training data extraction
- TruthfulQA
- Unbounded consumption
- Vector and embedding weaknesses
- Verbosity bias
- Access control
- Adversarial example
- Adversarial robustness
Where you learn it
Toxicity classification is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.