AI engineering glossary · Feature engineering data prep
What is text preprocessing?
Text preprocessing: The cleaning steps applied to raw text before modelling, such as lowercasing, removing punctuation and stop words, tokenising and stemming or lemmatising. It standardises text so equal words are treated equally.
Text preprocessing: an example
'Great Food!!' becomes ['great', 'food'] after lowercasing, stripping punctuation and tokenising.
Related terms
More Feature engineering data prep terms
- Threshold tuning
- Tomek links
- Training-serving skew
- Undersampling
- Unit vector scaling
- Unknown category handling
- Variance threshold
- Weight of evidence
Where you learn it
Text preprocessing is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.