AI engineering glossary · Feature engineering data prep
What is duplicate removal?
Duplicate removal (also called Deduplication): Finding and dropping repeated records in a dataset, whether exact copies or near-duplicates that differ only slightly. Duplicates overweight some cases, inflate accuracy when copies land in both training and test sets and skew statistics.
Duplicate removal: an example
A customer table with two rows for the same email and phone is reduced to one row, using drop_duplicates in pandas.
Related terms
More Feature engineering data prep terms
- Embedded method
- Entity embedding
- Equal-frequency binning
- Equal-width binning
- Feature crossing
- Feature engineering
- Feature extraction
- Feature scaling
Where you learn it
Duplicate removal is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.