AI engineering glossary · Data engineering
What is deduplication?
Deduplication (also called Dedupe): Identifying and removing repeated records that describe the same entity or event, using exact keys or fuzzy matching, and keeping one authoritative copy. It prevents double counting in reports and models.
Deduplication: an example
Two rows with the same order_id from a retried upload are collapsed into one, keeping the latest.
Related terms
More Data engineering terms
Where you learn it
Deduplication is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.