AI engineering glossary · Data engineering
What is shuffle?
Shuffle: The redistribution of data across machines so that records with the same key end up together, needed for joins, groupings and sorts. It involves disk and network transfer and is often the most expensive part of a distributed job.
Shuffle: an example
Grouping sales by city in Spark moves rows across the network so every row for Mumbai reaches the same worker.
Related terms
More Data engineering terms
- Slowly changing dimension
- Small files problem
- Snowflake
- Snowflake schema
- Star schema
- Stream processing
- Surrogate key
- Time travel
Where you learn it
Shuffle is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.