AI engineering glossary · Data engineering
What is partitioning?
Partitioning: Splitting a large dataset into separate pieces by the value of a chosen column, commonly date, so each piece is stored and processed independently. Queries that filter on that column read only the relevant pieces.
Partitioning: an example
Sales are stored in folders like date=2026-09-01/, so a query for one week touches seven folders, not years of data.
Related terms
More Data engineering terms
- Predicate pushdown
- Protocol Buffers
- Publish-subscribe
- Resilient Distributed Dataset
- Row-oriented storage
- Schema evolution
- Schema on read
- Schema on write
Where you learn it
Partitioning is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.