AI engineering glossary · Data engineering
What is compaction?
Compaction: Rewriting many small files or fragments of a table into fewer, larger ones, and dropping obsolete data. It improves read speed and cuts metadata load, and table formats provide commands to run it.
Compaction: an example
A nightly job merges thousands of small Parquet files in a partition into a few files of about 256 MB each.
Related terms
More Data engineering terms
Where you learn it
Compaction is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.