AI engineering glossary · Data engineering
What is small files problem?
Small files problem: The slowdown and overhead caused by storing very many tiny files in a distributed file or object store, since each file adds metadata handling and open costs. Jobs are usually fixed by merging files into larger ones.
Small files problem: an example
A streaming job writing every minute produces 1,440 small files per day per partition; queries slow down until they are merged.
Related terms
More Data engineering terms
Where you learn it
Small files problem is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.