AI engineering glossary · Data engineering
What is HDFS?
HDFS (also called Hadoop Distributed File System): The Hadoop Distributed File System, which splits large files into blocks and replicates each across several machines so data survives hardware failures and can be processed where it is stored.
HDFS: an example
A 1 GB file is split into 128 MB blocks, each copied to three different nodes.
Related terms
More Data engineering terms
- Hive
- Idempotency
- Incremental load
- JSON Lines
- Kappa architecture
- Lambda architecture
- Late-arriving data
- Lazy evaluation
Where you learn it
HDFS is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.