AI engineering glossary · Data engineering
What is RDD (Resilient Distributed Dataset)?
Resilient Distributed Dataset (also called RDD): Spark's low-level collection of records split across machines, immutable and rebuilt from a recorded chain of operations if a partition is lost. It gives fault tolerance without replicating data, though DataFrames are usually preferred.
Resilient Distributed Dataset: an example
An RDD of log lines is filtered and mapped; if a node fails, Spark recomputes only the lost partitions.
Related terms
More Data engineering terms
- Row-oriented storage
- Schema evolution
- Schema on read
- Schema on write
- Sensor
- Shuffle
- Slowly changing dimension
- Small files problem
Where you learn it
Resilient Distributed Dataset is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.