AI engineering glossary · Data engineering
What is Apache Spark?
Apache Spark (also called Spark, PySpark): An open-source engine that processes large datasets in parallel across a cluster, keeping intermediate data in memory where possible. It offers APIs in Python, SQL, Scala and Java for batch, SQL and streaming work.
Apache Spark: an example
A PySpark job groups 2 billion transaction rows by merchant across a cluster in minutes.
Related terms
More Data engineering terms
Where you learn it
Apache Spark is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.