AI engineering glossary · Data engineering
What is orchestration?
Orchestration: Coordinating when and in what order pipeline tasks run, handling dependencies, schedules, retries, alerts and resources. It is separate from the tasks' own logic, which does the actual data work.
Orchestration: an example
An orchestrator starts the transformation only after all three source extracts finish and pages the team if one fails twice.
Related terms
More Data engineering terms
- Parquet
- Partitioning
- Predicate pushdown
- Protocol Buffers
- Publish-subscribe
- Resilient Distributed Dataset
- Row-oriented storage
- Schema evolution
Where you learn it
Orchestration is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.