What is Workflow Orchestration (Airflow)

Definition

Workflow orchestration with Apache Airflow is the practice of defining data workflows as code in the form of directed acyclic graphs (DAGs), where Airflow schedules tasks, manages their dependencies, retries failures, and provides visibility into every run from a central web interface.

« Back to Glossary Index

Defines pipelines as version-controlled Python code rather than fragile cron scripts
Manages task dependencies through DAGs so steps always run in the right order
Provides built-in scheduling, retries, backfills, and alerting out of the box
Offers a rich UI to monitor run history, logs, and failures across all workflows

Real World Example

A data team defines an Airflow DAG that ingests source data, runs dbt models, and refreshes a dashboard each morning, with automatic retries on transient failures and email alerts when a task ultimately fails.

FAQs

What is a DAG in Airflow?

A DAG is a directed acyclic graph that defines tasks and their dependencies, ensuring each task runs only after its prerequisites complete.

Why use Airflow over cron?

Airflow adds dependency management, retries, backfills, logging, and a monitoring UI that simple cron jobs lack.

Is Airflow only for data pipelines?

It is most common for data workflows but can orchestrate any sequence of dependent tasks, including ML training and infrastructure jobs.