What is Data Pipeline

Definition

A data pipeline is an automated sequence of processing steps that moves data from one or more sources through transformation and validation stages to a destination, running on a schedule or in response to events so that data flows continuously without manual intervention.

« Back to Glossary Index

Replaces ad-hoc data copying with a repeatable, version-controlled flow that runs unattended
Encodes each transformation as a discrete step, making failures easy to isolate and re-run
Supports both scheduled and event-triggered execution to match the freshness each use case needs
Provides a single place to add validation, alerting, and lineage across the whole data journey

Real World Example

A subscription business runs a pipeline that pulls billing events hourly, joins them with usage logs, flags failed payments, and writes a churn-risk table that the retention team queries each morning.

FAQs

What are the core stages of a data pipeline?

Most pipelines include source extraction, transformation or enrichment, validation, and delivery to a target store or application.

What is the difference between a pipeline and ETL?

ETL is one type of pipeline focused on extract-transform-load; a pipeline is the broader concept covering any automated multi-step data flow, batch or streaming.

How are pipeline failures handled?

Well-built pipelines isolate steps so a failed stage can be retried or resumed without re-running the entire flow, with alerts notifying owners.