What is Data Ingestion

Definition

Data ingestion is the entry point of a data platform: the act of collecting data from source systems such as databases, APIs, files, and event streams and bringing it into a storage or processing environment, either in scheduled batches or as a continuous real-time feed.

« Back to Glossary Index

Connects disparate source systems to the platform through reusable, configurable connectors
Supports both batch and streaming intake so each source arrives at the cadence it requires
Captures source data in its original form, preserving fidelity for later reprocessing
Centralises intake monitoring so missing or delayed feeds are detected quickly

Real World Example

A media streaming service ingests clickstream events from millions of devices in real time through Kafka while batch-loading nightly content-catalog updates from a partner SFTP server.

FAQs

What is the difference between batch and streaming ingestion?

Batch ingestion collects data in scheduled chunks, while streaming ingestion captures records continuously as events occur for near-real-time availability.

Where does ingested data usually land?

It typically lands in a raw or staging zone of a data lake, lakehouse, or warehouse before any transformation is applied.

What makes ingestion reliable?

Idempotent intake, schema checks at the boundary, and monitoring for late or missing feeds are key to dependable ingestion.