What is Micro-batch

Definition

Micro-batch processing handles data in small, frequent batches at short intervals, combining the simplicity of batch processing with much lower latency, so it approximates streaming while retaining batch's easier exactly-once semantics and operational model.

« Back to Glossary Index

Delivers near-real-time results without full streaming complexity
Retains batch’s simpler recovery and exactly-once processing semantics
Tunes the interval to balance latency against throughput and cost
Reuses familiar batch tooling while cutting latency to seconds

Real World Example

A dashboard pipeline uses Spark Structured Streaming in micro-batch mode, processing new events every ten seconds, which keeps the dashboard nearly live while avoiding the operational overhead of true per-event streaming.

FAQs

How does micro-batch differ from true streaming?

Micro-batch groups events into small frequent batches, while true streaming processes each event individually, giving slightly higher latency but simpler semantics.

Why choose micro-batch?

It offers near-real-time latency with batch's easier recovery and exactly-once guarantees, a practical middle ground.

What is the trade-off?

Latency is bounded by the batch interval, so micro-batch cannot match the millisecond response of pure event streaming.