What is Idempotency

Definition

Idempotency is the property of an operation that produces the same result whether it is applied once or many times, ensuring that retries, duplicate messages, or reprocessing in a data pipeline do not create duplicate or inconsistent outcomes.

« Back to Glossary Index

Makes retries safe so a re-run never duplicates records or double-counts results
Allows pipelines to reprocess data confidently after failures or backfills
Protects against duplicate messages in at-least-once delivery systems
Simplifies recovery logic because re-running a step is always harmless

Real World Example

A payment pipeline uses an idempotency key on each transaction, so if a network timeout causes a charge request to be retried, the customer is billed exactly once rather than twice.

FAQs

Why is idempotency important in pipelines?

Because failures cause retries and duplicate messages, idempotency ensures reprocessing the same data does not corrupt results.

How is idempotency achieved?

Through techniques like unique keys, upserts, deduplication, and deterministic operations that can be safely repeated.

What is an idempotency key?

A unique identifier attached to a request so the receiving system recognises and ignores duplicate attempts of the same operation.