What is Compaction

Definition

Compaction is the maintenance process of merging many small data files into fewer, larger files in a table or storage system, reducing file-count overhead and metadata bloat so that queries run faster and storage is used more efficiently.

« Back to Glossary Index

Merges the many small files that streaming and frequent writes create into efficient large ones
Reduces query overhead caused by opening and listing thousands of tiny files
Lowers metadata pressure on the table format and catalog
Improves compression ratios by rewriting data into larger, better-organised blocks

Real World Example

A clickstream pipeline that writes thousands of tiny files per hour runs nightly compaction on its Delta tables, collapsing them into a few large files and cutting morning dashboard query times by more than half.

FAQs

Why do small files hurt performance?

Each file carries open, list, and metadata overhead, so thousands of tiny files force engines to do far more work than reading a few large ones.

What causes the small-file problem?

Streaming ingestion, micro-batches, and frequent small writes all produce many small files that accumulate over time.

Is compaction automatic?

Some lakehouse platforms offer auto-compaction or optimize commands, while others require scheduled maintenance jobs.