What is Partitioning

Definition

Partitioning is the technique of dividing a large dataset or table into smaller, independent segments based on a key such as date or region, so queries can scan only the relevant partitions and writes can target specific segments, improving performance and manageability.

« Back to Glossary Index

Cuts query cost and time by scanning only the partitions a query actually needs
Enables efficient pruning so filters on the partition key skip irrelevant data
Makes large tables manageable by isolating data into independently maintainable segments
Improves parallelism since partitions can be processed concurrently

Glossary: Lakehouse Optimization

Real World Example

An ad-tech company partitions its event table by date, so a query for the last seven days reads only seven partitions instead of scanning years of data, cutting both runtime and cloud query cost dramatically.

FAQs

How does partitioning speed up queries?

When a query filters on the partition key, the engine skips, or prunes, partitions that cannot match, scanning far less data.

What makes a good partition key?

A column frequently used in filters with moderate cardinality, such as date, works well; very high-cardinality keys create too many small partitions.

What is over-partitioning?

Splitting data into too many tiny partitions, which creates excessive small files and metadata overhead that can hurt performance.