What is Schema Evolution

Definition

Schema evolution is the capability of a data system to handle changes to a dataset's structure over time, such as adding, removing, or altering fields, without breaking existing data or downstream consumers, allowing schemas to adapt as business requirements change.

« Back to Glossary Index

Lets data structures change over time without rewriting historical files
Protects downstream consumers from breaking when producers add or modify fields
Supports compatibility rules so old and new data versions coexist safely
Reduces costly coordinated migrations every time a field needs to change

Glossary: Avro

Real World Example

A streaming platform adds an optional 'device_type' field to its event schema; thanks to backward-compatible schema evolution, existing consumers keep working untouched while new ones read the added field.

FAQs

Why is schema evolution needed?

Business needs change, so datasets must add or modify fields over time without forcing every consumer and historical file to be rewritten.

What is backward versus forward compatibility?

Backward compatibility lets new readers handle old data, while forward compatibility lets old readers handle new data; both reduce breakage.

Which formats support schema evolution well?

Formats like Avro and Parquet, combined with a schema registry or table format such as Delta, handle evolution gracefully.