What is Distributed Training

Definition

Distributed training is the practice of training a machine-learning model across multiple machines or accelerators in parallel, splitting the data or the model itself so that very large models and datasets can be trained in feasible time that a single device could not handle.

« Back to Glossary Index

Trains very large models and datasets in feasible time
Scales beyond the memory and compute of a single device
Speeds training through parallel data or model processing
Enables the largest modern models that no single GPU could train

Real World Example

A lab trains a large language model using distributed training across hundreds of GPUs, splitting both data and model so the training completes in days rather than being impossible on any single machine.

FAQs

What are data and model parallelism?

Data parallelism splits the training data across devices with model copies, while model parallelism splits the model itself across devices.

Why is distributed training necessary?

The largest models and datasets exceed any single device's memory and time limits, so training must be spread across many.

What are the challenges?

Coordinating devices, synchronising updates, and managing communication overhead efficiently are key challenges.