What is Quantization

Definition

Quantization is a model-compression technique that reduces the numerical precision of a model's weights and computations, for example from 32-bit to 8-bit or lower, shrinking memory use and speeding inference with minimal loss in accuracy.

« Back to Glossary Index

Shrinks model memory footprint by using lower-precision numbers
Speeds inference and reduces energy and hardware requirements
Enables large models to run on smaller or edge devices
Often preserves accuracy closely despite reduced precision

Real World Example

A team quantizes an LLM from 16-bit to 4-bit precision so it fits and runs on a single consumer GPU, dramatically cutting hardware cost while keeping output quality acceptable for their use case.

FAQs

What does quantization reduce?

It reduces the numerical precision of model weights and activations, lowering memory and compute needs.

Does quantization hurt accuracy?

It can cause slight accuracy loss, but modern techniques often keep degradation minimal relative to the efficiency gained.

How does quantization differ from distillation?

Quantization lowers numerical precision of the same model, while distillation trains a smaller separate model to mimic a larger one.