Techment — Site Header

What is On-device Inference

Definition

On-device inference is the practice of running a trained AI model directly on a local device such as a phone, sensor, or embedded system rather than in the cloud, delivering predictions locally for lower latency, offline capability, and stronger data privacy.
« Back to Glossary Index
  • Delivers low-latency predictions without a round trip to the cloud
  • Works offline, independent of network connectivity
  • Keeps sensitive data on the device, improving privacy
  • Reduces cloud compute and bandwidth costs

Real World Example

A smartphone runs an on-device model to recognise faces in photos locally, so tagging works instantly and offline while the user's images never leave the phone for cloud processing.

FAQs

Why run inference on-device?

For low latency, offline operation, and privacy, since data and computation stay local rather than going to the cloud.

What are the constraints of on-device inference?

Limited memory, compute, and power require optimised, often compressed or quantized, models.

How are models made to fit devices?

Techniques like quantization, distillation, and pruning shrink models so they run efficiently on constrained hardware.

Hello popup window