Shrinking Models for Embedded Devices: Quantisation and Pruning

Models are usually trained in 32-bit floating point, and that precision is rarely needed to run them. Quantisation stores weights and activations as small integers, and pruning removes weights that contribute little. Together they let a model fit into the memory of an embedded processor and run on integer hardware such as many NPUs. This article explains both techniques, what they cost in accuracy and when each one actually speeds things up.

At a glance

4x
Memory saving when float32 weights are stored as int8.
PTQ
Post-training quantisation, which converts a trained model using a small calibration set.
QAT
Quantisation-aware training, which simulates int8 during training to recover accuracy.
Sparsity
The fraction of weights set to zero by pruning.

The problem

A model that fits comfortably on a workstation may need several times the memory an embedded part has. Many accelerators also run only integer operations, so a floating-point model cannot use them at all.

Shrinking a model is not free. Some networks lose noticeable accuracy when quantised, and pruning individual weights saves no time unless the runtime and hardware can exploit the zeros.

Measure accuracy after every compression step, on the same test set, on the target runtime.

How int8 quantisation works

Each tensor is mapped to integers with a scale and a zero point, so that real value equals scale times the integer minus the zero point. Weight ranges are known after training; activation ranges are estimated by running a representative calibration set, so poor calibration data gives poor results.

Per-channel scales for weights usually preserve accuracy better than one scale per tensor. If post-training quantisation loses too much, quantisation-aware training usually recovers most of the gap. Sensitive layers, often the first and last, can also be left at higher precision where the runtime allows mixed precision.

What good looks like

Establish a baseline accuracy and latency for the float model on the agreed test set.

Apply post-training int8 quantisation with representative calibration data, then measure again on the target runtime.

If accuracy falls short, try quantisation-aware training, structured pruning or a smaller architecture, checking each change against the baseline.

Weight storage for a 5 million parameter model
MB
float3220
float1610
int85
int42.5
Weights only. Activations, runtime and code need memory as well.
Unstructured and structured pruning
UnstructuredStructured
Accuracy at a given sparsityUsually better retainedCan fall faster
Speed-up on ordinary hardwareLittle without sparse supportDirect
Memory savingNeeds a sparse storage formatDirect
Control over what is removedIndividual weightsWhole channels or filters
Structured pruning removes whole channels, so the result is simply a smaller dense model.

What it changes

Smaller Footprint

Weights take a quarter of the memory in int8 compared with float32.

Faster Inference

Integer arithmetic suits NPUs and microcontroller instruction sets.

Lower Power

Less data moved from memory means less energy per inference.

Known Trade-Off

Accuracy loss is measured rather than assumed.

Where to go next

Knowledge distillation, training a small model to imitate a larger one, is another route to a compact model and combines well with quantisation.

Lower precisions such as int4 are increasingly supported, but support varies by runtime and hardware. Check before planning around them, and expect the accuracy cost to be larger than at int8.

Problems this prevents

  • A model too large for on-chip memory
  • Calibration data that misses real conditions
  • Pruning that saves no inference time
  • Accuracy checked on the desktop only

Discuss your project

Email support@aavico.com or use Get in Touch when the website finishes loading.