Shrinking Models for Embedded Devices: Quantisation and Pruning
Models are usually trained in 32-bit floating point, and that precision is rarely needed to run them. Quantisation stores weights and activations as small integers, and pruning removes weights that contribute little. Together they let a model fit into the memory of an embedded processor and run on integer hardware such as many NPUs. This article explains both techniques, what they cost in accuracy and when each one actually speeds things up.
At a glance
- 4x
- Memory saving when float32 weights are stored as int8.
- PTQ
- Post-training quantisation, which converts a trained model using a small calibration set.
- QAT
- Quantisation-aware training, which simulates int8 during training to recover accuracy.
- Sparsity
- The fraction of weights set to zero by pruning.
The problem
A model that fits comfortably on a workstation may need several times the memory an embedded part has. Many accelerators also run only integer operations, so a floating-point model cannot use them at all.
Shrinking a model is not free. Some networks lose noticeable accuracy when quantised, and pruning individual weights saves no time unless the runtime and hardware can exploit the zeros.
Measure accuracy after every compression step, on the same test set, on the target runtime.
How int8 quantisation works
Each tensor is mapped to integers with a scale and a zero point, so that real value equals scale times the integer minus the zero point. Weight ranges are known after training; activation ranges are estimated by running a representative calibration set, so poor calibration data gives poor results.
Per-channel scales for weights usually preserve accuracy better than one scale per tensor. If post-training quantisation loses too much, quantisation-aware training usually recovers most of the gap. Sensitive layers, often the first and last, can also be left at higher precision where the runtime allows mixed precision.
What good looks like
Establish a baseline accuracy and latency for the float model on the agreed test set.
Apply post-training int8 quantisation with representative calibration data, then measure again on the target runtime.
If accuracy falls short, try quantisation-aware training, structured pruning or a smaller architecture, checking each change against the baseline.
| MB | |
|---|---|
| float32 | 20 |
| float16 | 10 |
| int8 | 5 |
| int4 | 2.5 |
| Unstructured | Structured | |
|---|---|---|
| Accuracy at a given sparsity | Usually better retained | Can fall faster |
| Speed-up on ordinary hardware | Little without sparse support | Direct |
| Memory saving | Needs a sparse storage format | Direct |
| Control over what is removed | Individual weights | Whole channels or filters |
What it changes
Smaller Footprint
Weights take a quarter of the memory in int8 compared with float32.
Faster Inference
Integer arithmetic suits NPUs and microcontroller instruction sets.
Lower Power
Less data moved from memory means less energy per inference.
Known Trade-Off
Accuracy loss is measured rather than assumed.
Where to go next
Knowledge distillation, training a small model to imitate a larger one, is another route to a compact model and combines well with quantisation.
Lower precisions such as int4 are increasingly supported, but support varies by runtime and hardware. Check before planning around them, and expect the accuracy cost to be larger than at int8.
Problems this prevents
- A model too large for on-chip memory
- Calibration data that misses real conditions
- Pruning that saves no inference time
- Accuracy checked on the desktop only
Discuss your project
Email support@aavico.com or use Get in Touch when the website finishes loading.