Running AI at the Edge: Microcontrollers, NPUs and Accelerators
Edge AI means running inference on the product itself rather than sending data to a server. It suits products that must respond quickly, work without a connection or keep sensor data private. The hardware ranges from a small microcontroller classifying vibration data to a module with a neural processing unit handling several camera streams. This article explains what each class of hardware suits and how to choose between them.
At a glance
- int8
- The usual numeric format for edge inference, a quarter of the size of float32 weights.
- NPU
- Dedicated hardware for the multiply-accumulate operations that dominate neural networks.
- Offline
- On-device inference keeps working when the network does not.
- Local data
- Raw images and audio can stay on the device instead of travelling to a server.
The problem
Teams often pick the processor first and the model second. A model developed on a desktop GPU can need far more memory and compute than the chosen part offers, and the gap is only found when the model is first ported.
Accelerators bring their own constraint. Each NPU supports a particular set of operators, and any layer outside that set falls back to the CPU, which can erase most of the expected speed-up.
Choose the model and the silicon together. Either one chosen alone is likely to be wrong.
Matching the model to the silicon
Microcontrollers run small networks for sensor, audio and simple vision tasks, usually through runtimes such as TensorFlow Lite for Microcontrollers and optimised kernel libraries like CMSIS-NN on Arm Cortex-M. Memory, not arithmetic, is usually the first limit.
Application processors with an NPU run larger vision models under Linux through the vendor toolchain. Check early that every operator in your model is supported, because coverage and tooling vary considerably between vendors.
What good looks like
Define the task, the input rate and the latency the product needs, then estimate model size and compute from a candidate architecture.
Shortlist hardware whose memory and operator support fit that model, and confirm the vendor toolchain converts it.
Benchmark the converted model on a development kit with real input data before the board design is committed.
| Platform | Memory available | Typical tasks |
|---|---|---|
| Microcontroller | Hundreds of kilobytes of RAM | Keyword spotting, vibration and sensor classification |
| Microcontroller with ML accelerator | Hundreds of kilobytes to a few megabytes | Small vision and audio models at low power |
| Application processor with NPU | External DRAM, typically hundreds of megabytes upward | Object detection and classification on camera streams |
| Separate accelerator module | Own memory plus a host over USB, M.2 or PCIe | Adding inference to an existing design |
| Embedded GPU module | Gigabytes of shared memory | Several cameras, larger models, mixed workloads |
| bytes | |
|---|---|
| float32 | 4 |
| float16 | 2 |
| int8 | 1 |
| int4 | 0.5 |
What it changes
Fast Response
Inference completes on the device, with no dependence on network latency.
Privacy
Sensitive data can be processed where it is captured.
Right-Sized Hardware
The processor fits the model, so the bill of materials carries no idle silicon.
Fewer Surprises
Operator support is checked before layout, not after.
Where to go next
Plan how models will be updated in the field. A new model is a firmware update and needs the same signing and rollback protection.
Measure power with inference running. Duty cycle and wake strategy often matter more than the accelerator itself.
Problems this prevents
- A model too large for the chosen part
- Layers that silently fall back to the CPU
- Hardware chosen before the model was sized
- Power measured without inference running
Discuss your project
Email support@aavico.com or use Get in Touch when the website finishes loading.