Running AI at the Edge: Microcontrollers, NPUs and Accelerators

Edge AI means running inference on the product itself rather than sending data to a server. It suits products that must respond quickly, work without a connection or keep sensor data private. The hardware ranges from a small microcontroller classifying vibration data to a module with a neural processing unit handling several camera streams. This article explains what each class of hardware suits and how to choose between them.

At a glance

int8
The usual numeric format for edge inference, a quarter of the size of float32 weights.
NPU
Dedicated hardware for the multiply-accumulate operations that dominate neural networks.
Offline
On-device inference keeps working when the network does not.
Local data
Raw images and audio can stay on the device instead of travelling to a server.

The problem

Teams often pick the processor first and the model second. A model developed on a desktop GPU can need far more memory and compute than the chosen part offers, and the gap is only found when the model is first ported.

Accelerators bring their own constraint. Each NPU supports a particular set of operators, and any layer outside that set falls back to the CPU, which can erase most of the expected speed-up.

Choose the model and the silicon together. Either one chosen alone is likely to be wrong.

Matching the model to the silicon

Microcontrollers run small networks for sensor, audio and simple vision tasks, usually through runtimes such as TensorFlow Lite for Microcontrollers and optimised kernel libraries like CMSIS-NN on Arm Cortex-M. Memory, not arithmetic, is usually the first limit.

Application processors with an NPU run larger vision models under Linux through the vendor toolchain. Check early that every operator in your model is supported, because coverage and tooling vary considerably between vendors.

What good looks like

Define the task, the input rate and the latency the product needs, then estimate model size and compute from a candidate architecture.

Shortlist hardware whose memory and operator support fit that model, and confirm the vendor toolchain converts it.

Benchmark the converted model on a development kit with real input data before the board design is committed.

Classes of edge AI hardware
PlatformMemory availableTypical tasks
MicrocontrollerHundreds of kilobytes of RAMKeyword spotting, vibration and sensor classification
Microcontroller with ML acceleratorHundreds of kilobytes to a few megabytesSmall vision and audio models at low power
Application processor with NPUExternal DRAM, typically hundreds of megabytes upwardObject detection and classification on camera streams
Separate accelerator moduleOwn memory plus a host over USB, M.2 or PCIeAdding inference to an existing design
Embedded GPU moduleGigabytes of shared memorySeveral cameras, larger models, mixed workloads
Broad categories. Memory, operator support and tooling vary considerably by vendor and part.
Storage per weight by numeric format
bytes
float324
float162
int81
int40.5
Bytes needed to store one weight. Lower precision shrinks memory use and bandwidth proportionally.

What it changes

Fast Response

Inference completes on the device, with no dependence on network latency.

Privacy

Sensitive data can be processed where it is captured.

Right-Sized Hardware

The processor fits the model, so the bill of materials carries no idle silicon.

Fewer Surprises

Operator support is checked before layout, not after.

Where to go next

Plan how models will be updated in the field. A new model is a firmware update and needs the same signing and rollback protection.

Measure power with inference running. Duty cycle and wake strategy often matter more than the accelerator itself.

Problems this prevents

  • A model too large for the chosen part
  • Layers that silently fall back to the CPU
  • Hardware chosen before the model was sized
  • Power measured without inference running

Discuss your project

Email support@aavico.com or use Get in Touch when the website finishes loading.