Machine learning on embedded devices

Focused primarily on running inference/prediction/feed-forward part on a microcontroller (or small embedded device). Training phase can run on a standard computer/server, using existing tools as much as possible.

Background

What and when to use machine learning

The defaults right now are to do conventional signal processing (no learning) in sensor, and stream raw data to the cloud for storage and processing. Machine learning happens in the cloud. If gateways are used, they mostly forward communication (no data processing).

On-edge processing valueable when

• Local response needed. Autonomy
• Adaptable response needed. Over time, in context.
• Low/predictable latency needed
• Sending raw sensor data has privacy implications. Audio, video.
• Unreliable connection
• High bandwidth sensor input. Audio, video, accelerometer/IMU, current sensor, radiowaves.
• Low bandwidth algorithm output
• Events of interest are rare
• Low energy usage needed
• Full/raw sensor data is not valuable to store
• Sensor system should be low cost

Example usecases

• Predictive maintenance, using audio/vibration data
• Activitity detection for people, using audio/accelerometer data. Assistive tech, medical
• Appliance disaggregation, using aggregated power consumption data. "Non-Intrusive Load Monitoring" (NILM)
• Anomaly/change detection for predictive maintenance, using audio/vibration data, or electrical data
• Gesture recognition as human input device, using accelerometer/gyro data.
• Speech/command recognition as human input device, using microphone. Keyword/Wake-word detection
• Battery saving in wireless sensors. Normally sending day/week aggregates, on event/anomaly detection send data immediately
• Health status of animals via activity detected using accelerometer
• Monitoring eating activity using accelerometer 1
• Environmental monitoring, using microphone to detect unwanted activity like cutting down trees
• Adaptive signalling and routing for wireless transmission in Wireless Sensor networks
• Electronic nose using arrays of MEMS detectors
• Material identification using reflecive spectrometer 1

More notes on Applications

Motivation

• Why the Future of Machine Learning is Tiny (devices) Tiny Computers are Already Cheap and Everywhere. Energy is the Limiting Factor.We Capture Much More Sensor Data Than We Use.
• embedded.com: Bringing machine learning to the edge Predictions are much lower bandwidth than the raw sensor data (e.g. video) It allows for local adaptation in the AI logic (L-DNN) It achieves lower latency between observed event and action resulting from AI logic "the most important question is what is the least amount accuracy and computation complexity we can do while still delivering the business value?" Top mistake: "Continuing with the top down approach ‘let’s make it perform the task first and then squeeze it on device` instead of switching to bottom up ‘let’s make it run on device and fulfill all hardware constraints first, and then tune it for the task at hand’."
• How to run deep learning model on microcontroller with CMSIS-NN. Why run deep learning model on a microcontroller? Sensitive data gets to the cloud, photos, and audio recordings. The company who sells this may charge a service fee to use its service and even worse sell your private data. It won't work without the network connection to the server. Data traveling back and forth between the device and server introduces lag. Require network and wireless hardware components on the circuit design which increase the cost. It might waste bandwidth sending useless data.

Own contributions

Presentations

• Embedding Data Science in IoT devices with MicroPython and emlearn. EuroSciPy 2026, Krakow.
• Machine Learning on Microcontrollers; Tiny, Useful and Fun. Open Source GenAI & ML Summit. Amsterdam, August 2025.
• Sensor data processing for scientific applications with MicroPython. EuroSciPy 2025, Krakow.
• Machine Learning on Microcontrollers With Zephyr and emlearn. Open Source Summit / Zephyr Developer Summit. Amsterdam, August 2025.
• Microcontrollers and Machine Learning with MicroPython and emlearn. FOSS-North. Gothenburg, April 2025.
• Sensor data processing on microcontrollers with MicroPython and emlearn. PyCon South Africa. Online, October 2024.
• emlearn - Machine Learning for Tiny Embedded Systems. Embedded Online Conference. April 2024.
• 6 years of open source TinyML with emlearn. TinyML EMEA, Milano. June 2024.
• emlearn - Machine Learning for Tiny Embedded Systems. May 2024.
• Machine Learning on microcontrollers using MicroPython and emlearn. PyDataBerlin & PyCon DE. April 2024.
• HVAC monitoring in commercial buildings using Anomaly Detection. TinyAI Forum on PdM and Anomaly Detection (online). December 2023.

Lecture notes

• Practical TinyML with emlearn. Aarhus University. March 2026.
• emlearn @ TinyMLDays Aarhus. August 2025.

Open source software

• emlearn. Machine Learning inference engine in C99 for Microcontrollers and Embedded devices. Since 2018.
• emlearn-micropython. Efficient and convenient Machine Learning and DSP for MicroPython. Since 2024.

Fun projects

• Automatic toothbrush timer using accelerometer. Github project. Video presentation.
• Spectrometer for edible oils Github project.

Commercial projects

• Acoustic detection of first crack in Roest coffee roasting machines. Shipping since 2019. Tech presentation.

Learning material

Books

• Learning in Embedded Systems, May 1993.
• TinyML: Machine Learning with TensorFlow on Arduino, and Ultra-Low Power Micro-Controllers. January, 2020.
• TinyML Cookbook.
• Machine Learning Systems - Principles and Practices of Engineering Artificially Intelligent Systems.

Articles

• Embedded.com: Applying machine learning in embedded systems
• Embedded.com: Transfer learning for the IoT

Tools

Open-source

• emlearn. Inference engine for microcontrollers. Supports converting scikit-learn models to plain C code. No dynamic allocations. No runtime needed.
• TensorFlow Lite for Microcontrollers. Supports neural network models made with TensorFlow (including Keras). Can run on wide range of platforms. Since November 2018. Supports ARM Cortex M, RISC-V, ESP32/Xtensa and Linux/MacOS host. Requires a runtime, aroud 20 kB.
• onnx2c. Allows generating C code from ONNX models.
• iree. Compiler. Has bare metal target. Around 30 kB runtime.
• executorch. C++ framework for deploying PyTorch models. Focused on custom backends, NPUs etc. Microcontroller CPU less in focus. Around 50 kB runtime. Supports bare-metal via "Portable" operations, or specialized for Cortex M.
• nnom - Fixed-point neural network compiler for microcontrollers. Supports wide range of networks. Outputs plain C code. Can use CMSIS-NN on ARM Cortex M.
• Embedded Learning Library by Microsoft. Discontinued in 2024 Set of C++ libraries for machine learning on embedded platforms. Includes code for kNN, RandomForest etc. Also has some node-based dataflow system in place it seems. JavaScript and Python bindings.
• uTensor. Export Tensorflow model to mbed/ARM microcontrollers. Not supported on ESP32 or RISC-V or similar. Lower activity since 2021.

• sklearn-porter. Can compile DecisionTreeClassifier and SVC models to C. Uses dynamic memory. Not optimized for use on embedded devices.

• microTVM. Depends only on the C standard library, and runs bare metal such as STM32 and NRF52 microcontrollers. Is under development. Which models are supported on microcontrollers not specified.

Low-level libraries

• gemmlowp. Low-precision matrix multiplication. Only general matrix multiplication ("GEMM"). C++11
• CMSIS-NN. Neural network operations for ARM Cortex M.
• esp-nn. Neural Network functions Espressif ESP32 (Xtensa and RISC-V).
• . Can be used bare-metal, but more focused on mobile/server grade (Cortex A+ etc).

Proprietary

• X-CUBE-AI for STM32

Historical

• nn4mc_cpp. Neural Networks for Microcontrollers. Supports Keras natively. Provides instructions for PyTorch et.c via ONNX. Documentation for using from Python is lacking, as well as the type of networks supported. Does not seem maintained since 2020.

• Embedded Classification Software Toolbox. Unmaintained since 2018.
• ONNC project has a backend for ARM Cortex M (using CMSIS-NN) and a C backend. Allows to convert an ONNX models to run on devices. Not updated since 2021.

• micromlgen. Generate C code for microcontrollers from Python's sklearn classifiers. Archived in 2024.

Models

A range of Machine Learning models are useful in an embedded devices setting. Classical methods are used when the amount of data is quite small, and neural networks for large datasets and complex inputs.

Below are notes on the various models in the context of embedded Machine Learning, including model size and compute-time optimization.

• Tree-based methods. Random Forest, Extratrees, Decision Trees, et.c.
• Neural Networks. Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Autoencoders
• Support Vector Machines (SVM).
• Mixture models. Gaussian Mixture Models (GMM).
• Nearest Neighbours. kNN et.c.

More topics

• Privacy
• Energy usage
• Model size
• On-device learning

State of the Art in 2019

Of ML inference on general-purpose microcontrollers.

TODO: update for 2023

• Deep models have efficient implementations for ARM Cortex-M. Ex: CNN and RNN in CMSIS-NN, FC in uTensor
• Some implementations available for non-neural models that can be used. Ex: SVM,RF,AdaBoost in sklearn-porter
• A few special-designed ML algorithms made. Ex: ProtoNN, Bonsai
• Basic tools available for converting Tensorflow models
• Keyword-spotting/wake-word on audio well established. Used in commercial products (Alexa etc)
• Human activity detecton on accelerometers.
• Computer vision is actively developed
• Lots of research and many announcements of low-power co-processors, but little on market yet

Limitations

• Neural models lacking for non-ARM micros. ESP8266/ESP32
• Non-neural models missing inference engines designed for microcontrollers
• "Small DNN" work mostly on computer vision for mobile phones (model size 1000x of uC)
• Few/no pretrained models available. Transfer learning little explored?
• Very little documentation of entire development process. From planning, data aquisition, model design
• Best practices underdocumented (or underdeveloped?)

Ways of advancing, make contributions

• Faster inference. Power saving, or bigger problems.
• Smaller models. Cheaper MCU, or bigger problems.
• Better accuracy on a problem. Better user experience, new usecases
• Solve a concrete usecase. Easier to deploy similar usecases
• Comparison between approaches. Microcontroller, ML model
• Libraries or tools. Lower time to market, enable more developers