I test a lot of tiny devices—fitness bands, smart rings, and the occasional prototype smartwatch—and one question always comes up: how do you meaningfully evaluate on-device AI when battery life is the limiting factor? It’s tempting to point to a single number (inference time, mAh drained) and call it a day, but battery-powered wearables demand a multidimensional approach. In this piece I walk through the practical benchmarks that matter, the traps to avoid, and a reproducible methodology you can apply to your own devices.
What “on-device AI” means for wearables
On wearables, on-device AI usually fits into two patterns: continuous sensing (step counting, heart rate anomaly detection, activity recognition) and on-event inference (wake word detection, fall detection, gesture recognition). Both classes share constraints: tiny batteries, constrained memory, limited thermal headroom, and often intermittent connectivity. That changes how we should evaluate models.
Key metrics I always measure
- Energy per inference — measured in microjoules or milliampere-hours per inference. This is the single most useful number for battery-driven devices because it connects model behavior to runtime.
- Inference latency — end-to-end time from sensor sample to model output. For real-time interactions you’ll need latency guarantees; for background sensing you can accept higher latency to save energy.
- Accuracy and robustness — standard model metrics (accuracy, F1, precision/recall) plus robustness to noisy inputs and model drift under real-world conditions.
- Memory footprint — flash storage usage (binary size) and runtime RAM. Many MCUs have < 1 MB RAM and flash constraints that rule out some models entirely.
- Thermal behavior — temperature rise on the device during sustained operation. Thermal throttling or user comfort (skin temperature) are practical limits.
- Duty cycle and baseline power — how often the model runs, and what the idle power consumption of the device is. A model that’s cheap per inference but runs constantly may still kill the battery.
- Startup and wake time — time and energy cost to spin up an AI pipeline from a low-power state.
- Developer and deployment ergonomics — toolchain maturity (TensorFlow Lite Micro, ONNX Runtime, Qualcomm SDK, Edge Impulse), quantization support, and over-the-air update complexity.
How I measure energy per inference
Measuring energy correctly is the hardest part and also where most people get it wrong. I use a three-step approach:
- Instrumented power measurement — place a high-resolution current shunt (e.g., 0.01 Ω) in series with the device’s VBAT and log current at a high sample rate (10 kHz+). For small devices I often use a Monsoon Power Monitor or a TI INA219/INA226 with an oscilloscope for microsecond resolution.
- Isolate the event — trigger a known number of inferences (say 1000) and capture the current trace. Subtract baseline idle power measured when the model isn’t running but the rest of the system is on.
- Compute energy per inference — integrate the over-baseline current over time, multiply by voltage and divide by the number of inferences. Report median and 95th percentile to show variance.
Important detail: if your model runs in bursts or as part of a sensor fusion pipeline, measure the full pipeline energy, not just the isolated MCU execution. For example, the power cost of sampling an IMU at 200 Hz may dwarf the model’s compute cost.
Latency: end-to-end, not just NN execution
I always separate three latency components:
- Sensor acquisition latency — time to collect the input window (e.g., 1 second of accelerometer data).
- Preprocessing and feature extraction — filtering, transforms (MFCC for audio), and quantization steps.
- Model execution — raw NN inference time on the device (including runtime overhead).
Measure these together and individually. A model with 5 ms inference time but that requires a 1 s input window won’t be suitable for real-time haptics, for instance.
Accuracy and real-world validation
Benchmarks on sanitized datasets are useful but insufficient. I recommend a two-phase approach:
- Standard dataset validation — train and test on common benchmarks (e.g., FSD50K subsets for audio, HAR datasets for motion). Report standard metrics and confusion matrices.
- In-situ testing — collect real-world data on target devices and users (skin contact variability, placement shifts). Measure false positives/negatives under everyday conditions.
I’ve often seen on-device models lose several percentage points of accuracy when the sensor stack changes (different accelerometer vendor or sampling jitter). If you can’t collect in-situ data, at least add noise augmentation and simulate sensor drift in training.
Memory and binary size
List both:
- Model binary size on flash (including runtime)
- Peak RAM usage during inference
For tiny MCUs, quantized models built with TensorFlow Lite Micro or CMSIS-NN often make the difference. If your model uses dynamic memory allocation or large buffers for preprocessing, that can disqualify an otherwise efficient architecture.
Thermals and human factors
Wearables are in contact with skin. I measure device surface temperature over time under continuous inference patterns. A rise of >3–4°C may feel warm and trigger user complaints. Some SoCs (Qualcomm Snapdragon W5, Apple S-series) are better at thermal management, but don’t assume—they differ across form factors.
Reproducible benchmark matrix
Here’s a minimal table I use to report results across competing models or devices. Use real numbers from your testing.
| Metric | Model A (quantized) | Model B (pruned) | Notes |
|---|---|---|---|
| Energy per inference (µJ) | 120 | 85 | Measured with Monsoon, 3.7V |
| Inference latency (ms) | 9 | 14 | End-to-end |
| Accuracy (real-world F1) | 0.92 | 0.88 | Field test with 20 users |
| Flash size (KB) | 320 | 240 | Includes runtime |
| Peak RAM (KB) | 64 | 48 | |
| Thermal rise (°C) | 2.1 | 3.8 | 10 min sustained |
| Duty cycle | 10% (10s windows) | Continuous | Operational mode |
Choice of tools and reference suites
For microcontroller-class devices I use TensorFlow Lite Micro, CMSIS-NN, and MLPerf Tiny where applicable. For more capable wearable SoCs, ONNX Runtime Mobile and vendor SDKs (Qualcomm’s Hexagon NN, MediaTek Neural Processing) are useful. Edge Impulse is great for building prototypes and provides power profiling hooks, while tools like the Monsoon Power Monitor or Nordic Power Profiler Kit give production-grade measurements.
Practical checklist before publishing a benchmark
- Document hardware revision, firmware version, and measurement rig.
- Provide raw power traces and a script to compute energy per inference.
- Publish the test dataset (or a sanitized subset) and training recipe.
- Report median and percentile metrics, not just averages.
- Note environmental conditions: room temperature, battery state of charge, antenna activity.
Trade-offs I watch for in real projects
There are always trade-offs. A highly optimized quantized model may improve energy and memory but become brittle; pruning may reduce size but increase latency due to irregular sparsity; an accelerometer-heavy pipeline can be far cheaper than continuous audio processing. The aha moment comes when you model the full system: inference energy, sensor energy, duty cycle, and user expectations together determine feasibility.
If you want, I can share the measurement scripts and a sample dataset I use for motion-based activity recognition—drop a note and I’ll post them. I’ve found that once teams see the energy-per-inference numbers in context (with duty cycle and sensor costs), the product decisions become obvious and the marketing claims about “AI on-device” either hold up—or they don’t.
