Optimizing AI models for deployment on resource-constrained IoT devices
This proof of concept explored how model quantization can make computer-vision models smaller and faster while preserving predictive performance. The application scenario focused on grape leaf disease classification, representing a practical smart-agriculture use case in which image analysis may need to be performed directly in the field.
The work was carried out by Žarko Peruničić as part of his MSc research at the University of Donja Gorica. It emerged from the broader educational and mentoring ecosystem around the MSc programme in Artificial Intelligence, developed with support from EuroCC / NCC Montenegro, with mentoring expertise from UDG and NCC Montenegro.
CHALLENGE
Edge AI brings artificial intelligence closer to the source of data, enabling local decision-making with lower latency and reduced dependence on cloud connectivity. However, deploying modern AI models on IoT and edge devices remains challenging because of limited processing power, memory, and energy resources.
SOLUTION
A MobileNetV2 computer-vision model was used as the reference architecture and optimized for edge deployment using TensorFlow Lite. Several variants were prepared and evaluated, including FP16 post-training quantization, dynamic INT8 quantization, full INT8 post-training quantization, and an INT8 model obtained through quantization-aware training (QAT).
The evaluation considered several factors that matter in real-world edge applications: model size, inference latency, execution stability, peak RAM usage, and classification accuracy. The benchmark was intentionally performed under controlled CPU-only conditions to approximate a resource-constrained edge scenario.
The results showed that quantization can substantially reduce model size and improve inference efficiency, but also demonstrated that not every quantization strategy leads to better practical performance.
On the more challenging field-oriented dataset, the baseline FP32 TensorFlow Lite model had a size of 8.47 MB, a median inference latency of 3.67 ms, and an accuracy of 89.44%. The QAT INT8 variant reduced the model to 2.59 MB, achieved a median latency of 2.93 ms, and reached 90.00% accuracy. Full INT8 post-training quantization produced a similarly compact model and the lowest median latency, at 2.91 ms.
An important finding was that a smaller model is not automatically a faster model. The dynamic INT8 variant reduced the model to 2.39 MB, but produced substantially higher inference latency on the evaluated platform. This underlines the need to benchmark optimization strategies on the actual execution environment rather than assuming that lower numerical precision will always improve performance.
The main conclusion is that there is no universally optimal quantization strategy. The best configuration depends on the model, data characteristics, calibration procedure, and target execution environment. The results therefore support an empirical, application-specific approach to Edge AI optimization.
BENEFITS
- Smaller AI models: INT8 quantization reduced the MobileNetV2 model from 8.47 MB to approximately 2.6 MB.
- Faster inference: full INT8 and QAT INT8 variants achieved lower inference latency than the baseline FP32 model.
- Preserved predictive quality: the QAT INT8 model achieved 90.0% accuracy on the more challenging dataset, slightly exceeding the FP32 baseline in this experiment.
- Reduced dependence on cloud connectivity: efficient models provide a basis for performing computer-vision inference closer to the data source.
- Practical relevance for smart agriculture: the approach can support future field applications where crop monitoring must operate under limited computing and connectivity conditions.
- Transferable Edge AI methodology: the study shows why optimization should be evaluated as a balance between accuracy, latency, model size, and execution environment rather than through a single metric.
The research contributes to the broader AI, IoT, and advanced-computing capacity-building activities of NCC Montenegro, showing how graduate education and mentoring can translate into practical experiments addressing real deployment constraints.