{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T17:15:50Z","timestamp":1774718150030,"version":"3.50.1"},"reference-count":44,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T00:00:00Z","timestamp":1730073600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>This paper proposes a new benchmark specifically designed for in-sensor digital machine learning computing to meet an ultra-low embedded memory requirement. With the exponential growth of edge devices, efficient local processing is essential to mitigate economic costs, latency, and privacy concerns associated with the centralized cloud processing. Emerging intelligent sensors equipped with computing assets to run neural network inferences and embedded in the same package, which hosts the sensing elements, present new challenges due to their limited memory resources and computational skills. This benchmark evaluates models trained with Quantization Aware Training (QAT) and compares their performance with Post-Training Quantization (PTQ) across three use cases: Human Activity Recognition (HAR) by means of the SHL dataset, Physical Activity Monitoring (PAM) by means of the PAMAP2 dataset, and superficial electromyography (sEMG) regression with the NINAPRO DB8 dataset. The results demonstrate the effectiveness of QAT over PTQ in most scenarios, highlighting the potential for deploying advanced AI models on highly resource-constrained sensors. The INT8 versions of the models always outperformed their FP32, regarding memory and latency reductions, except for the activations for CNN. The CNN model exhibited reduced memory usage and latency with respect to its Dense counterpart, allowing it to meet the stringent 8KiB data RAM and 32 KiB program RAM limits of the ISPU. The TCN model proved to be too large to fit within the memory constraints of the ISPU, primarily due to its greater capacity in terms of number of parameters, designed for processing more complex signals like EMG. This benchmark aims to guide the development of efficient AI solutions for In-Sensor Machine Learning Computing, fostering innovation in the field of Edge AI benchmarking, such as the one conducted by the MLCommons-Tiny working group.<\/jats:p>","DOI":"10.3390\/info15110674","type":"journal-article","created":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T08:39:07Z","timestamp":1730104747000},"page":"674","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Benchmarking In-Sensor Machine Learning Computing: An Extension to the MLCommons-Tiny Suite"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-2188-1436","authenticated-orcid":false,"given":"Fabrizio Maria","family":"Aymone","sequence":"first","affiliation":[{"name":"System Research and Applications, STMicroelectronics, Business Center Colleoni, Building Andromeda 3, at the 7th Floor, Via Cardano 20, 20864 Agrate Brianza, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1585-2313","authenticated-orcid":false,"given":"Danilo Pietro","family":"Pau","sequence":"additional","affiliation":[{"name":"System Research and Applications, STMicroelectronics, Business Center Colleoni, Building Andromeda 3, at the 7th Floor, Via Cardano 20, 20864 Agrate Brianza, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2024,10,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Al-Sarawi, S., Anbar, M., Abdullah, R., and Al Hawari, A.B. (2020, January 27\u201328). Internet of Things Market Analysis Forecasts, 2020\u20132030. Proceedings of the 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), London, UK.","DOI":"10.1109\/WorldS450073.2020.9210375"},{"key":"ref_2","first-page":"100224","article-title":"Study on artificial intelligence: The state of the art and future prospects","volume":"23","author":"Zhang","year":"2021","journal-title":"J. Ind. Inf. Integr."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"7789","DOI":"10.1109\/JIOT.2020.3039359","article-title":"Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things","volume":"8","author":"Zhang","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_4","unstructured":"Han, S., Mao, H., and Dally, W.J. (2016, January 2\u20134). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. (2019, January 15\u201320). HAQ: Hardware-Aware Automated Quantization with Mixed Precision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00881"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. (2017, January 22\u201329). Learning Efficient Convolutional Networks through Network Slimming. Proceedings of the ICCV, Venice, Italy.","DOI":"10.1109\/ICCV.2017.298"},{"key":"ref_7","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Cheng, H., Zhang, M., and Shi, J.Q. (2023). A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations. arXiv.","DOI":"10.1109\/TPAMI.2024.3447085"},{"key":"ref_9","unstructured":"Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (2022, January 17\u201323). DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks. Proceedings of the 39th International Conference on Machine Learning, Baltimore, MD, USA."},{"key":"ref_10","unstructured":"Lee, N., Ajanthan, T., and Torr, P.H. (2018). Snip: Single-shot network pruning based on connection sensitivity. arXiv."},{"key":"ref_11","first-page":"20390","article-title":"Sanity-checking pruning methods: Random tickets can win the jackpot","volume":"33","author":"Su","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_12","unstructured":"Wang, C., Zhang, G., and Grosse, R. (2020). Picking winning tickets before training by preserving gradient flow. arXiv."},{"key":"ref_13","unstructured":"Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. (2016). Learning structured sparsity in deep neural networks. Adv. Neural Inf. Process. Syst., 29."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Huang, Z., and Wang, N. (2018, January 8\u201314). Data-driven sparse structure selection for deep neural networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01270-0_19"},{"key":"ref_15","unstructured":"Bai, Y., Wang, H., Tao, Z., Li, K., and Fu, Y. (2022). Dual lottery ticket hypothesis. arXiv."},{"key":"ref_16","unstructured":"Chen, T., Zhang, Z., Liu, S., Chang, S., and Wang, Z. (2020, January 30). Long live the lottery: The existence of winning tickets in lifelong learning. Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia."},{"key":"ref_17","unstructured":"Chen, T., Sui, Y., Chen, X., Zhang, A., and Wang, Z. (2021, January 18\u201324). A unified lottery ticket hypothesis for graph neural networks. Proceedings of the International Conference on Machine Learning. PMLR, Virtual."},{"key":"ref_18","unstructured":"Huang, Y., Aloufi, R., Cadet, X., Zhao, Y., Barnaghi, P., and Haddadi, H. (2024). MicroT: Low-Energy and Adaptive Models for MCUs. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Choukroun, Y., Kravchik, E., Yang, F., and Kisilev, P. (2019, January 27\u201328). Low-bit quantization of neural networks for efficient inference. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Republic of Korea.","DOI":"10.1109\/ICCVW.2019.00363"},{"key":"ref_20","first-page":"1","article-title":"Quantized neural networks: Training neural networks with low precision weights and activations","volume":"18","author":"Hubara","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_21","unstructured":"Lin, D., Talathi, S., and Annapureddy, S. (2016, January 19\u201324). Fixed point quantization of deep convolutional networks. Proceedings of the International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_22","unstructured":"Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized neural networks. Adv. Neural Inf. Process. Syst., 29."},{"key":"ref_23","unstructured":"David, R., Duke, J., Jain, A., Reddi, V.J., Jeffries, N., Li, J., Kreeger, N., Nappier, I., Natraj, M., and Regev, S. (2021). TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems. arXiv."},{"key":"ref_24","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2016, January 2\u20134). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, Savannah, GA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ayaz, F., Zakariyya, I., Cano, J., Keoh, S.L., Singer, J., Pau, D., and Kharbouche-Harrari, M. (2023, January 18\u201323). Improving Robustness Against Adversarial Attacks with Deeply Quantized Neural Networks. Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), Gold Coast, Australia.","DOI":"10.1109\/IJCNN54540.2023.10191429"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1038\/s42256-021-00356-5","article-title":"Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors","volume":"3","author":"Coelho","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"ref_27","unstructured":"Coelho, C.N., Kuusela, A., Zhuang, H., Aarrestad, T., Loncar, V., Ngadiuba, J., Pierini, M., and Summers, S. (2020). Ultra low-latency, low-area inference accelerators using heterogeneous deep quantization with QKeras and hls4ml. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wang, E., Davis, J.J., Moro, D., Zielinski, P., Lim, J.J., Coelho, C., Chatterjee, S., Cheung, P.Y., and Constantinides, G.A. (2021, January 25). Enabling binary neural network training on the edge. Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning, Virtual.","DOI":"10.1145\/3469116.3470015"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MM.2020.2974843","article-title":"MLPerf: An Industry Standard Benchmark Suite for Machine Learning Performance","volume":"40","author":"Mattson","year":"2020","journal-title":"IEEE Micro"},{"key":"ref_30","unstructured":"Banbury, C., Reddi, V.J., Torelli, P., Holleman, J., Jeffries, N., Kiraly, C., Montino, P., Kanter, D., Ahmed, S., and Pau, D. (2021, January 6\u201314). MLPerf Tiny Benchmark. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Virtual."},{"key":"ref_31","unstructured":"Gal-On, S., and Levy, M. (2012). Exploring Coremark a Benchmark Maximizing Simplicity and Efficacy, The Embedded Microprocessor Benchmark Consortium."},{"key":"ref_32","unstructured":"Torelli, P., and Bangale, M. (2021, April 05). Measuring Inference Performance of Machine-Learning Frameworks on Edge-Class Devices with the Mlmark Benchmark. Techincal Report. Available online: https:\/\/api.semanticscholar.org\/CorpusID:232220731."},{"key":"ref_33","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv."},{"key":"ref_34","unstructured":"(2024, June 28). Benchmarking in Sensor Machine Learning: An Extension to MLCommons-Tiny Github Repository. Available online: https:\/\/github.com\/fabrizioaymone\/sensor."},{"key":"ref_35","unstructured":"(2024, May 28). Update: ISM330ISN and ISM330IS, Sensors with Intelligent Sensor Processing Unit for Greater AI at the Edge. Available online: https:\/\/www.st.com\/content\/st_com\/en\/campaigns\/ispu-ai-in-sensors.html."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"42592","DOI":"10.1109\/ACCESS.2018.2858933","article-title":"The University of Sussex-Huawei Locomotion and Transportation Dataset for Multimodal Analytics With Mobile Devices","volume":"6","author":"Gjoreski","year":"2018","journal-title":"IEEE Access"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"10870","DOI":"10.1109\/ACCESS.2019.2890793","article-title":"Enabling Reproducible Research in Sensor-Based Transportation Mode Recognition With the Sussex-Huawei Dataset","volume":"7","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Reiss, A., and Stricker, D. (2012, January 18\u201322). Introducing a New Benchmarked Dataset for Activity Monitoring. Proceedings of the 2012 16th International Symposium on Wearable Computers, Newcastle, UK.","DOI":"10.1109\/ISWC.2012.13"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Reiss, A., and Stricker, D. (2012, January 6\u20138). Creating and benchmarking a new dataset for physical activity monitoring. Proceedings of the 5th International Conference on Pervasive Technologies Related to Assistive Environments, Heraklion Crete, Greece.","DOI":"10.1145\/2413097.2413148"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Krasoulis, A., Vijayakumar, S., and Nazarpour, K. (2019). Effect of user practice on prosthetic finger control with an intuitive myoelectric decoder. Front. Neurosci., 13.","DOI":"10.3389\/fnins.2019.00891"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1109\/TBCAS.2019.2959160","article-title":"Robust Real-Time Embedded EMG Recognition Framework Using Temporal Convolutional Networks on a Multicore IoT Processor","volume":"14","author":"Zanghieri","year":"2020","journal-title":"IEEE Trans. Biomed. Circuits Syst."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Lea, C., Vidal, R., Reiter, A., and Hager, G.D. (2016). Temporal Convolutional Networks: A Unified Approach to Action Segmentation. arXiv.","DOI":"10.1007\/978-3-319-49409-8_7"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Zanghieri, M., Benatti, S., Burrello, A., Kartsch Morinigo, V.J., Meattini, R., Palli, G., Melchiorri, C., and Benini, L. (2021, January 23\u201325). sEMG-based Regression of Hand Kinematics with Temporal Convolutional Networks on a Low-Power Edge Microcontroller. Proceedings of the 2021 IEEE International Conference on Omni-Layer Intelligent Systems (COINS), Barcelona, Spain.","DOI":"10.1109\/COINS51742.2021.9524188"},{"key":"ref_44","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019, January 8\u201314). Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/11\/674\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:22:10Z","timestamp":1760113330000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/11\/674"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,28]]},"references-count":44,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2024,11]]}},"alternative-id":["info15110674"],"URL":"https:\/\/doi.org\/10.3390\/info15110674","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,28]]}}}