{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T17:32:25Z","timestamp":1778347945253,"version":"3.51.4"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"3-4","license":[{"start":{"date-parts":[[2024,5,22]],"date-time":"2024-05-22T00:00:00Z","timestamp":1716336000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,22]],"date-time":"2024-05-22T00:00:00Z","timestamp":1716336000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100011045","name":"Minist\u00e8re de l\u2019Enseignement sup\u00e9rieur, de la Recherche et de l\u2019Innovation","doi-asserted-by":"publisher","award":["ANR-20-CYAL-0005"],"award-info":[{"award-number":["ANR-20-CYAL-0005"]}],"id":[{"id":"10.13039\/501100011045","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschun","doi-asserted-by":"publisher","award":["16KIS1395K"],"award-info":[{"award-number":["16KIS1395K"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Ann. Telecommun."],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Practitioners in the field of TinyML lack so far a comprehensive, \u201cbatteries-included\u201d toolkit to streamline continuous integration, continuous deployment and performance assessments of executing diverse machine learning models on various low-power IoT hardware. Addressing this gap, our paper introduces RIOT-ML, a versatile toolkit crafted to assist IoT designers and researchers in these tasks. To this end, we designed RIOT-ML based on an integration of an array of functionalities from a low-power embedded OS, a universal model transpiler and compiler, a toolkit for TinyML performance measurement, and a low-power over-the-air secure update framework\u2014all of which usable on an open-access IoT testbed available to the community. Our open-source implementation of RIOT-ML and the initial experiments we report on showcase its utility in experimentally evaluating TinyML model performance across fleets of low-power IoT boards under test in the field, featuring a wide spectrum of heterogeneous microcontroller architectures and fleet network connectivity configurations. The existence of an open-source toolkit such as RIOT-ML is essential to expedite research combining artificial intelligence and IoT and to foster the full realization of edge computing\u2019s potential.<\/jats:p>","DOI":"10.1007\/s12243-024-01041-5","type":"journal-article","created":{"date-parts":[[2024,5,22]],"date-time":"2024-05-22T08:02:09Z","timestamp":1716364929000},"page":"283-297","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["RIOT-ML: toolkit for over-the-air secure updates and performance evaluation of TinyML models"],"prefix":"10.1007","volume":"80","author":[{"given":"Zhaolan","family":"Huang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Koen","family":"Zandberg","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kaspar","family":"Schleiser","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emmanuel","family":"Baccelli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,5,22]]},"reference":[{"issue":"12","key":"1041_CR1","doi-asserted-by":"publisher","first-page":"2295","DOI":"10.1109\/JPROC.2017.2761740","volume":"105","author":"V Sze","year":"2017","unstructured":"Sze V, Chen Y-H, Yang T-J, Emer JS (2017) Efficient processing of deep neural networks: a tutorial and survey. Proc IEEE 105(12):2295\u20132329","journal-title":"Proc IEEE"},{"issue":"9","key":"1041_CR2","doi-asserted-by":"publisher","first-page":"1012","DOI":"10.3390\/electronics10091012","volume":"10","author":"H Sharma","year":"2021","unstructured":"Sharma H, Haque A, Blaabjerg F (2021) Machine learning in wireless sensor networks for smart cities: a survey. Electronics 10(9):1012","journal-title":"Electronics"},{"issue":"3","key":"1041_CR3","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/MCAS.2020.3005467","volume":"20","author":"R Sanchez-Iborra","year":"2020","unstructured":"Sanchez-Iborra R, Skarmeta AF (2020) TinyML-enabled frugal smart objects: challenges and opportunities. IEEE Circuits Syst Mag 20(3):4\u201318","journal-title":"IEEE Circuits Syst Mag"},{"issue":"4","key":"1041_CR4","first-page":"1595","volume":"34","author":"PP Ray","year":"2022","unstructured":"Ray PP (2022) A review on TinyML: state-of-the-art and prospects. J King Saud Univ Comput Inf Sci 34(4):1595\u20131623","journal-title":"J King Saud Univ Comput Inf Sci"},{"key":"1041_CR5","doi-asserted-by":"crossref","unstructured":"Huang Z, Zandberg K, Schleiser K, Baccelli E (2023) U-TOE: universal TinyML on-board evaluation toolkit for low-power IoT. In 2023 12th ifip\/ieee inter-national conference on performance evaluation and modeling in wired and wireless networks (pemwn), IEEE, pp 1\u20136","DOI":"10.23919\/PEMWN58813.2023.10304946"},{"issue":"22","key":"1041_CR6","doi-asserted-by":"publisher","first-page":"21362","DOI":"10.1109\/JSEN.2022.3210773","volume":"22","author":"SS Saha","year":"2022","unstructured":"Saha SS, Sandha SS, Srivastava M (2022) Machine learning for microcontroller-class hardware-a review. IEEE Sensors J 22(22):21362\u201321390","journal-title":"IEEE Sensors J"},{"issue":"5","key":"1041_CR7","doi-asserted-by":"publisher","first-page":"720","DOI":"10.1109\/JIOT.2015.2505901","volume":"3","author":"O Hahm","year":"2015","unstructured":"Hahm O, Baccelli E, Petersen H, Tsiftes N (2015) Operating systems for low-end devices in the Internet of Things: a survey. IEEE Internet Things J 3(5):720\u2013734","journal-title":"IEEE Internet Things J"},{"issue":"6","key":"1041_CR8","doi-asserted-by":"publisher","first-page":"4428","DOI":"10.1109\/JIOT.2018.2815038","volume":"5","author":"E Baccelli","year":"2018","unstructured":"Baccelli E, G\u00fcndo\u011fan C, Hahm O, Kietzmann P, Lenders MS, Petersen H, W\u00e4hlisch M (2018) RIOT: an open source operating system for low-end embedded devices in the IoT. IEEE Internet Things J 5(6):4428\u20134440","journal-title":"IEEE Internet Things J"},{"issue":"9","key":"1041_CR9","doi-asserted-by":"publisher","first-page":"3203","DOI":"10.1109\/JSEN.2019.2894127","volume":"19","author":"LE Lima","year":"2019","unstructured":"Lima LE, Kimura BYL, Rosset V (2019) Experimental environments for the Internet of Things: a review. IEEE Sens J 19(9):3203\u20133211","journal-title":"IEEE Sens J"},{"issue":"11","key":"1041_CR10","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1109\/MCOM.2011.6069710","volume":"49","author":"A Gluhak","year":"2011","unstructured":"Gluhak A, Krco S, Nati M, Pfisterer D, Mitton N, Razafindralambo T (2011) A survey on facilities for experimental Internet of Things research. IEEE Commun Mag 49(11):58\u201367","journal-title":"IEEE Commun Mag"},{"key":"1041_CR11","doi-asserted-by":"crossref","unstructured":"Adjih C, Baccelli E, Fleury E, Harter G, Mitton N, Noel T, Vandaele J et al (2015) FIT IoT-LAB: a large scale open experimental IoT testbed. In: 2015 IEEE 2nd world forum on internet of things (wf-iot), IEEE, pp 459\u2013464","DOI":"10.1109\/WF-IoT.2015.7389098"},{"key":"1041_CR12","unstructured":"Banbury CR, Reddi VJ, Lam M, Fu W, Fazel A, Holleman J, Lokhmotov A et al (2020) Benchmarking TinyML systems: challenges and direction. Preprint at arXiv:2003.04821"},{"key":"1041_CR13","unstructured":"Banbury C, Reddi VJ, Torelli P, Holleman J, Jeffries N, Kiraly C, Pau D et al (2021) MLPerf tiny benchmark. Preprint at arXiv:2106.07597"},{"key":"1041_CR14","doi-asserted-by":"crossref","unstructured":"Osman A, Abid U, Gemma L, Perotto M, Brunelli D (2022) TinyML platforms benchmarking. In: Applications in electronics pervading industry, environment and society: Applepies 2021, Springer pp 139\u2013148","DOI":"10.1007\/978-3-030-95498-7_20"},{"key":"1041_CR15","doi-asserted-by":"crossref","unstructured":"Sudharsan B, Salerno S, Nguyen D-D, Yahya M, Wahid A, Yadav P, Ali MI (2021) TinyML benchmark: executing fully connected neural networks on commodity microcontrollers. In: 2021 IEEE 7th world forum on internet of things (wf-iot), IEEE, pp 883\u2013884","DOI":"10.1109\/WF-IoT51360.2021.9595024"},{"key":"1041_CR16","unstructured":"Chen T, Moreau T, Jiang Z, Zheng L, Yan E, Shen H, Ceze L et al (2018) TVM: an automated end-to-end optimizing compiler for deep learning, 578\u2013594"},{"key":"1041_CR17","first-page":"337","volume":"4","author":"H Qiu","year":"2022","unstructured":"Qiu H, Vavelidou I, Li J, Pergament E, Warden P, Chinchali S, Katti S (2022) ML-EXray: visibility into ml deployment on the edge. Proc Mach Learn Syst 4:337\u2013351","journal-title":"Proc Mach Learn Syst"},{"key":"1041_CR18","doi-asserted-by":"publisher","unstructured":"Yousefzadeh-Asl-Miandoab E, Robroek T, Tozun P (2023) Profiling and monitoring deep learning training tasks. In: Proceedings of the 3rd workshop on machine learning and systems, pp 18\u201325. https:\/\/doi.org\/10.1145\/3578356.3592589","DOI":"10.1145\/3578356.3592589"},{"key":"1041_CR19","doi-asserted-by":"publisher","first-page":"31866","DOI":"10.1109\/ACCESS.2023.3262138","volume":"11","author":"D Kreuzberger","year":"2023","unstructured":"Kreuzberger D, K\u00fchl N, Hirschl S (2023) Machine learning operations (MLOps): overview, definition, and architecture. IEEE Access 11:31866\u201331879. https:\/\/doi.org\/10.1109\/ACCESS.2023.3262138","journal-title":"IEEE Access"},{"key":"1041_CR20","doi-asserted-by":"crossref","unstructured":"L\u00ea MT, Arbel J (2023) TinyMLOps for real-time ultra-low power MCUs applied to frame-based event classification. In: Proceedings of the 3rd workshop on machine learning and systems, New York, NY, USA: Association for Computing Machinery, pp 148\u2013153","DOI":"10.1145\/3578356.3592586"},{"issue":"4","key":"1041_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3625289","volume":"56","author":"J Diaz-de Arcaya","year":"2023","unstructured":"Diaz-de Arcaya J, Torre-Bastida AI, Z\u00e1rate G, Mi\u00f1\u00f3n R, Almeida A (2023) A joint study of the challenges, opportunities, and roadmap of MLOps and AIOps: a systematic survey. ACM Comput Surv 56(4):1\u201330. https:\/\/doi.org\/10.1145\/3625289","journal-title":"ACM Comput Surv"},{"key":"1041_CR22","doi-asserted-by":"publisher","first-page":"71907","DOI":"10.1109\/ACCESS.2019.2919760","volume":"7","author":"K Zandberg","year":"2019","unstructured":"Zandberg K, Schleiser K, Acosta F, Tschofenig H, Baccelli E (2019) Secure firmware updates for constrained IoT devices using open standards: a reality check. IEEE Access 7:71907\u201371920","journal-title":"IEEE Access"},{"key":"1041_CR23","doi-asserted-by":"crossref","unstructured":"Mtetwa NS, Tarwireyi P, Abu-Mahfouz AM, Adigun MO (2019) Secure firmware updates in the Internet of Things: a survey. In 2019 international multidisciplinary information technology and engineering conference (imitec), IEEE, pp 1\u20137","DOI":"10.1109\/IMITEC45504.2019.9015845"},{"key":"1041_CR24","doi-asserted-by":"crossref","unstructured":"Banegas G, Zandberg K, Baccelli E, Herrmann A, Smith B (2022) Quantum-resistant software update security on low-power networked embedded devices. In International conference on applied cryptography and network security, Springer, pp 872\u2013891","DOI":"10.1007\/978-3-031-09234-3_43"},{"issue":"2","key":"1041_CR25","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1109\/MCOM.001.1900125","volume":"58","author":"J Bauwens","year":"2020","unstructured":"Bauwens J, Ruckebusch P, Giannoulis S, Moerman I, De Poorter E (2020) Over-the-air software updates in the Internet of Things: an overview of key principles. IEEE Commun Mag 58(2):35\u201341","journal-title":"IEEE Commun Mag"},{"issue":"2","key":"1041_CR26","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1109\/MIC.2012.29","volume":"16","author":"C Bormann","year":"2012","unstructured":"Bormann C, Castellani AP, Shelby Z (2012) CoAP: an application protocol for billions of tiny Internet nodes. IEEE Internet Comput 16(2):62\u201367","journal-title":"IEEE Internet Comput"},{"key":"1041_CR27","doi-asserted-by":"crossref","unstructured":"Shelby Z, Hartke K, Bormann C (2014) RFC 7252: The constrained application protocol (CoAP). IETF Request for Comments","DOI":"10.17487\/rfc7252"},{"key":"1041_CR28","doi-asserted-by":"publisher","unstructured":"Moran B, Tschofenig H, Brown D, Meriac M (2021) A firmware update architecture for Internet of Things. RFC 9019. https:\/\/doi.org\/10.17487\/RFC9019","DOI":"10.17487\/RFC9019"},{"key":"1041_CR29","doi-asserted-by":"publisher","unstructured":"Hansen T 3rd, D. E. E. (2006) US secure hash algorithms (SHA and HMACSHA). RFC 4634. https:\/\/doi.org\/10.17487\/RFC4634","DOI":"10.17487\/RFC4634"},{"key":"1041_CR30","doi-asserted-by":"publisher","unstructured":"Josefsson S, Liusvaara I (2017) Edwards-curve digital signature algorithm (EdDSA). RFC 8032. https:\/\/doi.org\/10.17487\/RFC8032","DOI":"10.17487\/RFC8032"},{"key":"1041_CR31","unstructured":"Tschofenig H, Housley R, Moran B, Brown D, Takayama K (2023) Encrypted payloads in SUIT Manifests (Internet-Draft No. draft-ietf-suit-firmwareencryption-18). Work in Progress. Internet Engineering Task Force. Retrieved from https:\/\/datatracker.ietf.org\/doc\/draft-ietfsuit-firmware-encryption\/18\/"},{"key":"1041_CR32","doi-asserted-by":"publisher","unstructured":"Selander G, Mattsson JP, Palombini F (2024) Ephemeral Diffie-Hellman Over COSE (EDHOC). RFC 9528. https:\/\/doi.org\/10.17487\/RFC9528","DOI":"10.17487\/RFC9528"},{"issue":"94720\u20131776","key":"1041_CR33","first-page":"11","volume":"754","author":"W Kahan","year":"1996","unstructured":"Kahan W (1996) IEEE standard 754 for binary floating-point arithmetic. Lecture Notes Status IEEE 754(94720\u20131776):11","journal-title":"Lecture Notes Status IEEE"},{"key":"1041_CR34","doi-asserted-by":"publisher","unstructured":"Schaad J (2017) CBOR object signing and encryption (COSE). RFC 8152. https:\/\/doi.org\/10.17487\/RFC8152","DOI":"10.17487\/RFC8152"}],"container-title":["Annals of Telecommunications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12243-024-01041-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12243-024-01041-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12243-024-01041-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,11]],"date-time":"2025-03-11T06:58:54Z","timestamp":1741676334000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12243-024-01041-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,22]]},"references-count":34,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["1041"],"URL":"https:\/\/doi.org\/10.1007\/s12243-024-01041-5","relation":{},"ISSN":["0003-4347","1958-9395"],"issn-type":[{"value":"0003-4347","type":"print"},{"value":"1958-9395","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,22]]},"assertion":[{"value":"17 January 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 May 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 May 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"The paper reflects only the authors\u2019 views. MESRI and BMBF are not responsible for any use that may be made of the information it contains.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclaimer"}}]}}