{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T17:16:48Z","timestamp":1783099008296,"version":"3.54.6"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"5s","license":[{"start":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T00:00:00Z","timestamp":1694217600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Institute of Information and Communications Technology Planning and Evaluation"},{"name":"Development of Core Technology for Autonomous Energy-driven Computing System SW in Power-Instable Environment","award":["2021-0-00360"],"award-info":[{"award-number":["2021-0-00360"]}]},{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"crossref","award":["2021R1A2C200497612"],"award-info":[{"award-number":["2021R1A2C200497612"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2023,10,31]]},"abstract":"<jats:p>The use of deep neural network (DNN) applications in microcontroller unit (MCU) embedded systems is getting popular. However, the DNN models in such systems frequently suffer from accuracy loss due to the dataset shift problem. On-device learning resolves this problem by updating the model parameters on-site with the real-world data, thus localizing the model to its surroundings. However, the backpropagation step during on-device learning requires the output of every layer computed during the forward pass to be stored in memory. This is usually infeasible in MCU devices as they are equipped only with a few KBs of SRAM. Given their energy limitation and the timeliness requirements, using flash memory to store the output of every layer is not practical either. Although there have been proposed a few research results to enable on-device learning under stringent memory conditions, they require the modification of the target models or the use of non-conventional gradient computation strategies. This paper proposes DaCapo, a backpropagation scheme that enables on-device learning in memory-constrained embedded systems. DaCapo stores only the output of certain layers, known as checkpoints, in SRAM, and discards the others. The discarded outputs are recomputed during backpropagation from the nearest checkpoint in front of them. In order to minimize the recomputation occurrences, DaCapo optimally plans the checkpoints to be stored in the SRAM area at a particular phase of the backpropagation and thus replaces the checkpoints stored in memory as the backpropagation progresses. We implemented the proposed scheme in an STM32F429ZI board and evaluated it with five representative DNN models. Our evaluation showed that DaCapo improved backpropagation time by up to 22% and saved energy consumption by up to 28% in comparison to AIfES, a machine learning platform optimized for MCU devices. In addition, our proposed approach enabled the training of MobileNet, which the MCU device had been previously unable to train.<\/jats:p>","DOI":"10.1145\/3609121","type":"journal-article","created":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T13:33:18Z","timestamp":1694266398000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["DaCapo: An On-Device Learning Scheme for Memory-Constrained Embedded Systems"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-0380-5678","authenticated-orcid":false,"given":"Osama","family":"Khan","sequence":"first","affiliation":[{"name":"Sungkyunkwan University, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7369-3949","authenticated-orcid":false,"given":"Gwanjong","family":"Park","sequence":"additional","affiliation":[{"name":"Sungkyunkwan University, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2103-8019","authenticated-orcid":false,"given":"Euiseong","family":"Seo","sequence":"additional","affiliation":[{"name":"Sungkyunkwan University, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,9]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2022. AIfES: Aritifical Intelligence for Embedded Systems. https:\/\/github.com\/Fraunhofer-IMS\/AIfES_for_Arduino"},{"key":"e_1_3_1_3_2","unstructured":"2022. FOMO: Faster Objects More Objects. https:\/\/docs.edgeimpulse.com\/docs\/edge-impulse-studio\/learning-blocks\/object-detection\/fomo-object-detection-for-constrained-devices"},{"key":"e_1_3_1_4_2","article-title":"MLPerf tiny benchmark","author":"Banbury Colby","year":"2021","unstructured":"Colby Banbury et\u00a0al. 2021. MLPerf tiny benchmark. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2021).","journal-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks"},{"key":"e_1_3_1_5_2","first-page":"517","article-title":"Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers","volume":"3","author":"Banbury Colby","year":"2021","unstructured":"Colby Banbury, Chuteng Zhou, Igor Fedorov, Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, and Paul Whatmough. 2021. Micronets: Neural network architectures for deploying tinyml applications on commodity microcontrollers. Proceedings of Machine Learning and Systems 3 (2021), 517\u2013532.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/NER.2019.8717169"},{"key":"e_1_3_1_7_2","first-page":"1","volume-title":"2021 IEEE Sensors Applications Symposium (SAS\u201921)","author":"Burrello Alessio","year":"2021","unstructured":"Alessio Burrello, Marcello Zanghieri, Cristian Sarti, Leonardo Ravaglia, Simone Benatt, and Luca Benini. 2021. Tackling time-variability in sEMG-based gesture recognition with on-device incremental learning and temporal convolutional networks. In 2021 IEEE Sensors Applications Symposium (SAS\u201921). IEEE, 1\u20136."},{"key":"e_1_3_1_8_2","first-page":"11285","volume-title":"Advances in Neural Information Processing Systems","author":"Cai Han","year":"2020","unstructured":"Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020. TinyTL: Reduce memory, not parameters for efficient on-device learning. In Advances in Neural Information Processing Systems, Vol. 33. 11285\u201311297."},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989752"},{"key":"e_1_3_1_10_2","article-title":"Training deep nets with sublinear memory cost","author":"Chen Tianqi","year":"2016","unstructured":"Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016).","journal-title":"arXiv preprint arXiv:1604.06174"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-79078-5_2"},{"key":"e_1_3_1_12_2","first-page":"800","article-title":"Tensorflow lite micro: Embedded machine learning for tinyml systems","volume":"3","author":"David Robert","year":"2021","unstructured":"Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Rocky Rhodes, Tiezhen Wang, and Pete Warden. 2021. Tensorflow lite micro: Embedded machine learning for tinyml systems. Proceedings of Machine Learning and Systems 3 (2021), 800\u2013811.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/iThings-GreenCom-CPSCom-SmartData-Cybermatics55523.2022.00018","volume-title":"2022 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics)","author":"Vita Fabrizio De","year":"2022","unstructured":"Fabrizio De Vita, Giorgio Nocera, Dario Bruneo, Valeria Tomaselli, and Mirko Falchetto. 2022. On-device training of deep learning models on edge microcontrollers. In 2022 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics (Cybermatics). IEEE, 62\u201369."},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3450494"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.3390\/s21092975"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.3390\/ani11020485"},{"key":"e_1_3_1_17_2","article-title":"The reversible residual network: Backpropagation without storing activations","volume":"30","author":"Gomez Aidan N.","year":"2017","unstructured":"Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. 2017. The reversible residual network: Backpropagation without storing activations. Advances in Neural Information Processing Systems 30 (2017).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_18_2","first-page":"4125","article-title":"Memory-efficient backpropagation through time","volume":"29","author":"Gruslys Audrunas","year":"2016","unstructured":"Audrunas Gruslys, R\u00e9mi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. 2016. Memory-efficient backpropagation through time. Advances in Neural Information Processing Systems 29 (2016), 4125\u20134133.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/MNET.2019.1800214"},{"key":"e_1_3_1_21_2","first-page":"1627","volume-title":"International Conference on Machine Learning","author":"Jaderberg Max","year":"2017","unstructured":"Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. 2017. Decoupled neural interfaces using synthetic gradients. In International Conference on Machine Learning. PMLR, 1627\u20131635."},{"key":"e_1_3_1_22_2","first-page":"497","volume-title":"Proceedings of the Machine Learning and Systems","volume":"2","author":"Jain Paras","year":"2020","unstructured":"Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. 2020. Checkmate: Breaking the memory wall with optimal tensor rematerialization. In Proceedings of the Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze (Eds.), Vol. 2. 497\u2013511."},{"key":"e_1_3_1_23_2","first-page":"264","volume-title":"Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics","author":"Jiang Jing","year":"2007","unstructured":"Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 264\u2013271."},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/PerComWorkshops53856.2022.9767250"},{"key":"e_1_3_1_25_2","article-title":"Efficient rematerialization for deep networks","volume":"32","author":"Kumar Ravi","year":"2019","unstructured":"Ravi Kumar, Manish Purohit, Zoya Svitkina, Erik Vee, and Joshua Wang. 2019. Efficient rematerialization for deep networks. Advances in Neural Information Processing Systems 32 (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_26_2","article-title":"A graph theoretic framework of recomputation algorithms for memory-efficient backpropagation","volume":"32","author":"Kusumoto Mitsuru","year":"2019","unstructured":"Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, and Masanori Koyama. 2019. A graph theoretic framework of recomputation algorithms for memory-efficient backpropagation. Advances in Neural Information Processing Systems 32 (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3356250.3360030"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3417313.3429382"},{"key":"e_1_3_1_29_2","article-title":"On-device training under 256KB memory","author":"Lin Ji","year":"2022","unstructured":"Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. 2022. On-device training under 256KB memory. arXiv preprint arXiv:2206.15472 (2022).","journal-title":"arXiv preprint arXiv:2206.15472"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/BIOROB.2018.8487838"},{"key":"e_1_3_1_31_2","unstructured":"Pramod Kaushik Mudrakarta Mark Sandler Andrey Zhmoginov and Andrew Howard. 2019. K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning. arxiv:1810.10703 [cs.LG]"},{"key":"e_1_3_1_32_2","first-page":"17573","volume-title":"International Conference on Machine Learning","author":"Patil Shishir G.","year":"2022","unstructured":"Shishir G. Patil, Paras Jain, Prabal Dutta, Ion Stoica, and Joseph Gonzalez. 2022. POET: Training neural networks on tiny devices with integrated rematerialization and paging. In International Conference on Machine Learning. PMLR, 17573\u201317583."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3152544"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN52387.2021.9533927"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCOM.2012.6384464"},{"key":"e_1_3_1_36_2","volume-title":"STM32F427xx, STM32F429xx Datasheet, DocID024030 Rev. 10","author":"STMicroelectronics","year":"2008","unstructured":"STMicroelectronics 2008. STM32F427xx, STM32F429xx Datasheet, DocID024030 Rev. 10. STMicroelectronics."},{"key":"e_1_3_1_37_2","volume-title":"STM32F405\/415, STM32F407\/417, STM32F427\/437 and STM32F429\/439 Advanced Arm\u00ae-based 32-bit MCUs Reference Manual, RM0090 Rev. 19","author":"STMicroelectronics","year":"2021","unstructured":"STMicroelectronics 2021. STM32F405\/415, STM32F407\/417, STM32F427\/437 and STM32F429\/439 Advanced Arm\u00ae-based 32-bit MCUs Reference Manual, RM0090 Rev. 19. STMicroelectronics. pp. 77."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3410992.3411014"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2017.2761740"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.123"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.sigpro.2009.06.001"}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3609121","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3609121","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:58Z","timestamp":1750182538000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3609121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,9]]},"references-count":40,"journal-issue":{"issue":"5s","published-print":{"date-parts":[[2023,10,31]]}},"alternative-id":["10.1145\/3609121"],"URL":"https:\/\/doi.org\/10.1145\/3609121","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"value":"1539-9087","type":"print"},{"value":"1558-3465","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,9]]},"assertion":[{"value":"2023-03-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-07-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}