{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,11]],"date-time":"2025-12-11T07:42:52Z","timestamp":1765438972713,"version":"3.41.0"},"reference-count":84,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,6,29]],"date-time":"2024-06-29T00:00:00Z","timestamp":1719619200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Hellenic Foundation for Research and Innovation (HFRI) under the 3rd Call for HFRI PhD Fellowships","award":["5578"],"award-info":[{"award-number":["5578"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>\n            The relentless expansion of deep learning applications in recent years has prompted a pivotal shift toward on-device execution, driven by the urgent need for real-time processing, heightened privacy concerns, and reduced latency across diverse domains. This article addresses the challenges inherent in optimising the execution of deep neural networks (DNNs) on mobile devices, with a focus on device heterogeneity, multi-DNN execution, and dynamic runtime adaptation. We introduce\n            <jats:monospace>CARIn<\/jats:monospace>\n            , a novel framework designed for the optimised deployment of both single- and multi-DNN applications under user-defined service-level objectives. Leveraging an expressive multi-objective optimisation framework and a runtime-aware sorting and search algorithm (\n            <jats:monospace>RASS<\/jats:monospace>\n            ) as the MOO solver,\n            <jats:monospace>CARIn<\/jats:monospace>\n            facilitates efficient adaptation to dynamic conditions while addressing resource contention issues associated with multi-DNN execution. Notably,\n            <jats:monospace>RASS<\/jats:monospace>\n            generates a set of configurations, anticipating subsequent runtime adaptation, ensuring rapid, low-overhead adjustments in response to environmental fluctuations. Extensive evaluation across diverse tasks, including text classification, scene recognition, and face analysis, showcases the versatility of\n            <jats:monospace>CARIn<\/jats:monospace>\n            across various model architectures, such as Convolutional Neural Networks and Transformers, and realistic use cases. We observe a substantial enhancement in the fair treatment of the problem\u2019s objectives, reaching 1.92\u00d7 when compared to single-model designs and up to 10.69\u00d7 in contrast to the state-of-the-art OODIn framework. Additionally, we achieve a significant gain of up to 4.06\u00d7 over hardware-unaware designs in multi-DNN applications. Finally, our framework sustains its performance while effectively eliminating the time overhead associated with identifying the optimal design in response to environmental challenges.\n          <\/jats:p>","DOI":"10.1145\/3665868","type":"journal-article","created":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T11:22:53Z","timestamp":1716463373000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads"],"prefix":"10.1145","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-5364-4410","authenticated-orcid":false,"given":"Ioannis","family":"Panopoulos","sequence":"first","affiliation":[{"name":"National Technical University of Athens, Zografou, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5181-6251","authenticated-orcid":false,"given":"Stylianos","family":"Venieris","sequence":"additional","affiliation":[{"name":"Samsung AI Center-Cambridge, London, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3011-3746","authenticated-orcid":false,"given":"Iakovos","family":"Venieris","sequence":"additional","affiliation":[{"name":"National Technical University of Athens, Zografou Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,6,29]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3325413.3329793"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3487552.3487863"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01123"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.23919\/DATE56975.2023.10137095"},{"key":"e_1_3_2_6_2","first-page":"1877","article-title":"Language models are few-shot learners","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS'20). Curran Associates, Inc., 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS'20)"},{"key":"e_1_3_2_7_2","volume-title":"Proceedings of the 8th International Conference on Learning Representations (ICLR\u201920)","author":"Cai Han","year":"2020","unstructured":"Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2020. Once-for-all: Train one network and specialize it for efficient deployment. In Proceedings of the 8th International Conference on Learning Representations (ICLR\u201920). OpenReview.net."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2023.3265198"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/PERCOM50583.2021.9439111"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01166"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00098"},{"key":"e_1_3_2_12_2","volume-title":"Proceedings of the 7th International Conference on Learning Representations (ICLR\u201919)","author":"Elsken Thomas","year":"2019","unstructured":"Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Efficient multi-objective neural architecture search via lamarckian evolution. In Proceedings of the 7th International Conference on Learning Representations (ICLR\u201919). OpenReview.net."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2008.44"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613424.3614263"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2017.7952261"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1080\/23311916.2018.1502242"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM53939.2023.10228928"},{"key":"e_1_3_2_18_2","first-page":"705","volume-title":"Proceedings of the 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201921)","author":"Guo Peizhen","year":"2021","unstructured":"Peizhen Guo, Bo Hu, and Wenjun Hu. 2021. Mistify: Automating DNN model porting for on-device inference at the edge. In Proceedings of the 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201921), James Mickens and Renata Teixeira (Eds.). USENIX Association, 705\u2013719."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58574-7_27"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/PACT52795.2021.00014"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447993.3483249"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2021.3124306"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00447"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1613\/JAIR.1.14139"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3498361.3538948"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3498361.3538932"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/DAC56929.2023.10247989"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2022.3218991"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/1145\/3302424.3303950"},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the International Conference on Computer-Aided Design (ICCAD\u201923)","author":"Kouris Alexandros","year":"2023","unstructured":"Alexandros Kouris, Stylianos I. Venieris, Stefanos Laskaridis, and Nicholas D. Lane. 2023. Fluid batching: Exit-aware preemptive serving of early-exit neural networks on edge NPUs. In Proceedings of the International Conference on Computer-Aided Design (ICCAD\u201923)."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/EDGE60047.2023.00017"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520134"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_2_34_2","unstructured":"Stefanos Laskaridis Kleomenis Kateveas Lorenzo Minto and Hamed Haddadi. 2024. MELTing point: Mobile evaluation of language transformers. Retrieved from https:\/\/arxiv.org\/abs\/2403.12844"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3400302.3415698"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2021.3134843"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3386901.3388947"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485730.3485938"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448125"},{"key":"e_1_3_2_40_2","volume-title":"Proceedings of the 10th International Conference on Learning Representations (ICLR\u201922)","author":"Mehta Sachin","year":"2022","unstructured":"Sachin Mehta and Mohammad Rastegari. 2022. MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer. In Proceedings of the 10th International Conference on Learning Representations (ICLR\u201922). OpenReview.net."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528692"},{"key":"e_1_3_2_42_2","unstructured":"Subhabrata Mukherjee Ahmed Hassan Awadallah and Jianfeng Gao. 2021. Xtreme Distil Transformers: Task Transfer for Task-agnostic Distillation. arxiv:2106.04563 [cs.CL]. Retrieved from https:\/\/arxiv.org\/abs\/2106.04563"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISCC58397.2023.10217850"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/MLCAD52597.2021.9531281"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11831-021-09663-x"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206537"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01044"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00474"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33016949"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/d18-1404"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460352"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11831-022-09778-9"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/IWQOS57198.2023.10188769"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/MDAT.2020.2982629"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.ACL-MAIN.195"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00293"},{"key":"e_1_3_2_58_2","first-page":"6105","volume-title":"Proceedings of the 36th International Conference on Machine Learning (ICML\u201919),","volume":"97","author":"Tan Mingxing","year":"2019","unstructured":"Mingxing Tan and Quoc V. Le. 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML\u201919), Proceedings of Machine Learning Research, Vol. 97, Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 6105\u20136114. http:\/\/proceedings.mlr.press\/v97\/tan19a.html"},{"key":"e_1_3_2_59_2","first-page":"10096","volume-title":"Proceedings of the 38th International Conference on Machine Learning (ICML'21), Proceedings Machine Learning Research","volume":"139","author":"Tan Mingxing","year":"2021","unstructured":"Mingxing Tan and Quoc V. Le. 2021. EfficientNetV2: Smaller models and faster training. In Proceedings of the 38th International Conference on Machine Learning (ICML'21), Proceedings Machine Learning Research, Marina Meila and Tong Zhang (Eds.). Vol. 139, PMLR, 10096\u201310106. http:\/\/proceedings.mlr.press\/v139\/tan21a.html"},{"key":"e_1_3_2_60_2","article-title":"LLaMa: Open and efficient foundation language models","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, et\u00a0al. 2023. LLaMa: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971","journal-title":"arXiv:2302.13971"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2022.3176845"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMARTCOMP52413.2021.00021"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2019.2944584"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCI.2021.3084393"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/1145\/3570361.3592529"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01099"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2019.00048"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3464298.3493391"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-19803-8_7"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313591"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2021.3107424"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3627703.3650074"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_18"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372224.3419192"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCAD51958.2021.9643501"},{"key":"e_1_3_2_76_2","article-title":"A survey of multi-tenant deep learning inference on GPU","author":"Yu Fuxun","year":"2022","unstructured":"Fuxun Yu, Di Wang, Longfei Shangguan, Minjia Zhang, Chenchen Liu, and Xiang Chen. 2022. A survey of multi-tenant deep learning inference on GPU. MLSys'22 Workshop on Cloud Intelligence\/AIOps. arXiv:2203.09040. Retrieved from https:\/\/arxiv.org\/abs\/2203.09040","journal-title":"MLSys'22 Workshop on Cloud Intelligence\/AIOps"},{"key":"e_1_3_2_77_2","first-page":"521","volume-title":"Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201922)","author":"Yu Gyeong-In","year":"2022","unstructured":"Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. ORCA: A distributed serving system for transformer-based generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201922). USENIX Association, 521\u2013538."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00189"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2023.3283780"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458864.3467882"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00354"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.463"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2021.3070203"},{"key":"e_1_3_2_84_2","doi-asserted-by":"crossref","unstructured":"Ziyang Zhang Huan Li Yang Zhao Changyao Lin and Jie Liu. 2023. BCEdge: SLO-aware DNN inference services with adaptive batching on edge platforms. arXiv:2305.01519. Retrieved from https:\/\/arxiv.org\/abs\/2305.01519","DOI":"10.36227\/techrxiv.22724099"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-48424-7_18"}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3665868","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3665868","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T23:44:27Z","timestamp":1750290267000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3665868"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,29]]},"references-count":84,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3665868"],"URL":"https:\/\/doi.org\/10.1145\/3665868","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"type":"print","value":"1539-9087"},{"type":"electronic","value":"1558-3465"}],"subject":[],"published":{"date-parts":[[2024,6,29]]},"assertion":[{"value":"2023-11-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}