{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T11:36:50Z","timestamp":1776166610473,"version":"3.50.1"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2026,5,31]]},"abstract":"<jats:p>\n                    The rapid evolution of Artificial Intelligence (AI) and Machine Learning (ML) has significantly heightened computational demands, particularly for inference-serving workloads. While traditional cloud-based deployments offer scalability, they face challenges such as network congestion, high energy consumption, and privacy concerns. In contrast, edge computing provides low-latency and sustainable alternatives but is constrained by limited computational resources. In this work, we introduce\n                    <jats:monospace>SynergAI<\/jats:monospace>\n                    , a novel framework designed for performance- and architecture-aware inference serving across heterogeneous edge-to-cloud infrastructures. Built upon a comprehensive performance characterization of modern inference engines,\n                    <jats:monospace>SynergAI<\/jats:monospace>\n                    integrates a combination of offline and online decision-making policies to deliver intelligent, lightweight, and architecture-aware scheduling. By dynamically allocating workloads across diverse hardware architectures, it effectively minimizes Quality of Service (QoS) violations. We implement\n                    <jats:monospace>SynergAI<\/jats:monospace>\n                    within a Kubernetes-based ecosystem and evaluate its efficiency. Our results demonstrate that architecture-driven inference serving enables optimized and architecture-aware deployments on emerging hardware platforms, achieving an average reduction of 2.4\u00d7 in QoS violations compared to a State-of-the-Art (SotA) solution.\n                  <\/jats:p>","DOI":"10.1145\/3800959","type":"journal-article","created":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T21:01:22Z","timestamp":1773867682000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["SynergAI: Edge-to-Cloud Synergy for Architecture-Driven High-Performance Orchestration"],"prefix":"10.1145","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-9695-8905","authenticated-orcid":false,"given":"Foteini","family":"Stathopoulou","sequence":"first","affiliation":[{"name":"Electrical and Computer Engineering, National Technical University of Athens","place":["Zografou, Greece"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9188-2930","authenticated-orcid":false,"given":"Aggelos","family":"Ferikoglou","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, National Technical University of Athens","place":["Zografou, Greece"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8116-3503","authenticated-orcid":false,"given":"Manolis","family":"Katsaragakis","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, National Technical University of Athens","place":["Zografou, Greece"]},{"name":"Katholieke Universiteit Leuven","place":["Zografou, Greece"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6147-6908","authenticated-orcid":false,"given":"Dimosthenis","family":"Masouros","sequence":"additional","affiliation":[{"name":"National Technical University of Athens School of Electrical and Computer Engineering","place":["Athens, Greece"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3151-2730","authenticated-orcid":false,"given":"Sotirios","family":"Xydis","sequence":"additional","affiliation":[{"name":"School of Electrical and Computer Engineering, National Technical University of Athens","place":["Athens, Greece"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6930-6847","authenticated-orcid":false,"given":"Dimitrios","family":"Soudris","sequence":"additional","affiliation":[{"name":"ECE, NTUA","place":["Athens, Greece"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,4,14]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"2017. Amazon SageMaker. Retrieved March 24 2025 fromhttps:\/\/aws.amazon.com\/sagemaker\/"},{"key":"e_1_3_2_3_2","unstructured":"2018. Azure Machine Learning. Retrieved March 24 2025 fromhttps:\/\/azure.microsoft.com\/en-us\/products\/machine-learning"},{"key":"e_1_3_2_4_2","unstructured":"2021. NVidia AI Enterprise. Retrieved March 24 2025 fromhttps:\/\/www.nvidia.com\/en-eu\/data-center\/products\/ai-enterprise\/"},{"key":"e_1_3_2_5_2","unstructured":"2018. ONNX Runtime. Retrieved March 25 2025 from https:\/\/onnxruntime.ai\/"},{"key":"e_1_3_2_6_2","unstructured":"2017. Tensorflow Lite. Retrieved March 25 2025 from https:\/\/github.com\/tensorflow\/tflite-micro"},{"key":"e_1_3_2_7_2","unstructured":"2016. Tensorflow Serving. Retrieved March 25 2025 from https:\/\/www.tensorflow.org\/tfx\/guide\/serving"},{"key":"e_1_3_2_8_2","unstructured":"2018. Triton Inference Server. Retrieved March 25 2025 fromhttps:\/\/developer.nvidia.com\/nvidia-triton-inference-server"},{"key":"e_1_3_2_9_2","unstructured":"2021. Vertex AI. Retrieved March 24 2025 fromhttps:\/\/cloud.google.com\/vertex-ai"},{"issue":"1","key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"2467","DOI":"10.1038\/s41598-022-06218-3","article-title":"A proficient approach to forecast COVID-19 spread via optimized dynamic machine learning models","volume":"12","author":"Alali Yasminah","year":"2022","unstructured":"Yasminah Alali, Fouzi Harrou, and Ying Sun. 2022. A proficient approach to forecast COVID-19 spread via optimized dynamic machine learning models. Scientific Reports 12, 1 (2022), 2467.","journal-title":"Scientific Reports"},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/DSD60849.2023.00015","volume-title":"Proceedings of the 2023 26th Euromicro Conference on Digital System Design (DSD)","author":"Archet Agathe","year":"2023","unstructured":"Agathe Archet, Nicolas Ventroux, Nicolas Gac, and Fran\u00e7ois Orieux. 2023. Energy-efficient use of an embedded heterogeneous SoC for the inference of CNNs. In Proceedings of the 2023 26th Euromicro Conference on Digital System Design (DSD). IEEE, 30\u201338."},{"key":"e_1_3_2_12_2","unstructured":"Lichang Chen Shiyang Li Jun Yan Hai Wang Kalpa Gunaratna Vikas Yadav Zheng Tang Vijay Srinivasan Tianyi Zhou Heng Huang et\u00a0al. 2023. Alpagasus: Training a better alpaca with fewer data. arXiv:2307.08701. Retrieved from https:\/\/arxiv.org\/abs\/2307.08701"},{"key":"e_1_3_2_13_2","first-page":"1","article-title":"Pearson correlation coefficient","author":"Cohen Israel","year":"2009","unstructured":"Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise Reduction in Speech Processing (2009), 1\u20134.","journal-title":"Noise Reduction in Speech Processing"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/2499368.2451125"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2644865.2541941"},{"key":"e_1_3_2_16_2","first-page":"200","volume-title":"Proceedings of the 2019 IEEE International Conference on Autonomic Computing (ICAC)","author":"Ding Jianru","year":"2019","unstructured":"Jianru Ding, Ruiqi Cao, Indrajeet Saravanan, Nathaniel Morris, and Christopher Stewart. 2019. Characterizing service level objectives for cloud services: Realities and myths. In Proceedings of the 2019 IEEE International Conference on Autonomic Computing (ICAC). IEEE, 200\u2013206."},{"key":"e_1_3_2_17_2","first-page":"1","volume-title":"Proceedings of the 2023 IEEE 16th International Conference on Cloud Computing (CLOUD)","author":"Ferikoglou Aggelos","year":"2023","unstructured":"Aggelos Ferikoglou, Panos Chrysomeris, Achilleas Tzenetopoulos, Manolis Katsaragakis, Dimosthenis Masouros, and Dimitrios Soudris. 2023. IRIS: Interference and resource aware predictive orchestration for ML inference serving. In Proceedings of the 2023 IEEE 16th International Conference on Cloud Computing (CLOUD). IEEE, 1\u201312."},{"key":"e_1_3_2_18_2","first-page":"4","volume-title":"Proceedings of the 13th EuroSys Conference","author":"Garefalakis Panagiotis","year":"2018","unstructured":"Panagiotis Garefalakis, Konstantinos Karanasos, Peter Pietzuch, Arun Suresh, and Sriram Rao. 2018. M edea: Scheduling of long running applications in shared production clusters. In Proceedings of the 13th EuroSys Conference. ACM, 4."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10614-021-10094-w"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3361525.3361543"},{"key":"e_1_3_2_21_2","article-title":"Resource-efficient DNN inference with early exiting in serverless edge computing","author":"Guo Xiaolin","year":"2024","unstructured":"Xiaolin Guo, Fang Dong, Dian Shen, Zhaowu Huang, and Jinghui Zhang. 2024. Resource-efficient DNN inference with early exiting in serverless edge computing. IEEE Transactions on Mobile Computing (2024).","journal-title":"IEEE Transactions on Mobile Computing"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2024.3415661"},{"key":"e_1_3_2_23_2","first-page":"16","article-title":"The era of cognitive systems: An inside look at IBM Watson and how it works","volume":"1","author":"High Rob","year":"2012","unstructured":"Rob High. 2012. The era of cognitive systems: An inside look at IBM Watson and how it works. IBM Corporation, Redbooks 1 (2012), 16.","journal-title":"IBM Corporation, Redbooks"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-57077-4_10"},{"key":"e_1_3_2_25_2","unstructured":"Hakan Inan Kartikeya Upasani Jianfeng Chi Rashi Rungta Krithika Iyer Yuning Mao Michael Tontchev Qing Hu Brian Fuller Davide Testuggine et\u00a0al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv:2312.06674. Retrieved from https:\/\/arxiv.org\/abs\/2312.06674"},{"key":"e_1_3_2_26_2","unstructured":"Intel. 2025. Intel Xeon Specs. Retrieved June 28 2025 fromhttps:\/\/www.intel.com\/content\/www\/us\/en\/products\/sku\/86067\/intel-xeon-processor-e52658a-v3-30m-cache-2-20-ghz\/specifications.html"},{"key":"e_1_3_2_27_2","unstructured":"Intel. 2025. Intel Xeon Specs. Retrieved March 28 2026 fromhttps:\/\/www.intel.com\/content\/www\/us\/en\/products\/sku\/86067\/intel-xeon-processor-e52658a-v3-30m-cache-2-20-ghz\/specifications.html"},{"key":"e_1_3_2_28_2","first-page":"1","volume-title":"Proceedings of the 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)","author":"Kakolyris Andreas Kosmas","year":"2023","unstructured":"Andreas Kosmas Kakolyris, Manolis Katsaragakis, Dimosthenis Masouros, and Dimitrios Soudris. 2023. RoaD-RuNNer: Collaborative DNN partitioning and offloading on heterogeneous edge systems. In Proceedings of the 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 1\u20136."},{"issue":"32","key":"e_1_3_2_29_2","article-title":"Machine learning in computer vision: A review.","volume":"8","author":"Khan Abdullah Ayub","year":"2021","unstructured":"Abdullah Ayub Khan, Asif Ali Laghari, and Shafique Ahmed Awan. 2021. Machine learning in computer vision: A review. EAI Endorsed Transactions on Scalable Information Systems 8, 32 (2021), 1\u201311.","journal-title":"EAI Endorsed Transactions on Scalable Information Systems"},{"key":"e_1_3_2_30_2","unstructured":"Kubernetes Project. 2024. Kubernetes: Production-Grade Container Orchestration. Retrieved March 28 2025 from https:\/\/kubernetes.io"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSAC.2021.3126076"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TWC.2019.2946140"},{"key":"e_1_3_2_33_2","first-page":"409","volume-title":"Proceedings of the 2021 58th ACM\/IEEE Design Automation Conference (DAC)","author":"Li Min","year":"2021","unstructured":"Min Li, Yu Li, Ye Tian, Li Jiang, and Qiang Xu. 2021. AppealNet: An efficient and highly-accurate edge\/cloud collaborative architecture for DNN inference. In Proceedings of the 2021 58th ACM\/IEEE Design Automation Conference (DAC). IEEE, 409\u2013414."},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","first-page":"6204","DOI":"10.1109\/ICPR48806.2021.9413186","volume-title":"Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR)","author":"Limonova Elena","year":"2021","unstructured":"Elena Limonova, Daniil Alfonso, Dmitry Nikolaev, and Vladimir V. Arlazarov. 2021. ResNet-like architecture with low hardware requirements. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 6204\u20136211."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1109\/HPCA47549.2020.00025","volume-title":"Proceedings of the 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)","author":"Patel Tirthak","year":"2020","unstructured":"Tirthak Patel and Devesh Tiwari. 2020. Clite: Efficient and qos-aware co-location of multiple latency-critical jobs for warehouse scale computers. In Proceedings of the 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 193\u2013206."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3234150"},{"key":"e_1_3_2_37_2","unstructured":"PyTorch. 2024. Int8 Quantization for x86 CPU in PyTorch. Retrieved March 28 2025 from https:\/\/pytorch.org\/blog\/int8-quantization\/?utm_source=chatgpt.com"},{"key":"e_1_3_2_38_2","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et\u00a0al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML)."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660605.3660941"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA45697.2020.00045"},{"key":"e_1_3_2_41_2","first-page":"1","volume-title":"Proceedings of the 2022 IEEE High Performance Extreme Computing Conference (HPEC)","author":"Reuther Albert","year":"2022","unstructured":"Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner. 2022. AI and ML accelerator survey and trends. In Proceedings of the 2022 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 1\u201310."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460352"},{"key":"e_1_3_2_43_2","article-title":"Applications of artificial intelligence (AI) in healthcare: A review","author":"Shaheen Mohammed Yousef","year":"2021","unstructured":"Mohammed Yousef Shaheen. 2021. Applications of artificial intelligence (AI) in healthcare: A review. ScienceOpen Preprints (2021).","journal-title":"ScienceOpen Preprints"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520129"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12083-021-01223-1"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.iotcps.2023.02.004"},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1007\/978-3-031-04580-6_8","volume-title":"Embedded Computer Systems: Architectures, Modeling, and Simulation: 21st International Conference, SAMOS 2021, Virtual Event, July 4\u20138, 2021, Proceedings","author":"Tzenetopoulos Achilleas","year":"2022","unstructured":"Achilleas Tzenetopoulos, Dimosthenis Masouros, Sotirios Xydis, and Dimitrios Soudris. 2022. Interference-aware workload placement for improving latency distribution of converged HPC\/Big Data cloud infrastructures. In Embedded Computer Systems: Architectures, Modeling, and Simulation: 21st International Conference, SAMOS 2021, Virtual Event, July 4\u20138, 2021, Proceedings. Springer, 108\u2013123."},{"key":"e_1_3_2_48_2","unstructured":"Mart Van Baalen Andrey Kuzmin Suparna S. Nair Yuwei Ren Eric Mahurin Chirag Patel Sundar Subramanian Sanghyuk Lee Markus Nagel Joseph Soriaga et\u00a0al. 2023. FP8 versus INT8 for efficient deep learning inference. arXiv:2303.17951. Retrieved from https:\/\/arxiv.org\/abs\/2303.17951"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSAC.2022.3180772"},{"key":"e_1_3_2_50_2","unstructured":"Yu Emma Wang Gu-Yeon Wei and David Brooks. 2019. Benchmarking TPU GPU and CPU platforms for deep learning. arXiv:1907.10701. Retrieved from https:\/\/arxiv.org\/abs\/1907.10701"},{"issue":"2","key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"864","DOI":"10.1109\/TCOMM.2022.3229033","article-title":"Reinforcement learning based energy-efficient collaborative inference for mobile edge computing","volume":"71","author":"Xiao Yilin","year":"2022","unstructured":"Yilin Xiao, Liang Xiao, Kunpeng Wan, Helin Yang, Yi Zhang, Yi Wu, and Yanyong Zhang. 2022. Reinforcement learning based energy-efficient collaborative inference for mobile edge computing. IEEE Transactions on Communications 71, 2 (2022), 864\u2013876.","journal-title":"IEEE Transactions on Communications"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TGCN.2021.3111731"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2021.3116597"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3009876"}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3800959","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T10:27:35Z","timestamp":1776162455000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3800959"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,14]]},"references-count":53,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,5,31]]}},"alternative-id":["10.1145\/3800959"],"URL":"https:\/\/doi.org\/10.1145\/3800959","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"value":"1539-9087","type":"print"},{"value":"1558-3465","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,14]]},"assertion":[{"value":"2025-07-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}