{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T08:36:22Z","timestamp":1777106182002,"version":"3.51.4"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T00:00:00Z","timestamp":1684454400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2211018, 1912495, 1909004, 1714389, 1912495, 1629915, 1629129, 1763681"],"award-info":[{"award-number":["2211018, 1912495, 1909004, 1714389, 1912495, 1629915, 1629129, 1763681"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Meas. Anal. Comput. Syst."],"published-print":{"date-parts":[[2023,5,19]]},"abstract":"<jats:p>The growing adoption of hardware accelerators driven by their intelligent compiler and runtime system counterparts has democratized ML services and precipitously reduced their execution times. This motivates us to shift our attention to efficiently serve these ML services under distributed settings and characterize the overheads imposed by the RPC mechanism ('RPC tax') when serving them on accelerators. The RPC implementations designed over the years implicitly assume the host CPU services the requests, and we focus on expanding such works towards accelerator-based services. While recent proposals calling for SmartNICs to take on this task are reasonable for simple kernels, serving complex ML models requires a more nuanced view to optimize both the data-path and the control\/orchestration of these accelerators. We program today's commodity network interface cards (NICs) to split the control and data paths for effective transfer of control while efficiently transferring the payload to the accelerator. As opposed to unified approaches that bundle these paths together, limiting the flexibility in each of these paths, we design and implement SplitRPC - a control + data path optimizing RPC mechanism for ML inference serving. SplitRPC allows us to optimize the datapath to the accelerator while simultaneously allowing the CPU to maintain full orchestration capabilities. We implement SplitRPC on both commodity NICs and SmartNICs and demonstrate how GPU-based ML services running different compiler\/runtime systems can benefit. For a variety of ML models served using different inference runtimes, we demonstrate that SplitRPC is effective in minimizing the RPC tax while providing significant gains in throughput and latency over existing kernel by-pass approaches, without requiring expensive SmartNIC devices.<\/jats:p>","DOI":"10.1145\/3589974","type":"journal-article","created":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T20:01:16Z","timestamp":1684785676000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["SplitRPC: A {Control + Data} Path Splitting RPC Stack for ML Inference Serving"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-3876-0624","authenticated-orcid":false,"given":"Adithya","family":"Kumar","sequence":"first","affiliation":[{"name":"The Pennsylvania State University, University Park, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6173-687X","authenticated-orcid":false,"given":"Anand","family":"Sivasubramaniam","sequence":"additional","affiliation":[{"name":"Penn State University, University Park, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8394-8953","authenticated-orcid":false,"given":"Timothy","family":"Zhu","sequence":"additional","affiliation":[{"name":"The Pennsylvania State University, University park, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,5,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/9\/02\/C02023"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.1991.148661"},{"key":"e_1_2_1_3_1","volume-title":"bRPC Framework. https:\/\/brpc.apache.org\/. [Online","year":"2022","unstructured":"Apache. 2020. bRPC Framework. https:\/\/brpc.apache.org\/. [Online; accessed 09-Aug-2022]."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3482898.3483365"},{"key":"e_1_2_1_5_1","volume-title":"ONNX: Open Neural Network Exchange. https:\/\/github.com\/onnx\/onnx.","author":"Bai Junjie","year":"2019","unstructured":"Junjie Bai, Fang Lu, Ke Zhang, et al. 2019. ONNX: Open Neural Network Exchange. https:\/\/github.com\/onnx\/onnx."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MICRO56248.2022.00083"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3422575.3422778"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/800217.806609"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association","author":"Brunella Marco Spaziani","year":"2020","unstructured":"Marco Spaziani Brunella, Giacomo Belocchi, Marco Bonola, Salvatore Pontarelli, Giuseppe Siracusano, Giuseppe Bianchi, Aniello Cammarano, Alessandro Palumbo, Luca Petrucci, and Roberto Bifulco. 2020. hXDP: Efficient Software Packet Processing on FPGA NICs. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Boston, MA, USA, 973--990."},{"key":"e_1_2_1_10_1","unstructured":"Tianqi Chen Thierry Moreau Ziheng Jiang Lianmin Zheng Eddie Yan Haichen Shen Meghan Cowan Leyuan Wang Yuwei Hu Luis Ceze et al. 2018. {TVM}: An automated end-to-end optimizing compiler for deep learning. In 13th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 18). USENIX Association Boston MA 578--594."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001177"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS47774.2020.00029"},{"key":"e_1_2_1_13_1","unstructured":"Eric Chung Jeremy Fowers Kalin Ovtcharov Michael Papamichael Adrian Caulfield Todd Massengill Ming Liu Daniel Lo Shlomi Alkalay Michael Haselman et al. 2018. Serving dnns in real time at datacenter scale with project brainwave. iEEE Micro Vol. 38 2 (2018) 8--20."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419111.3421285"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX Association","author":"Crankshaw Daniel","year":"2017","unstructured":"Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. Clipper: A low-latency online prediction serving system. In Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX Association, Boston, MA, USA, 613--627."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931088.2931091"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419111.3421284"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association","author":"Eran Haggai","year":"2019","unstructured":"Haggai Eran, Lior Zeno, Maroun Tork, Gabi Malka, and Mark Silberstein. 2019. {NICA}: An infrastructure for inline acceleration of network applications. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association, Boston, MA, USA, 345--362."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX Association","author":"Firestone Daniel","year":"2018","unstructured":"Daniel Firestone, Andrew Putnam, Sambhrama Mundkur, Derek Chiou, Alireza Dabagh, Mike Andrewartha, Hari Angepat, Vivek Bhanu, Adrian Caulfield, Eric Chung, et al. 2018. Azure Accelerated Networking: SmartNICs in the Public Cloud. In Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX Association, Boston, MA, USA, 51--66."},{"key":"e_1_2_1_20_1","volume-title":"https:\/\/grpc.io\/. [Online","author":"Framework GRPC","year":"2022","unstructured":"Google. 2018. GRPC Framework. https:\/\/grpc.io\/. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3135974.3135993"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association","author":"Gujarati Arpan","year":"2020","unstructured":"Arpan Gujarati, Reza Karimi, Safya Alzayat, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Boston, MA, USA. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/gujarati"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/InPar.2012.6339596"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS.2019.00012"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274808.3274813"},{"key":"e_1_2_1_26_1","volume-title":"ONNX Runtime inference engine. https:\/\/github.com\/Microsoft\/onnxruntime\/. [Online","author":"Microsoft Inc. 2016.","year":"2022","unstructured":"Microsoft Inc. 2016. ONNX Runtime inference engine. https:\/\/github.com\/Microsoft\/onnxruntime\/. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_27_1","unstructured":"DPDK Intel. 2014. Data plane development kit."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359630"},{"key":"e_1_2_1_29_1","volume-title":"GPU Technology Conference Presentation","volume":"338","author":"Jones Stephen","year":"2012","unstructured":"Stephen Jones. 2012. Introduction to dynamic parallelism. In GPU Technology Conference Presentation, Vol. 338. NVIDIA, Santa Clara, CA, 2012."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX Association","author":"Kalia Anuj","year":"2019","unstructured":"Anuj Kalia, Michael Kaminsky, and David Andersen. 2019. Datacenter {RPCs} can be General and Fast. In Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX Association, Boston, MA, USA, 1--16."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3466752.3480051"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483565"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association","author":"Kogias Marios","year":"2019","unstructured":"Marios Kogias, George Prekas, Adrien Ghosn, Jonas Fietz, and Edouard Bugnion. 2019. {R2P2}: Making {RPCs} first-class datacenter citizens. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association, Boston, MA, USA, 863--880."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3445814.3446696"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association","author":"Lee Yunseong","year":"2018","unstructured":"Yunseong Lee, Alberto Scolari, Byung-Gon Chun, Marco Domenico Santambrogio, Markus Weimer, and Matteo Interlandi. 2018. {PRETZEL}: Opening the Black Box of Machine Learning Prediction Serving Systems. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Boston, MA, USA, 611--626."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341302.3342079"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association","author":"Liu Ming","year":"2019","unstructured":"Ming Liu, Simon Peter, Arvind Krishnamurthy, and Phitchaya Mangpo Phothilimthana. 2019b. E3: Energy-Efficient Microservices on SmartNIC-Accelerated Servers. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association, Renton, WA, 363--378. https:\/\/www.usenix.org\/conference\/atc19\/presentation\/liu-ming"},{"key":"e_1_2_1_38_1","volume-title":"2019 {USENIX} Annual Technical Conference (USENIX ATC 19)","author":"Liu Yizhi","unstructured":"Yizhi Liu, Yao Wang, Ruofei Yu, Mu Li, Vin Sharma, and Yida Wang. 2019c. Optimizing {CNN} Model Inference on CPUs. In 2019 {USENIX} Annual Technical Conference (USENIX ATC 19). USENIX Association, Boston, MA, 1025--1040."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association","author":"Ma Lingxiao","year":"2020","unstructured":"Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. Rammer: Enabling Holistic Deep Learning Compiler Optimizations with rTasks. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Boston, MA, USA, 881--897. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/ma"},{"key":"e_1_2_1_40_1","volume-title":"https:\/\/github.com\/Mellanox\/libvma. [Online","author":"LIBVMA.","year":"2022","unstructured":"Mellanox. 2022a. LIBVMA. https:\/\/github.com\/Mellanox\/libvma. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_41_1","volume-title":"Mellanox NV Peer memory. https:\/\/github.com\/Mellanox\/nv_peer_memory. [Online","year":"2022","unstructured":"Mellanox. 2022b. Mellanox NV Peer memory. https:\/\/github.com\/Mellanox\/nv_peer_memory. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454083"},{"key":"e_1_2_1_43_1","volume-title":"CUDA GPUDirect RDMA. https:\/\/docs.nvidia.com\/cuda\/gpudirect-rdma\/index.html. [Online","author":"NVIDIA.","year":"2022","unstructured":"NVIDIA. 2022a. CUDA GPUDirect RDMA. https:\/\/docs.nvidia.com\/cuda\/gpudirect-rdma\/index.html. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_44_1","volume-title":"CUDA Graphs API. https:\/\/docs.nvidia.com\/cuda\/cuda-runtime-api\/group__CUDART__GRAPH.html. [Online","author":"NVIDIA.","year":"2022","unstructured":"NVIDIA. 2022b. CUDA Graphs API. https:\/\/docs.nvidia.com\/cuda\/cuda-runtime-api\/group__CUDART__GRAPH.html. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_45_1","volume-title":"CUDA VMM Aliasing. https:\/\/docs.nvidia.com\/cuda\/cuda-c-programming-guide\/index.html##virtual-aliasing-support. [Online","author":"NVIDIA.","year":"2022","unstructured":"NVIDIA. 2022c. CUDA VMM Aliasing. https:\/\/docs.nvidia.com\/cuda\/cuda-c-programming-guide\/index.html##virtual-aliasing-support. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_46_1","unstructured":"Christopher Olston Noah Fiedel Kiril Gorovoy Jeremiah Harmsen Li Lao Fangwei Li Vinu Rajashekhar Sukriti Ramesh and Jordan Soyke. 2017. TensorFlow-Serving: Flexible High-Performance ML Serving. arxiv: 1712.06139 [cs.DC]"},{"key":"e_1_2_1_47_1","volume-title":"Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15)","author":"Ousterhout Kay","year":"2015","unstructured":"Kay Ousterhout, Ryan Rasti, Sylvia Ratnasamy, Scott Shenker, and Byung-Gon Chun. 2015. Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). USENIX Association, Oakland, CA, 293--307. https:\/\/www.usenix.org\/conference\/nsdi15\/technical-sessions\/presentation\/ousterhout"},{"key":"e_1_2_1_48_1","volume-title":"Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems. Curran Associates","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems. Curran Associates, Inc., NY, USA, 8026--8037."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association","author":"Phothilimthana Phitchaya Mangpo","year":"2018","unstructured":"Phitchaya Mangpo Phothilimthana, Ming Liu, Antoine Kaufmann, Simon Peter, Rastislav Bodik, and Thomas Anderson. 2018. Floem: A Programming System for NIC-Accelerated Network Applications. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Carlsbad, CA, 663--679. https:\/\/www.usenix.org\/conference\/osdi18\/presentation\/phothilimthana"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association","author":"Pirelli Solal","year":"2020","unstructured":"Solal Pirelli and George Candea. 2020. A Simpler and Faster {NIC} Driver Model for Network Functions. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Boston, MA, USA, 225--241."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503222.3507711"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the International Symposium on Microarchitecture (MICRO). IEEE","author":"Zarandi Arash Pourhabibi","year":"2021","unstructured":"Arash Pourhabibi Zarandi, Mark Johnathon Sutherland, Alexandros Daglis, and Babak Falsafi. 2021. Cerebros: Evading the RPC Tax in Datacenters. In Proceedings of the International Symposium on Microarchitecture (MICRO). IEEE, New York, NY, USA, -."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458336.3465287"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3457388.3458661"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA45697.2020.00045"},{"key":"e_1_2_1_56_1","volume-title":"INFaaS: A Model-less and Managed Inference Serving System. arxiv","author":"Romero Francisco","year":"1905","unstructured":"Francisco Romero, Qian Li, Neeraja J. Yadwadkar, and Christos Kozyrakis. 2020. INFaaS: A Model-less and Managed Inference Serving System. arxiv: 1905.13348 [cs.DC]"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483555"},{"key":"e_1_2_1_58_1","volume-title":"USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)","author":"Shashidhara Rajath","year":"2022","unstructured":"Rajath Shashidhara, Tim Stamler, Antoine Kaufmann, and Simon Peter. 2022. {FlexTOE}: Flexible {TCP} Offload with {Fine-Grained} Parallelism. In USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). USENIX, Bostom, MA, USA, 87--102."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359658"},{"key":"e_1_2_1_60_1","volume-title":"Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference. arxiv","author":"Shen Haichen","year":"2021","unstructured":"Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, and Yida Wang. 2021. Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference. arxiv: 2006.03031 [cs.PL]"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2963098"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA45697.2020.00027"},{"key":"e_1_2_1_63_1","volume-title":"Tensorflow Frozen Graph. https:\/\/docs.snap.com\/lens-studio\/references\/guides\/lens-features\/machine-learning\/ml-frameworks\/export-from-tensorflow\/. [Online","year":"2022","unstructured":"Tensorflow. 2022. Tensorflow Frozen Graph. https:\/\/docs.snap.com\/lens-studio\/references\/guides\/lens-features\/machine-learning\/ml-frameworks\/export-from-tensorflow\/. [Online; accessed 17-Apr-2022]."},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of The Web Conference (WWW). Association for Computing Machinery","author":"Narayanan Iyswarya","year":"2020","unstructured":"textbfKumar, A, Iyswarya Narayanan, Timothy Zhu, and Anand Sivasubramaniam. 2020. The Fast and The Frugal: Tail latency aware provisioning for coping with load variations. In Proceedings of The Web Conference (WWW). Association for Computing Machinery, New York, NY, USA, 314--326."},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 15th ACM International Systems and Storage Conference (SYSTOR ). Association for Computing Machinery","author":"Sivasubramaniam Anand","year":"2022","unstructured":"textbfKumar, A, Anand Sivasubramaniam, and Timothy Zhu. 2022. Overflowing Emerging Neural Network Inference Tasks from the GPU to the CPU on Heterogeneous Servers. In Proceedings of the 15th ACM International Systems and Storage Conference (SYSTOR ). Association for Computing Machinery, New York, NY, USA."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378528"},{"key":"e_1_2_1_67_1","volume-title":"Wikibench: A distributed, wikipedia based web application benchmark. Master's thesis","author":"van Baaren Erik-Jan","year":"2009","unstructured":"Erik-Jan van Baaren. 2009. Wikibench: A distributed, wikipedia based web application benchmark. Master's thesis, VU University Amsterdam, Vol. 1 (2009)."},{"key":"e_1_2_1_68_1","unstructured":"Han Vanholder. 2016. Efficient Inference with TensorRT."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3492321.3519569"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458336.3465283"},{"key":"e_1_2_1_72_1","volume-title":"Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications. arxiv","author":"Yu Peifeng","year":"2019","unstructured":"Peifeng Yu and Mosharaf Chowdhury. 2019. Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications. arxiv: 1902.04610 [cs.DC]"},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association","author":"Zhang Chengliang","year":"2019","unstructured":"Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. Mark: Exploiting cloud services for cost-effective, slo-aware machine learning inference serving. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association, Boston, MA, USA."},{"key":"e_1_2_1_74_1","volume-title":"12th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 20)","author":"Zhang Jeff","unstructured":"Jeff Zhang, Sameh Elnikety, Shuayb Zarar, Atul Gupta, and Siddharth Garg. 2020. Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-Service Systems. In 12th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 20). USENIX Association, Boston, MA, USA."},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association","author":"Zhang Minjia","year":"2018","unstructured":"Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, and Yuxiong He. 2018. Deepcpu: Serving rnn-based deep learning models 10x faster. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association, Boston, MA, 951--965. https:\/\/www.usenix.org\/conference\/atc18\/presentation\/zhang-minjia"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503222.3507721"},{"key":"e_1_2_1_77_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association","author":"Zheng Lianmin","year":"2020","unstructured":"Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. 2020. Ansor: Generating High-Performance Tensor Programs for Deep Learning. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Boston, MA, USA, 863--879. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/zheng"}],"container-title":["Proceedings of the ACM on Measurement and Analysis of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3589974","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3589974","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3589974","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:15Z","timestamp":1750183755000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3589974"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,19]]},"references-count":76,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,5,19]]}},"alternative-id":["10.1145\/3589974"],"URL":"https:\/\/doi.org\/10.1145\/3589974","relation":{},"ISSN":["2476-1249"],"issn-type":[{"value":"2476-1249","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,19]]},"assertion":[{"value":"2023-05-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}