{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,24]],"date-time":"2025-08-24T01:36:35Z","timestamp":1755999395100,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":67,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,6,6]],"date-time":"2022-06-06T00:00:00Z","timestamp":1654473600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["1909004"],"award-info":[{"award-number":["1909004"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,6]]},"DOI":"10.1145\/3534056.3534935","type":"proceedings-article","created":{"date-parts":[[2022,6,6]],"date-time":"2022-06-06T14:56:12Z","timestamp":1654527372000},"page":"26-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Overflowing emerging neural network inference tasks from the GPU to the CPU on heterogeneous servers"],"prefix":"10.1145","author":[{"given":"Adithya","family":"Kumar","sequence":"first","affiliation":[{"name":"The Pennsylvania State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anand","family":"Sivasubramaniam","sequence":"additional","affiliation":[{"name":"The Pennsylvania State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timothy","family":"Zhu","sequence":"additional","affiliation":[{"name":"The Pennsylvania State University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,6,6]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , 2016 . {TensorFlow}: A System for {Large-Scale} Machine Learning . In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX , Boston, MA, 265--283. Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. {TensorFlow}: A System for {Large-Scale} Machine Learning. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). USENIX, Boston, MA, 265--283."},{"key":"e_1_3_2_1_2_1","volume-title":"NIPS Machine Learning for Systems Workshop. NeurIPS","author":"Addanki Ravichandra","year":"2018","unstructured":"Ravichandra Addanki , Shaileshh Bojja Venkatakrishnan , Shreyan Gupta , Hongzi Mao , and Mohammad Alizadeh . 2018 . Placeto: Efficient progressive device placement optimization . In NIPS Machine Learning for Systems Workshop. NeurIPS , San Diego, CA, USA. Ravichandra Addanki, Shaileshh Bojja Venkatakrishnan, Shreyan Gupta, Hongzi Mao, and Mohammad Alizadeh. 2018. Placeto: Efficient progressive device placement optimization. In NIPS Machine Learning for Systems Workshop. NeurIPS, San Diego, CA, USA."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_3_1","DOI":"10.1002\/cpe.1631"},{"key":"e_1_3_2_1_4_1","volume-title":"ONNX: Open Neural Network Exchange. https:\/\/github.com\/onnx\/onnx.","author":"Bai Junjie","year":"2019","unstructured":"Junjie Bai , Fang Lu , Ke Zhang , 2019 . ONNX: Open Neural Network Exchange. https:\/\/github.com\/onnx\/onnx. Junjie Bai, Fang Lu, Ke Zhang, et al. 2019. ONNX: Open Neural Network Exchange. https:\/\/github.com\/onnx\/onnx."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_5_1","DOI":"10.1145\/3422575.3422778"},{"volume-title":"https:\/\/github.com\/cameron314\/concurrentqueue. [Online","year":"2021","unstructured":"Cameron. 2016. Moodycamel::ConcurrentQueue. https:\/\/github.com\/cameron314\/concurrentqueue. [Online ; accessed 27- May - 2021 ]. Cameron. 2016. Moodycamel::ConcurrentQueue. https:\/\/github.com\/cameron314\/concurrentqueue. [Online; accessed 27-May-2021].","key":"e_1_3_2_1_6_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_7_1","DOI":"10.1145\/3456727.3463766"},{"unstructured":"Tianqi Chen Thierry Moreau Ziheng Jiang Lianmin Zheng Eddie Yan Haichen Shen Meghan Cowan Leyuan Wang Yuwei Hu Luis Ceze etal 2018. {TVM}: An automated end-to-end optimizing compiler for deep learning. In 13th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 18). USENIX Association Boston MA 578--594.  Tianqi Chen Thierry Moreau Ziheng Jiang Lianmin Zheng Eddie Yan Haichen Shen Meghan Cowan Leyuan Wang Yuwei Hu Luis Ceze et al. 2018. {TVM}: An automated end-to-end optimizing compiler for deep learning. In 13th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 18). USENIX Association Boston MA 578--594.","key":"e_1_3_2_1_8_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_9_1","DOI":"10.1145\/3007787.3001177"},{"key":"e_1_3_2_1_10_1","volume-title":"Advances in Neural Information Processing Systems","author":"Chorowski Jan K","year":"2015","unstructured":"Jan K Chorowski , Dzmitry Bahdanau , Dmitriy Serdyuk , Kyunghyun Cho , and Yoshua Bengio . 2015. Attention-Based Models for Speech Recognition . In Advances in Neural Information Processing Systems , Vol. 28 . Curran Associates, Inc., NY , USA. https:\/\/proceedings.neurips.cc\/paper\/ 2015 \/file\/1068c6e4c8051cfd4e9ea8072e3189e2-Paper.pdf Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-Based Models for Speech Recognition. In Advances in Neural Information Processing Systems, Vol. 28. Curran Associates, Inc., NY, USA. https:\/\/proceedings.neurips.cc\/paper\/2015\/file\/1068c6e4c8051cfd4e9ea8072e3189e2-Paper.pdf"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_11_1","DOI":"10.1109\/MM.2018.022071131"},{"key":"e_1_3_2_1_12_1","volume-title":"Clipper: A low-latency online prediction serving system. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17)","author":"Crankshaw Daniel","year":"2017","unstructured":"Daniel Crankshaw , Xin Wang , Guilio Zhou , Michael J Franklin , Joseph E Gonzalez , and Ion Stoica . 2017 . Clipper: A low-latency online prediction serving system. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17) . USENIX Association , Boston, MA, USA , 613--627. Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. Clipper: A low-latency online prediction serving system. In 14th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 17). USENIX Association, Boston, MA, USA, 613--627."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_13_1","DOI":"10.1145\/3419111.3421284"},{"key":"e_1_3_2_1_14_1","volume-title":"International Conference on Machine Learning. PMLR","author":"Gao Yuanxiang","year":"2018","unstructured":"Yuanxiang Gao , Li Chen , and Baochun Li . 2018 . Spotlight: Optimizing device placement for training deep neural networks . In International Conference on Machine Learning. PMLR , Princeton University, Princeton, NJ, USA, 1676--1684. Yuanxiang Gao, Li Chen, and Baochun Li. 2018. Spotlight: Optimizing device placement for training deep neural networks. In International Conference on Machine Learning. PMLR, Princeton University, Princeton, NJ, USA, 1676--1684."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_15_1","DOI":"10.1109\/IPDPS.2013.66"},{"volume-title":"Deep learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Yoshua Bengio , and Aaron Courville . 2016. Deep learning . MIT press , Cambridge, MA, USA . Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press, Cambridge, MA, USA.","key":"e_1_3_2_1_16_1"},{"unstructured":"Ga\u00ebl Guennebaud and Benoit Jacob. 2010. Eigen. http:\/\/eigen.tuxfamily.org.  Ga\u00ebl Guennebaud and Benoit Jacob. 2010. Eigen. http:\/\/eigen.tuxfamily.org.","key":"e_1_3_2_1_17_1"},{"key":"e_1_3_2_1_18_1","volume-title":"14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)","author":"Gujarati Arpan","year":"2020","unstructured":"Arpan Gujarati , Reza Karimi , Safya Alzayat , Antoine Kaufmann , Ymir Vigfusson , and Jonathan Mace . 2020 . Serving DNNs like Clockwork: Performance Predictability from the Bottom Up . In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) . USENIX Association, Boston, MA, USA. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/gujarati Arpan Gujarati, Reza Karimi, Safya Alzayat, Antoine Kaufmann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Boston, MA, USA. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/gujarati"},{"key":"e_1_3_2_1_19_1","volume-title":"Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX","author":"Gunasekaran Jashwant Raj","year":"2022","unstructured":"Jashwant Raj Gunasekaran , Cyan Subhra Mishra , Prashanth Thinakaran , Bikash Sharma , Mahmut Taylan Kandemir , and Chita R Das . 2022 . Cocktail: A Multidimensional Optimization for Model Serving in Cloud . In Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX , Boston, MA, 1041--1057. Jashwant Raj Gunasekaran, Cyan Subhra Mishra, Prashanth Thinakaran, Bikash Sharma, Mahmut Taylan Kandemir, and Chita R Das. 2022. Cocktail: A Multidimensional Optimization for Model Serving in Cloud. In Proceedings of the Conference on Networked Systems Design and Implementation (NSDI). USENIX, Boston, MA, 1041--1057."},{"unstructured":"Incorporate Gurobi Optimization. 2018. Gurobi optimizer reference manual. http:\/\/www.gurobi.com.  Incorporate Gurobi Optimization. 2018. Gurobi optimizer reference manual. http:\/\/www.gurobi.com.","key":"e_1_3_2_1_20_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_21_1","DOI":"10.1109\/ISPASS.2019.00012"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_22_1","DOI":"10.1109\/HPCA.2018.00059"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_23_1","DOI":"10.1109\/CVPR.2016.90"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_24_1","DOI":"10.1145\/3274808.3274813"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_25_1","DOI":"10.1145\/3373376.3378530"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_26_1","DOI":"10.1137\/0218016"},{"key":"e_1_3_2_1_27_1","volume-title":"ONNX Runtime inference engine. https:\/\/github.com\/Microsoft\/onnxruntime\/. [Online","author":"Microsoft Inc. 2016.","year":"2021","unstructured":"Microsoft Inc. 2016. ONNX Runtime inference engine. https:\/\/github.com\/Microsoft\/onnxruntime\/. [Online ; accessed 27- May - 2021 ]. Microsoft Inc. 2016. ONNX Runtime inference engine. https:\/\/github.com\/Microsoft\/onnxruntime\/. [Online; accessed 27-May-2021]."},{"key":"e_1_3_2_1_28_1","volume-title":"ONNX Runtime inference engine C APIs. https:\/\/github.com\/microsoft\/onnxruntime\/blob\/master\/docs\/C_API.md. [Online","author":"Microsoft Inc. 2016.","year":"2021","unstructured":"Microsoft Inc. 2016. ONNX Runtime inference engine C APIs. https:\/\/github.com\/microsoft\/onnxruntime\/blob\/master\/docs\/C_API.md. [Online ; accessed 27- May - 2021 ]. Microsoft Inc. 2016. ONNX Runtime inference engine C APIs. https:\/\/github.com\/microsoft\/onnxruntime\/blob\/master\/docs\/C_API.md. [Online; accessed 27-May-2021]."},{"key":"e_1_3_2_1_29_1","volume-title":"ONNXRuntime Graph Optimizations. https:\/\/github.com\/microsoft\/onnxruntime\/blob\/master\/docs\/ONNX_Runtime_Graph_Optimizations.md. [Online","author":"Microsoft Inc. 2016.","year":"2021","unstructured":"Microsoft Inc. 2016. ONNXRuntime Graph Optimizations. https:\/\/github.com\/microsoft\/onnxruntime\/blob\/master\/docs\/ONNX_Runtime_Graph_Optimizations.md. [Online ; accessed 27- May - 2021 ]. Microsoft Inc. 2016. ONNXRuntime Graph Optimizations. https:\/\/github.com\/microsoft\/onnxruntime\/blob\/master\/docs\/ONNX_Runtime_Graph_Optimizations.md. [Online; accessed 27-May-2021]."},{"unstructured":"Mike Innes Alan Edelman Keno Fischer Chris Rackauckas Elliot Saba Viral B Shah and Will Tebbutt. 2019. A Differentiable Programming System to Bridge Machine Learning and Scientific Computing. arXiv:cs.PL\/1907.07587  Mike Innes Alan Edelman Keno Fischer Chris Rackauckas Elliot Saba Viral B Shah and Will Tebbutt. 2019. A Differentiable Programming System to Bridge Machine Learning and Scientific Computing. arXiv:cs.PL\/1907.07587","key":"e_1_3_2_1_30_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_31_1","DOI":"10.1145\/3419111.3421302"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_32_1","DOI":"10.1145\/3341301.3359630"},{"volume-title":"14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20)","author":"Jiang Yimin","unstructured":"Yimin Jiang , Yibo Zhu , Chang Lan , Bairen Yi , Yong Cui , and Chuanxiong Guo . 2020. A Unified Architecture for Accelerating Distributed {DNN} Training in Heterogeneous GPU\/CPU Clusters . In 14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20) . USENIX Association , Boston, MA, USA , 463--479. Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. 2020. A Unified Architecture for Accelerating Distributed {DNN} Training in Heterogeneous GPU\/CPU Clusters. In 14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20). USENIX Association, Boston, MA, USA, 463--479.","key":"e_1_3_2_1_33_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_34_1","DOI":"10.1145\/3079856.3080246"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_35_1","DOI":"10.1145\/2628071.2628088"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_36_1","DOI":"10.1109\/CVPR.2017.106"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_37_1","DOI":"10.1007\/978-3-319-10602-1_48"},{"volume-title":"2019 {USENIX} Annual Technical Conference (USENIX ATC 19)","author":"Liu Yizhi","unstructured":"Yizhi Liu , Yao Wang , Ruofei Yu , Mu Li , Vin Sharma , and Yida Wang . 2019. Optimizing {CNN} Model Inference on CP Us . In 2019 {USENIX} Annual Technical Conference (USENIX ATC 19) . USENIX Association , Boston, MA , 1025--1040. Yizhi Liu, Yao Wang, Ruofei Yu, Mu Li, Vin Sharma, and Yida Wang. 2019. Optimizing {CNN} Model Inference on CPUs. In 2019 {USENIX} Annual Technical Conference (USENIX ATC 19). USENIX Association, Boston, MA, 1025--1040.","key":"e_1_3_2_1_38_1"},{"key":"e_1_3_2_1_39_1","volume-title":"Applications of deep learning in biomedicine. Molecular pharmaceutics 13, 5","author":"Mamoshina Polina","year":"2016","unstructured":"Polina Mamoshina , Armando Vieira , Evgeny Putin , and Alex Zhavoronkov . 2016. Applications of deep learning in biomedicine. Molecular pharmaceutics 13, 5 ( 2016 ), 1445--1454. Polina Mamoshina, Armando Vieira, Evgeny Putin, and Alex Zhavoronkov. 2016. Applications of deep learning in biomedicine. Molecular pharmaceutics 13, 5 (2016), 1445--1454."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_40_1","DOI":"10.1145\/3154842.3154843"},{"key":"e_1_3_2_1_41_1","volume-title":"International Conference on Learning Representations. Openreview.net, Online.","author":"Mirhoseini Azalia","year":"2018","unstructured":"Azalia Mirhoseini , Anna Goldie , Hieu Pham , Benoit Steiner , Quoc V Le , and Jeff Dean . 2018 . A hierarchical model for device placement . In International Conference on Learning Representations. Openreview.net, Online. Azalia Mirhoseini, Anna Goldie, Hieu Pham, Benoit Steiner, Quoc V Le, and Jeff Dean. 2018. A hierarchical model for device placement. In International Conference on Learning Representations. Openreview.net, Online."},{"volume-title":"nvml. https:\/\/docs.nvidia.com\/deploy\/nvml-api\/nvml-api-reference.html. [Online","year":"2021","unstructured":"Nvidia. 2020. nvml. https:\/\/docs.nvidia.com\/deploy\/nvml-api\/nvml-api-reference.html. [Online ; accessed 27- May - 2021 ]. Nvidia. 2020. nvml. https:\/\/docs.nvidia.com\/deploy\/nvml-api\/nvml-api-reference.html. [Online; accessed 27-May-2021].","key":"e_1_3_2_1_42_1"},{"unstructured":"Christopher Olston Noah Fiedel Kiril Gorovoy Jeremiah Harmsen Li Lao Fangwei Li Vinu Rajashekhar Sukriti Ramesh and Jordan Soyke. 2017. TensorFlow-Serving: Flexible High-Performance ML Serving. arXiv:cs.DC\/1712.06139  Christopher Olston Noah Fiedel Kiril Gorovoy Jeremiah Harmsen Li Lao Fangwei Li Vinu Rajashekhar Sukriti Ramesh and Jordan Soyke. 2017. TensorFlow-Serving: Flexible High-Performance ML Serving. arXiv:cs.DC\/1712.06139","key":"e_1_3_2_1_43_1"},{"key":"e_1_3_2_1_44_1","volume-title":"https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/faster-rcnn. [Online","author":"Model ONNX.","year":"2021","unstructured":"ONNX. 2016. FasterRCNN Model . https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/faster-rcnn. [Online ; accessed 27- May - 2021 ]. ONNX. 2016. FasterRCNN Model. https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/faster-rcnn. [Online; accessed 27-May-2021]."},{"key":"e_1_3_2_1_45_1","volume-title":"https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/mask-rcnn. [Online","author":"Model ONNX.","year":"2021","unstructured":"ONNX. 2016. Mask-RCNN Model . https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/mask-rcnn. [Online ; accessed 27- May - 2021 ]. ONNX. 2016. Mask-RCNN Model. https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/mask-rcnn. [Online; accessed 27-May-2021]."},{"key":"e_1_3_2_1_46_1","volume-title":"https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/ssd. [Online","author":"Model ONNX.","year":"2021","unstructured":"ONNX. 2016. SSD Model . https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/ssd. [Online ; accessed 27- May - 2021 ]. ONNX. 2016. SSD Model. https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/object_detection_segmentation\/ssd. [Online; accessed 27-May-2021]."},{"unstructured":"Aditya Paliwal Felix Gimeno Vinod Nair Yujia Li Miles Lubin Pushmeet Kohli and Oriol Vinyals. 2020. Reinforced Genetic Algorithm Learning for Optimizing Computation Graphs. arXiv:cs.LG\/1905.02494  Aditya Paliwal Felix Gimeno Vinod Nair Yujia Li Miles Lubin Pushmeet Kohli and Oriol Vinyals. 2020. Reinforced Genetic Algorithm Learning for Optimizing Computation Graphs. arXiv:cs.LG\/1905.02494","key":"e_1_3_2_1_47_1"},{"unstructured":"Jongsoo Park Maxim Naumov Protonu Basu Summer Deng Aravind Kalaiah Daya Khudia James Law Parth Malani Andrey Malevich Satish Nadathur Juan Pino Martin Schatz Alexander Sidorov Viswanath Sivakumar Andrew Tulloch Xiaodong Wang Yiming Wu Hector Yuen Utku Diril Dmytro Dzhulgakov Kim Hazelwood Bill Jia Yangqing Jia Lin Qiao Vijay Rao Nadav Rotem Sungjoo Yoo and Mikhail Smelyanskiy. 2018. Deep Learning Inference in Facebook Data Centers: Characterization Performance Optimizations and Hardware Implications. arXiv:cs.LG\/1811.09886  Jongsoo Park Maxim Naumov Protonu Basu Summer Deng Aravind Kalaiah Daya Khudia James Law Parth Malani Andrey Malevich Satish Nadathur Juan Pino Martin Schatz Alexander Sidorov Viswanath Sivakumar Andrew Tulloch Xiaodong Wang Yiming Wu Hector Yuen Utku Diril Dmytro Dzhulgakov Kim Hazelwood Bill Jia Yangqing Jia Lin Qiao Vijay Rao Nadav Rotem Sungjoo Yoo and Mikhail Smelyanskiy. 2018. Deep Learning Inference in Facebook Data Centers: Characterization Performance Optimizations and Hardware Implications. arXiv:cs.LG\/1811.09886","key":"e_1_3_2_1_48_1"},{"key":"e_1_3_2_1_49_1","volume-title":"Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems. Curran Associates","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , 2019 . Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems. Curran Associates , Inc., NY, USA , 8026--8037. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems. Curran Associates, Inc., NY, USA, 8026--8037."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_50_1","DOI":"10.1145\/3373376.3378505"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_51_1","DOI":"10.1145\/3457388.3458661"},{"volume-title":"Advances in neural information processing systems. Curran Associates","author":"Ren Shaoqing","unstructured":"Shaoqing Ren , Kaiming He , Ross Girshick , and Jian Sun . 2015. Faster r-cnn: Towards real-time object detection with region proposal networks . In Advances in neural information processing systems. Curran Associates , Inc., NY, USA , 91--99. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems. Curran Associates, Inc., NY, USA, 91--99.","key":"e_1_3_2_1_52_1"},{"key":"e_1_3_2_1_53_1","volume-title":"Global Machine Learning Chip Market. https:\/\/www.globenewswire.com\/news-release\/2020\/02\/18\/1986370\/0\/en\/Global-Machine-Learning-Chip-Market-to-Garner-37-85-Billion-by-2025-at-40-8-CAGR.html. [Online","author":"Research Allied Market","year":"2021","unstructured":"Allied Market Research . 2020. Global Machine Learning Chip Market. https:\/\/www.globenewswire.com\/news-release\/2020\/02\/18\/1986370\/0\/en\/Global-Machine-Learning-Chip-Market-to-Garner-37-85-Billion-by-2025-at-40-8-CAGR.html. [Online ; accessed 27- May - 2021 ]. Allied Market Research. 2020. Global Machine Learning Chip Market. https:\/\/www.globenewswire.com\/news-release\/2020\/02\/18\/1986370\/0\/en\/Global-Machine-Learning-Chip-Market-to-Garner-37-85-Billion-by-2025-at-40-8-CAGR.html. [Online; accessed 27-May-2021]."},{"unstructured":"Francisco Romero Qian Li Neeraja J. Yadwadkar and Christos Kozyrakis. 2020. INFaaS: A Model-less and Managed Inference Serving System. arXiv:cs.DC\/1905.13348  Francisco Romero Qian Li Neeraja J. Yadwadkar and Christos Kozyrakis. 2020. INFaaS: A Model-less and Managed Inference Serving System. arXiv:cs.DC\/1905.13348","key":"e_1_3_2_1_54_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_55_1","DOI":"10.1145\/2517349.2522715"},{"key":"e_1_3_2_1_56_1","volume-title":"Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference. arXiv:cs.PL\/2006.03031","author":"Shen Haichen","year":"2021","unstructured":"Haichen Shen , Jared Roesch , Zhi Chen , Wei Chen , Yong Wu , Mu Li , Vin Sharma , Zachary Tatlock , and Yida Wang . 2021 . Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference. arXiv:cs.PL\/2006.03031 Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, and Yida Wang. 2021. Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference. arXiv:cs.PL\/2006.03031"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_57_1","DOI":"10.1145\/3374888.3374897"},{"volume-title":"Advances in neural information processing systems. Curran Associates","author":"Szegedy Christian","unstructured":"Christian Szegedy , Alexander Toshev , and Dumitru Erhan . 2013. Deep neural networks for object detection . In Advances in neural information processing systems. Curran Associates , Inc., NY, USA , 2553--2561. Christian Szegedy, Alexander Toshev, and Dumitru Erhan. 2013. Deep neural networks for object detection. In Advances in neural information processing systems. Curran Associates, Inc., NY, USA, 2553--2561.","key":"e_1_3_2_1_58_1"},{"unstructured":"Han Vanholder. 2016. Efficient Inference with TensorRT.  Han Vanholder. 2016. Efficient Inference with TensorRT.","key":"e_1_3_2_1_59_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_60_1","DOI":"10.1145\/3302424.3303953"},{"key":"e_1_3_2_1_61_1","volume-title":"Condconv: Conditionally parameterized convolutions for efficient inference. Advances in Neural Information Processing Systems 32","author":"Yang Brandon","year":"2019","unstructured":"Brandon Yang , Gabriel Bender , Quoc V Le , and Jiquan Ngiam . 2019 . Condconv: Conditionally parameterized convolutions for efficient inference. Advances in Neural Information Processing Systems 32 (2019). Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam. 2019. Condconv: Conditionally parameterized convolutions for efficient inference. Advances in Neural Information Processing Systems 32 (2019)."},{"key":"e_1_3_2_1_62_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association","author":"Zhang Chengliang","year":"2019","unstructured":"Chengliang Zhang , Minchen Yu , Wei Wang , and Feng Yan . 2019 . Mark: Exploiting cloud services for cost-effective, slo-aware machine learning inference serving . In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association , Boston, MA, USA. Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. 2019. Mark: Exploiting cloud services for cost-effective, slo-aware machine learning inference serving. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association, Boston, MA, USA."},{"volume-title":"12th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 20)","author":"Zhang Jeff","unstructured":"Jeff Zhang , Sameh Elnikety , Shuayb Zarar , Atul Gupta , and Siddharth Garg . 2020. Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-Service Systems . In 12th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 20) . USENIX Association , Boston, MA, USA . Jeff Zhang, Sameh Elnikety, Shuayb Zarar, Atul Gupta, and Siddharth Garg. 2020. Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-Service Systems. In 12th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 20). USENIX Association, Boston, MA, USA.","key":"e_1_3_2_1_63_1"},{"key":"e_1_3_2_1_64_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association","author":"Zhang Minjia","year":"2018","unstructured":"Minjia Zhang , Samyam Rajbhandari , Wenhan Wang , and Yuxiong He . 2018 . Deepcpu: Serving rnn-based deep learning models 10x faster . In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association , Boston, MA, 951--965. https:\/\/www.usenix.org\/conference\/atc18\/presentation\/zhang-minjia Minjia Zhang, Samyam Rajbhandari, Wenhan Wang, and Yuxiong He. 2018. Deepcpu: Serving rnn-based deep learning models 10x faster. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (ATC). USENIX Association, Boston, MA, 951--965. https:\/\/www.usenix.org\/conference\/atc18\/presentation\/zhang-minjia"},{"key":"e_1_3_2_1_65_1","volume-title":"Dynet: Dynamic convolution for accelerating convolutional neural networks. arXiv preprint arXiv:2004.10694 1","author":"Zhang Yikang","year":"2020","unstructured":"Yikang Zhang , Jian Zhang , Qiang Wang , and Zhao Zhong . 2020 . Dynet: Dynamic convolution for accelerating convolutional neural networks. arXiv preprint arXiv:2004.10694 1 (2020). Yikang Zhang, Jian Zhang, Qiang Wang, and Zhao Zhong. 2020. Dynet: Dynamic convolution for accelerating convolutional neural networks. arXiv preprint arXiv:2004.10694 1 (2020)."},{"key":"e_1_3_2_1_66_1","volume-title":"Ansor: Generating High-Performance Tensor Programs for Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)","author":"Zheng Lianmin","year":"2020","unstructured":"Lianmin Zheng , Chengfan Jia , Minmin Sun , Zhao Wu , Cody Hao Yu , Ameer Haj-Ali , Yida Wang , Jun Yang , Danyang Zhuo , Koushik Sen , Joseph E. Gonzalez , and Ion Stoica . 2020 . Ansor: Generating High-Performance Tensor Programs for Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) . USENIX Association, Boston, MA, USA, 863--879. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/zheng Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. 2020. Ansor: Generating High-Performance Tensor Programs for Deep Learning. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, Boston, MA, USA, 863--879. https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/zheng"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_67_1","DOI":"10.1145\/3373376.3378508"}],"event":{"sponsor":["SIGOPS ACM Special Interest Group on Operating Systems","Technion Israel Institute of Technology","USENIX Assoc USENIX Assoc"],"acronym":"SYSTOR '22","name":"SYSTOR '22: The 15th ACM International Systems and Storage Conference","location":"Haifa Israel"},"container-title":["Proceedings of the 15th ACM International Conference on Systems and Storage"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534056.3534935","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3534056.3534935","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3534056.3534935","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:09Z","timestamp":1750186809000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534056.3534935"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,6]]},"references-count":67,"alternative-id":["10.1145\/3534056.3534935","10.1145\/3534056"],"URL":"https:\/\/doi.org\/10.1145\/3534056.3534935","relation":{},"subject":[],"published":{"date-parts":[[2022,6,6]]},"assertion":[{"value":"2022-06-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}