{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T22:44:53Z","timestamp":1781131493706,"version":"3.54.1"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,6,15]],"date-time":"2020-06-15T00:00:00Z","timestamp":1592179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100017548","name":"National Defense Science and Technology Innovation Special Zone Project of China,","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100017548","id-type":"DOI","asserted-by":"crossref"}]},{"name":"General Research Fund of the Research Grants Council of Hong Kong","award":["PolyU 152221\/19E"],"award-info":[{"award-number":["PolyU 152221\/19E"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61772559, 61872310, 61672543, 61972145"],"award-info":[{"award-number":["61772559, 61872310, 61672543, 61972145"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2020,6,15]]},"abstract":"<jats:p>Deep neural networks (DNNs) have been widely used in many intelligent applications such as object recognition and automatic driving due to their superior performance in conducting inference tasks. However, DNN models are usually heavyweight in computation, hindering their utilization on the resource-constraint Internet of Things (IoT) end devices. To this end, cooperative deep inference is proposed, in which a DNN model is adaptively partitioned into two parts and different parts are executed on different devices (cloud or edge end devices) to minimize the total inference latency. One important issue is thus to find the optimal partition of the deep model subject to network dynamics in a real-time manner. In this paper, we formulate the optimal DNN partition as a min-cut problem in a directed acyclic graph (DAG) specially derived from the DNN and propose a novel two-stage approach named quick deep model partition (QDMP) to solve it. QDMP exploits the fact that the optimal partition of a DNN model must be between two adjacent cut vertices in the corresponding DAG. It first identifies the two cut vertices and considers only the subgraph in between when calculating the min-cut. QDMP can find the optimal model partition with response time less than 300ms even for large DNN models containing hundreds of layers (up to 66.3x faster than the state-of-the-art solution), and thus enables real-time cooperative deep inference over the cloud and edge end devices. Moreover, we observe one important fact that is ignored in all previous works: As many deep learning frameworks optimize the execution of DNN models, the execution latency of a series of layers in a DNN does not equal to the summation of each layer's independent execution latency. This results in inaccurate inference latency estimation in existing works. We propose a new execution latency measurement method, with which the inference latency can be accurately estimated in practice. We implement QDMP on real hardware and use a real-world self-driving car video dataset to evaluate its performance. Experimental results show that QDMP outperforms the state-of-the-art solution, reducing inference latency by up to 1.69x and increasing throughput by up to 3.81x.<\/jats:p>","DOI":"10.1145\/3397315","type":"journal-article","created":{"date-parts":[[2020,6,15]],"date-time":"2020-06-15T22:30:37Z","timestamp":1592260237000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":60,"title":["Towards Real-time Cooperative Deep Inference over the Cloud and Edge End Devices"],"prefix":"10.1145","volume":"4","author":[{"given":"Shigeng","family":"Zhang","sequence":"first","affiliation":[{"name":"Central South University, Changsha, P. R. China and the State Key Laboratory for Novel Software Technology, Nanjing University, P.R China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yinggang","family":"Li","sequence":"additional","affiliation":[{"name":"Central South University, Changsha, P. R. China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuan","family":"Liu","sequence":"additional","affiliation":[{"name":"Hunan University, Changsha, P. R. China and Science and Technology on Parallel and Distributed Processing Laboratory (PDL)"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Song","family":"Guo","sequence":"additional","affiliation":[{"name":"Hong Kong Polytechnic University, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiping","family":"Wang","sequence":"additional","affiliation":[{"name":"Central South University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianxin","family":"Wang","sequence":"additional","affiliation":[{"name":"Central South University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bo","family":"Ding","sequence":"additional","affiliation":[{"name":"National University of Defense Technology, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Di","family":"Wu","sequence":"additional","affiliation":[{"name":"Hunan University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,6,15]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 265--283","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , 2016 . Tensorflow: A system for large-scale machine learning . In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 265--283 . Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 265--283."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSEC.2018.2888775"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000049"},{"key":"e_1_2_1_4_1","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Jill Burstein, Christy Doran , and Thamar Solorio (Eds.). Association for Computational Linguistics , 4171--4186. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171--4186."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3277893.3277899"},{"key":"e_1_2_1_6_1","volume-title":"Mohammad Saeed Abrishami, and Massoud Pedram","author":"Eshratifar Amir Erfan","year":"2019","unstructured":"Amir Erfan Eshratifar , Mohammad Saeed Abrishami, and Massoud Pedram . 2019 . JointDNN: an efficient training and inference engine for intelligent mobile cloud computing services. IEEE Transactions on Mobile Computing pp (2019), 1--12. Amir Erfan Eshratifar, Mohammad Saeed Abrishami, and Massoud Pedram. 2019. JointDNN: an efficient training and inference engine for intelligent mobile cloud computing services. IEEE Transactions on Mobile Computing pp (2019), 1--12."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_8_1","volume-title":"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR abs\/1704.04861","author":"Howard Andrew G.","year":"2017","unstructured":"Andrew G. Howard , Menglong Zhu , Bo Chen , Dmitry Kalenichenko , Weijun Wang , Tobias Weyand , Marco Andreetto , and Hartwig Adam . 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR abs\/1704.04861 ( 2017 ). arXiv: 1704.04861 http:\/\/arxiv.org\/abs\/1704.04861 Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR abs\/1704.04861 (2017). arXiv: 1704.04861 http:\/\/arxiv.org\/abs\/1704.04861"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2019.8737614"},{"key":"e_1_2_1_10_1","volume-title":"SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt;1MB model size. CoRR abs\/1602.07360","author":"Iandola Forrest N.","year":"2016","unstructured":"Forrest N. Iandola , Matthew W. Moskewicz , Khalid Ashraf , Song Han , William J. Dally , and Kurt Keutzer . 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt;1MB model size. CoRR abs\/1602.07360 ( 2016 ). Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and &lt;1MB model size. CoRR abs\/1602.07360 (2016)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_2_1_12_1","volume-title":"Trevor Mudge, Jason Mars, and Lingjia Tang.","author":"Kang Yiping","year":"2017","unstructured":"Yiping Kang , Johann Hauswald , Cao Gao , Austin Rovinski , Trevor Mudge, Jason Mars, and Lingjia Tang. 2017 . Neurosurgeon : Collaborative intelligence between the cloud and mobile edge. 45, 1 (2017), 615--629. Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. 2017. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. 45, 1 (2017), 615--629."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/AVSS.2018.8639121"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPSN.2016.7460664"},{"key":"e_1_2_1_15_1","volume-title":"Deep learning. Nature 521, 7553","author":"LeCun Yann","year":"2015","unstructured":"Yann LeCun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. Nature 521, 7553 ( 2015 ), 436--444. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436--444."},{"key":"e_1_2_1_16_1","volume-title":"Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing","author":"Li En","year":"2019","unstructured":"En Li , Liekang Zeng , Zhi Zhou , and Xu Chen . 2019 . Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing . IEEE Transactions on Wireless Communications ( 2019). En Li, Liekang Zeng, Zhi Zhou, and Xu Chen. 2019. Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing. IEEE Transactions on Wireless Communications (2019)."},{"key":"e_1_2_1_17_1","volume-title":"Proceedindgs of the 5th International Conference on Learning Representations(ICLR).","author":"Li Hao","year":"2017","unstructured":"Hao Li , Asim Kadav , Igor Durdanovic , Hanan Samet , and Hans Peter Graf . 2017 . Pruning Filters for Efficient ConvNets . In Proceedindgs of the 5th International Conference on Learning Representations(ICLR). Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. 2017. Pruning Filters for Efficient ConvNets. In Proceedindgs of the 5th International Conference on Learning Representations(ICLR)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2018.2881295"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 5th International Conference on Learning Representations (ICLR).","author":"Molchanov Pavlo","year":"2017","unstructured":"Pavlo Molchanov , Stephen Tyree , Tero Karras , Timo Aila , and Jan Kautz . 2017 . Pruning Convolutional Neural Networks for Resource Efficient Inference . In Proceedings of the 5th International Conference on Learning Representations (ICLR). Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning Convolutional Neural Networks for Resource Efficient Inference. In Proceedings of the 5th International Conference on Learning Representations (ICLR)."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of Advances in Neural Information Processing Systems (NIPS). 8024--8035","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , 2019 . PyTorch: An imperative style, high-performance deep learning library . In Proceedings of Advances in Neural Information Processing Systems (NIPS). 8024--8035 . Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of Advances in Neural Information Processing Systems (NIPS). 8024--8035."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.91"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.690"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations (ICLR), Yoshua Bengio and Yann LeCun (Eds.).","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman . 2015 . Very Deep Convolutional Networks for Large-Scale Image Recognition . In Proceedings of the 3rd International Conference on Learning Representations (ICLR), Yoshua Bengio and Yann LeCun (Eds.). Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), Yoshua Bengio and Yann LeCun (Eds.)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0190(74)90003-9"},{"key":"e_1_2_1_26_1","volume-title":"Data structures and network algorithms","author":"Tarjan Robert Endre","unstructured":"Robert Endre Tarjan . 1983. Data structures and network algorithms . Vol. 44 . Siam . Robert Endre Tarjan. 1983. Data structures and network algorithms. Vol. 44. Siam."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2016.7900006"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2017.226"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1561\/1300000054"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/mnet.2019.1800286"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of Advances in neural information processing systems (NIPS). 802--810","author":"Xingjian SHI","year":"2015","unstructured":"SHI Xingjian , Zhourong Chen , Hao Wang , Dit-Yan Yeung , Wai-Kin Wong , and Wang-chun Woo. 2015 . Convolutional LSTM network: A machine learning approach for precipitation nowcasting . In Proceedings of Advances in neural information processing systems (NIPS). 802--810 . SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proceedings of Advances in neural information processing systems (NIPS). 802--810."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01146"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of International Conference on Learning Representations (ICLR). https:\/\/openreview.net\/forum?id=BylBr3C9K7","author":"Yang Haichuan","year":"2019","unstructured":"Haichuan Yang , Yuhao Zhu , and Ji Liu . 2019 . Energy-Constrained Compression for Deep Neural Networks via Weighted Sparse Projection and Layer Input Masking . In Proceedings of International Conference on Learning Representations (ICLR). https:\/\/openreview.net\/forum?id=BylBr3C9K7 Haichuan Yang, Yuhao Zhu, and Ji Liu. 2019. Energy-Constrained Compression for Deep Neural Networks via Weighted Sparse Projection and Layer Input Masking. In Proceedings of International Conference on Learning Representations (ICLR). https:\/\/openreview.net\/forum?id=BylBr3C9K7"},{"key":"e_1_2_1_34_1","volume-title":"Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems. 5754--5764.","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang , Zihang Dai , Yiming Yang , Jaime Carbonell , Russ R Salakhutdinov , and Quoc V Le . 2019 . Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems. 5754--5764. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems. 5754--5764."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 34th International Conference on Machine Learning (ICML). 3958--3966","author":"Yoon Jaehong","year":"2017","unstructured":"Jaehong Yoon and Sung Ju Hwang . 2017 . Combined group and exclusive sparsity for deep neural networks . In Proceedings of the 34th International Conference on Machine Learning (ICML). 3958--3966 . Jaehong Yoon and Sung Ju Hwang. 2017. Combined group and exclusive sparsity for deep neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML). 3958--3966."},{"key":"e_1_2_1_36_1","volume-title":"BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling. CoRR abs\/1805.04687","author":"Yu Fisher","year":"2018","unstructured":"Fisher Yu , Wenqi Xian , Yingying Chen , Fangchen Liu , Mike Liao , Vashisht Madhavan , and Trevor Darrell . 2018. BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling. CoRR abs\/1805.04687 ( 2018 ). arXiv:1805.04687 http:\/\/arxiv.org\/abs\/1805.04687 Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. 2018. BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling. CoRR abs\/1805.04687 (2018). arXiv:1805.04687 http:\/\/arxiv.org\/abs\/1805.04687"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2019.00057"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00716"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397315","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3397315","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:12:47Z","timestamp":1750201967000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397315"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,15]]},"references-count":38,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6,15]]}},"alternative-id":["10.1145\/3397315"],"URL":"https:\/\/doi.org\/10.1145\/3397315","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,15]]},"assertion":[{"value":"2020-06-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}