{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T11:02:31Z","timestamp":1775041351089,"version":"3.50.1"},"reference-count":34,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,15]],"date-time":"2022-08-15T00:00:00Z","timestamp":1660521600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Key Program Special Fund in XJTLU","award":["KSF-E-64"],"award-info":[{"award-number":["KSF-E-64"]}]},{"name":"Key Program Special Fund in XJTLU","award":["RDF-19-01-14"],"award-info":[{"award-number":["RDF-19-01-14"]}]},{"name":"Key Program Special Fund in XJTLU","award":["RDF-20-01-15"],"award-info":[{"award-number":["RDF-20-01-15"]}]},{"name":"Key Program Special Fund in XJTLU","award":["52175030"],"award-info":[{"award-number":["52175030"]}]},{"name":"XJTLU Research Development Funding","award":["KSF-E-64"],"award-info":[{"award-number":["KSF-E-64"]}]},{"name":"XJTLU Research Development Funding","award":["RDF-19-01-14"],"award-info":[{"award-number":["RDF-19-01-14"]}]},{"name":"XJTLU Research Development Funding","award":["RDF-20-01-15"],"award-info":[{"award-number":["RDF-20-01-15"]}]},{"name":"XJTLU Research Development Funding","award":["52175030"],"award-info":[{"award-number":["52175030"]}]},{"name":"National Natural Science Foundation of China (NSFC)","award":["KSF-E-64"],"award-info":[{"award-number":["KSF-E-64"]}]},{"name":"National Natural Science Foundation of China (NSFC)","award":["RDF-19-01-14"],"award-info":[{"award-number":["RDF-19-01-14"]}]},{"name":"National Natural Science Foundation of China (NSFC)","award":["RDF-20-01-15"],"award-info":[{"award-number":["RDF-20-01-15"]}]},{"name":"National Natural Science Foundation of China (NSFC)","award":["52175030"],"award-info":[{"award-number":["52175030"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Recent advances in both lightweight deep learning algorithms and edge computing increasingly enable multiple model inference tasks to be conducted concurrently on resource-constrained edge devices, allowing us to achieve one goal collaboratively rather than getting high quality in each standalone task. However, the high overall running latency for performing multi-model inferences always negatively affects the real-time applications. To combat latency, the algorithms should be optimized to minimize the latency for multi-model deployment without compromising the safety-critical situation. This work focuses on the real-time task scheduling strategy for multi-model deployment and investigating the model inference using an open neural network exchange (ONNX) runtime engine. Then, an application deployment strategy is proposed based on the container technology and inference tasks are scheduled to different containers based on the scheduling strategies. Experimental results show that the proposed solution is able to significantly reduce the overall running latency in real-time applications.<\/jats:p>","DOI":"10.3390\/s22166097","type":"journal-article","created":{"date-parts":[[2022,8,15]],"date-time":"2022-08-15T23:44:03Z","timestamp":1660607043000},"page":"6097","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Multi-Model Running Latency Optimization in an Edge Computing Paradigm"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6697-4041","authenticated-orcid":false,"given":"Peisong","family":"Li","sequence":"first","affiliation":[{"name":"School of Advanced Technology, Xi\u2019an Jiaotong-Liverpool University, Suzhou 215123, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8771-8901","authenticated-orcid":false,"given":"Xinheng","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Advanced Technology, Xi\u2019an Jiaotong-Liverpool University, Suzhou 215123, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kaizhu","family":"Huang","sequence":"additional","affiliation":[{"name":"Data Science Research Center, Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7774-1024","authenticated-orcid":false,"given":"Yi","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool L69 3BX, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shancang","family":"Li","sequence":"additional","affiliation":[{"name":"School of Computer Science and Informatics, Cardiff University, Cardiff CF10 3AT, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muddesar","family":"Iqbal","sequence":"additional","affiliation":[{"name":"Renewable Energy Laboratory, Communications and Networks Engineering Department, College of Engineering, Prince Sultan University, Riyadh 11586, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2131","DOI":"10.1109\/COMST.2021.3106401","article-title":"Resource scheduling in edge computing: A survey","volume":"23","author":"Luo","year":"2021","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_2","unstructured":"Holler, J., Tsiatsis, V., Mulligan, C., Karnouskos, S., Avesand, S., and Boyle, D. (2014). Internet of Things, Academic Press."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/MAES.2020.3043072","article-title":"Artificial intelligence and data fusion at the edge","volume":"36","author":"Munir","year":"2021","journal-title":"IEEE Aerosp. Electron. Syst. Mag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1109\/TWC.2019.2946140","article-title":"Edge AI: On-demand accelerating deep neural network inference via edge computing","volume":"19","author":"Li","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Brandalero, M., Ali, M., Le Jeune, L., Hernandez, H.G.M., Veleski, M., da Silva, B., Lemeire, J., Van Beeck, K., Touhafi, A., and Goedem\u00e9, T. (2020, January 23\u201325). AITIA: Embedded AI Techniques for Embedded Industrial Applications. Proceedings of the International Conference on Omni-layer Intelligent Systems (COINS), Barcelona, Spain.","DOI":"10.1109\/COINS49042.2020.9191672"},{"key":"ref_6","unstructured":"Sleight, M. (2021, June 22). How Do Self-Driving Cars Work?. Available online: https:\/\/www.bankrate.com\/insurance\/car\/how-do-self-driving-cars-work\/."},{"key":"ref_7","unstructured":"Gupta, A. (2021, April 04). Machine Learning Algorithms in Autonomous Driving. Available online: https:\/\/www.iiot-world.com\/artificial-intelligence-ml\/machine-learning\/machine-learning-algorithms-in-autonomous-driving\/."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Zhou, S., Xie, M., Jin, Y., Miao, F., and Ding, C. (2021, January 7\u20138). An End-to-end Multi-task Object Detection using Embedded GPU in Autonomous Driving. Proceedings of the 22nd International Symposium on Quality Electronic Design (ISQED), Santa Clara, CA, USA.","DOI":"10.1109\/ISQED51717.2021.9424308"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"6469","DOI":"10.1109\/JIOT.2020.3043716","article-title":"Computing Systems for Autonomous Driving: State of the Art and Challenges","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1002\/sys.21528","article-title":"Autonomous driving systems hardware and software architecture exploration: Optimizing latency and cost under safety constraints","volume":"23","author":"Collin","year":"2020","journal-title":"Syst. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Dong, Z., Shi, W., Tong, G., and Yang, K. (2020, January 27\u201328). Collaborative autonomous driving: Vision and challenges. Proceedings of the International Conference on Connected and Autonomous Driving (MetroCAD), Detroit, MI, USA.","DOI":"10.1109\/MetroCAD48866.2020.00010"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Verucchi, M., Brilli, G., Sapienza, D., Verasani, M., Arena, M., Gatti, F., Capotondi, A., Cavicchioli, R., Bertogna, M., and Solieri, M. (2020, January 8\u201311). A systematic assessment of embedded neural networks for object detection. Proceedings of the 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Vienna, Austria.","DOI":"10.1109\/ETFA46521.2020.9212130"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lin, C., Zhang, Z., Li, H., and Liu, J. (2021, January 15\u201317). ECSRL: A Learning-Based Scheduling Framework for AI Workloads in Heterogeneous Edge-Cloud Systems. Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, Coimbra, Portugal.","DOI":"10.1145\/3485730.3492886"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hao, C., and Chen, D. (2021, January 6\u20139). Software\/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems. Proceedings of the IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS), Washington, DC, USA.","DOI":"10.1109\/AICAS51828.2021.9458577"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wang, X., Han, Y., Leung, V.C., Niyato, D., Yan, X., and Chen, X. (2020). Edge AI: Convergence of Edge Computing and Artificial Intelligence, Springer Nature.","DOI":"10.1007\/978-981-15-6186-3"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1738","DOI":"10.1109\/JPROC.2019.2918951","article-title":"Edge intelligence: Paving the last mile of artificial intelligence with edge computing","volume":"107","author":"Zhou","year":"2019","journal-title":"Proc. IEEE"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Calo, S.B., Touna, M., Verma, D.C., and Cullen, A. (2017, January 11\u201314). Edge computing architecture for applying AI to IoT. Proceedings of the IEEE International Conference on Big Data (Big Data), Boston, MA, USA.","DOI":"10.1109\/BigData.2017.8258272"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Campolo, C., Genovese, G., Iera, A., and Molinaro, A. (2021). Virtualizing AI at the distributed edge towards intelligent IoT applications. J. Sens. Actuator Netw., 10.","DOI":"10.3390\/jsan10010013"},{"key":"ref_19","unstructured":"Chen, J., Li, K., Deng, Q., Li, K., and Philip, S.Y. (2019). Distributed deep learning model for intelligent video surveillance systems with edge computing. IEEE Trans. Ind. Inform."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"9372","DOI":"10.1109\/JIOT.2020.2986015","article-title":"Intelligent cooperative edge computing in internet of things","volume":"7","author":"Gong","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_21","unstructured":"Bi, J. (2020). Improving Training and Inference for Embedded Machine Learning. [Ph.D. Thesis, University of Southampton]."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"774","DOI":"10.1111\/mice.12449","article-title":"Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures","volume":"34","author":"Wu","year":"2019","journal-title":"Comput.-Aided Civ. Infrastruct. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1016\/j.ins.2021.06.039","article-title":"Neural network quantization in federated learning at the edge","volume":"575","author":"Tonellotto","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Minakova, S., Tang, E., and Stefanov, T. (2020, January 5\u20139). Combining task-and data-level parallelism for high-throughput cnn inference on embedded cpus-gpus mpsocs. Proceedings of the International Conference on Embedded Computer Systems, Samos, Greece.","DOI":"10.1007\/978-3-030-60939-9_2"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Dey, S., Mukherjee, A., and Pal, A. (2019, January 10). Embedded Deep Inference in Practice: Case for Model Partitioning. Proceedings of the 1st Workshop on Machine Learning on Edge in Sensor Systems, 2019, New York, NY, USA.","DOI":"10.1145\/3362743.3362964"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Verma, G., Gupta, Y., Malik, A.M., and Chapman, B. (2021, January 17\u201321). Performance evaluation of deep learning compilers for edge inference. Proceedings of the IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Portland, OR, USA.","DOI":"10.1109\/IPDPSW52791.2021.00128"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hao, C., Sarwari, A., Jin, Z., Abu-Haimed, H., Sew, D., Li, Y., Liu, X., Wu, B., Fu, D., and Gu, J. (2019, January 20\u201323). A hybrid GPU+ FPGA system design for autonomous driving cars. Proceedings of the IEEE International Workshop on Signal Processing Systems (SiPS), Nanjing, China.","DOI":"10.1109\/SiPS47522.2019.9020540"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mujica, G., Rodriguez-Zurrunero, R., Wilby, M.R., Portilla, J., Rodr\u00edguez Gonz\u00e1lez, A.B., Araujo, A., Riesgo, T., and Vinagre Diaz, J.J. (2018). Edge and fog computing platform for data fusion of complex heterogeneous sensors. Sensors, 18.","DOI":"10.3390\/s18113630"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Fu, Y., Tian, D., Duan, X., Zhou, J., Lang, P., Lin, C., and You, X. (2020, January 19\u201323). A Camera\u2013Radar Fusion Method Based on Edge Computing. Proceedings of the IEEE International Conference on Edge Computing (EDGE), Beijing, China.","DOI":"10.1109\/EDGE50951.2020.00009"},{"key":"ref_30","unstructured":"Fadadu, S., Pandey, S., Hegde, D., Shi, Y., Chou, F.C., Djuric, N., and Vallespi-Gonzalez, C. (2020). Multi-view fusion of sensor data for improved perception and prediction in autonomous driving. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Mendez, J., Molina, M., Rodriguez, N., Cuellar, M.P., and Morales, D.P. (2021). Camera-LiDAR Multi-Level Sensor Fusion for Target Detection at the Network Edge. Sensors, 21.","DOI":"10.3390\/s21123992"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Warakagoda, N., Dirdal, J., and Faxvaag, E. (2019, January 2\u20135). Fusion of lidar and camera images in end-to-end deep learning for steering an off-road unmanned ground vehicle. Proceedings of the 22th International Conference on Information Fusion (FUSION), Ottawa, ON, Canada.","DOI":"10.23919\/FUSION43075.2019.9011341"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Yang, M., Wang, S., Bakita, J., Vu, T., Smith, F.D., Anderson, J.H., and Frahm, J.M. (2019, January 16\u201318). Re-thinking CNN frameworks for time-sensitive autonomous-driving applications: Addressing an industrial challenge. Proceedings of the IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), Montreal, QC, Canada.","DOI":"10.1109\/RTAS.2019.00033"},{"key":"ref_34","unstructured":"Microsoft (2022, July 21). ONNX Runtime. Available online: https:\/\/microsoft.github.io\/onnxruntime\/."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/16\/6097\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:08:57Z","timestamp":1760141337000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/16\/6097"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,15]]},"references-count":34,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["s22166097"],"URL":"https:\/\/doi.org\/10.3390\/s22166097","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,15]]}}}