{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,26]],"date-time":"2025-12-26T07:09:24Z","timestamp":1766732964131,"version":"build-2065373602"},"reference-count":50,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2022,9,5]],"date-time":"2022-09-05T00:00:00Z","timestamp":1662336000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010418","name":"Institute of Information &amp; Communications Technology Planning &amp; Evaluation","doi-asserted-by":"publisher","award":["2021-0-01578"],"award-info":[{"award-number":["2021-0-01578"]}],"id":[{"id":"10.13039\/501100010418","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>As deep learning technology paves its way, real-world applications that make use of it become popular these days. Edge computing architecture is one of the service architectures to realize the deep learning based service, which makes use of the resources near the data source or client. In Edge computing architecture it becomes important to manage resource usage, and there is research on optimization of deep learning, such as pruning or binarization, which makes deep learning models more lightweight, along with the research for the efficient distribution of workloads on cloud or edge resources. Those are to reduce the workload on edge resources. In this paper, a usage optimization method with batch and model management is proposed. The proposed method is to increase the utilization of GPU resource by modifying the batch size of the input of an inference application. To this end, the inference pipelines are identified to see how the different kinds of resources are used, and then the effect of batch inference on GPU is measured. The proposed method consists of a few modules, including a tool for batch size management which is able to change a batch size with respect to the available resources, and another one for model management which supports on-the-fly update of a model. The proposed methods are implemented on a real-time video analysis application and deployed in the Kubernetes cluster as a Docker container. The result shows that the proposed method can optimize the usage of edge resources for real-time video analysis deep learning applications.<\/jats:p>","DOI":"10.3390\/s22176717","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T04:18:32Z","timestamp":1662610712000},"page":"6717","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Optimization of Edge Resources for Deep Learning Application with Batch and Model Management"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6834-9255","authenticated-orcid":false,"given":"Seungwoo","family":"Kum","sequence":"first","affiliation":[{"name":"Korea Electronics Technology Institute, Seongnam 13509, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6679-6166","authenticated-orcid":false,"given":"Seungtaek","family":"Oh","sequence":"additional","affiliation":[{"name":"Korea Electronics Technology Institute, Seongnam 13509, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2504-0792","authenticated-orcid":false,"given":"Jeongcheol","family":"Yeom","sequence":"additional","affiliation":[{"name":"Korea Electronics Technology Institute, Seongnam 13509, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7451-6411","authenticated-orcid":false,"given":"Jaewon","family":"Moon","sequence":"additional","affiliation":[{"name":"Korea Electronics Technology Institute, Seongnam 13509, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,5]]},"reference":[{"key":"ref_1","unstructured":"(2022, July 18). NVIDIA Jetson Platforms. Available online: https:\/\/www.nvidia.com\/en-us\/autonomous-machines\/embedded-systems\/."},{"key":"ref_2","unstructured":"(2022, July 18). Google Coral. Available online: https:\/\/coral.ai."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tsaramirsis, G., Kantaros, A., Al-Darraji, I., Piromalis, D., Apostolopoulos, C., Pavlopoulou, A., Alrammal, M., Ismail, Z., Buhari, S.M., and Stojmenovic, M. (2022). A Modern Approach towards an Industry 4.0 Model: From Driving Technologies to Management. J. Sens., 2022.","DOI":"10.1155\/2022\/5023011"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ying, Z., Ahmad, I., Mateen, S., Zia, A., Nazir, S., and Mukhtar, N. (2021). An Overview of Computational Models for Industrial Internet of Things to Enhance Usability. Complexity, 2021.","DOI":"10.1155\/2021\/5554685"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Liang, F., Zhou, Y., Chen, X., Liu, F., Zhang, C., and Wu, X. (2021, January 14\u201316). Review of Target Detection Technology based on Deep Learning. Proceedings of the 5th International Conference on Control Engineering and Artificial Intelligence, Sanya, China.","DOI":"10.1145\/3448218.3448234"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Hou, D., Liu, T., Pan, Y.T., and Hou, J. (2019, January 7\u20139). AI on edge device for laser chip defect detection. Proceedings of the 2019 IEEE 9th Annual Computing and Communication Workshop and Conference, CCWC 2019, Las Vegas, NV, USA.","DOI":"10.1109\/CCWC.2019.8666503"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Qi, S., Yang, J., and Zhong, Z. (2020, January 18\u201320). A Review on Industrial Surface Defect Detection Based on Deep Learning Technology. Proceedings of the 2020 The 3rd International Conference on Machine Learning and Machine Intelligence, Hangzhou, China.","DOI":"10.1145\/3426826.3426832"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Larsson, L., Gustafsson, H., Klein, C., and Elmroth, E. (2020, January 7\u201310). Decentralized Kubernetes Federation Control Plane. Proceedings of the 2020 IEEE\/ACM 13th International Conference on Utility and Cloud Computing (UCC), Leicester, UK.","DOI":"10.1109\/UCC48980.2020.00056"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1109\/TVT.2021.3127011","article-title":"AirEdge: A Dependency-Aware Multi-Task Orchestration in Federated Aerial Computing","volume":"71","author":"Awada","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"958","DOI":"10.1109\/TNSM.2021.3052837","article-title":"Machine Learning-Based Scaling Management for Kubernetes Edge Clusters","volume":"18","author":"Toka","year":"2021","journal-title":"IEEE Trans. Netw. Serv. Manag."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Han, Y., Shen, S., Wang, X., Wang, S., and Leung, V.C. (2021, January 10\u201313). Tailored Learning-Based Scheduling for Kubernetes-Oriented Edge-Cloud System. Proceedings of the IEEE INFOCOM 2021\u2014IEEE Conference on Computer Communications, Vancouver, BC, Canada.","DOI":"10.1109\/INFOCOM42981.2021.9488701"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Hadidi, R., Cao, J., Xie, Y., Asgari, B., Krishna, T., and Kim, H. (2019, January 3\u20135). Characterizing the Deployment of Deep Neural Networks on Commercial Edge Devices. Proceedings of the 2019 IEEE International Symposium on Workload Characterization, IISWC 2019, Orlando, FL, USA.","DOI":"10.1109\/IISWC47752.2019.9041955"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tuli, S., Basumatary, N., and Buyya, R. (2019, January 21\u201322). EdgeLens: Deep Learning based Object Detection in Integrated IoT, Fog and Cloud Computing Environments. Proceedings of the 2019 4th International Conference on Information Systems and Computer Networks, ISCON 2019, Mathura, India.","DOI":"10.1109\/ISCON47742.2019.9036216"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Vater, J., Schlaak, P., and Knoll, A. (2020, January 13\u201317). A Modular Edge-\/Cloud-Solution for Automated Error Detection of Industrial Hairpin Weldings using Convolutional Neural Networks. Proceedings of the 2020 IEEE 44th Annual Computers, Software, and Applications Conference, COMPSAC 2020, Madrid, Spain.","DOI":"10.1109\/COMPSAC48688.2020.0-202"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Hou, X., Guan, Y., Han, T., and Zhang, N. (June, January 30). DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices. Proceedings of the 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), Lyon, France.","DOI":"10.1109\/IPDPS53621.2022.00110"},{"key":"ref_16","unstructured":"Merkel, D. (2014). Docker: Lightweight linux containers for consistent development and deployment. Linux J., 2014."},{"key":"ref_17","unstructured":"Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Jeong, E., Kim, J., and Ha, S. (2022). TensorRT-based Framework and Optimization Methodology for Deep Learning Inference on Jetson Boards. ACM Trans. Embed. Comput. Syst.","DOI":"10.1145\/3508391"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mao, Y., Fu, Y., Gu, S., Vhaduri, S., Cheng, L., and Liu, Q. (2020). Resource Management Schemes for Cloud-Native Platforms with Computing Containers of Docker and Kubernetes. arXiv.","DOI":"10.36227\/techrxiv.13146548.v1"},{"key":"ref_20","unstructured":"Rabenstein, B., and Volz, J. (2015). Prometheus: A Next-Generation Monitoring System (Talk), USENIX Association."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1109\/MS.2018.2141035","article-title":"Using Microservices for Legacy Software Modernization","volume":"35","author":"Knoche","year":"2018","journal-title":"IEEE Softw."},{"key":"ref_22","first-page":"76","article-title":"The Design and Architecture of Microservices","volume":"3","author":"Sill","year":"2016","journal-title":"IEEE Cloud Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"869","DOI":"10.1109\/COMST.2020.2970550","article-title":"Convergence of Edge Computing and Deep Learning: A Comprehensive Survey","volume":"22","author":"Wang","year":"2020","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_24","first-page":"2046","article-title":"Collaborative edge and cloud neural networks for real-time video processing","volume":"11","author":"Grulich","year":"2018","journal-title":"Assoc. Comput. Mach."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mudassar, B.A., Ko, J.H., and Mukhopadhyay, S. (2018, January 24\u201329). Edge-cloud collaborative processing for intelligent internet of things: A case study on smart surveillance. Proceedings of the 55th Annual Design Automation Conference, San Francisco, CA, USA.","DOI":"10.1145\/3195970.3196036"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Dey, S., and Mukherjee, A. (2018, January 19\u201323). Implementing Deep Learning and Inferencing on Fog and Edge Computing Systems. Proceedings of the 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Athens, Greece.","DOI":"10.1109\/PERCOMW.2018.8480168"},{"key":"ref_27","unstructured":"Han, S., Mao, H., and Dally, W.J. (2015). Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv."},{"key":"ref_28","unstructured":"(2022, July 18). TensorRT. Available online: http:\/\/https:\/\/developer.nvidia.com\/tensorrt."},{"key":"ref_29","unstructured":"Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or \u22121. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1109\/TWC.2019.2946140","article-title":"Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing","volume":"19","author":"Li","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_31","unstructured":"Jiang, Z., Chen, T., and Li, M. (2018, January 15\u201316). Efficient Deep Learning Inference on Edge Devices. Proceedings of the SysML\u201918, Stanford, CA, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Li, H., Hu, C., Jiang, J., Wang, Z., Wen, Y., and Zhu, W. (2018, January 11\u201313). JALAD: Joint Accuracy-And Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution. Proceedings of the 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS), Singapore.","DOI":"10.1109\/PADSW.2018.8645013"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Goel, A., Tung, C., Hu, X., Thiruvathukal, G.K., Davis, J.C., and Lu, Y.H. (2022, January 17\u201320). Efficient Computer Vision on Edge Devices with Pipeline-Parallel Hierarchical Neural Networks. Proceedings of the 2022 27th Asia and South Pacific Design Automation Conference (ASP-DAC), Taipei, Taiwan.","DOI":"10.1109\/ASP-DAC52403.2022.9712574"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Hsu, K.J., Bhardwaj, K., and Gavrilovska, A. (2019, January 7\u20139). Couper: DNN model slicing for visual analytics containers at the edge. Proceedings of the 4th ACM\/IEEE Symposium on Edge Computing, Arlington, VA, USA.","DOI":"10.1145\/3318216.3363309"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Coello, C., Sanchez, R., de Lange, S., Halvorsen, J., Bertani-\u00d8kland, M., Myrvang, V., and Heitmann, S. (2021, January 20\u201323). Workflowfor training and serving deep learning models for image classification and object detection\u2014Application to fault detection on electric poles. Proceedings of the CIRED 2021\u2014The 26th International Conference and Exhibition on Electricity Distribution, Geneva, Switzerland.","DOI":"10.1049\/icp.2021.1557"},{"key":"ref_36","unstructured":"Meloni, P., Loi, D., Busia, P., Deriu, G., Pimentel, A.D., Sapra, D., Stefanov, T., Minakova, S., Conti, F., and Benini, L. (May, January 30). Optimization and deployment of CNNs at the Edge: The ALOHA experience. Proceedings of the 16th ACM International Conference on Computing Frontiers, Alghero, Italy."},{"key":"ref_37","unstructured":"(2022, July 18). KubeFlow. Available online: http:\/\/www.kubeflow.org."},{"key":"ref_38","unstructured":"(2022, July 18). BentoML. Available online: https:\/\/github.com\/bentoml\/BentoML."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.jpdc.2020.08.006","article-title":"DLHub: Simplifying publication, discovery, and use of machine learning models in science","volume":"147","author":"Li","year":"2021","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Crankshaw, D., Sela, G.E., Mo, X., Zumar, C., Stoica, I., Gonzalez, J., and Tumanov, A. (2020, January 19\u201321). InferLine: Latency-aware provisioning and scaling for prediction serving pipelines. Proceedings of the 11th ACM Symposium on Cloud Computing, Virtual.","DOI":"10.1145\/3419111.3421285"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Lemay, M., Li, S., and Guo, T. (2020, January 21\u201324). PERSEUS: Characterizing Performance and Cost of Multi-Tenant Serving for CNN Models. Proceedings of the 2020 IEEE International Conference on Cloud Engineering (IC2E), Sydney, Australia.","DOI":"10.1109\/IC2E48712.2020.00014"},{"key":"ref_42","unstructured":"Romero, F., Li, Q., Yadwadkar, N.J., and Kozyrakis, C. (2019). INFaaS: A Model-less and Managed Inference Serving System. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Yadwadkar, N.J., Romero, F., Li, Q., and Kozyrakis, C. (2019, January 13\u201315). A Case for Managed and Model-less Inference Serving. Proceedings of the Workshop on Hot Topics in Operating Systems, Bertinoro, Italy.","DOI":"10.1145\/3317550.3321443"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Yu, M., Jiang, Z., Ng, H.C., Wang, W., Chen, R., and Li, B. (2021, January 7\u201310). Gillis: Serving large neural networks in serverless functions with automatic model partitioning. Proceedings of the 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS), Washington, DC, USA.","DOI":"10.1109\/ICDCS51616.2021.00022"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Xiang, Y., and Kim, H. (2019, January 3\u20136). Pipelined data-parallel CPU\/GPU scheduling for multi-DNN real-time inference. Proceedings of the 2019 IEEE Real-Time Systems Symposium (RTSS), Hong Kong, China.","DOI":"10.1109\/RTSS46320.2019.00042"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Hu, Y., Rallapalli, S., Ko, B., and Govindan, R. (2018, January 10\u201314). Olympian: Scheduling GPU usage in a deep neural network model serving system. Proceedings of the 19th International Middleware Conference, Rennes, France.","DOI":"10.1145\/3274808.3274813"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Al-Rakhami, M., Alsahli, M., Hassan, M.M., Alamri, A., Guerrieri, A., and Fortino, G. (2018, January 12\u201315). Cost efficient edge intelligence framework using docker containers. Proceedings of the 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC\/PiCom\/DataCom\/CyberSciTech), Athens, Greece.","DOI":"10.1109\/DASC\/PiCom\/DataCom\/CyberSciTec.2018.00138"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Verucchi, M., Brilli, G., Sapienza, D., Verasani, M., Arena, M., Gatti, F., Capotondi, A., Cavicchioli, R., Bertogna, M., and Solieri, M. (2020, January 8\u201311). A Systematic Assessment of Embedded Neural Networks for Object Detection. Proceedings of the IEEE International Conference on Emerging Technologies and Factory Automation, ETFA, Vienna, Austria.","DOI":"10.1109\/ETFA46521.2020.9212130"},{"key":"ref_49","unstructured":"Armani, V., Faticanti, F., Cretti, S., Kum, S., and Siracusa, D. (2021). A Cost-Effective Workload Allocation Strategy for Cloud-Native Edge Services. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Kum, S., Kim, Y., Siracusa, D., and Moon, J. (2020, January 9\u201311). Artificial Intelligence Service Architecture for Edge Device. Proceedings of the 2020 IEEE 10th International Conference on Consumer Electronics (ICCE-Berlin), Berlin, Germany.","DOI":"10.1109\/ICCE-Berlin50680.2020.9352184"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/17\/6717\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:23:56Z","timestamp":1760142236000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/17\/6717"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,5]]},"references-count":50,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2022,9]]}},"alternative-id":["s22176717"],"URL":"https:\/\/doi.org\/10.3390\/s22176717","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,9,5]]}}}