{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T11:59:39Z","timestamp":1780401579782,"version":"3.54.1"},"reference-count":45,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,2,16]],"date-time":"2021-02-16T00:00:00Z","timestamp":1613433600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["NRF-2020R1A2C2008447"],"award-info":[{"award-number":["NRF-2020R1A2C2008447"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010418","name":"Institute for Information and Communications Technology Promotion","doi-asserted-by":"publisher","award":["IITP-2018-2015-0-00742"],"award-info":[{"award-number":["IITP-2018-2015-0-00742"]}],"id":[{"id":"10.13039\/501100010418","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Containers virtually package a piece of software and share the host Operating System (OS) upon deployment. This makes them notably light weight and suitable for dynamic service deployment at the network edge and Internet of Things (IoT) devices for reduced latency and energy consumption. Data collection, computation, and now intelligence is included in variety of IoT devices which have very tight latency and energy consumption conditions. Recent studies satisfy latency condition through containerized services deployment on IoT devices and gateways. They fail to account for the limited energy and computing resources of these devices which limit the scalability and concurrent services deployment. This paper aims to establish guidelines and identify critical factors for containerized services deployment on resource constrained IoT devices. For this purpose, two container orchestration tools (i.e., Docker Swarm and Kubernetes) are tested and compared on a baseline IoT gateways testbed. Experiments use Deep Learning driven data analytics and Intrusion Detection System services, and evaluate the time it takes to prepare and deploy a container (creation time), Central Processing Unit (CPU) utilization for concurrent containers deployment, memory usage under different traffic loads, and energy consumption. The results indicate that container creation time and memory usage are decisive factors for containerized micro service architecture.<\/jats:p>","DOI":"10.3390\/s21041378","type":"journal-article","created":{"date-parts":[[2021,2,16]],"date-time":"2021-02-16T08:09:09Z","timestamp":1613462949000},"page":"1378","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Empirical Performance and Energy Consumption Evaluation of Container Solutions on Resource Constrained IoT Gateways"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6580-3232","authenticated-orcid":false,"given":"Syed M.","family":"Raza","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaeyeop","family":"Jeong","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2692-6883","authenticated-orcid":false,"given":"Moonseong","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Liberal Arts, Seoul Theological University, Bucheon 14754, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Byungseok","family":"Kang","sequence":"additional","affiliation":[{"name":"Department of Electronics, Computing and Mathematics, University of Derby, Derby DE22 1FX, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyunseung","family":"Choo","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Sungkyunkwan University, Suwon 16419, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"59172","DOI":"10.1109\/ACCESS.2019.2915020","article-title":"iDiSC: A New Approach to IoT-Data-Intensive Service Components Deployment in Edge-Cloud-Hybrid System","volume":"7","author":"Chen","year":"2019","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kumar, A., Narendra, N.C., and Bellur, U. (2016, January 19). Uploading and Replicating Internet of Things (IoT) Data on Distributed Cloud Storage. Proceedings of the 2016 IEEE 9th International Conference on Cloud Computing (CLOUD), San Francisco, CA, USA.","DOI":"10.1109\/CLOUD.2016.0094"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1016\/j.future.2017.02.014","article-title":"Exploiting smart e-Health gateways at the edge of healthcare Internet-of-Things: A fog computing approach","volume":"78","author":"Rahmani","year":"2018","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1109\/JIOT.2017.2788802","article-title":"On Reducing IoT Service Delay via Fog Offloading","volume":"5","author":"Yousefpour","year":"2018","journal-title":"IEEE Internet Things J."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Chang, Z., Liu, L., Guo, X., and Sheng, Q. (2020). Dynamic Resource Allocation and Computation Offloading for IoT Fog Computing System. IEEE Trans. Ind. Inform.","DOI":"10.1109\/TII.2020.2978946"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1109\/TETCI.2019.2902869","article-title":"Resource Sharing and Task Offloading in IoT Fog Computing: A Contract-Learning Approach","volume":"4","author":"Zhou","year":"2020","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Yang, Y., Luo, X., Chu, X., and Zhou, M.T. (2020). Fog Computing Architecture and Technologies. Fog-Enabled Intelligent IoT Systems, Springer International Publishing.","DOI":"10.1007\/978-3-030-23185-9"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.comnet.2018.07.020","article-title":"IoT survey: An SDN and fog computing perspective","volume":"143","author":"Ola","year":"2018","journal-title":"Comput. Networks"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"(2019). A survey of edge computing based designs for IoT security. Digit. Commun. Netw., 6, 195\u2013202.","DOI":"10.1016\/j.dcan.2019.08.006"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"6882","DOI":"10.1109\/JIOT.2020.2970501","article-title":"Passban IDS: An Intelligent Anomaly Based Intrusion Detection System for IoT Edge Devices","volume":"7","author":"Eskandari","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"4660","DOI":"10.1109\/JSEN.2019.2900097","article-title":"NETRA: Enhancing IoT Security Using NFV-Based Edge Traffic Analysis","volume":"19","author":"Sairam","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1109\/MC.2016.245","article-title":"Fog Computing: Helping the Internet of Things Realize Its Potential","volume":"49","author":"Dastjerdi","year":"2016","journal-title":"Computer"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ismail, B.I., Mostajeran Goortani, E., Ab Karim, M.B., Ming Tat, W., Setapa, S., Luke, J.Y., and Hong Hoe, O. (2016, January 11). Evaluation of Docker as Edge computing platform. Proceedings of the 2015 IEEE Conference on Open Systems (ICOS), Melaka, Malaysia.","DOI":"10.1109\/ICOS.2015.7377291"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.comcom.2020.04.061","article-title":"Geo-distributed efficient deployment of containers with Kubernetes","volume":"159","author":"Fabiana","year":"2020","journal-title":"Comput. Commun."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Sheoran, A., Bu, X., Cao, L., Sharma, P., and Fahmy, S. (2017, January 8). An empirical case for container-driven fine-grained VNF resource flexing. Proceedings of the 2016 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), Palo Alto, CA, USA.","DOI":"10.1109\/NFV-SDN.2016.7919486"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Dolui, K., and Kiraly, C. (2019, January 21). Towards Multi-Container Deployment on IoT Gateways. Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, United Arab Emirates.","DOI":"10.1109\/GLOCOM.2018.8647688"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Hong, H., Tsai, P., and Hsu, C. (2016, January 10). Dynamic module deployment in a fog computing platform. Proceedings of the 2016 18th Asia-Pacific Network Operations and Management Symposium (APNOMS), Kanazawa, Japan.","DOI":"10.1109\/APNOMS.2016.7737202"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Chen, J.L., Pang, A.C., Deng, D.J., and Lin, C.C. (2019). Container-Based Virtualization for Real-Time Data Streaming Processing on the Edge Computing Architecture. Wireless Internet, Springer International Publishing.","DOI":"10.1007\/978-3-030-06158-6"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hoque, S., De Brito, M.S., Willner, A., Keil, O., and Magedanz, T. (2017, January 11). Towards Container Orchestration in Fog Computing Infrastructures. Proceedings of the 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC), Turin, Italy.","DOI":"10.1109\/COMPSAC.2017.248"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Khazaei, H., Bannazadeh, H., and Leon-Garcia, A. (2017, January 15). End-to-end management of IoT applications. Proceedings of the 2017 IEEE Conference on Network Softwarization (NetSoft), Bologna, Italy.","DOI":"10.1109\/NETSOFT.2017.8004252"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Bellavista, P., and Zanni, A. (2017, January 7). Feasibility of Fog Computing Deployment Based on Docker Containerization over RaspberryPi. Proceedings of the 18th International Conference on Distributed Computing and Networking (ICDCN \u201917), Hyderabad, India.","DOI":"10.1145\/3007748.3007777"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"8835","DOI":"10.1109\/ACCESS.2017.2704444","article-title":"Virtualization on Internet of Things Edge Devices With Container Technologies: A Performance Evaluation","volume":"5","author":"Morabito","year":"2017","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kaup, F., Gottschling, P., and Hausheer, D. (2014, January 16). PowerPi: Measuring and modeling the power consumption of the Raspberry Pi. Proceedings of the 39th Annual IEEE Conference on Local Computer Networks, Edmonton, Canada.","DOI":"10.1109\/LCN.2014.6925777"},{"key":"ref_24","unstructured":"StackRox (2021, February 01). Kubernetes and Container Security and Adoption Trends. Available online: https:\/\/www.stackrox.com\/kubernetes-adoption-security-and-market-share-for-containers\/."},{"key":"ref_25","unstructured":"Zabbix (2020, April 20). Zabbix features. Available online: https:\/\/www.zabbix.com\/features."},{"key":"ref_26","unstructured":"MarketsandMarkets (2021, February 01). Container Orchestration Market by Component (Platform and Services), Organization Size (Large Enterprises and SMEs), Vertical (Telecommunications and IT, BFSI, Government and Public Sector, Manufacturing), and Region - Global Forecast to 2023. Available online: https:\/\/www.marketsandmarkets.com\/Market-Reports\/container-orchestration-market-13241043.html."},{"key":"ref_27","unstructured":"\u201cG2\u201d (2021, February 01). Best Container Orchestration Software. Available online: https:\/\/www.g2.com\/categories\/container-orchestration."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Truyen, E., Van Landuyt, D., Preuveneers, D., Lagaisse, B., and Joosen, W. (2019). A Comprehensive Feature Comparison Study of Open-Source Container Orchestration Frameworks. Appl. Sci., 9.","DOI":"10.3390\/app9050931"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Jawarneh, I.M.A., Bellavista, P., Bosi, F., Foschini, L., Martuscelli, G., Montanari, R., and Palopoli, A. (2019, January 15). Container Orchestration Engines: A Thorough Functional and Performance Comparison. Proceedings of the ICC 2019\u20142019 IEEE International Conference on Communications (ICC), Shanghai, China.","DOI":"10.1109\/ICC.2019.8762053"},{"key":"ref_30","unstructured":"Tugba, G., G\u00f6ker, C., and \u00d6zg\u00fcn, S. (2020, January 18). MARKA: A Microservice Architecture-Based Application Performance Comparison Between Docker Swarm and Kubernetes. Proceedings of the ICSEA - 2020 The Fifteenth International Conference on Software Engineering Advances, Porto, Portugal."},{"key":"ref_31","unstructured":"(2020, October 08). Marka.club. Available online: https:\/\/github.com\/kodkafa\/marka.club."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"102788","DOI":"10.1016\/j.jnca.2020.102788","article-title":"Performance comparison of container orchestration platforms with low cost devices in the fog, assisting Internet of Things applications","volume":"169","author":"Cobos","year":"2020","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_33","first-page":"35","article-title":"Measurement of Performance and Energy Consumption of OpenCV Programs on Raspberry Pi","volume":"9","author":"Kadota","year":"2020","journal-title":"Bull. Netw. Comput. Syst. Softw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Jeong, J., Raza, S.M., Kim, M., Kang, B., Jang, B., and Choo, H. (2020, January 23). Empirical Analysis of Containers on Resource Constrained IoT Gateway. Proceedings of the 2020 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, CA, USA.","DOI":"10.1109\/ICCE46568.2020.9043056"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1109\/COMST.2015.2477041","article-title":"Network Function Virtualization: State-of-the-Art and Research Challenges","volume":"18","author":"Mijumbi","year":"2016","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_36","unstructured":"Rotter, C., Farkas, L., Nyiri, G., Csatari, G., and J\u00e1nosi, L. (2016, January 3). Using Linux Containers in Telecom Applications. Proceedings of the 19th International ICIN Conference\u2014Innovations in Clouds, Internet and Networks, Paris, France."},{"key":"ref_37","unstructured":"Sayfan, G. (2021, February 01). Mastering Kubernetes: Large Scale Container Deployment and Management. Packt Publishing, Available online: https:\/\/books.google.co.kr\/books?id=nPBZDwAAQBAJ&lpg=PP1&dq=Mastering%20kubernetes&pg=PP1#v=onepage&q=Mastering%20kubernetes&f=false."},{"key":"ref_38","unstructured":"TensorFlow (2020, April 20). Introduction to TensorFlow. Available online: https:\/\/www.tensorflow.org\/learn."},{"key":"ref_39","unstructured":"Keras (2020, April 20). Keras documentation. Available online: https:\/\/www.tensorflow.org\/learn."},{"key":"ref_40","unstructured":"Pytorch (2020, April 20). Pytorch features. Available online: https:\/\/pytorch.org\/features."},{"key":"ref_41","unstructured":"Snort (2020, April 20). Snort Users Manual. Available online: https:\/\/www.snort.org\/documents."},{"key":"ref_42","unstructured":"Cisco (2020, April 20). What Is a Firewall. Available online: https:\/\/www.cisco.com\/c\/en\/us\/products\/security\/firewalls\/what-is-a-firewall.html?dtid=osscdc000283."},{"key":"ref_43","unstructured":"TcpReplay (2020, April 20). Overview. Available online: http:\/\/tcpreplay.appneta.com\/wiki\/captures.html#overview."},{"key":"ref_44","unstructured":"\u201cTcpReplay\u201d (2021, February 01). bigFlows.pcap. Available online: http:\/\/tcpreplay.appneta.com\/wiki\/captures.html#bigflows-pcap."},{"key":"ref_45","first-page":"18","article-title":"Energy Consumption in Mobile Phones","volume":"9","author":"Javed","year":"2017","journal-title":"Int. J. Comput. Netw. Inf. Secur. (IJCNIS)"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1378\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:24:46Z","timestamp":1760160286000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1378"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,16]]},"references-count":45,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["s21041378"],"URL":"https:\/\/doi.org\/10.3390\/s21041378","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,16]]}}}