{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T20:44:31Z","timestamp":1792701871801,"version":"4.5.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,11,16]],"date-time":"2021-11-16T00:00:00Z","timestamp":1637020800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,16]],"date-time":"2021-11-16T00:00:00Z","timestamp":1637020800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002809","name":"Generalitat de Catalunya","doi-asserted-by":"publisher","award":["2017-SGR-1414"],"award-info":[{"award-number":["2017-SGR-1414"]}],"id":[{"id":"10.13039\/501100002809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002809","name":"Generalitat de Catalunya","doi-asserted-by":"publisher","award":["2020 FI-B 00257"],"award-info":[{"award-number":["2020 FI-B 00257"]}],"id":[{"id":"10.13039\/501100002809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100011033","name":"Agencia Estatal de Investigaci\u00f3n","doi-asserted-by":"publisher","award":["PID2019-107255GB"],"award-info":[{"award-number":["PID2019-107255GB"]}],"id":[{"id":"10.13039\/501100011033","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100014374","name":"Universitat Polit\u00e8cnica de Catalunya","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100014374","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cluster Comput"],"published-print":{"date-parts":[[2022,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Containerization technology offers an appealing alternative for encapsulating and operating applications (and all their dependencies) without being constrained by the performance penalties of using Virtual Machines and, as a result, has got the interest of the High-Performance Computing (HPC) community to obtain fast, customized, portable, flexible, and reproducible deployments of their workloads. Previous work on this area has demonstrated that containerized HPC applications can exploit InfiniBand networks, but has ignored the potential of multi-container deployments which partition the processes that belong to each application into multiple containers in each host. Partitioning HPC applications has demonstrated to be useful when using virtual machines by constraining them to a single NUMA (Non-Uniform Memory Access) domain. This paper conducts a systematical study on the performance of multi-container deployments with different network fabrics and protocols, focusing especially on Infiniband networks. We analyze the impact of container granularity and its potential to exploit processor and memory affinity to improve applications\u2019 performance. Our results show that default Singularity can achieve near bare-metal performance but does not support fine-grain multi-container deployments. Docker and Singularity-instance have similar behavior in terms of the performance of deployment schemes with different container granularity and affinity. This behavior differs for the several network fabrics and protocols, and depends as well on the application communication patterns and the message size. Moreover, deployments on Infiniband are also more impacted by the computation and memory allocation, and because of that, they can exploit the affinity better.<\/jats:p>","DOI":"10.1007\/s10586-021-03460-8","type":"journal-article","created":{"date-parts":[[2021,11,16]],"date-time":"2021-11-16T19:04:01Z","timestamp":1637089441000},"page":"847-868","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Performance characterization of containerization for HPC workloads on InfiniBand clusters: an empirical study"],"prefix":"10.1007","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0058-8732","authenticated-orcid":false,"given":"Peini","family":"Liu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jordi","family":"Guitart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,11,16]]},"reference":[{"issue":"6","key":"3460_CR1","doi-asserted-by":"publisher","first-page":"931","DOI":"10.1109\/TPDS.2011.66","volume":"22","author":"A Iosup","year":"2011","unstructured":"Iosup, A., Ostermann, S., Yigitbasi, M.N., Prodan, R., Fahringer, T., Epema, D.: Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Trans. Parallel Distrib. Syst. 22(6), 931\u2013945 (2011). https:\/\/doi.org\/10.1109\/TPDS.2011.66","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"3460_CR2","doi-asserted-by":"publisher","unstructured":"Beltre, A.M., Saha, P., Govindaraju, M., Younge, A., Grant, R.E.: Enabling HPC workloads on cloud infrastructure using Kubernetes container orchestration mechanisms. In: Proceedings of CANOPIE-HPC 2019: 1st International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC, pp. 11\u201320 (2019). https:\/\/doi.org\/10.1109\/CANOPIE-HPC49598.2019.00007","DOI":"10.1109\/CANOPIE-HPC49598.2019.00007"},{"key":"3460_CR3","doi-asserted-by":"publisher","unstructured":"Liu, P., Guitart, J.: Performance comparison of multi-container deployment schemes for HPC workloads: an empirical study. Journal of Supercomputing (2020). https:\/\/doi.org\/10.1007\/s11227-020-03518-1","DOI":"10.1007\/s11227-020-03518-1"},{"key":"3460_CR4","doi-asserted-by":"publisher","unstructured":"Zhang, J., Lu, X., Panda, D.K.: High performance MPI library for container-based HPC cloud on InfiniBand clusters. In: 45th International Conference on Parallel Processing (ICPP), pp. 268\u2013277. IEEE (2016). https:\/\/doi.org\/10.1109\/ICPP.2016.38","DOI":"10.1109\/ICPP.2016.38"},{"issue":"10","key":"3460_CR5","doi-asserted-by":"publisher","first-page":"2683","DOI":"10.1109\/TPDS.2013.242","volume":"25","author":"KZ Ibrahim","year":"2014","unstructured":"Ibrahim, K.Z., Hofmeyr, S., Iancu, C.: The case for partitioning virtual machines on multicore architectures. IEEE Trans. Parallel Distrib. Syst. 25(10), 2683\u20132696 (2014). https:\/\/doi.org\/10.1109\/TPDS.2013.242","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"3460_CR6","unstructured":"Arango, C., Dernat, R., Sanabria, J.: Performance evaluation of container-based virtualization for high performance computing environments. CoRR (2017). arXiv:1709.10140"},{"key":"3460_CR7","doi-asserted-by":"publisher","unstructured":"Saha, P., Beltre, A., Uminski, P., Govindaraju, M.: Evaluation of Docker Containers for Scientific Workloads in the Cloud. In: Proceedings of Practice and Experience on Advanced Research Computing (PEARC18). ACM, New York (2018). https:\/\/doi.org\/10.1145\/3219104.3229280","DOI":"10.1145\/3219104.3229280"},{"key":"3460_CR8","doi-asserted-by":"publisher","unstructured":"Zhang, J., Lu, X., Panda, D.K.: Performance characterization of hypervisor-and container-based virtualization for HPC on SR-IOV enabled infiniband clusters. In: Proceedings of 30th International on Parallel and Distributed Processing Symposium (IPDPS16), pp. 1777\u20131784. IEEE (2016). https:\/\/doi.org\/10.1109\/IPDPSW.2016.178","DOI":"10.1109\/IPDPSW.2016.178"},{"key":"3460_CR9","doi-asserted-by":"publisher","unstructured":"Zhang, J., Lu, X., Panda, D.K.: Is singularity-based container technology ready for running MPI applications on HPC clouds? In: Proceedings of 10th International Conference on Utility and Cloud Computing (UCC17), pp. 151\u2013160. ACM, New York (2017). https:\/\/doi.org\/10.1145\/3147213.3147231","DOI":"10.1145\/3147213.3147231"},{"key":"3460_CR10","unstructured":"Saha, P., Beltre, A., Govindaraju, M.: Scylla: a mesos framework for container based MPI jobs. CoRR (2019). arXiv:1905.08386"},{"key":"3460_CR11","doi-asserted-by":"publisher","unstructured":"Chung, M.T., Quang-Hung, N., Nguyen, M.T., Thoai, N.: Using Docker in high performance computing applications. In: Proceedings of 6th International Conference on Communications and Electronics (ICCE16), pp. 52\u201357. IEEE (2016). https:\/\/doi.org\/10.1109\/CCE.2016.7562612","DOI":"10.1109\/CCE.2016.7562612"},{"key":"3460_CR12","doi-asserted-by":"publisher","unstructured":"Chung, M.T., Le, A., Quang-Hung, N., Nguyen, D., Thoai, N.: Provision of Docker and InfiniBand in high performance computing. In: Proceedings of the 2016 International Conference on Advanced Computing and Applications, ACOMP16, pp. 127\u2013134. IEEE (2016). https:\/\/doi.org\/10.1109\/ACOMP.2016.027","DOI":"10.1109\/ACOMP.2016.027"},{"issue":"7","key":"3460_CR13","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1145\/3140607.3050765","volume":"52","author":"J Zhang","year":"2017","unstructured":"Zhang, J., Lu, X., Panda, D.K.: Designing locality and NUMA aware MPI runtime for nested virtualization based HPC cloud with SR-IOV enabled InfiniBand. SIGPLAN Not. 52(7), 187\u2013200 (2017). https:\/\/doi.org\/10.1145\/3140607.3050765","journal-title":"SIGPLAN Not."},{"issue":"5","key":"3460_CR14","doi-asserted-by":"publisher","first-page":"e0177459","DOI":"10.1371\/journal.pone.0177459","volume":"12","author":"GM Kurtzer","year":"2017","unstructured":"Kurtzer, G.M., Sochat, V., Bauer, M.W.: Singularity: scientific containers for mobility of compute. PLoS ONE 12(5), e0177459 (2017). https:\/\/doi.org\/10.1371\/journal.pone.0177459","journal-title":"PLOS ONE"},{"key":"3460_CR15","unstructured":"HPC Wire: Sylabs releases singularity 3.0 container platform. Cites AI Support (2018). https:\/\/www.hpcwire.com\/2018\/10\/08\/sylabs-releases-singularity-3-0-container-platform-cites-ai-support\/"},{"issue":"7","key":"3460_CR16","doi-asserted-by":"publisher","first-page":"1087","DOI":"10.1002\/spe.2793","volume":"50","author":"C Sauvanaud","year":"2020","unstructured":"Sauvanaud, C., Dholakia, A., Guitart, J., Kim, C., Mayes, P.: Big data deployment in containerized infrastructures through the interconnection of network namespaces. Softw. Pract. Exp. 50(7), 1087\u20131113 (2020). https:\/\/doi.org\/10.1002\/spe.2793","journal-title":"Software: Practice and Experience"},{"key":"3460_CR17","volume-title":"InfiniBand Network Architecture","author":"T Shanley","year":"2002","unstructured":"Shanley, T.: InfiniBand Network Architecture. Addison Wesley, Boston (2002)"},{"key":"3460_CR18","doi-asserted-by":"publisher","unstructured":"Subramoni, H., Lai, P., Luo, M., Panda, D.K.: RDMA over Ethernet: a preliminary study. In: Proceedings of the IEEE International Conference on Cluster Computing and Workshops (CLUSTER09), pp. 1\u20139 (2009). https:\/\/doi.org\/10.1109\/CLUSTR.2009.5289144","DOI":"10.1109\/CLUSTR.2009.5289144"},{"key":"3460_CR19","unstructured":"Grun, P.: Introduction to Infiniband for end users. Technical Report. InfiniBand Trade Association (2010)"},{"key":"3460_CR20","doi-asserted-by":"publisher","unstructured":"Luszczek, P.R., Bailey, D.H., Dongarra, J.J., Kepner, J., Lucas, R.F., Rabenseifner, R., Takahashi, D.: The HPC Challenge (HPCC) benchmark suite. In: Proceedings of the 2006 ACM\/IEEE Conference on Supercomputing (SC06), pp. 213\u2013es. ACM, New York (2006). https:\/\/doi.org\/10.1145\/1188455.1188677","DOI":"10.1145\/1188455.1188677"},{"key":"3460_CR21","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1016\/j.procs.2014.05.019","volume":"29","author":"F Xing","year":"2014","unstructured":"Xing, F., You, H., Lu, C.: HPC benchmark assessment with statistical analysis. Procedia Comput. Sci. 29, 210\u2013219 (2014). https:\/\/doi.org\/10.1016\/j.procs.2014.05.019","journal-title":"Procedia Computer Science"},{"key":"3460_CR22","unstructured":"HPC Advisory Council: HPCC performance benchmark and profiling (2015). https:\/\/hpcadvisorycouncil.com\/pdf\/HPCC_Analysis_and_Profiling_Intel_E5-2697v3.pdf"},{"key":"3460_CR23","doi-asserted-by":"publisher","unstructured":"Tudor, B.M., Teo, Y.M.: A practical approach for performance analysis of shared-memory programs. In: Proceedings of the 2011 IEEE International Parallel Distributed Processing Symposium, pp. 652\u2013663 (2011). https:\/\/doi.org\/10.1109\/IPDPS.2011.68","DOI":"10.1109\/IPDPS.2011.68"},{"key":"3460_CR24","unstructured":"HPC Advisory Council: Interconnect analysis: 10GigE and InfiniBand in high performance computing. White Paper (2009). https:\/\/www.hpcadvisorycouncil.com\/pdf\/IB_and_10GigE_in_HPC.pdf"},{"key":"3460_CR25","doi-asserted-by":"publisher","unstructured":"Younge, A.J., Pedretti, K., Grant, R.E., Brightwell, R.: A tale of two systems: using containers to deploy HPC applications on supercomputers and clouds. In: Proceedings of the 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 74\u201381 (2017). https:\/\/doi.org\/10.1109\/CloudCom.2017.40","DOI":"10.1109\/CloudCom.2017.40"},{"key":"3460_CR26","doi-asserted-by":"publisher","unstructured":"Rudyy, O., Garcia-Gasulla, M., Mantovani, F., Santiago, A., Sirvent, R., V\u00e1zquez, M.: Containers in HPC: a scalability and portability study in production biological simulations. In: Proceedings of the 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 567\u2013577 (2019). https:\/\/doi.org\/10.1109\/IPDPS.2019.00066","DOI":"10.1109\/IPDPS.2019.00066"}],"container-title":["Cluster Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03460-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10586-021-03460-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03460-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,3,19]],"date-time":"2022-03-19T20:07:29Z","timestamp":1647720449000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10586-021-03460-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,16]]},"references-count":26,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4]]}},"alternative-id":["3460"],"URL":"https:\/\/doi.org\/10.1007\/s10586-021-03460-8","relation":{},"ISSN":["1386-7857","1573-7543"],"issn-type":[{"value":"1386-7857","type":"print"},{"value":"1573-7543","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,16]]},"assertion":[{"value":"15 February 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 October 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 October 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 November 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}