{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T12:03:33Z","timestamp":1780401813865,"version":"3.54.1"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T00:00:00Z","timestamp":1623196800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T00:00:00Z","timestamp":1623196800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100012470","name":"CERN","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100012470","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cluster Comput"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The Compact Muon Solenoid (CMS) experiment heavily relies on the CMSWEB cluster to host critical services for its operational needs. The cluster is deployed on virtual machines (VMs) from the CERN OpenStack cloud and is manually maintained by operators and developers. The release cycle is composed of several steps, from building RPMs to their deployment, validation, and integration tests. To enhance the sustainability of the CMSWEB cluster, CMS decided to migrate its cluster to a containerized solution based on Docker and orchestrated with Kubernetes (K8s). This allows us to significantly speed up the release upgrade cycle, follow the end-to-end deployment procedure, and reduce operational cost. In this paper, we give an overview of the CMSWEB VM cluster and the issues we discovered during this migration. We discuss the architecture and the implementation strategy in the CMSWEB Kubernetes cluster. Even though Kubernetes provides horizontal pod autoscaling based on CPUs and memory, in this paper, we provide details of horizontal pod autoscaling based on the custom metrics of CMSWEB services. We also discuss automated deployment procedure based on the best practices of continuous integration\/continuous deployment (CI\/CD) workflows. We present performance analysis between Kubernetes and VM based CMSWEB deployments. Finally, we describe various issues found during the implementation in Kubernetes and report on lessons learned during the migration process.<\/jats:p>","DOI":"10.1007\/s10586-021-03325-0","type":"journal-article","created":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T06:45:13Z","timestamp":1623221113000},"page":"3085-3099","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Migration of CMSWEB cluster at CERN to Kubernetes: a comprehensive study"],"prefix":"10.1007","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2979-564X","authenticated-orcid":false,"given":"Muhammad","family":"Imran","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Valentin","family":"Kuznetsov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katarzyna Maria","family":"Dziedziniewicz-Wojcik","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreas","family":"Pfeiffer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Panos","family":"Paparrigopoulos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Spyridon","family":"Trigazis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tommaso","family":"Tedeschi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diego","family":"Ciangottini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,6,9]]},"reference":[{"key":"3325_CR1","first-page":"S08004","volume":"3","author":"CMS Collaboration","year":"2008","unstructured":"Collaboration, C.M.S.: The CMS experiment at the CERN LHC. JINST 3, S08004 (2008)","journal-title":"JINST"},{"key":"3325_CR2","unstructured":"CMS experiment at CERN. https:\/\/home.cern\/science\/experiments\/cms. Accessed 13 April 2020"},{"issue":"3","key":"3325_CR3","first-page":"228","volume":"17","author":"BB Rad","year":"2017","unstructured":"Rad, B.B., Bhatti, H.J., Ahmadi, M.: An introduction to Docker and analysis of its performance. Int. J. Comput. Sci. Netw. Secur. 17(3), 228 (2017)","journal-title":"Int. J. Comput. Sci. Netw. Secur."},{"key":"3325_CR4","volume-title":"Kubernetes in Action","author":"M Luksa","year":"2018","unstructured":"Luksa, M.: Kubernetes in Action. Manning Publications, Shelter Island (2018)"},{"key":"3325_CR5","doi-asserted-by":"publisher","unstructured":"Imran, M., Kuznetsov, V., Marcella, L., Maria Dziedziniewicz-Wojcik, K., Pfeiffer, A., Paparrigopoulos, P.: Migration of CMSWEB cluster at CERN to Kubernetes. In: PoS ICHEP2020, p 911 (2021). https:\/\/doi.org\/10.22323\/1.390.0911","DOI":"10.22323\/1.390.0911"},{"key":"3325_CR6","volume-title":"CouchDB: The Definitive Guide: Time to Relax","author":"JC Anderson","year":"2010","unstructured":"Anderson, J.C., Lehnardt, J., Slater, N.: CouchDB: The Definitive Guide: Time to Relax. O\u2019Reilly Media, Inc., Newton (2010)"},{"key":"3325_CR7","doi-asserted-by":"publisher","first-page":"062038","DOI":"10.1088\/1742-6596\/664\/6\/062038","volume":"664","author":"M Mascheroni","year":"2015","unstructured":"Mascheroni, M., Balcas, J., Belforte, S., Bockelman, B., Hern\u00e1ndez, J., Ciangottini, D., Konstantinov, P., Silva, J., Ali, M., Melo, A.: CMS distributed data analysis with CRAB3. J. Phys. Conf. Ser. 664, 062038 (2015)","journal-title":"J. Phys. Conf. Ser."},{"key":"3325_CR8","doi-asserted-by":"publisher","first-page":"032026","DOI":"10.1088\/1742-6596\/396\/3\/032026","volume":"396","author":"M Cinquilli","year":"2012","unstructured":"Cinquilli, M., Spiga, D., Grandi, C., Hernandez, J.M., Konstantinov, P., Mascheroni, M., Riahi, H., Vaandering, E.: CRAB3: establishing a new generation of services for distributed analysis at CMS. J. Phys. Conf. Ser. 396, 032026 (2012)","journal-title":"J. Phys. Conf. Ser."},{"key":"3325_CR9","doi-asserted-by":"publisher","first-page":"042052","DOI":"10.1088\/1742-6596\/513\/4\/042052","volume":"513","author":"M Giffels","year":"2014","unstructured":"Giffels, M., Guo, Y., Kuznetsov, V., Magini, N., Wildish, T.: The CMS data management system. J. Phys. Conf. Ser. 513, 042052 (2014)","journal-title":"J. Phys. Conf. Ser."},{"key":"3325_CR10","doi-asserted-by":"publisher","first-page":"072001","DOI":"10.1088\/1742-6596\/119\/7\/072001","volume":"119","author":"A Afaq","year":"2008","unstructured":"Afaq, A., Dolgert, A., Guo, Y., Jones, C., Kosyakov, S., Kuznetsov, V., Lueking, L., Riley, D., Sekhri, V.: The CMS dataset bookkeeping service. J. Phys. Conf. Ser. 119, 072001 (2008)","journal-title":"J. Phys. Conf. Ser."},{"key":"3325_CR11","doi-asserted-by":"publisher","first-page":"032024","DOI":"10.1088\/1742-6596\/513\/3\/032024","volume":"513","author":"F De Guio","year":"2014","unstructured":"De Guio, F.: The CMS data quality monitoring software: experience and future prospects. J. Phys. Conf. Ser. 513, 032024 (2014)","journal-title":"J. Phys. Conf. Ser."},{"key":"3325_CR12","volume-title":"Apache: The Definitive Guide","author":"B Laurie","year":"2003","unstructured":"Laurie, B., Laurie, P.: Apache: The Definitive Guide. O\u2019Reilly Media, Inc., Newton (2003)"},{"key":"3325_CR13","unstructured":"I.\u00a0MongoDB, Mongodb, URL https:\/\/www. mongodb. com\/. Cited on (2014) 9 (2014)"},{"key":"3325_CR14","unstructured":"Rehn, J., Barrass, T., Bonacorsi, D., Hernandez, J., Semeniouk, I., Tuura, L., Wu, Y.: Computing in High Energy and Nuclear Physics (CHEP), vol. 2006 (Citeseer, 2006)"},{"key":"3325_CR15","doi-asserted-by":"publisher","first-page":"072018","DOI":"10.1088\/1742-6596\/664\/7\/072018","volume":"664","author":"G Boudoul","year":"2015","unstructured":"Boudoul, G., Franzoni, G., Norkus, A., Pol, A., Srimanobhas, P., Vlimant, J.: Monte Carlo production management at CMS. J. Phys. Conf. Ser. 664, 072018 (2015)","journal-title":"J. Phys. Conf. Ser."},{"key":"3325_CR16","doi-asserted-by":"publisher","first-page":"032092","DOI":"10.1088\/1742-6596\/513\/3\/032092","volume":"513","author":"V Spinoso","year":"2014","unstructured":"Spinoso, V., Missiato, M.: A flexible monitoring infrastructure for the simulation requests. J. Phys. Conf. Ser. 513, 032092 (2014)","journal-title":"J. Phys. Conf. Ser."},{"key":"3325_CR17","unstructured":"CMSWEB environment at a glance. https:\/\/cms-http-group.web.cern.ch\/cms-http-group\/activity.html. Accessed 4 April 2020"},{"key":"3325_CR18","unstructured":"CMS repository for Docker. https:\/\/github.com\/dmwm\/CMSKubernetes\/tree\/master\/docker. Accessed 4 April 2020"},{"key":"3325_CR19","unstructured":"CMSWEB Kubernetes repository. https:\/\/github.com\/dmwm\/CMSKubernetes\/tree\/master\/kubernetes. Accessed 4 April 2020"},{"key":"3325_CR20","unstructured":"CMSDIST repository. https:\/\/github.com\/cms-sw\/cmsdist. Accessed 4 April 2020"},{"key":"3325_CR21","unstructured":"Docker Hub repository for CMSSW. https:\/\/hub.docker.com\/u\/cmssw. Accessed 5 May 2020"},{"key":"3325_CR22","doi-asserted-by":"publisher","first-page":"08031","DOI":"10.1051\/epjconf\/201921408031","volume":"214","author":"A Aimar","year":"2019","unstructured":"Aimar, A., Corman, A.A., Andrade, P., Fernandez, J.D., Bear, B.G., Karavakis, E., Kulikowski, D.M., Magnoni, L.: MONIT: monitoring the CERN data centres and the WLCG infrastructure. EPJ Web Conf. 214, 08031 (2019)","journal-title":"EPJ Web Conf."},{"key":"3325_CR23","unstructured":"Monitoring of CMSWEB frontend nodes. https:\/\/bit.ly\/2BIKvJf. Accessed 6 Sep 2020"},{"key":"3325_CR24","unstructured":"Cinder storage in OpenStack. https:\/\/wiki.openstack.org\/wiki\/Cinder. Accessed 14 April 2020"},{"key":"3325_CR25","doi-asserted-by":"publisher","first-page":"04038","DOI":"10.1051\/epjconf\/202024504038","volume":"245","author":"L Mascetti","year":"2020","unstructured":"Mascetti, L., Rios, M.A., Bocchi, E., Vicente, J.C., Cheong, B.C.K., Castro, D., Collet, J., Contescu, C., Labrador, H.G., Iven, J.: CERN disk storage services: report from last data taking, evolution and future outlook towards Exabyte-scale storage. EPJ Web Conf. 245, 04038 (2020)","journal-title":"EPJ Web Conf."},{"key":"3325_CR26","volume-title":"Monitoring with Prometheus","author":"J Turnbull","year":"2018","unstructured":"Turnbull, J., et al.: Monitoring with Prometheus. Turnbull Press, Brooklyn (2018)"},{"key":"3325_CR27","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s41781-020-00051-x","volume":"5","author":"C Ariza-Porras","year":"2021","unstructured":"Ariza-Porras, C., Kuznetsov, V., Legger, F.: The CMS monitoring infrastructure and applications. Comput. Softw. Big Sci. 5, 5 (2021). https:\/\/doi.org\/10.1007\/s41781-020-00051-x","journal-title":"Comput. Softw. Big Sci."},{"key":"3325_CR28","unstructured":"Horizontal pod autoscaler. https:\/\/kubernetes.io\/docs\/tasks\/run-application\/horizontal-pod-autoscale. Accessed 17 Sep 2020"},{"key":"3325_CR29","unstructured":"Horizontal pod autoscaler walkthrough. https:\/\/kubernetes.io\/docs\/tasks\/run-application\/horizontal-pod-autoscale-walkthrough. Accessed 17 Sep 2020"},{"key":"3325_CR30","unstructured":"Prometheus adapter. https:\/\/github.com\/DirectXMan12\/k8s-prometheus-adapter. Accessed 17 Sep 2020"},{"key":"3325_CR31","unstructured":"General rules for writing Prometheus exporters. https:\/\/prometheus.io\/docs\/instrumenting\/writing_exporters\/. Accessed 7 Dec 2020"},{"key":"3325_CR32","unstructured":"Prometheus exporter exposition formats. https:\/\/prometheus.io\/docs\/instrumenting\/exposition_formats\/. Accessed 7 Dec 2020"},{"key":"3325_CR33","unstructured":"Prometheus exporters. https:\/\/prometheus.io\/docs\/instrumenting\/exporters. Accessed 17 Sep 2020"},{"key":"3325_CR34","unstructured":"Prometheus exporters assigned ports. https:\/\/github.com\/prometheus\/prometheus\/wiki\/Default-port-allocations. Accessed 17 Sep 2020"},{"key":"3325_CR35","unstructured":"HPA via Prometheus walkthrough. https:\/\/github.com\/Cloud-PG\/prometheus-hpa. Accessed 17 Sep 2020"},{"key":"3325_CR36","unstructured":"ReCaS data center. https:\/\/www.recas-bari.it\/index.php\/en\/. Accessed 5 Dec 2020"},{"key":"3325_CR37","doi-asserted-by":"crossref","unstructured":"Kinsman, T., Wessel, M., Gerosa, M.A., Treude, C.: How Do Software Developers Use GitHub Actions to Automate Their Workflows? arXiv preprint (2021). arXiv:2103.12224","DOI":"10.1109\/MSR52588.2021.00054"},{"key":"3325_CR38","unstructured":"Imagebot. https:\/\/github.com\/vkuznet\/imagebot. Accessed 7 Dec 2020"},{"key":"3325_CR39","unstructured":"Hey tool. https:\/\/github.com\/rakyll\/hey. Accessed 4 April 2020"},{"key":"3325_CR40","unstructured":"To run or not to run a database on Kubernetes: what to consider. https:\/\/bit.ly\/37eB9k9. Accessed 27 April 2021"}],"container-title":["Cluster Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03325-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10586-021-03325-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10586-021-03325-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,30]],"date-time":"2021-10-30T18:19:27Z","timestamp":1635617967000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10586-021-03325-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,9]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["3325"],"URL":"https:\/\/doi.org\/10.1007\/s10586-021-03325-0","relation":{},"ISSN":["1386-7857","1573-7543"],"issn-type":[{"value":"1386-7857","type":"print"},{"value":"1573-7543","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,9]]},"assertion":[{"value":"22 January 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 April 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 May 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 June 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}