{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T07:34:41Z","timestamp":1779176081435,"version":"3.51.4"},"reference-count":26,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2026,1,31]]},"abstract":"<jats:p>MLOps has emerged as a key solution to address many socio-technical challenges of bringing ML models to production, such as integrating ML models with non-ML software, continuous monitoring, maintenance, and retraining of deployed models. Despite the utility of MLOps, an integrated body of knowledge regarding MLOps remains elusive because of its extensive scope due to the diversity of ML productionalization challenges it addresses. Whilst the existing literature reviews provide valuable snapshots of specific practices, tools, and research prototypes related to MLOps at various times, they focus on particular facets of MLOps, thus fail to offer a comprehensive and invariant framework that can weave these perspectives into a unified understanding of MLOps. This article presents a Multivocal Literature Review that systematically analyzes a corpus of 150 peer-reviewed and 48 grey literature to synthesize a unified conceptualization of MLOps and develop a snapshot of its best practices, adoption challenges, and solutions.<\/jats:p>","DOI":"10.1145\/3747346","type":"journal-article","created":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T06:08:41Z","timestamp":1751954921000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["A Multivocal Review of MLOps Practices, Challenges and Open Issues"],"prefix":"10.1145","volume":"58","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6824-2765","authenticated-orcid":false,"given":"Beyza","family":"Eken","sequence":"first","affiliation":[{"name":"Sakarya University","place":["Sakarya, Turkey"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5342-9551","authenticated-orcid":false,"given":"Samodha","family":"Pallewatta","sequence":"additional","affiliation":[{"name":"The University of Adelaide","place":["Adelaide, Australia"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9538-7476","authenticated-orcid":false,"given":"Nguyen","family":"Tran","sequence":"additional","affiliation":[{"name":"The University of Adelaide","place":["Adelaide, Australia"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1859-7872","authenticated-orcid":false,"given":"Ayse","family":"Tosun","sequence":"additional","affiliation":[{"name":"Istanbul Technical University","place":["Istanbul, Turkey"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9696-3626","authenticated-orcid":false,"given":"Muhammad Ali","family":"Babar","sequence":"additional","affiliation":[{"name":"The University of Adelaide","place":["Adelaide, Australia"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,9,8]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Kent Beck Mike Beedle Arie Van Bennekum Alistair Cockburn Ward Cunningham Martin Fowler James Grenning Jim Highsmith Andrew Hunt Ron Jeffries et\u00a0al. 2001. The agile manifesto. Software Development 9 8 (2001) 28\u201335. http:\/\/agilemanifesto.org"},{"key":"e_1_3_2_3_2","unstructured":"Joseph Bradley Rafi Kurlansik Matt Thomson and Niall Turbitt. 2023. The Big Book of MLOps. Databricks (E-book)."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3171202"},{"key":"e_1_3_2_6_2","unstructured":"IBM Data and AI Team. 2023. MLOps and the evolution of data science. Retrieved November 2023 from https:\/\/www.ibm.com\/think\/topics\/mlops"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106771"},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Beyza Eken Samodha Pallewatta Nguyen Khoi Tran Ayse Tosun and Muhammad Ali Babar. 2025. Online appendix of A Multivocal Review of MLOps Practices Challenges and Open Issues. https:\/\/github.com\/beken\/MLOpsMLR","DOI":"10.1145\/3747346"},{"key":"e_1_3_2_9_2","unstructured":"Leonhard Faubel and Klaus Schmid. 2023. Review protocol: A systematic literature review of MLOps. (2023)."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.09.006"},{"key":"e_1_3_2_11_2","unstructured":"IBM. 2022. IBM Global AI Adoption Index 2022. Retrieved February 2024 from https:\/\/www.ibm.com\/downloads\/documents\/us-en\/107a02e94a48f5c1"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA53835.2021.00050"},{"key":"e_1_3_2_13_2","unstructured":"Barbara Kitchenham Stuart Charters et\u00a0al. 2007. Guidelines for performing systematic literature reviews in software engineering: Technical report EBSE 2007-001. (2007)."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3526073.3527584"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","unstructured":"Dominik Kreuzberger Niklas K\u00fchl and Sebastian Hirschl. 2023. Machine learning operations (mlops): Overview definition and architecture. IEEE Access 11 (2023) 31866\u201331879.","DOI":"10.1109\/ACCESS.2023.3262138"},{"key":"e_1_3_2_16_2","first-page":"308","article-title":"MLOps: Practices, maturity models, roles, tools, and challenges-A systematic literature review. In","author":"Lima Anderson","year":"2022","unstructured":"Anderson Lima, Luciano Monteiro, and Ana Paula Furtado. 2022. MLOps: Practices, maturity models, roles, tools, and challenges-A systematic literature review. In Proceedings of the International Conference on Enterprise Information Systems (ICEIS). 1 (2022), 308\u2013320.","journal-title":"Proceedings of the International Conference on Enterprise Information Systems (ICEIS)"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-64148-1_12"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECET55527.2022.9872968"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3555041.3589717"},{"key":"e_1_3_2_20_2","unstructured":"Khalid Salama Jarek Kazmierczak and Donna Schut. 2021. Practitioners guide to MLOps: A framework for continuous delivery and automation of machine learning. Google Could White Paper (2021)."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3595360.3595859"},{"key":"e_1_3_2_22_2","unstructured":"David Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In Proceedings of the 29th International Conference on Neural Information Processing Systems 2 (2015) 2503\u20132511."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111615"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA64295.2024.00054"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599776"},{"key":"e_1_3_2_26_2","article-title":"Guide to the software engineering body of knowledge (SWEBOK guide), version 4.0","author":"Washizaki Hironori","year":"2024","unstructured":"Hironori Washizaki. 2024. Guide to the software engineering body of knowledge (SWEBOK guide), version 4.0. IEEE Computer Society, Waseda University, Japan. http:\/\/www. swebok. org IEEE Computer Society.","journal-title":"IEEE Computer Society, Waseda University, Japan. http:\/\/www. swebok. org IEEE Computer Society"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3747346","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T03:27:51Z","timestamp":1757474871000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3747346"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,8]]},"references-count":26,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,1,31]]}},"alternative-id":["10.1145\/3747346"],"URL":"https:\/\/doi.org\/10.1145\/3747346","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,8]]},"assertion":[{"value":"2024-06-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-13","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}