{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T21:52:49Z","timestamp":1791582769971,"version":"4.3.4"},"reference-count":111,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100006374","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DGE2243822"],"award-info":[{"award-number":["DGE2243822"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,4,17]]},"abstract":"<jats:p>Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationalization of machine learning, or MLOps, requires MLEs to have proficiency in data science and engineering. When considered holistically, the job seems staggering---how do MLEs do MLOps, and what are their unaddressed challenges? To address these questions, we conducted semi-structured ethnographic interviews with 18 MLEs working on various applications, including chatbots, autonomous vehicles, and finance. We find that MLEs engage in a workflow of (i) data preparation, (ii) experimentation, (iii) evaluation throughout a multi-staged deployment, and (iv) continual monitoring and response. Throughout this workflow, MLEs collaborate extensively with data scientists, product stakeholders, and one another, supplementing routine verbal exchanges with communication tools ranging from Slack to organization-wide ticketing and reporting systems. We introduce the 3Vs of MLOps: velocity, visibility, and versioning --- three virtues of successful ML deployments that MLEs learn to balance and grow as they mature. Finally, we discuss design implications and opportunities for future work.<\/jats:p>","DOI":"10.1145\/3653697","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T10:05:31Z","timestamp":1714385131000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["\"We Have No Idea How Models will Behave in Production until Production\": How Engineers Operationalize Machine Learning"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0919-9672","authenticated-orcid":false,"given":"Shreya","family":"Shankar","sequence":"first","affiliation":[{"name":"University of California, Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2876-8498","authenticated-orcid":false,"given":"Rolando","family":"Garcia","sequence":"additional","affiliation":[{"name":"University of California, Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0802-4069","authenticated-orcid":false,"given":"Joseph M.","family":"Hellerstein","sequence":"additional","affiliation":[{"name":"University of California, Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4538-4752","authenticated-orcid":false,"given":"Aditya G.","family":"Parameswaran","sequence":"additional","affiliation":[{"name":"University of California, Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,26]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Ease.ML: A Lifecycle Management System for MLDev and MLOps. In Conference on Innovative Data Systems Research (CIDR","author":"Aguilar Leonel","year":"2021","unstructured":"Leonel Aguilar, David Dao, Shaoduo Gan, Nezihe Merve Gurel, Nora Hollenstein, Jiawei Jiang, Bojan Karlas, Thomas Lemmin, Tian Li, Yang Li, Susie Rao, Johannes Rausch, Cedric Renggli, Luka Rimanic, Maurice Weber, Shuai Zhang, Zhikuan Zhao, Kevin Schawinski, Wentao Wu, and Ce Zhang. 2021. Ease.ML: A Lifecycle Management System for MLDev and MLOps. In Conference on Innovative Data Systems Research (CIDR 2021). https:\/\/www.microsoft.com\/en-us\/research\/publication\/ease-ml-a-lifecycle-management-system-for-mldev-and-mlops\/"},{"key":"e_1_2_1_2_1","volume-title":"What is mlops? In Beginning MLOps with MLFlow","author":"Alla Sridhar","unstructured":"Sridhar Alla and Suman Kalyan Adari. 2021. What is mlops? In Beginning MLOps with MLFlow. Springer, 79--124."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_2_1_4_1","unstructured":"Anonymous. 2021. ML Reproducibility Systems: Status and Research Agenda. https:\/\/openreview.net\/forum?id=v-6XBItNld2"},{"key":"e_1_2_1_5_1","volume-title":"2020 USENIX Conference on Operational Machine Learning (OpML 20)","author":"Banerjee Amitabha","year":"2020","unstructured":"Amitabha Banerjee, Chien-Chia Chen, Chien-Chun Hung, Xiaobo Huang, Yifan Wang, and Razvan Chevesaran. 2020. Challenges and Experiences with $$MLOps$$ for Performance Diagnostics in $$Hybrid-Cloud$$ Enterprise Software Deployments. In 2020 USENIX Conference on Operational Machine Learning (OpML 20)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1177\/1525822X221107053"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3320246"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of SysML. https:\/\/mlsys.org\/Conferences\/2019\/doc\/2019\/167","author":"Breck Eric","year":"2019","unstructured":"Eric Breck, Marty Zinkevich, Neoklis Polyzotis, Steven Whang, and Sudip Roy. 2019. Data Validation for Machine Learning. In Proceedings of SysML. https:\/\/mlsys.org\/Conferences\/2019\/doc\/2019\/167.pdf"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142473.1142574"},{"key":"e_1_2_1_10_1","volume-title":"4th CRISP-DM SIG Workshop in Brussels in March","volume":"1999","author":"Chapman Pete","year":"1999","unstructured":"Pete Chapman, Julian Clinton, Randy Kerber, Thomas Khabaza, Thomas Reinartz, Colin Shearer, and R\u00fcdiger Wirth. 1999. The CRISP-DM user guide. In 4th CRISP-DM SIG Workshop in Brussels in March, Vol. 1999. sn."},{"key":"e_1_2_1_11_1","volume-title":"7th $$USENIX$$ Workshop on the Theory and Practice of Provenance (TaPP 15).","author":"Chavan Amit","unstructured":"Amit Chavan, Silu Huang, Amol Deshpande, Aaron Elmore, Samuel Madden, and Aditya Parameswaran. 2015. Towards a unified query language for provenance and versioning. In 7th $$USENIX$$ Workshop on the Theory and Practice of Provenance (TaPP 15)."},{"key":"e_1_2_1_12_1","volume-title":"Reducing confusion about grounded theory and qualitative content analysis: Similarities and differences. Qualitative report","author":"Cho Ji Young","year":"2014","unstructured":"Ji Young Cho and Eun-Hee Lee. 2014. Reducing confusion about grounded theory and qualitative content analysis: Similarities and differences. Qualitative report, Vol. 19, 32 (2014)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0065-2458(03)62001-2"},{"key":"e_1_2_1_14_1","volume-title":"14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17)","author":"Crankshaw Daniel","year":"2017","unstructured":"Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J Franklin, Joseph E Gonzalez, and Ion Stoica. 2017. Clipper: A $$Low-Latency$$ Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 613--627."},{"key":"e_1_2_1_15_1","volume-title":"Qualitative inquiry and research design: Choosing among five approaches","author":"Creswell John W","unstructured":"John W Creswell and Cheryl N Poth. 2016. Qualitative inquiry and research design: Choosing among five approaches. Sage publications."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376772"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2016.68"},{"key":"e_1_2_1_18_1","unstructured":"Mihail Eric. [n. d.]. MLOps is a mess but that's to be expected. https:\/\/www.mihaileric.com\/posts\/mlops-is-a-mess\/"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01437-7_16"},{"key":"e_1_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Rolando Garcia Eric Liu Vikram Sreekanti Bobby Yan Anusha Dandamudi Joseph E. Gonzalez Joseph M Hellerstein and Koushik Sen. 2021. Hindsight Logging for Model Training. In VLDB.","DOI":"10.14778\/3436905.3436925"},{"key":"e_1_2_1_21_1","volume-title":"Context: The missing piece in the machine learning lifecycle. In CMI.","author":"Garcia Rolando","year":"2018","unstructured":"Rolando Garcia, Vikram Sreekanti, Neeraja Yadwadkar, Daniel Crankshaw, Joseph E Gonzalez, and Joseph M Hellerstein. 2018. Context: The missing piece in the machine learning lifecycle. In CMI."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/AIKE52691.2021.00010"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_2_1_24_1","unstructured":"Samadrita Ghosh. 2021. Mlops challenges and how to face them. https:\/\/neptune.ai\/blog\/mlops-challenges-and-how-to-face-them"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/WAIN52551.2021.00019"},{"key":"e_1_2_1_26_1","volume-title":"How many interviews are enough? An experiment with data saturation and variability. Field methods","author":"Guest Greg","year":"2006","unstructured":"Greg Guest, Arwen Bunce, and Laura Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field methods, Vol. 18, 1 (2006), 59--82."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047205"},{"key":"e_1_2_1_28_1","volume-title":"Ground: A Data Context Service.. In CIDR.","author":"Hellerstein Joseph M","year":"2017","unstructured":"Joseph M Hellerstein, Vikram Sreekanti, Joseph E Gonzalez, James Dalton, Akon Dey, Sreyashi Nag, Krishna Ramachandran, Sudhanshu Arora, Arka Bhattacharyya, Shirshanka Das, et al. 2017. Ground: A Data Context Service.. In CIDR."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376177"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134688"},{"key":"e_1_2_1_33_1","volume-title":"Taverna: a tool for building and running workflows of services. Nucleic acids research","author":"Hull Duncan","year":"2006","unstructured":"Duncan Hull, Katy Wolstencroft, Robert Stevens, Carole Goble, Mathew R Pocock, Peter Li, and Tom Oinn. 2006. Taverna: a tool for building and running workflows of services. Nucleic acids research , Vol. 34, suppl_2 (2006), W729--W732."},{"key":"e_1_2_1_34_1","unstructured":"Chip Huyen. 2020. Machine learning tools landscape V2 (84 new tools). https:\/\/huyenchip.com\/2020\/12\/30\/mlops-v2.html"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA53835.2021.00050"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979444"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.219"},{"key":"e_1_2_1_38_1","unstructured":"Daniel Kang Deepti Raghavan Peter Bailis and Matei Zaharia. [n. d.]. Model assertions for debugging machine learning."},{"key":"e_1_2_1_39_1","first-page":"3025453","article-title":"Variolite: Supporting Exploratory Programming by Data Scientists","volume":"10","author":"Kery Mary Beth","year":"2017","unstructured":"Mary Beth Kery, Amber Horvath, and Brad A Myers. 2017. Variolite: Supporting Exploratory Programming by Data Scientists.. In CHI, Vol. 10. 3025453--3025626.","journal-title":"CHI"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884783"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2754374"},{"key":"e_1_2_1_42_1","volume-title":"Clive Cox, Giovanni Vacanti, and Alexandru Coca.","author":"Klaise Janis","year":"2020","unstructured":"Janis Klaise, Arnaud Van Looveren, Clive Cox, Giovanni Vacanti, and Alexandru Coca. 2020. Monitoring and explainability of models in production. ArXiv , Vol. abs\/2007.06299 (2020)."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bts480"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","unstructured":"Dominik Kreuzberger Niklas K\u00fchl and Sebastian Hirschl. 2022. Machine Learning Operations (MLOps): Overview Definition and Architecture. https:\/\/doi.org\/10.48550\/ARXIV.2205.02302","DOI":"10.48550\/ARXIV.2205.02302"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077257.3077271"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476052"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300493"},{"key":"e_1_2_1_48_1","volume-title":"Willem Jan Van Den Heuvel, and Damian Andrew Tamburri.","author":"Kumara Indika","year":"2022","unstructured":"Indika Kumara, Rowan Arts, Dario Di Nucci, Willem Jan Van Den Heuvel, and Damian Andrew Tamburri. 2022. Requirements and Reference Architecture for MLOps: Insights from Industry. (2022)."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2019.2962338"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","unstructured":"Angela Lee Doris Xin Doris Lee and Aditya Parameswaran. 2020. Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development. https:\/\/doi.org\/10.48550\/ARXIV.2005.01520","DOI":"10.48550\/ARXIV.2005.01520"},{"key":"e_1_2_1_51_1","unstructured":"Cheng Han Lee. 2020. 3 data careers decoded and what it means for you. https:\/\/www.udacity.com\/blog\/2014\/12\/data-analyst-vs-data-scientist-vs-data-engineer.html"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359981"},{"key":"e_1_2_1_53_1","volume-title":"Maturity Models, Roles, Tools, and Challenges-A Systematic Literature Review. ICEIS (1)","author":"Lima Anderson","year":"2022","unstructured":"Anderson Lima, Luciano Monteiro, and Ana Paula Furtado. 2022. MLOps: Practices, Maturity Models, Roles, Tools, and Challenges-A Systematic Literature Review. ICEIS (1) (2022), 308--320."},{"key":"e_1_2_1_54_1","unstructured":"Zhiqiu Lin Jia Shi Deepak Pathak and Deva Ramanan. 2021. The CLEAR Benchmark: Continual LEArning on Real-World Imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https:\/\/openreview.net\/forum?id=43mYF598ZDB"},{"key":"e_1_2_1_55_1","volume-title":"What is DevOps? \" O'Reilly Media","author":"Loukides Mike","unstructured":"Mike Loukides. 2012. What is DevOps? \" O'Reilly Media, Inc.\"."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.06.010"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376445"},{"key":"e_1_2_1_58_1","volume-title":"2021 IEEE\/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN). IEEE, 109--112","author":"Sasu","year":"2021","unstructured":"Sasu M\"akinen, Henrik Skogstr\u00f6m, Eero Laaksonen, and Tommi Mikkonen. 2021. Who needs MLOps: What data scientists seek to accomplish and how can MLOps help?. In 2021 IEEE\/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN). IEEE, 109--112."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3522664.3528611"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_2_1_61_1","unstructured":"MLReef. 2021. Global mlops and ML Tools Landscape: Mlreef. https:\/\/about.mlreef.com\/blog\/global-mlops-and-ml-tools-landscape\/"},{"key":"e_1_2_1_62_1","volume-title":"TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. In KDD","author":"Naresh Akshay","year":"2017","unstructured":"Akshay Naresh Modi et al. 2017. TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. In KDD 2017."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2011.06.019"},{"key":"e_1_2_1_64_1","volume-title":"Jukka K Nurminen, and Tommi Mikkonen.","author":"Muiruri Dennis","year":"2022","unstructured":"Dennis Muiruri, Lucy Ellen Lwakatare, Jukka K Nurminen, and Tommi Mikkonen. 2022. Practices and Infrastructures for ML Systems--An Interview Study in Finnish Organizations. (2022)."},{"key":"e_1_2_1_65_1","volume-title":"Ways of Knowing in HCI","author":"Muller Michael","unstructured":"Michael Muller. 2014. Curiosity, creativity, and surprise as analytic tools: Grounded theory method. In Ways of Knowing in HCI. Springer, 25--48."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300356"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403205"},{"key":"e_1_2_1_68_1","unstructured":"Andrew Ng Eddy Shyu Aarti Bagul and Geoff Ladwig. [n. d.]. Evaluating a model - advice for applying machine learning. https:\/\/www.coursera.org\/lecture\/advanced-learning-algorithms\/evaluating-a-model-26yGi"},{"key":"e_1_2_1_69_1","unstructured":"Yaniv Ovadia Emily Fertig J. Ren Zachary Nado D. Sculley Sebastian Nowozin Joshua V. Dillon Balaji Lakshminarayanan and Jasper Snoek. 2019. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In NeurIPS."},{"key":"e_1_2_1_70_1","volume-title":"Benchmarks & Analysis. Machine Learning Journal","author":"Paduraru Cosmin","year":"2021","unstructured":"Cosmin Paduraru, Daniel J. Mankowitz, Gabriel Dulac-Arnold, Jerry Li, Nir Levine, Sven Gowal, and Todd Hester. 2021. Challenges of Real-World Reinforcement Learning:Definitions, Benchmarks & Analysis. Machine Learning Journal (2021)."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","unstructured":"Andrei Paleyes Raoul-Gabriel Urma and Neil D. Lawrence. 2022. Challenges in Deploying Machine Learning: A Survey of Case Studies. ACM Comput. Surv. (apr 2022). https:\/\/doi.org\/10.1145\/3533378 Just Accepted.","DOI":"10.1145\/3533378"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998331"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274405"},{"key":"e_1_2_1_74_1","volume-title":"International Conference on Human Factors in Computing Systems.","author":"Patel Kayur","unstructured":"Kayur Patel, James Fogarty, James A. Landay, and Beverly L. Harrison. 2008. Investigating statistical machine learning as a tool for software development. In International Conference on Human Factors in Computing Systems."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3054782"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299887.3299891"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.2196\/29301"},{"key":"e_1_2_1_78_1","volume-title":"Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In NeurIPS.","author":"Rabanser Stephan","year":"2019","unstructured":"Stephan Rabanser, Stephan G\u00fcnnemann, and Zachary Chase Lipton. 2019. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In NeurIPS."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA56994.2022.00021"},{"key":"e_1_2_1_81_1","volume-title":"Bojan Karlavs, Wentao Wu, and Ce Zhang.","author":"Renggli Cedric","year":"2021","unstructured":"Cedric Renggli, Luka Rimanic, Nezihe Merve G\u00fcrel, Bojan Karlavs, Wentao Wu, and Ce Zhang. 2021. A data quality-driven view of mlops. arXiv preprint arXiv:2102.07750 (2021)."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2019.2946162"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.3390\/app11198861"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415556"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445518"},{"key":"e_1_2_1_86_1","doi-asserted-by":"crossref","unstructured":"Sebastian Schelter et al. 2018. Automating Large-Scale Data Quality Verification. In PVLDB'19.","DOI":"10.14778\/3229863.3229867"},{"key":"e_1_2_1_87_1","unstructured":"D. Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Fran\u00e7ois Crespo and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In NIPS."},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3382494.3410681"},{"key":"e_1_2_1_89_1","volume-title":"Parameswaran","author":"Shankar Shreya","year":"2022","unstructured":"Shreya Shankar, Bernease Herman, and Aditya G. Parameswaran. 2022. Rethinking Streaming Machine Learning Evaluation. ArXiv , Vol. abs\/2205.11473 (2022)."},{"key":"e_1_2_1_90_1","volume-title":"Proc. VLDB Endow. (sep","author":"Shankar Shreya","year":"2023","unstructured":"Shreya Shankar, Stephen Macke, Sarah Chasins, Andrew Head, and Aditya Parameswaran. 2023. Bolt-on, Compact, and Rapid Program Slicing for Notebooks. Proc. VLDB Endow. (sep 2023)."},{"key":"e_1_2_1_91_1","volume-title":"Parameswaran","author":"Shankar Shreya","year":"2022","unstructured":"Shreya Shankar and Aditya G. Parameswaran. 2022. Towards Observability for Production Machine Learning Pipelines. ArXiv , Vol. abs\/2108.13557 (2022)."},{"key":"e_1_2_1_92_1","volume-title":"The ethnographic interview","author":"Spradley James P","unstructured":"James P Spradley. 2016. The ethnographic interview. Waveland Press."},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403379"},{"key":"e_1_2_1_94_1","volume-title":"Why AI investments fail to deliver. https:\/\/www.infoworld.com\/article\/3639028\/why-ai-investments-fail-to-deliver.html [Online","author":"Nunez Steve","year":"2022","unstructured":"Steve Nunez. 2022. Why AI investments fail to deliver. https:\/\/www.infoworld.com\/article\/3639028\/why-ai-investments-fail-to-deliver.html [Online; accessed 15-September-2022]."},{"key":"e_1_2_1_95_1","unstructured":"Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology: An overview. (1994)."},{"key":"e_1_2_1_96_1","volume-title":"Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters, and Klaus-Robert M\u00fcller.","author":"Studer Stefan","year":"2021","unstructured":"Stefan Studer, Thanh Binh Bui, Christian Drescher, Alexander Hanuschkin, Ludwig Winkler, Steven Peters, and Klaus-Robert M\u00fcller. 2021. Towards CRISP-ML (Q): a machine learning process model with quality assurance methodology. Machine learning and knowledge extraction , Vol. 3, 2 (2021), 392--413."},{"key":"e_1_2_1_97_1","doi-asserted-by":"crossref","unstructured":"Masashi Sugiyama et al. 2007. Covariate Shift Adaptation by Importance Weighted Cross Validation. In JMLR.","DOI":"10.1007\/11861898_36"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCWC54503.2022.9720902"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1109\/SYNASC51798.2020.00015"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3181730"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939502.2939516"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-01685-1"},{"key":"e_1_2_1_104_1","volume-title":"The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=MLOps&oldid=1109828739 [Online","author":"Wikipedia Wikipedia","year":"2022","unstructured":"Wikipedia contributors. 2022. MLOps -- Wikipedia, The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=MLOps&oldid=1109828739 [Online; accessed 15-September-2022]."},{"key":"e_1_2_1_105_1","volume-title":"A Fine-Grained Analysis on Distribution Shift. ArXiv","author":"Wiles Olivia","year":"2021","unstructured":"Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dvijotham, and Ali Taylan Cemgil. 2021. A Fine-Grained Analysis on Distribution Shift. ArXiv , Vol. abs\/2110.11328 (2021)."},{"key":"e_1_2_1_106_1","volume-title":"Process, and Challenges of Exploratory Data Analysis: An Interview Study. ArXiv","author":"Wongsuphasawat Kanit","year":"2019","unstructured":"Kanit Wongsuphasawat, Yang Liu, and Jeffrey Heer. 2019. Goals, Process, and Challenges of Exploratory Data Analysis: An Interview Study. ArXiv , Vol. abs\/1911.00568 (2019)."},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457566"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445306"},{"key":"e_1_2_1_109_1","first-page":"39","article-title":"Accelerating the Machine Learning Lifecycle with MLflow","volume":"41","author":"M. Zaharia","year":"2018","unstructured":"M. Zaharia et al. 2018. Accelerating the Machine Learning Lifecycle with MLflow. IEEE Data Eng. Bull. , Vol. 41 (2018), 39--45.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392826"},{"key":"e_1_2_1_111_1","volume-title":"OneLabeler: A Flexible System for Building Data Labeling Tools. In CHI Conference on Human Factors in Computing Systems. 1--22","author":"Zhang Yu","year":"2022","unstructured":"Yu Zhang, Yun Wang, Haidong Zhang, Bin Zhu, Siming Chen, and Dongmei Zhang. 2022. OneLabeler: A Flexible System for Building Data Labeling Tools. In CHI Conference on Human Factors in Computing Systems. 1--22. io"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3653697","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3653697","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T17:22:46Z","timestamp":1755883366000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3653697"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,17]]},"references-count":111,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2024,4,17]]}},"alternative-id":["10.1145\/3653697"],"URL":"https:\/\/doi.org\/10.1145\/3653697","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,17]]},"assertion":[{"value":"2024-04-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}