{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T21:30:21Z","timestamp":1778275821498,"version":"3.51.4"},"reference-count":159,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T00:00:00Z","timestamp":1670371200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100012338","name":"Alan Turing Institute","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100012338","id-type":"DOI","asserted-by":"crossref"}]},{"name":"UK Research & Innovation"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>In recent years, machine learning has transitioned from a field of academic research interest to a field capable of solving real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries, and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. By mapping found challenges to the steps of the machine learning deployment workflow, we show that practitioners face issues at each stage of the deployment process. The goal of this article is to lay out a research agenda to explore approaches addressing these challenges.<\/jats:p>","DOI":"10.1145\/3533378","type":"journal-article","created":{"date-parts":[[2022,4,30]],"date-time":"2022-04-30T11:11:21Z","timestamp":1651317081000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":428,"title":["Challenges in Deploying Machine Learning: A Survey of Case Studies"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3703-8163","authenticated-orcid":false,"given":"Andrei","family":"Paleyes","sequence":"first","affiliation":[{"name":"University of Cambridge, Cambridge, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8245-4425","authenticated-orcid":false,"given":"Raoul-Gabriel","family":"Urma","sequence":"additional","affiliation":[{"name":"Cambridge Spark, Cambridge, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9258-1030","authenticated-orcid":false,"given":"Neil D.","family":"Lawrence","sequence":"additional","affiliation":[{"name":"University of Cambridge, Cambridge, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2022,12,7]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"Global AI survey: AI proves its worth, but few scale impact","author":"Cam Arif","year":"2019","unstructured":"Arif Cam, Michael Chui, and Bryce Hall. 2019. Global AI survey: AI proves its worth, but few scale impact. McKinsey Analytics (2019). https:\/\/www.mckinsey.com\/featured-insights\/artificial-intelligence\/global-ai-survey-ai-proves-its-worth-but-few-scale-impact. Accessed from 13\/06\/2022.","journal-title":"McKinsey Analytics"},{"issue":"1","key":"e_1_3_2_3_2","first-page":"108","article-title":"Artificial intelligence for the real world","volume":"96","author":"Davenport Thomas H.","year":"2018","unstructured":"Thomas H. Davenport and Rajeev Ronanki. 2018. Artificial intelligence for the real world. Harv. Bus. Rev. 96, 1 (2018), 108\u2013116.","journal-title":"Harv. Bus. Rev."},{"key":"e_1_3_2_4_2","volume-title":"Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction","author":"Britain) Royal Society (Great","year":"2017","unstructured":"Royal Society (Great Britain). 2017. Machine Learning: The Power and Promise of Computers that Learn by Example: an Introduction. Royal Society."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-008-9050-0"},{"key":"e_1_3_2_6_2","unstructured":"Kyle Wiggers. 2019. Algorithmia: 50% of companies spend between 8 and 90 days deploying a single AI model. Retrieved from https:\/\/venturebeat.com\/2019\/12\/11\/algorithmia-50-of-companies-spend-upwards-of-three-months-deploying-a-single-ai-model\/."},{"key":"e_1_3_2_7_2","unstructured":"Lawrence E. Hecht. 2019. Add It Up: How Long Does a Machine Learning Deployment Take? Retrieved from https:\/\/thenewstack.io\/add-it-up-how-long-does-a-machine-learning-deployment-take\/."},{"key":"e_1_3_2_8_2","unstructured":"Kyle Wiggers. 2019. IDC: For 1 in 4 companies half of all AI projects fail. Retrieved from https:\/\/venturebeat.com\/2019\/07\/08\/idc-for-1-in-4-companies-half-of-all-ai-projects-fail\/."},{"key":"e_1_3_2_9_2","volume-title":"The State of Machine Learning Adoption in the Enterprise","author":"Lorica Ben","year":"2018","unstructured":"Ben Lorica and Nathan Paco. 2018. The State of Machine Learning Adoption in the Enterprise. O\u2019Reilly Media."},{"key":"e_1_3_2_10_2","unstructured":"Dotscience 2019 The state of development and operations of AI applications"},{"key":"e_1_3_2_11_2","unstructured":"Dimens. Res. 2019 Artificial intelligence and machine learning projects are obstructed by data issues"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3375624"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_14_2","volume-title":"Proceedings of the 27th European Conference on Information Systems","author":"Baier Lucas","year":"2019","unstructured":"Lucas Baier, Fabian J\u00f6hren, and Stefan Seebacher. 2019. Challenges in the deployment and operation of machine learning in practice. In Proceedings of the 27th European Conference on Information Systems."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453444"},{"issue":"4","key":"e_1_3_2_16_2","first-page":"13","article-title":"The CRISP-DM model: The new blueprint for data mining","volume":"5","author":"Shearer Colin","year":"2000","unstructured":"Colin Shearer. 2000. The CRISP-DM model: The new blueprint for data mining. J. Data Warehous. 5, 4 (2000), 13\u201322.","journal-title":"J. Data Warehous."},{"key":"e_1_3_2_17_2","article-title":"What is the team data science process?","author":"Severtson W. A. R. Roald Bradley","year":"2017","unstructured":"W. A. R. Roald Bradley Severtson, L. Franks, and G. Ericson. 2017. What is the team data science process?Microsoft Azure (2017). https:\/\/docs.microsoft.com\/en-us\/azure\/architecture\/data-science-process\/overview. Accessed from 13\/06\/2022.","journal-title":"Microsoft Azure"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2020.09.140"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3299887.3299891"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2016.2619369"},{"key":"e_1_3_2_21_2","volume-title":"Data Readiness: Lessons from an Emergency","author":"Initiative The DELVE","year":"2020","unstructured":"The DELVE Initiative. 2020. Data Readiness: Lessons from an Emergency. Technical Report. Retrieved from https:\/\/rs-delve.github.io\/reports\/2020\/11\/24\/data-readiness-lessons-from-an-emergency.html."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2481244.2481247"},{"key":"e_1_3_2_23_2","first-page":"149","article-title":"The single responsibility principle","author":"Martin Robert C.","year":"2002","unstructured":"Robert C. Martin. 2002. The single responsibility principle. Princ., Patterns, Pract. Agile Softw. Devel. (2002), 149\u2013154.","journal-title":"Princ., Patterns, Pract. Agile Softw. Devel."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994518"},{"issue":"5","key":"e_1_3_2_25_2","first-page":"5212","article-title":"A review of data cleaning algorithms for data warehouse systems","volume":"3","author":"Patil Rajashree Y.","year":"2012","unstructured":"Rajashree Y. Patil and R. V. Kulkarni. 2012. A review of data cleaning algorithms for data warehouse systems. Int. J. Comput. Sci. Inf. Technol. 3, 5 (2012), 5212\u20135214.","journal-title":"Int. J. Comput. Sci. Inf. Technol."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2019.11.177"},{"key":"e_1_3_2_27_2","article-title":"Data engineering for data analytics: A classification of the issues, and case studies","author":"Nazabal Alfredo","year":"2020","unstructured":"Alfredo Nazabal, Christopher K. I. Williams, Giovanni Colavizza, Camila Rangel Smith, and Angus Williams. 2020. Data engineering for data analytics: A classification of the issues, and case studies. arXiv preprint arXiv:2004.12929 (2020).","journal-title":"arXiv preprint arXiv:2004.12929"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939682"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMST.2018.2883147"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2010.11.006"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102062"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2010.114"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.554"},{"key":"e_1_3_2_34_2","article-title":"Denoising multi-source weak supervision for neural text classification","author":"Ren Wendi","year":"2020","unstructured":"Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. 2020. Denoising multi-source weak supervision for neural text classification. arXiv preprint arXiv:2010.04582 (2020).","journal-title":"arXiv preprint arXiv:2010.04582"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-021-05961-4"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2019.2962338"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.219"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2754374"},{"key":"e_1_3_2_39_2","unstructured":"Diego Charrez. 2019. NeurIPS 2019 Stats. Retrieved from https:\/\/medium.com\/@dcharrezt\/neurips-2019-stats-c91346d31c8f."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330658"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330656"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/MCOM.001.1900664"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022643204877"},{"issue":"1","key":"e_1_3_2_44_2","first-page":"1","article-title":"Machine learning algorithms in heavy process manufacturing","volume":"6","author":"Hansson Karl","year":"2016","unstructured":"Karl Hansson, Siril Yella, Mark Dougherty, and Hasan Fleyeh. 2016. Machine learning algorithms in heavy process manufacturing. Amer. J. Intell. Syst. 6, 1 (2016), 1\u201313.","journal-title":"Amer. J. Intell. Syst."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40854-016-0029-6"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1155\/2017\/3296874"},{"key":"e_1_3_2_47_2","article-title":"The cost of training NLP models: A concise overview","author":"Sharir Or","year":"2020","unstructured":"Or Sharir, Barak Peleg, and Yoav Shoham. 2020. The cost of training NLP models: A concise overview. arXiv preprint arXiv:2004.08900 (2020).","journal-title":"arXiv preprint arXiv:2004.08900"},{"key":"e_1_3_2_48_2","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics."},{"key":"e_1_3_2_49_2","unstructured":"Nathan Benaich and Ian Hogarth. 2020. State of AI report 2020. Retrieved from www.stateof.ai."},{"key":"e_1_3_2_50_2","doi-asserted-by":"crossref","unstructured":"Emma Strubell Ananya Ganesh and Andrew McCallum. 2020. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 34 13693\u201313696.","DOI":"10.1609\/aaai.v34i09.7123"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2018.00035"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.41"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.2478\/popets-2020-0060"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/11787006_1"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/1536414.1536440"},{"key":"e_1_3_2_57_2","article-title":"Federated optimization: Distributed optimization beyond the datacenter","author":"Kone\u010dn\u1ef3 Jakub","year":"2015","unstructured":"Jakub Kone\u010dn\u1ef3, Brendan McMahan, and Daniel Ramage. 2015. Federated optimization: Distributed optimization beyond the datacenter. In Proceedings of the OPT Workshop on Optimization for Machine Learning, NIPS.","journal-title":"I"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2020.07.061"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3242042"},{"key":"e_1_3_2_60_2","first-page":"2951","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Snoek Jasper","year":"2012","unstructured":"Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical Bayesian optimization of machine learning algorithms. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 2951\u20132959."},{"key":"e_1_3_2_61_2","first-page":"1168","volume-title":"Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research)","volume":"51","author":"Shahriari Bobak","year":"2016","unstructured":"Bobak Shahriari, Alexandre Bouchard-Cote, and Nando Freitas. 2016. Unbounded Bayesian optimization via regularization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Arthur Gretton and Christian C. Robert (Eds.), Vol. 51. PMLR, 1168\u20131176. Retrieved from http:\/\/proceedings.mlr.press\/v51\/shahriari16.html."},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240765.3243479"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330744"},{"key":"e_1_3_2_64_2","unstructured":"Danilo Sato Arif Wider and Christoph Windheuser. 2019. Continuous Delivery for Machine Learning. Retrieved from https:\/\/martinfowler.com\/articles\/cd4ml.html."},{"key":"e_1_3_2_65_2","article-title":"Opening the \u201cBlack Box\u201d: The path to deployment of AI models in banking, white paper","author":"Ananth Ram","year":"2019","unstructured":"Ram Ananth, Seph Mard, and Peter Simon. 2019. Opening the \u201cBlack Box\u201d: The path to deployment of AI models in banking, white paper. DataRobot (2019). https:\/\/www.datarobot.com\/resources\/opening-the-black-box\/. Accessed from 13\/06\/2022.","journal-title":"DataRobot"},{"key":"e_1_3_2_66_2","unstructured":"Prudential Regulation Authority. 2018. Model risk management principles for stress testing. https:\/\/www.bankofengland.co.uk\/prudential-regulation\/publication\/2018\/model-risk-management-principles-for-stress-testing-ss. Accessed from 13\/06\/2022."},{"key":"e_1_3_2_67_2","article-title":"Guide for the targeted review of internal models (TRIM)","author":"Guide ECB TRIM","year":"2017","unstructured":"ECB TRIM Guide. 2017. Guide for the targeted review of internal models (TRIM). European Central Bank. https:\/\/www.bankingsupervision.europa.eu\/ecb\/pub\/pdf\/trim_guide.en.pdf. Accessed from 13\/06\/2022.","journal-title":"European Central Bank"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCCN.2018.2878202"},{"key":"e_1_3_2_69_2","volume-title":"Proceedings of the Conference on Systems and Machine Learning","author":"Breck Eric","year":"2019","unstructured":"Eric Breck, Marty Zinkevich, Neoklis Polyzotis, Steven Whang, and Sudip Roy. 2019. Data validation for machine learning. In Proceedings of the Conference on Systems and Machine Learning. Retrieved from https:\/\/mlsys.org\/Conferences\/2019\/doc\/2019\/167.pdf."},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00023"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330739"},{"key":"e_1_3_2_72_2","first-page":"2503","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Sculley David","year":"2015","unstructured":"David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 2503\u20132511."},{"key":"e_1_3_2_73_2","article-title":"Monitoring and explainability of models in production","author":"Klaise Janis","year":"2020","unstructured":"Janis Klaise, Arnaud Van Looveren, Clive Cox, Giovanni Vacanti, and Alexandru Coca. 2020. Monitoring and explainability of models in production. In Proceedings of the Workshop on Challenges in Deploying and Monitoring Machine Learning Systems.","journal-title":"I"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219911"},{"key":"e_1_3_2_75_2","article-title":"Continual learning in practice","author":"Diethe Tom","year":"2018","unstructured":"Tom Diethe, Tom Borchert, Eno Thereska, Borja Balle, and Neil Lawrence. 2018. Continual learning in practice. In Proceedings of the Workshop on Continual Learning, NeurIPS.","journal-title":"Proceedings of the Workshop on Continual Learning, NeurIPS"},{"key":"e_1_3_2_76_2","volume-title":"Dataset Shift in Machine Learning","author":"Qui\u00f1onero-Candela Joaquin","year":"2009","unstructured":"Joaquin Qui\u00f1onero-Candela, Masashi Sugiyama, Neil D. Lawrence, and Anton Schwaighofer. 2009. Dataset Shift in Machine Learning. The MIT Press."},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2020.0110127"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3062900"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.3233\/IDA-194515"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.3389\/fmars.2020.00506"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2019.106031"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116895"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11411"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.23919\/CCC50068.2020.9189106"},{"key":"e_1_3_2_85_2","unstructured":"Arif Wider and Christian Deger. 2017. Getting Smart: Applying Continuous Delivery to Data Science to Drive Car Sales. Retrieved from https:\/\/www.thoughtworks.com\/insights\/blog\/getting-smart-applying-continuous-delivery-data-science-drive-car-sales."},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012429"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403379"},{"key":"e_1_3_2_88_2","article-title":"Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector","author":"Leslie David","year":"2019","unstructured":"David Leslie. 2019. Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector. Available at Retrieved from: SSRN 3403301 (2019).","journal-title":"Available at Retrieved from: SSRN 3403301"},{"key":"e_1_3_2_89_2","volume-title":"Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy","author":"O\u2019Neil Cathy","year":"2016","unstructured":"Cathy O\u2019Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Broadway Books."},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-019-04144-6"},{"key":"e_1_3_2_91_2","unstructured":"Robert Soden Dennis Wagenaar Dave Luo and Annegien Tijssen. 2019. Taking Ethics Fairness and Bias Seriously in Machine Learning for Disaster Risk Management. arxiv:cs.CY\/1912.05538."},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939709"},{"key":"e_1_3_2_93_2","first-page":"77","volume-title":"Proceedings of the Conference on Fairness, Accountability and Transparency","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the Conference on Fairness, Accountability and Transparency. PMLR, 77\u201391."},{"key":"e_1_3_2_94_2","article-title":"Fairness in machine learning","volume":"1","author":"Barocas Solon","year":"2017","unstructured":"Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017. Fairness in machine learning. NIPS Tutor. 1 (2017).","journal-title":"NIPS Tutor."},{"key":"e_1_3_2_95_2","first-page":"1","article-title":"Artificial intelligence in the creative industries: A review","author":"Anantrasirichai Nantheera","year":"2021","unstructured":"Nantheera Anantrasirichai and David Bull. 2021. Artificial intelligence in the creative industries: A review. Artif. Intell. Rev. (2021), 1\u201368.","journal-title":"Artif. Intell. Rev."},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/3425780"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.7108"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2011.11.004"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-49186-4_27"},{"key":"e_1_3_2_100_2","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1007\/978-94-007-1356-7_2","volume-title":"The Growing Gap between Emerging Technologies and Legal-ethical Oversight","author":"Marchant Gary E.","year":"2011","unstructured":"Gary E. Marchant. 2011. The growing gap between emerging technologies and the law. In The Growing Gap between Emerging Technologies and Legal-ethical Oversight. Springer, 19\u201333."},{"key":"e_1_3_2_101_2","volume-title":"Proceedings of the World Economic Forum","author":"Malan D.","year":"2016","unstructured":"D. Malan et\u00a0al. 2016. Values and the fourth industrial revolution: Connecting the dots between value, values, profit and purpose. In Proceedings of the World Economic Forum."},{"key":"e_1_3_2_102_2","volume-title":"Proceedings of the World Economic Forum","author":"Malan Daniel","year":"2018","unstructured":"Daniel Malan. 2018. The law can\u2019t keep up with new tech. Here\u2019s how to close the gap. In Proceedings of the World Economic Forum."},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1093\/idpl\/ipx005"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.1093\/jlb\/lsaa002"},{"key":"e_1_3_2_105_2","article-title":"FDA releases artificial intelligence\/machine learning action plan","author":"Stephens Keri","year":"2021","unstructured":"Keri Stephens. 2021. FDA releases artificial intelligence\/machine learning action plan. AXIS Imag. News (2021). https:\/\/www.fda.gov\/news-events\/press-announcements\/fda-releases-artificial-intelligencemachine-learning-action-plan. Accessed from 13\/06\/2022.","journal-title":"AXIS Imag. News"},{"key":"e_1_3_2_106_2","unstructured":"Mustafa Suleyman and Dominic King. 2017. The Information Commissioner the Royal Free and what we\u2019ve learned. (2017). https:\/\/www.deepmind.com\/blog\/the-information-commissioner-the-royal-free-and-what-weve-learned. Accessed from 13\/06\/2022."},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12967-019-02204-y"},{"key":"e_1_3_2_108_2","unstructured":"Ramprakash Ramamoorthy P. Satya Madhuri and Malini Christina Raj. 2019. AI from labs to production\u2014challenges and learnings. USENIX Association. https:\/\/www.usenix.org\/conference\/opml19\/presentation\/madhuri. Accessed from 13\/06\/2022."},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372827"},{"key":"e_1_3_2_110_2","unstructured":"Information Commissioner\u2019s Office. 2019. Project ExplAIn interim report. Retrieved from https:\/\/ico.org.uk\/about-the-ico\/research-and-reports\/project-explain-interim-report\/."},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13322"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0048-x"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i08.7055"},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445432"},{"key":"e_1_3_2_115_2","article-title":"Failure modes in machine learning systems","author":"Kumar Ram Shankar Siva","year":"2019","unstructured":"Ram Shankar Siva Kumar, David O. Brien, Kendra Albert, Salom\u00e9 Vilj\u00f6en, and Jeffrey Snover. 2019. Failure modes in machine learning systems. arXiv preprint arXiv:1911.11034 (2019).","journal-title":"arXiv preprint arXiv:1911.11034"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2018.07.023"},{"key":"e_1_3_2_117_2","article-title":"Adversarial machine learning at scale","author":"Kurakin Alexey","year":"2017","unstructured":"Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017. Adversarial machine learning at scale. In Proceedings of the International Conference on Learning Representations.","journal-title":"Proceedings of the International Conference on Learning Representations"},{"key":"e_1_3_2_118_2","article-title":"Adversarial machine learning-industry perspectives","author":"Kumar Ram Shankar Siva","year":"2020","unstructured":"Ram Shankar Siva Kumar, Magnus Nystrom, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. 2020. Adversarial machine learning-industry perspectives. Retrieved from SSRN 3532474 (2020).","journal-title":"Retrieved from SSRN 3532474"},{"key":"e_1_3_2_119_2","first-page":"1","article-title":"Exploiting machine learning to subvert your spam filter.","author":"Nelson Blaine","year":"2008","unstructured":"Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D. Joseph, Benjamin I. P. Rubinstein, Udam Saini, Charles Sutton, J. Doug Tygar, and Kai Xia. 2008. Exploiting machine learning to subvert your spam filter.In Proceedings of the 1st Usenix Workshop on Large-Scale Exploits and Emergent Threats. 1\u20139.","journal-title":"I"},{"key":"e_1_3_2_120_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00057"},{"key":"e_1_3_2_121_2","article-title":"In 2016 Microsoft\u2019s racist chatbot revealed the dangers of online conversation","volume":"11","author":"Schwartz Oscar","year":"2019","unstructured":"Oscar Schwartz. 2019. In 2016 Microsoft\u2019s racist chatbot revealed the dangers of online conversation. IEEE Spectr. 11 (2019).","journal-title":"IEEE Spectr."},{"key":"e_1_3_2_122_2","first-page":"601","volume-title":"Proceedings of the 25th USENIX Security Symposium (USENIX Security\u201916)","author":"Tram\u00e8r Florian","year":"2016","unstructured":"Florian Tram\u00e8r, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. 2016. Stealing machine learning models via prediction APIs. In Proceedings of the 25th USENIX Security Symposium (USENIX Security\u201916). 601\u2013618."},{"key":"e_1_3_2_123_2","doi-asserted-by":"publisher","DOI":"10.1145\/2810103.2813677"},{"key":"e_1_3_2_124_2","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2018.0083"},{"key":"e_1_3_2_125_2","unstructured":"Chip Huyen. 2020. What I learned from looking at 200 machine learning tools. Retrieved from https:\/\/huyenchip.com\/2020\/06\/22\/mlops.html."},{"key":"e_1_3_2_126_2","unstructured":"Kumar Venkateswar. 2019. Using Amazon SageMaker to operationalize machine learning. USENIX Association. https:\/\/www.usenix.org\/conference\/opml19\/presentation\/venkateswar. Accessed from 13\/06\/2022."},{"key":"e_1_3_2_127_2","first-page":"1","volume-title":"Proceedings of the Conference on Predictive APIs and Apps","year":"2016","unstructured":"AML Team. 2016. AzureML: Anatomy of a machine learning service. In Proceedings of the Conference on Predictive APIs and Apps. 1\u201313."},{"key":"e_1_3_2_128_2","article-title":"Meet Michelangelo: Uber\u2019s machine learning platform","author":"Hermann Jeremy","year":"2017","unstructured":"Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber\u2019s machine learning platform. Retrieved from https:\/\/eng.uber.com\/michelangelo.","journal-title":"https:\/\/eng.uber.com\/michelangelo"},{"key":"e_1_3_2_129_2","first-page":"51","volume-title":"Proceedings of the USENIX Conference on Operational Machine Learning (OpML\u201919)","author":"Baylor Denis","year":"2019","unstructured":"Denis Baylor, Kevin Haas, Konstantinos Katsiapis, Sammy Leong, Rose Liu, Clemens Menwald, Hui Miao, Neoklis Polyzotis, Mitchell Trott, and Martin Zinkevich. 2019. Continuous training for production ML in the TensorFlow extended (TFX) platform. In Proceedings of the USENIX Conference on Operational Machine Learning (OpML\u201919). 51\u201353."},{"issue":"4","key":"e_1_3_2_130_2","first-page":"39","article-title":"Accelerating the machine learning lifecycle with MLflow.","volume":"41","author":"Zaharia Matei","year":"2018","unstructured":"Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et\u00a0al. 2018. Accelerating the machine learning lifecycle with MLflow.IEEE Data Eng. Bull. 41, 4 (2018), 39\u201345.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_3_2_131_2","unstructured":"Sebastian Schelter Tammo Rukat and Felix Biessmann. 2021. JENGA-A framework to study the impact of data errors on the predictions of machine learning models. (2021). https:\/\/www.amazon.science\/publications\/jenga-a-framework-to-study-the-impact-of-data-errors-on-the-predictions-of-machine-learning-models. Accessed from 13\/06\/2022."},{"key":"e_1_3_2_132_2","volume-title":"Association for Computational Linguistics (ACL\u201920)","author":"Ribeiro Marco Tulio","year":"2020","unstructured":"Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Association for Computational Linguistics (ACL\u201920)."},{"key":"e_1_3_2_133_2","volume-title":"Proceedings of the NIPS MLSys Workshop","author":"Hynes Nick","year":"2017","unstructured":"Nick Hynes, D. Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, automated sanity checking for ML data sets. In Proceedings of the NIPS MLSys Workshop."},{"key":"e_1_3_2_134_2","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314036"},{"key":"e_1_3_2_135_2","doi-asserted-by":"publisher","DOI":"10.14778\/3291264.3291268"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12125"},{"key":"e_1_3_2_137_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3470827"},{"key":"e_1_3_2_138_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330648"},{"key":"e_1_3_2_139_2","volume-title":"Advances in Neural Information Processing Systems","author":"Feurer Matthias","year":"2015","unstructured":"Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2015\/file\/11d0e6287202fced83f79975ec59a3a6-Paper.pdf."},{"key":"e_1_3_2_140_2","first-page":"66","volume-title":"Proceedings of the Workshop on Automatic Machine Learning","author":"Olson Randal S.","year":"2016","unstructured":"Randal S. Olson and Jason H. Moore. 2016. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning. PMLR, 66\u201374."},{"key":"e_1_3_2_141_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2019.00196"},{"key":"e_1_3_2_142_2","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(19)30108-6"},{"key":"e_1_3_2_143_2","volume-title":"Proceedings of the Neural Information Processing Systems Foundation Conference","author":"Rabanser Stephan","year":"2019","unstructured":"Stephan Rabanser, Stephan G\u00fcnnemann, and Zachary Chase Lipton. 2019. Failing loudly: An empirical study of methods for detecting dataset shift. In Proceedings of the Neural Information Processing Systems Foundation Conference."},{"key":"e_1_3_2_144_2","unstructured":"Arnaud Van Looveren Janis Klaise Giovanni Vacanti Oliver Cobb Ashley Scillitoe and Robert Samoilescu. 2022. Alibi Detect: Algorithms for outlier adversarial and drift detection. (32022). Retrieved from https:\/\/github.com\/SeldonIO\/alibi-detect. Accessed from 13\/06\/2022."},{"key":"e_1_3_2_145_2","doi-asserted-by":"publisher","DOI":"10.1145\/2523813"},{"key":"e_1_3_2_146_2","first-page":"1126","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Finn Chelsea","year":"2017","unstructured":"Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning. PMLR, 1126\u20131135."},{"key":"e_1_3_2_147_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2014.06.003"},{"key":"e_1_3_2_148_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-59081-3_42"},{"key":"e_1_3_2_149_2","first-page":"2019","article-title":"AI project development\u2013how project managers should prepare","volume":"31","author":"Zujus A.","year":"2018","unstructured":"A. Zujus. 2018. AI project development\u2013how project managers should prepare. Tratto il giorno Agosto 31 (2018), 2019.","journal-title":"Tratto il giorno Agosto"},{"key":"e_1_3_2_150_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER50967.2021.00046"},{"key":"e_1_3_2_151_2","article-title":"Data readiness levels","author":"Lawrence Neil D.","year":"2017","unstructured":"Neil D. Lawrence. 2017. Data readiness levels. arXiv preprint arXiv:1705.02245 (2017).","journal-title":"arXiv preprint arXiv:1705.02245"},{"key":"e_1_3_2_152_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_3_2_153_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6195393"},{"key":"e_1_3_2_154_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_155_2","unstructured":"Neil Lawrence. 2019. Modern Data-oriented Programming. Retrieved from http:\/\/inverseprobability.com\/talks\/notes\/modern-data-oriented-programming.html."},{"key":"e_1_3_2_156_2","unstructured":"Tom Borchert. 2020. Milan: An Evolution of Data-oriented Programming. Retrieved from https:\/\/tborchertblog.wordpress.com\/2020\/02\/13\/28\/."},{"key":"e_1_3_2_157_2","article-title":"Towards better data discovery and collection with flow-based programming","author":"Paleyes Andrei","year":"2021","unstructured":"Andrei Paleyes, Christian Cabrera, and Neil D. Lawrence. 2021. Towards better data discovery and collection with flow-based programming. In Proceedings of the Data-centric AI Workshop, NeurIPS.","journal-title":"Proceedings of the Data-centric AI Workshop, NeurIPS"},{"key":"e_1_3_2_158_2","doi-asserted-by":"crossref","unstructured":"Alexander Lavin Ciar\u00e1n M. Gilligan-Lee Alessya Visnjic Siddha Ganju Dava Newman Sujoy Ganguly Danny Lange At\u0131l\u0131m G\u00fcne\u015f Baydin Amit Sharma Adam Gibson Yarin Gal Eric P. Xing Chris Mattmann and James Parr. 2021. Technology Readiness Levels for Machine Learning Systems. arxiv:cs.LG\/2101.03989.","DOI":"10.21203\/rs.3.rs-133138\/v1"},{"key":"e_1_3_2_159_2","unstructured":"Martin Zinkevich. 2017. Rules of machine learning: Best practices for ML engineering. Retrieved from https:\/\/developers.google.com\/machine-learning\/guides\/rules-of-ml."},{"key":"e_1_3_2_160_2","unstructured":"Verein Deutscher Ingenieure (VDI). 2019. Einsatz von Big Data in produzierenden Industrien. Retrieved from https:\/\/www.vdi.de\/news\/detail\/einsatz-von-big-data-in-produzierenden-industrien."}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533378","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533378","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:17Z","timestamp":1750186817000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533378"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,7]]},"references-count":159,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3533378"],"URL":"https:\/\/doi.org\/10.1145\/3533378","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,7]]},"assertion":[{"value":"2021-03-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-23","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}