{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T11:33:38Z","timestamp":1778672018438,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":61,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,5]],"date-time":"2020-10-05T00:00:00Z","timestamp":1601856000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,5]]},"DOI":"10.1145\/3382494.3410681","type":"proceedings-article","created":{"date-parts":[[2020,10,23]],"date-time":"2020-10-23T16:59:24Z","timestamp":1603472364000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":92,"title":["Adoption and Effects of Software Engineering Best Practices in Machine Learning"],"prefix":"10.1145","author":[{"given":"Alex","family":"Serban","sequence":"first","affiliation":[{"name":"ICIS, Radboud University, Software Improvement Group, Amsterdam, Netherlands"}]},{"given":"Koen","family":"van der Blom","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, Netherlands"}]},{"given":"Holger","family":"Hoos","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, Netherlands"}]},{"given":"Joost","family":"Visser","sequence":"additional","affiliation":[{"name":"LIACS, Leiden University, Leiden, Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2020,10,23]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Best Practices in Machine Learning Infrastructure. https:\/\/algorithmia.com\/blog\/best-practices-in-machine-learning-infrastructure. [Online","year":"2019","unstructured":"Algorithmia. 2019. Best Practices in Machine Learning Infrastructure. https:\/\/algorithmia.com\/blog\/best-practices-in-machine-learning-infrastructure. [Online ; accessed 15-12- 2019 ]. Algorithmia. 2019. Best Practices in Machine Learning Infrastructure. https:\/\/algorithmia.com\/blog\/best-practices-in-machine-learning-infrastructure. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_2_1","volume-title":"How to organize data labelling for Machine Learning. https:\/\/www.altexsoft.com\/blog\/datascience\/how-to-organize-data-labeling-for-machine-learning-approaches-and-tools\/. [Online","year":"2019","unstructured":"Altexsoft. 2018. How to organize data labelling for Machine Learning. https:\/\/www.altexsoft.com\/blog\/datascience\/how-to-organize-data-labeling-for-machine-learning-approaches-and-tools\/. [Online ; accessed 15-12- 2019 ]. Altexsoft. 2018. How to organize data labelling for Machine Learning. https:\/\/www.altexsoft.com\/blog\/datascience\/how-to-organize-data-labeling-for-machine-learning-approaches-and-tools\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_3_1","volume-title":"Software engineering for machine learning: A case study","author":"Amershi Saleema","unstructured":"Saleema Amershi , Andrew Begel , Christian Bird , Robert DeLine , Harald Gall , Ece Kamar , Nachiappan Nagappan , Besmira Nushi , and Thomas Zimmermann . 2019. Software engineering for machine learning: A case study . In ICSE-SEIP. IEEE , 291--300. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In ICSE-SEIP. IEEE, 291--300."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2018.00018"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098021"},{"key":"e_1_3_2_1_6_1","volume-title":"Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. In 2019 USENIX Conference on Operational Machine Learning (OpML 19)","author":"Baylor Denis","year":"2019","unstructured":"Denis Baylor , Kevin Haas , Konstantinos Katsiapis , Sammy Leong , Rose Liu , Clemens Menwald , Hui Miao , Neoklis Polyzotis , Mitchell Trott , and Martin Zinkevich . 2019 . Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. In 2019 USENIX Conference on Operational Machine Learning (OpML 19) . 51--53. Denis Baylor, Kevin Haas, Konstantinos Katsiapis, Sammy Leong, Rose Liu, Clemens Menwald, Hui Miao, Neoklis Polyzotis, Mitchell Trott, and Martin Zinkevich. 2019. Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform. In 2019 USENIX Conference on Operational Machine Learning (OpML 19). 51--53."},{"key":"e_1_3_2_1_7_1","volume-title":"https:\/\/ai.googleblog.com\/2019\/12\/fairness-indicators-scalable.html. [Online","author":"Blog Google AI","year":"2019","unstructured":"Google AI Blog . 2019. Fairness Indicators . https:\/\/ai.googleblog.com\/2019\/12\/fairness-indicators-scalable.html. [Online ; accessed 15-12- 2019 ]. Google AI Blog. 2019. Fairness Indicators. https:\/\/ai.googleblog.com\/2019\/12\/fairness-indicators-scalable.html. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_8_1","volume-title":"Responsible AI practices. https:\/\/ai.google\/responsibilities\/responsible-ai-practices\/. [Online","author":"Blog Google AI","year":"2019","unstructured":"Google AI Blog . 2019. Responsible AI practices. https:\/\/ai.google\/responsibilities\/responsible-ai-practices\/. [Online ; accessed 15-12- 2019 ]. Google AI Blog. 2019. Responsible AI practices. https:\/\/ai.google\/responsibilities\/responsible-ai-practices\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_9_1","volume-title":"How do teams work together on automated ML projects. https:\/\/azure.microsoft.com\/en-us\/blog\/how-do-teams-work-together-on-an-automated-machine-learning-project\/. [Online","author":"Blog Microsoft","year":"2019","unstructured":"Microsoft Blog . 2019. How do teams work together on automated ML projects. https:\/\/azure.microsoft.com\/en-us\/blog\/how-do-teams-work-together-on-an-automated-machine-learning-project\/. [Online ; accessed 15-12- 2019 ]. Microsoft Blog. 2019. How do teams work together on automated ML projects. https:\/\/azure.microsoft.com\/en-us\/blog\/how-do-teams-work-together-on-an-automated-machine-learning-project\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_10_1","volume-title":"Collaborative development environments. Advances in computers 59, 1","author":"Booch Grady","year":"2003","unstructured":"Grady Booch and Alan W Brown . 2003. Collaborative development environments. Advances in computers 59, 1 ( 2003 ), 1--27. Grady Booch and Alan W Brown. 2003. Collaborative development environments. Advances in computers 59, 1 (2003), 1--27."},{"key":"e_1_3_2_1_11_1","volume-title":"Reliable Machine Learning in the Wild - NeurIPS Workshop.","author":"Breck Eric","unstructured":"Eric Breck , Shanqing Cai , Eric Nielsen , Michael Salib , and D. Sculley . 2016. What's your ML test score? A rubric for ML production systems . In Reliable Machine Learning in the Wild - NeurIPS Workshop. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. 2016. What's your ML test score? A rubric for ML production systems. In Reliable Machine Learning in the Wild - NeurIPS Workshop."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2017.8258038"},{"key":"e_1_3_2_1_13_1","volume-title":"ML Ops: Machine Learning as an engineered disciplined. https:\/\/towardsdatascience.com\/ml-ops-machine-learning-as-an-engineering-discipline-b86ca4874a3f. [Online","author":"Breuel Cristiano","year":"2019","unstructured":"Cristiano Breuel . 2019. ML Ops: Machine Learning as an engineered disciplined. https:\/\/towardsdatascience.com\/ml-ops-machine-learning-as-an-engineering-discipline-b86ca4874a3f. [Online ; accessed 15-12- 2019 ]. Cristiano Breuel. 2019. ML Ops: Machine Learning as an engineered disciplined. https:\/\/towardsdatascience.com\/ml-ops-machine-learning-as-an-engineering-discipline-b86ca4874a3f. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_2_1_15_1","volume-title":"Empirical methods and studies in software engineering","author":"Ciolkowski Marcus","unstructured":"Marcus Ciolkowski , Oliver Laitenberger , Sira Vegas , and Stefan Biffl . 2003. Practical experiences in the design and conduct of surveys in empirical software engineering . In Empirical methods and studies in software engineering . Springer , 104--128. Marcus Ciolkowski, Oliver Laitenberger, Sira Vegas, and Stefan Biffl. 2003. Practical experiences in the design and conduct of surveys in empirical software engineering. In Empirical methods and studies in software engineering. Springer, 104--128."},{"key":"e_1_3_2_1_16_1","volume-title":"The ultimate guide to data labeling for ML. https:\/\/www.cloudfactory.com\/data-labeling-guide. [Online","year":"2019","unstructured":"Cloudfactory. 2019. The ultimate guide to data labeling for ML. https:\/\/www.cloudfactory.com\/data-labeling-guide. [Online ; accessed 15-12- 2019 ]. Cloudfactory. 2019. The ultimate guide to data labeling for ML. https:\/\/www.cloudfactory.com\/data-labeling-guide. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_17_1","volume-title":"Igor Steinmacher, and Tayana Conte.","author":"de Souza Nascimento Elizamary","year":"2019","unstructured":"Elizamary de Souza Nascimento , Iftekhar Ahmed , Edson Oliveira , M\u00e1rcio Piedade Palheta , Igor Steinmacher, and Tayana Conte. 2019 . Understanding Development Process of Machine Learning Systems: Challenges and Solutions. In ESEM. IEEE , 1--6. Elizamary de Souza Nascimento, Iftekhar Ahmed, Edson Oliveira, M\u00e1rcio Piedade Palheta, Igor Steinmacher, and Tayana Conte. 2019. Understanding Development Process of Machine Learning Systems: Challenges and Solutions. In ESEM. IEEE, 1--6."},{"key":"e_1_3_2_1_18_1","volume-title":"Testing and Debugging in Machine Learning. https:\/\/developers.google.com\/machine-learning\/testing-debugging\/pipeline\/production. [Online","author":"Devs Google","year":"2019","unstructured":"Google Devs . 2019. Testing and Debugging in Machine Learning. https:\/\/developers.google.com\/machine-learning\/testing-debugging\/pipeline\/production. [Online ; accessed 15-12- 2019 ]. Google Devs. 2019. Testing and Debugging in Machine Learning. https:\/\/developers.google.com\/machine-learning\/testing-debugging\/pipeline\/production. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_19_1","volume-title":"Team Data Science Process Documentation. https:\/\/docs.microsoft.com\/en-us\/azure\/machine-learning\/team-data-science-process\/. [Online","author":"Docs Microsoft","year":"2019","unstructured":"Microsoft Docs . 2019. Team Data Science Process Documentation. https:\/\/docs.microsoft.com\/en-us\/azure\/machine-learning\/team-data-science-process\/. [Online ; accessed 15-12- 2019 ]. Microsoft Docs. 2019. Team Data Science Process Documentation. https:\/\/docs.microsoft.com\/en-us\/azure\/machine-learning\/team-data-science-process\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_20_1","volume-title":"Machine Learning Logistics. https:\/\/mapr.com\/ebook\/machine-learning-logistics\/. [Online","author":"Dunning Ted","year":"2019","unstructured":"Ted Dunning and Ellen Friedman . 2019. Machine Learning Logistics. https:\/\/mapr.com\/ebook\/machine-learning-logistics\/. [Online ; accessed 15-12- 2019 ]. Ted Dunning and Ellen Friedman. 2019. Machine Learning Logistics. https:\/\/mapr.com\/ebook\/machine-learning-logistics\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_21_1","volume-title":"Coordinating expertise in software development teams. Management science 46, 12","author":"Faraj Samer","year":"2000","unstructured":"Samer Faraj and Lee Sproull . 2000. Coordinating expertise in software development teams. Management science 46, 12 ( 2000 ), 1554--1568. Samer Faraj and Lee Sproull. 2000. Coordinating expertise in software development teams. Management science 46, 12 (2000), 1554--1568."},{"key":"e_1_3_2_1_22_1","volume-title":"NeurIPS","author":"Feurer Matthias","year":"2015","unstructured":"Matthias Feurer , Aaron Klein , Katharina Eggensperger , Jost Springenberg , Manuel Blum , and Frank Hutter . 2015. Efficient and Robust Automated Machine Learning . In NeurIPS 2015 . 2962--2970. Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. 2015. Efficient and Robust Automated Machine Learning. In NeurIPS 2015. 2962--2970."},{"key":"e_1_3_2_1_23_1","volume-title":"Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16)","author":"Fr\u00e9chette Alexandre","year":"2016","unstructured":"Alexandre Fr\u00e9chette , Lars Kotthoff , Tomasz P. Michalak , Talal Rahwan , Holger H. Hoos , and Kevin Leyton-Brown . 2016 . Using the Shapley Value to Analyze Algorithm Portfolios . In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16) . AAAI Press, 3397--3403. http:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI16\/paper\/view\/12495 Alexandre Fr\u00e9chette, Lars Kotthoff, Tomasz P. Michalak, Talal Rahwan, Holger H. Hoos, and Kevin Leyton-Brown. 2016. Using the Shapley Value to Analyze Algorithm Portfolios. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16). AAAI Press, 3397--3403. http:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI16\/paper\/view\/12495"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.09.006"},{"key":"e_1_3_2_1_25_1","volume-title":"Benefitting from the Grey Literature in Software Engineering Research. arXiv:1911.12038","author":"Garousi Vahid","year":"2019","unstructured":"Vahid Garousi , Michael Felderer , Mika V M\u00e4ntyl\u00e4 , and Austen Rainer . 2019. Benefitting from the Grey Literature in Software Engineering Research. arXiv:1911.12038 ( 2019 ). Vahid Garousi, Michael Felderer, Mika V M\u00e4ntyl\u00e4, and Austen Rainer. 2019. Benefitting from the Grey Literature in Software Engineering Research. arXiv:1911.12038 (2019)."},{"key":"e_1_3_2_1_26_1","volume-title":"Principled Machine Learning: Practices and Tools for Efficient Collaboration. https:\/\/dev.to\/robogeek\/principled-machine-learning-4eho. [Online","author":"Herron David","year":"2019","unstructured":"David Herron . 2019. Principled Machine Learning: Practices and Tools for Efficient Collaboration. https:\/\/dev.to\/robogeek\/principled-machine-learning-4eho. [Online ; accessed 15-12- 2019 ]. David Herron. 2019. Principled Machine Learning: Practices and Tools for Efficient Collaboration. https:\/\/dev.to\/robogeek\/principled-machine-learning-4eho. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_27_1","first-page":"97","article-title":"Bias and changes in perceived intensity of verbal qualifiers effected by scale orientation","volume":"1","author":"Hofmans Joeri","year":"2007","unstructured":"Joeri Hofmans , Peter Theuns , Sven Baekelandt , Olivier Mairesse , Niels Schillewaert , and Walentina Cools . 2007 . Bias and changes in perceived intensity of verbal qualifiers effected by scale orientation . In Survey Research Methods , Vol. 1. 97 -- 108 . Joeri Hofmans, Peter Theuns, Sven Baekelandt, Olivier Mairesse, Niels Schillewaert, and Walentina Cools. 2007. Bias and changes in perceived intensity of verbal qualifiers effected by scale orientation. In Survey Research Methods, Vol. 1. 97--108.","journal-title":"Survey Research Methods"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC2E.2019.00025"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Frank Hutter Lars Kotthoff and Joaquin Vanschoren (Eds.). 2019. Automated Machine Learning: Methods Systems Challenges. Springer.  Frank Hutter Lars Kotthoff and Joaquin Vanschoren (Eds.). 2019. Automated Machine Learning: Methods Systems Challenges. Springer.","DOI":"10.1007\/978-3-030-05318-5"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/CESSER-IP.2019.00009"},{"key":"e_1_3_2_1_31_1","volume-title":"Software development best practices in a deep learning environment. https:\/\/towardsdatascience.com\/software-development-best-practices-in-a-deep-learning-environment-a1769e9859b1. [Online","author":"John Per","year":"2019","unstructured":"Per John . 2019. Software development best practices in a deep learning environment. https:\/\/towardsdatascience.com\/software-development-best-practices-in-a-deep-learning-environment-a1769e9859b1. [Online ; accessed 15-12- 2019 ]. Per John. 2019. Software development best practices in a deep learning environment. https:\/\/towardsdatascience.com\/software-development-best-practices-in-a-deep-learning-environment-a1769e9859b1. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2018.3571224"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/566493.566495"},{"key":"e_1_3_2_1_34_1","volume-title":"10 Best Practices for Deep Learning. https:\/\/nanonets.com\/blog\/10-best-practices-deep-learning\/. [Online","author":"James Le.","year":"2019","unstructured":"James Le. 2019. 10 Best Practices for Deep Learning. https:\/\/nanonets.com\/blog\/10-best-practices-deep-learning\/. [Online ; accessed 15-12- 2019 ]. James Le. 2019. 10 Best Practices for Deep Learning. https:\/\/nanonets.com\/blog\/10-best-practices-deep-learning\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_35_1","volume-title":"From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1","author":"Lundberg Scott M","year":"2020","unstructured":"Scott M Lundberg , Gabriel Erion , Hugh Chen , Alex DeGrave , Jordan M Prutkin , Bala Nair , Ronit Katz , Jonathan Himmelfarb , Nisha Bansal , and Su-In Lee . 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 ( 2020 ), 2522--5839. Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. 2020. From local explanations to global understanding with explainable AI for trees. Nature machine intelligence 2, 1 (2020), 2522--5839."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-19034-7_14"},{"key":"e_1_3_2_1_37_1","volume-title":"The Current State of Automated Machine Learning. https:\/\/www.kdnuggets.com\/2017\/01\/current-state-automated-machine-learning.html. [Online","author":"Mayo Matthew","year":"2020","unstructured":"Matthew Mayo . 2017. The Current State of Automated Machine Learning. https:\/\/www.kdnuggets.com\/2017\/01\/current-state-automated-machine-learning.html. [Online ; accessed 12- May - 2020 ]. Matthew Mayo. 2017. The Current State of Automated Machine Learning. https:\/\/www.kdnuggets.com\/2017\/01\/current-state-automated-machine-learning.html. [Online; accessed 12-May-2020]."},{"key":"e_1_3_2_1_38_1","volume-title":"Managing Machine Learning Projects. https:\/\/d1.awsstatic.com\/whitepapers\/aws-managing-ml-projects.pdf. [Online","author":"Megler V.M.","year":"2019","unstructured":"V.M. Megler . 2019. Managing Machine Learning Projects. https:\/\/d1.awsstatic.com\/whitepapers\/aws-managing-ml-projects.pdf. [Online ; accessed 15-12- 2019 ]. V.M. Megler. 2019. Managing Machine Learning Projects. https:\/\/d1.awsstatic.com\/whitepapers\/aws-managing-ml-projects.pdf. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_39_1","volume-title":"The curse of big data labeling and three ways to solve it. https:\/\/aws.amazon.com\/blogs\/apn\/the-curse-of-big-data-labeling-and-three-ways-to-solve-it\/. [Online","author":"Prendki Jennifer","year":"2019","unstructured":"Jennifer Prendki . 2018. The curse of big data labeling and three ways to solve it. https:\/\/aws.amazon.com\/blogs\/apn\/the-curse-of-big-data-labeling-and-three-ways-to-solve-it\/. [Online ; accessed 15-12- 2019 ]. Jennifer Prendki. 2018. The curse of big data labeling and three ways to solve it. https:\/\/aws.amazon.com\/blogs\/apn\/the-curse-of-big-data-labeling-and-three-ways-to-solve-it\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_40_1","volume-title":"A survey on data collection for machine learning: a big data-ai integration perspective","author":"Roh Yuji","year":"2019","unstructured":"Yuji Roh , Geon Heo , and Steven Euijong Whang . 2019. A survey on data collection for machine learning: a big data-ai integration perspective . IEEE Transactions on Knowledge and Data Engineering ( 2019 ). Yuji Roh, Geon Heo, and Steven Euijong Whang. 2019. A survey on data collection for machine learning: a big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering (2019)."},{"key":"e_1_3_2_1_41_1","volume-title":"Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data","author":"Rohrmann Bernd","year":"2007","unstructured":"Bernd Rohrmann . 2007. Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data . University of Melbourne , Australia ( 2007 ). Bernd Rohrmann. 2007. Verbal qualifiers for rating scales: Sociolinguistic considerations and psychometric data. University of Melbourne, Australia (2007)."},{"key":"e_1_3_2_1_42_1","volume-title":"Zinkevich Steven Whang Sudip Roy","author":"Alkis Polyzotis","year":"2017","unstructured":"Alkis Polyzotis Martin A. Zinkevich Steven Whang Sudip Roy . 2017 . Data management challenges in production machine learning. https:\/\/storage.googleapis.com\/pub-tools-public-publication-data\/pdf\/45a9dcf23dbdfa24dbced358f825636c58518afa.pdf. [Online; accessed 15-12-2019]. Alkis Polyzotis Martin A. Zinkevich Steven Whang Sudip Roy. 2017. Data management challenges in production machine learning. https:\/\/storage.googleapis.com\/pub-tools-public-publication-data\/pdf\/45a9dcf23dbdfa24dbced358f825636c58518afa.pdf. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_43_1","volume-title":"Preparing and Architecting for Machine Learning. https:\/\/www.gartner.com. [Online","author":"Sapp Carlton","year":"2019","unstructured":"Carlton Sapp . 2017. Preparing and Architecting for Machine Learning. https:\/\/www.gartner.com. [Online ; accessed 15-12- 2019 ]. Carlton Sapp. 2017. Preparing and Architecting for Machine Learning. https:\/\/www.gartner.com. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_44_1","volume-title":"Continuous Delivery for Machine Learning. https:\/\/martinfowler.com\/articles\/cd4ml.html. [Online","author":"Sato Danilo","year":"2019","unstructured":"Danilo Sato , Arif Wider , and Christoph Windheuser . 2019. Continuous Delivery for Machine Learning. https:\/\/martinfowler.com\/articles\/cd4ml.html. [Online ; accessed 15-12- 2019 ]. Danilo Sato, Arif Wider, and Christoph Windheuser. 2019. Continuous Delivery for Machine Learning. https:\/\/martinfowler.com\/articles\/cd4ml.html. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_45_1","volume-title":"NeurIPS","author":"Sculley David","year":"2015","unstructured":"David Sculley , Gary Holt , Daniel Golovin , Eugene Davydov , Todd Phillips , Dietmar Ebner , Vinay Chaudhary , Michael Young , Jean-Francois Crespo , and Dan Dennison . 2015. Hidden technical debt in machine learning systems . In NeurIPS 2015 . 2503--2511. David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In NeurIPS 2015. 2503--2511."},{"key":"e_1_3_2_1_46_1","volume-title":"The product backlog","author":"Sedano Todd","unstructured":"Todd Sedano , Paul Ralph , and C\u00e9cile P\u00e9raire . 2019. The product backlog . In ICSE. IEEE , 200--211. Todd Sedano, Paul Ralph, and C\u00e9cile P\u00e9raire. 2019. The product backlog. In ICSE. IEEE, 200--211."},{"key":"#cr-split#-e_1_3_2_1_47_1.1","doi-asserted-by":"crossref","unstructured":"Alex Serban Koen van der Blom Holger Hoos and Joost Visser. 2020. Adoption and Effects of Software Engineering Best Practices in Machine Learning - Supplementary Material. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.3946453 10.5281\/zenodo.3946453","DOI":"10.1145\/3382494.3410681"},{"key":"#cr-split#-e_1_3_2_1_47_1.2","doi-asserted-by":"crossref","unstructured":"Alex Serban Koen van der Blom Holger Hoos and Joost Visser. 2020. Adoption and Effects of Software Engineering Best Practices in Machine Learning - Supplementary Material. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.3946453","DOI":"10.1145\/3382494.3410681"},{"key":"e_1_3_2_1_48_1","volume-title":"Machine learning: Moving from experiments to production. https:\/\/blog.codecentric.de\/en\/2019\/03\/machine-learning-experiments-production\/. [Online","author":"Seyffarth Roman","year":"2019","unstructured":"Roman Seyffarth . 2019. Machine learning: Moving from experiments to production. https:\/\/blog.codecentric.de\/en\/2019\/03\/machine-learning-experiments-production\/. [Online ; accessed 15-12- 2019 ]. Roman Seyffarth. 2019. Machine learning: Moving from experiments to production. https:\/\/blog.codecentric.de\/en\/2019\/03\/machine-learning-experiments-production\/. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_49_1","volume-title":"2018 USENIX Annual Technical Conference (USENIX ATC 18)","author":"Sridhar Vinay","year":"2018","unstructured":"Vinay Sridhar , Sriram Subramanian , Dulcardo Arteaga , Swaminathan Sundararaman , Drew Roselli , and Nisha Talagala . 2018 . Model governance: Reducing the anarchy of production ML . In 2018 USENIX Annual Technical Conference (USENIX ATC 18) . 351--358. Vinay Sridhar, Sriram Subramanian, Dulcardo Arteaga, Swaminathan Sundararaman, Drew Roselli, and Nisha Talagala. 2018. Model governance: Reducing the anarchy of production ML. In 2018 USENIX Annual Technical Conference (USENIX ATC 18). 351--358."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2584053"},{"key":"e_1_3_2_1_51_1","volume-title":"The scrum guide. The definitive guide to scrum: The rules of the game. Scrum. org 268","author":"Sutherland Jeff","year":"2013","unstructured":"Jeff Sutherland and Ken Schwaber . 2013. The scrum guide. The definitive guide to scrum: The rules of the game. Scrum. org 268 ( 2013 ). Jeff Sutherland and Ken Schwaber. 2013. The scrum guide. The definitive guide to scrum: The rules of the game. Scrum. org 268 (2013)."},{"key":"e_1_3_2_1_52_1","volume-title":"Operational Machine Learning. https:\/\/www.kdnuggets.com\/2018\/04\/operational-machine-learning-successful-mlops.htmls. [Online","author":"Talagala Nisha","year":"2019","unstructured":"Nisha Talagala . 2018. Operational Machine Learning. https:\/\/www.kdnuggets.com\/2018\/04\/operational-machine-learning-successful-mlops.htmls. [Online ; accessed 15-12- 2019 ]. Nisha Talagala. 2018. Operational Machine Learning. https:\/\/www.kdnuggets.com\/2018\/04\/operational-machine-learning-successful-mlops.htmls. [Online; accessed 15-12-2019]."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40669-0_33"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3076246.3076248"},{"key":"e_1_3_2_1_55_1","volume-title":"C# Edition: Ten Guidelines for Future-Proof Code","author":"Visser Joost","unstructured":"Joost Visser , Sylvan Rigal , Gijs Wijnholds , Pascal van Eck , and Rob van der Leek . 2016. Building Maintainable Software , C# Edition: Ten Guidelines for Future-Proof Code . O'Reilly Media, Inc. Joost Visser, Sylvan Rigal, Gijs Wijnholds, Pascal van Eck, and Rob van der Leek. 2016. Building Maintainable Software, C# Edition: Ten Guidelines for Future-Proof Code. O'Reilly Media, Inc."},{"key":"e_1_3_2_1_56_1","volume-title":"How does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering","author":"Wan Zhiyuan","year":"2019","unstructured":"Zhiyuan Wan , Xin Xia , David Lo , and Gail C Murphy . 2019. How does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering ( 2019 ). Zhiyuan Wan, Xin Xia, David Lo, and Gail C Murphy. 2019. How does Machine Learning Change Software Development Practices? IEEE Transactions on Software Engineering (2019)."},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWESEP49350.2019.00017"},{"key":"e_1_3_2_1_58_1","volume-title":"Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining. Springer, 29--39","author":"Wirth R\u00fcdiger","year":"2000","unstructured":"R\u00fcdiger Wirth and Jochen Hipp . 2000 . CRISP-DM: Towards a standard process model for data mining . In Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining. Springer, 29--39 . R\u00fcdiger Wirth and Jochen Hipp. 2000. CRISP-DM: Towards a standard process model for data mining. In Proceedings of the 4th international conference on the practical applications of knowledge discovery and data mining. Springer, 29--39."},{"key":"e_1_3_2_1_59_1","volume-title":"Machine learning testing: Survey, landscapes and horizons","author":"Zhang Jie M","year":"2020","unstructured":"Jie M Zhang , Mark Harman , Lei Ma , and Yang Liu . 2020. Machine learning testing: Survey, landscapes and horizons . IEEE Transactions on Software Engineering ( 2020 ). Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering (2020)."},{"key":"e_1_3_2_1_60_1","unstructured":"Martin Zinkevich. 2019. Rules of Machine Learning: Best Practices for ML Engineering. https:\/\/developers.google.com\/machine-learning\/guides\/rules-of-ml. [Online;accessed15-12-2019].  Martin Zinkevich. 2019. Rules of Machine Learning: Best Practices for ML Engineering. https:\/\/developers.google.com\/machine-learning\/guides\/rules-of-ml. [Online;accessed15-12-2019]."}],"event":{"name":"ESEM '20: ACM \/ IEEE International Symposium on Empirical Software Engineering and Measurement","location":"Bari Italy","acronym":"ESEM '20","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the 14th ACM \/ IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3382494.3410681","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3382494.3410681","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:26Z","timestamp":1750199906000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3382494.3410681"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,5]]},"references-count":61,"alternative-id":["10.1145\/3382494.3410681","10.1145\/3382494"],"URL":"https:\/\/doi.org\/10.1145\/3382494.3410681","relation":{},"subject":[],"published":{"date-parts":[[2020,10,5]]},"assertion":[{"value":"2020-10-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}