{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T23:30:38Z","timestamp":1781220638134,"version":"3.54.1"},"publisher-location":"Cham","reference-count":23,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783030190330","type":"print"},{"value":"9783030190347","type":"electronic"}],"license":[{"start":{"date-parts":[[2019,1,1]],"date-time":"2019-01-01T00:00:00Z","timestamp":1546300800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Artificial intelligence enabled systems have been an inevitable part of everyday life. However, efficient software engineering principles and processes need to be considered and extended when developing AI- enabled systems. The objective of this study is to identify and classify software engineering challenges that are faced by different companies when developing software-intensive systems that incorporate machine learning components. Using case study approach, we explored the development of machine learning systems from six different companies across various domains and identified main software engineering challenges. The challenges are mapped into a proposed taxonomy that depicts the evolution of use of ML components in software-intensive system in industrial settings. Our study provides insights to software engineering community and research to guide discussions and future research into applied machine learning.<\/jats:p>","DOI":"10.1007\/978-3-030-19034-7_14","type":"book-chapter","created":{"date-parts":[[2019,5,10]],"date-time":"2019-05-10T23:08:11Z","timestamp":1557529691000},"page":"227-243","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":147,"title":["A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation"],"prefix":"10.1007","author":[{"given":"Lucy Ellen","family":"Lwakatare","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aiswarya","family":"Raj","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan","family":"Bosch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Helena Holmstr\u00f6m","family":"Olsson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ivica","family":"Crnkovic","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,4,27]]},"reference":[{"key":"14_CR1","doi-asserted-by":"publisher","unstructured":"Arpteg, A., Brinne, B., Crnkovic-Friis, L., Bosch, J.: Software engineering challenges of deep learning. In: 44th Euromicro Conference on Software Engineering and Advanced Applications, pp. 50\u201359. IEEE (2018). https:\/\/doi.org\/10.1109\/SEAA.2018.00018","DOI":"10.1109\/SEAA.2018.00018"},{"issue":"2","key":"14_CR2","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/1964897.1964906","volume":"12","author":"J Attenberg","year":"2011","unstructured":"Attenberg, J., Provost, F.: Inactive learning? Difficulties employing active learning in practice. ACM SIGKDD Explor. Newsl. 12(2), 36\u201341 (2011)","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"14_CR3","unstructured":"Bosch, J., Olsson, H.H., Crnkovic, I.: It takes three to tango: requirement, outcome\/data, and AI driven. In: International Workshop on Software-Intensive Business: Start-Ups, Ecosystems and Platforms, pp. 177\u2013192 (2018)"},{"key":"14_CR4","doi-asserted-by":"publisher","unstructured":"Hains, G., Jakobsson, A., Khmelevsky, Y.: Towards formal methods and software engineering for deep learning: security, safety and productivity for dl systems development. In: 2018 Annual IEEE International Systems Conference, pp. 1\u20135. IEEE, April 2018. https:\/\/doi.org\/10.1109\/SYSCON.2018.8369576","DOI":"10.1109\/SYSCON.2018.8369576"},{"key":"14_CR5","doi-asserted-by":"publisher","unstructured":"Hazelwood, K., et al.: Applied machine learning at Facebook: a datacenter infrastructure perspective. In: International Symposium on High Performance Computer Architecture, pp. 620\u2013629. IEEE (2018). https:\/\/doi.org\/10.1109\/HPCA.2018.00059","DOI":"10.1109\/HPCA.2018.00059"},{"key":"14_CR6","doi-asserted-by":"publisher","unstructured":"Hill, C., Bellamy, R., Erickson, T., Burnett, M.: Trials and tribulations of developers of intelligent systems: a field study. In: Symposium on Visual Languages and Human-Centric Computing, pp. 162\u2013170. IEEE (2016). https:\/\/doi.org\/10.1109\/VLHCC.2016.7739680","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"14_CR7","doi-asserted-by":"publisher","unstructured":"Kumar, R.S.S., Wicker, A., Swann, M.: Practical machine learning for cloud intrusion detection: challenges and the way forward. In: 10th Workshop on Artificial Intelligence and Security, pp. 81\u201390. ACM (2017). https:\/\/doi.org\/10.1145\/3128572.3140445","DOI":"10.1145\/3128572.3140445"},{"key":"14_CR8","unstructured":"Lefortier, D., Truchet, A., de Rijke, M.: Sources of variability in large-scale machine learning systems. In: Machine Learning Systems (NIPS 2015 Workshop) (2015)"},{"key":"14_CR9","doi-asserted-by":"publisher","unstructured":"Lin, J., Kolcz, A.: Large-scale machine learning at Twitter. In: SIGMOD International Conference on Management of Data, pp. 793\u2013804. ACM (2012). https:\/\/doi.org\/10.1145\/2213836.2213958","DOI":"10.1145\/2213836.2213958"},{"key":"14_CR10","doi-asserted-by":"publisher","unstructured":"Liu, S., Xiao, F., Ou, W., Si, L.: Cascade ranking for operational e-commerce search. In: International Conference on Knowledge Discovery and Data Mining, pp. 1557\u20131565. ACM (2017). https:\/\/doi.org\/10.1145\/3097983.3098011","DOI":"10.1145\/3097983.3098011"},{"key":"14_CR11","unstructured":"Murphy, C., Kaiser, G.E., Arias, M.: An approach to software testing of machine learning applications. In: 19th International Conference on Software Engineering and Knowledge Engineering, pp. 167\u2013172. Knowledge Systems Institute Graduate School (2007)"},{"key":"14_CR12","doi-asserted-by":"crossref","unstructured":"NVIDIA: Nvidia drive hardware for self-driving cars. https:\/\/www.nvidia.com\/en-us\/self-driving-cars\/drive-platform\/hardware\/. Accessed 11 Jan 2019","DOI":"10.1201\/9780429316777-2"},{"key":"14_CR13","doi-asserted-by":"publisher","unstructured":"Polyzotis, N., Roy, S., Whang, S.E., Zinkevich, M.: Data management challenges in production machine learning. In: International Conference on Management of Data, pp. 1723\u20131726. ACM (2017). https:\/\/doi.org\/10.1145\/3035918.3054782","DOI":"10.1145\/3035918.3054782"},{"issue":"2","key":"14_CR14","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1023\/A:1007442505281","volume":"30","author":"F Provost","year":"1998","unstructured":"Provost, F., Kohavi, R.: Guest editors\u2019 introduction: on applied research in machine learning. Mach. Learn. 30(2), 127\u2013132 (1998). https:\/\/doi.org\/10.1023\/A:1007442505281","journal-title":"Mach. Learn."},{"key":"14_CR15","doi-asserted-by":"crossref","unstructured":"Raeder, T., Stitelman, O., Dalessandro, B., Perlich, C., Provost, F.: Design principles of massive, robust prediction systems. In: International Conference on Knowledge Discovery and Data Mining, pp. 1357\u20131365. ACM (2012)","DOI":"10.1145\/2339530.2339740"},{"key":"14_CR16","doi-asserted-by":"crossref","unstructured":"Rana, R., Staron, M., Hansson, J., Nilsson, M., Meding, W.: A framework for adoption of machine learning in industry for software defect prediction. In: 9th International Conference on Software Engineering and Applications, pp. 383\u2013392. IEEE (2014)","DOI":"10.5220\/0005099303830392"},{"key":"14_CR17","doi-asserted-by":"crossref","unstructured":"Runeson, P., H\u00f6st, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14(2), (2008)","DOI":"10.1007\/s10664-008-9102-8"},{"key":"14_CR18","unstructured":"Schelter, S., B\u00f6se, J.H., Kirschnick, J., Klein, T., Seufert, S.: Automatically tracking metadata and provenance of machine learning experiments. In: NIPS Workshop on Machine Learning Systems (2017)"},{"key":"14_CR19","doi-asserted-by":"publisher","unstructured":"Schleier-Smith, J.: An architecture for agile machine learning in real-time applications. In: International Conference on Knowledge Discovery and Data Mining, pp. 2059\u20132068. ACM (2015). https:\/\/doi.org\/10.1145\/2783258.2788628","DOI":"10.1145\/2783258.2788628"},{"key":"14_CR20","unstructured":"Sculley, D., et al.: Hidden technical debt in machine learning systems. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, pp. 2503\u20132511. Curran Associates, Inc. (2015)"},{"key":"14_CR21","doi-asserted-by":"publisher","unstructured":"Tata, S., et al.: Quick access: building a smart experience for Google drive. In: 23rd International Conference on Knowledge Discovery and Data Mining, pp. 1643\u20131651. ACM (2017). https:\/\/doi.org\/10.1145\/3097983.3098048","DOI":"10.1145\/3097983.3098048"},{"key":"14_CR22","doi-asserted-by":"publisher","unstructured":"van der Weide, T., Papadopoulos, D., Smirnov, O., Zielinski, M., van Kasteren, T.: Versioning for end-to-end machine learning pipelines. In: 1st Workshop on Data Management for End-to-End Machine Learning, pp. 2:1\u20132:9. ACM (2017). https:\/\/doi.org\/10.1145\/3076246.3076248","DOI":"10.1145\/3076246.3076248"},{"issue":"2","key":"14_CR23","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1023\/A:1023760326768","volume":"11","author":"D Zhang","year":"2003","unstructured":"Zhang, D., Tsai, J.J.: Machine learning and software engineering. Softw. Qual. J. 11(2), 87\u2013119 (2003). https:\/\/doi.org\/10.1023\/A:1023760326768","journal-title":"Softw. Qual. J."}],"container-title":["Lecture Notes in Business Information Processing","Agile Processes in Software Engineering and Extreme Programming"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-19034-7_14","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,12]],"date-time":"2024-03-12T13:40:58Z","timestamp":1710250858000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-030-19034-7_14"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"ISBN":["9783030190330","9783030190347"],"references-count":23,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-19034-7_14","relation":{},"ISSN":["1865-1348","1865-1356"],"issn-type":[{"value":"1865-1348","type":"print"},{"value":"1865-1356","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019]]},"assertion":[{"value":"27 April 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"XP","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Agile Software Development","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Montreal, QC","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Canada","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2019","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"21 May 2019","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"25 May 2019","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"20","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"xpu2019","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/www.agilealliance.org\/xp2019\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"45","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"15","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"33% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"4","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"4","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}