{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T13:19:35Z","timestamp":1770729575097,"version":"3.49.0"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,12,28]],"date-time":"2024-12-28T00:00:00Z","timestamp":1735344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,1,31]]},"abstract":"<jats:p>\n            Effective model management is crucial for ensuring performance and reliability in Machine Learning (ML) systems, given the dynamic nature of data and operational environments. However, standard practices are lacking, often resulting in\n            <jats:italic>ad hoc<\/jats:italic>\n            approaches. To address this, our research provides a clear definition of ML model management activities, processes, and techniques. Analyzing 227 ML repositories, we propose a taxonomy of 16 model management activities and identify 12 unique challenges. We find that 57.9% of the identified activities belong to the maintenance category, with activities like refactoring (20.5%) and documentation (18.3%) dominating. Our findings also reveal significant challenges in documentation maintenance (15.3%) and bug management (14.9%), emphasizing the need for robust versioning tools and practices in the ML pipeline. Additionally, we conducted a survey that underscores a shift toward automation, particularly in data, model, and documentation versioning, as key to managing ML models effectively. Our contributions include a detailed taxonomy of model management activities, a mapping of challenges to these activities, practitioner-informed solutions for challenge mitigation, and a publicly available dataset of model management activities and challenges. This work aims to equip ML developers with knowledge and best practices essential for the robust management of ML models.\n          <\/jats:p>","DOI":"10.1145\/3688841","type":"journal-article","created":{"date-parts":[[2024,8,16]],"date-time":"2024-08-16T15:45:14Z","timestamp":1723823114000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["An Exploratory Study on Machine Learning Model Management"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0481-8331","authenticated-orcid":false,"given":"Jasmine","family":"Latendresse","sequence":"first","affiliation":[{"name":"Concordia University, Montreal, Quebec, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0472-4514","authenticated-orcid":false,"given":"Samuel","family":"Abedu","sequence":"additional","affiliation":[{"name":"Concordia University, Montreal, Quebec, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1863-9147","authenticated-orcid":false,"given":"Ahmad","family":"Abdellatif","sequence":"additional","affiliation":[{"name":"University of Calgary, Calgary, Alberta, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1285-9878","authenticated-orcid":false,"given":"Emad","family":"Shihab","sequence":"additional","affiliation":[{"name":"Concordia University, Montreal, Quebec, Canada"}]}],"member":"320","published-online":{"date-parts":[[2024,12,28]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387473"},{"key":"e_1_3_2_3_2","first-page":"291","volume-title":"2019 IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP \u201919)","author":"Amershi S.","year":"2019","unstructured":"S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann. 2019. Software engineering for machine learning: A case study. In 2019 IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP \u201919). IEEE, 291\u2013300."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453478"},{"issue":"7","key":"e_1_3_2_5_2","first-page":"1","article-title":"Challenges in deploying machine learning: A survey of case studies","volume":"55","author":"Paleyes A.","year":"2022","unstructured":"A. Paleyes, R.-G. Urma, and N. D. Lawrence. 2022. Challenges in deploying machine learning: A survey of case studies. ACM Computing Surveys 55, 7 (2022), 1\u201329.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_6_2","first-page":"1","volume-title":"Workshop on Human-In-the-Loop Data Analytics","author":"Vartak M.","year":"2016","unstructured":"M. Vartak, H. Subramanyam, W.-E. Lee, S. Viswanathan, S. Husnoo, S. Madden, and M. Zaharia. 2016. Modeldb: A system for machine learning model management. In Workshop on Human-In-the-Loop Data Analytics, 1\u20133."},{"key":"e_1_3_2_7_2","first-page":"5","article-title":"On challenges in machine learning model management","volume":"41","author":"Schelter S.","year":"2018","unstructured":"S. Schelter, F. Biessmann, T. Januschowski, D. Salinas, S. Seufert, and G. Szarvas. 2018. On challenges in machine learning model management. IEEE Database Engineering Bulletin 41 (2018), 5\u201315.","journal-title":"IEEE Database Engineering Bulletin"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543847"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","unstructured":"N. Nahar H. Zhang G. Lewis S. Zhou and C. K\u00e4stner. 2023. A meta-summary of challenges in building products with ml components\u2013collecting experiences from 4758+ practitioners. DOI: 10.1109\/CAIN58948.2023.00034","DOI":"10.1109\/CAIN58948.2023.00034"},{"issue":"4","key":"e_1_3_2_10_2","first-page":"16","article-title":"Modeldb: Opportunities and challenges in managing machine learning models","volume":"41","author":"Vartak M.","year":"2018","unstructured":"M. Vartak and S. Madden. 2018. Modeldb: Opportunities and challenges in managing machine learning models. IEEE Data Engineering Bulletin 41, 4 (2018), 16\u201325.","journal-title":"IEEE Data Engineering Bulletin"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9393-5"},{"key":"e_1_3_2_12_2","first-page":"69","volume-title":"16th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement","author":"Rzig D. E.","year":"2022","unstructured":"D. E. Rzig, F. Hassan, C. Bansal, and N. Nagappan. 2022. Characterizing the usage of ci tools in ml projects. In 16th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement, 69\u201379."},{"key":"e_1_3_2_13_2","first-page":"501","volume-title":"2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C \u201917)","author":"Gousios G.","year":"2017","unstructured":"G. Gousios and D. Spinellis. 2017. Mining software engineering data from github. In 2017 IEEE\/ACM 39th International Conference on Software Engineering Companion (ICSE-C \u201917). IEEE, 501\u2013502."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9512-6"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597073.2597074"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.09.016"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510057"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397366"},{"key":"e_1_3_2_19_2","first-page":"63","volume-title":"Personal Opinion Surveys","author":"Kitchenham B. A.","year":"2008","unstructured":"B. A. Kitchenham and S. L. Pfleeger. 2008. Personal Opinion Surveys. Springer, London, 63\u201392."},{"key":"e_1_3_2_20_2","first-page":"147","volume-title":"International Conference on Software Engineering","author":"Guan H.","year":"2023","unstructured":"H. Guan, Y. Xiao, J. Li, Y. Liu, and G. Bai. 2023. A comprehensive study of real-world bugs in machine learning model optimization. In International Conference on Software Engineering, 147\u2013158."},{"key":"e_1_3_2_21_2","unstructured":"S. Team. 2023. Documentation as code: Why you need it & how to get started. Retrieved from https:\/\/swimm.io\/learn\/code-documentation\/documentation-as-code-why-you-need-it-and-how-to-get-started"},{"key":"e_1_3_2_22_2","first-page":"152","volume-title":"2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER \u201922)","author":"Serban A.","year":"2022","unstructured":"A. Serban and J. Visser. 2022. Adapting software architectures to machine learning challenges. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER \u201922). IEEE, 152\u2013163."},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1109\/VLHCC.2016.7739680","volume-title":"2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC \u201916)","author":"Hill C.","year":"2016","unstructured":"C. Hill, R. Bellamy, T. Erickson, and M. Burnett. 2016. Trials and tribulations of developers of intelligent systems: A field study. In 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC \u201916). IEEE, 162\u2013170."},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1007\/978-3-030-62136-0_17","volume-title":"Model-Based Engineering of Collaborative Embedded Systems","author":"Hillemacher S.","year":"2021","unstructured":"S. Hillemacher, N. J\u00e4ckel, C. Kugler, P. Orth, D. Schmalzing, and L. Wachtmeister. 2021. Artifact-based analysis for the development of collaborative embedded systems. In Model-Based Engineering of Collaborative Embedded Systems. Springer, 315\u2013331."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543847"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316447303300309"},{"key":"e_1_3_2_27_2","first-page":"2503","article-title":"Hidden technical debt in machine learning systems","volume":"28","author":"Sculley D.","year":"2015","unstructured":"D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison. 2015. Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems 28 (2015), 2503\u20132511.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10073-7"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110542"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379177.3388905"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517607"},{"key":"e_1_3_2_32_2","first-page":"334","article-title":"Data validation for machine learning","author":"Breck E.","year":"2019","unstructured":"E. Breck, N. Polyzotis, S. Roy, S. Whang, and M. Zinkevich. 2019. Data validation for machine learning. In MLSys, 334\u2013347.","journal-title":"MLSys"},{"key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/SEAA.2018.00018","volume-title":"2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA \u201918)","author":"Arpteg A.","year":"2018","unstructured":"A. Arpteg, B. Brinne, L. Crnkovic-Friis, and J. Bosch. 2018. Software engineering challenges of deep learning. In 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA \u201918). IEEE, 50\u201359."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-09993-1"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1109\/IC2E.2019.00025","volume-title":"2019 IEEE International Conference on Cloud Engineering (IC2E \u201919)","author":"Hummer W.","year":"2019","unstructured":"W. Hummer, V. Muthusamy, T. Rausch, P. Dube, K. E. l. Maghraoui, A. Murthi, and P. Oum. 2019. Modelops: Cloud-based lifecycle management for reliable and trusted ai. In 2019 IEEE International Conference on Cloud Engineering (IC2E \u201919). IEEE, 113\u2013120."},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","first-page":"634","DOI":"10.1109\/ICSME52107.2021.00071","volume-title":"2021 IEEE International Conference on Software Maintenance and Evolution (ICSME \u201921)","author":"Lewis G. A.","year":"2021","unstructured":"G. A. Lewis, I. Ozkaya, and X. Xu. 2021. Software architecture challenges for ml systems. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME \u201921). IEEE, 634\u2013638."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","unstructured":"S. Shankar R. Garcia J. M. Hellerstein and A. G. Parameswaran. 2022. Operationalizing machine learning: An interview study. DOI: 10.48550\/arXiv.2209.09125","DOI":"10.48550\/arXiv.2209.09125"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502102"},{"key":"e_1_3_2_39_2","first-page":"1","volume-title":"1st International Conference on AI Engineering: Software Engineering for AI","author":"Golendukhina V.","year":"2022","unstructured":"V. Golendukhina, V. Lenarduzzi, and M. Felderer. 2022. What is software quality for ai engineers? towards a thinning of the fog. In 1st International Conference on AI Engineering: Software Engineering for AI, 1\u20139."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2754374"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510209"},{"issue":"9","key":"e_1_3_2_42_2","first-page":"1857","article-title":"How does machine learning change software development practices?","volume":"47","author":"Wan Z.","year":"2019","unstructured":"Z. Wan, X. Xia, D. Lo, and G. C. Murphy. 2019. How does machine learning change software development practices? IEEE Transactions on Software Engineering 47, 9 (2019), 1857\u20131871.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_43_2","first-page":"1","volume-title":"14th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201920)","author":"Serban A.","year":"2020","unstructured":"A. Serban, K. van der Blom, H. Hoos, and J. Visser. 2020. Adoption and effects of software engineering best practices in machine learning. In 14th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201920), 1\u201312."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","unstructured":"M. Saidur Rahman F. Khomh A. Hamidi J. Cheng G. Antoniol and H. Washizaki. 2021. Machine learning application development: Practitioners\u2019 insights. DOI: 10.48550\/arXiv.2112.15277","DOI":"10.48550\/arXiv.2112.15277"},{"key":"e_1_3_2_45_2","doi-asserted-by":"crossref","first-page":"666","DOI":"10.1109\/ICSE.2015.82","volume-title":"2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering","volume":"1","author":"Salman I.","year":"2015","unstructured":"I. Salman, A. T. Misirli, and N. Juristo. 2015. Are students representatives of professionals in software engineering experiments?\u201d In 2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering, Vol. 1, 666\u2013676."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026586415054"},{"key":"e_1_3_2_47_2","first-page":"3","article-title":"Hyperparameter optimization","volume":"1","author":"Feurer M.","year":"2019","unstructured":"M. Feurer and F. Hutter. 2019. Hyperparameter optimization. Automated Machine Learning: Methods, Systems, Challenges 1 (2019), 3\u201333.","journal-title":"Automated Machine Learning: Methods, Systems, Challenges"},{"key":"e_1_3_2_48_2","unstructured":"A. Grigorev. 2021. Deploying machine learning models part 3: managing dependencies. Retrieved from https:\/\/freecontent.manning.com\/deploying-machine-learning-models-part-3-managing-dependencies\/"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.3390\/info11070363"},{"key":"e_1_3_2_50_2","unstructured":"OpenSourceForU. 2021. Why python is popular for machine learning implementations. Retrieved November 09 2023 from https:\/\/www.opensourceforu.com\/2021\/02\/why-python-is-popular-for-machine-learning-implementations\/"},{"key":"e_1_3_2_51_2","first-page":"165","volume-title":"31st ACM International Conference on Design of Communication","author":"Watson R.","year":"2013","unstructured":"R. Watson, M. Stamnes, J. Jeannot-Schroeder, and J. H. Spyridakis. 2013. Api documentation and software community values: A survey of open-source api documentation. In 31st ACM International Conference on Design of Communication, 165\u2013174."},{"key":"e_1_3_2_52_2","author":"Sonnenburg S.","year":"2007","unstructured":"S. Sonnenburg, M. L. Braun, C. S. Ong, S. Bengio, L. Bottou, G. Holmes, Y. LeCunn, K.-R. Muller, F. Pereira, C. E. Rasmussen. 2007. The need for open source software in machine learning.","journal-title":"The need for open source software in machine learning"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1088\/1749-4699\/6\/1\/015010"},{"key":"e_1_3_2_54_2","doi-asserted-by":"crossref","unstructured":"Z. Yang C. Wang J. Shi T. Hoang P. Kochhar Q. Lu Z. Xing and D. Lo. 2023. What do users ask in open-source ai repositories? An empirical study of github issues. arXiv:2303.09795.","DOI":"10.1109\/MSR59073.2023.00024"},{"key":"e_1_3_2_55_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1007\/978-3-319-10247-4_3","article-title":"Data preparation basic models","volume":"72","author":"Garc\u00eda S.","year":"2015","unstructured":"S. Garc\u00eda, J. Luengo, F. Herrera, S. Garc\u00eda, J. Luengo, and F. Herrera. 2015. Data preparation basic models. Data Preprocessing in Data Mining 72 (2015), 39\u201357.","journal-title":"Data Preprocessing in Data Mining"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-021-00492-0"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2022.03.004"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10291-1"},{"key":"e_1_3_2_59_2","unstructured":"NVIDIA DEVELOPER. Jun 2023. Retrieved from https:\/\/developer.nvidia.com\/blog\/a-guide-to-monitoring-machine-learning-models-in-production\/"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-016-0043-6"},{"key":"e_1_3_2_61_2","unstructured":"B. John. 2023. Ml model packaging [the ultimate guide]. Retrieved from https:\/\/neptune.ai\/blog\/ml-model-packaging"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-019-04507-z"},{"key":"e_1_3_2_63_2","unstructured":"[Online]. Available: https:\/\/cloud.google.com\/architecture\/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning"},{"key":"e_1_3_2_64_2","first-page":"28 747","article-title":"Is a modular architecture enough?\u201d","volume":"35","author":"Mittal S.","year":"2022","unstructured":"S. Mittal, Y. Bengio, and G. Lajoie. 2022. Is a modular architecture enough?\u201d Advances in Neural Information Processing Systems 35 (2022), 28 747\u201328 760.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","unstructured":"J. Zhang S. H. Yeung Y. Shu B. He and W. Wang. 2019. Efficient memory management for gpu-based deep learning systems. DOI: 10.48550\/arXiv.1903.06631","DOI":"10.48550\/arXiv.1903.06631"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.sysarc.2019.101635"},{"key":"e_1_3_2_67_2","first-page":"101","volume-title":"2019 IEEE International Conference on Artificial Intelligence Testing (AITest \u201919)","author":"Marijan D.","year":"2019","unstructured":"D. Marijan, A. Gotlieb, and M. K. Ahuja. 2019. Challenges of testing machine learning based systems. In 2019 IEEE International Conference on Artificial Intelligence Testing (AITest \u201919). IEEE, 101\u2013102."},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","unstructured":"S. Ackerman O. Raz M. Zalmanovici and A. Zlotnick. 2021. Automatically detecting data drift in machine learning classifiers. DOI: 10.48550\/arXiv.2111.05672","DOI":"10.48550\/arXiv.2111.05672"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106665"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2901751"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477314.3507123"},{"key":"e_1_3_2_72_2","unstructured":"Y. Zhao. 2020. Mlops and data versioning in machine learning project."},{"key":"e_1_3_2_73_2","first-page":"42\u201351","volume-title":"32nd Annual International Conference on Computer Science and Software Engineering","author":"Njomou A. T.","year":"2022","unstructured":"A. T. Njomou, M. Fokaefs, D. F. Silatchom Kamga, and B. Adams. 2022. On the challenges of migrating to machine learning life cycle management platforms. In 32nd Annual International Conference on Computer Science and Software Engineering, ser. (CASCON \u201922). IBM Corp., 42\u201351."},{"key":"e_1_3_2_74_2","first-page":"750","volume-title":"28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Chen Z.","year":"2020","unstructured":"Z. Chen, Y. Cao, Y. Liu, H. Wang, T. Xie, and X. Liu. 2020. A comprehensive study on challenges in deploying deep learning based software. In 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 750\u2013762."},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111047"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2022.111515"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539783"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","unstructured":"S. Cao X. Sun R. Widyasari D. Lo X. Wu L. Bo J. Zhang B. Li W. Liu and D. Wu. 2024. A systematic literature review on explainability for machine\/deep learning-based software engineering research. DOI: 10.48550\/arXiv.2401.14617","DOI":"10.48550\/arXiv.2401.14617"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3173346"},{"key":"e_1_3_2_80_2","first-page":"1","volume-title":"4th International Workshop on Data Management for End-to-End Machine Learning","author":"Chen A.","year":"2020","unstructured":"A. Chen, A. Chow, A. Davidson, A. DCunha, A. Ghodsi, S. A. Hong, A. Konwinski, C. Mewald, S. Murching, and T. Nykodym. 2020. Developments in mlflow: A system to accelerate the machine learning lifecycle. In 4th International Workshop on Data Management for End-to-End Machine Learning, 1\u20134."},{"key":"e_1_3_2_81_2","first-page":"91","volume-title":"2020 IEEE 24th International Enterprise Distributed Object Computing Workshop (EDOCW \u201920)","author":"Weber C.","year":"2020","unstructured":"C. Weber and P. Reimann. 2020. Mmp - a platform to manage machine learning models in industry 4.0 environments. In 2020 IEEE 24th International Enterprise Distributed Object Computing Workshop (EDOCW \u201920), 91\u201394."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549088"},{"key":"e_1_3_2_83_2","first-page":"9","volume-title":"1st ACM International Workshop on AI and Software Testing\/Analysis","author":"Sztwiertnia S.","year":"2021","unstructured":"S. Sztwiertnia, M. Gr\u00fcbel, A. Chouchane, D. Sokolowski, K. Narasimhan, and M. Mezini. 2021. Impact of programming languages on machine learning bugs. In 1st ACM International Workshop on AI and Software Testing\/Analysis, 9\u201312."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3688841","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3688841","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:10Z","timestamp":1750291450000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3688841"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,28]]},"references-count":82,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1,31]]}},"alternative-id":["10.1145\/3688841"],"URL":"https:\/\/doi.org\/10.1145\/3688841","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,28]]},"assertion":[{"value":"2024-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-30","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}