{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:25:50Z","timestamp":1783383950032,"version":"3.54.6"},"reference-count":77,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2024,5,29]],"date-time":"2024-05-29T00:00:00Z","timestamp":1716940800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,29]],"date-time":"2024-05-29T00:00:00Z","timestamp":1716940800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004063","name":"Knut och Alice Wallenbergs Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004063","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine Learning (ML) experiment management tools support ML practitioners and software engineers when building intelligent software systems. By managing large numbers of ML experiments comprising many different ML assets, they not only facilitate engineering ML models and ML-enabled systems, but also managing their evolution\u2014for instance, tracing system behavior to concrete experiments when the model performance drifts. However, while ML experiment management tools have become increasingly popular, little is known about their effectiveness in practice, as well as their actual benefits and challenges. We present a mixed-methods empirical study of experiment management tools and the support they provide to users. First, our survey of 81 ML practitioners sought to determine the benefits and challenges of ML experiment management and of the existing tool landscape. Second, a controlled experiment with 15 student developers investigated the effectiveness of ML experiment management tools. We learned that 70% of our survey respondents perform ML experiments using specialized tools, while out of those who do not use such tools, 52% are unaware of experiment management tools or of their benefits. The controlled experiment showed that experiment management tools offer valuable support to users to systematically track and retrieve ML assets. Using ML experiment management tools reduced error rates and increased completion rates. By presenting a user\u2019s perspective on experiment management tools, and the first controlled experiment in this area, we hope that our results foster the adoption of these tools in practice, as well as they direct tool builders and researchers to improve the tool landscape overall.<\/jats:p>","DOI":"10.1007\/s10664-024-10444-w","type":"journal-article","created":{"date-parts":[[2024,5,29]],"date-time":"2024-05-29T07:02:18Z","timestamp":1716966138000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Machine learning experiment management tools: a mixed-methods empirical study"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4143-322X","authenticated-orcid":false,"given":"Samuel","family":"Idowu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Osman","family":"Osman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Str\u00fcber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thorsten","family":"Berger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,5,29]]},"reference":[{"key":"10444_CR1","doi-asserted-by":"crossref","unstructured":"Alberti M, Pondenkandath V, Wursch M, Ingold R, Liwicki M (2018) DeepDIVA: a highly-functional python framework for reproducible experiments. ICFHR, pp 423\u2013428","DOI":"10.1109\/ICFHR-2018.2018.00080"},{"key":"10444_CR2","unstructured":"Amazon SageMaker. Available: https:\/\/aws.amazon.com\/sagemaker\/"},{"key":"10444_CR3","doi-asserted-by":"crossref","unstructured":"Amershi S, Begel A, Bird C, DeLine R, Gall H, Kamar E, Nagappan N, Nushi B, Zimmermann T (2019) Software engineering for machine learning: a case study. In: International conference on software engineering: software engineering in practice (ICSE-SEIP). IEEE, pp 291\u2013300","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"10444_CR4","unstructured":"Appendix (2022). Available: https:\/\/github.com\/isselab\/2024-appendix-ml_exp_mgmt_study"},{"issue":"2","key":"10444_CR5","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1109\/TSE.2007.17","volume":"33","author":"E Arisholm","year":"2007","unstructured":"Arisholm E, Gallis H, Dyba T, Sjoberg DI (2007) Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Trans Software Eng 33(2):65\u201386","journal-title":"IEEE Trans Software Eng"},{"key":"10444_CR6","doi-asserted-by":"crossref","unstructured":"Arpteg A, Brinne B, Crnkovic-Friis L, Bosch J (2018) Software engineering challenges of deep learning. In SEAA","DOI":"10.1109\/SEAA.2018.00018"},{"key":"10444_CR7","unstructured":"Azure ai | microsoft cloud (2022). Available: https:\/\/azure.microsoft.com\/"},{"key":"10444_CR8","unstructured":"Berg G (2022) Image classification with machine learning as a service:-a comparison between azure, sagemaker, and vertex ai"},{"key":"10444_CR9","doi-asserted-by":"crossref","unstructured":"Berger T, V\u00f6lter M, Jensen HP, Dangprasert T, Siegmund J (2016) Efficiency of projectional editing: a controlled experiment. In: FSE, pp 763\u2013774","DOI":"10.1145\/2950290.2950315"},{"key":"10444_CR10","doi-asserted-by":"crossref","unstructured":"Bosch J (2022) Introduction to the ai engineering theme. Accelerating Digital Transformation: 10 Years of Software Center, p 399","DOI":"10.1007\/978-3-031-10873-0_17"},{"issue":"6","key":"10444_CR11","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1109\/MS.2022.3199621","volume":"39","author":"J Bosch","year":"2022","unstructured":"Bosch J, Olsson HH, Brinne B, Crnkovic I (2022) AI engineering: realizing the potential of AI. IEEE Softw 39(6):23\u201327","journal-title":"IEEE Softw"},{"key":"10444_CR12","volume-title":"Survey of machine-learning experimental methods at neurips2019 and iclr2020","author":"X Bouthillier","year":"2020","unstructured":"Bouthillier X, Varoquaux G (2020) Survey of machine-learning experimental methods at neurips2019 and iclr2020. Tech, Rep"},{"key":"10444_CR13","doi-asserted-by":"crossref","unstructured":"Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. In: HealthCom, pp 239\u2013249","DOI":"10.1007\/978-3-540-45143-3_6"},{"key":"10444_CR14","unstructured":"Control DV (2023) What is dvc?. Available: https:\/\/dvc.org\/doc\/user-guide\/what-is-dvc"},{"key":"10444_CR15","doi-asserted-by":"crossref","unstructured":"Counsell S (2008) Do student developers differ from industrial developers?. In: ITI, pp 477\u2013482","DOI":"10.1109\/ITI.2008.4588457"},{"key":"10444_CR16","first-page":"15","volume":"2469","author":"DN da Silva","year":"2019","unstructured":"da Silva DN, Sim\u00f5es A, Cardoso C, de Oliveira DE, Rittmeyer JN, Wehmuth K, Lustosa H, Pereira RS, Souto Y, Vignoli LE, Salles R, de Heleno SC, Ziviani A, Ogasawara E, Delicato FC, de Pires PF, da Pinto HLC, Maia L, Porto F (2019) A conceptual vision toward the management of machine learning models. In CEUR Workshop Proceedings 2469:15\u201327","journal-title":"In CEUR Workshop Proceedings"},{"key":"10444_CR17","unstructured":"DVC (2021) Dvc. https:\/\/dvc.org\/"},{"key":"10444_CR18","unstructured":"Dvc extension for visual studio code (2022). Available: https:\/\/marketplace.visualstudio.com\/items?itemName=Iterative.dvc"},{"key":"10444_CR19","doi-asserted-by":"crossref","unstructured":"Falessi D, Juristo N, Wohlin C, Turhan B, M\u00fcnch J, Jedlitschka A, Oivo M (2018) Empirical software engineering experts on the use of students and professionals in experiments. ESE, pp 452\u2013489","DOI":"10.1007\/s10664-017-9523-3"},{"key":"10444_CR20","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1145\/240455.240464","volume":"39","author":"U Fayyad","year":"1996","unstructured":"Fayyad U, Piatetsky-Shapiro G, Smyth P (1996) The KDD process for extracting useful knowledge from volumes of data. Commun ACM 39:27\u201334","journal-title":"Commun ACM"},{"key":"10444_CR21","doi-asserted-by":"publisher","first-page":"100551","DOI":"10.1016\/j.softx.2020.100551","volume":"12","author":"R Ferenc","year":"2020","unstructured":"Ferenc R, Viszkok T, Aladics T, J\u00e1sz J, Heged\u00fcs P (2020) Deep-water framework: the Swiss army knife of humans working with machine learning models. SoftwareX 12:100551","journal-title":"SoftwareX"},{"key":"10444_CR22","doi-asserted-by":"crossref","unstructured":"Gharibi G, Walunj V, Rella S, Lee Y (2019) ModelKB: towards automated management of the modeling lifecycle in deep learning. RAISE, pp 28\u201334","DOI":"10.1109\/RAISE.2019.00013"},{"issue":"1","key":"10444_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10664-021-10057-7","volume":"27","author":"NE Gold","year":"2022","unstructured":"Gold NE, Krinke J (2022) Ethics in the mining of software repositories. Empir Softw Eng 27(1):1\u201349","journal-title":"Empir Softw Eng"},{"key":"10444_CR24","doi-asserted-by":"crossref","unstructured":"Hill C, Bellamy R, Erickson T, Burnett M (2016) Trials and tribulations of developers of intelligent systems: a field study. In VL\/HCC, pp 162\u2013170","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"10444_CR25","doi-asserted-by":"crossref","unstructured":"Hohman F, Wongsuphasawat K, Kery MB, Patel K (2020) Understanding and visualizing data iteration in machine learning. In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp 1\u201313","DOI":"10.1145\/3313831.3376177"},{"issue":"3","key":"10444_CR26","first-page":"201","volume":"5","author":"M H\u00f6st","year":"2000","unstructured":"H\u00f6st M, Regnell B, Wohlin C (2000) Using students as subjects-a comparative study of students and professionals in lead-time impact assessment. ESE 5(3):201\u2013214","journal-title":"ESE"},{"key":"10444_CR27","doi-asserted-by":"crossref","unstructured":"Idowu S, Sens Y, Berger T, Krueger J, Vierhauser M (2024) A large-scale study of ml-related python projects. In: 39th ACM\/SIGAPP symposium on applied computing (SAC)","DOI":"10.1145\/3605098.3636056"},{"key":"10444_CR28","doi-asserted-by":"crossref","unstructured":"Idowu S, Str\u00fcber D, Berger T (2021) Asset management in machine learning: a survey. In: ICSE-SEIP. IEEE, pp 51\u201360","DOI":"10.1109\/ICSE-SEIP52600.2021.00014"},{"key":"10444_CR29","doi-asserted-by":"crossref","unstructured":"Idowu S, Str\u00fcber D, Berger T (2022a) Asset management in machine learning: state-of-research and state-of-practice. ACM Computing Surveys (CSUR)","DOI":"10.1145\/3543847"},{"key":"10444_CR30","doi-asserted-by":"crossref","unstructured":"Idowu S, Strueber D, Berger T (2022b) Emmm: a unified meta-model for tracking machine learning experiments. In: Euromicro conference on software engineering and advanced applications (SEAA)","DOI":"10.1109\/SEAA56994.2022.00016"},{"key":"10444_CR31","doi-asserted-by":"crossref","unstructured":"Isdahl R, Gundersen OE (2019) Out-of-the-box reproducibility: a survey of machine learning platforms. In: eScience. IEEE","DOI":"10.1109\/eScience.2019.00017"},{"key":"10444_CR32","first-page":"188","volume":"171","author":"P Janardhanan","year":"2020","unstructured":"Janardhanan P (2020) Project repositories for machine learning with tensorflow. Procedia CS 171:188\u2013196","journal-title":"Procedia CS"},{"issue":"6245","key":"10444_CR33","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1126\/science.aaa8415","volume":"349","author":"MI Jordan","year":"2015","unstructured":"Jordan MI, Mitchell TM (2015) Machine learning: trends, perspectives, and prospects. Science 349(6245):255\u2013260","journal-title":"Science"},{"issue":"5","key":"10444_CR34","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1109\/MS.2018.3571224","volume":"35","author":"F Khomh","year":"2018","unstructured":"Khomh F, Adams B, Cheng J, Fokaefs M, Antoniol G (2018) Software engineering for machine-learning applications: the road ahead. IEEE Softw 35(5):81\u201384","journal-title":"IEEE Softw"},{"key":"10444_CR35","doi-asserted-by":"publisher","first-page":"463","DOI":"10.3233\/IDT-190160","volume":"13","author":"F Kumeno","year":"2020","unstructured":"Kumeno F (2020) Sofware engineering challenges for machine learning applications: a literature review. Intelligent Decision Technologies 13:463\u2013476","journal-title":"Intelligent Decision Technologies"},{"key":"10444_CR36","doi-asserted-by":"crossref","unstructured":"Lewis GA, Bellomo S, Ozkaya I (2021) Characterizing and detecting mismatch in machine-learning-enabled systems. In 2021 IEEE\/ACM 1st workshop on AI engineering-software engineering for AI (WAIN). IEEE, pp 133\u2013140","DOI":"10.1109\/WAIN52551.2021.00028"},{"issue":"4","key":"10444_CR37","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1177\/2168479017739269","volume":"52","author":"KJ Lui","year":"2018","unstructured":"Lui KJ (2018) Sample size determination for a 3-treatment 3-period crossover trial in frequency data. Therapeutic innovation & regulatory science 52(4):407\u2013415","journal-title":"Therapeutic innovation & regulatory science"},{"key":"10444_CR38","unstructured":"Microsoft (2017) Team Data Science Process Documentation. Available: https:\/\/docs.microsoft.com\/en-us\/azure\/machine-learning\/team-data-science-process\/"},{"key":"10444_CR39","doi-asserted-by":"publisher","unstructured":"Miotto R, Wang F, Wang S, Jiang X, Dudley JT (2017) Deep learning for healthcare: review, opportunities and challenges. Briefings in Bioinformatics 19(6):1236\u20131246. Available: https:\/\/doi.org\/10.1093\/bib\/bbx044","DOI":"10.1093\/bib\/bbx044"},{"key":"10444_CR40","unstructured":"MLflow (2021) Mlflow. https:\/\/mlflow.org\/"},{"key":"10444_CR41","unstructured":"Ml-Tooling (2023) Ml-tooling\/best-of-ml-python: a ranked list of awesome machine learning python libraries. updated weekly. Available: https:\/\/github.com\/ml-tooling\/best-of-ml-python"},{"key":"10444_CR42","unstructured":"Most popular machine learning libraries 2014\/2021. Available: https:\/\/statisticsanddata.org\/data\/most-popular-machine-learning-libraries"},{"key":"10444_CR43","doi-asserted-by":"crossref","unstructured":"Nahar N, Zhou S, Lewis G, K\u00e4stner C (2022) Collaboration challenges in building ml-enabled systems: communication, documentation, engineering, and process. In: Proceedings of the 44th international conference on software engineering, pp 413\u2013425","DOI":"10.1145\/3510003.3510209"},{"key":"10444_CR44","doi-asserted-by":"crossref","unstructured":"Namaki MH, Floratou A, Psallidas F, Krishnan S, Agrawal A, Wu Y (2020) Vamsa: tracking provenance in data science scripts","DOI":"10.1145\/3394486.3403205"},{"key":"10444_CR45","doi-asserted-by":"crossref","unstructured":"Nayak A, Dutta K (2017) Impacts of machine learning and artificial intelligence on mankind. In: 2017 international conference on intelligent computing and control (I2C2), 2017, pp 1\u20133","DOI":"10.1109\/I2C2.2017.8321908"},{"key":"10444_CR46","doi-asserted-by":"publisher","first-page":"111860","DOI":"10.1016\/j.jss.2023.111860","volume":"207","author":"R Nazir","year":"2024","unstructured":"Nazir R, Bucaioni A, Pelliccione P (2024) Architecting ml-enabled systems: challenges, best practices, and design decisions. J Syst Softw 207:111860","journal-title":"J Syst Softw"},{"key":"10444_CR47","unstructured":"Neptune (2021) Neptune.ai. https:\/\/neptune.ai\/"},{"key":"10444_CR48","unstructured":"Ormenisan AA, Ismail M, Haridi S, Dowling J (2020) Implicit Provenance for Machine Learning Artifacts. MLSys\u201920, p\u00a03"},{"key":"10444_CR49","unstructured":"Polyaxon-machine learning at scale. Available: https:\/\/polyaxon.com\/"},{"key":"10444_CR50","unstructured":"Quaranta L, Calefato F, Lanubile F (2021) A taxonomy of tools for reproducible machine learning experiments"},{"key":"10444_CR51","unstructured":"Raschka S, Mirjalili V (2019) Python machine learning: machine learning and deep learning with Python, scikit-learn, and TensorFlow 2. Packt Publishing Ltd"},{"key":"10444_CR52","doi-asserted-by":"crossref","unstructured":"Rashidi HH, Tran N, Albahra S, Dang LT (2021) Machine learning in health care and laboratory medicine: general overview of supervised learning and Auto-ML. International Journal of Laboratory Hematology, vol\u00a043, no\u00a0S1, pp 15\u201322. Available: https:\/\/onlinelibrary.wiley.com\/doi\/abs\/10.1111\/ijlh.13537","DOI":"10.1111\/ijlh.13537"},{"key":"10444_CR53","unstructured":"Runeson P (2003) Using students as experiment subjects\u2013an analysis on graduate and freshmen student data. In: EASE, pp 95\u2013102"},{"key":"10444_CR54","first-page":"666","volume":"1","author":"I Salman","year":"2015","unstructured":"Salman I, Misirli AT, Juristo N (2015) Are students representatives of professionals in software engineering experiments? In ICSE 1:666\u2013676","journal-title":"In ICSE"},{"key":"10444_CR55","first-page":"55","volume":"9","author":"IH Sarker","year":"2015","unstructured":"Sarker IH, Faruque F, Hossen U, Rahman A (2015) A survey of software development process models in software engineering. IJSEA 9:55\u201370","journal-title":"IJSEA"},{"key":"10444_CR56","first-page":"3","volume":"2018","author":"S Schelter","year":"2018","unstructured":"Schelter S, B\u00f6se JH, Kirschnick J, Klein T, Seufert S (2018) Declarative metadata management: a missing piece in end-to-end machine learning. SysML 2018:3","journal-title":"SysML"},{"key":"10444_CR57","unstructured":"Schlegel M, Sattler KU (2022) Management of machine learning lifecycle artifacts: a survey. arXiv:2210.11831"},{"key":"10444_CR58","unstructured":"Scikit Learn (2021) Datasets: Boston and diabetes. https:\/\/scikit-learn.org\/stable\/datasets\/toy_dataset, california, https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.datasets.fetch_california_housing.html"},{"key":"10444_CR59","first-page":"2503","volume":"28","author":"D Sculley","year":"2015","unstructured":"Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M, Crespo JF, Dennison D (2015) Hidden technical debt in machine learning systems. NIPS 28:2503\u20132511","journal-title":"NIPS"},{"key":"10444_CR60","doi-asserted-by":"crossref","unstructured":"Serban A, van\u00a0der Blom K, Hoos H, Visser J (2020) Adoption and effects of software engineering best practices in machine learning. ESEM","DOI":"10.1145\/3382494.3410681"},{"key":"10444_CR61","doi-asserted-by":"crossref","unstructured":"Sharma R, Kamble SS, Gunasekaran A, Kumar V, Kumar A (2020) A systematic literature review on machine learning applications for sustainable agriculture supply chain performance. Computers & Operations Research 119:104926. Available: https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0305054820300435","DOI":"10.1016\/j.cor.2020.104926"},{"key":"10444_CR62","doi-asserted-by":"crossref","unstructured":"Siegmund J, Siegmund N, Apel S (2015) Views on internal and external validity in empirical software engineering. In: 2015 IEEE\/ACM 37th IEEE international conference on software engineering. vol.\u00a01. IEEE, pp 9\u201319","DOI":"10.1109\/ICSE.2015.24"},{"key":"10444_CR63","unstructured":"Tsay J, Mummert T, Bobroff N, Braz A, Westerink P (2018) Runway: machine learning model experiment management tool. SysML, pp. 1\u20133"},{"key":"10444_CR64","doi-asserted-by":"crossref","unstructured":"Tuggener L, Amirian M, Rombach K, L\u00f6rwald S, Varlet A, Westermann C, Stadelmann T (2019) Automated machine learning in practice: State of the art and recent results. In: 2019 6th Swiss Conference on Data Science (SDS). pp 31\u201336","DOI":"10.1109\/SDS.2019.00-11"},{"key":"10444_CR65","doi-asserted-by":"crossref","unstructured":"Turner JR (2013) Crossover Design, New York, pp 521","DOI":"10.1007\/978-1-4419-1005-9_1009"},{"key":"10444_CR66","unstructured":"Vartak M, Subramanyam H, Lee WEE, Viswanathan S, Husnoo S, Madden S, Zaharia M (2016) ModelDB: a system for machine learning model management. In the Workshop. ACM Press, pp. 1\u20133"},{"key":"10444_CR67","unstructured":"Vertex ai | google cloud (2022). Available: https:\/\/cloud.google.com\/vertex-ai"},{"key":"10444_CR68","unstructured":"Visengeriyeva L, Kammer A, B\u00e4r I, Pl\u00f6d A (2021) ml-ops.org. Available: https:\/\/ml-ops.org\/content\/end-to-end-ml-workflow"},{"key":"10444_CR69","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1109\/MNET.2017.1700200","volume":"32","author":"M Wang","year":"2017","unstructured":"Wang M, Cui Y, Wang X, Xiao S, Jiang J (2017) Machine learning for networking: workflow, advances and opportunities. IEEE Network 32:92\u201399","journal-title":"IEEE Network"},{"key":"10444_CR70","doi-asserted-by":"crossref","unstructured":"Waring J, Lindvall C, Umeton R (2020) Automated machine learning: review of the state-of-the-art and opportunities for healthcare. Artificial Intelligence in Medicine, vol 104, pp 101822. Available: https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0933365719310437","DOI":"10.1016\/j.artmed.2020.101822"},{"key":"10444_CR71","doi-asserted-by":"crossref","unstructured":"Weber T, Hu\u00dfmann H (2022) Tooling for developing data-driven applications: overview and outlook. Proceedings of Mensch und Computer 2022:66\u201377","DOI":"10.1145\/3543758.3543779"},{"key":"10444_CR72","unstructured":"Wels S (2012) Test driven development. In: Proceedings of Agile Seminar 2012"},{"key":"10444_CR73","first-page":"29","volume":"24959","author":"R Wirth","year":"2000","unstructured":"Wirth R (2000) CRISP-DM: towards a standard process model for data mining. ICKDDM, 24959:29\u201339","journal-title":"ICKDDM,"},{"issue":"1","key":"10444_CR74","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1080\/21693277.2016.1192517","volume":"4","author":"T Wuest","year":"2016","unstructured":"Wuest T, Weimer D, Irgens C, Thoben KD (2016) Machine learning in manufacturing: advantages, challenges, and applications. Production & Manufacturing Research 4(1):23\u201345","journal-title":"Production & Manufacturing Research"},{"key":"10444_CR75","doi-asserted-by":"publisher","unstructured":"Xin D, Ma L, Liu J, Macke S, Song S, Parameswaran A (2018) Accelerating human-in-the-loop machine learning: challenges and opportunities. In: Proceedings of the second workshop on data management for end-to-end machine learning, ser. DEEM\u201918. New York, USA: Association for Computing Machinery. Available: https:\/\/doi.org\/10.1145\/3209889.3209897","DOI":"10.1145\/3209889.3209897"},{"issue":"4","key":"10444_CR76","first-page":"39","volume":"41","author":"M Zaharia","year":"2018","unstructured":"Zaharia M, Chen A, Davidson A, Ghodsi A, Hong SA, Konwinski A, Murching S, Nykodym T, Ogilvie P, Parkhe M et al (2018) Accelerating the machine learning lifecycle with mlflow. IEEE Data Eng. Bull 41(4):39\u201345","journal-title":"IEEE Data Eng. Bull"},{"key":"10444_CR77","doi-asserted-by":"crossref","unstructured":"Zhang D, Shen Y, Huang Z, Xie X (2022) Auto machine learning-based modelling and prediction of excavation-induced tunnel displacement. Journal of Rock Mechanics and Geotechnical Engineering, vol\u00a014, no\u00a04, pp 1100\u20131114. Available: https:\/\/www.sciencedirect.com\/science\/article\/pii\/S1674775522000786","DOI":"10.1016\/j.jrmge.2022.03.005"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10444-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10444-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10444-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,5]],"date-time":"2024-07-05T15:12:30Z","timestamp":1720192350000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10444-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,29]]},"references-count":77,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["10444"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10444-w","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,29]]},"assertion":[{"value":"3 January 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 May 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declared that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"74"}}