{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T16:28:15Z","timestamp":1784996895567,"version":"3.55.0"},"reference-count":89,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>Machine learning components are essential for today\u2019s software systems, causing a need to adapt traditional software engineering practices when developing machine-learning-based systems. This need is pronounced due to many development-related challenges of machine learning components such as asset, experiment, and dependency management. Recently, many asset management tools addressing these challenges have become available. It is essential to understand the support such tools offer to facilitate research and practice on building new management tools with native supports for machine learning and software engineering assets.<\/jats:p>\n          <jats:p>This article positions machine learning asset management as a discipline that provides improved methods and tools for performing operations on machine learning assets. We present a feature-based survey of 18 state-of-practice and 12 state-of-research tools supporting machine-learning asset management. We overview their features for managing the types of assets used in machine learning experiments. Most state-of-research tools focus on tracking, exploring, and retrieving assets to address development concerns such as reproducibility, while the state-of-practice tools also offer collaboration and workflow-execution-related operations. In addition, assets are primarily tracked intrusively from the source code through APIs and managed via web dashboards or command-line interfaces (CLIs). We identify asynchronous collaboration and asset reusability as directions for new tools and techniques.<\/jats:p>","DOI":"10.1145\/3543847","type":"journal-article","created":{"date-parts":[[2022,6,16]],"date-time":"2022-06-16T10:05:58Z","timestamp":1655373958000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":38,"title":["Asset Management in Machine Learning: State-of-research and State-of-practice"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4143-322X","authenticated-orcid":false,"given":"Samuel","family":"Idowu","sequence":"first","affiliation":[{"name":"Chalmers | University of Gothenburg, G\u00f6teborg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5969-3521","authenticated-orcid":false,"given":"Daniel","family":"Str\u00fcber","sequence":"additional","affiliation":[{"name":"Radboud University Nijmegen, GL Nijmegen, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3870-5167","authenticated-orcid":false,"given":"Thorsten","family":"Berger","sequence":"additional","affiliation":[{"name":"Chalmers | University of Gothenburg, G\u00f6teborg, Sweden and Ruhr University Bochum, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,15]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2018.00049"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICFHR-2018.2018.00080"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.5555\/2541773"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3412841.3442046"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/SEAA.2018.00018"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2019.20866"},{"key":"e_1_3_2_9_2","volume-title":"Deep Learning","author":"Bengio Yoshua","year":"2017","unstructured":"Yoshua Bengio, Ian Goodfellow, and Aaron Courville. 2017. Deep Learning. MIT Press Massachusetts."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/2791060.2791108"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09787-6"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.11365"},{"key":"e_1_3_2_13_2","doi-asserted-by":"crossref","unstructured":"Sumon Biswas Mohammad Wardat and Hridesh Rajan. 2021. The art and practice of data science pipelines: A comprehensive study of data science pipelines in theory in-the-small and in-the-large. In IEEE\/ACM 44th International Conference on Software Engineering (ICSE\u201922) . IEEE 2091\u20132103.","DOI":"10.1145\/3510003.3510057"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3143021"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015338"},{"key":"e_1_3_2_16_2","volume-title":"Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020","author":"Bouthillier Xavier","year":"2020","unstructured":"Xavier Bouthillier and Ga\u00ebl Varoquaux. 2020. Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020. Technical Report."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1613\/JAIR.1.12228"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics8080832"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2019.00038"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376729"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1147\/sj.453.0621"},{"key":"e_1_3_2_22_2","first-page":"15","volume-title":"Proceedings of the CEUR Workshop.","author":"Silva Daniel N. R. da","year":"2019","unstructured":"Daniel N. R. da Silva, Adolfo Sim\u00f5es, Carlos Cardoso, Douglas E. M. de Oliveira, Jo\u00e3o N. Rittmeyer, Klaus Wehmuth, Hermano Lustosa, Rafael S. Pereira, Yania Souto, Luciana E. G. Vignoli, Rebecca Salles, S. C. de Heleno, Artur Ziviani, Eduardo Ogasawara, Flavia C. Delicato, Paulo F. de Pires, Hardy Leonardo C. P. da Pinto, Luciano Maia, and Fabio Porto. 2019. A conceptual vision toward the management of machine learning models. In Proceedings of the CEUR Workshop.15\u201327. Retrieved from http:\/\/www.master.iag.usp.br\/lab\/."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.558708"},{"key":"e_1_3_2_24_2","unstructured":"Mihail Dimitrov and Ibrahim Osman. 2012. The Impact of Cloud Computing on Organizations in Regard to Cost and Security. In Informatik Student Paper Master (INFSPM\u201912) . Ume\u00e5 University Department of Informatics. 28 pages."},{"key":"e_1_3_2_25_2","unstructured":"Chris Drummond. 2009. Replicability is not reproducibility: nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML Vol. 1. National Research Council of Canada Montreal Canada."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-02654-1_11"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/240455.240464"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.softx.2020.100551"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3329486.3329495"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/RAISE.2019.00013"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336294.3336321"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_2_33_2","volume-title":"Practical Machine Learning","author":"Gollapudi S.","year":"2016","unstructured":"S. Gollapudi. 2016. Practical Machine Learning. Packt Publishing."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387473"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/SECSE.2009.5069155"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/FOSM.2008.4659248"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-62136-0_17"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4842-3933-9"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/IC2E.2019.00025"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/PerComW.2014.6815206"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP55303.2022.9794036"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/SmartGridComm.2014.7007705"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.enbuild.2016.09.068"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00014"},{"key":"e_1_3_2_46_2","volume-title":"Proceedings of the Euromicro Conference on Software Engineering and Advanced Applications","author":"Idowu Samuel","year":"2022","unstructured":"Samuel Idowu, Daniel Str\u00fcber, and Thorsten Berger. 2022. EMMM: A unified meta-model for tracking machine learning experiments. In Proceedings of the Euromicro Conference on Software Engineering and Advanced Applications."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/escience.2019.00017"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/IEEESTD.2010.5551093"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA235785"},{"key":"e_1_3_2_50_2","unstructured":"B. Kitchenham and S. Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. Technical report ver. 2.3 ebse technical report. ebse. Retrieved from http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.117.471."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.3233\/978-1-61499-649-1-87"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.3233\/idt-190160"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3426428.3426922"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3136040.3136054"},{"key":"e_1_3_2_55_2","unstructured":"Hui Miao Ang Li Larry S. Davis and Amol Deshpande. 2016. ModelHub : Lifecycle management for deep learning. CoRR abs\/1611.06224. http:\/\/arxiv.org\/abs\/1611.06224."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.112"},{"key":"e_1_3_2_57_2","unstructured":"Microsoft. 2017. Team Data Science Process Documentation. Retrieved 23 Feb. 2021 from https:\/\/docs.microsoft.com\/en-us\/azure\/machine-learning\/team-data-science-process\/."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939502.2939516"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.3390\/bdcc5020020"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.3390\/bdcc5020020"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403205"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338974"},{"key":"e_1_3_2_64_2","volume-title":"Proceedings of the FSE","author":"Ne\u0161i\u0107 Damir","year":"2019","unstructured":"Damir Ne\u0161i\u0107, Jacob Kr\u00fcger, Stefan St\u0103nciulescu, and Thorsten Berger. 2019. Principles of feature modeling. In Proceedings of the FSE."},{"key":"e_1_3_2_65_2","unstructured":"Alexandru A. Ormenisan Mahmoud Ismail Seif Haridi and Jim Dowling. 2020. Implicit provenance for machine learning artifacts. In Proceedings of the Machine Learning and Systems (MLSys) Workshop vol. 20."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/icccbda.2018.8386577"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3001867.3001874"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2976199"},{"key":"e_1_3_2_69_2","first-page":"45","volume-title":"Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop","author":"P. Jagadeesh Chandra Bose R.","year":"2019","unstructured":"Jagadeesh Chandra Bose R. P., Kapil Singi, Vikrant Kaulgud, Kanchanjot Kaur Phokela, and Sanjay Podder. 2019. Framework for trustworthy software development. In Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop. IEEE, 45\u201348."},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.14257\/ijseia.2015.9.11.05"},{"key":"e_1_3_2_71_2","unstructured":"Sebastian Schelter Joos-Hendrik B\u00f6se Johannes Kirschnick Thoralf Klein and Stephan Seufert. 2017. Automatically tracking metadata and provenance of machine learning experiments. In Machine Learning Systems Workshop at NIPS . 27\u201329."},{"key":"e_1_3_2_72_2","unstructured":"Sebastian Schelter Joos-Hendrik B\u00f6se Johannes Kirschnick Thoralf Klein and Stephan Seufert. 2018. Declarative metadata management: A missing piece in end-to-end machine learning. In Proceedings of the Conference on Systems and Machine Learning (SysML\u201918) Vol. 18. 3 pages."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3382494.3410681"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3382494.3410680"},{"key":"e_1_3_2_75_2","first-page":"84","volume-title":"Proceedings of the CEUR Workshop Proceedings","author":"Souza Renan","year":"2018","unstructured":"Renan Souza, Liliane Neves, Leonardo Azeredo, Ricardo Luiz, Elaine Tady, Paulo Cavalin, and Marta Mattoso. 2018. Towards a human-in-the-loop library for tracking hyperparameter tuning in deep learning development. In Proceedings of the CEUR Workshop Proceedings. 84\u201387. Retrieved from https:\/\/github.com\/hpcdb\/DL-Steer."},{"key":"e_1_3_2_76_2","first-page":"351","volume-title":"Proceedings of the 2018 USENIX Annual Technical Conference","author":"Sridhar Vinay","year":"2018","unstructured":"Vinay Sridhar, Sriram Subramanian, Dulcardo Arteaga, Swaminathan Sundararaman, Drew Roselli, and Nisha Talagala. 2018. Model governance: Reducing the anarchy of production ML. In Proceedings of the 2018 USENIX Annual Technical Conference. 351\u2013358. Retrieved from https:\/\/www.usenix.org\/conference\/atc18\/presentation\/sridhar."},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3382025.3414960"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/1183236.1183260"},{"key":"e_1_3_2_79_2","article-title":"A practical taxonomy of reproducibility for machine learning research","author":"Tatman Rachael","year":"2018","unstructured":"Rachael Tatman, Jake Vanderplas, and Sohier Dane. 2018. A practical taxonomy of reproducibility for machine learning research. In Reproducibility in ML Workshop, ICML\u201918Ml (2018). Retrieved from https:\/\/hub.docker.com\/r\/pangwei\/tf1.1\/.","journal-title":"Reproducibility in ML Workshop, ICML\u201918"},{"key":"e_1_3_2_80_2","unstructured":"J. Tsay T. Mummert N. Bobroff A. Braz and P. Westerink. 2018. Runway: Machine learning model experiment management tool. In Inaugural Conference on Systems and Machine Learning (SysML\u201918) . 1\u20133."},{"key":"e_1_3_2_81_2","first-page":"1","volume-title":"Proceedings of the 2021 IEEE\/ACM 1st Workshop on AI Engineering-Software Engineering for AI","author":"Oort Bart van","year":"2021","unstructured":"Bart van Oort, Lu\u00eds Cruz, Maur\u00edcio Aniche, and Arie van Deursen. 2021. The prevalence of code smells in machine learning projects. In Proceedings of the 2021 IEEE\/ACM 1st Workshop on AI Engineering-Software Engineering for AI. IEEE, 1\u20138."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/2641190.2641198"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939502.2939516"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1515\/itit-2018-0022"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1.1.198.5133"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-018-2508-4"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1080\/21693277.2016.1192517"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209889.3209897"},{"issue":"1","key":"e_1_3_2_90_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3392826","article-title":"How do data science workers collaborate? Roles, workflows, and tools","volume":"4","author":"Zhang Amy X.","year":"2020","unstructured":"Amy X. Zhang, Michael Muller, and Dakuo Wang. 2020. How do data science workers collaborate? Roles, workflows, and tools. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1 (2020), 1\u201323.","journal-title":"Proceedings of the ACM on Human-Computer Interaction"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3543847","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3543847","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:49:39Z","timestamp":1750268979000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3543847"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,15]]},"references-count":89,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3543847"],"URL":"https:\/\/doi.org\/10.1145\/3543847","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,15]]},"assertion":[{"value":"2021-09-10","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-06-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}