{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T12:16:38Z","timestamp":1780488998313,"version":"3.54.1"},"reference-count":207,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T00:00:00Z","timestamp":1724112000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T00:00:00Z","timestamp":1724112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004351","name":"Cisco Systems","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004351","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2107230"],"award-info":[{"award-number":["2107230"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2229703"],"award-info":[{"award-number":["2229703"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2107020"],"award-info":[{"award-number":["2107020"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2104319"],"award-info":[{"award-number":["2104319"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Context<\/jats:title>\n                <jats:p>Many engineering organizations are reimplementing and extending deep neural networks from the research community. We describe this process as deep learning model reengineering. Deep learning model reengineering \u2014 reusing, replicating, adapting, and enhancing state-of-the-art deep learning approaches \u2014 is challenging for reasons including under-documented reference models, changing requirements, and the cost of implementation and testing.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Objective<\/jats:title>\n                <jats:p>Prior work has characterized the challenges of deep learning model development, but as yet we know little about the deep learning model reengineering process and its common challenges. Prior work has examined DL systems from a \u201cproduct\u201d view, examining defects from projects regardless of the engineers\u2019 purpose. Our study is focused on reengineering activities from a \u201cprocess\u201d view, and focuses on engineers specifically engaged in the reengineering process.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Method<\/jats:title>\n                <jats:p>Our goal is to understand the characteristics and challenges of deep learning model reengineering. We conducted a mixed-methods case study of this phenomenon, focusing on the context of computer vision. Our results draw from two data sources: defects reported in open-source reeengineering projects, and interviews conducted with practitioners and the leaders of a reengineering team. From the defect data source, we analyzed 348 defects from 27 open-source deep learning projects. Meanwhile, our reengineering team replicated 7 deep learning models over two years; we interviewed 2 open-source contributors, 4 practitioners, and 6 reengineering team leaders to understand their experiences.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Our results describe how deep learning-based computer vision techniques are reengineered, quantitatively analyze the distribution of defects in this process, and qualitatively discuss challenges and practices. We found that most defects (58%) are reported by re-users, and that reproducibility-related defects tend to be discovered during training (68% of them are). Our analysis shows that most environment defects (88%) are interface defects, and most environment defects (46%) are caused by API defects. We found that training defects have diverse symptoms and root causes. We identified four main challenges in the DL reengineering process: model operationalization, performance debugging, portability of DL operations, and customized data pipeline. Integrating our quantitative and qualitative data, we propose a novel reengineering workflow.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Our findings inform several conclusion, including: standardizing model reengineering practices, developing validation tools to support model reengineering, automated support beyond manual model reengineering, and measuring additional unknown aspects of model reengineering.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s10664-024-10521-0","type":"journal-article","created":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T08:02:25Z","timestamp":1724140945000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Challenges and practices of deep learning model reengineering: A case study on computer vision"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2608-8576","authenticated-orcid":false,"given":"Wenxin","family":"Jiang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6444-9806","authenticated-orcid":false,"given":"Vishnu","family":"Banna","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6740-8260","authenticated-orcid":false,"given":"Naveen","family":"Vivek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1827-1389","authenticated-orcid":false,"given":"Abhinav","family":"Goel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0413-4594","authenticated-orcid":false,"given":"Nicholas","family":"Synovic","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0452-5571","authenticated-orcid":false,"given":"George K.","family":"Thiruvathukal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2495-686X","authenticated-orcid":false,"given":"James C.","family":"Davis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,8,20]]},"reference":[{"key":"10521_CR1","unstructured":"ONNX (2019a) | Home. https:\/\/onnx.ai\/"},{"key":"10521_CR2","unstructured":"Portability between deep learning frameworks \u2013 with ONNX (2019b) https:\/\/blog.codecentric.de\/en\/2019\/08\/portability-deep-learning-frameworks-onnx\/"},{"key":"10521_CR3","unstructured":"Managing labels (2020) https:\/\/docs.github.com\/en\/issues\/using-labels-and-milestones-to-track-work\/managing-labels"},{"key":"10521_CR4","unstructured":"Papers with Code - ML Reproducibility Challenge 2021 Edition (2020) https:\/\/paperswithcode.com\/rc2021"},{"key":"10521_CR5","unstructured":"Being a Computer Vision Engineer in 2021 (2021) https:\/\/viso.ai\/computer-vision\/computer-vision-engineer\/"},{"key":"10521_CR6","unstructured":"Machine Learning Operations (2021) https:\/\/ml-ops.org\/"},{"key":"10521_CR7","unstructured":"TensorFlow (2021) https:\/\/www.tensorflow.org\/"},{"key":"10521_CR8","doi-asserted-by":"crossref","unstructured":"Abdullah M, Madain A, Jararweh Y (2022) Chatgpt: Fundamentals, applications and social impacts. In: 2022 Ninth International conference on social networks analysis, management and security (SNAMS), IEEE, pp 1\u20138","DOI":"10.1109\/SNAMS58071.2022.10062688"},{"key":"10521_CR9","doi-asserted-by":"crossref","unstructured":"Ait A, Izquierdo JLC, Cabot J (2023) Hfcommunity: A tool to analyze the hugging face hub community. In: 2023 IEEE International conference on software analysis, evolution and reengineering (SANER), IEEE, pp 728\u2013732","DOI":"10.1109\/SANER56733.2023.00080"},{"key":"10521_CR10","doi-asserted-by":"crossref","unstructured":"Alahmari SS, Goldgof DB, Mouton PR, Hall LO (2020) Challenges for the Repeatability of Deep Learning Models. IEEE Access","DOI":"10.1109\/ACCESS.2020.3039833"},{"key":"10521_CR11","unstructured":"AlDanial (2022) cloc. https:\/\/github.com\/AlDanial\/cloc"},{"key":"10521_CR12","doi-asserted-by":"crossref","unstructured":"Ali Q, Riganelli O, Mariani L (2024) Testing in the evolving world of dl systems: Insights from python github projects. arXiv:2405.19976","DOI":"10.1109\/QRS62785.2024.00013"},{"key":"10521_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-021-00444-8","volume":"8","author":"L Alzubaidi","year":"2021","unstructured":"Alzubaidi L, Zhang J, Humaidi AJ, Al-Dujaili A, Duan Y, Al-Shamma O, Santamar\u00eda J, Fadhel MA, Al-Amidie M, Farhan L (2021) Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions. J Big Data 8:1\u201374","journal-title":"J Big Data"},{"key":"10521_CR14","doi-asserted-by":"crossref","unstructured":"Amershi S, Begel A, Bird C, DeLine R, Gall H (2019) Software Engineering for Machine Learning: A Case Study. In: International conference on software engineering: software engineering in practice (ICSE-SEIP)","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"10521_CR15","doi-asserted-by":"crossref","unstructured":"Amusuo P, Sharma A, Rao SR, Vincent A, Davis JC (2022) Reflections on software failure analysis. In: ACM Joint european software engineering conference and symposium on the foundations of software engineering \u2014 Ideas, Visions, and Reflections track (ESEC\/FSE-IVR)","DOI":"10.1145\/3540250.3560879"},{"key":"10521_CR16","doi-asserted-by":"crossref","unstructured":"Anandayuvaraj D, Davis JC (2022) Reflecting on recurring failures in iot development. In: Proceedings of the 37th IEEE\/ACM International conference on automated software engineering, pp 1\u20135","DOI":"10.1145\/3551349.3559545"},{"key":"10521_CR17","doi-asserted-by":"crossref","unstructured":"Aranda J, Venolia G (2009) The secret life of bugs: Going past the errors and omissions in software repositories. In: International conference on software engineering (ICSE)","DOI":"10.1109\/ICSE.2009.5070530"},{"key":"10521_CR18","doi-asserted-by":"crossref","unstructured":"Arya D, Wang W, Guo JL, Cheng J (2019) Analysis and detection of information types of open source software issue discussions. In: 2019 IEEE\/ACM 41st International conference on software engineering (ICSE), IEEE, pp 454\u2013464","DOI":"10.1109\/ICSE.2019.00058"},{"key":"10521_CR19","unstructured":"Bahdanau D, Cho KH, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: International conference on learning representations (ICLR)"},{"key":"10521_CR20","unstructured":"Banna V, Chinnakotla A, Yan Z, Vegesana A, Vivek N, Krishnappa K, Jiang W, Lu YH, Thiruvathukal GK, Davis JC (2021) An experience report on machine learning reproducibility: Guidance for practitioners and TensorFlow model garden contributors. https:\/\/arxiv.org\/abs\/2107.00821"},{"key":"10521_CR21","doi-asserted-by":"crossref","unstructured":"Baysal O, Kononenko O, Holmes R, Godfrey MW (2012) The secret life of patches: A firefox case study. In: 2012 19th working conference on reverse engineering, IEEE, pp 447\u2013455","DOI":"10.1109\/WCRE.2012.54"},{"key":"10521_CR22","doi-asserted-by":"crossref","unstructured":"Bennett KH, Rajlich VT (2000) Software maintenance and evolution: a roadmap. In: Proceedings of the conference on the future of software engineering, pp 73\u201387","DOI":"10.1145\/336512.336534"},{"key":"10521_CR23","unstructured":"Berner C, Brockman G, Chan B, Cheung V, D\u0229biak P, Dennison C, Farhi D, Fischer Q, Hashme S, Hesse C, J\u00f3zefowicz R, Gray S, Olsson C, Pachocki J, Petrov M, Pinto HPdO, Raiman J, Salimans T, Schlatter J, Schneider J, Sidor S, Sutskever I, Tang J, Wolski F, Zhang S (2019) Dota 2 with Large Scale Deep Reinforcement Learning. arXiv:1912.06680"},{"issue":"3","key":"10521_CR24","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1007\/s10664-022-10282-8","volume":"28","author":"A Bhatia","year":"2023","unstructured":"Bhatia A, Eghan EE, Grichi M, Cavanagh WG, Jiang ZM, Adams B (2023) Towards a change taxonomy for machine learning pipelines: Empirical study of ml pipelines and forks related to academic publications. Empirical Softw Eng 28(3):60","journal-title":"Empirical Softw Eng"},{"issue":"2","key":"10521_CR25","doi-asserted-by":"publisher","first-page":"4492","DOI":"10.35940\/ijeat.B4563.129219","volume":"9","author":"K Bhavsar","year":"2020","unstructured":"Bhavsar K, Shah V, Gopalan S (2020) Machine learning: a software process reengineering in software development organization. Int J Eng Advanced Technol 9(2):4492\u20134500","journal-title":"Int J Eng Advanced Technol"},{"key":"10521_CR26","unstructured":"Bibal A, Fr\u00e9nay B (2016) Interpretability of Machine Learning Models and Representations: an Introduction. In: European symposium on artificial neural networks"},{"issue":"13","key":"10521_CR27","doi-asserted-by":"publisher","first-page":"1802","DOI":"10.1177\/1049732316654870","volume":"26","author":"L Birt","year":"2016","unstructured":"Birt L, Scott S, Cavers D, Campbell C, Walter F (2016) Member checking: a tool to enhance trustworthiness or merely a nod to validation? Qualitative Health Res 26(13):1802\u20131811","journal-title":"Qualitative Health Res"},{"key":"10521_CR28","doi-asserted-by":"crossref","unstructured":"Boehm B, Beck K (2010) The changing nature of software evolution; The inevitability of evolution. In: IEEE Software","DOI":"10.1109\/MS.2010.103"},{"key":"10521_CR29","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.09.016","author":"H Borges","year":"2018","unstructured":"Borges H, Valente MT (2018) What\u2019s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform In: Journal of systems and software (JSS). https:\/\/doi.org\/10.1016\/j.jss.2018.09.016","journal-title":"Understanding Repository Starring Practices in a Social Coding Platform. In: Journal of systems and software (JSS)"},{"key":"10521_CR30","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110542","volume":"164","author":"HB Braiek","year":"2020","unstructured":"Braiek HB, Khomh F (2020) On testing machine learning programs. J Syst Softw (JSS) 164:110542","journal-title":"J Syst Softw (JSS)"},{"key":"10521_CR31","doi-asserted-by":"publisher","unstructured":"Breck E, Cai S, Nielsen E, Salib M, Sculley D (2017) The ML test score: A rubric for ML production readiness and technical debt reduction. In: 2017 IEEE International conference on big data (big data), pp 1123\u20131132, https:\/\/doi.org\/10.1109\/BigData.2017.8258038","DOI":"10.1109\/BigData.2017.8258038"},{"key":"10521_CR32","unstructured":"Breck E, Polyzotis N, Roy S, Whang S, Zinkevich M (2019) Data Validation for Machine Learning. In: the Conference on machine learning and systems (MLSys)"},{"key":"10521_CR33","unstructured":"Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language Models are Few-Shot Learners. Tech Rep arXiv:2005.14165"},{"key":"10521_CR34","unstructured":"Bubeck S, Chandrasekaran V, Eldan R, Gehrke J, Horvitz E, Kamar E, Lee P, Lee YT, Li Y, Lundberg S et\u00a0al (2023) Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712"},{"key":"10521_CR35","unstructured":"Byrne E (1992) A conceptual foundation for software re-engineering. In: Conference on software maintenance"},{"key":"10521_CR36","doi-asserted-by":"crossref","unstructured":"Chang X, Dou W, Gao Y, Wang J, Wei J, Huang T (2019) Detecting atomicity violations for event-driven node. js applications. In: 2019 IEEE\/ACM 41st International conference on software engineering (ICSE), IEEE, pp 631\u2013642","DOI":"10.1109\/ICSE.2019.00073"},{"key":"10521_CR37","doi-asserted-by":"publisher","unstructured":"Chen B, Wen M, Shi Y, Lin D, Rajbahadur GK, Ming Z, Jiang (2022a) Towards Training Reproducible Deep Learning Models. In: International conference on software engineering (ICSE), pp 2202\u20132214, https:\/\/doi.org\/10.1145\/3510003.3510163","DOI":"10.1145\/3510003.3510163"},{"key":"10521_CR38","doi-asserted-by":"crossref","unstructured":"Chen C, Liu MY, Tuzel O, Xiao J (2017) R-cnn for small object detection. In: Computer Vision\u2013ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part V 13, Springer, pp 214\u2013230","DOI":"10.1007\/978-3-319-54193-8_14"},{"key":"10521_CR39","unstructured":"Chen J, Liang Y, Shen Q, Jiang J (2022b) Toward Understanding Deep Learning Framework Bugs. arXiv:2203.04026"},{"key":"10521_CR40","unstructured":"Chen K, Wang J, Pang J, Cao Y, Xiong Y, Li X, Sun S, Feng W, Liu Z, Xu J et\u00a0al (2019) Mmdetection: Open mmlab detection toolbox and benchmark. arXiv:1906.07155"},{"key":"10521_CR41","doi-asserted-by":"crossref","unstructured":"Cheng B, Misra I, Schwing AG, Kirillov A, Girdhar R (2022) Masked-attention Mask Transformer for Universal Image Segmentation. arXiv:2112.01527","DOI":"10.1109\/CVPR52688.2022.00135"},{"issue":"03","key":"10521_CR42","first-page":"1","volume":"62","author":"D Cohen","year":"2004","unstructured":"Cohen D, Lindvall M, Costa P (2004) An introduction to agile methods. Advanced Comput 62(03):1\u201366","journal-title":"Advanced Comput"},{"issue":"1","key":"10521_CR43","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychological Measurement 20(1):37\u201346","journal-title":"Educ Psychological Measurement"},{"key":"10521_CR44","doi-asserted-by":"crossref","unstructured":"Davis JC, Jajal P, Jiang W, Schorlemmer TR, Synovic N, Thiruvathukal GK (2023) Reusing deep learning models: Challenges and directions in software engineering. In: Proceedings of the IEEE john vincent atanasoff symposium on modern computing (JVA\u201923)","DOI":"10.1109\/JVA60410.2023.00015"},{"key":"10521_CR45","unstructured":"Devanbu P, Dwyer M, Elbaum S, Lowry M, Moran K, Poshyvanyk D, Ray B, Singh R, Zhang X (2020) Deep Learning & Software Engineering: State of Research and Future Directions. arXiv:2009.08525"},{"key":"10521_CR46","doi-asserted-by":"crossref","unstructured":"Dhanya V, Subeesh A, Kushwaha N, Vishwakarma DK, Kumar TN, Ritika G, Singh A (2022) Deep learning based computer vision approaches for smart agricultural applications. Artif Intell Agric","DOI":"10.1016\/j.aiia.2022.09.007"},{"key":"10521_CR47","unstructured":"Ding Z, Reddy A, Joshi A (2021) Reproducibility. https:\/\/blog.ml.cmu.edu\/2020\/08\/31\/5-reproducibility\/"},{"key":"10521_CR48","unstructured":"Doshi-Velez F, Kim B (2017) Towards A Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608"},{"key":"10521_CR49","doi-asserted-by":"crossref","unstructured":"Eghbali A, Pradel M (2020) No strings attached: an empirical study of string-related software bugs. In: International conference on automated software engineering (ASE)","DOI":"10.1145\/3324884.3416576"},{"key":"10521_CR50","unstructured":"Face H (2024) Hugging Face Documentation: timm. https:\/\/huggingface.co\/docs\/timm\/index"},{"key":"10521_CR51","doi-asserted-by":"crossref","unstructured":"Fitzgerald B (2006) The transformation of open source software. MIS Quarterly pp 587\u2013598","DOI":"10.2307\/25148740"},{"key":"10521_CR52","unstructured":"Forsyth DA, Ponce J (2002) Computer vision: a modern approach. Prentice Hall Professional Technical Reference"},{"key":"10521_CR53","doi-asserted-by":"crossref","unstructured":"Garcia J, Feng Y, Shen J, Almanee S, Xia Y, Chen QA (2020) A comprehensive study of autonomous vehicle bugs. In: International Conference on Software Engineering (ICSE), https:\/\/dl.acm.org\/doi\/10.1145\/3377811.3380397","DOI":"10.1145\/3377811.3380397"},{"key":"10521_CR54","doi-asserted-by":"crossref","unstructured":"Gharehyazie M, Ray B, Filkov V (2017) Some from here, some from there: Cross-project code reuse in github. In: 2017 IEEE\/ACM 14th International conference on mining software repositories (MSR), IEEE, pp 291\u2013301","DOI":"10.1109\/MSR.2017.15"},{"key":"10521_CR55","doi-asserted-by":"crossref","unstructured":"Goel A, Tung C, Lu YH, Thiruvathukal GK (2020) A Survey of Methods for Low-Power Deep Learning and Computer Vision. In: IEEE World forum on internet of things (WF-IoT)","DOI":"10.1109\/WF-IoT48130.2020.9221198"},{"key":"10521_CR56","first-page":"821","volume":"11763","author":"M Gong","year":"2021","unstructured":"Gong M, Wang D, Zhao X, Guo H, Luo D, Song M (2021) A review of non-maximum suppression algorithms for deep learning target detection. Seventh symposium on novel photoelectronic detection technology and applications, SPIE 11763:821\u2013828","journal-title":"Seventh symposium on novel photoelectronic detection technology and applications, SPIE"},{"key":"10521_CR57","unstructured":"Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press"},{"key":"10521_CR58","unstructured":"Google (2022) Tensorflow model garden. https:\/\/github.com\/tensorflow\/models"},{"key":"10521_CR59","doi-asserted-by":"crossref","unstructured":"Gopalakrishna NK, Anandayuvaraj D, Detti A, Bland FL, Rahaman S, Davis JC (2022) \u201cIf security is required\u201d: engineering and security practices for machine learning-based IoT devices. In: Proceedings of the 4th international workshop on software engineering research & practices for the internet of things (SERP4IoT), pp 1\u20138","DOI":"10.1145\/3528227.3528565"},{"key":"10521_CR60","unstructured":"Goyal P, Doll\u00e1r P, Girshick R, Noordhuis P, Wesolowski L, Kyrola A, Tulloch A, Jia Y, He K (2018) Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv:1706.02677"},{"key":"10521_CR61","doi-asserted-by":"crossref","unstructured":"Grima-Farrell C (2017) The rtp model: An interactive research to practice framework. What Matters in a Research to Practice Cycle? Teachers as Researchers pp 237\u2013250","DOI":"10.1007\/978-981-10-2087-2_9"},{"key":"10521_CR62","doi-asserted-by":"crossref","unstructured":"Guan H, Xiao Y, Li J, Liu Y, Bai G (2023) A comprehensive study of real-world bugs in machine learning model optimization. In: Proceedings of the international conference on software engineering","DOI":"10.1109\/ICSE48619.2023.00024"},{"key":"10521_CR63","doi-asserted-by":"crossref","unstructured":"Gundersen OE, Kjensmo S (2018) State of the art: Reproducibility in artificial intelligence. AAAI Conference on Artif Intell (AAAI)","DOI":"10.1609\/aaai.v32i1.11503"},{"key":"10521_CR64","doi-asserted-by":"crossref","unstructured":"Gundersen OE, Gil Y, Aha DW (2018) On reproducible AI: Towards reproducible research, open science, and digital scholarship in AI publications. AI Magazine","DOI":"10.1609\/aimag.v39i3.2816"},{"key":"10521_CR65","doi-asserted-by":"crossref","unstructured":"Guo J, Jiang Y, Zhao Y, Chen Q, Sun J (2018) DLFuzz: Differential Fuzzing Testing of Deep Learning Systems. In: European Software engineering conference and symposium on the foundations of software engineering (ESEC\/FSE)","DOI":"10.1145\/3236024.3264835"},{"key":"10521_CR66","unstructured":"Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W (2021) Lora: Low-rank adaptation of large language models. arXiv:2106.09685"},{"key":"10521_CR67","doi-asserted-by":"crossref","unstructured":"Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: International conference on software engineering (ICSE)","DOI":"10.1145\/3377811.3380395"},{"issue":"6377","key":"10521_CR68","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1126\/science.359.6377.725","volume":"359","author":"M Hutson","year":"2018","unstructured":"Hutson M (2018) Artificial intelligence faces reproducibility crisis. American Assoc Advancement Sci 359(6377):725\u2013726. https:\/\/doi.org\/10.1126\/science.359.6377.725","journal-title":"American Assoc Advancement Sci"},{"key":"10521_CR69","doi-asserted-by":"crossref","unstructured":"Imtiaz SM, Batole F, Singh A, Pan R, Cruz BD, Rajan H (2023) Decomposing a recurrent neural network into modules for enabling reusability and replacement. In: 2023 IEEE\/ACM 45th International conference on software engineering (ICSE), IEEE, pp 1020\u20131032","DOI":"10.1109\/ICSE48619.2023.00093"},{"key":"10521_CR70","doi-asserted-by":"crossref","unstructured":"Islam JF, Mondal M, Roy CK (2016) Bug replication in code clones: An empirical study. In: International conference on software analysis, evolution, and reengineering (SANER), IEEE, vol\u00a01, pp 68\u201378","DOI":"10.1109\/SANER.2016.78"},{"key":"10521_CR71","doi-asserted-by":"crossref","unstructured":"Islam MJ, Nguyen G, Pan R, Rajan H (2019) A comprehensive study on deep learning bug characteristics. In: European software engineering conference and symposium on the foundations of software engineering (ESEC\/FSE)","DOI":"10.1145\/3338906.3338955"},{"key":"10521_CR72","doi-asserted-by":"crossref","unstructured":"Islam MJ, Pan R, Nguyen G, Rajan H (2020) Repairing deep neural networks: fix patterns and challenges. In: International conference on software engineering (ICSE)","DOI":"10.1145\/3377811.3380378"},{"key":"10521_CR73","unstructured":"Jajal P, Jiang W, Tewari A, Woo J, Lu YH, Thiruvathukal GK, Davis JC (2023) Analysis of failures and risks in deep learning model converters: A case study in the onnx ecosystem. arXiv:2303.17708"},{"key":"10521_CR74","unstructured":"Jarzabek S (1993) Software reengineering for reusability. In: International computer software and applications conference (COMPSAC)"},{"key":"10521_CR75","doi-asserted-by":"crossref","unstructured":"Jiang W, Synovic N, Sethi R (2022a) An Empirical Study of Artifacts and Security Risks in the Pre-trained Model Supply Chain. Los Angeles p\u00a010","DOI":"10.1145\/3560835.3564547"},{"key":"10521_CR76","doi-asserted-by":"publisher","unstructured":"Jiang W, Synovic N, Sethi R, Indarapu A, Hyatt M, Schorlemmer TR, Thiruvathukal GK, Davis JC (2022b) An empirical study of artifacts and security risks in the pre-trained model supply chain. In: ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses (SCORED\u201922), p 105\u2013114, https:\/\/doi.org\/10.1145\/3560835.3564547","DOI":"10.1145\/3560835.3564547"},{"key":"10521_CR77","unstructured":"Jiang W, Cheung C, Kim M, Kim H, Thiruvathukal GK, Davis JC (2023a) Naming practices of pre-trained models in hugging face. arXiv:2310.01642"},{"key":"10521_CR78","doi-asserted-by":"crossref","unstructured":"Jiang W, Synovic N, Hyatt M, Schorlemmer TR, Sethi R, Lu YH, Thiruvathukal GK, Davis JC (2023b) An empirical study of pre-trained model reuse in the hugging face deep learning model registry. In: IEEE\/ACM 45th International conference on software engineering (ICSE\u201923)","DOI":"10.1109\/ICSE48619.2023.00206"},{"key":"10521_CR79","doi-asserted-by":"crossref","unstructured":"Jiang W, Synovic N, Jajal P, Schorlemmer TR, Tewari A, Pareek B, Thiruvathukal GK, Davis JC (2023c) Ptmtorrent: A dataset for mining open-source pre-trained model packages. Proceedings of the 20th International Conference on Mining Software Repositories (MSR\u201923)","DOI":"10.1109\/MSR59073.2023.00021"},{"key":"10521_CR80","doi-asserted-by":"crossref","unstructured":"Jiang W, Yasmin J, Jones J, Synovic N, Kuo J, Bielanski N, Tian Y, , Thiruvathukal GK, Davis JC (2024) Peatmoss: A dataset and initial analysis of pre-trained models in open-source software. In: International conference on mining software repositories (MSR)","DOI":"10.1145\/3643991.3644907"},{"key":"10521_CR81","unstructured":"Jing YK (2021) Model Zoo - Deep learning code and pretrained models. https:\/\/modelzoo.co\/"},{"key":"10521_CR82","doi-asserted-by":"crossref","unstructured":"Johnson RB, Onwuegbuzie AJ (2004) Mixed methods research: a research paradigm whose time has come. Educ Res 33(7):14\u201326","DOI":"10.3102\/0013189X033007014"},{"key":"10521_CR83","doi-asserted-by":"publisher","unstructured":"K\u00e4ding C, Rodner E, Freytag A, Denzler J (2017) Fine-Tuning Deep Neural Networks in Continuous Learning Scenarios. In: Chen CS, Lu J, Ma KK (eds) Computer Vision \u2013 ACCV 2016 Workshops, vol 10118, Springer International Publishing, Cham, pp 588\u2013605, https:\/\/doi.org\/10.1007\/978-3-319-54526-4_43, http:\/\/link.springer.com\/10.1007\/978-3-319-54526-4_43, series Title: Lecture Notes in Computer Science","DOI":"10.1007\/978-3-319-54526-4_43"},{"key":"10521_CR84","unstructured":"Keras (2022) Keras applications. https:\/\/keras.io\/api\/applications\/"},{"key":"10521_CR85","doi-asserted-by":"crossref","unstructured":"Keshta N, Morgan Y (2017) Comparison between traditional plan-based and agile software processes according to team size & project domain (a systematic literature review). In: 2017 8th IEEE Annual information technology, electronics and mobile communication conference (IEMCON), IEEE, pp 567\u2013575","DOI":"10.1109\/IEMCON.2017.8117128"},{"issue":"3","key":"10521_CR86","doi-asserted-by":"publisher","DOI":"10.1111\/exsy.12400","volume":"36","author":"A Khamparia","year":"2019","unstructured":"Khamparia A, Singh KM (2019) A systematic review on deep learning architectures and applications. Expert Syst 36(3):e12400","journal-title":"Expert Syst"},{"key":"10521_CR87","unstructured":"Kim J, Li J (2020) Introducing the model garden for tensorflow 2. https:\/\/blog.tensorflow.org\/2020\/03\/introducing-model-garden-for-tensorflow-2.html"},{"key":"10521_CR88","first-page":"84","volume":"6","author":"A Krizhevsky","year":"2012","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet Classification with Deep Convolutional Neural Networks. Advances in neural information processing systems (NeurIPS) 6:84\u201390","journal-title":"Advances in neural information processing systems (NeurIPS)"},{"key":"10521_CR89","doi-asserted-by":"crossref","unstructured":"Kumar A, Boehm M, Yang J (2017) Data Management in Machine Learning: Challenges, Techniques, and Systems. In: International conference on management of data","DOI":"10.1145\/3035918.3054775"},{"issue":"2","key":"10521_CR90","doi-asserted-by":"publisher","first-page":"712","DOI":"10.1109\/TITS.2019.2962338","volume":"22","author":"S Kuutti","year":"2020","unstructured":"Kuutti S, Bowden R, Jin Y, Barber P, Fallah S (2020) A survey of deep learning applications to autonomous vehicle control. IEEE Trans Intell Transportation Syst 22(2):712\u2013733","journal-title":"IEEE Trans Intell Transportation Syst"},{"key":"10521_CR91","volume-title":"Safeware: System safety and computers","author":"NG Leveson","year":"1995","unstructured":"Leveson NG (1995) Safeware: System safety and computers. ACM, New York, NY, USA"},{"key":"10521_CR92","unstructured":"Leveson NG (2016) Engineering a safer world: Systems thinking applied to safety. The MIT Press"},{"key":"10521_CR93","doi-asserted-by":"publisher","unstructured":"Li R, Jiao Q, Cao W, Wong HS, Wu S (2020) Model Adaptation: Unsupervised Domain Adaptation without Source Data. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition pp 9638\u20139647,https:\/\/doi.org\/10.1109\/CVPR42600.2020.00966","DOI":"10.1109\/CVPR42600.2020.00966"},{"key":"10521_CR94","doi-asserted-by":"crossref","unstructured":"Li Z, Liu F, Yang W, Peng S, Zhou J (2021) A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Trans Neural Netw Learn Syst","DOI":"10.1109\/TNNLS.2020.3007412"},{"key":"10521_CR95","doi-asserted-by":"crossref","unstructured":"Lin TY, Maire M, Belongie S et\u00a0al (2014) Microsoft COCO: Common Objects in Context. In: European conference on computer vision (ECCV)","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"10521_CR96","unstructured":"Linda D, Rosenberg H, Hyatt LE (1996) Software Re-engineering. Softw Assurance Technol Center"},{"issue":"1","key":"10521_CR97","first-page":"1","volume":"31","author":"C Liu","year":"2021","unstructured":"Liu C, Gao C, Xia X, Lo D, Grundy J, Yang X (2021) On the Replicability and Reproducibility of Deep Learning in Software Engineering. ACM Trans Softw Eng Methodol 31(1):1\u201346","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10521_CR98","doi-asserted-by":"crossref","unstructured":"Liu J, Lin J, Ruffy F, Tan C, Li J, Panda A, Zhang L (2023) Nnsmith: Generating diverse and valid test cases for deep learning compilers. In: Proceedings of the 28th ACM international conference on architectural support for programming languages and operating systems, Volume 2, pp 530\u2013543","DOI":"10.1145\/3575693.3575707"},{"key":"10521_CR99","doi-asserted-by":"crossref","unstructured":"Liu X, Ji K, Fu Y, Tam W, Du Z, Yang Z, Tang J (2022) P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In: Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 2: Short Papers), pp 61\u201368","DOI":"10.18653\/v1\/2022.acl-short.8"},{"key":"10521_CR100","doi-asserted-by":"publisher","unstructured":"Liu Y, Xu C, Cheung SC (2014) Characterizing and detecting performance bugs for smartphone applications. In: Proceedings of the 36th International Conference on Software Engineering, ACM, Hyderabad India, pp 1013\u20131024, https:\/\/doi.org\/10.1145\/2568225.2568229, https:\/\/dl.acm.org\/doi\/10.1145\/2568225.2568229","DOI":"10.1145\/2568225.2568229"},{"key":"10521_CR101","doi-asserted-by":"crossref","unstructured":"Liu Y, Chen C, Zhang R, Qin T, Ji X, Lin H, Yang M (2020) Enhancing the interoperability between deep learning frameworks by model conversion. In: European software engineering conference\/foundations of software engineering (ESEC\/FSE)","DOI":"10.1145\/3368089.3417051"},{"key":"10521_CR102","unstructured":"Lorenzoni G, Alencar P, Nascimento N, Cowan D (2021) Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review. arXiv:2102.07574"},{"key":"10521_CR103","doi-asserted-by":"publisher","unstructured":"Lu B, Yang J, Chen LY, Ren S (2019) Automating Deep Neural Network Model Selection for Edge Inference. In: 2019 IEEE First International conference on cognitive machine intelligence (CogMI), pp 184\u2013193, https:\/\/doi.org\/10.1109\/CogMI48466.2019.00035","DOI":"10.1109\/CogMI48466.2019.00035"},{"key":"10521_CR104","doi-asserted-by":"crossref","unstructured":"Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, Chen C, Su T, Li L, Liu Y et\u00a0al (2018a) Deepgauge: Multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM\/IEEE international conference on automated software engineering, pp 120\u2013131","DOI":"10.1145\/3238147.3238202"},{"key":"10521_CR105","doi-asserted-by":"crossref","unstructured":"Ma S, Liu Y, Lee WC, Zhang X, Grama A (2018b) Mode: automated neural network model debugging via state differential analysis and input selection. In: Proceedings of the 2018 26th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 175\u2013186","DOI":"10.1145\/3236024.3236082"},{"key":"10521_CR106","doi-asserted-by":"crossref","unstructured":"Majthoub M, Qutqut MH, Odeh Y (2018) Software re-engineering: An overview. In: 2018 8th International conference on computer science and information technology (CSIT), IEEE, pp 266\u2013270","DOI":"10.1109\/CSIT.2018.8486173"},{"issue":"3","key":"10521_CR107","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh ML (2012) Interrater reliability: the kappa statistic. Biochemia Medica 22(3):276\u2013282","journal-title":"Biochemia Medica"},{"key":"10521_CR108","unstructured":"Mckeeman WM (1998) Differential Testing for Software. Digital Technical J"},{"key":"10521_CR109","unstructured":"Meta (2022) Torchvision. https:\/\/github.com\/pytorch\/vision"},{"key":"10521_CR110","unstructured":"Meta (2024a) Detectron. https:\/\/ai.meta.com\/tools\/detectron\/"},{"key":"10521_CR111","unstructured":"Meta (2024b) Detectron2. https:\/\/ai.meta.com\/tools\/detectron2\/"},{"key":"10521_CR112","doi-asserted-by":"publisher","unstructured":"Michael LG, Donohue J, Davis JC, Lee D, Servant F (2019) Regexes are Hard: Decision-Making, Difficulties, and Risks in Programming Regular Expressions. In: 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE), pp 415\u2013426, https:\/\/doi.org\/10.1109\/ASE.2019.00047, iSSN: 2643-1572","DOI":"10.1109\/ASE.2019.00047"},{"key":"10521_CR113","doi-asserted-by":"publisher","unstructured":"Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, Spitzer E, Raji ID, Gebru T (2019) Model Cards for Model Reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, ACM, Atlanta GA USA, pp 220\u2013229, https:\/\/doi.org\/10.1145\/3287560.3287596, https:\/\/dl.acm.org\/doi\/10.1145\/3287560.3287596","DOI":"10.1145\/3287560.3287596"},{"key":"10521_CR114","doi-asserted-by":"crossref","unstructured":"Montes D, Peerapatanapokin P, Schultz J, Guo C, Jiang W, Davis JC (2022a) Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability. In: European software engineering conference and symposium on the foundations of software engineering (ESEC\/FSE-IVR track)","DOI":"10.1145\/3540250.3560881"},{"key":"10521_CR115","doi-asserted-by":"crossref","unstructured":"Montes D, Pongpatapee P, Schultz J, Guo C, Jiang W, Davis J (2022b) Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability. In: ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering \u2014 Ideas, Visions, and Reflections track (ESEC\/FSE-IVR)","DOI":"10.1145\/3540250.3560881"},{"key":"10521_CR116","doi-asserted-by":"crossref","unstructured":"Nahar N, Zhou S, Lewis G, K\u00e4stner C (2022) Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process. In: International conference on software engineering (ICSE)","DOI":"10.1145\/3510003.3510209"},{"issue":"2","key":"10521_CR117","doi-asserted-by":"publisher","first-page":"464","DOI":"10.3390\/s22020464","volume":"22","author":"U Nepal","year":"2022","unstructured":"Nepal U, Eslamiat H (2022) Comparing yolov3, yolov4 and yolov5 for autonomous landing spot detection in faulty uavs. Sensors 22(2):464","journal-title":"Sensors"},{"key":"10521_CR118","doi-asserted-by":"crossref","unstructured":"Nikanjam A, Khomh F (2021) Design Smells in Deep Learning Programs: An Empirical Study. In: IEEE International conference on software maintenance and evolution (ICSME)","DOI":"10.26226\/morressier.613b5418842293c031b5b61d"},{"key":"10521_CR119","doi-asserted-by":"crossref","unstructured":"Nikitin NO, Vychuzhanin P, Sarafanov M, Polonskaia IS, Revin I, Barabanova IV, Maximov G, Kalyuzhnaya AV, Boukhanovsky A (2022) Automated evolutionary approach for the design of composite machine learning pipelines. Future Generation Computer Systems","DOI":"10.1016\/j.future.2021.08.022"},{"key":"10521_CR120","unstructured":"O\u2019Connor R (2023) Pytorch vs tensorflow in 2023. https:\/\/www.assemblyai.com\/blog\/pytorch-vs-tensorflow-in-2023\/"},{"key":"10521_CR121","doi-asserted-by":"publisher","unstructured":"Openja M, Nikanjam A, Yahmed AH, Khomh F, Jiang ZMJ (2022) An Empirical Study of Challenges in Converting Deep Learning Models. In: 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 13\u201323, https:\/\/doi.org\/10.1109\/ICSME55016.2022.00010, iSSN: 2576-3148","DOI":"10.1109\/ICSME55016.2022.00010"},{"key":"10521_CR122","unstructured":"O\u2019Shea K, Nash R (2015) An introduction to convolutional neural networks. arXiv:1511.08458"},{"key":"10521_CR123","doi-asserted-by":"crossref","unstructured":"Pan R, Rajan H (2020) On decomposing a deep neural network into modules. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 889\u2013900","DOI":"10.1145\/3368089.3409668"},{"key":"10521_CR124","doi-asserted-by":"crossref","unstructured":"Pan R, Rajan H (2022) Decomposing convolutional neural networks into reusable and replaceable modules. In: International conference on software engineering (ICSE), ACM, Pittsburgh Pennsylvania","DOI":"10.1145\/3510003.3510051"},{"key":"10521_CR125","unstructured":"Pan R, Ibrahimzada AR, Krishna R, Sankar D, Wassi LP, Merler M, Sobolev B, Pavuluri R, Sinha S, Jabbarvand R (2023) Understanding the effectiveness of large language models in code translation. arXiv:2308.03109"},{"key":"10521_CR126","doi-asserted-by":"crossref","unstructured":"Panchal D, Baran I, Musgrove D, Lu D (2023) Mlops: Automatic, zero-touch and reusable machine learning training and serving pipelines. In: 2023 IEEE International conference on internet of things and intelligence systems (IoTaIS), IEEE, pp 175\u2013181","DOI":"10.1109\/IoTaIS60147.2023.10346079"},{"key":"10521_CR127","doi-asserted-by":"crossref","unstructured":"Panchal D, Verma P, Baran I, Musgrove D, Lu D (2024) Reusable mlops: Reusable deployment, reusable infrastructure and hot-swappable machine learning models and services. arXiv:2403.00787","DOI":"10.1109\/ICSCC62041.2024.10690392"},{"key":"10521_CR128","doi-asserted-by":"crossref","unstructured":"Pei K, Cao Y, Yang J, Jana S (2017) DeepXplore: Automated Whitebox Testing of Deep Learning Systems. In: Symposium on operating systems principles (SOSP)","DOI":"10.1145\/3132747.3132785"},{"key":"10521_CR129","unstructured":"Perry D, Sim S, Easterbrook S (2004) Case studies for software engineers. In: International conference on software engineering (ICSE)"},{"key":"10521_CR130","doi-asserted-by":"publisher","unstructured":"Pham HV, Qian S, Wang J, Lutellier T, Rosenthal J, Tan L, Yu Y, Nagappan N (2020) Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance. In: International conference on automated software engineering (ASE), https:\/\/doi.org\/10.1145\/3324884.3416545","DOI":"10.1145\/3324884.3416545"},{"key":"10521_CR131","unstructured":"Pineau J (2022) How the AI community can get serious about reproducibility. https:\/\/ai.facebook.com\/blog\/how-the-ai-community-can-get-serious-about-reproducibility\/"},{"key":"10521_CR132","unstructured":"Pineau J, Vincent-Lamarre P, Sinha K, Lariviere V, Beygelzimer A (2020) Improving Reproducibility in Machine Learning Research. J Machine Learn Res"},{"issue":"1","key":"10521_CR133","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-020-18073-9","volume":"11","author":"M Popel","year":"2020","unstructured":"Popel M, Tomkova M, Tomek J, Kaiser \u0141, Uszkoreit J, Bojar O, \u017dabokrtsk\u1ef3 Z (2020) Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. Nature Commun 11(1):1\u201315","journal-title":"Nature Commun"},{"key":"10521_CR134","volume-title":"Software engineering: a practitioner\u2019s approach","author":"RS Pressman","year":"2005","unstructured":"Pressman RS (2005) Software engineering: a practitioner\u2019s approach. Palgrave Macmillan"},{"key":"10521_CR135","unstructured":"Pytorch (2021) Pytorch hub. https:\/\/pytorch.org\/hub\/"},{"key":"10521_CR136","doi-asserted-by":"publisher","unstructured":"Qi B, Sun H, Gao X, Zhang H, Li Z, Liu X (2023) Reusing deep neural network models through model re-engineering. In: International Conference on Software Engineering, IEEE Press, p 983\u2013994, https:\/\/doi.org\/10.1109\/ICSE48619.2023.00090","DOI":"10.1109\/ICSE48619.2023.00090"},{"key":"10521_CR137","doi-asserted-by":"publisher","unstructured":"Rahman S, River E, Khomh F, Guhneuc YG, Lehnert B (2019) Machine learning software engineering in practice: An industrial case study. arXiv preprint https:\/\/doi.org\/10.48550\/arXiv.1906.07154","DOI":"10.48550\/arXiv.1906.07154"},{"key":"10521_CR138","unstructured":"Ralph P, Ali Nb, Baltes S, Bianculli D, Diaz J, Dittrich Y, Ernst N, Felderer M, Feldt R, Filieri A, de\u00a0Fran\u00e7a BBN, Furia CA, Gay G, Gold N, Graziotin D, He P, Hoda R, Juristo N, Kitchenham B, Lenarduzzi V, Mart\u00ednez J, Melegati J, Mendez D, Menzies T, Molleri J, Pfahl D, Robbes R, Russo D, Saarim\u00e4ki N, Sarro F, Taibi D, Siegmund J, Spinellis D, Staron M, Stol K, Storey MA, Taibi D, Tamburri D, Torchiano M, Treude C, Turhan B, Wang X, Vegas S (2021) Empirical Standards for Software Engineering Research. arXiv:2010.03525"},{"key":"10521_CR139","doi-asserted-by":"crossref","unstructured":"Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 779\u2013788","DOI":"10.1109\/CVPR.2016.91"},{"key":"10521_CR140","doi-asserted-by":"crossref","unstructured":"Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans Pattern Anal Machine Intell (TPAMI)","DOI":"10.1109\/TPAMI.2016.2577031"},{"key":"10521_CR141","doi-asserted-by":"crossref","unstructured":"Ritchie J, Spencer L (2002) Qualitative data analysis for applied policy research. In: Analyzing qualitative data, Routledge, pp 187\u2013208","DOI":"10.4324\/9780203413081-14"},{"key":"10521_CR142","unstructured":"Ritchie J, Lewis J, Nicholls CM, Ormston R et\u00a0al (2013) Qualitative research practice: A guide for social science students and researchers. Sage"},{"key":"10521_CR143","unstructured":"Rosenberg LH, Hyatt LE (1996) Software re-engineering. Software Assurance Technology Center pp 2\u20133"},{"key":"10521_CR144","doi-asserted-by":"crossref","unstructured":"Runeson P, H\u00f6st M (2009) Guidelines for conducting and reporting case study research in software engineering. Empirical Softw Eng (EMSE)","DOI":"10.1007\/s10664-008-9102-8"},{"key":"10521_CR145","doi-asserted-by":"publisher","unstructured":"Saha RK, Khurshid S, Perry DE (2014) An empirical study of long lived bugs. 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering, CSMR-WCRE 2014 - Proceedings pp 144\u2013153, https:\/\/doi.org\/10.1109\/CSMR-WCRE.2014.6747164","DOI":"10.1109\/CSMR-WCRE.2014.6747164"},{"key":"10521_CR146","unstructured":"Schelter S, Boese JH, Kirschnick J, Klein T, Seufert S (2017) Automatically tracking metadata and provenance of machine learning experiments. In: Machine learning systems workshop at NIPS"},{"key":"10521_CR147","unstructured":"Schelter S, Biessmann F, Januschowski T, Salinas D, Seufert S, Szarvas G (2018) On Challenges in Machine Learning Model Management. Bullet IEEE Computer Soc Technical Committee Data Eng"},{"key":"10521_CR148","doi-asserted-by":"crossref","unstructured":"Schmidhuber J (2015) Deep learning in neural networks: An overview. Neural Netw","DOI":"10.1016\/j.neunet.2014.09.003"},{"key":"10521_CR149","unstructured":"Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M (2014) Machine Learning : The High-Interest Credit Card of Technical Debt. In: NIPS Workshop on software engineering for machine learning (SE4ML)"},{"key":"10521_CR150","unstructured":"Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M, Crespo JF, Dennison D (2015) Hidden Technical Debt in Machine Learning Systems. In: Advances in Neural Information Processing Systems, Curran Associates, Inc., vol\u00a028, https:\/\/proceedings.neurips.cc\/paper\/2015\/hash\/86df7dcfd896fcaf2674f757a2463eba-Abstract.html"},{"key":"10521_CR151","doi-asserted-by":"crossref","unstructured":"Seaman CB, Shull F, Regardie M, Elbert D, Feldmann RL, Guo Y, Godfrey S (2008) Defect categorization: making use of a decade of widely varying historical data. In: Empirical software engineering and measurement (ESEM)","DOI":"10.1145\/1414004.1414030"},{"issue":"1145\/3382494","key":"10521_CR152","first-page":"3410681","volume":"10","author":"A Serban","year":"2020","unstructured":"Serban A, Van Der Blom K, Hoos H, Visser J (2020) Adoption and effects of software engineering best practices in machine learning. Int Symposium on Empirical Softw Eng Measurement 10(1145\/3382494):3410681","journal-title":"Int Symposium on Empirical Softw Eng Measurement"},{"key":"10521_CR153","doi-asserted-by":"crossref","unstructured":"Shen Q, Ma H, Chen J, Tian Y, Cheung SC, Chen X (2021) A comprehensive study of deep learning compiler bugs. In: European software engineering conference and symposium on the foundations of software engineering (ESEC\/FSE)","DOI":"10.1145\/3468264.3468591"},{"key":"10521_CR154","doi-asserted-by":"crossref","unstructured":"Shorten C, Khoshgoftaar TM (2019) A survey on Image Data Augmentation for Deep Learning. J Big Data","DOI":"10.1186\/s40537-019-0197-0"},{"key":"10521_CR155","doi-asserted-by":"publisher","first-page":"53040","DOI":"10.1109\/ACCESS.2019.2912200","volume":"7","author":"A Shrestha","year":"2019","unstructured":"Shrestha A, Mahmood A (2019) Review of deep learning algorithms and architectures. IEEE access 7:53040\u201353065","journal-title":"IEEE access"},{"key":"10521_CR156","first-page":"14274","volume":"35","author":"M Shu","year":"2022","unstructured":"Shu M, Nie W, Huang DA, Yu Z, Goldstein T, Anandkumar A, Xiao C (2022) Test-time prompt tuning for zero-shot generalization in vision-language models. Adv Neural Inf Process Syst 35:14274\u201314289","journal-title":"Adv Neural Inf Process Syst"},{"issue":"3","key":"10521_CR157","doi-asserted-by":"publisher","first-page":"2041","DOI":"10.11591\/ijece.v9i3.pp2041-2048","volume":"9","author":"J Singh","year":"2019","unstructured":"Singh J, Singh K, Singh J (2019) Reengineering framework for open source software using decision tree approach. Int J Electrical Computer Eng (IJECE) 9(3):2041\u20132048","journal-title":"Int J Electrical Computer Eng (IJECE)"},{"key":"10521_CR158","unstructured":"Srivastava A, Thomson S (2009) Framework analysis: A qualitative methodology for applied policy research"},{"key":"10521_CR159","doi-asserted-by":"crossref","unstructured":"Sun X, Zhou T, Li G, Hu J, Yang H, Li B (2017) An Empirical Study on Real Bugs for Machine Learning Programs. In: Asia-Pacific SOFTWARE ENGINEERING CONFERENCE (APSEC)","DOI":"10.1109\/APSEC.2017.41"},{"key":"10521_CR160","doi-asserted-by":"crossref","unstructured":"Szeliski R (2022) Computer vision: algorithms and applications. Springer Nature","DOI":"10.1007\/978-3-030-34372-9"},{"key":"10521_CR161","doi-asserted-by":"crossref","unstructured":"Taecharungroj V (2023) \u201cwhat can chatgpt do?\u201d analyzing early reactions to the innovative ai chatbot on twitter. Big Data Cognitive Comput 7(1):35","DOI":"10.3390\/bdcc7010035"},{"key":"10521_CR162","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101693","volume":"63","author":"N Tajbakhsh","year":"2020","unstructured":"Tajbakhsh N, Jeyaseelan L, Li Q, Chiang JN, Wu Z, Ding X (2020) Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Med Image Anal 63:101693","journal-title":"Med Image Anal"},{"key":"10521_CR163","unstructured":"Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C (2018) A Survey on Deep Transfer Learning. IEEE Trans Knowl Data Eng"},{"key":"10521_CR164","doi-asserted-by":"crossref","unstructured":"Tan L, Liu C, Li Z, Wang X, Zhou Y, Zhai C (2014) Bug characteristics in open source software. Empirical Softw Eng (EMSE)","DOI":"10.1007\/s10664-013-9258-8"},{"key":"10521_CR165","unstructured":"Tatman R, Vanderplas J, Dane S (2018) A Practical Taxonomy of Reproducibility for Machine Learning Research. In: Reproducibility in machine learning workshop at ICML"},{"key":"10521_CR166","doi-asserted-by":"crossref","unstructured":"Thiruvathukal GK, Lu YH, Kim J, Chen Y, Chen B (2022) Low-power Computer Vision: Improve the Efficiency of Artificial Intelligence","DOI":"10.1201\/9781003162810"},{"key":"10521_CR167","doi-asserted-by":"crossref","unstructured":"Thung F, Wang S, Lo D, Jiang L (2012) An empirical study of bugs in machine learning systems. In: International symposium on software reliability engineering (ISSRE)","DOI":"10.1109\/ISSRE.2012.22"},{"key":"10521_CR168","doi-asserted-by":"crossref","unstructured":"Tian Y, Pei K, Jana S, Ray B (2018) DeepTest: automated testing of deep-neural-network-driven autonomous cars. In: International conference on software engineering (ICSE)","DOI":"10.1145\/3180155.3180220"},{"key":"10521_CR169","unstructured":"Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, Rozi\u00e8re B, Goyal N, Hambro E, Azhar F et\u00a0al (2023a) Llama: Open and efficient foundation language models. arXiv:2302.13971"},{"key":"10521_CR170","unstructured":"Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S et\u00a0al (2023b) Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288"},{"key":"10521_CR171","doi-asserted-by":"publisher","unstructured":"Tsay J, Braz A, Hirzel M, Shinnar A, Mummert T (2020) AIMMX: Artificial Intelligence Model Metadata Extractor. In: International conference on mining software repositories (MSR), https:\/\/doi.org\/10.1145\/3379597.3387448","DOI":"10.1145\/3379597.3387448"},{"key":"10521_CR172","doi-asserted-by":"publisher","unstructured":"Tsay J, Braz A, Hirzel M, Shinnar A, Mummert T (2022) Extracting enhanced artificial intelligence model metadata from software repositories. Empirical Softw Eng 27(7):176. https:\/\/doi.org\/10.1007\/s10664-022-10206-6, https:\/\/link.springer.com\/10.1007\/s10664-022-10206-6","DOI":"10.1007\/s10664-022-10206-6"},{"key":"10521_CR173","doi-asserted-by":"crossref","unstructured":"Tucker DC, Devon MS (2010) A Case Study in Software Reengineering. In: International conference on informatio (itng)n technology: New Generations","DOI":"10.1109\/ITNG.2010.189"},{"key":"10521_CR174","doi-asserted-by":"crossref","unstructured":"Unceta I, Nin J, Pujol O (2020) Environmental adaptation and differential replication in machine learning. Entropy 22(10):1122","DOI":"10.3390\/e22101122"},{"key":"10521_CR175","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1016\/0164-1212(89)90016-2","volume":"9","author":"JD Valett","year":"1989","unstructured":"Valett JD, McGarry FE (1989) A Summary of Software Measurement Experiences in the Software Engineering Laboratory. J Syst Softw 9:137\u2013148","journal-title":"J Syst Softw"},{"key":"10521_CR176","unstructured":"Vartak M, Subramanyam H, Lee WE, Viswanathan S, Husnoo S, Madden S, Zaharia M (2016) Modeldb: a system for machine learning model management. In: the workshop on human-in-the-loop data analytics"},{"key":"10521_CR177","unstructured":"Villa J, Zimmerman Y (2018) Reproducibility in ML: why it matters and how to achieve it. https:\/\/determined.ai\/blog\/reproducibility-in-ml"},{"key":"10521_CR178","doi-asserted-by":"crossref","unstructured":"Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P et\u00a0al (2019) Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature","DOI":"10.1038\/s41586-019-1724-z"},{"key":"10521_CR179","doi-asserted-by":"crossref","unstructured":"Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep Learning for Computer Vision: A Brief Review. Comput Intell Neurosci","DOI":"10.1155\/2018\/7068349"},{"key":"10521_CR180","doi-asserted-by":"publisher","unstructured":"Wang J, Dou W, Gao Y, Gao C, Qin F, Yin K, Wei J (2017) A comprehensive study on real world concurrency bugs in Node.js. In: 2017 32nd IEEE\/ACM International conference on automated software engineering (ASE), pp 520\u2013531, https:\/\/doi.org\/10.1109\/ASE.2017.8115663","DOI":"10.1109\/ASE.2017.8115663"},{"key":"10521_CR181","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1016\/j.jmsy.2018.01.003","volume":"48","author":"J Wang","year":"2018","unstructured":"Wang J, Ma Y, Zhang L, Gao RX, Wu D (2018) Deep learning for smart manufacturing: Methods and applications. J Manufac Syst 48:144\u2013156","journal-title":"J Manufac Syst"},{"key":"10521_CR182","unstructured":"Wang J, Lu Y, Yuan B, Chen B, Liang P, De\u00a0Sa C, Re C, Zhang C (2023) Cocktailsgd: Fine-tuning foundation models over 500mbps networks. In: International conference on machine learning, PMLR, pp 36058\u201336076"},{"key":"10521_CR183","doi-asserted-by":"crossref","unstructured":"Wang P, Brown C, Jennings JA, Stolee KT (2020a) An Empirical Study on Regular Expression Bugs. In: International conference on mining software repositories (MSR)","DOI":"10.1145\/3379597.3387464"},{"key":"10521_CR184","unstructured":"Wang S, Huang L, Ge J, Zhang T, Feng H, Li M, Zhang H, Ng V (2020b) Synergy between machine\/deep learning and software engineering: How far are we? arXiv:2008.05515"},{"key":"10521_CR185","doi-asserted-by":"publisher","unstructured":"Wardat M, Le W, Rajan H (2021) DeepLocalize: Fault Localization for Deep Neural Networks. In: 2021 IEEE\/ACM 43rd International conference on software engineering (ICSE), pp 251\u2013262, https:\/\/doi.org\/10.1109\/ICSE43902.2021.00034","DOI":"10.1109\/ICSE43902.2021.00034"},{"key":"10521_CR186","unstructured":"Wei J, Bosma M, Zhao VY, Guu K, Yu AW, Lester B, Du N, Dai AM, Le QV (2021) Finetuned language models are zero-shot learners. arXiv:2109.01652"},{"key":"10521_CR187","doi-asserted-by":"crossref","unstructured":"Wei Z, Wang H, Yang Z, Chan W (2022) Sebox4dl: a modular software engineering toolbox for deep learning models. In: Proceedings of the ACM\/IEEE 44th International conference on software engineering: companion proceedings, pp 193\u2013196","DOI":"10.1109\/ICSE-Companion55297.2022.9793795"},{"key":"10521_CR188","doi-asserted-by":"crossref","unstructured":"Willemink MJ, Koszek WA, Hardell C, Wu J, Fleischmann D, Harvey H, Folio LR, Summers RM, Rubin DL, Lungren MP (2020) Preparing Medical Imaging Data for Machine Learning. Radiological Society of North America","DOI":"10.1148\/radiol.2020192224"},{"key":"10521_CR189","doi-asserted-by":"crossref","unstructured":"Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Le\u00a0Scao T, Gugger S, Drame M, Lhoest Q, Rush A (2020) Transformers: State-of-the-Art Natural Language Processing. In: Conference on empirical methods in natural language processing: system demonstrations","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"10521_CR190","unstructured":"Wu C, Yin S, Qi W, Wang X, Tang Z, Duan N (2023) Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv:2303.04671"},{"key":"10521_CR191","unstructured":"Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K et\u00a0al (2016) Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv"},{"key":"10521_CR192","doi-asserted-by":"crossref","unstructured":"Xin D, Miao H, Parameswaran A, Polyzotis N (2021) Production machine learning pipelines: Empirical analysis and optimization opportunities. In: Proceedings of the 2021 international conference on management of data, pp 2639\u20132652","DOI":"10.1145\/3448016.3457566"},{"key":"10521_CR193","doi-asserted-by":"crossref","unstructured":"Xu S, Wang J, Shou W, Ngo T, Sadick AM, Wang X (2021) Computer Vision Techniques in Construction: A Critical Review. Archives of Computational Methods in Engineering","DOI":"10.1007\/s11831-020-09504-3"},{"key":"10521_CR194","doi-asserted-by":"crossref","unstructured":"Yang J, Yan C, Wan C, Lu S, Cheung A (2019) View-centric performance optimization for database-backed web applications. In: 2019 IEEE\/ACM 41st International conference on software engineering (ICSE), IEEE, pp 994\u20131004","DOI":"10.1109\/ICSE.2019.00104"},{"key":"10521_CR195","unstructured":"You K, Liu Y, Wang J, Jordan MI, Long M (2021a) Ranking and Tuning Pre-trained Models: A New Paradigm of Exploiting Model Hubs. J Machine Learn Res (JMLR) 23(1):9400\u20139446, arXiv:2110.10545"},{"key":"10521_CR196","unstructured":"You K, Liu Y, Wang J, Long M (2021b) LogME: Practical Assessment of Pre-trained Models for Transfer Learning. In: International conference on machine learning (ICML), PMLR, pp 12133\u201312143, https:\/\/proceedings.mlr.press\/v139\/you21b.html"},{"key":"10521_CR197","unstructured":"Yuan L, Chen D, Chen YL, Codella N, Dai X, Gao J, Hu H, Huang X, Li B, Li C et\u00a0al (2021) Florence: A new foundation model for computer vision. arXiv:2111.11432"},{"key":"10521_CR198","unstructured":"Yuan Y (2023) On the power of foundation models. In: International conference on machine learning, PMLR, pp 40519\u201340530"},{"issue":"1","key":"10521_CR199","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TSE.2019.2962027","volume":"48","author":"JM Zhang","year":"2020","unstructured":"Zhang JM, Harman M, Ma L, Liu Y (2020) Machine learning testing: Survey, landscapes and horizons. IEEE Trans Softw Eng 48(1):1\u201336","journal-title":"IEEE Trans Softw Eng"},{"key":"10521_CR200","doi-asserted-by":"crossref","unstructured":"Zhang R, Xiao W, Zhang H, Liu Y, Lin H, Yang M (2020b) An empirical study on program failures of deep learning jobs. In: International conference on software engineering (ICSE)","DOI":"10.1145\/3377811.3380362"},{"key":"10521_CR201","doi-asserted-by":"crossref","unstructured":"Zhang T, Gao C, Ma L, Lyu M, Kim M (2019) An Empirical Study of Common Challenges in Developing Deep Learning Applications. In: International symposium on software reliability engineering (ISSRE)","DOI":"10.1109\/ISSRE.2019.00020"},{"key":"10521_CR202","doi-asserted-by":"crossref","unstructured":"Zhang X, Liu J, Sun N, Fang C, Liu J, Wang J, Chai D, Chen Z (2021) Duo: Differential Fuzzing for Deep Learning Operators. IEEE Trans Reliability","DOI":"10.1109\/TR.2021.3107165"},{"key":"10521_CR203","doi-asserted-by":"crossref","unstructured":"Zhang Y, Chen Y, Cheung SC, Xiong Y, Zhang L (2018) An empirical study on TensorFlow program bugs. Int Symposium Soft Testing Anal (ISSTA)","DOI":"10.1145\/3213846.3213866"},{"key":"10521_CR204","doi-asserted-by":"crossref","unstructured":"Zhang Y, Ren L, Chen L, Xiong Y, Cheung SC, Xie T (2020c) Detecting numerical bugs in neural network architectures. European software engineering conference and symposium on the foundations of software engineering (ESEC\/FSE)","DOI":"10.1145\/3368089.3409720"},{"key":"10521_CR205","unstructured":"Zhou C, Li Q, Li C, Yu J, Liu Y, Wang G, Zhang K, Ji C, Yan Q, He L et\u00a0al (2023) A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv:2302.09419"},{"issue":"1","key":"10521_CR206","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1109\/TSMC.2022.3166397","volume":"53","author":"L Zhou","year":"2022","unstructured":"Zhou L, Zhang L, Konz N (2022) Computer vision techniques in manufacturing. IEEE Trans Syst, Man, and Cybernetics: Syst 53(1):105\u2013117","journal-title":"IEEE Trans Syst, Man, and Cybernetics: Syst"},{"key":"10521_CR207","unstructured":"Zou X, Yang J, Zhang H, Li F, Li L, Gao J, Lee YJ (2023) Segment Everything Everywhere All at Once. arXiv:2304.06718"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10521-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10521-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10521-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,6]],"date-time":"2024-11-06T04:50:04Z","timestamp":1730868604000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10521-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,20]]},"references-count":207,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["10521"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10521-0","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,20]]},"assertion":[{"value":"25 June 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 August 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This work was supported by Google and Cisco and by NSF awards #2107230, #2229703, #2107020, and #2104319. We acknowledge that Google has an interest in promoting the use of TensorFlow, hence their investment in the TensorFlow Model Garden. Although this interest enabled the collection of some of the data (the student team), Google did not otherwise engage in the project. No Google employees participated in the project, neither as subjects nor as researchers.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosure of potential conflicts of interest"}},{"value":"Our human subjects work (interviews) followed a protocol that was approved by Purdue University\u2019s Institutional Review Board (IRB). The IRB protocol number is #<i>IRB-2021-366<\/i>.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Research involving Human Participants"}}],"article-number":"142"}}