{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T02:50:25Z","timestamp":1777344625396,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":59,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF (National Science Foundation)","doi-asserted-by":"publisher","award":["CNS-21-20448, CCF-19-34884"],"award-info":[{"award-number":["CNS-21-20448, CCF-19-34884"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,11,7]]},"DOI":"10.1145\/3540250.3549088","type":"proceedings-article","created":{"date-parts":[[2022,11,9]],"date-time":"2022-11-09T20:46:22Z","timestamp":1668026782000},"page":"734-746","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":31,"title":["23 shades of self-admitted technical debt: an empirical study on machine learning software"],"prefix":"10.1145","author":[{"given":"David","family":"OBrien","sequence":"first","affiliation":[{"name":"Iowa State University, USA"}]},{"given":"Sumon","family":"Biswas","sequence":"additional","affiliation":[{"name":"Iowa State University, USA"}]},{"given":"Sayem","family":"Imtiaz","sequence":"additional","affiliation":[{"name":"Iowa State University, USA"}]},{"given":"Rabe","family":"Abdalkareem","sequence":"additional","affiliation":[{"name":"Carleton University, Canada"}]},{"given":"Emad","family":"Shihab","sequence":"additional","affiliation":[{"name":"Concordia University, Canada"}]},{"given":"Hridesh","family":"Rajan","sequence":"additional","affiliation":[{"name":"Iowa State University, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,11,9]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3387906.3388630"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/MTD.2014.9"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2901742"},{"key":"e_1_3_2_1_5_1","volume-title":"The Open-Closed Principle of Modern Machine Learning Frameworks. In 2018 IEEE\/ACM 15th International Conference on Mining Software Repositories (MSR). 353\u2013363","author":"Braiek Houssem Ben","year":"2018","unstructured":"Houssem Ben Braiek , Foutse Khomh , and Bram Adams . 2018 . The Open-Closed Principle of Modern Machine Learning Frameworks. In 2018 IEEE\/ACM 15th International Conference on Mining Software Repositories (MSR). 353\u2013363 . Houssem Ben Braiek, Foutse Khomh, and Bram Adams. 2018. The Open-Closed Principle of Modern Machine Learning Frameworks. In 2018 IEEE\/ACM 15th International Conference on Mining Software Repositories (MSR). 353\u2013363."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2009.5069475"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00086"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409704"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468536"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510057"},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of IEEE Big Data.","author":"Breck Eric","unstructured":"Eric Breck , Shanqing Cai , Eric Nielsen , Michael Salib , and D. Sculley . 2017. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction . In Proceedings of IEEE Big Data. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. 2017. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. In Proceedings of IEEE Big Data."},{"key":"e_1_3_2_1_12_1","volume-title":"Managing Technical Debt in Software-reliant Systems. In FSE\/SDP Workshop on Future of Software Engineering Research. ACM, 47\u201352","author":"Brown N.","unstructured":"N. Brown , Y. Cai , Y. Guo , R. Kazman , M. Kim , P. Kruchten , E. Lim , A. MacCormack , R. Nord , I. Ozkaya , R. Sangwan , C. Seaman , K. Sullivan , and N. Zazworka . 2010 . Managing Technical Debt in Software-reliant Systems. In FSE\/SDP Workshop on Future of Software Engineering Research. ACM, 47\u201352 . N. Brown, Y. Cai, Y. Guo, R. Kazman, M. Kim, P. Kruchten, E. Lim, A. MacCormack, R. Nord, I. Ozkaya, R. Sangwan, C. Seaman, K. Sullivan, and N. Zazworka. 2010. Managing Technical Debt in Software-reliant Systems. In FSE\/SDP Workshop on Future of Software Engineering Research. ACM, 47\u201352."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453478"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510225"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606588"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2803171"},{"key":"e_1_3_2_1_17_1","volume-title":"Proceedings of the 36th International Conference on Software Engineering (ICSE\u201914)","author":"Dyer Robert","unstructured":"Robert Dyer , Hridesh Rajan , Hoan Anh Nguyen , and Tien N. Nguyen . 2014. Mining Billions of AST Nodes to Study Actual and Potential Usage of Java Language Features . In Proceedings of the 36th International Conference on Software Engineering (ICSE\u201914) . 779\u2013790. Robert Dyer, Hridesh Rajan, Hoan Anh Nguyen, and Tien N. Nguyen. 2014. Mining Billions of AST Nodes to Study Actual and Potential Usage of Java Language Features. In Proceedings of the 36th International Conference on Software Engineering (ICSE\u201914). 779\u2013790."},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 12th International Conference on Generative Programming: Concepts & Experiences (GPCE). 23\u201332","author":"Dyer Robert","unstructured":"Robert Dyer , Hridesh Rajan , and Tien N. Nguyen . 2013. Declarative Visitors to Ease Fine-grained Source Code Mining with Full History on Billions of AST Nodes . In Proceedings of the 12th International Conference on Generative Programming: Concepts & Experiences (GPCE). 23\u201332 . Robert Dyer, Hridesh Rajan, and Tien N. Nguyen. 2013. Declarative Visitors to Ease Fine-grained Source Code Mining with Full History on Billions of AST Nodes. In Proceedings of the 12th International Conference on Generative Programming: Concepts & Experiences (GPCE). 23\u201332."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786848"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642982"},{"key":"e_1_3_2_1_21_1","volume-title":"The measurement of interrater agreement. Statistical methods for rates and proportions, 2, 212-236","author":"Fleiss Joseph L","year":"1981","unstructured":"Joseph L Fleiss , Bruce Levin , and Myunghee Cho Paik . 1981. The measurement of interrater agreement. Statistical methods for rates and proportions, 2, 212-236 ( 1981 ), 22\u201323. Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 1981. The measurement of interrater agreement. Statistical methods for rates and proportions, 2, 212-236 (1981), 22\u201323."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00022"},{"key":"e_1_3_2_1_23_1","volume-title":"2022 IEEE\/ACM 1st International Conference on AI Engineering \u2013 Software Engineering for AI (CAIN). IEEE Computer Society","author":"Foidl H.","unstructured":"H. Foidl , M. Felderer , and R. Ramler . 2022. Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems . In 2022 IEEE\/ACM 1st International Conference on AI Engineering \u2013 Software Engineering for AI (CAIN). IEEE Computer Society , Los Alamitos, CA, USA. 229\u2013239. https:\/\/doi.ieeecomputersociety.org\/ H. Foidl, M. Felderer, and R. Ramler. 2022. Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systems. In 2022 IEEE\/ACM 1st International Conference on AI Engineering \u2013 Software Engineering for AI (CAIN). IEEE Computer Society, Los Alamitos, CA, USA. 229\u2013239. https:\/\/doi.ieeecomputersociety.org\/"},{"key":"e_1_3_2_1_24_1","volume-title":"International Workshop on Managing Technical Debt. IEEE, 15\u201322","author":"Fontana F. A.","unstructured":"F. A. Fontana , V. Ferme , and S. Spinelli . 2012. Investigating the impact of code smells debt on quality code evaluation . In International Workshop on Managing Technical Debt. IEEE, 15\u201322 . F. A. Fontana, V. Ferme, and S. Spinelli. 2012. Investigating the impact of code smells debt on quality code evaluation. In International Workshop on Managing Technical Debt. IEEE, 15\u201322."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00052"},{"key":"e_1_3_2_1_26_1","volume-title":"Proceedings of the 17th International Conference on Mining Software Repositories.","author":"Gonzalez Danielle","unstructured":"Danielle Gonzalez , T. Zimmermann , and N. Nagappan . 2020. The State of the ML-universe: 10 Years of Artificial Intelligence & Machine Learning Software Development on GitHub . Proceedings of the 17th International Conference on Mining Software Repositories. Danielle Gonzalez, T. Zimmermann, and N. Nagappan. 2020. The State of the ML-universe: 10 Years of Artificial Intelligence & Machine Learning Software Development on GitHub. Proceedings of the 17th International Conference on Mining Software Repositories."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9522-4"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_2_1_29_1","unstructured":"Nick Hynes D. Sculley and Michael Terry. 2017. The Data Linter: Lightweight Automated Sanity Checking for ML Data Sets. http:\/\/learningsys.org\/nips17\/assets\/papers\/paper_19.pdf \t\t\t\t  Nick Hynes D. Sculley and Michael Terry. 2017. The Data Linter: Lightweight Automated Sanity Checking for ML Data Sets. http:\/\/learningsys.org\/nips17\/assets\/papers\/paper_19.pdf"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380378"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507288.2507326"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110827"},{"key":"e_1_3_2_1_34_1","unstructured":"Jiakun Liu Qiao Huang Xin Xia Emad Shihab David Lo and Shanping Li. 2020. Is Using Deep Learning Frameworks Free? Characterizing Technical Debt in Deep Learning Frameworks. ICSE-SEIS \u201920. 1\u201310. \t\t\t\t  Jiakun Liu Qiao Huang Xin Xia Emad Shihab David Lo and Shanping Li. 2020. Is Using Deep Learning Frameworks Free? Characterizing Technical Debt in Deep Learning Frameworks. ICSE-SEIS \u201920. 1\u201310."},{"key":"e_1_3_2_1_35_1","volume-title":"Wait For It: Identifying \"On-Hold\" Self-Admitted Technical Debt. CoRR, abs\/1901.09511","author":"Maipradit Rungroj","year":"2019","unstructured":"Rungroj Maipradit , Christoph Treude , Hideaki Hata , and Kenichi Matsumoto . 2019. Wait For It: Identifying \"On-Hold\" Self-Admitted Technical Debt. CoRR, abs\/1901.09511 ( 2019 ), arXiv:1901.09511. arxiv:1901.09511 Rungroj Maipradit, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2019. Wait For It: Identifying \"On-Hold\" Self-Admitted Technical Debt. CoRR, abs\/1901.09511 (2019), arXiv:1901.09511. arxiv:1901.09511"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2654244"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.8"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510052"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409668"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510051"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00077"},{"key":"e_1_3_2_1_42_1","volume-title":"An Exploratory Study on Self-Admitted Technical Debt. In International Conference on Software Maintenance and Evolution. IEEE Computer Society, 91\u2013100","author":"Potdar Aniket","year":"2014","unstructured":"Aniket Potdar and Emad Shihab . 2014 . An Exploratory Study on Self-Admitted Technical Debt. In International Conference on Software Maintenance and Evolution. IEEE Computer Society, 91\u2013100 . Aniket Potdar and Emad Shihab. 2014. An Exploratory Study on Self-Admitted Technical Debt. In International Conference on Software Maintenance and Evolution. IEEE Computer Society, 91\u2013100."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324916"},{"key":"e_1_3_2_1_44_1","volume-title":"Machine Learning: The High Interest Credit Card of Technical Debt. In SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop).","author":"Sculley D.","year":"2014","unstructured":"D. Sculley , Gary Holt , Daniel Golovin , Eugene Davydov , Todd Phillips , Dietmar Ebner , Vinay Chaudhary , and Michael Young . 2014 . Machine Learning: The High Interest Credit Card of Technical Debt. In SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop). D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, and Michael Young. 2014. Machine Learning: The High Interest Credit Card of Technical Debt. In SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop)."},{"key":"e_1_3_2_1_45_1","unstructured":"D. Sculley Gary Holt D. Golovin Eugene Davydov Todd Phillips D. Ebner Vinay Chaudhary Michael Young J. Crespo and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In NIPS. \t\t\t\t  D. Sculley Gary Holt D. Golovin Eugene Davydov Todd Phillips D. Ebner Vinay Chaudhary Michael Young J. Crespo and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In NIPS."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.799955"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-385512-1.00002-5"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.02.056"},{"key":"e_1_3_2_1_49_1","volume-title":"Marco Tulio Valente, and Ricardo Terra","author":"Oliveira Silva Marcelino Campos","year":"2016","unstructured":"Marcelino Campos Oliveira Silva , Marco Tulio Valente, and Ricardo Terra . 2016 . Does technical debt lead to the rejection of pull requests? arXiv preprint arXiv:1604.01450. Marcelino Campos Oliveira Silva, Marco Tulio Valente, and Ricardo Terra. 2016. Does technical debt lead to the rejection of pull requests? arXiv preprint arXiv:1604.01450."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3464968.3468408"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106738"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00033"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/IS.2018.8710521"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2016.72"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510071"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00034"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387459"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196423"},{"key":"e_1_3_2_1_59_1","volume-title":"Investigating the Impact of Design Debt on Software Quality. In International Workshop on Managing Technical Debt. ACM, 17\u201323","author":"Zazworka Nico","year":"2011","unstructured":"Nico Zazworka , Michele A. Shaw , Forrest Shull , and Carolyn Seaman . 2011 . Investigating the Impact of Design Debt on Software Quality. In International Workshop on Managing Technical Debt. ACM, 17\u201323 . Nico Zazworka, Michele A. Shaw, Forrest Shull, and Carolyn Seaman. 2011. Investigating the Impact of Design Debt on Software Quality. In International Workshop on Managing Technical Debt. ACM, 17\u201323."}],"event":{"name":"ESEC\/FSE '22: 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Singapore Singapore","acronym":"ESEC\/FSE '22","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","NUS NUS"]},"container-title":["Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3540250.3549088","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3540250.3549088","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3540250.3549088","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:51:01Z","timestamp":1750182661000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3540250.3549088"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":59,"alternative-id":["10.1145\/3540250.3549088","10.1145\/3540250"],"URL":"https:\/\/doi.org\/10.1145\/3540250.3549088","relation":{},"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}