{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T16:23:43Z","timestamp":1780331023100,"version":"3.54.1"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,6,28]],"date-time":"2024-06-28T00:00:00Z","timestamp":1719532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62302430 and No. 62202341"],"award-info":[{"award-number":["No. 62302430 and No. 62202341"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LQ24F020017"],"award-info":[{"award-number":["LQ24F020017"]}]},{"name":"ARC Laureate Fellowship","award":["FL190100035"],"award-info":[{"award-number":["FL190100035"]}]},{"name":"Zhejiang Province \u201cJianBingLingYan+X\u201d Research and Development Plan","award":["2024C01114"],"award-info":[{"award-number":["2024C01114"]}]},{"name":"Joint Funds of the Zhejiang Provincial Natural Science Foundation of China","award":["LHZSD24F020001"],"award-info":[{"award-number":["LHZSD24F020001"]}]},{"name":"Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study","award":["SN-ZJU-SIAS-001"],"award-info":[{"award-number":["SN-ZJU-SIAS-001"]}]},{"name":"Shanghai Sailing Program","award":["23YF1446900"],"award-info":[{"award-number":["23YF1446900"]}]},{"name":"Zhejiang Provincial Engineering Research Center for Real-time SmartTech in Urban Security Governance"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>Software development is a collaborative process that involves various interactions among individuals and teams. TODO comments in source code play a critical role in managing and coordinating diverse tasks during this process. However, this study finds that a large proportion of open-source project TODO comments are left unresolved or take a long time to be resolved. About 46.7% of TODO comments in open-source repositories are of low-quality (e.g., TODOs that are ambiguous, lack information, or are useless to developers). This highlights the need for better TODO practices. In this study, we investigate four aspects regarding the quality of TODO comments in open-source projects: (1) the prevalence of low-quality TODO comments; (2) the key characteristics of high-quality TODO comments; (3) how are TODO comments of different quality managed in practice; and (4) the feasibility of automatically assessing TODO comment quality. Examining 2,863 TODO comments from Top100 GitHub Java repositories, we propose criteria to identify high-quality TODO comments and provide insights into their optimal composition. We discuss the lifecycle of TODO comments with varying quality. To assist developers, we construct deep learning-based methods that show promising performance in identifying the quality of TODO comments, potentially enhancing development efficiency and code quality.<\/jats:p>","DOI":"10.1145\/3664811","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T11:34:28Z","timestamp":1715600068000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["What Makes a Good TODO Comment?"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3314-0427","authenticated-orcid":false,"given":"Haoye","family":"Wang","sequence":"first","affiliation":[{"name":"School of Computer &amp; Computing Science, Hangzhou City University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3030-9917","authenticated-orcid":false,"given":"Zhipeng","family":"Gao","sequence":"additional","affiliation":[{"name":"Shanghai Institute for Advanced Study of Zhejiang University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2748-1249","authenticated-orcid":false,"given":"Tingting","family":"Bi","sequence":"additional","affiliation":[{"name":"The University of Western Australia, Perth Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4928-7076","authenticated-orcid":false,"given":"John","family":"Grundy","sequence":"additional","affiliation":[{"name":"Software Systems and Cybersecurity, Monash University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5507-6569","authenticated-orcid":false,"given":"Xinyu","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8179-7119","authenticated-orcid":false,"given":"Minghui","family":"Wu","sequence":"additional","affiliation":[{"name":"Hangzhou City University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4111-4189","authenticated-orcid":false,"given":"Xiaohu","family":"Yang","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,6,28]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","unstructured":"Zenodo. 2023. What Makes a Good TODO Comment. Retrieved from https:\/\/zenodo.org\/records\/1087800210.5281\/zenodo.7898592","DOI":"10.5281\/zenodo.7898592"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340571"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10183-w"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510125"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2008.12.036"},{"key":"e_1_3_3_8_2","volume-title":"Convolutional neural network for sentence classification","author":"Chen Yahui","year":"2015","unstructured":"Yahui Chen. 2015. Convolutional neural network for sentence classification. Master\u2019s thesis. University of Waterloo."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1085313.1085331"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2009.5090040"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3565971"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCKE60553.2023.10326279"},{"key":"e_1_3_3_14_2","article-title":"Codebert: A pre-trained model for programming and natural languages","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et\u00a0al. 2020. Codebert: A pre-trained model for programming and natural languages. Retrieved from https:\/\/arXiv:2002.08155","journal-title":"R"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","unstructured":"Jerome H. Friedman. 2001. Greedy function approximation: A gradient boosting machine. The Annals of Statistics 29 5 (2001) 1189\u20131232.","DOI":"10.1214\/aos\/1013203451"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00070"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3522674"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3401026"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3412845"},{"key":"e_1_3_3_21_2","article-title":"Automating the removal of obsolete TODO comments","author":"Gao Zhipeng","year":"2021","unstructured":"Zhipeng Gao, Xin Xia, David Lo, John Grundy, and Thomas Zimmermann. 2021. Automating the removal of obsolete TODO comments. Retrieved from https:\/\/arXiv:2108.05846","journal-title":"R"},{"key":"e_1_3_3_22_2","article-title":"The state of open source software","year":"2023","unstructured":"GitHub. 2023. The state of open source software. Retrieved from https:\/\/octoverse.github.com\/","journal-title":"R"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2011.22"},{"key":"e_1_3_3_24_2","unstructured":"Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 2 (1979) 65\u201370."},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9522-4"},{"key":"e_1_3_3_26_2","article-title":"TODO Bot","year":"2022","unstructured":"JasonEtco. 2022. TODO Bot. Retrieved from https:\/\/github.com\/apps\/todo","journal-title":"R"},{"key":"e_1_3_3_27_2","first-page":"68","volume-title":"Proceedings of the International Workshop on Quantitative Approaches to Software Quality and International Workshop on Technical Debt Analytics at the Asia-Pacific Software Engineering Conference (QuASoQ\/TDA@ APSEC\u201916)","author":"Kamei Yasutaka","year":"2016","unstructured":"Yasutaka Kamei, Everton da S. Maldonado, Emad Shihab, and Naoyasu Ubayashi. 2016. Using analytics to quantify interest of self-admitted technical debt. In Proceedings of the International Workshop on Quantitative Approaches to Software Quality and International Workshop on Technical Debt Analytics at the Asia-Pacific Software Engineering Conference (QuASoQ\/TDA@ APSEC\u201916). 68\u201371."},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13881-2_7"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106257"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2014.12.027"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-023-2771-z"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00040"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2018.00028"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416581"},{"key":"e_1_3_3_35_2","article-title":"All your TODO comments in one place","author":"LLC Innobuilt Software","year":"2019","unstructured":"Innobuilt Software LLC. 2019. All your TODO comments in one place. Retrieved from https:\/\/imdone.io","journal-title":"R"},{"key":"e_1_3_3_36_2","article-title":"Nltk: The natural language toolkit","author":"Loper Edward","year":"2002","unstructured":"Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. Retrieved from https:\/\/cs\/0205028","journal-title":"R"},{"key":"e_1_3_3_37_2","unstructured":"Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. ArXiv abs\/1711.05101 (2017)."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.8"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/MTD.2015.7332619"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2904957"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3528228.3528408"},{"key":"e_1_3_3_42_2","volume-title":"Proceedings of the Workshops at the 32nd AAAI Conference on Artificial Intelligence","author":"Nie Pengyu","year":"2018","unstructured":"Pengyu Nie, Junyi Jessy Li, Sarfraz Khurshid, Raymond Mooney, and Milos Gligoric. 2018. Natural language processing and program analysis for supporting todo comments as software evolves. In Proceedings of the Workshops at the 32nd AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338965"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/WICSA-ECSA.212.17"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.4249\/scholarpedia.1883"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2014.22"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.31"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3519312"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635922"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324916"},{"key":"e_1_3_3_51_2","article-title":"scikit-learn","author":"package The replicate","year":"2023","unstructured":"The replicate package. 2023. scikit-learn. Retrieved from https:\/\/scikit-learn.org\/stable\/","journal-title":"R"},{"key":"e_1_3_3_52_2","first-page":"7","article-title":"Multi layer perceptron","author":"Riedmiller Martin","year":"2014","unstructured":"Martin Riedmiller and A. Lernen. 2014. Multi layer perceptron. Machine Learning Lab Special Lecture, University of Freiburg (2014), 7\u201324.","journal-title":"Machine Learning Lab Special Lecture, University of Freiburg"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528469"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/78.650093"},{"key":"e_1_3_3_55_2","volume-title":"Introduction to Information Retrieval","author":"Sch\u00fctze Hinrich","year":"2008","unstructured":"Hinrich Sch\u00fctze, Christopher D. Manning, and Prabhakar Raghavan. 2008. Introduction to Information Retrieval. Vol. 39. Cambridge University Press, Cambridge, UK."},{"key":"e_1_3_3_56_2","unstructured":"Justin Searls. 2019. todo_or_die. Retrieved from https:\/\/github.com\/searls\/todo_or_die"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/2856636.2856638"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2013.6613836"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368123"},{"key":"e_1_3_3_60_2","unstructured":"Adam Svensson. 2015. Reducing outdated and inconsistent code comments during software development: The comment validator program. https:\/\/api.semanticscholar.org\/CorpusID:61830050"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/1294261.1294276"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510205"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.7653"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2016.72"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4380-9_16"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387459"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3093761"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2017.8115681"},{"key":"e_1_3_3_70_2","doi-asserted-by":"crossref","unstructured":"Meng Yan Xin Xia Yuanrui Fan Ahmed E. Hassan David Lo and Shanping Li. 2020. Just-in-time defect identification and localization: A two-phase framework. IEEE Transactions on Software Engineering 48 1 (2020) 82\u2013101.","DOI":"10.1109\/TSE.2020.2978819"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3312662.3312710"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524610.3528387"},{"issue":"4","key":"e_1_3_3_73_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1082983.1083152","article-title":"Source code that talks: An exploration of Eclipse task comments and their implication to repository mining","volume":"30","author":"Ying Annie T. T.","year":"2005","unstructured":"Annie T. T. Ying, James L. Wright, and Steven Abrams. 2005. Source code that talks: An exploration of Eclipse task comments and their implication to repository mining. ACM SIGSOFT Softw. Eng. Notes 30, 4 (2005), 1\u20135.","journal-title":"ACM SIGSOFT Softw. Eng. Notes"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2019.8870174"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10031-3"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196423"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER48275.2020.9054868"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/2460999.2461005"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2018.00032"},{"key":"e_1_3_3_80_2","article-title":"A sensitivity analysis of (and practitioners\u2019 guide to) convolutional neural networks for sentence classification","author":"Zhang Ye","year":"2015","unstructured":"Ye Zhang and Byron Wallace. 2015. A sensitivity analysis of (and practitioners\u2019 guide to) convolutional neural networks for sentence classification. Retrieved from https:\/\/arXiv:1510.03820","journal-title":"R"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678720"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.11"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664811","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3664811","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:29Z","timestamp":1750295849000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664811"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,28]]},"references-count":81,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3664811"],"URL":"https:\/\/doi.org\/10.1145\/3664811","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,28]]},"assertion":[{"value":"2023-11-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}