{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T04:58:57Z","timestamp":1774673937570,"version":"3.50.1"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2020,8,4]],"date-time":"2020-08-04T00:00:00Z","timestamp":1596499200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,4]],"date-time":"2020-08-04T00:00:00Z","timestamp":1596499200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["PE17039"],"award-info":[{"award-number":["PE17039"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2020,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Self-admitted technical debt refers to situations where a software developer knows that their current implementation is not optimal and indicates this using a source code comment. In this work, we hypothesize that it is possible to develop automated techniques to understand a subset of these comments in more detail, and to propose tool support that can help developers manage self-admitted technical debt more effectively. Based on a qualitative study of 333 comments indicating self-admitted technical debt, we first identify one particular class of debt amenable to automated management: on-hold self-admitted technical debt (on-hold SATD), i.e., debt which contains a condition to indicate that a developer is waiting for a certain event or an updated functionality having been implemented elsewhere. We then design and evaluate an automated classifier which can identify these on-hold instances with an area under the receiver operating characteristic curve (AUC) of 0.98 as well as detect the specific conditions that developers are waiting for. Our work presents a first step towards automated tool support that is able to indicate when certain instances of self-admitted technical debt are ready to be addressed.<\/jats:p>","DOI":"10.1007\/s10664-020-09854-3","type":"journal-article","created":{"date-parts":[[2020,8,4]],"date-time":"2020-08-04T01:02:26Z","timestamp":1596502946000},"page":"3770-3798","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Wait for it: identifying \u201cOn-Hold\u201d self-admitted technical debt"],"prefix":"10.1007","volume":"25","author":[{"given":"Rungroj","family":"Maipradit","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6919-2149","authenticated-orcid":false,"given":"Christoph","family":"Treude","sequence":"additional","affiliation":[]},{"given":"Hideaki","family":"Hata","sequence":"additional","affiliation":[]},{"given":"Kenichi","family":"Matsumoto","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,8,4]]},"reference":[{"issue":"1","key":"9854_CR1","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/S1570-8667(03)00065-0","volume":"2","author":"MI Abouelhoda","year":"2004","unstructured":"Abouelhoda MI, Kurtz S, Ohlebusch E (2004) Replacing suffix trees with enhanced suffix arrays. Journal of Discrete Algorithms 2(1):53\u201386","journal-title":"Journal of Discrete Algorithms"},{"key":"9854_CR2","doi-asserted-by":"crossref","unstructured":"Al Omran FNA, Treude C (2017) Choosing an nlp library for analyzing software documentation: A systematic literature review and a series of experiments. In: Proceedings of the international conference on mining software repositories, pp 187\u2013197","DOI":"10.1109\/MSR.2017.42"},{"key":"9854_CR3","doi-asserted-by":"crossref","unstructured":"Bavota G, Russo B (2016) A large-scale empirical study on self-admitted technical debt. In: Proceedings of the working conference on mining software repositories, pp 315\u2013326","DOI":"10.1145\/2901739.2901742"},{"key":"9854_CR4","doi-asserted-by":"crossref","unstructured":"Bazrafshan S, Koschke R (2013) An empirical study of clone removals. In: Proceedings of the international conference on software maintenance, pp 50\u201359","DOI":"10.1109\/ICSM.2013.16"},{"key":"9854_CR5","doi-asserted-by":"crossref","unstructured":"de Freitas Farias MA, de Mendon\u00e7a Neto MG, da Silva AB, Sp\u00ednola RO (2015) A contextualized vocabulary model for identifying technical debt on code comments. In: 2015 IEEE 7th international workshop on managing technical debt (MTD), pp 25\u201332","DOI":"10.1109\/MTD.2015.7332621"},{"key":"9854_CR6","doi-asserted-by":"crossref","unstructured":"de Freitas Farias MA, Santos JA, Kalinowski M, Mendon\u00e7a M, Sp\u00ednola R O (2016) Investigating the identification of technical debt through code comment analysis. In: International Conference onEnterprise Information Systems, Springer, pp 284\u2013309","DOI":"10.1007\/978-3-319-62386-3_14"},{"key":"9854_CR7","doi-asserted-by":"crossref","unstructured":"Ernst NA, Bellomo S, Ozkaya I, Nord RL, Gorton I (2015) Measure it? Manage it? Ignore it? Software practitioners and technical debt. In: Proceedings of the joint meeting on foundations of software engineering, pp 50\u201360","DOI":"10.1145\/2786805.2786848"},{"key":"9854_CR8","unstructured":"Feurer M, Klein A, Eggensperger K, Springenberg J, Blum M, Hutter F (2015) Efficient and robust automated machine learning. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems 28, Curran Associates, Inc., pp 2962\u20132970"},{"key":"9854_CR9","doi-asserted-by":"publisher","first-page":"106475","DOI":"10.1109\/ACCESS.2019.2933318","volume":"7","author":"J Flisar","year":"2019","unstructured":"Flisar J, Podgorelec V (2019) Identification of self-admitted technical debt using enhanced feature selection based on word embedding. IEEE Access 7:106475\u2013106494","journal-title":"IEEE Access"},{"key":"9854_CR10","unstructured":"Honnibal M, Montani I (2017) Spacy - industrial-strength natural language processing in python. https:\/\/spacy.io\/, (Accessed on 13\/04\/2019)"},{"issue":"1","key":"9854_CR11","doi-asserted-by":"publisher","first-page":"418","DOI":"10.1007\/s10664-017-9522-4","volume":"23","author":"Q Huang","year":"2018","unstructured":"Huang Q, Shihab E, Xia X, Lo D, Li S (2018) Identifying self-admitted technical debt in open source projects using text mining. Empir Softw Eng 23(1):418\u2013451","journal-title":"Empir Softw Eng"},{"key":"9854_CR12","doi-asserted-by":"crossref","unstructured":"Ichinose T, Uemura K, Tanaka D, Hata H, Iida H, Matsumoto K (2016) ROCAT on KATARIBE: Code visualization for communities. In: Proceedings of the international conference on applied computing and information technology, pp 158\u2013163","DOI":"10.1109\/ACIT-CSII-BCD.2016.040"},{"key":"9854_CR13","first-page":"68","volume":"1771","author":"Y Kamei","year":"2016","unstructured":"Kamei Y, Maldonado E, Shihab E, Ubayashi N (2016) Using analytics to quantify the interest of self-admitted technical debt. CEUR Workshop Proceedings 1771:68\u201371","journal-title":"CEUR Workshop Proceedings"},{"key":"9854_CR14","unstructured":"Kniberg H (2013) Good and bad technical debt (and how TDD helps). http:\/\/blog.crisp.se\/2013\/10\/11\/henrikkniberg\/good-and-bad-technical-debt"},{"issue":"6","key":"9854_CR15","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MS.2012.130","volume":"29","author":"E Lim","year":"2012","unstructured":"Lim E, Taksande N, Seaman C (2012) A balancing act: What software practitioners have to say about technical debt. IEEE Softw 29(6):22\u201327","journal-title":"IEEE Softw"},{"key":"9854_CR16","doi-asserted-by":"crossref","unstructured":"Liu Z, Huang Q, Xia X, Shihab E, Lo D, Li S (2018) SATD detector: a text-mining-based self-admitted technical debt detection tool. In: Proceedings of the 40th international conference on software engineering: companion proceeedings (ICSE \u201918)","DOI":"10.1145\/3183440.3183478"},{"key":"9854_CR17","doi-asserted-by":"crossref","unstructured":"Maldonado E, Shihab E (2015) Detecting and quantifying different types of self-admitted technical debt. In: Proceedings of the international workshop on managing technical debt, pp 9\u201315","DOI":"10.1109\/MTD.2015.7332619"},{"key":"9854_CR18","doi-asserted-by":"crossref","unstructured":"Maldonado E, Abdalkareem R, Shihab E, Serebrenik A (2017a) An empirical study on the removal of self-admitted technical debt. In: Proceedings of the international conference on software maintenance and evolution, pp 238\u2013248","DOI":"10.1109\/ICSME.2017.8"},{"key":"9854_CR19","doi-asserted-by":"crossref","unstructured":"Maldonado E, Shihab E, Tsantalis N (2017b) Using natural language processing to automatically detect self-admitted technical debt. IEEE Trans Softw Eng 43(11):1044\u20131062","DOI":"10.1109\/TSE.2017.2654244"},{"key":"9854_CR20","unstructured":"McConnell S (2007) Technical debt. http:\/\/www.construx.com\/10x_Software_Development\/Technical_Debt\/"},{"key":"9854_CR21","first-page":"72","volume":"1771","author":"S Mensah","year":"2016","unstructured":"Mensah S, Keung J, Bosu M, Bennin K (2016) Rework effort estimation of self-admitted technical debt. CEUR Workshop Proceedings 1771:72\u201375","journal-title":"CEUR Workshop Proceedings"},{"issue":"C","key":"9854_CR22","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.jss.2017.09.026","volume":"135","author":"S Mensah","year":"2018","unstructured":"Mensah S, Keung J, Svajlenko J, Bennin KE, Mi Q (2018) On the value of a prioritization scheme for resolving self-admitted technical debt. J Syst Softw 135(C):37\u201354","journal-title":"J Syst Softw"},{"key":"9854_CR23","doi-asserted-by":"crossref","unstructured":"Palomba F, Zaidman A, Oliveto R, De Lucia A (2017) An exploratory study on the relationship between changes and refactoring. In: Proceedings of the international conference on program comprehension, pp 176\u2013185","DOI":"10.1109\/ICPC.2017.38"},{"key":"9854_CR24","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825\u20132830","journal-title":"J Mach Learn Res"},{"key":"9854_CR25","doi-asserted-by":"crossref","unstructured":"Potdar A, Shihab E (2014) An exploratory study on self-admitted technical debt. In: Proceedings of the international conference on software maintenance and evolution, pp 91\u2013100","DOI":"10.1109\/ICSME.2014.31"},{"key":"9854_CR26","doi-asserted-by":"crossref","unstructured":"Prana GAA, Treude C, Thung F, Atapattu T, Lo D (2019) Categorizing the content of github readme files. Empirical Software Engineering","DOI":"10.1007\/s10664-018-9660-3"},{"key":"9854_CR27","doi-asserted-by":"crossref","unstructured":"Ren X, Xing Z, Xia X, Lo D, Wang X, Grundy J (2019) Neural network based detection of self-admitted technical debt: from performance to explainability. In: ACM transactions on software engineering and methodology (TOSEM)","DOI":"10.1145\/3324916"},{"issue":"5","key":"9854_CR28","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1016\/0306-4573(88)90021-0","volume":"24","author":"G Salton","year":"1988","unstructured":"Salton G, Buckley C (1988) Term-weighting approaches in automatic text retrieval. Inf Process Manag 24(5):513\u2013523","journal-title":"Inf Process Manag"},{"key":"9854_CR29","unstructured":"Shirakawa M (2017) Github - iwnsew\/ngweight: N-gram weighting scheme. https:\/\/github.com\/iwnsew\/ngweight, (Accessed on 04\/13\/2019)"},{"key":"9854_CR30","doi-asserted-by":"crossref","unstructured":"Shirakawa M, Hara T, Nishio S (2015) N-gram idf: A global term weighting scheme based on information distance. In: Proceedings of the international conference on world wide Web, pp 960\u2013970","DOI":"10.1145\/2736277.2741628"},{"issue":"1","key":"9854_CR31","doi-asserted-by":"publisher","first-page":"5:1","DOI":"10.1145\/3052775","volume":"36","author":"M Shirakawa","year":"2017","unstructured":"Shirakawa M, Hara T, Nishio S (2017) Idf for word n-grams. ACM Transactions on Information Systems 36(1):5:1\u20135:38","journal-title":"ACM Transactions on Information Systems"},{"issue":"2","key":"9854_CR32","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s10664-008-9060-1","volume":"13","author":"FJ Shull","year":"2008","unstructured":"Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211\u2013218","journal-title":"Empir Softw Eng"},{"key":"9854_CR33","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1016\/j.jss.2019.02.056","volume":"152","author":"G Sierra","year":"2019","unstructured":"Sierra G, Shihab E, Kamei Y (2019) A survey of self-admitted technical debt. J Syst Softw 152:70\u201382","journal-title":"J Syst Softw"},{"key":"9854_CR34","doi-asserted-by":"crossref","unstructured":"Silva MC, Valente MT, Terra R (2016) Does technical debt lead to the rejection of pull requests?. In: Proceedings of the 12th Brazilian symposium on information systems, ser. SBSI \u201916, pp 248\u2013254","DOI":"10.5753\/sbsi.2016.5969"},{"key":"9854_CR35","doi-asserted-by":"crossref","unstructured":"Terdchanakul P, Hata H, Phannachitta P, Matsumoto K (2017) Bug or not? bug report classification using n-gram idf. In: Proceedings of the international conference on software maintenance and evolution, pp 534\u2013538","DOI":"10.1109\/ICSME.2017.14"},{"issue":"5","key":"9854_CR36","first-page":"360","volume":"37","author":"AJ Viera","year":"2005","unstructured":"Viera AJ, Garrett JM (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360\u2013363","journal-title":"Fam Med"},{"key":"9854_CR37","doi-asserted-by":"crossref","unstructured":"Wattanakriengkrai S, Maipradit R, Hata H, Choetkiertikul M, Sunetnanta T, Matsumoto K (2018) Identifying design and requirement self-admitted technical debt using n-gram idf. In: Proceedings of the international workshop on empirical software engineering in practice, pp 7\u201312","DOI":"10.1109\/IWESEP.2018.00010"},{"key":"9854_CR38","doi-asserted-by":"crossref","unstructured":"Wehaibi S, Shihab E, Guerrouj L (2016) Examining the impact of self-admitted technical debt on software quality. In: Proceedings of the international conference on software analysis, evolution, and reengineering, pp 179\u2013188","DOI":"10.1109\/SANER.2016.72"},{"key":"9854_CR39","doi-asserted-by":"crossref","unstructured":"Yan M, Xia X, Shihab E, Lo D, Yin J, Yang X (2019) Automating change-level self-admitted technical debt determination. IEEE Transactions on Software Engineering","DOI":"10.1109\/TSE.2018.2831232"},{"key":"9854_CR40","doi-asserted-by":"crossref","unstructured":"Zampetti F, Noiseux C, Antoniol G, Khomh F, di Penta M (2017) Recommending when design technical debt should be self-admitted. In: Proceedings of the international conference on software maintenance and evolution, pp 216\u2013226","DOI":"10.1109\/ICSME.2017.44"},{"key":"9854_CR41","doi-asserted-by":"crossref","unstructured":"Zampetti F, Serebrenik A, Di Penta M (2018) Was self-admitted technical debt removal a real removal? an in-depth perspective. In: Proceedings of the international conference on mining software repositories, pp 526\u2013536","DOI":"10.1145\/3196398.3196423"}],"updated-by":[{"DOI":"10.1007\/s10664-021-09939-7","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2021,3,16]],"date-time":"2021-03-16T00:00:00Z","timestamp":1615852800000}}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09854-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-020-09854-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09854-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,8,4]],"date-time":"2021-08-04T00:03:43Z","timestamp":1628035423000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-020-09854-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,4]]},"references-count":41,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,9]]}},"alternative-id":["9854"],"URL":"https:\/\/doi.org\/10.1007\/s10664-020-09854-3","relation":{"correction":[{"id-type":"doi","id":"10.1007\/s10664-021-09939-7","asserted-by":"object"}]},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,4]]},"assertion":[{"value":"4 August 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 March 2021","order":2,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":3,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":4,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1007\/s10664-021-09939-7","URL":"https:\/\/doi.org\/10.1007\/s10664-021-09939-7","order":5,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}}]}}