{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:40:36Z","timestamp":1785537636520,"version":"3.56.0"},"reference-count":53,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,7,10]],"date-time":"2022-07-10T00:00:00Z","timestamp":1657411200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,10]],"date-time":"2022-07-10T00:00:00Z","timestamp":1657411200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"ITEA3","award":["No. 17038 VISDOM"],"award-info":[{"award-number":["No. 17038 VISDOM"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Technical debt is a metaphor indicating sub-optimal solutions implemented for short-term benefits by sacrificing the long-term maintainability and evolvability of software. A special type of technical debt is explicitly admitted by software engineers (e.g. using a TODO comment); this is called Self-Admitted Technical Debt or SATD. Most work on automatically identifying SATD focuses on source code comments. In addition to source code comments, issue tracking systems have shown to be another rich source of SATD, but there are no approaches specifically for automatically identifying SATD in issues. In this paper, we first create a training dataset by collecting and manually analyzing 4,200 issues (that break down to 23,180 sections of issues) from seven open-source projects (i.e., Camel, Chromium, Gerrit, Hadoop, HBase, Impala, and Thrift) using two popular issue tracking systems (i.e., Jira and Google Monorail). We then propose and optimize an approach for automatically identifying SATD in issue tracking systems using machine learning. Our findings indicate that: 1) our approach outperforms baseline approaches by a wide margin with regard to the F1-score; 2) transferring knowledge from suitable datasets can improve the predictive performance of our approach; 3) extracted SATD keywords are intuitive and potentially indicating types and indicators of SATD; 4) projects using different issue tracking systems have less common SATD keywords compared to projects using the same issue tracking system; 5) a small amount of training data is needed to achieve good accuracy.<\/jats:p>","DOI":"10.1007\/s10664-022-10128-3","type":"journal-article","created":{"date-parts":[[2022,7,11]],"date-time":"2022-07-11T12:03:34Z","timestamp":1657541014000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Identifying self-admitted technical debt in issue tracking systems using machine learning"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1566-725X","authenticated-orcid":false,"given":"Yikun","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamed","family":"Soliman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paris","family":"Avgeriou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,7,10]]},"reference":[{"key":"10128_CR1","doi-asserted-by":"crossref","unstructured":"Alves N S, Ribeiro L F, Caires V, Mendes T S, Sp\u00ednola RO (2014) Towards an ontology of terms on technical debt. In: 2014 Sixth international workshop on managing technical debt. IEEE, pp 1\u20137","DOI":"10.1109\/MTD.2014.9"},{"key":"10128_CR2","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.infsof.2015.10.008","volume":"70","author":"NS Alves","year":"2016","unstructured":"Alves N S, Mendes T S, de Mendon\u00e7a M G, RO Sp\u00ednola, Shull F, Seaman C (2016) Identification and management of technical debt: a systematic mapping study. Inf Softw Technol 70:100\u2013121","journal-title":"Inf Softw Technol"},{"issue":"4","key":"10128_CR3","doi-asserted-by":"publisher","first-page":"110","DOI":"10.4230\/DagRep.6.4.110","volume":"6","author":"P Avgeriou","year":"2016","unstructured":"Avgeriou P, Kruchten P, Ozkaya I, Seaman C (2016) Managing technical debt in software engineering (Dagstuhl Seminar 16162). Dagstuhl Rep 6 (4):110\u2013138. https:\/\/doi.org\/10.4230\/DagRep.6.4.110","journal-title":"Dagstuhl Rep"},{"key":"10128_CR4","doi-asserted-by":"crossref","unstructured":"Bavota G, Russo B (2016) A large-scale empirical study on self-admitted technical debt. In: Proceedings of the 13th international conference on mining software repositories, pp 315\u2013326","DOI":"10.1145\/2901739.2901742"},{"key":"10128_CR5","doi-asserted-by":"crossref","unstructured":"Bellomo S, Nord R L, Ozkaya I, Popeck M (2016) Got technical debt? Surfacing elusive technical debt in issue trackers. In: 2016 IEEE\/ACM 13th working conference on mining software repositories (MSR). IEEE, pp 327\u2013338","DOI":"10.1145\/2901739.2901754"},{"key":"10128_CR6","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1162\/tacl_a_00051","volume":"5","author":"P Bojanowski","year":"2017","unstructured":"Bojanowski P, Grave E, Joulin A, Mikolov T (2017) Enriching word vectors with subword information. Trans Assoc Comput Linguist 5:135\u2013146","journal-title":"Trans Assoc Comput Linguist"},{"issue":"3","key":"10128_CR7","doi-asserted-by":"publisher","first-page":"1352","DOI":"10.1007\/s10664-017-9546-9","volume":"23","author":"F Calefato","year":"2018","unstructured":"Calefato F, Lanubile F, Maiorano F, Novielli N (2018) Sentiment polarity detection for software development. Empir Softw Eng 23(3):1352\u20131382","journal-title":"Empir Softw Eng"},{"key":"10128_CR8","unstructured":"Dai K, Kruchten P (2017) Detecting technical debt through issue trackers. In: QuASoq@ APSEC, pp 59\u201365"},{"key":"10128_CR9","doi-asserted-by":"crossref","unstructured":"de Freitas Farias M A, Santos J A, Kalinowski M, Mendon\u00e7a M, Sp\u00ednola RO (2016) Investigating the identification of technical debt through code comment analysis. In: International conference on enterprise information systems. Springer, pp 284\u2013309","DOI":"10.1007\/978-3-319-62386-3_14"},{"key":"10128_CR10","doi-asserted-by":"crossref","unstructured":"Efstathiou V, Chatzilenas C, Spinellis D (2018) Word embeddings for the software engineering domain. In: Proceedings of the 15th international conference on mining software repositories, pp 38\u201341","DOI":"10.1145\/3196398.3196448"},{"key":"10128_CR11","doi-asserted-by":"crossref","unstructured":"Ernst N A (2012) On the role of requirements in understanding and managing technical debt. In: 2012 Third international workshop on managing technical debt (MTD). IEEE, pp 61\u201364","DOI":"10.1109\/MTD.2012.6226002"},{"key":"10128_CR12","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-98074-4","volume-title":"Learning from imbalanced data sets","author":"A Fern\u00e1ndez","year":"2018","unstructured":"Fern\u00e1ndez A, Garc\u00eda S, Galar M, Prati RC, Krawczyk B, Herrera F (2018) Learning from imbalanced data sets. Springer, Berlin"},{"issue":"212\u2013236","key":"10128_CR13","first-page":"22","volume":"2","author":"JL Fleiss","year":"1981","unstructured":"Fleiss J L, Levin B, Paik M C, et al. (1981) The measurement of interrater agreement. Statistical Methods for Rates and Proportions 2(212\u2013236):22\u201323","journal-title":"Statistical Methods for Rates and Proportions"},{"key":"10128_CR14","doi-asserted-by":"publisher","first-page":"106475","DOI":"10.1109\/ACCESS.2019.2933318","volume":"7","author":"J Flisar","year":"2019","unstructured":"Flisar J, Podgorelec V (2019) Identification of self-admitted technical debt using enhanced feature selection based on word embedding. IEEE Access 7:106475\u2013106494","journal-title":"IEEE Access"},{"issue":"3","key":"10128_CR15","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1198\/004017007000000245","volume":"49","author":"A Genkin","year":"2007","unstructured":"Genkin A, Lewis D D, Madigan D (2007) Large-scale bayesian logistic regression for text categorization. Technometrics 49(3):291\u2013304","journal-title":"Technometrics"},{"issue":"19","key":"10128_CR16","doi-asserted-by":"publisher","first-page":"2811","DOI":"10.1093\/bioinformatics\/btu393","volume":"30","author":"Z Gu","year":"2014","unstructured":"Gu Z, Gu L, Eils R, Schlesner M, Brors B (2014) Circlize implements and enhances circular visualization in r. Bioinformatics 30(19):2811\u20132812","journal-title":"Bioinformatics"},{"issue":"1","key":"10128_CR17","doi-asserted-by":"publisher","first-page":"418","DOI":"10.1007\/s10664-017-9522-4","volume":"23","author":"Q Huang","year":"2018","unstructured":"Huang Q, Shihab E, Xia X, Lo D, Li S (2018) Identifying self-admitted technical debt in open source projects using text mining. Empir Softw Eng 23(1):418\u2013451","journal-title":"Empir Softw Eng"},{"key":"10128_CR18","doi-asserted-by":"crossref","unstructured":"Joulin A, Grave \u00c9 , Bojanowski P, Mikolov T (2017) Bag of tricks for efficient text classification. In: Proceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2. Short Papers, pp 427\u2013431","DOI":"10.18653\/v1\/E17-2068"},{"key":"10128_CR19","unstructured":"Kamei Y, Maldonado EDS, Shihab E, Ubayashi N (2016) Using analytics to quantify interest of self-admitted technical debt. In: QuASoq\/TDA@ APSEC, pp 68\u201371"},{"key":"10128_CR20","doi-asserted-by":"crossref","unstructured":"Kim Y (2014) Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP), pp 1746\u20131751","DOI":"10.3115\/v1\/D14-1181"},{"issue":"4","key":"10128_CR21","doi-asserted-by":"publisher","first-page":"150","DOI":"10.3390\/info10040150","volume":"10","author":"K Kowsari","year":"2019","unstructured":"Kowsari K, Jafari Meimandi K, Heidarysafa M, Mendu S, Barnes L, Brown D (2019) Text classification algorithms: a survey. Information 10(4):150","journal-title":"Information"},{"key":"10128_CR22","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1016\/j.jss.2014.12.027","volume":"101","author":"Z Li","year":"2015","unstructured":"Li Z, Avgeriou P, Liang P (2015) A systematic mapping study on technical debt and its management. J Syst Softw 101:193\u2013220","journal-title":"J Syst Softw"},{"key":"10128_CR23","doi-asserted-by":"publisher","unstructured":"Li Y, Soliman M, Avgeriou P (2020) Identification and remediation of self-admitted technical debt in issue trackers. In: 2020 46th Euromicro conference on software engineering and advanced applications (SEAA). https:\/\/doi.org\/10.1109\/SEAA51224.2020.00083, pp 495\u2013503","DOI":"10.1109\/SEAA51224.2020.00083"},{"key":"10128_CR24","doi-asserted-by":"crossref","unstructured":"Liu Z, Huang Q, Xia X, Shihab E, Lo D, Li S (2018) Satd detector: a text-mining-based self-admitted technical debt detection tool. In: Proceedings of the 40th international conference on software engineering: companion proceedings, pp 9\u201312","DOI":"10.1145\/3183440.3183478"},{"key":"10128_CR25","doi-asserted-by":"publisher","unstructured":"Maldonado EDS, Shihab E (2015) Detecting and quantifying different types of self-admitted technical debt. In: 2015 IEEE 7th international workshop on managing technical debt (MTD). https:\/\/doi.org\/10.1109\/MTD.2015.7332619, pp 9\u201315","DOI":"10.1109\/MTD.2015.7332619"},{"issue":"11","key":"10128_CR26","doi-asserted-by":"publisher","first-page":"1044","DOI":"10.1109\/TSE.2017.2654244","volume":"43","author":"EDS Maldonado","year":"2017","unstructured":"Maldonado EDS, Shihab E, Tsantalis N (2017) Using natural language processing to automatically detect self-admitted technical debt. IEEE Trans Softw Eng 43(11):1044\u20131062. https:\/\/doi.org\/10.1109\/TSE.2017.2654244https:\/\/doi.org\/10.1109\/TSE.2017.2654244","journal-title":"IEEE Trans Softw Eng"},{"key":"10128_CR27","unstructured":"McCallum A, Nigam K et al (1998) A comparison of event models for naive bayes text classification. In: AAAI-98 workshop on learning for text categorization. Citeseer, vol 752, pp 41\u201348"},{"key":"10128_CR28","unstructured":"Merten T, Mager B, H\u00fcbner P, Quirchmayr T, Paech B, B\u00fcrsner S (2015) Requirements communication in issue tracking systems in four open-source projects. In: REFSQ workshops, pp 114\u2013125"},{"key":"10128_CR29","unstructured":"Mikolov T, Grave E, Bojanowski P, Puhrsch C, Joulin A (2018) Advances in pre-training distributed word representations. In: Proceedings of the international conference on language resources and evaluation (LREC 2018)"},{"key":"10128_CR30","doi-asserted-by":"crossref","unstructured":"Ortu M, Murgia A, Destefanis G, Tourani P, Tonelli R, Marchesi M, Adams B (2016) The emotional side of software developers in jira. In: 2016 IEEE\/ACM 13th working conference on mining software repositories (MSR). IEEE, pp 480\u2013483","DOI":"10.1145\/2901739.2903505"},{"key":"10128_CR31","first-page":"6452","volume":"2","author":"DN Perkins","year":"1992","unstructured":"Perkins D N, Salomon G, et al. (1992) Transfer of learning. International Encyclopedia of Education 2:6452\u20136457","journal-title":"International Encyclopedia of Education"},{"key":"10128_CR32","doi-asserted-by":"crossref","unstructured":"Phan H, Krawczyk-Becker M, Gerkmann T, Mertins A (2017) Dnn and cnn with weighted and multi-task loss functions for audio event detection. arXiv:170803211","DOI":"10.1109\/ICASSP.2018.8461353"},{"key":"10128_CR33","doi-asserted-by":"crossref","unstructured":"Potdar A, Shihab E (2014) An exploratory study on self-admitted technical debt. In: 2014 IEEE international conference on software maintenance and evolution. IEEE, pp 91\u2013100","DOI":"10.1109\/ICSME.2014.31"},{"issue":"3","key":"10128_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3324916","volume":"28","author":"X Ren","year":"2019","unstructured":"Ren X, Xing Z, Xia X, Lo D, Wang X, Grundy J (2019) Neural network-based detection of self-admitted technical debt: from performance to explainability. ACM Trans Software Eng Methodol (TOSEM) 28(3):1\u201345","journal-title":"ACM Trans Software Eng Methodol (TOSEM)"},{"key":"10128_CR35","doi-asserted-by":"publisher","DOI":"10.1002\/9781118181034","volume-title":"Case study research in software engineering: guidelines and examples","author":"P Runeson","year":"2012","unstructured":"Runeson P, Host M, Rainer A, Regnell B (2012) Case study research in software engineering: guidelines and examples. Wiley, New York"},{"key":"10128_CR36","doi-asserted-by":"crossref","unstructured":"Semwal T, Yenigalla P, Mathur G, Nair S B (2018) A practitioners\u2019 guide to transfer learning for text classification using convolutional neural networks. In: Proceedings of the 2018 SIAM international conference on data mining. SIAM, pp 513\u2013521","DOI":"10.1137\/1.9781611975321.58"},{"key":"10128_CR37","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781107298019","volume-title":"Understanding machine learning: from theory to algorithms","author":"S Shalev-Shwartz","year":"2014","unstructured":"Shalev-Shwartz S, Ben-David S (2014) Understanding machine learning: from theory to algorithms. Cambridge University Press, Cambridge"},{"key":"10128_CR38","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1016\/j.jss.2019.02.056","volume":"152","author":"G Sierra","year":"2019","unstructured":"Sierra G, Shihab E, Kamei Y (2019) A survey of self-admitted technical debt. J Syst Softw 152:70\u201382. https:\/\/doi.org\/10.1016\/j.jss.2019.02.056. http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121219300457","journal-title":"J Syst Softw"},{"key":"10128_CR39","unstructured":"Smith T (2018) The most important players in the open source ecosystem. https:\/\/dzone.com\/articles\/the-most-important-players-in-the-open-source-ecos"},{"key":"10128_CR40","doi-asserted-by":"crossref","unstructured":"Steidl D, Hummel B, Juergens E (2013) Quality analysis of source code comments. In: 2013 21St international conference on program comprehension (icpc). IEEE, pp 83\u201392","DOI":"10.1109\/ICPC.2013.6613836"},{"issue":"1","key":"10128_CR41","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/j.dss.2009.07.011","volume":"48","author":"A Sun","year":"2009","unstructured":"Sun A, Lim E P, Liu Y (2009) On strategies for imbalanced text classification using svm: a comparative study. Decis Support Syst 48(1):191\u2013201","journal-title":"Decis Support Syst"},{"issue":"2","key":"10128_CR42","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1016\/j.eswa.2005.07.019","volume":"30","author":"S Tan","year":"2006","unstructured":"Tan S (2006) An effective refinement strategy for knn text classifier. Expert Syst Appl 30(2):290\u2013298","journal-title":"Expert Syst Appl"},{"issue":"11","key":"10128_CR43","doi-asserted-by":"publisher","first-page":"1063","DOI":"10.1109\/TSE.2017.2653105","volume":"43","author":"M Tufano","year":"2017","unstructured":"Tufano M, Palomba F, Bavota G, Oliveto R, Di Penta M, De Lucia A, Poshyvanyk D (2017) When and why your code starts to smell bad (and whether the smells go away). IEEE Trans Softw Eng 43(11):1063\u20131088","journal-title":"IEEE Trans Softw Eng"},{"key":"10128_CR44","doi-asserted-by":"crossref","unstructured":"van Solingen R, Basili V, Caldiera G, Rombach H D (2002) Goal Question Metric (GQM) approach. In: Encyclopedia of software engineering. Wiley, Hoboken, pp 528\u2013532","DOI":"10.1002\/0471028959.sof142"},{"key":"10128_CR45","doi-asserted-by":"crossref","unstructured":"Wehaibi S, Shihab E, Guerrouj L (2016) Examining the impact of self-admitted technical debt on software quality. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), vol 1. IEEE, pp 179\u2013188","DOI":"10.1109\/SANER.2016.72"},{"key":"10128_CR46","doi-asserted-by":"crossref","unstructured":"Wei J, Zou K (2019) Eda: easy data augmentation techniques for boosting performance on text classification tasks. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 6383\u2013 6389","DOI":"10.18653\/v1\/D19-1670"},{"key":"10128_CR47","unstructured":"Wieting J, Bansal M, Gimpel K, Livescu K (2015) Towards universal paraphrastic sentence embeddings. arXiv:151108198"},{"key":"10128_CR48","doi-asserted-by":"crossref","unstructured":"Xavier L, Ferreira F, Brito R, Valente MT (2020) Beyond the code: mining self-admitted technical debt in issue tracker systems. arXiv:200309418","DOI":"10.1145\/3379597.3387459"},{"issue":"12","key":"10128_CR49","first-page":"2913","volume":"7","author":"B Xu","year":"2012","unstructured":"Xu B, Guo X, Ye Y, Cheng J (2012) An improved random forest classifier for text categorization. JCP 7(12):2913\u20132920","journal-title":"JCP"},{"key":"10128_CR50","doi-asserted-by":"crossref","unstructured":"Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 7370\u20137377","DOI":"10.1609\/aaai.v33i01.33017370"},{"key":"10128_CR51","doi-asserted-by":"crossref","unstructured":"Zampetti F, Serebrenik A, Di Penta M (2018) Was self-admitted technical debt removal a real removal? An in-depth perspective. In: 2018 IEEE\/ACM 15th international conference on mining software repositories (MSR). IEEE, pp 526\u2013536","DOI":"10.1145\/3196398.3196423"},{"key":"10128_CR52","unstructured":"Zhang Y, Wallace B C (2017) A sensitivity analysis of (and practitioners\u2019 guide to) convolutional neural networks for sentence classification. In: Proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), pp 253\u2013263"},{"key":"10128_CR53","unstructured":"Zhang X, Zhao J, LeCun Y (2015) Character-level convolutional networks for text classification. In: Advances in neural information processing systems, pp 649\u2013657"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10128-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10128-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10128-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,26]],"date-time":"2022-09-26T08:18:21Z","timestamp":1664180301000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10128-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,10]]},"references-count":53,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["10128"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10128-3","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,10]]},"assertion":[{"value":"31 January 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 July 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Ethics approval and consent to participate"}},{"value":"Not applicable","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Consent for Publication"}},{"value":"The authors have no conflicts of interest to declare that are relevant to the content of this article.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"131"}}