{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T04:24:20Z","timestamp":1785731060669,"version":"3.56.0"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,4,15]],"date-time":"2023-04-15T00:00:00Z","timestamp":1681516800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,15]],"date-time":"2023-04-15T00:00:00Z","timestamp":1681516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009077","name":"ITEA3","doi-asserted-by":"publisher","award":["No. 17038 VISDOM"],"award-info":[{"award-number":["No. 17038 VISDOM"]}],"id":[{"id":"10.13039\/501100009077","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2023,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Technical debt refers to taking shortcuts to achieve short-term goals while sacrificing the long-term maintainability and evolvability of software systems. A large part of technical debt is explicitly reported by the developers themselves; this is commonly referred to as <jats:italic>Self-Admitted Technical Debt<\/jats:italic> or <jats:italic>SATD<\/jats:italic>. Previous work has focused on identifying SATD from source code comments and issue trackers. However, there are no approaches available for automatically identifying SATD from other sources such as commit messages and pull requests, or by combining multiple sources. Therefore, we propose and evaluate an approach for automated SATD identification that integrates four sources: source code comments, commit messages, pull requests, and issue tracking systems. Our findings show that our approach outperforms baseline approaches and achieves an average F1-score of 0.611 when detecting four types of SATD (i.e., code\/design debt, requirement debt, documentation debt, and test debt) from the four aforementioned sources. Thereafter, we analyze 23.6M code comments, 1.3M commit messages, 3.7M issue sections, and 1.7M pull request sections to characterize SATD in 103 open-source projects. Furthermore, we investigate the SATD keywords and relations between SATD in different sources. The findings indicate, among others, that: 1) SATD is evenly spread among all sources; 2) issues and pull requests are the two most similar sources regarding the number of shared SATD keywords, followed by commit messages, and then followed by code comments; 3) there are four kinds of relations between SATD items in the different sources.<\/jats:p>","DOI":"10.1007\/s10664-023-10297-9","type":"journal-article","created":{"date-parts":[[2023,4,15]],"date-time":"2023-04-15T07:02:14Z","timestamp":1681542134000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":40,"title":["Automatic identification of self-admitted technical debt from four different sources"],"prefix":"10.1007","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1566-725X","authenticated-orcid":false,"given":"Yikun","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6638-3732","authenticated-orcid":false,"given":"Mohamed","family":"Soliman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7101-0754","authenticated-orcid":false,"given":"Paris","family":"Avgeriou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,4,15]]},"reference":[{"key":"10297_CR1","unstructured":"Aaron Stannard (2021) How to use Github professionally: Best practices for working with Github in team settings. https:\/\/petabridge.com\/blog\/use-github-professionally\/ . Accessed 1 March 2022"},{"key":"10297_CR2","unstructured":"Akira Ajisaka (2021) Hadoop contributor guide - how to contribute. https:\/\/cwiki.apache.org\/confluence\/display\/HADOOP\/How+To+Contribute. Accessed 1 March 2022"},{"issue":"5","key":"10297_CR3","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1145\/2160718.2160733","volume":"55","author":"E Allman","year":"2012","unstructured":"Allman E (2012) Managing technical debt. Commun ACM 55 (5):50\u201355","journal-title":"Commun ACM"},{"key":"10297_CR4","doi-asserted-by":"crossref","unstructured":"Alves NS, Ribeiro LF, Caires V, Mendes TS, Sp\u00ednola RO (2014) Towards an ontology of terms on technical debt. In: 2014 6th international workshop on managing technical debt. IEEE, pp 1\u20137","DOI":"10.1109\/MTD.2014.9"},{"key":"10297_CR5","unstructured":"Apache Software Foundation (2021) Briefing: The apache way. http:\/\/www.apache.org\/theapacheway\/index.html. Accessed 1 March 2022"},{"issue":"4","key":"10297_CR6","doi-asserted-by":"publisher","first-page":"110","DOI":"10.4230\/DagRep.6.4.110","volume":"6","author":"P Avgeriou","year":"2016","unstructured":"Avgeriou P, Kruchten P, Ozkaya I, Seaman C (2016) Managing technical debt in software engineering (Dagstuhl Seminar 16162). Dagstuhl Rep 6 (4):110\u2013138. https:\/\/doi.org\/10.4230\/DagRep.6.4.110","journal-title":"Dagstuhl Rep"},{"key":"10297_CR7","doi-asserted-by":"crossref","unstructured":"Bellomo S, Nord R L, Ozkaya I, Popeck M (2016) Got technical debt? surfacing elusive technical debt in issue trackers. In: 2016 IEEE\/ACM 13th working conference on mining software repositories (MSR). IEEE, pp 327\u2013338","DOI":"10.1145\/2901739.2901754"},{"issue":"1","key":"10297_CR8","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L (2001) Random forests. Mach Learn 45(1):5\u201332","journal-title":"Mach Learn"},{"key":"10297_CR9","doi-asserted-by":"crossref","unstructured":"Chen X, Yu D, Fan X, Wang L, Chen J (2021) Multiclass classification for self-admitted technical debt based on XGboost. IEEE Transactions on Reliability","DOI":"10.1109\/TR.2021.3087864"},{"key":"10297_CR10","doi-asserted-by":"crossref","unstructured":"Collobert R, Weston J (2008) A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th international conference on Machine learning, pp 160\u2013167","DOI":"10.1145\/1390156.1390177"},{"issue":"11","key":"10297_CR11","doi-asserted-by":"publisher","first-page":"1044","DOI":"10.1109\/TSE.2017.2654244","volume":"43","author":"E da Silva Maldonado","year":"2017","unstructured":"da Silva Maldonado E, Shihab E, Tsantalis N (2017) Using natural language processing to automatically detect self-admitted technical debt. IEEE Trans Softw Eng 43(11):1044\u20131062","journal-title":"IEEE Trans Softw Eng"},{"key":"10297_CR12","unstructured":"Dai K, Kruchten P (2017) Detecting technical debt through issue trackers. In: QuASoQ@ APSEC, pp 59\u201365"},{"issue":"3","key":"10297_CR13","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1198\/004017007000000245","volume":"49","author":"A Genkin","year":"2007","unstructured":"Genkin A, Lewis D D, Madigan D (2007) Large-scale Bayesian logistic regression for text categorization. Technometrics 49(3):291\u2013304","journal-title":"Technometrics"},{"issue":"4","key":"10297_CR14","doi-asserted-by":"publisher","first-page":"436","DOI":"10.2307\/798843","volume":"12","author":"BG Glaser","year":"1965","unstructured":"Glaser B G (1965) The constant comparative method of qualitative analysis. Soc Probl 12(4):436\u2013445","journal-title":"Soc Probl"},{"issue":"4","key":"10297_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3447247","volume":"30","author":"Z Guo","year":"2021","unstructured":"Guo Z, Liu S, Liu J, Li Y, Chen L, Lu H, Zhou Y (2021) How far have we progressed in identifying self-admitted technical debts? a comprehensive empirical study. ACM Trans Softw Eng Method (TOSEM) 30(4):1\u201356","journal-title":"ACM Trans Softw Eng Method (TOSEM)"},{"issue":"1","key":"10297_CR16","doi-asserted-by":"publisher","first-page":"418","DOI":"10.1007\/s10664-017-9522-4","volume":"23","author":"Q Huang","year":"2018","unstructured":"Huang Q, Shihab E, Xia X, Lo D, Li S (2018) Identifying self-admitted technical debt in open source projects using text mining. Empir Softw Eng 23(1):418\u2013451","journal-title":"Empir Softw Eng"},{"key":"10297_CR17","doi-asserted-by":"crossref","unstructured":"Iammarino M, Zampetti F, Aversano L, Di Penta M (2019) Self-admitted technical debt removal and refactoring actions: Co-occurrence or more?. In: 2019 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 186\u2013190","DOI":"10.1109\/ICSME.2019.00029"},{"key":"10297_CR18","doi-asserted-by":"publisher","first-page":"110976","DOI":"10.1016\/j.jss.2021.110976","volume":"178","author":"M Iammarino","year":"2021","unstructured":"Iammarino M, Zampetti F, Aversano L, Di Penta M (2021) An empirical study on the co-occurrence between refactoring actions and self-admitted technical debt removal. J Syst Softw 178:110976","journal-title":"J Syst Softw"},{"key":"10297_CR19","unstructured":"Kamei Y, Maldonado EdS, Shihab E, Ubayashi N (2016) Using analytics to quantify interest of self-admitted technical debt. In: QuASoQ\/TDA@ APSEC, pp 68\u201371"},{"key":"10297_CR20","doi-asserted-by":"crossref","unstructured":"Kim Y (2014) Convolutional neural networks for sentence classification. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1746\u20131751","DOI":"10.3115\/v1\/D14-1181"},{"key":"10297_CR21","doi-asserted-by":"crossref","unstructured":"Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics pp 159\u2013174","DOI":"10.2307\/2529310"},{"key":"10297_CR22","doi-asserted-by":"publisher","unstructured":"Li Y, Soliman M, Avgeriou P (2020a) Identification and remediation of self-admitted technical debt in issue trackers. In: 2020 46th Euromicro conference on software engineering and advanced applications (SEAA), pp 495\u2013503. https:\/\/doi.org\/10.1109\/SEAA51224.2020.00083","DOI":"10.1109\/SEAA51224.2020.00083"},{"key":"10297_CR23","doi-asserted-by":"publisher","unstructured":"Li Y, Soliman M, Avgeriou P (2022b) Identifying self-admitted technical debt in issue tracking systems using machine learning. Empir Softw Eng 27(131). https:\/\/doi.org\/10.1007\/s10664-022-10128-3","DOI":"10.1007\/s10664-022-10128-3"},{"key":"10297_CR24","doi-asserted-by":"publisher","unstructured":"Li Y, Soliman M, Avgeriou P, Lou S (2022c) Self-Admitted Technical Debt in the Embedded Systems Industry: An Exploratory Case Study. IEEE Transactions on Software Engineering. https:\/\/doi.org\/10.1109\/TSE.2022.3224378","DOI":"10.1109\/TSE.2022.3224378"},{"key":"10297_CR25","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1016\/j.jss.2014.12.027","volume":"101","author":"Z Li","year":"2015","unstructured":"Li Z, Avgeriou P, Liang P (2015) A systematic mapping study on technical debt and its management. J Syst Softw 101:193\u2013220","journal-title":"J Syst Softw"},{"key":"10297_CR26","doi-asserted-by":"crossref","unstructured":"Liu X, Gao J, He X, Deng L, Duh K, Wang YY (2015) Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In: Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 912\u2013921","DOI":"10.3115\/v1\/N15-1092"},{"key":"10297_CR27","doi-asserted-by":"crossref","unstructured":"Maldonado EdS, Shihab E (2015) Detecting and quantifying different types of self-admitted technical debt. In: 2015 IEEE 7th international workshop on managing technical debt (MTD). IEEE, pp 9\u201315","DOI":"10.1109\/MTD.2015.7332619"},{"key":"10297_CR28","unstructured":"Mikolov T, Grave E, Bojanowski P, Puhrsch C, Joulin A (2018) Advances in pre-training distributed word representations. In: Proceedings of the international conference on language resources and evaluation (LREC)"},{"key":"10297_CR29","unstructured":"Monson-Haefel R (2021) It\u2019s easy! your first Tomee pull-request: Using jira. https:\/\/www.tomitribe.com\/blog\/its-easy-requesting-and-discovering-jira-tickets-in-tomee\/. Accessed 1 March 2022"},{"key":"10297_CR30","doi-asserted-by":"crossref","unstructured":"Potdar A, Shihab E (2014) An exploratory study on self-admitted technical debt. In: 2014 IEEE international conference on software maintenance and evolution. IEEE, pp 91\u2013100","DOI":"10.1109\/ICSME.2014.31"},{"issue":"3","key":"10297_CR31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3324916","volume":"28","author":"X Ren","year":"2019","unstructured":"Ren X, Xing Z, Xia X, Lo D, Wang X, Grundy J (2019) Neural network-based detection of self-admitted technical debt: From performance to explainability. ACM Trans Softw Eng Method (TOSEM) 28(3):1\u201345","journal-title":"ACM Trans Softw Eng Method (TOSEM)"},{"key":"10297_CR32","doi-asserted-by":"publisher","DOI":"10.1002\/9781118181034","volume-title":"Case study research in software engineering: Guidelines and examples","author":"P Runeson","year":"2012","unstructured":"Runeson P, Host M, Rainer A, Regnell B (2012) Case study research in software engineering: Guidelines and examples. Wiley, New York"},{"key":"10297_CR33","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1016\/j.jss.2019.02.056","volume":"152","author":"G Sierra","year":"2019","unstructured":"Sierra G, Shihab E, Kamei Y (2019) A survey of self-admitted technical debt. J Syst Softw 152:70\u201382. https:\/\/doi.org\/10.1016\/j.jss.2019.02.056, http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121219300457","journal-title":"J Syst Softw"},{"key":"10297_CR34","doi-asserted-by":"crossref","unstructured":"Student (1908) The probable error of a mean. Biometrika pp 1\u201325","DOI":"10.2307\/2331554"},{"issue":"1","key":"10297_CR35","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1016\/j.dss.2009.07.011","volume":"48","author":"A Sun","year":"2009","unstructured":"Sun A, Lim E P, Liu Y (2009) On strategies for imbalanced text classification using SVM: A comparative study. Decis Support Syst 48(1):191\u2013201","journal-title":"Decis Support Syst"},{"key":"10297_CR36","doi-asserted-by":"crossref","unstructured":"van Solingen R, Basili V, Caldiera G, Rombach HD (2002) Goal Question Metric (GQM) approach. In: Encyclopedia of Software Eng. Wiley, Hoboken, pp 528\u2013532","DOI":"10.1002\/0471028959.sof142"},{"issue":"2","key":"10297_CR37","first-page":"101","volume":"25","author":"A Vargha","year":"2000","unstructured":"Vargha A, Delaney H D (2000) A critique and improvement of the CL common language effect size statistics of MCGraw and Wong. J Educ Behav Stat 25(2):101\u2013132","journal-title":"J Educ Behav Stat"},{"key":"10297_CR38","doi-asserted-by":"crossref","unstructured":"Wang X, Liu J, Li L, Chen X, Liu X, Wu H (2020) Detecting and explaining self-admitted technical debts with attention-based neural networks. In: Proceedings of the 35th IEEE\/ACM international conference on automated software engineering, pp 871\u2013882","DOI":"10.1145\/3324884.3416583"},{"key":"10297_CR39","doi-asserted-by":"crossref","unstructured":"Xavier L, Ferreira F, Brito R, Valente MT (2020) Beyond the code: Mining self-admitted technical debt in issue tracker systems. In: Proceedings of the 17th international conference on mining software repositories, pp 137\u2013146","DOI":"10.1145\/3379597.3387459"},{"key":"10297_CR40","doi-asserted-by":"crossref","unstructured":"Zampetti F, Serebrenik A, Di Penta M (2018) Was self-admitted technical debt removal a real removal? an in-depth perspective. In: 2018 IEEE\/ACM 15th international conference on mining software repositories (MSR). IEEE, pp 526\u2013536","DOI":"10.1145\/3196398.3196423"},{"issue":"6","key":"10297_CR41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10664-021-10031-3","volume":"26","author":"F Zampetti","year":"2021","unstructured":"Zampetti F, Fucci G, Serebrenik A, Di Penta M (2021) Self-admitted technical debt practices: a comparison between industry and open-source. Empir Softw Eng 26(6):1\u201332","journal-title":"Empir Softw Eng"},{"key":"10297_CR42","unstructured":"Zhang Y, Wallace BC (2017) A sensitivity analysis of (and practitioners\u2019 guide to) convolutional neural networks for sentence classification. In: Proceedings of the 8th international joint conference on natural language processing (vol 1: Long Papers), pp 253\u2013263"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10297-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-023-10297-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10297-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T09:47:16Z","timestamp":1685094436000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-023-10297-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,15]]},"references-count":42,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,5]]}},"alternative-id":["10297"],"URL":"https:\/\/doi.org\/10.1007\/s10664-023-10297-9","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,15]]},"assertion":[{"value":"31 January 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 April 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflicts of interest to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}],"article-number":"65"}}