{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:31:02Z","timestamp":1792787462335,"version":"4.5.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"10","license":[{"start":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T00:00:00Z","timestamp":1789603200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Natural Sciences and Engineering Research Council of Canada","award":["RGPIN-2019-05071"],"award-info":[{"award-number":["RGPIN-2019-05071"]}],"id":[{"id":"https:\/\/ror.org\/01h531d29","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,10,31]]},"abstract":"<jats:p>Self-Admitted Technical Debt (SATD), cases where developers intentionally acknowledge suboptimal solutions in code through comments, poses a significant challenge to software maintainability. Left unresolved, SATD can degrade code quality, increase maintenance costs, and hinder long-term software sustainability. Manually identifying and resolving SATD is time-consuming and error-prone, often competing with feature development and release deadlines. Automating SATD repayment is therefore essential to alleviate developer burden. While Large Language Models (LLMs) have shown promise in tasks like code generation and program repair, their potential in automated SATD repayment remains underexplored.<\/jats:p>\n                  <jats:p>\n                    In this article, we identify three key technical challenges in training and evaluating LLMs for SATD repayment: (1) dataset representativeness and scalability, (2) removal of irrelevant SATD repayment samples, and (3) limitations of existing evaluation metrics, i.e., BLEU and CrystalBLEU. To address the first two dataset-related challenges, we adopt a language-independent SATD tracing tool and design a 10-step filtering pipeline to extract SATD repayments from repository commit histories, leveraging LLM-as-judge for relevance filtering. This results in two new large-scale SATD repayment datasets: 58,722 items for Python and 97,347 items for Java. To improve evaluation, we introduce two diff-based metrics, BLEU-diff and CrystalBLEU-diff, which measure code changes rather than whole code, thereby mitigating the impact of SATD-irrelevant source code. Additionally, we propose a new metric, Line-Level Exact Match on Diff (LEMOD), which is both interpretable and informative, providing fine-grained insights into SATD repayment quality. Using our new benchmarks and evaluation metrics, we evaluate two types of automated SATD repayment methods: fine-tuning smaller models (a few hundred million parameters) and prompt engineering with large-scale models, including four SOTA open source LLMs and GPT-4o-mini. Our results reveal that while fine-tuned smaller models achieve Exact Match (EM) scores comparable to prompt-based approaches, they underperform on BLEU-based metrics and LEMOD. The best EM performance is achieved by Gemma-2-9B, correctly addressing 10.1% of Python SATDs and 8.1% of Java SATDs with a simple prompt. The only previous study in this domain focused solely on Java, achieving an EM score of 2.3%. When using BLEU-diff, CrystalBLEU-diff, and LEMOD metrics, Llama-3.1-70B-Instruct and GPT-4o-mini deliver the highest performance. While our three proposed metrics demonstrate strong correlations with EM (0.65\n                    <jats:inline-formula content-type=\"math\/tex\">\n                      <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\sim\\)<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    0.84), both BLEU and CrystalBLEU show almost no correlation with this intuitive metric (0.01\n                    <jats:inline-formula content-type=\"math\/tex\">\n                      <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\sim\\)<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    0.08), highlighting their largely unpredictable behavior when applied directly to the entire code rather than the diff. Our work contributes a robust benchmark, improved evaluation metrics, and a comprehensive evaluation of LLMs, advancing research on automated SATD repayment.\n                  <\/jats:p>","DOI":"10.1145\/3796704","type":"journal-article","created":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T15:42:55Z","timestamp":1770738175000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["A Large-Scale Empirical Evaluation of LLMs for Automated Self-Admitted Technical Debt Repayment"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4219-642X","authenticated-orcid":false,"given":"Mohammad Sadegh","family":"Sheikhaei","sequence":"first","affiliation":[{"name":"Queen\u2019s University, Kingston, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2208-3893","authenticated-orcid":false,"given":"Yuan","family":"Tian","sequence":"additional","affiliation":[{"name":"Queen\u2019s University, Kingston, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3823-1771","authenticated-orcid":false,"given":"Shaowei","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Manitoba, Winnipeg, Manitoba, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1006-8493","authenticated-orcid":false,"given":"Bowen","family":"Xu","sequence":"additional","affiliation":[{"name":"North Carolina State University at Raleigh, Raleigh, North Carolina, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,9,17]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107376"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2021.102693"},{"key":"e_1_3_2_4_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_2_5_2","unstructured":"Aaditya Bhatia Foutse Khomh Bram Adams and Ahmed E. Hassan. 2023. An empirical study of self-admitted technical debt in machine learning software. arXiv:2311.12019. Retrieved from https:\/\/arxiv.org\/abs\/2311.12019"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3267446"},{"key":"e_1_3_2_7_2","unstructured":"Chatbot Arena LLM Leaderboard. 2024. LMarena AI. Retrieved July 31 2024 from https:\/\/lmarena.ai\/"},{"key":"e_1_3_2_8_2","unstructured":"Banghao Chen Zhaofeng Zhang Nicolas Langren\u00e9 and Shengxin Zhu. 2024. Unleashing the potential of prompt engineering in large language models: A comprehensive review. arXiv:2310.14735. Retrieved from https:\/\/arxiv.org\/abs\/2310.14735"},{"key":"e_1_3_2_9_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pond\u00e9 Jared Kaplan Harrison Edwards Yura Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_10_2","unstructured":"Neil Chowdhury James Aung Chan Jun Shern Oliver Jaffe Dane Sherburn Giulio Starace Evan Mays Rachel Dias Marwan Aljubeh Mia Glaese et al. 2024. Introducing SWE-Bench Verified. Retrieved from https:\/\/openai.com\/index\/introducing-swe-bench-verified\/"},{"key":"e_1_3_2_11_2","unstructured":"Shaiful Chowdhury Hisham Kidwai and Muhammad Asaduzzaman. 2024. Evidence is all we need: Do self-admitted technical debts impact method-level maintenance? arXiv:2411.13777. Retrieved from https:\/\/arxiv.org\/abs\/2411.13777"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/157709.157715"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00074"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556903"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468553"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447247"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3729379"},{"key":"e_1_3_2_19_2","volume-title":"The 12th International Conference on Learning Representations","author":"Jimenez Carlos E.","year":"2024","unstructured":"Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The 12th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=VTF8yNQM66"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_2_23_2","unstructured":"Yikun Li Mohamed Soliman and Paris Avgeriou. 2023. Automatically estimating the effort required to repay self-admitted technical debt. arXiv:2309.06020. Retrieved from https:\/\/arxiv.org\/abs\/2309.06020"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380345"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09917-5"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00125"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","unstructured":"Lin Long Rui Wang Ruixuan Xiao Junbo Zhao Xiao Ding Gang Chen and Haobo Wang. 2024. On llms-driven synthetic data generation curation and evaluation: A survey. arXiv:2406.15126. Retrieved from https:\/\/arxiv.org\/abs\/2406.15126","DOI":"10.18653\/v1\/2024.findings-acl.658"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650105.3652301"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.8"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/MTD.2015.7332619"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00103"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549088"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639176"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524843.3528096"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.3390\/fi15070232"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.31"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3135465"},{"key":"e_1_3_2_38_2","unstructured":"Python Software Foundation. 2024. difflib - Helpers for Computing Deltas. Retrieved from https:\/\/docs.python.org\/3\/library\/difflib.htmlPython standard library"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","unstructured":"Xiaoxue Ren Zhenchang Xing Xin Xia David Lo Xinyu Wang and John Grundy. 2019. Neural network-based detection of self-admitted technical debt: From performance to explainability. 28 3 (Jul. 2019) Article 15 1\u201345. DOI: 10.1145\/3324916","DOI":"10.1145\/3324916"},{"key":"e_1_3_2_40_2","unstructured":"Pranab Sahoo Ayush Kumar Singh Sriparna Saha Vinija Jain Samrat Sohel Mondal and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv:2402.07927. Retrieved from https:\/\/arxiv.org\/abs\/2402.07927"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TechDebt59074.2023.00009"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-024-10548-3"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00063"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME52107.2021.00029"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3405005"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.68"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3715109"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524610.3528387"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196423"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER48275.2020.9054868"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556955"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3796704","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T15:19:34Z","timestamp":1789658374000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3796704"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,17]]},"references-count":51,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2026,10,31]]}},"alternative-id":["10.1145\/3796704"],"URL":"https:\/\/doi.org\/10.1145\/3796704","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,17]]},"assertion":[{"value":"2025-01-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-09-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}