{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,12]],"date-time":"2026-07-12T03:14:00Z","timestamp":1783826040886,"version":"3.55.0"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"5","funder":[{"name":"JSPS for the KAKENHI grants","award":["JP24K02921, JP25K03100"],"award-info":[{"award-number":["JP24K02921, JP25K03100"]}]},{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"crossref","award":["JPMJAP2415"],"award-info":[{"award-number":["JPMJAP2415"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Inamori Research Institute for Science for supporting Yasutaka Kamei"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,5,31]]},"abstract":"<jats:p>\n                    Log statements play a critical role in modern software development, capturing essential run-time information necessary for software maintenance. Recently, new techniques have been developed to automate logging activities, allowing log statements to be injected into code by identifying specific code locations, selecting the appropriate log level, and generating meaningful log messages that describe the behavior being logged. Although automated logging in production code has attracted significant attention, little focus has been given to the injection of logs in test code. To fill this gap, we conduct an empirical study on 5,206,759 Java test methods collected from 6,405 GitHub projects to explore and disclose the effectiveness and limitations of Pre-Trained Language Models (PLMs) and Large Language Models (LLMs) for generating and injecting test log statements. Our findings demonstrate that general-purpose LLMs like GPT-3.5-Turbo, when properly instructed to inject logging statements in test methods, performs comparably to the best-performing PLMs on predicting log level. Additionally, GPT-3.5-Turbo substantially outperforms the best in PLMs on predicting log position, with a 33.97% improvement while also achieving superior performance in predicting log messages in terms of\n                    <jats:italic toggle=\"yes\">BLEU<\/jats:italic>\n                    and\n                    <jats:italic toggle=\"yes\">ROUGE<\/jats:italic>\n                    . This work takes the first step toward evaluating the capability of PLMs and LLMs to generate test log statement. To facilitate future research, we have open sourced all data and source code used in this work.\n                  <\/jats:p>","DOI":"10.1145\/3759915","type":"journal-article","created":{"date-parts":[[2025,8,12]],"date-time":"2025-08-12T19:39:02Z","timestamp":1755027542000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["An Empirical Study on Language Models for Generating Log Statements in Test Code"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-7311-7060","authenticated-orcid":false,"given":"Honglin","family":"Shu","sequence":"first","affiliation":[{"name":"Tianjin University, Tianjin, China and Kyushu University, Fukuoka, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2004-0902","authenticated-orcid":false,"given":"Dong","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7965-7712","authenticated-orcid":false,"given":"Antonio","family":"Mastropaolo","sequence":"additional","affiliation":[{"name":"Computer Science, William and Mary, Williamsburg, VA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2216-3148","authenticated-orcid":false,"given":"Gabriele","family":"Bavota","sequence":"additional","affiliation":[{"name":"Universita della Svizzera Italiana, Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7058-1045","authenticated-orcid":false,"given":"Yasutaka","family":"Kamei","sequence":"additional","affiliation":[{"name":"Kyushu University, Fukuoka, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,4,24]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"[n.d.]. Javaparser. Retrieved from https:\/\/javaparser.org"},{"key":"e_1_3_1_3_2","doi-asserted-by":"crossref","unstructured":"Saima Afrin Joseph Call Khai-Nguyen Nguyen Oscar Chaparro and Antonio Mastropaolo. 2025. Resource-efficient and effective code summarization. arXiv:2502.03617. Retrieved from https:\/\/arxiv.org\/abs\/2502.03617","DOI":"10.1109\/Forge66646.2025.00032"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00055"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380408"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238214"},{"key":"e_1_3_1_7_2","first-page":"4299","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","author":"Christiano Paul F.","year":"2017","unstructured":"Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4299\u20134307."},{"key":"e_1_3_1_8_2","unstructured":"Ozren Dabi\u0107 Emad Aghajani and Gabriele Bavota. 2023. SEART Data Hub. Retrieved from https:\/\/seart-dh.si.usi.ch"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-0441"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00051"},{"key":"e_1_3_1_11_2","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Romain Sauvestre Tal Remez et al. 2023. Code Llama: Open foundation models for code. arXiv:2308.12950. Retrieved from https:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2591062.2591175"},{"key":"e_1_3_1_14_2","unstructured":"Sina Gholamian and Paul A. S. Ward. 2021. A comprehensive survey of logging in software: From logging statements automation to log mining and analysis. arXiv:2110.12489. Retrieved from https:\/\/arxiv.org\/abs\/2110.12489"},{"key":"e_1_3_1_15_2","unstructured":"Sreyan Ghosh Chandra Kiran Reddy Evuru Sonal Kumar Ramaneswaran S Deepali Aneja Zeyu Jin Ramani Duraiswami and Dinesh Manocha. 2024. A closer look at the limitations of instruction tuning. arXiv:2402.05119. Retrieved from https:\/\/arxiv.org\/abs\/2402.05119"},{"key":"e_1_3_1_16_2","unstructured":"Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Alex Vaughan et al. 2024. The llama 3 herd of models. arXiv:2407.21783. Retrieved from https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3166924"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238193"},{"key":"e_1_3_1_19_2","unstructured":"Homepage. 2024. Retrieved from https:\/\/github.com\/SpanShu96\/Empirical-Test-Logging-Generation\/tree\/main"},{"key":"e_1_3_1_20_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv:2308.10620. Retrieved from https:\/\/arxiv.org\/abs\/2308.10620"},{"key":"e_1_3_1_21_2","unstructured":"Haolin Jin Linghan Huang Haipeng Cai Jun Yan Bo Li and Huaming Chen. 2024. From llms to llm-based agents for software engineering: A survey of current challenges and future. arXiv:2408.02479. Retrieved from https:\/\/arxiv.org\/abs\/2408.02479"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9518-0"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678927"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316447003000308"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2970422"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9456-2"},{"key":"e_1_3_1_27_2","first-page":"1","article-title":"Structured chain-of-thought prompting for code generation","author":"Li Jia","year":"2025","unstructured":"Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2025. Structured chain-of-thought prompting for code generation. ACM Transactions on Software Engineering and Methodology 34, 2 (2025), 1\u201323.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3475375"},{"key":"e_1_3_1_29_2","unstructured":"Zhenhao Li An Ran Chen Xing Hu Xin Xia Tse-Hsun Chen and Weiyi Shang. 2023. Are they all good? Studying practitioners\u2019 expectations on the readability of log messages. arXiv:2308.08836. Retrieved from https:\/\/arxiv.org\/abs\/2308.08836"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416636"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00131"},{"key":"e_1_3_1_32_2","first-page":"74","volume-title":"Annual Meeting of the Association for Computational Linguistics","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics, 74\u201381."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534379"},{"issue":"9","key":"e_1_3_1_34_2","first-page":"2012","article-title":"Which variables should i log","volume":"47","author":"Liu Zhongxin","year":"2019","unstructured":"Zhongxin Liu, Xin Xia, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shanping Li. 2019. Which variables should i log? IEEE Transactions on Software Engineering 47, 9 (2019), 2012\u20132031.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_35_2","unstructured":"Wei Ma Shangqing Liu Wenhan Wang Qiang Hu Ye Liu Cen Zhang Liming Nie and Yang Liu. 2023. The scope of ChatGPT in software engineering: A thorough investigation. arXiv:2305.12138. Retrieved from https:\/\/arxiv.org\/abs\/2305.12138"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111947"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3511561"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF02295996"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/658"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","unstructured":"Marius Mosbach Tiago Pimentel Shauli Ravfogel Dietrich Klakow and Yanai Elazar. 2023. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. arXiv:2305.16938. Retrieved from https:\/\/arxiv.org\/abs\/2305.16938","DOI":"10.18653\/v1\/2023.findings-acl.779"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2076450.2076466"},{"key":"e_1_3_1_42_2","unstructured":"OpenAI. 2022. GPT-3.5. Retrieved from https:\/\/platform.openai.com\/docs\/models\/gpt-3-5"},{"key":"e_1_3_1_43_2","unstructured":"OpenAI. 2023. ChatGPT. Retrieved from https:\/\/openai.com\/blog\/chatgpt\/"},{"key":"e_1_3_1_44_2","unstructured":"OpenAI. 2024. Retrieved from https:\/\/openai.com\/"},{"key":"e_1_3_1_45_2","unstructured":"OpenAI. 2024. Introducing GPT-4o and more tools to ChatGPT free users. Retrieved from https:\/\/openai.com\/index\/gpt-4o-and-more-tools-to-chatgpt-free\/"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.5555\/3600270.3602281"},{"key":"e_1_3_1_47_2","first-page":"311","volume-title":"Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.145"},{"key":"e_1_3_1_49_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21 (2020), Article 140, 1\u201367.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_3_1_50_2","unstructured":"Pranab Sahoo Ayush Kumar Singh Sriparna Saha Vinija Jain Samrat Mondal and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv:2402.07927. Retrieved from https:\/\/arxiv.org\/abs\/2402.07927"},{"key":"e_1_3_1_51_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_3_1_53_2","first-page":"196","article-title":"Individual comparisons by ranking methods","volume":"1","author":"Wilcoxon Frank","year":"1945","unstructured":"Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics 1 (1945), 196\u2013202.","journal-title":"Biometrics"},{"key":"e_1_3_1_54_2","unstructured":"Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault R\u00e9mi Louf Morgan Funtowicz et al. 2019. Huggingface\u2019s transformers: State-of-the-art natural language processing. arXiv:1910.03771. Retrieved from https:\/\/arxiv.org\/abs\/1910.03771"},{"key":"e_1_3_1_55_2","unstructured":"Xiaoyuan Xie Zhipeng Cai Songqiang Chen and Jifeng Xuan. 2023. Generate complete logging statements with an efficient end-to-end approach. arXiv:2311.02862. Retrieved from https:\/\/arxiv.org\/abs\/2311.02862"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623326"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227202"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10139-0"},{"key":"e_1_3_1_59_2","unstructured":"Shengyu Zhang Linfeng Dong Xiaoya Li Sen Zhang Xiaofei Sun Shuhe Wang Jiwei Li Runyi Hu Tianwei Zhang Fei Wu et al. 2023. Instruction tuning for large language models: A survey. arXiv:2308.10792. Retrieved from https:\/\/arxiv.org\/abs\/2308.10792"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338931"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132778"},{"key":"e_1_3_1_62_2","unstructured":"Xin Zhou Sicong Cao Xiaobing Sun and David Lo. 2024. Large language model for vulnerability detection and repair: Literature review and roadmap. arXiv:2404.02525. Retrieved from https:\/\/arxiv.org\/abs\/2404.02525"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338961"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.60"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3759915","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T06:33:06Z","timestamp":1777098786000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3759915"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,24]]},"references-count":63,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,5,31]]}},"alternative-id":["10.1145\/3759915"],"URL":"https:\/\/doi.org\/10.1145\/3759915","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,24]]},"assertion":[{"value":"2024-09-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}