{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T16:33:48Z","timestamp":1783096428616,"version":"3.54.6"},"reference-count":102,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>Accurate classification of issues is essential for effective project management and timely responses, as the volume of issue reports continues to grow. Manual classification is labor-intensive and error-prone, necessitating automated solutions. While large language models (LLMs) show promise in automated issue labeling, most research focuses on broad categorization (e.g., bugs, feature requests), with limited attention to fine-grained categorization. Understanding specific bug types is crucial, as different bugs require tailored resolution strategies.<\/jats:p>\n                  <jats:p>This study addresses this gap by evaluating LLMs and prompt engineering strategies for fine-grained bug report categorization. We analyze 221,184 fine-grained bug report category labels generated by selected LLMs using various prompt engineering strategies for 1,024 bug reports. We examine how LLMs and prompt engineering influence output characteristics, control over outputs, and categorization performance. Our findings highlight that LLMs and prompt engineering significantly impact output consistency and classification capability, with some yielding consistent results and others introducing variability. Based on these findings, we analyze the agreements and disagreements between LLM-generated labels and human annotations to assess category correctness. Our results suggest that examining label consistency and discrepancies can serve as a complementary method for validating bug report categories, identifying unclear reports, and detecting misclassifications in human annotations.<\/jats:p>","DOI":"10.1145\/3736408","type":"journal-article","created":{"date-parts":[[2025,5,20]],"date-time":"2025-05-20T13:03:21Z","timestamp":1747746201000},"page":"1-47","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Exploring Fine-Grained Bug Report Categorization with Large Language Models and Prompt Engineering: An\u00a0Empirical Study"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6975-6752","authenticated-orcid":false,"given":"Anil","family":"Koyuncu","sequence":"first","affiliation":[{"name":"Computer Engineering, Bilkent University, Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,2,13]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3069248"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00065"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643787.3648041"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MOBILSoft59058.2023.00012"},{"key":"e_1_3_3_6_2","unstructured":"Xavier Amatriain. 2024. Prompt design and engineering: Introduction and advanced methods. arXiv:2401.14423. Retrieved from https:\/\/arxiv.org\/abs\/2401.14423"},{"key":"e_1_3_3_7_2","first-page":"304","volume-title":"Proceedings of the 2008 Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds","author":"Antoniol Giuliano","year":"2008","unstructured":"Giuliano Antoniol, Kamel Ayari, Massimiliano Di Penta, Foutse Khomh, and Yann-Ga\u00ebl Gu\u00e9h\u00e9neuc. 2008. Is it a bug or an enhancement? A text-based approach to classify change requests. In Proceedings of the 2008 Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds, 304\u2013318."},{"key":"e_1_3_3_8_2","doi-asserted-by":"crossref","unstructured":"Gabriel Aracena Kyle Luster Fabio Santos Igor Steinmacher and Marco A. Gerosa. 2024. Applying large language models API to issue classification problem. arXiv:2401.04637. Retrieved from https:\/\/arxiv.org\/abs\/2401.04637","DOI":"10.1145\/3643787.3648043"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.15346\/hc.v1i1.3"},{"key":"e_1_3_3_10_2","unstructured":"Viraat Aryabumi John Dang Dwarak Talupuru Saurabh Dash David Cairuz Hangyu Lin Bharat Venkitesh Madeline Smith Kelly Marchisio Sebastian Ruder et\u00a0al. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv:2405.15032. Retrieved from https:\/\/arxiv.org\/abs\/2405.15032"},{"key":"e_1_3_3_11_2","unstructured":"Yuntao Bai Andy Jones Kamal Ndousse Amanda Askell Anna Chen Nova DasSarma Dawn Drain Stanislav Fort Deep Ganguli Tom Henighan et\u00a0al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. Retrieved from https:\/\/arxiv.org\/abs\/2204.05862"},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1145\/3528588.3528663","volume-title":"Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering","author":"Bharadwaj Shikhar","year":"2022","unstructured":"Shikhar Bharadwaj and Tushar Kadam. 2022. Github issue classification using BERT-style models. In Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering, 40\u201343."},{"key":"e_1_3_3_13_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","unstructured":"Gemma Catolino Fabio Palomba Andy Zaidman and Filomena Ferrucci. 2019. Not all bugs are the same: Understanding characterizing and classifying bug types. 152 (2019) 165\u2013181. DOI: 10.1016\/j.jss.2019.03.002","DOI":"10.1016\/j.jss.2019.03.002"},{"key":"e_1_3_3_15_2","first-page":"376","volume-title":"Proceedings of the 2014 7th International Conference on Contemporary Computing (IC3)","author":"Chawla Indu","year":"2014","unstructured":"Indu Chawla and Sandeep K. Singh. 2014. Automatic bug labeling using semantic information from LSI. In Proceedings of the 2014 7th International Conference on Contemporary Computing (IC3). IEEE, 376\u2013381."},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09893-w"},{"key":"e_1_3_3_17_2","unstructured":"Robert Chew John Bollenbacher Michael Wenger Jessica Speer and Annice Kim. 2023. LLM-assisted content analysis: Using large language models to support deductive coding. arXiv:2306.14924. Retrieved from https:\/\/arxiv.org\/abs\/2306.14924"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","unstructured":"Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv:2305.01937. DOI: 10.48550\/arXiv.2305.01937","DOI":"10.48550\/arXiv.2305.01937"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.177364"},{"key":"e_1_3_3_20_2","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1145\/3528588.3528659","volume-title":"Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering","author":"Colavito Giuseppe","year":"2022","unstructured":"Giuseppe Colavito, Filippo Lanubile, and Nicole Novielli. 2022. Issue report classification using pre-trained language models. In Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering, 29\u201332."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/NLBSE59153.2023.00011"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111838"},{"key":"e_1_3_3_23_2","first-page":"469","volume-title":"Proceedings of the 2024 IEEE\/ACM 21st International Conference on Mining Software Repositories (MSR)","author":"Colavito Giuseppe","year":"2024","unstructured":"Giuseppe Colavito, Filippo Lanubile, Nicole Novielli, and Luigi Quaranta. 2024. Leveraging GPT-like LLMs to automate issue labeling. In Proceedings of the 2024 IEEE\/ACM 21st International Conference on Mining Software Repositories (MSR), 469\u2013480."},{"issue":"2","key":"e_1_3_3_24_2","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1080\/07481756.2024.2303715","article-title":"Intercoder reliability for use in qualitative research and evaluation","volume":"57","author":"Coleman Monica L.","year":"2024","unstructured":"Monica L. Coleman, Moira Ragan, and Tahani Dari. 2024. Intercoder reliability for use in qualitative research and evaluation. Measurement and Evaluation in Counseling and Development 57, 2 (2024), 136\u2013146.","journal-title":"Measurement and Evaluation in Counseling and Development"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-024-10494-0"},{"key":"e_1_3_3_26_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from https:\/\/arxiv.org\/abs\/1810.04805"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-024-09675-3"},{"key":"e_1_3_3_28_2","first-page":"259","volume-title":"Proceedings of the 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)","author":"Du Xiaoting","year":"2017","unstructured":"Xiaoting Du, Zheng Zheng, Guanping Xiao, and Beibei Yin. 2017. The automatic classification of fault trigger based bug report. In Proceedings of the 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 259\u2013265."},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643787.3648039"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502004"},{"key":"e_1_3_3_32_2","first-page":"344","volume-title":"Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS)","author":"Goseva-Popstojanova Katerina","year":"2018","unstructured":"Katerina Goseva-Popstojanova and Jacob Tyo. 2018. Identification of security related bug reports via text mining using supervised and unsupervised classification. In Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 344\u2013355."},{"key":"e_1_3_3_33_2","first-page":"3","volume-title":"Symposium on Reliability in Distributed Software and Database Systems","author":"Jim Gray","year":"1986","unstructured":"Jim Gray. 1986. Why do computers stop and what can be done about it?. In Symposium on Reliability in Distributed Software and Database Systems, 3\u201312."},{"key":"e_1_3_3_34_2","first-page":"425","article-title":"A classification of software faults","volume":"27","author":"Grottke Michael","year":"2005","unstructured":"Michael Grottke and Kishor S. Trivedi. 2005. A classification of software faults. Journal of Reliability Engineering Association of Japan 27, 7 (2005), 425\u2013438.","journal-title":"Journal of Reliability Engineering Association of Japan"},{"key":"e_1_3_3_35_2","first-page":"1","volume-title":"Authorea Preprints","author":"Usman Hadi Muhammad","year":"2023","unstructured":"Muhammad Usman Hadi, Qasem Al-Tashi, Rizwan Qureshi, Abbas Shah, Amgad Muneer, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, et\u00a0al. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints 1 (2023), 1\u201326."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","unstructured":"Xingwei He Zhenghao Lin Yeyun Gong A.-Long Jin Hang Zhang Chen Lin Jian Jiao Siu Ming Yiu Nan Duan and Weizhu Chen. 2024. AnnoLLM: Making large language models to be better crowdsourced annotators. arXiv:2303.16854. DOI: 10.48550\/arXiv.2303.16854","DOI":"10.48550\/arXiv.2303.16854"},{"key":"e_1_3_3_37_2","first-page":"79568","article-title":"A comparison of pretrained models for classifying issue reports","author":"Heo Jueun","year":"2024","unstructured":"Jueun Heo, Gibeom Kwon, Changwon Kwak, and Seonah Lee. 2024. A comparison of pretrained models for classifying issue reports. IEEE Access 12 (2024), 79568\u201379584.","journal-title":"IEEE Access"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.5555\/2486788.2486840"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.array.2022.100189"},{"key":"e_1_3_3_40_2","volume-title":"ACM Transactions on Software Engineering and Methodology","author":"Hou Xinyi","year":"2023","unstructured":"Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology (2023)."},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.5555\/1564131.1564137"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-014-0155-1"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","unstructured":"Dave Hulbert. 2023. Using Tree-of-Thought Prompting to boost ChatGPT's Reasoning. Zenodo. DOI: 10.5281\/zenodo.10323452","DOI":"10.5281\/zenodo.10323452"},{"key":"e_1_3_3_44_2","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1145\/3528588.3528662","volume-title":"Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering","author":"Maliheh Izadi","year":"2022","unstructured":"Maliheh Izadi. 2022. Catiss: An intelligent tool for categorizing issues reports using transformers. In Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering, 44\u201347."},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/1595696.1595715"},{"key":"e_1_3_3_46_2","unstructured":"Albert Q. Jiang Alexandre Sablayrolles Arthur Mensch Chris Bamford Devendra Singh Chaplot Diego de las Casas Florian Bressand Gianna Lengyel Guillaume Lample Lucile Saulnier et\u00a0al. 2023. Mistral 7B. arXiv:2310.06825. Retrieved from https:\/\/arxiv.org\/abs\/2310.06825"},{"key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"406","DOI":"10.1109\/ICSME.2019.00070","volume-title":"Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Kallis Rafael","year":"2019","unstructured":"Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2019. Ticket tagger: Machine learning driven issue classification. In Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 406\u2013409."},{"key":"e_1_3_3_48_2","first-page":"126","volume-title":"Proceedings of the 2014 19th International Conference on Engineering of Complex Computer Systems","author":"Singh Kochhar Pavneet","year":"2014","unstructured":"Pavneet Singh Kochhar, Ferdian Thung, and David Lo. 2014. Automatic fine-grained issue report reclassification. In Proceedings of the 2014 19th International Conference on Engineering of Complex Computer Systems. IEEE, 126\u2013135."},{"key":"e_1_3_3_49_2","unstructured":"Takeshi Kojima Shixiang Shane Gu Machel Reid Yutaka Matsuo and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In\u00a0Proceedings of the 36th International Conference on Neural Information Processing Systems. Curran Associates Inc. New Orleans LA Article 1613 15 pages."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.3390\/app12010338"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASEW52652.2021.00015"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2024.112588"},{"key":"e_1_3_3_53_2","unstructured":"Tian Liang Zhiwei He Wenxiang Jiao Xing Wang Yan Wang Rui Wang Yujiu Yang Zhaopeng Tu and Shuming Shi. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv:2305.19118. Retrieved from https:\/\/arxiv.org\/abs\/2305.19118"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","unstructured":"Jieyi Long. 2023. Large language model guided tree-of-thought. arXiv:2305.08291. DOI: 10.48550\/arXiv.2305.08291","DOI":"10.48550\/arXiv.2305.08291"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650105.3652301"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3651640.3651648"},{"key":"e_1_3_3_57_2","doi-asserted-by":"crossref","unstructured":"Rajiv Movva Pang Wei Koh and Emma Pierson. 2024. Annotation alignment: Comparing LLM and human annotations of conversational safety. arXiv:2406.06369. Retrieved from https:\/\/arxiv.org\/abs\/2406.06369","DOI":"10.18653\/v1\/2024.emnlp-main.511"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW53611.2021.00113"},{"key":"e_1_3_3_59_2","unstructured":"Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Nick Barnes and Ajmal Mian. 2023. A comprehensive overview of large language models. arXiv:2307.06435. Retrieved from https:\/\/arxiv.org\/abs\/2307.06435"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695330"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","unstructured":"Wendk\u00fbuni C. Ou\u00e9draogo Yinghua Li Kader Kabor\u00e9 Xunzhu Tang Anil Koyuncu Jacques Klein David Lo and Tegawend\u00e9 F. Bissyand\u00e9. 2024. Test smells in LLM-generated unit tests. arXiv:2410.10628. DOI: 10.48550\/arXiv.2410.10628","DOI":"10.48550\/arXiv.2410.10628"},{"key":"e_1_3_3_62_2","first-page":"47","volume-title":"Proceedings of the 2021 IEEE\/ACM 29th International Conference on Program Comprehension (ICPC)","author":"Perez Quentin","year":"2021","unstructured":"Quentin Perez, Pierre-Antoine Jean, Christelle Urtado, and Sylvain Vauttier. 2021. Bug or not bug? That is the question. In Proceedings of the 2021 IEEE\/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 47\u201358."},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3536168.3543301"},{"key":"e_1_3_3_64_2","volume-title":"Proceedings of the 2013 20th Asia-Pacific Software Engineering Conference (APSEC)","author":"Pingclasai Natthakul","year":"2013","unstructured":"Natthakul Pingclasai, Hideaki Hata, and Ken-ichi Matsumoto. 2013. Classifying bug reports to bugs and other requests using topic modeling. In Proceedings of the 2013 20th Asia-Pacific Software Engineering Conference (APSEC), Vol. 2. IEEE, 13\u201318."},{"issue":"140","key":"e_1_3_3_65_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1\u201367.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643787.3648042"},{"key":"e_1_3_3_67_2","unstructured":"Pranab Sahoo Ayush Kumar Singh Sriparna Saha Vinija Jain Samrat Mondal and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv:2402.07927. Retrieved from https:\/\/arxiv.org\/abs\/2402.07927"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/1414004.1414030"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3624724"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","unstructured":"Jiho Shin Clark Tang Tahmineh Mohati Maleknaz Nayebi Song Wang and Hadi Hemmati. 2023. Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks. arXiv:2310.10508. DOI: 10.48550\/arXiv.2310.10508","DOI":"10.48550\/arXiv.2310.10508"},{"key":"e_1_3_3_72_2","unstructured":"Chenglei Si Zhe Gan Zhengyuan Yang Shuohang Wang Jianfeng Wang Jordan Boyd-Graber and Lijuan Wang. 2022. Prompting gpt-3 to be reliable. arXiv:2210.09150. Retrieved from https:\/\/arxiv.org\/abs\/2210.09150"},{"key":"e_1_3_3_73_2","first-page":"33","volume-title":"Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering","author":"Latif Siddiq Mohammed","year":"2022","unstructured":"Mohammed Latif Siddiq and Joanna C. S. Santos. 2022. Bert-based github issue report classification. In Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering, 33\u201336."},{"key":"e_1_3_3_74_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Sun Hao","year":"2023","unstructured":"Hao Sun, Alihan H\u00fcy\u00fck, and Mihaela van der Schaar. 2023. Query-dependent prompt evaluation and optimization with offline inverse RL. In Proceedings of the 12th International Conference on Learning Representations."},{"key":"e_1_3_3_75_2","first-page":"1665","volume-title":"Empirical Software Engineering","volume":"19","author":"Tan Lin","year":"2014","unstructured":"Lin Tan, Chen Liu, Zhenmin Li, Xuanhui Wang, Yuanyuan Zhou, and Chengxiang Zhai. 2014. Bug characteristics in open source software. Empirical Software Engineering 19 (2014), 1665\u20131705."},{"key":"e_1_3_3_76_2","unstructured":"Gemma Team Morgane Riviere Shreya Pathak Pier Giuseppe Sessa Cassidy Hardin Surya Bhupatiraju L\u00e9onard Hussenot Thomas Mesnard Bobak Shahriari Alexandre Ram\u00e9 et\u00a0al. 2024. Gemma 2: Improving open language models at a practical size. arXiv:2408.00118. Retrieved from https:\/\/arxiv.org\/abs\/2408.00118"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2012.30"},{"key":"e_1_3_3_78_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et\u00a0al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3528588.3528661"},{"key":"e_1_3_3_80_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"5","key":"e_1_3_3_81_2","first-page":"360","article-title":"Understanding interobserver agreement: The kappa statistic","volume":"37","author":"Viera Anthony J.","year":"2005","unstructured":"Anthony J. Viera and Joanne M. Garrett. 2005. Understanding interobserver agreement: The kappa statistic. Family Medicine 37, 5 (2005), 360\u2013363.","journal-title":"Family Medicine"},{"key":"e_1_3_3_82_2","doi-asserted-by":"crossref","unstructured":"Andreas Vogelsang and Jannik Fischbach. 2024. Using large language models for natural language processing tasks in requirements engineering: A systematic guideline. arXiv:2402.13823. Retrieved from https:\/\/arxiv.org\/abs\/2402.13823","DOI":"10.1007\/978-3-031-73143-3_16"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3368208"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.107476"},{"key":"e_1_3_3_85_2","doi-asserted-by":"crossref","unstructured":"Lei Wang Wanyu Xu Yihuai Lan Zhiqiang Hu Yunshi Lan Roy Ka-Wei Lee and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv:2305.04091. Retrieved from https:\/\/arxiv.org\/abs\/2305.04091","DOI":"10.18653\/v1\/2023.acl-long.147"},{"key":"e_1_3_3_86_2","unstructured":"Xuezhi Wang Jason Wei Dale Schuurmans Quoc Le Ed Chi Sharan Narang Aakanksha Chowdhery and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv:2203.11171. Retrieved from https:\/\/arxiv.org\/abs\/2203.11171"},{"key":"e_1_3_3_87_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106530"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3063727"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCS.2014.14"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2021.3074512"},{"key":"e_1_3_3_92_2","unstructured":"An Yang Baosong Yang Binyuan Hui Bo Zheng Bowen Yu Chang Zhou Chengpeng Li Chengyuan Li Dayiheng Liu Fei Huang et\u00a0al. 2024. Qwen2 technical report. arXiv:2407.10671. Retrieved from https:\/\/arxiv.org\/abs\/2407.10671"},{"key":"e_1_3_3_93_2","doi-asserted-by":"publisher","unstructured":"Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas L. Griffiths Yuan Cao and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv:2305.10601. DOI: 10.48550\/arXiv.2305.10601","DOI":"10.48550\/arXiv.2305.10601"},{"key":"e_1_3_3_94_2","doi-asserted-by":"crossref","first-page":"54608","DOI":"10.1109\/ACCESS.2024.3389497","article-title":"GPT (generative pre-trained transformer)\u2013a comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions","author":"Yenduri Gokul","year":"2024","unstructured":"Gokul Yenduri, M. Ramalingam, G. Chemmalar Selvi, Y. Supriya, Gautam Srivastava, Praveen Kumar Reddy Maddikunta, G. Deepti Raj, Rutvij H. Jhaveri, B. Prabadevi, Weizheng Wang, et\u00a0al. 2024. GPT (generative pre-trained transformer)\u2013a comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions. IEEE Access 12 (2024), 54608\u201354649.","journal-title":"IEEE Access"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/SERA61261.2024.10685583"},{"key":"e_1_3_3_96_2","unstructured":"Denny Zhou Nathanael Sch\u00e4rli Le Hou Jason Wei Nathan Scales Xuezhi Wang Dale Schuurmans Claire Cui Olivier Bousquet Quoc Le et\u00a0al. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv:2205.10625. Retrieved from https:\/\/arxiv.org\/abs\/2205.10625"},{"key":"e_1_3_3_97_2","volume-title":"Large language models are human-level prompt engineers","author":"Zhou Yongchao","year":"2022","unstructured":"Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv:2211.01910. Retrieved from https:\/\/arxiv.org\/abs\/2211.01910"},{"key":"e_1_3_3_98_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.1770"},{"key":"e_1_3_3_99_2","volume-title":"First Conference on Language Modeling","author":"Zhu Banghua","year":"2024","unstructured":"Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang and Jiantao Jiao. 2024. Starling-7B: Improving LLM helpfulness & harmlessness with RLAIF. In First Conference on Language Modeling. Retrieved from https:\/\/openreview.net\/forum?id=GqDntYTTbk"},{"key":"e_1_3_3_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2892646"},{"key":"e_1_3_3_101_2","unstructured":"Daniel M. Ziegler Nisan Stiennon Jeffrey Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv:1909.08593. Retrieved from https:\/\/arxiv.org\/abs\/1909.08593"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.63"},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.845"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3736408","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T14:37:00Z","timestamp":1770993420000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3736408"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,13]]},"references-count":102,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3736408"],"URL":"https:\/\/doi.org\/10.1145\/3736408","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,13]]},"assertion":[{"value":"2024-11-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}