{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T02:23:06Z","timestamp":1778034186283,"version":"3.51.4"},"reference-count":123,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T00:00:00Z","timestamp":1763424000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T00:00:00Z","timestamp":1763424000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2026,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Conversational large-language models (LLMs), such as ChatGPT, are extensively used for issue resolution tasks, particularly for generating ideas to implement new features or resolve bugs. However, not all developer-LLM conversations are useful for effective issue resolution and it is still unknown what makes some of these conversations not helpful. In this paper, we analyze 686 developer-ChatGPT conversations shared within GitHub issue threads to identify characteristics that make these conversations effective for issue resolution. First, we empirically analyze the conversations and their corresponding issue threads to distinguish helpful from unhelpful conversations. We begin by categorizing the types of tasks developers seek help with (e.g.,\n                    <jats:italic>code generation<\/jats:italic>\n                    ,\n                    <jats:italic>bug identification and fixing<\/jats:italic>\n                    ,\n                    <jats:italic>test generation<\/jats:italic>\n                    ), to better understand the scenarios in which ChatGPT is most effective. Next, we examine a wide range of conversational, project, and issue-related metrics to uncover statistically significant factors associated with helpful conversations. Finally, we identify common deficiencies in unhelpful ChatGPT responses to highlight areas that could inform the design of more effective developer-facing tools. We found that only 62% of the ChatGPT conversations were helpful for successful issue resolution. Among different tasks related to issue resolution, ChatGPT was most helpful in assisting with code generation, and tool\/library\/API recommendations, but struggled with generating code explanations. Our conversational metrics reveal that helpful conversations are shorter, more readable, and exhibit higher semantic and linguistic alignment. Our project metrics reveal that larger, more popular projects and experienced developers benefit more from ChatGPT\u2019s assistance. Our issue metrics indicate that ChatGPT is more effective on simpler issues characterized by limited developer activity and faster resolution times. These typically involve well-scoped technical problems such as compilation errors and tool feature requests. In contrast, it performs less effectively on complex issues that demand deep project-specific understanding, such as system-level code debugging and refactoring. The most common deficiencies in unhelpful ChatGPT responses include incorrect information and lack of comprehensiveness. Our findings have wide implications including guiding developers on effective interaction strategies for issue resolution, informing the development of tools or frameworks to support optimal prompt design, and providing insights on fine-tuning LLMs for issue resolution tasks.\n                  <\/jats:p>","DOI":"10.1007\/s10664-025-10745-8","type":"journal-article","created":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T10:28:13Z","timestamp":1763461693000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["What characteristics make ChatGPT effective for software issue resolution? An empirical study of task, project, and conversational signals in GitHub issues"],"prefix":"10.1007","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1517-7135","authenticated-orcid":false,"given":"Ramtin","family":"Ehsani","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sakshi","family":"Pathak","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Esteban","family":"Parra","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sonia","family":"Haiduc","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Preetha","family":"Chatterjee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,11,18]]},"reference":[{"key":"10745_CR1","doi-asserted-by":"crossref","unstructured":"Ahmed T, Devanbu P, Treude C, Pradel M (2025) Can llms replace manual annotation of software engineering artifacts? arXiv:2408.05534","DOI":"10.1109\/MSR66628.2025.00086"},{"key":"10745_CR2","doi-asserted-by":"publisher","unstructured":"Al-Msie\u2019deen R (2024) Bushradbr: An automatic approach to retrieving duplicate bug reports. Int J Comput Digi Syst 15(1):221\u2013238. https:\/\/doi.org\/10.12785\/ijcds\/150118","DOI":"10.12785\/ijcds\/150118"},{"issue":"5","key":"10745_CR3","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1111\/opo.12131","volume":"34","author":"RA Armstrong","year":"2014","unstructured":"Armstrong RA (2014) When to use the Bonferroni correction. Ophthalmic Physiol Opt J Br Coll Ophthalmic Opticians (Optometrists) 34(5):502\u2013508. https:\/\/doi.org\/10.1111\/opo.12131","journal-title":"Ophthalmic Physiol Opt J Br Coll Ophthalmic Opticians (Optometrists)"},{"key":"10745_CR4","doi-asserted-by":"publisher","unstructured":"Ayala O, Bechard P (2024) Reducing hallucination in structured outputs via retrieval-augmented generation. In: Yang Y, Davani A, Sil A, Kumar A (eds) Proceedings of the 2024 conference of the North American Chapter of the association for computational linguistics: Human language technologies (Volume 6: Industry Track), association for computational linguistics, Mexico City, Mexico, pp 228\u2013238.https:\/\/doi.org\/10.18653\/v1\/2024.naacl-industry.19","DOI":"10.18653\/v1\/2024.naacl-industry.19"},{"issue":"6","key":"10745_CR5","doi-asserted-by":"publisher","DOI":"10.1115\/1.4056539","volume":"145","author":"S Banerjee","year":"2023","unstructured":"Banerjee S, Sussman M, Lian Y (2023) Dimensional analysis in error reduction for prediction of nucleate boiling heat flux by artificial neural networks for limited dataset. ASME J Heat Mass Transfer 145(6):061602","journal-title":"ASME J Heat Mass Transfer"},{"key":"10745_CR6","unstructured":"Barrus T (2024) pyspellchecker: Pure python spell checker based on work by Peter Norvig"},{"issue":"1","key":"10745_CR7","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","volume":"57","author":"Y Benjamini","year":"1995","unstructured":"Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc: Ser B (Methodol) 57(1):289\u2013300. https:\/\/doi.org\/10.1111\/j.2517-6161.1995.tb02031.x","journal-title":"J Roy Stat Soc: Ser B (Methodol)"},{"key":"10745_CR8","unstructured":"Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3(null):993\u20131022"},{"key":"10745_CR9","unstructured":"Boyd RL (2022) The Development and Psychometric Properties of LIWC-22"},{"key":"10745_CR10","doi-asserted-by":"crossref","unstructured":"Champa AI, Rabbi MF, Nachuma C, Zibran MF (2024) Chatgpt in action: Analyzing its use in software development","DOI":"10.1145\/3643991.3645077"},{"key":"10745_CR11","doi-asserted-by":"publisher","unstructured":"Chatterjee P, Gause B, Hedinger H, Pollock L (2017) Extracting code segments and their descriptions from research articles. In: 2017 IEEE\/ACM 14th international conference on mining software repositories (MSR), pp 91\u2013101. https:\/\/doi.org\/10.1109\/MSR.2017.10","DOI":"10.1109\/MSR.2017.10"},{"key":"10745_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.110454","volume":"159","author":"P Chatterjee","year":"2020","unstructured":"Chatterjee P, Kong M, Pollock L (2020) Finding help with programming errors: An exploratory study of novice software engineers\u2019 focus in stack overflow posts. J Syst Softw 159:110454","journal-title":"J Syst Softw"},{"key":"10745_CR13","doi-asserted-by":"publisher","unstructured":"Chatterjee P, Damevski K, Kraft NA, Pollock L (2021) Automatically identifying the quality of developer chats for post hoc use. ACM Trans Softw Eng Methodol 30(4). https:\/\/doi.org\/10.1145\/3450503","DOI":"10.1145\/3450503"},{"key":"10745_CR14","doi-asserted-by":"publisher","unstructured":"Chen C, Xing Z, Wang X (2017) Unsupervised software-specific morphological forms inference from informal discussions. In: 2017 IEEE\/ACM 39th international conference on software engineering (ICSE), pp 450\u2013461. https:\/\/doi.org\/10.1109\/ICSE.2017.48","DOI":"10.1109\/ICSE.2017.48"},{"key":"10745_CR15","doi-asserted-by":"publisher","unstructured":"Chen Z, Jiang L (2025) Evaluating software development agents: Patch patterns, code quality, and issue complexity in real-world github scenarios. In: 2025 IEEE international conference on software analysis, evolution and reengineering (SANER), pp 657\u2013668. https:\/\/doi.org\/10.1109\/SANER64311.2025.00068","DOI":"10.1109\/SANER64311.2025.00068"},{"key":"10745_CR16","doi-asserted-by":"crossref","unstructured":"Chouchen M, Bessghaier N, Begoug M, Ouni A, AlOmar EA, Wiem\u00a0Mkaouer M (2024) How do software developers use chatgpt? an exploratory study on github pull requests. In: 2024 IEEE\/ACM 21st international conference on mining software repositories (MSR), pp 212\u2013216","DOI":"10.1145\/3643991.3645084"},{"key":"10745_CR17","unstructured":"Clark HH, Brennan S (1991) Grounding in communication. In: Perspectives on socially shared cognition"},{"key":"10745_CR18","volume-title":"Sampling Techniques","author":"WG Cochran","year":"1977","unstructured":"Cochran WG (1977) Sampling Techniques, 3rd edn. John Wiley & Sons, New York, NY","edition":"3"},{"key":"10745_CR19","doi-asserted-by":"publisher","unstructured":"Colavito G, Lanubile F, Novielli N, Quaranta L (2024) Leveraging gpt-like llms to automate issue labeling. In: Proceedings of the 21st international conference on mining software repositories, association for computing machinery, New York, NY, USA, MSR \u201924, p 469\u2013480. https:\/\/doi.org\/10.1145\/3643991.3644903","DOI":"10.1145\/3643991.3644903"},{"key":"10745_CR20","doi-asserted-by":"publisher","unstructured":"Cong G, Wang L, Lin CY, Song YI, Sun Y (2008) Finding question-answer pairs from online forums. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval, association for computing machinery, New York, NY, USA, SIGIR \u201908, p 467\u2013474. https:\/\/doi.org\/10.1145\/1390334.1390415","DOI":"10.1145\/1390334.1390415"},{"key":"10745_CR21","doi-asserted-by":"crossref","unstructured":"Corbin J, Strauss A (2008) Basics of Qualitative Research, 3rd edn. SAGE Publications Inc, techniques and procedures for developing grounded theory","DOI":"10.4135\/9781452230153"},{"key":"10745_CR22","doi-asserted-by":"publisher","unstructured":"Cordy JR, Roy CK (2011) The nicad clone detector. In: Proceedings of the 2011 IEEE 19th International Conference on Program Comprehension. IEEE Comput Soc, USA, ICPC \u201911, p 219\u2013220. https:\/\/doi.org\/10.1109\/ICPC.2011.26","DOI":"10.1109\/ICPC.2011.26"},{"key":"10745_CR23","doi-asserted-by":"publisher","unstructured":"Alencar\u00a0da Costa D, Abebe SL, Mcintosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: 2014 IEEE international conference on software maintenance and evolution, pp 281\u2013290. https:\/\/doi.org\/10.1109\/ICSME.2014.50","DOI":"10.1109\/ICSME.2014.50"},{"key":"10745_CR24","doi-asserted-by":"publisher","unstructured":"Danescu-Niculescu-Mizil C, Gamon M, Dumais S (2011) Mark my words! linguistic style accommodation in social media. In: Proceedings of the 20th international conference on world wide web, association for computing machinery, New York, NY, USA, WWW \u201911, p 745\u2013754. https:\/\/doi.org\/10.1145\/1963405.1963509","DOI":"10.1145\/1963405.1963509"},{"key":"10745_CR25","doi-asserted-by":"publisher","unstructured":"Danial A (2021) cloc: v1.92. https:\/\/doi.org\/10.5281\/zenodo.5760077","DOI":"10.5281\/zenodo.5760077"},{"key":"10745_CR26","doi-asserted-by":"crossref","unstructured":"Das JK, Mondal S, Roy CK (2024) Investigating the utility of chatgpt in the issue tracking system: An exploratory study. arXiv:2402.03735","DOI":"10.1145\/3643991.3645083"},{"key":"10745_CR27","doi-asserted-by":"crossref","unstructured":"Das JK, Mondal S, Roy CK (2025) Why do developers engage with chatgpt in issue-tracker? investigating usage and reliance on chatgpt-generated code","DOI":"10.1109\/SANER64311.2025.00015"},{"key":"10745_CR28","doi-asserted-by":"publisher","unstructured":"Delile Z, Radel S, Godinez J, Engstrom G, Brucker T, Young K, Ghanavati S (2023) Evaluating privacy questions from stack overflow: Can chatgpt compete? In: 2023 IEEE 31st international requirements engineering conference workshops (REW), pp 239\u201324https:\/\/doi.org\/10.1109\/REW57809.2023.00048","DOI":"10.1109\/REW57809.2023.00048"},{"key":"10745_CR29","doi-asserted-by":"publisher","unstructured":"Dhasade AB, Venigalla ASM, Chimalakonda S (2020) Towards prioritizing github issues. In: Proceedings of the 13th innovations in software engineering conference (formerly known as india software engineering conference), association for computing Machinery, New York, NY, USA, ISEC \u201920. https:\/\/doi.org\/10.1145\/3385032.3385052","DOI":"10.1145\/3385032.3385052"},{"key":"10745_CR30","doi-asserted-by":"crossref","unstructured":"Duijn M, Kucera A, Bacchelli A (2015) Quality questions need quality code: Classifying code fragments on stack overflow. In: 2015 IEEE\/ACM 12th working conference on mining software repositories","DOI":"10.1109\/MSR.2015.51"},{"key":"10745_CR31","unstructured":"Eastman D (2023) How conversational programming will democratize computing. https:\/\/thenewstack.io\/how-conversational-programming-will-democratize-computing"},{"key":"10745_CR32","doi-asserted-by":"crossref","unstructured":"Ehsani R, Parra E, Haiduc S, Chatterjee P (2025a) Hierarchical knowledge injection for improving llm-based program repair. In: 2025 IEEE\/ACM 40th international conference on automated software engineering (ASE) arXiv:2506.24015","DOI":"10.1109\/ASE63991.2025.00122"},{"key":"10745_CR33","doi-asserted-by":"publisher","unstructured":"Ehsani R, Pathak S, Chatterjee P (2025b) Towards detecting prompt knowledge gaps for improved llm-guided issue resolution. In: 2025 IEEE\/ACM 22nd international conference on mining software repositories (MSR), pp 699\u2013711. https:\/\/doi.org\/10.1109\/MSR66628.2025.00107","DOI":"10.1109\/MSR66628.2025.00107"},{"key":"10745_CR34","doi-asserted-by":"publisher","unstructured":"Eklund A, Forsman M (2022) Topic modeling by clustering language model embeddings: Human validation on an industry dataset. In: Li Y, Lazaridou A (eds) Proceedings of the 2022 conference on empirical methods in natural language processing: Industry track, association for computational linguistics, Abu Dhabi, UAE, pp 635\u2013643. https:\/\/doi.org\/10.18653\/v1\/2022.emnlp-industry.65","DOI":"10.18653\/v1\/2022.emnlp-industry.65"},{"key":"10745_CR35","doi-asserted-by":"crossref","unstructured":"Ertugrul\u00a0Sagdic MRI Arda\u00a0Bayram (2024) On the taxonomy of developers\u2019 discussion topics with chatgpt","DOI":"10.1145\/3643991.3645080"},{"key":"10745_CR36","doi-asserted-by":"crossref","unstructured":"Fan A, Gokkaya B, Harman M, Lyubarskiy M, Sengupta S, Yoo S, Zhang JM (2023) Large language models for software engineering: Survey and open problems. arXiv:2310.03533","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"10745_CR37","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2025.107778","volume":"185","author":"HL Fran\u00e7a","year":"2025","unstructured":"Fran\u00e7a HL, Goseva-Popstojanova K, Teixeira C, Laranjeiro N (2025) GPTs are not the silver bullet: Performance and challenges of using GPTs for security bug report identification. Inf Softw Technol 185:107778. https:\/\/doi.org\/10.1016\/j.infsof.2025.107778","journal-title":"Inf Softw Technol"},{"key":"10745_CR38","unstructured":"GitHub (2023a) Bug fix example. https:\/\/github.com\/blonestar\/wp-theme-vite-tailwind\/issues\/11"},{"key":"10745_CR39","unstructured":"GitHub (2023b) Code comment example. https:\/\/github.com\/OpenVoiceOS\/ovos-technical-manual\/issues\/4"},{"key":"10745_CR40","unstructured":"GitHub (2023c) Code enhancements example.https:\/\/github.com\/karthink\/gptel\/issues\/76"},{"key":"10745_CR41","unstructured":"GitHub (2023d) Code explanation example. https:\/\/github.com\/LukeberryPi\/phived\/issues\/55"},{"key":"10745_CR42","unstructured":"GitHub (2023e) Code generation example. https:\/\/github.com\/akorb\/SteamShutdown\/issues\/52"},{"key":"10745_CR43","unstructured":"GitHub (2023f) Explicit positive comment example. https:\/\/github.com\/jasonacox\/Powerwall-Dashboard\/issues\/288"},{"key":"10745_CR44","unstructured":"GitHub (2023g) Explicit positive comment example. https:\/\/github.com\/WordPress\/wordpress-playground\/issues\/1096"},{"key":"10745_CR45","unstructured":"GitHub (2023h) Explicit positive comment example. https:\/\/github.com\/hackforla\/website\/issues\/5086"},{"key":"10745_CR46","unstructured":"GitHub (2023i) Failed answers in conversation with chatgpt. https:\/\/github.com\/swc-project\/swc\/issues\/7754"},{"key":"10745_CR47","unstructured":"GitHub (2023j) Generic answers in conversation with chatgpt. https:\/\/github.com\/christofmuc\/KnobKraft-orm\/issues\/73"},{"key":"10745_CR48","unstructured":"GitHub (2023k) Hallucinated answers in conversation with chatgpt. https:\/\/github.com\/chrisjpatty\/flume\/issues\/237"},{"key":"10745_CR49","unstructured":"GitHub (2023l) Hallucinated answers in conversation with chatgpt. https:\/\/github.com\/NCATSTranslator\/Feedback\/issues\/198"},{"key":"10745_CR50","unstructured":"GitHub (2023m) Helpful issue providing a link to chatgpt. https:\/\/github.com\/OpenAdaptAI\/OpenAdapt\/issues\/570"},{"key":"10745_CR51","unstructured":"GitHub (2023n) Information retreival of chatgpt. https:\/\/github.com\/winglang\/wing\/issues\/2869"},{"key":"10745_CR52","unstructured":"GitHub (2023o) Information seeking example. https:\/\/github.com\/meixger\/hackernews-daily\/issues\/251"},{"key":"10745_CR53","unstructured":"GitHub (2023p) Irrelevant answers in conversation with chatgpt. https:\/\/github.com\/dubinc\/dub\/issues\/479"},{"key":"10745_CR54","unstructured":"GitHub (2023q) Knowledge seeking from chatgpt. https:\/\/github.com\/meta-llama\/llama\/issues\/639"},{"key":"10745_CR55","unstructured":"GitHub (2023r) Lsm example in conversation with chatgpt. https:\/\/github.com\/moses-palmer\/pynput\/issues\/588"},{"key":"10745_CR56","unstructured":"GitHub (2023s) Outdated answers in conversation with chatgpt. https:\/\/github.com\/nv-morpheus\/Morpheus\/issues\/1210"},{"key":"10745_CR57","unstructured":"GitHub (2023t) Semantic similarity example in conversation with chatgpt. https:\/\/github.com\/sikalabs\/slu\/issues\/235"},{"key":"10745_CR58","unstructured":"GitHub (2023u) Shift in conversation with chatgpt. https:\/\/github.com\/udayhello\/handwritten1\/issues\/2"},{"key":"10745_CR59","unstructured":"GitHub (2023v) System design example. https:\/\/github.com\/sky-unlimited\/wef_2023\/issues\/26"},{"key":"10745_CR60","unstructured":"GitHub (2023w) Test generation example. https:\/\/github.com\/simonw\/symbex\/issues\/18"},{"key":"10745_CR61","unstructured":"GitHub (2023x) Tool\/library example. https:\/\/github.com\/TryQuiet\/quiet\/issues\/1495"},{"key":"10745_CR62","unstructured":"GitHub (2023y) Toxic language using chatgpt. https:\/\/github.com\/sleuthkit\/sleuthkit\/issues\/2872"},{"key":"10745_CR63","unstructured":"GitHub (2023z) Trust in conversation with chatgpt. https:\/\/github.com\/winglang\/wing\/issues\/2869"},{"key":"10745_CR64","unstructured":"GitHub (2024) Octoverse 2023: The state of open source. https:\/\/octoverse.github.com\/"},{"key":"10745_CR65","unstructured":"GitHubIssue (2023a) Example of positive sentiment conversation. https:\/\/github.com\/darklang\/dark\/issues\/5314"},{"key":"10745_CR66","unstructured":"GitHubIssue (2023b) Unclear answers in conversation with chatgpt. https:\/\/github.com\/webtorrent\/webtorrent\/issues\/2536"},{"key":"10745_CR67","unstructured":"GitHubIssue (2023c) Wrong answers in conversation with chatgpt. https:\/\/github.com\/kreait\/firebase-php\/issues\/802"},{"key":"10745_CR68","doi-asserted-by":"publisher","unstructured":"Han Y, Wang Z, Feng Y, Zhao Z, Wang Y (2024) Characterizing developers\u2019 linguistic behaviors in open source development across their social statuses. Proc ACM Hum-Comput Interact 8(CSCW1). https:\/\/doi.org\/10.1145\/3637306","DOI":"10.1145\/3637306"},{"key":"10745_CR69","doi-asserted-by":"crossref","unstructured":"Hao H, Hasan KA, Qin H, Macedo M, Tian Y, Ding SHH, Hassan AE (2024) An empirical study on developers shared conversations with chatgpt in github pull requests and issues. 2403.10468","DOI":"10.1007\/s10664-024-10540-x"},{"key":"10745_CR70","doi-asserted-by":"publisher","unstructured":"Harper FM, Moy D, Konstan JA (2009) Facts or friends? distinguishing informational and conversational questions in social q&a sites. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI \u201909, p 759\u2013768. https:\/\/doi.org\/10.1145\/1518701.1518819","DOI":"10.1145\/1518701.1518819"},{"key":"10745_CR71","volume-title":"Nonparametric Statistical Methods","author":"M Hollander","year":"2013","unstructured":"Hollander M, Wolfe DA, Chicken E (2013) Nonparametric Statistical Methods, 3rd edn. John Wiley & Sons","edition":"3"},{"key":"10745_CR72","doi-asserted-by":"publisher","unstructured":"Hossain SB, Jiang N, Zhou Q, Li X, Chiang WH, Lyu Y, Nguyen H, Tripp O (2024) A deep dive into large language models for automated bug localization and repair. Proc ACM Softw Eng 1(FSE). https:\/\/doi.org\/10.1145\/3660773","DOI":"10.1145\/3660773"},{"key":"10745_CR73","doi-asserted-by":"crossref","unstructured":"Hou X, Zhao Y, Liu Y, Yang Z, Wang K, Li L, Luo X, Lo D, Grundy J, Wang H (2024) Large language models for software engineering: A systematic literature review. arXiv:1203.81062","DOI":"10.1145\/3695988"},{"key":"10745_CR74","doi-asserted-by":"crossref","unstructured":"Imran MM, Jain Y, Chatterjee P, Damevski K (2022) Data augmentation for improving emotion recognition in software engineering communication. In: Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering","DOI":"10.1145\/3551349.3556925"},{"key":"10745_CR75","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1016\/j.jss.2018.08.030","volume":"145","author":"MR Islam","year":"2018","unstructured":"Islam MR, Zibran MF (2018) Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text. J Syst Softw 145:125\u2013146. https:\/\/doi.org\/10.1016\/j.jss.2018.08.030","journal-title":"J Syst Softw"},{"key":"10745_CR76","unstructured":"Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan KR (2024) Swe-bench: Can language models resolve real-world github issues? In: ICLR, https:\/\/openreview.net\/forum?id=VTF8yNQM66"},{"key":"10745_CR77","doi-asserted-by":"publisher","unstructured":"Jin K, Wang CY, Pham HV, Hemmati H (2024) Can ChatGPT support developers? An empirical evaluation of large language models for code generation. https:\/\/doi.org\/10.1145\/3643991.3645074, arXiv:2402.11702 [cs]","DOI":"10.1145\/3643991.3645074"},{"key":"10745_CR78","doi-asserted-by":"crossref","unstructured":"Kabir S, Udo-Imeh DN, Kou B, Zhang T (2024) Is stack overflow obsolete? An empirical study of the characteristics of chatgpt answers to stack overflow questions. arXiv:2308.02312","DOI":"10.1145\/3613904.3642596"},{"key":"10745_CR79","doi-asserted-by":"publisher","unstructured":"Kavaler D, Sirovica S, Hellendoorn V, Aranovich R, Filkov V (2017) Perceived language complexity in github issue discussions and their effect on issue resolution. In: 2017 32nd IEEE\/ACM international conference on automated software engineering (ASE), pp 72\u20138https:\/\/doi.org\/10.1109\/ASE.2017.8115620","DOI":"10.1109\/ASE.2017.8115620"},{"key":"10745_CR80","unstructured":"Khandelwal T (2025) Using llm-based approaches to enhance and automate topic labeling. arXiv:2502.18469"},{"key":"10745_CR81","unstructured":"Li J, Mynatt ED, Mishra V, Bell J (2023) Always nice and confident, sometimes wrong: Developer\u2019s experiences engaging generative ai chatbots versus human-powered q&a platforms. ArXiv abs\/2309.13684"},{"key":"10745_CR82","doi-asserted-by":"crossref","unstructured":"Liu D, Gong C, Zhang S, Ma Y (2022) The influence of firm\u2019s feedbacks on user-generated content\u2019s linguistic style matching\u2013an explanation based on communication accommodation theory. Frontiers Psychol 13","DOI":"10.3389\/fpsyg.2022.949968"},{"key":"10745_CR83","unstructured":"Liu F, Liu Y, Shi L, Huang H, Wang R, Yang Z, Zhang L, Li Z, Ma Y (2024) Exploring and evaluating hallucinations in llm-powered code generation. arXiv:2404.00971"},{"key":"10745_CR84","unstructured":"Liu J, Tang X, Li L, Chen P, Liu Y (2023) Which is a better programming assistant? a comparative study between chatgpt and stack overflow. arXiv:2308.13851"},{"key":"10745_CR85","first-page":"276","volume":"22","author":"M McHugh","year":"2012","unstructured":"McHugh M (2012) Interrater reliability: The kappa statistic. Biochemia medica \u010dasopis Hrvatskoga dru\u0161tva medicinskih biokemi\u010dara \/ HDMB 22:276\u201382","journal-title":"Biochemia medica \u010dasopis Hrvatskoga dru\u0161tva medicinskih biokemi\u010dara \/ HDMB"},{"key":"10745_CR86","doi-asserted-by":"publisher","unstructured":"Miller JK, Alexander TJ (2025) Human-interpretable clustering of short text using large language models. Royal Soc Open Sci 12(1). https:\/\/doi.org\/10.1098\/rsos.241692","DOI":"10.1098\/rsos.241692"},{"key":"10745_CR87","doi-asserted-by":"crossref","unstructured":"Mohamed S, Parvin A, Parra E (2024) Chatting with ai: Deciphering developer conversations with chatgpt","DOI":"10.1145\/3643991.3645078"},{"key":"10745_CR88","doi-asserted-by":"crossref","unstructured":"Mondal S, Bappon SD, Roy CK (2024) Enhancing user interaction in chatgpt: Characterizing and consolidating multiple prompts for issue resolution. arXiv:2402.04568","DOI":"10.1145\/3643991.3645085"},{"key":"10745_CR89","doi-asserted-by":"crossref","unstructured":"Nachar N (2008) The mann-whitney u: A test for assessing whether two independent samples come from the same distribution. Tutorials in Quantitative Methods for Psychology 4","DOI":"10.20982\/tqmp.04.1.p013"},{"key":"10745_CR90","doi-asserted-by":"crossref","unstructured":"Novielli N, Calefato F, Dongiovanni D, Girardi D, Lanubile F (2020) Can we use se-specific sentiment analysis tools in a cross-platform setting? In: Proceedings of the 17th International Conference on Mining Software Repositories","DOI":"10.1145\/3379597.3387446"},{"key":"10745_CR91","doi-asserted-by":"publisher","unstructured":"Oedingen M, Engelhardt RC, Denz R, Hammer M, Konen W (2024) Chatgpt code detection: Techniques for uncovering the source of code. AI 5(3). https:\/\/doi.org\/10.3390\/ai5030053","DOI":"10.3390\/ai5030053"},{"key":"10745_CR92","unstructured":"OpenAI (2023) Chatgpt. https:\/\/www.openai.com\/"},{"key":"10745_CR93","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1146\/annurev.psych.54.101601.145041","volume":"54","author":"JW Pennebaker","year":"2003","unstructured":"Pennebaker JW, Mehl MR, Niederhoffer KG (2003) Psychological aspects of natural language. Use: Our words, our selves. Ann Rev Psychol 54:547\u2013577","journal-title":"Ann Rev Psychol"},{"key":"10745_CR94","doi-asserted-by":"publisher","unstructured":"Ponzanelli L, Mocci A, Bacchelli A, Lanza M, Fullerton D (2014) Improving low quality stack overflow post detection. In: 2014 IEEE international conference on software maintenance and evolution, pp 541\u2013544. https:\/\/doi.org\/10.1109\/ICSME.2014.90","DOI":"10.1109\/ICSME.2014.90"},{"key":"10745_CR95","unstructured":"Possel H (2024) 50 Popular texting abbreviations & internet acronyms. https:\/\/www.smart-words.org\/abbreviations\/text.html"},{"key":"10745_CR96","doi-asserted-by":"publisher","unstructured":"Qian C, Zhang M, Nie Y, Lu S, Cao H (2023) A survey on bug deduplication and triage methods from multiple points of view. Applied Sciences 13(15). https:\/\/doi.org\/10.3390\/app13158788,","DOI":"10.3390\/app13158788"},{"key":"10745_CR97","unstructured":"Reimers N, Gurevych I (2024) all-minilm-l6-v2. https:\/\/huggingface.co\/sentence-transformers\/all-MiniLM-L6-v2"},{"key":"10745_CR98","unstructured":"Ren Y, Gay G, K\u00e4stner C, Jamshidi P (2020) Understanding the nature of system-related issues in machine learning frameworks: An exploratory study. arXiv:2005.06091"},{"key":"10745_CR99","doi-asserted-by":"publisher","unstructured":"Rigby PC, Hassan AE (2007) What can oss mailing lists tell us? a preliminary psychometric text analysis of the apache developer mailing list. In: Fourth International Workshop on Mining Software Repositories (MSR\u201907:ICSE Workshops 2007), pp 23\u201323. https:\/\/doi.org\/10.1109\/MSR.2007.35","DOI":"10.1109\/MSR.2007.35"},{"key":"10745_CR100","doi-asserted-by":"crossref","unstructured":"R\u00f6der M, Both A, Hinneburg A (2015) Exploring the space of topic coherence measures. In: Proceedings of the 8th ACM international conference on web search and data mining, association for computing machinery, New York, NY, USA, WSDM \u201915, p 399\u2013408","DOI":"10.1145\/2684822.2685324"},{"key":"10745_CR101","doi-asserted-by":"crossref","unstructured":"Ross SI, Martinez F, Houde S, Muller M, Weisz JD (2023) The programmer\u2019s assistant: Conversational interaction with a large language model for software development. In: Proceedings of the 28th international conference on intelligent user interfaces, ACM, IUI \u201923","DOI":"10.1145\/3581641.3584037"},{"key":"10745_CR102","doi-asserted-by":"crossref","unstructured":"Sandler M, Choung H, Ross A, David P (2024) A linguistic comparison between human and chatgpt-generated conversations. arXiv:2401.16587","DOI":"10.1007\/978-981-97-8702-9_25"},{"issue":"1","key":"10745_CR103","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1109\/TSE.2023.3334955","volume":"50","author":"M Sch\u00e4fer","year":"2024","unstructured":"Sch\u00e4fer M, Nadi S, Eghbali A, Tip F (2024) An empirical evaluation of using large language models for automated unit test generation. IEEE Trans Software Eng 50(1):85\u2013105. https:\/\/doi.org\/10.1109\/TSE.2023.3334955","journal-title":"IEEE Trans Software Eng"},{"key":"10745_CR104","unstructured":"Scott B (2023) The SMOG Readability Formula, a Simple Measure of Gobbledygook. https:\/\/readabilityformulas.com\/the-smog-readability-formula\/, Section: Readability Formulas Help"},{"key":"10745_CR105","doi-asserted-by":"publisher","unstructured":"Shuster K, Poff S, Chen M, Kiela D, Weston J (2021) Retrieval augmentation reduces hallucination in conversation. In: Moens MF, Huang X, Specia L, Yih SWt (eds) Findings of the association for computational linguistics: EMNLP 2021, association for computational linguistics, Punta Cana, Dominican Republic, pp 3784\u20133803. https:\/\/doi.org\/10.18653\/v1\/2021.findings-emnlp.320","DOI":"10.18653\/v1\/2021.findings-emnlp.320"},{"key":"10745_CR106","unstructured":"Silva LD, Samhi J, Khomh F (2024) Chatgpt vs llama: Impact, reliability, and challenges in stack overflow discussions. arXiv:2402.08801"},{"key":"10745_CR107","unstructured":"spaCy (2024) spacy $$\\cdot$$ PyPI. https:\/\/pypi.org\/project\/spacy\/"},{"key":"10745_CR108","unstructured":"Stahl PM (2024) pemistahl\/lingua-py. https:\/\/github.com\/pemistahl\/lingua-py"},{"key":"10745_CR109","doi-asserted-by":"crossref","unstructured":"Tan Z, Li D, Wang S, Beigi A, Jiang B, Bhattacharjee A, Karami M, Li J, Cheng L, Liu H (2024) Large language models for data annotation and synthesis: A survey. arXiv:2402.13446","DOI":"10.18653\/v1\/2024.emnlp-main.54"},{"key":"10745_CR110","doi-asserted-by":"crossref","unstructured":"Tang N, Chen M, Ning Z, Bansal A, Huang Y, McMillan C, Li TJJ (2023) An empirical study of developer behaviors for validating and repairing ai-generated code. Plateau Workshop","DOI":"10.1109\/VL\/HCC60511.2024.00015"},{"key":"10745_CR111","unstructured":"Tao W, Zhou Y, Wang Y, Zhang W, Zhang H, Cheng Y (2024) Magis: Llm-based multi-agent framework for github issue resolution. arXiv:2403.17927"},{"key":"10745_CR112","unstructured":"TechTerms (2024) Dictionary of Software Terms. https:\/\/techterms.com\/category\/software"},{"key":"10745_CR113","doi-asserted-by":"publisher","unstructured":"Umargono E, Suseno JE, Gunawan SV (2020) K-means clustering optimization using the elbow method and early centroid determination based on mean and median formula. In: Proceedings of the 2nd international seminar on science and technology (ISSTEC 2019), Atlantis Press, pp 121\u2013129. https:\/\/doi.org\/10.2991\/assehr.k.201010.019","DOI":"10.2991\/assehr.k.201010.019"},{"key":"10745_CR114","doi-asserted-by":"crossref","unstructured":"Vakulenko S, de\u00a0Rijke M, Cochez M, Savenkov V, Polleres A (2018) Measuring semantic coherence of a conversation. arXiv:1806.06411","DOI":"10.1007\/978-3-030-00671-6_37"},{"key":"10745_CR115","doi-asserted-by":"crossref","unstructured":"Wang Q, Saha K, Gregori E, Joyner D, Goel A (2021) Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assistant. In: Proceedings of the 2021 CHI conference on human factors in computing systems, association for computing machinery, New York, NY, USA, CHI \u201921","DOI":"10.1145\/3411764.3445645"},{"key":"10745_CR116","unstructured":"Wu Y, Li Z, Zhang JM, Papadakis M, Harman M, Liu Y (2023) Large language models in fault localisation. arXiv:2308.15276"},{"key":"10745_CR117","doi-asserted-by":"publisher","unstructured":"Xiao T, Treude C, Hata H, Matsumoto K (2024) DevGPT: Studying Developer-ChatGPT Conversations. https:\/\/doi.org\/10.1145\/3643991.3648400","DOI":"10.1145\/3643991.3648400"},{"key":"10745_CR118","doi-asserted-by":"publisher","unstructured":"Xu B, Nguyen TD, Le-Cong T, Hoang T, Liu J, Kim K, Gong C, Niu C, Wang C, Le B, Lo D (2023) Are we ready to embrace generative ai for software q&a? In: 2023 38th IEEE\/ACM international conference on automated software engineering (ASE), pp 1713\u20131717. https:\/\/doi.org\/10.1109\/ASE56229.2023.00023","DOI":"10.1109\/ASE56229.2023.00023"},{"key":"10745_CR119","doi-asserted-by":"crossref","unstructured":"Yang J, Jin H, Tang R, Han X, Feng Q, Jiang H, Zhong S, Yin B, Hu X (2024) Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Trans Knowl Discov Data 18(6)","DOI":"10.1145\/3649506"},{"key":"10745_CR120","doi-asserted-by":"publisher","unstructured":"Zhang T, Xu B, Thung F, Haryono SA, Lo D, Jiang L (2020) Sentiment analysis for software engineering: How far can pre-trained transformer models go? In: 2020 IEEE international conference on software maintenance and evolution (ICSME), pp 70\u201380. https:\/\/doi.org\/10.1109\/ICSME46990.2020.00017","DOI":"10.1109\/ICSME46990.2020.00017"},{"key":"10745_CR121","doi-asserted-by":"crossref","unstructured":"Zhang Y, Ruan H, Fan Z, Roychoudhury A (2024) Autocoderover: Autonomous program improvement. arXiv:2404.05427","DOI":"10.1145\/3650212.3680384"},{"key":"10745_CR122","doi-asserted-by":"publisher","unstructured":"Zhou J, Zhang Y, Luo Q, Parker AG, De\u00a0Choudhury M (2023) Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions. In: Proceedings of the 2023 CHI conference on human factors in computing systems, association for computing machinery, New York, NY, USA, CHI \u201923. https:\/\/doi.org\/10.1145\/3544548.3581318,","DOI":"10.1145\/3544548.3581318"},{"key":"10745_CR123","doi-asserted-by":"crossref","unstructured":"Zhou X, Liang P, Zhang B, Li Z, Ahmad A, Shahin M, Waseem M (2024) Exploring the problems, their causes and solutions of ai pair programming: A study with practitioners of github copilot. arXiv:2311.01020","DOI":"10.1016\/j.jss.2024.112204"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10745-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-025-10745-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10745-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T11:26:52Z","timestamp":1770809212000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-025-10745-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,18]]},"references-count":123,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1]]}},"alternative-id":["10745"],"URL":"https:\/\/doi.org\/10.1007\/s10664-025-10745-8","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,18]]},"assertion":[{"value":"28 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 September 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 November 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Declaration"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}],"article-number":"22"}}