{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T11:14:37Z","timestamp":1778152477375,"version":"3.51.4"},"reference-count":113,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T00:00:00Z","timestamp":1778112000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T00:00:00Z","timestamp":1778112000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"NEVADA NASA EPSCOR","award":["AWD2536\/GR20392"],"award-info":[{"award-number":["AWD2536\/GR20392"]}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2519136"],"award-info":[{"award-number":["2519136"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    The rapid adoption of large language models (LLMs) like ChatGPT has introduced new dynamics in software development, particularly within pull request workflows. While prior research has examined the quality of AI-generated code, less is known about how developers evaluate, adapt, and integrate these suggestions in real-world collaboration. We analyze 338 pull requests from 255 GitHub repositories containing self-admitted ChatGPT usage, comprising 645 AI-generated snippets and 3,486 developer-authored patches. To support this analysis at scale, we use PatchTrack, an automated classifier that identifies whether AI-generated patches were applied, partially reused, or not integrated. Our findings reveal that full adoption of ChatGPT-generated code is uncommon: the median integration rate is 25%. Qualitative analysis of 89 pull requests with integrated patches reveals recurring patterns of\n                    <jats:italic>structural integration<\/jats:italic>\n                    ,\n                    <jats:italic>selective extraction<\/jats:italic>\n                    , and\n                    <jats:italic>iterative refinement<\/jats:italic>\n                    , indicating that developers typically treat AI output as a starting point rather than a final implementation. Even when code is not directly adopted, ChatGPT influences workflows through\n                    <jats:italic>conceptual guidance<\/jats:italic>\n                    ,\n                    <jats:italic>documentation<\/jats:italic>\n                    , and\n                    <jats:italic>debugging strategies<\/jats:italic>\n                    . Integration decisions reflect\n                    <jats:italic>contextual fit<\/jats:italic>\n                    ,\n                    <jats:italic>integration effort<\/jats:italic>\n                    ,\n                    <jats:italic>maintainer trust<\/jats:italic>\n                    , and established pull request review norms rather than serving as direct indicators of code correctness. Overall, this study provides empirical insight into AI-mediated decision-making in collaborative software development, showing that the influence of generative AI extends beyond patch generation to how developers reason about, adapt, and negotiate code during review within pull request workflows. These findings inform the design of AI-assisted tools and support more transparent and effective use of LLMs in practice.\n                  <\/jats:p>","DOI":"10.1007\/s10664-026-10869-5","type":"journal-article","created":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T10:41:02Z","timestamp":1778150462000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["PatchTrack: A comprehensive analysis of ChatGPT\u2019s influence on pull request outcomes"],"prefix":"10.1007","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0133-8164","authenticated-orcid":false,"given":"Daniel","family":"Ogenrwot","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3206-7085","authenticated-orcid":false,"given":"John","family":"Businge","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,5,7]]},"reference":[{"key":"10869_CR1","doi-asserted-by":"publisher","unstructured":"Abedu S, Abdellatif A, Shihab E (2024) Llm-based chatbots for mining software repositories: Challenges and opportunities. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. EASE \u201924, pp 201\u2013210. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3661167.3661218","DOI":"10.1145\/3661167.3661218"},{"key":"10869_CR2","doi-asserted-by":"publisher","unstructured":"Azeem MI, Panichella S, Di\u00a0Sorbo A, Serebrenik A, Wang Q (2020) Action-based recommendation in pull-request development. In: Proceedings of the International Conference on Software and System Processes. ICSSP \u201920, pp 115\u2013124. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3379177.3388904","DOI":"10.1145\/3379177.3388904"},{"key":"10869_CR3","doi-asserted-by":"publisher","unstructured":"Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: 2013 35th International Conference on Software Engineering (ICSE), pp 712\u2013721. https:\/\/doi.org\/10.1109\/ICSE.2013.6606617","DOI":"10.1109\/ICSE.2013.6606617"},{"key":"10869_CR4","doi-asserted-by":"crossref","unstructured":"Baker BS (1995) On finding duplication and near-duplication in large software systems. In: Proceedings of 2nd working conference on reverse engineering, pp 86\u201395. IEEE","DOI":"10.1109\/WCRE.1995.514697"},{"key":"10869_CR5","doi-asserted-by":"crossref","unstructured":"Baxter ID, Yahin A, Moura L, Sant\u2019Anna M, Bier L (1998) Clone detection using abstract syntax trees. In: Proceedings international conference on software maintenance (Cat. No. 98CB36272), pp 368\u2013377. IEEE","DOI":"10.1109\/ICSM.1998.738528"},{"issue":"7","key":"10869_CR6","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1145\/362686.362692","volume":"13","author":"BH Bloom","year":"1970","unstructured":"Bloom BH (1970) Space\/time trade-offs in hash coding with allowable errors. Commun ACM 13(7):422\u2013426. https:\/\/doi.org\/10.1145\/362686.362692","journal-title":"Commun ACM"},{"key":"10869_CR7","doi-asserted-by":"publisher","unstructured":"Bowman B, Huang HH (2020) Vgraph: A robust vulnerable code clone detection system using code property triplets. In: 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pp 53\u201369. https:\/\/doi.org\/10.1109\/EuroSP48549.2020.00012","DOI":"10.1109\/EuroSP48549.2020.00012"},{"key":"10869_CR8","doi-asserted-by":"publisher","unstructured":"Businge J, Kawuma S, Openja M, Bainomugisha E, Serebrenik A (2019) How stable are eclipse application framework internal interfaces? In: 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp 117\u2013127. https:\/\/doi.org\/10.1109\/SANER.2019.8668018","DOI":"10.1109\/SANER.2019.8668018"},{"key":"10869_CR9","doi-asserted-by":"publisher","unstructured":"Businge J, Decan A, Zerouali A, Mens T, Demeyer S, De\u00a0Roover C (2022) Variant forks \u2013 motivations and impediments. In: Proceedings of the 29th edition of the IEEE international conference on software analysis, evolution and reengineering, pp 867\u2013877. IEEE Computer Society. https:\/\/doi.org\/10.1109\/SANER53432.2022.00105","DOI":"10.1109\/SANER53432.2022.00105"},{"key":"10869_CR10","doi-asserted-by":"publisher","unstructured":"Cassee N, Vasilescu B, Serebrenik A (200) The Silent Helper: The Impact of Continuous Integration on Code Reviews . In: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp 423\u2013434. IEEE Computer Society, Los Alamitos, CA, USA. https:\/\/doi.org\/10.1109\/SANER48275.2020.9054818, https:\/\/doi.ieeecomputersociety.org\/10.1109\/SANER48275.2020.9054818","DOI":"10.1109\/SANER48275.2020.9054818"},{"key":"10869_CR11","doi-asserted-by":"publisher","unstructured":"Champa AI, Rabbi MF, Nachuma C, Zibran MF (2024) Chatgpt in action: Analyzing its use in software development. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR \u201924, pp 182\u2013186. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3643991.3645077","DOI":"10.1145\/3643991.3645077"},{"issue":"5","key":"10869_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3652155","volume":"33","author":"D Chen","year":"2024","unstructured":"Chen D, Liu Y, Zhou M, Zhao Y, Wang S, Wang X, Chen X, Bissyand\u00e9 T, Klein J (2024) Llm for mobile: An initial roadmap. ACM Trans Softw Eng Methodol 33(5):1\u201344","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10869_CR13","doi-asserted-by":"publisher","unstructured":"Chouchen M, Bessghaier N, Begoug M, Ouni A, Alomar E, Mkaouer MW (2024) How do software developers use chatgpt? an exploratory study on github pull requests. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR \u201924, pp 212\u2013216. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3643991.3645084","DOI":"10.1145\/3643991.3645084"},{"key":"10869_CR14","unstructured":"Cormen TH, Leiserson CE, Rivest RL, Stein C (2009) Introduction to Algorithms, 3rd edn. The MIT Press"},{"key":"10869_CR15","doi-asserted-by":"publisher","unstructured":"Deng Y, Xia CS, Yang C, Zhang SD, Yang S, Zhang L (2024) Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries. In: Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. ICSE \u201924, pp 1\u201313. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3597503.3623343","DOI":"10.1145\/3597503.3623343"},{"key":"10869_CR16","doi-asserted-by":"publisher","unstructured":"Dey T, Mockus A (2020) Effect of technical and social factors on pull request quality for the npm ecosystem. In: Proceedings of the 14th ACM \/ IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). ESEM \u201920. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3382494.3410685","DOI":"10.1145\/3382494.3410685"},{"issue":"4","key":"10869_CR17","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1109\/MS.2023.3265877","volume":"40","author":"C Ebert","year":"2023","unstructured":"Ebert C, Louridas P (2023) Generative ai for software practitioners. IEEE Softw 40(4):30\u201338. https:\/\/doi.org\/10.1109\/MS.2023.3265877","journal-title":"IEEE Softw"},{"key":"10869_CR18","doi-asserted-by":"crossref","unstructured":"Ehsani R, Pathak S, Chatterjee P (2025) Towards detecting prompt knowledge gaps for improved llm-guided issue resolution. In: Proceedings of the 22nd International Conference on Mining Software Repositories (MSR 2025). ACM, Ottawa, Canada. To appear","DOI":"10.1109\/MSR66628.2025.00107"},{"key":"10869_CR19","doi-asserted-by":"publisher","unstructured":"Feng S, Suo W, Wu Y, Zou D, Liu Y, Jin H (2024) Machine learning is all you need: A simple token-based approach for effective code clone detection. In: Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. ICSE \u201924, pp 1\u201313. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3597503.3639114","DOI":"10.1145\/3597503.3639114"},{"key":"10869_CR20","doi-asserted-by":"publisher","unstructured":"Ford D, Behroozi M, Serebrenik A, Parnin C (2019) Beyond the code itself: how programmers really look at pull requests. In: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Society. ICSE-SEIS \u201919, pp 51\u201360. IEEE Press. https:\/\/doi.org\/10.1109\/ICSE-SEIS.2019.17","DOI":"10.1109\/ICSE-SEIS.2019.17"},{"issue":"1","key":"10869_CR21","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1186\/1471-2288-13-117","volume":"13","author":"NK Gale","year":"2013","unstructured":"Gale NK, Heath G, Cameron E, Rashid S, Redwood S (2013) Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 13(1):117. https:\/\/doi.org\/10.1186\/1471-2288-13-117","journal-title":"BMC Med Res Methodol"},{"key":"10869_CR22","doi-asserted-by":"publisher","unstructured":"Ghorbani A, Cassee N, Robinson D, Alami A, Ernst NA, Serebrenik A, Wasowski A (2023) Autonomy is an acquired taste: Exploring developer preferences for github bots. In: 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE), pp 1405\u20131417. https:\/\/doi.org\/10.1109\/ICSE48619.2023.00123","DOI":"10.1109\/ICSE48619.2023.00123"},{"key":"10869_CR23","unstructured":"GitHub pull request (2023). https:\/\/github.com\/darklang\/dark\/pull\/5068"},{"key":"10869_CR24","unstructured":"GitHub pull request (2023a). https:\/\/chatgpt.com\/share\/8cb16814-2855-4fbd-87e5-bde8ba349728"},{"key":"10869_CR25","unstructured":"GitHub pull request (2023b). https:\/\/github.com\/faker-js\/faker\/pull\/2405"},{"key":"10869_CR26","unstructured":"GitHub pull request (2023c). https:\/\/github.com\/laravel-json-api\/core\/pull\/12"},{"key":"10869_CR27","unstructured":"GitHub pull request (2023d). https:\/\/github.com\/ory\/elements\/pull\/171"},{"key":"10869_CR28","unstructured":"GitHub pull request (2023e). https:\/\/github.com\/darklang\/dark\/pull\/5063"},{"key":"10869_CR29","unstructured":"GitHub pull request (2023f). https:\/\/github.com\/sveltejs\/learn.svelte.dev\/pull\/522"},{"key":"10869_CR30","unstructured":"GitHub pull request (2023g). https:\/\/github.com\/darklang\/dark\/pull\/5058"},{"key":"10869_CR31","unstructured":"GitHub pull request (2023h). https:\/\/github.com\/Bananapus\/nana-core\/pull\/37"},{"key":"10869_CR32","unstructured":"GitHub pull request (2023i). https:\/\/github.com\/codecrafters-io\/frontend\/pull\/1061"},{"key":"10869_CR33","unstructured":"GitHub pull request (2023j). https:\/\/github.com\/faker-js\/faker\/pull\/2230"},{"key":"10869_CR34","unstructured":"GitHub pull request (2023k). https:\/\/github.com\/digitalbitbox\/bitbox-wallet-app\/pull\/2415"},{"key":"10869_CR35","unstructured":"GitHub pull request (2024a). https:\/\/github.com\/pokt-network\/poktroll\/pull\/185"},{"key":"10869_CR36","unstructured":"GitHub pull request (2024b). https:\/\/github.com\/Mudlet\/Mudlet\/pull\/7123"},{"key":"10869_CR37","unstructured":"GitHub pull request (2024c). https:\/\/github.com\/nylas\/nylas-python\/pull\/279"},{"key":"10869_CR38","unstructured":"GitHub pull request (2024d). https:\/\/github.com\/alshedivat\/al-folio\/pull\/2059"},{"key":"10869_CR39","unstructured":"GitHub pull request (2024e). https:\/\/github.com\/gemini-hlsw\/scheduler\/pull\/428"},{"key":"10869_CR40","unstructured":"GitHub pull request (2024f). https:\/\/github.com\/theosanderson\/taxonium\/pull\/534"},{"key":"10869_CR41","unstructured":"GitHub pull request (2024g). https:\/\/github.com\/open-learning-exchange\/myplanet\/pull\/2214"},{"key":"10869_CR42","unstructured":"GitHub pull request (2024h). https:\/\/github.com\/open-learning-exchange\/myplanet\/pull\/2212"},{"key":"10869_CR43","unstructured":"GitHub pull request (2024i). https:\/\/github.com\/labdao\/plex\/pull\/468"},{"key":"10869_CR44","unstructured":"GitHub pull request (2024j). https:\/\/github.com\/plausible\/analytics\/pull\/3792"},{"key":"10869_CR45","unstructured":"GitHub pull request (2024k). https:\/\/github.com\/open-learning-exchange\/myplanet\/pull\/2213"},{"key":"10869_CR46","unstructured":"GitHub: GitHub REST API Documentation (2025) GitHub. https:\/\/docs.github.com\/en\/rest?apiVersion=2022-11-28"},{"key":"10869_CR47","unstructured":"GitHub: Online Appendix (2025) GitHub. https:\/\/www.gnu.org\/software\/diffutils\/manual\/html_node\/Hunks.html"},{"key":"10869_CR48","unstructured":"GitHub: The State of Open Source: Octoverse 2024 (2024). https:\/\/github.blog\/news-insights\/octoverse\/octoverse-2024\/"},{"key":"10869_CR49","unstructured":"Golzadeh M, Decan A, Mens T (2019) On the effect of discussions on pull request decisions. In: BENEVOL"},{"key":"10869_CR50","doi-asserted-by":"publisher","unstructured":"Gousios G, Pinzger M, Deursen Av (2014) An exploratory study of the pull-based software development model. In: Proceedings of the 36th International Conference on Software Engineering. ICSE 2014, pp 345\u2013355. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2568225.2568260","DOI":"10.1145\/2568225.2568260"},{"key":"10869_CR51","doi-asserted-by":"publisher","unstructured":"Gousios G, Storey M-A, Bacchelli A (2016) Work practices and challenges in pull-based development: the contributor\u2019s perspective. In: Proceedings of the 38th International Conference on Software Engineering. ICSE \u201916, pp 285\u2013296. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2884781.2884826","DOI":"10.1145\/2884781.2884826"},{"key":"10869_CR52","doi-asserted-by":"publisher","unstructured":"Grewal B, Lu W, Nadi S, Bezemer C-P (2024) Analyzing developer use of chatgpt generated code in open source github projects. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR \u201924, pp 157\u2013161. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3643991.3645072","DOI":"10.1145\/3643991.3645072"},{"key":"10869_CR53","doi-asserted-by":"publisher","unstructured":"Guo Q, Cao J, Xie X, Liu S, Li X, Chen B, Peng X (2024) Exploring the potential of chatgpt in automated code refinement: An empirical study. In: Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. ICSE \u201924, pp 1\u201313. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3597503.3623306","DOI":"10.1145\/3597503.3623306"},{"key":"10869_CR54","doi-asserted-by":"crossref","unstructured":"Hao H, Hasan KA, Qin H, Macedo M, Tian Y, Ding SHH, Hassan AE (2024) An empirical study on developers\u2019 shared conversations with chatgpt in github pull requests and issues. arXiv preprint arXiv:2403.10468. License: CC BY-NC-SA 4.0","DOI":"10.1007\/s10664-024-10540-x"},{"key":"10869_CR55","doi-asserted-by":"crossref","unstructured":"Hassan AE, Lin D, Rajbahadur GK, Gallaba K, Cogo FR, Chen B, Zhang H, Thangarajah K, Oliva GA, Lin J, Abdullah WM, Jiang ZM (2024) Rethinking software engineering in the foundation model era: A curated catalogue of challenges in the development of trustworthy fmware. arXiv preprint arXiv:2402.15943","DOI":"10.1145\/3663529.3663849"},{"key":"10869_CR56","unstructured":"Hou X, Zhao Y, Liu Y, Yang Z, Wang K, Li L, Luo X, Lo D, Grundy J, Wang H (2023) Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620"},{"key":"10869_CR57","doi-asserted-by":"crossref","unstructured":"Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 328\u2013339. Association for Computational Linguistics, Melbourne, Australia. https:\/\/doi.org\/10.18653\/v1\/P18-1031, https:\/\/aclanthology.org\/P18-1031","DOI":"10.18653\/v1\/P18-1031"},{"key":"10869_CR58","doi-asserted-by":"publisher","unstructured":"Huang Y, Chen Y, Chen X, Chen J, Peng R, Tang Z, Huang J, Xu F, Zheng Z (2024) Generative software engineering. arXiv preprint arXiv:2403.02583, https:\/\/doi.org\/10.48550\/arXiv.2403.02583. Submitted on 5 Mar 2024, last revised 3 Apr 2024 (this version, v2)","DOI":"10.48550\/arXiv.2403.02583"},{"key":"10869_CR59","doi-asserted-by":"publisher","unstructured":"Jang J, Agrawal A, Brumley D (2012) Redebug: Finding unpatched code clones in entire os distributions. In: 2012 IEEE symposium on security and privacy, pp 48\u201362. https:\/\/doi.org\/10.1109\/SP.2012.13","DOI":"10.1109\/SP.2012.13"},{"key":"10869_CR60","doi-asserted-by":"publisher","unstructured":"Jiang L, Misherghi G, Su Z, Glondu S (2007) Deckard: Scalable and accurate tree-based detection of code clones. In: 29th International Conference on Software Engineering (ICSE\u201907), pp 96\u2013105. https:\/\/doi.org\/10.1109\/ICSE.2007.30","DOI":"10.1109\/ICSE.2007.30"},{"key":"10869_CR61","doi-asserted-by":"publisher","unstructured":"Jiang N, Liu K, Lutellier T, Tan L (2023) Impact of code language models on automated program repair. In: Proceedings of the 45th International Conference on Software Engineering. ICSE \u201923, pp 1430\u20131442. IEEE Press. https:\/\/doi.org\/10.1109\/ICSE48619.2023.00125","DOI":"10.1109\/ICSE48619.2023.00125"},{"key":"10869_CR62","doi-asserted-by":"publisher","unstructured":"Jin K, Wang C-Y, Pham HV, Hemmati H (2024) Can chatgpt support developers? an empirical evaluation of large language models for code generation. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR \u201924, pp 167\u2013171. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3643991.3645074","DOI":"10.1145\/3643991.3645074"},{"key":"10869_CR63","doi-asserted-by":"crossref","unstructured":"Ju J, Yu L, Li X, Yang L, Zuo C (2023) Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In: 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), pp 647\u2013658. IEEE","DOI":"10.1109\/ISSRE59848.2023.00026"},{"issue":"7","key":"10869_CR64","doi-asserted-by":"publisher","first-page":"654","DOI":"10.1109\/TSE.2002.1019480","volume":"28","author":"T Kamiya","year":"2002","unstructured":"Kamiya T, Kusumoto S, Inoue K (2002) Ccfinder: A multilinguistic token-based code clone detection system for large scale source code. IEEE Trans Software Eng 28(7):654\u2013670","journal-title":"IEEE Trans Software Eng"},{"key":"10869_CR65","doi-asserted-by":"publisher","unstructured":"Kawuma S, Businge J, Bainomugisha E (2016) Can we find stable alternatives for unstable eclipse interfaces? In: 2016 IEEE 24th International Conference on Program Comprehension (ICPC), pp 1\u201310. https:\/\/doi.org\/10.1109\/ICPC.2016.7503716","DOI":"10.1109\/ICPC.2016.7503716"},{"key":"10869_CR66","doi-asserted-by":"publisher","unstructured":"Khatoonabadi S, Costa DE, Mujahid S, Shihab E (2023) Understanding the helpfulness of stale bot for pull-based development: An empirical study of 20 large open-source projects. ACM Trans Softw Eng Methodol 33(2). https:\/\/doi.org\/10.1145\/3624739","DOI":"10.1145\/3624739"},{"key":"10869_CR67","doi-asserted-by":"publisher","unstructured":"Kim S, Woo S, Lee H, Oh H (2017a) Vuddy: A scalable approach for vulnerable code clone discovery. In: 2017 IEEE Symposium on Security and Privacy (SP), pp 595\u2013614. https:\/\/doi.org\/10.1109\/SP.2017.62","DOI":"10.1109\/SP.2017.62"},{"key":"10869_CR68","doi-asserted-by":"publisher","unstructured":"Kim S, Woo S, Lee H, Oh H (2017b) Vuddy: A scalable approach for vulnerable code clone discovery. In: 2017 IEEE Symposium on Security and Privacy (SP), pp 595\u2013614. https:\/\/doi.org\/10.1109\/SP.2017.62","DOI":"10.1109\/SP.2017.62"},{"key":"10869_CR69","doi-asserted-by":"publisher","unstructured":"Latendresse J, Khatoonabadi S, Abdellatif A, Shihab E (2024) Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations. https:\/\/doi.org\/10.48550\/arXiv.2408.05128","DOI":"10.48550\/arXiv.2408.05128"},{"key":"10869_CR70","unstructured":"Legay D, Decan A, Mens T (2018) On the impact of pull request decisions on future contributions. arXiv preprint arXiv:1812.06269"},{"key":"10869_CR71","doi-asserted-by":"publisher","unstructured":"Li S, Cheng Y, Chen J, Xuan J, He S, Shang W (2024) Assessing the performance of ai-generated code: A case study on github copilot. In: 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE), pp 216\u2013227. https:\/\/doi.org\/10.1109\/ISSRE62328.2024.00030","DOI":"10.1109\/ISSRE62328.2024.00030"},{"key":"10869_CR72","doi-asserted-by":"publisher","unstructured":"Liang JT, Yang C, Myers BA (2024) A large-scale survey on the usability of ai programming assistants: Successes and challenges. In: Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. ICSE \u201924, pp 1\u201313. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3597503.3608128","DOI":"10.1145\/3597503.3608128"},{"key":"10869_CR73","unstructured":"Lincoln YS, Guba EG (1985) Naturalistic Inquiry, pp 393\u2013408. Sage Publications, Beverly Hills, Calif. https:\/\/archive.org\/details\/naturalisticinqu00linc"},{"key":"10869_CR74","doi-asserted-by":"crossref","unstructured":"Luo L, Ning J, Zhao Y, Wang Z, Ding Z, Chen P, Fu W, Han Q, Xu G, Qiu Y et al (2024) Taiyi: a bilingual fine-tuned large language model for diverse biomedical tasks. J Am Med Inf Assoc, 037","DOI":"10.1093\/jamia\/ocae037"},{"issue":"4","key":"10869_CR75","doi-asserted-by":"publisher","first-page":"459","DOI":"10.1007\/s10515-020-00280-2","volume":"27","author":"T Menzies","year":"2020","unstructured":"Menzies T, Pecheur C (2020) Software engineering with ai\/ml: State of the art and future prospects. Autom Softw Eng 27(4):459\u2013489. https:\/\/doi.org\/10.1007\/s10515-020-00280-2","journal-title":"Autom Softw Eng"},{"key":"10869_CR76","doi-asserted-by":"publisher","unstructured":"Min H, Li\u00a0Ping Z (2019) Survey on software clone detection research. In: Proceedings of the 2019 3rd International Conference on Management Engineering, Software Engineering and Service Sciences. ICMSS 2019, pp 9\u201316. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3312662.3312707","DOI":"10.1145\/3312662.3312707"},{"key":"10869_CR77","doi-asserted-by":"crossref","unstructured":"Moumoula M, Kabore A, Klein J, Bissyand\u00e9 T (2024) Cross-lingual code clone detection: When llms fall short against embedding-based classifier. In: Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","DOI":"10.1145\/3691620.3695335"},{"key":"10869_CR78","doi-asserted-by":"crossref","unstructured":"Nashid N, Ding D, Gallaba K, Hassan AE, Mesbah A (2025) Characterizing multi-hunk patches: Divergence, proximity, and llm repair challenges. arXiv preprint arXiv:2506.04418","DOI":"10.1109\/ASE63991.2025.00137"},{"issue":"3","key":"10869_CR79","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1007\/s10664-021-10052-y","volume":"27","author":"L Ochoa","year":"2022","unstructured":"Ochoa L, Degueule T, Falleri J-R, Vinju J (2022) Breaking bad? semantic versioning and impact of breaking changes in maven central: An external and differentiated replication study. Empir Softw Eng 27(3):61","journal-title":"Empir Softw Eng"},{"key":"10869_CR80","doi-asserted-by":"publisher","first-page":"100732","DOI":"10.1016\/j.cosrev.2025.100732","volume":"57","author":"L Ochoa","year":"2025","unstructured":"Ochoa L, Hammad M, Giray G, Babur O, Bennin K (2025) Characterising harmful api uses and repair techniques: Insights from a systematic review. Comput Sci Rev 57:100732","journal-title":"Comput Sci Rev"},{"key":"10869_CR81","doi-asserted-by":"publisher","unstructured":"Ogenrwot D, Businge J (2025) Replication Package for PatchTrack: A Comprehensive Analysis of ChatGPT\u2019s Influence on Pull Request Outcomes. https:\/\/doi.org\/10.5281\/zenodo.14978624","DOI":"10.5281\/zenodo.14978624"},{"key":"10869_CR82","unstructured":"OpenAI (2025) Terms of Use. OpenAI. https:\/\/openai.com\/policies\/terms-of-use"},{"key":"10869_CR83","doi-asserted-by":"crossref","unstructured":"Ou\u00e9draogo W, Kabore K, Tian H, Song Y, Koyuncu A, Klein J, Lo D, Bissyand\u00e9 T (2024) Llms and prompting for unit test generation: A large-scale evaluation. In: Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","DOI":"10.1145\/3691620.3695330"},{"key":"10869_CR84","doi-asserted-by":"publisher","unstructured":"Ozpolat Z, Yildirim Karabatak M (2023) Artificial intelligence-based tools in software development processes: Application of chatgpt. Eur J Technic. https:\/\/doi.org\/10.36222\/ejt.1330631","DOI":"10.36222\/ejt.1330631"},{"key":"10869_CR85","doi-asserted-by":"publisher","unstructured":"Peng S, Kalliamvakou E, Cihon P, Demirer M (2023) The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590, https:\/\/doi.org\/10.48550\/arXiv.2302.06590 . Submitted on 13 Feb 2023","DOI":"10.48550\/arXiv.2302.06590"},{"key":"10869_CR86","doi-asserted-by":"publisher","unstructured":"Ramkisoen PK, Businge J, Bladel B, Decan A, Demeyer S, De\u00a0Roover C, Khomh F (2022) Pareco: patched clones and missed patches among the divergent variants of a software family. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ESEC\/FSE 2022, pp 646\u2013658. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3540250.3549112","DOI":"10.1145\/3540250.3549112"},{"key":"10869_CR87","volume-title":"The Programmer\u2019s Apprentice","author":"C Rich","year":"1990","unstructured":"Rich C, Waters RC (1990) The Programmer\u2019s Apprentice. ACM Press\/Addison-Wesley, Reading, MA"},{"key":"10869_CR88","doi-asserted-by":"publisher","unstructured":"Rigby PC, Bird C (2013) Convergent contemporary software peer review practices. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ESEC\/FSE 2013, pp 202\u2013212. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2491411.2491444","DOI":"10.1145\/2491411.2491444"},{"key":"10869_CR89","doi-asserted-by":"publisher","unstructured":"Russo D (2024) Navigating the complexity of generative ai adoption in software engineering. ACM Trans Softw Eng Methodol. https:\/\/doi.org\/10.1145\/3652154. Just Accepted","DOI":"10.1145\/3652154"},{"key":"10869_CR90","doi-asserted-by":"publisher","unstructured":"Sajnani H, Saini V, Svajlenko J, Roy CK, Lopes CV (2016) Sourcerercc: scaling code clone detection to big-code. In: Proceedings of the 38th International Conference on Software Engineering. ICSE \u201916, pp 1157\u20131168. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2884781.2884877","DOI":"10.1145\/2884781.2884877"},{"key":"10869_CR91","doi-asserted-by":"publisher","unstructured":"Sauvola J, Tarkoma S, Klemettinen M, Riekki J, Doermann D (2024) Future of software development with generative ai. Automated Softw Eng 31(26). https:\/\/doi.org\/10.1007\/s10515-024-00330-1","DOI":"10.1007\/s10515-024-00330-1"},{"key":"10869_CR92","doi-asserted-by":"crossref","unstructured":"Siddiq ML, Santos J, Tanvir RH, Ulfat N, Rifat FA, Lopes VC (2023) Exploring the effectiveness of large language models in generating unit tests. arXiv preprint arXiv:2305.00418","DOI":"10.1145\/3661167.3661216"},{"key":"10869_CR93","doi-asserted-by":"publisher","unstructured":"Siddiq ML, Roney L, Zhang J, Santos JCDS (2024) Quality assessment of chatgpt generated code and their use by developers. In: Proceedings of the 21st International Conference on Mining Software Repositories. MSR \u201924, pp 152\u2013156. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3643991.3645071","DOI":"10.1145\/3643991.3645071"},{"key":"10869_CR94","doi-asserted-by":"publisher","unstructured":"Soares DM, Lima\u00a0J\u00fanior ML, Murta L, Plastino A (2015) Acceptance factors of pull requests in open-source projects. In: Proceedings of the 30th Annual ACM Symposium on Applied Computing. SAC \u201915, pp 1541\u20131546. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2695664.2695856","DOI":"10.1145\/2695664.2695856"},{"key":"10869_CR95","volume-title":"Card Sorting: Designing Usable Categories","author":"D Spencer","year":"2009","unstructured":"Spencer D (2009) Card Sorting: Designing Usable Categories. Rosenfeld Media, New York"},{"issue":"2","key":"10869_CR96","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1109\/TSE.2016.2584053","volume":"43","author":"M-A Storey","year":"2017","unstructured":"Storey M-A, Zagalsky A, Filho FF, Singer L, German DM (2017) How social and communication channels shape and challenge a participatory culture in software development. IEEE Trans Softw Eng 43(2):185\u2013204. https:\/\/doi.org\/10.1109\/TSE.2016.2584053","journal-title":"IEEE Trans Softw Eng"},{"key":"10869_CR97","doi-asserted-by":"publisher","unstructured":"Tanzil MH, Khan JY, Uddin G (2024) Chatgpt incorrectness detection in software reviews. In: Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. ICSE \u201924. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3597503.3639194","DOI":"10.1145\/3597503.3639194"},{"key":"10869_CR98","doi-asserted-by":"publisher","unstructured":"Tsay J, Dabbish L, Herbsleb J (2014) Influence of social and technical factors for evaluating contribution in github. In: Proceedings of the 36th International Conference on Software Engineering. ICSE 2014, pp 356\u2013366. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2568225.2568315","DOI":"10.1145\/2568225.2568315"},{"key":"10869_CR99","doi-asserted-by":"crossref","unstructured":"Tufano R, Mastropaolo A, Pepe F, Dabi\u0107 O, Di\u00a0Penta M, Bavota G (2024a) Unveiling chatgpt\u2019s usage in open source projects: A mining-based study. arXiv preprint arXiv:2402.13456. Paper accepted for publication at 21st International Conference on Mining Software Repositories (MASR\u201924)","DOI":"10.1145\/3643991.3644918"},{"key":"10869_CR100","doi-asserted-by":"publisher","unstructured":"Tufano R, Dabi\u0107 O, Mastropaolo A, Ciniselli M, Bavota G (2024b) Code review automation: Strengths and weaknesses of the state of the art. IEEE Trans Softw Eng 50(2):338\u2013353. https:\/\/doi.org\/10.1109\/TSE.2023.3348172","DOI":"10.1109\/TSE.2023.3348172"},{"key":"10869_CR101","doi-asserted-by":"publisher","unstructured":"Vaithilingam P, Zhang T, Glassman EL (2022) Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems. CHI EA \u201922, pp 1\u20137. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3491101.3519665","DOI":"10.1145\/3491101.3519665"},{"key":"10869_CR102","doi-asserted-by":"publisher","unstructured":"Wang L, Zheng Z, Wu X, Sang B, Zhang J, Tao X (2023) Fork entropy: Assessing the diversity of open source software projects\u2019 forks. In: 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE), pp 204\u2013216. https:\/\/doi.org\/10.1109\/ASE56229.2023.00168","DOI":"10.1109\/ASE56229.2023.00168"},{"key":"10869_CR103","doi-asserted-by":"publisher","unstructured":"Weeraddana NR, Xu X, Alfadel M et al (2023)An empirical comparison of ethnic and gender diversity of devops and non-devops contributions to open-source projects. Empirical Softw Eng 28:150 https:\/\/doi.org\/10.1007\/s10664-023-10394-9. Accepted 11 Sept 2023","DOI":"10.1007\/s10664-023-10394-9"},{"key":"10869_CR104","doi-asserted-by":"publisher","unstructured":"Wermelinger M (2023) Using github copilot to solve simple programming problems. In: Proceedings of the 54th ACM technical symposium on computer science education, p 7. ACM, Toronto, Canada. https:\/\/doi.org\/10.1145\/3545945.3569830","DOI":"10.1145\/3545945.3569830"},{"key":"10869_CR105","unstructured":"White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, Elnashar A, Spencer-Smith J, Schmidt DC (2023) A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382"},{"key":"10869_CR106","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.8304091","author":"T Xiao","year":"2023","unstructured":"Xiao T, Treude C, Hata H, Matsumoto K (2023) Devgpt: Studying developer-chatgpt conversations. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.8304091","journal-title":"Zenodo"},{"key":"10869_CR107","doi-asserted-by":"publisher","unstructured":"Xiao T, Hata H, Treude C, Matsumoto K (2024) Generative ai for pull request descriptions: Adoption, impact, and developer interventions. Proc ACM Softw Eng 1(FSE). https:\/\/doi.org\/10.1145\/3643773","DOI":"10.1145\/3643773"},{"key":"10869_CR108","doi-asserted-by":"crossref","unstructured":"Yu Y, Wang H, Filkov V, Devanbu P, Vasilescu B (2015) Wait for it: determinants of pull request evaluation latency on github. In: Proceedings of the 12th Working Conference on Mining Software Repositories. MSR \u201915, pp 367\u2013371. IEEE Press","DOI":"10.1109\/MSR.2015.42"},{"issue":"5","key":"10869_CR109","doi-asserted-by":"publisher","first-page":"1676","DOI":"10.1109\/TSE.2020.3031401","volume":"48","author":"Z Yu","year":"2022","unstructured":"Yu Z, Fahid FM, Tu H, Menzies T (2022) Identifying self-admitted technical debts with jitterbug: A two-step approach. IEEE Trans Softw Eng 48(5):1676\u20131691. https:\/\/doi.org\/10.1109\/TSE.2020.3031401","journal-title":"IEEE Trans Softw Eng"},{"key":"10869_CR110","doi-asserted-by":"publisher","first-page":"48157","DOI":"10.1109\/ACCESS.2021.3065872","volume":"9","author":"H Zhang","year":"2021","unstructured":"Zhang H, Sakurai K (2021) A survey of software clone detection from security perspective. IEEE Access 9:48157\u201348173. https:\/\/doi.org\/10.1109\/ACCESS.2021.3065872","journal-title":"IEEE Access"},{"issue":"2","key":"10869_CR111","doi-asserted-by":"publisher","first-page":"849","DOI":"10.1109\/TSE.2022.3165056","volume":"49","author":"X Zhang","year":"2022","unstructured":"Zhang X, Yu Y, Gousios G, Rastogi A (2022) Pull request decisions explained: An empirical overview. IEEE Trans Software Eng 49(2):849\u2013871","journal-title":"IEEE Trans Software Eng"},{"issue":"4","key":"10869_CR112","doi-asserted-by":"publisher","first-page":"2140","DOI":"10.1007\/s10664-019-09696-8","volume":"24","author":"G Zhao","year":"2019","unstructured":"Zhao G, Costa DA, Zou Y (2019) Improving the pull requests review process using learning-to-rank algorithms. Empir Softw Eng 24(4):2140\u20132170. https:\/\/doi.org\/10.1007\/s10664-019-09696-8","journal-title":"Empir Softw Eng"},{"key":"10869_CR113","doi-asserted-by":"publisher","unstructured":"Zhu J, Zhou M, Mockus A (2016) Effectiveness of code contribution: from patch-based to pull-request-based tools. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. FSE 2016, pp 871\u2013882. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2950290.2950364","DOI":"10.1145\/2950290.2950364"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10869-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-026-10869-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10869-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T10:41:48Z","timestamp":1778150508000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-026-10869-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,7]]},"references-count":113,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,9]]}},"alternative-id":["10869"],"URL":"https:\/\/doi.org\/10.1007\/s10664-026-10869-5","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,7]]},"assertion":[{"value":"18 August 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 April 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"The authors declare that there are no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Clinical trial number in the manuscript"}}],"article-number":"136"}}