{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T19:42:32Z","timestamp":1795203752959,"version":"4.10.3"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>Empirical software engineering research on production systems has brought forth a better understanding of the software engineering process for practitioners and researchers alike. However, only a small subset of production systems is studied, limiting the impact of this research. While software engineering practitioners could benefit from replicating research on their own data, this poses its own set of challenges, since performing replications requires a deep understanding of research methodologies and subtle nuances in software engineering data. Given that large language models (LLMs), such as GPT-4, show promise in tackling both software engineering- and science-related tasks, these models could help replicate and thus democratize empirical software engineering research.<\/jats:p>\n                  <jats:p>In this paper, we examine GPT-4\u2019s abilities to perform replications of empirical software engineering research on new data. We specifically study their ability to surface assumptions made in empirical software engineering research methodologies, as well as their ability to plan and generate code for analysis pipelines on seven empirical software engineering papers. We perform a user study with 14 participants with software engineering research expertise, who evaluate GPT-4-generated assumptions and analysis plans (i.e., a list of module specifications) from the papers. We find that GPT-4 is able to surface correct assumptions, but struggles to generate ones that apply common knowledge about software engineering data. In a manual analysis of the generated code, we find that the GPT-4-generated code contains correct high-level logic, given a subset of the methodology. However, the code contains many small implementation-level errors, reflecting a lack of software engineering knowledge. Our findings have implications for leveraging LLMs for software engineering research as well as practitioner data scientists in software teams.<\/jats:p>","DOI":"10.1145\/3660767","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"1330-1353","source":"Crossref","is-referenced-by-count":25,"title":["Can GPT-4 Replicate Empirical Software Engineering Research?"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6722-9959","authenticated-orcid":false,"given":"Jenny T.","family":"Liang","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2249-6371","authenticated-orcid":false,"given":"Carmen","family":"Badea","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7774-0387","authenticated-orcid":false,"given":"Christian","family":"Bird","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8885-8367","authenticated-orcid":false,"given":"Robert","family":"DeLine","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0654-4335","authenticated-orcid":false,"given":"Denae","family":"Ford","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2263-9326","authenticated-orcid":false,"given":"Nicole","family":"Forsgren","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4905-1469","authenticated-orcid":false,"given":"Thomas","family":"Zimmermann","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2023. ChatGPT Plugins. Retrieved September 25 2023 from https:\/\/openai.com\/blog\/chatgpt-plugins#code-interpreter."},{"key":"e_1_3_1_3_2","unstructured":"2023. Standards | Empirical Standards. Retrieved September 25 2023 from https:\/\/sigsoft.org\/EmpiricalStandards\/docs\/?standard=Replication."},{"key":"e_1_3_1_4_2","unstructured":"Rohan Anil Andrew M Dai Orhan Firat Melvin Johnson Dmitry Lepikhin Alexandre Passos Siamak Shakeri Emanuel Taropa Paige Bailey Zhifeng Chen et al. 2023. PaLM 2 Technical Report. arXiv preprint arXiv:2305.10403 (2023)."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-33607-z"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-demo.9"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","unstructured":"Sebastian Baltes and Stephan Diehl. 2018. Towards a Theory of Software Development Expertise. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 187\u2013200. https:\/\/doi.org\/10.1145\/3236024.3236061 10.1145\/3236024.3236061","DOI":"10.1145\/3236024.3236061"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3586030"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","unstructured":"Andrew Begel and Thomas Zimmermann. 2014. Analyze This! 145 Questions for Data Scientists in Software Engineering. In IEEE\/ACM International Conference on Software Engineering (ICSE). 12\u201323. https:\/\/doi.org\/10.1145\/2568225.2568233 10.1145\/2568225.2568233","DOI":"10.1145\/2568225.2568233"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070550"},{"issue":"2020","key":"e_1_3_1_11_2","first-page":"1877","article-title":"Language Models Are Few-shot Learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models Are Few-shot Learners. Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","unstructured":"J. Carver J. VanVoorhis and V. Basili. 2004. Understanding the Impact of Assumptions on Experimental Validity. In International Symposium on Empirical Software Engineering (ISESE). 251\u2013260. https:\/\/doi.org\/10.1109\/ISESE.2004.1334912 10.1109\/ISESE.2004.1334912","DOI":"10.1109\/ISESE.2004.1334912"},{"key":"e_1_3_1_13_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_3_1_14_2","unstructured":"Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann et al. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.02311 (2022)."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","unstructured":"Robert A DeLine. 2021. Glinda: Supporting Data Science with Live Programming GUIs and a Domain-specific Language. In ACM CHI Conference on Human Factors in Computing Systems. 1\u201311. https:\/\/doi.org\/10.1145\/3411764.3445267 10.1145\/3411764.3445267","DOI":"10.1145\/3411764.3445267"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338112"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","unstructured":"Jon Eyolfson Lin Tan and Patrick Lam. 2011. Do Time of Day and Developer Experience Affect Commit Bugginess?. In Working Conference on Mining Software Repositories (MSR). 153\u2013162. https:\/\/doi.org\/10.1145\/1985441.1985464 10.1145\/1985441.1985464","DOI":"10.1145\/1985441.1985464"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","unstructured":"Sakina Fatima Taher A Ghaleb and Lionel Briand. 2022. Flakify: A Black-box Language Model-based Predictor for Flaky Tests. IEEE Transactions on Software Engineering (2022). https:\/\/doi.org\/10.1109\/TSE.2022.3201209 10.1109\/TSE.2022.3201209","DOI":"10.1109\/TSE.2022.3201209"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.70731"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1177\/1077800405284363"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3487567"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","unstructured":"Enrico Fregnan Larissa Braz Marco D\u2019Ambros G\u00fcl \u00c7alikli and Alberto Bacchelli. 2022. First Come First Served: The Impact of File Position on Code Review. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 483\u2013494. https:\/\/doi.org\/10.1145\/3540250.3549177 10.1145\/3540250.3549177","DOI":"10.1145\/3540250.3549177"},{"key":"e_1_3_1_24_2","unstructured":"Fan Gao Hang Jiang Moritz Blum Jinghui Lu Yuang Jiang and Irene Li. 2023. Large Language Models on Wikipedia- Style Survey Generation: an Evaluation in NLP Concepts. arXiv preprint arXiv:2308.10410 (2023)."},{"key":"e_1_3_1_25_2","doi-asserted-by":"crossref","unstructured":"Ken Gu Madeleine Grunde-McLaughlin Andrew M McNutt Jeffrey Heer and Tim Althoff. 2024. How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. (2024).","DOI":"10.1145\/3613904.3641891"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","unstructured":"Philip J Guo Thomas Zimmermann Nachiappan Nagappan and Brendan Murphy. 2010. Characterizing and Predicting which Bugs Get Fixed: An Empirical Study of Microsoft Windows. In ACM\/IEEE International Conference on Software Engineering (ICSE). 495\u2013504. https:\/\/doi.org\/10.1145\/1806799.1806871 10.1145\/1806799.1806871","DOI":"10.1145\/1806799.1806871"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","unstructured":"Suchin Gururangan Ana Marasovic Swabha Swayamdipta Kyle Lo Iz Beltagy Doug Downey and Noah A. Smith. Don\u2019t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Annual Meeting of the Association for Computational Linguistics (ACL). 8342\u20138360. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.740 10.18653\/v1\/2020.acl-main.740","DOI":"10.18653\/v1\/2020.acl-main.740"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","unstructured":"Emitza Guzman David Az\u00f3car and Yang Li. 2014. Sentiment Analysis of Commit Comments in GitHub: An Empirical Study. In Working Conference on Mining Software Repositories (MSR). 352\u2013355. https:\/\/doi.org\/10.1145\/2597073.2597118 10.1145\/2597073.2597118","DOI":"10.1145\/2597073.2597118"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","unstructured":"Perttu H\u00e4m\u00e4l\u00e4inen Mikke Tavast and Anton Kunnari. 2023. Evaluating Large Language Models in Generating Synthetic HCI Research Data: A Case Study. In ACM CHI Conference on Human Factors in Computing Systems (CHI). 1\u201319. https:\/\/doi.org\/10.1145\/3544548.3580688 10.1145\/3544548.3580688","DOI":"10.1145\/3544548.3580688"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1080\/10508406.2013.802652"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","unstructured":"Tobias Hey Jan Keim Anne Koziolek and Walter F Tichy. 2020. Norbert: Transfer Learning for Requirements Classification. In IEEE International Requirements Engineering Conference (RE). IEEE 169\u2013179. https:\/\/doi.org\/10.1109\/RE48521.2020.00028 10.1109\/RE48521.2020.00028","DOI":"10.1109\/RE48521.2020.00028"},{"key":"e_1_3_1_32_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv preprint arXiv:2308.10620 (2023)."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","unstructured":"Hennie Huijgens Ayushi Rastogi Ernst Mulders Georgios Gousios and Arie van Deursen. 2020. Questions for Data Scientists in Software Engineering: A Replication. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 568\u2013579. https:\/\/doi.org\/10.1145\/3368089.3409717 10.1145\/3368089.3409717","DOI":"10.1145\/3368089.3409717"},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","unstructured":"Brittany Johnson Rahul Pandita Justin Smith Denae Ford Sarah Elder Emerson Murphy-Hill Sarah Heckman and Caitlin Sadowski. 2016. A Cross-tool Communication Study on Program Analysis Tool nNtifications. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). 73\u201384.","DOI":"10.1145\/2950290.2950304"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","unstructured":"Eunice Jun Maureen Daum Jared Roesch Sarah Chasins Emery Berger Rene Just and Katharina Reinecke. 2019. Tea: A High-level Language and Runtime System for Automating Statistical Analysis. In ACM Symposium on User Interface Software and Technology (UIST). 591\u2013603. https:\/\/doi.org\/10.1145\/3332165.3347940 10.1145\/3332165.3347940","DOI":"10.1145\/3332165.3347940"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","unstructured":"Eunice Jun Audrey Seo Jeffrey Heer and Ren\u00e9 Just. 2022. Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships. In ACM CHI Conference on Human Factors in Computing Systems (CHI). 1\u201316. https:\/\/doi.org\/10.1145\/3491102.3501888 10.1145\/3491102.3501888","DOI":"10.1145\/3491102.3501888"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","unstructured":"Miryung Kim Dongxiang Cai and Sunghun Kim. 2011. An Empirical Investigation into the Role of API-level Refactorings during Software Evolution. In IEEE\/ACM International Conference on Software Engineering (ICSE). 151\u2013160. https:\/\/doi.org\/10.1145\/1985793.1985815 10.1145\/1985793.1985815","DOI":"10.1145\/1985793.1985815"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","unstructured":"Miryung Kim Thomas Zimmermann Robert DeLine and Andrew Begel. 2016. The Emerging Role of Data Scientists on Software Development Teams. In IEEE\/ACM International Conference on Software Engineering (ICSE). 96\u2013107. https:\/\/doi.org\/10.1109\/TSE.2017.2754374 10.1109\/TSE.2017.2754374","DOI":"10.1109\/TSE.2017.2754374"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2754374"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2021.3071086"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_3"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9279-3"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","unstructured":"Jenny T Liang Maryam Arab Minhyuk Ko Amy J Ko and Thomas D LaToza. 2023. A Qualitative Study on the Implementation Design Decisions of Developers. In IEEE\/ACM International Conference on Software Engineering (ICSE). 435\u2013447. https:\/\/doi.org\/10.1109\/ICSE48619.2023.00047 10.1109\/ICSE48619.2023.00047","DOI":"10.1109\/ICSE48619.2023.00047"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","unstructured":"Jenny T. Liang Carmen Badea Christian Bird Robert DeLine Denae Ford Nicole Forsgren and Thomas Zimmermann. 2024. Supplemental Materials to \u201cCan Large Language Models Replicate Empirical Software Engineering Research\u201d. https:\/\/doi.org\/10.6084\/m9.figshare.24210468 10.6084\/m9.figshare.24210468","DOI":"10.6084\/m9.figshare.24210468"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","unstructured":"Jenny T Liang Chenyang Yang and Brad A Myers. 2024. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In IEEE\/ACM International Conference on Software Engineering (ICSE). 1\u201313. https:\/\/doi.org\/10.1145\/3597503.3608128 10.1145\/3597503.3608128","DOI":"10.1145\/3597503.3608128"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","unstructured":"Jenny T Liang Thomas Zimmermann and Denae Ford. 2022. Understanding Skills for OSS Communities on GitHub. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 170\u2013182. https:\/\/doi.org\/10.1145\/3540250.3549082 10.1145\/3540250.3549082","DOI":"10.1145\/3540250.3549082"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00638"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","unstructured":"David Lo Nachiappan Nagappan and Thomas Zimmermann. 2015. How Practitioners Perceive the Relevance of Software Engineering Research. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 415\u2013425. https:\/\/doi.org\/10.1145\/2786805.2786809 10.1145\/2786805.2786809","DOI":"10.1145\/2786805.2786809"},{"key":"e_1_3_1_49_2","unstructured":"Shantanu Mandal Adhrik Chethan Vahid Janfaza SM Mahmud Todd A Anderson Javier Turek Jesmin Jahan Tithi and Abdullah Muzahid. 2023. Large Language Models Based Automatic Synthesis of Software Specifications. arXiv preprint arXiv:2304.09181 (2023)."},{"key":"e_1_3_1_50_2","doi-asserted-by":"crossref","unstructured":"Hussein Mozannar Gagan Bansal Adam Fourney and Eric Horvitz. 2024. Reading between the Lines: Modeling User Behavior and Costs in AI-assisted Programming. In ACM CHI Conference on Human Factors in Computing Systems (CHI).","DOI":"10.1145\/3613904.3641936"},{"key":"e_1_3_1_51_2","unstructured":"OpenAI. 2023. GPT-4 Technical Report. ArXiv abs\/2303.08774 (2023)."},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","unstructured":"Daniel Pletea Bogdan Vasilescu and Alexander Serebrenik. 2014. Security and Emotion: Sentiment Analysis of Security Discussions on GitHub. In Working Conference on Mining Software Repositories (MSR). 348\u2013351. https:\/\/doi.org\/10.1145\/2597073.2597117 10.1145\/2597073.2597117","DOI":"10.1145\/2597073.2597117"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","unstructured":"Lutz Prechelt. 2021. On Implicit Assumptions Underlying Software Engineering Research. In International Conference on Evaluation and Assessment in Software Engineering (EASE). 336\u2013339. https:\/\/doi.org\/10.1145\/3463274.3463356 10.1145\/3463274.3463356","DOI":"10.1145\/3463274.3463356"},{"key":"e_1_3_1_54_2","unstructured":"Aditya Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen and Ilya Sutskever. 2021. Zero-shotTtext-to-image Generation. In International Conference on Machine Learning (ICML). PMLR 8821\u20138831."},{"key":"e_1_3_1_55_2","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Tal Remez J\u00e9r\u00e9my Rapin et al. 2023. Code LLaMA: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023)."},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3188720"},{"key":"e_1_3_1_57_2","unstructured":"Morgan Klaus Scheuerman Katta Spiel Oliver L Haimson Foad Hamidi and Stacy M Branham. 2020. HCI Guidelines for Gender Equity and Inclusivity. In UMBC Faculty Collection."},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","unstructured":"Marija Selakovic and Michael Pradel. 2016. Performance Issues and Optimizations in JavaScript: An Empirical Study. In IEEE\/ACM International Conference on Software Engineering (ICSE). 61\u201372. https:\/\/doi.org\/10.1145\/2884781.2884829 10.1145\/2884781.2884829","DOI":"10.1145\/2884781.2884829"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","unstructured":"Mikke Tavast Anton Kunnari and Perttu H\u00e4m\u00e4l\u00e4inen. 2022. Language Models Can Generate Human-like Self-reports of Emotion. In ACM International Conference on Intelligent User Interfaces (IUI). 69\u201372. https:\/\/doi.org\/10.1145\/3490100.3516464 10.1145\/3490100.3516464","DOI":"10.1145\/3490100.3516464"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","unstructured":"Yingchen Tian Yuxia Zhang Klaas-Jan Stol Lin Jiang and Hui Liu. 2022. What Makes a Good Commit Message?. In IEEE\/ACM International Conference on Software Engineering (ICSE). 2389\u20132401. https:\/\/doi.org\/10.1145\/3510003.3510205 10.1145\/3510003.3510205","DOI":"10.1145\/3510003.3510205"},{"key":"e_1_3_1_61_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_3_1_62_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. LLaMA 2: Open Foundation and Fine-tuned Chat Models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","unstructured":"David Wadden Kyle Lo Lucy Lu Wang Arman Cohan Iz Beltagy and Hannaneh Hajishirzi. 2022. MultiVerS: Improving Scientific Claim Verification with Weak Supervision and Full-document Context. In Findings of the Association for Computational Linguistics: NAACL. 61\u201376. https:\/\/doi.org\/10.18653\/v1\/2022.findings-naacl.6 10.18653\/v1\/2022.findings-naacl.6","DOI":"10.18653\/v1\/2022.findings-naacl.6"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","unstructured":"Bingcheng Wang Pei-Luen Patrick Rau and Tianyi Yuan. 2022. Measuring User Competence in Using Artificial Intelligence: Validity and Reliability of Artificial Intelligence Literacy Scale. Behaviour & Information Technology (2022) 1\u201314. https:\/\/doi.org\/10.1080\/0144929X.2022.2072768 10.1080\/0144929X.2022.2072768","DOI":"10.1080\/0144929X.2022.2072768"},{"key":"e_1_3_1_65_2","first-page":"24824","article-title":"Chain-of-thought Prompting Elicits Reasoning in Large Language Models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","unstructured":"Moshi Wei Nima Shiri Harzevili Yuchao Huang Junjie Wang and Song Wang. 2022. Clear: Contrastive Learning for API Recommendation. In IEEE\/ACM International Conference on Software Engineering (ICSE). 376\u2013387. https:\/\/doi.org\/10.1145\/3510003.3510159 10.1145\/3510003.3510159","DOI":"10.1145\/3510003.3510159"},{"key":"e_1_3_1_67_2","unstructured":"Tongshuang Wu Haiyi Zhu Maya Albayrak Alexis Axon Amanda Bertsch Wenxing Deng Ziqi Ding Bill Guo Sireesh Gururaja Tzu-Sheng Kuo et al. 2023. LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs. arXiv preprint arXiv:2307.10168 (2023)."},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","unstructured":"Ziang Xiao Xingdi Yuan Q Vera Liao Rania Abdelghani and Pierre-Yves Oudeyer. 2023. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In ACM International Conference on Intelligent User Interfaces (IUI). 75\u201378. https:\/\/doi.org\/10.1145\/3581754.3584136 10.1145\/3581754.3584136","DOI":"10.1145\/3581754.3584136"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","unstructured":"Frank F Xu Uri Alon Graham Neubig and Vincent Josua Hellendoorn. 2022. A Systematic Evaluation of Large Language Models of Code. In ACM SIGPLAN International Symposium on Machine Programming (MAPS). 1\u201310. https:\/\/doi.org\/10.1145\/3520312.3534862 10.1145\/3520312.3534862","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","unstructured":"Tao Yu Rui Zhang Kai Yang Michihiro Yasunaga Dongxu Wang Zifan Li James Ma Irene Li Qingning Yao Shanelle Roman Zilin Zhang and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Conference on Empirical Methods in Natural Language Processing (EMNLP). 3911\u20133921. https:\/\/doi.org\/10.18653\/v1\/D18-1425 10.18653\/v1\/D18-1425","DOI":"10.18653\/v1\/D18-1425"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","unstructured":"Jialu Zhang Todd Mytkowicz Mike Kaufman Ruzica Piskac and Shuvendu K Lahiri. 2022. Using Pre-trained Language Models to Resolve Textual and Semantic Merge Conflicts (Experience Paper). In ACM International Symposium on Software Testing and Analysis (ISSTA). 77\u201388. https:\/\/doi.org\/10.1145\/3533767.3534396 10.1145\/3533767.3534396","DOI":"10.1145\/3533767.3534396"},{"key":"e_1_3_1_72_2","doi-asserted-by":"crossref","unstructured":"Zibin Zheng Kaiwen Ning Jiachi Chen Yanlin Wang Wenqing Chen Lianghong Guo and Weicheng Wang. 2023. Towards an Understanding of Large Language Models in Software Engineering Tasks. arXiv preprint arXiv:2308.11396 (2023).","DOI":"10.1007\/s10664-024-10602-0"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660767","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3660767","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T08:06:07Z","timestamp":1770192367000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":71,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3660767"],"URL":"https:\/\/doi.org\/10.1145\/3660767","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}