{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,19]],"date-time":"2026-11-19T03:20:10Z","timestamp":1795058410110,"version":"4.10.2"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62076232, 62172049"],"award-info":[{"award-number":["62076232, 62172049"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>Recently, large language models (LLM) based generative AI has been gaining momentum for their impressive high-quality performances in multiple domains, particularly after the release of the ChatGPT. Many believe that they have the potential to perform general-purpose problem-solving in software development and replace human software developers. Nevertheless, there are in a lack of serious investigation into the capability of these LLM techniques in fulfilling software development tasks. In a controlled 2 x 2 between-subject experiment with 109 participants, we examined whether and to what degree working with ChatGPT was helpful in the coding task and typical software development task and how people work with ChatGPT. We found that while ChatGPT performed well in solving simple coding problems, its performance in supporting typical software development tasks was not that good. We also observed the interactions between participants and ChatGPT and found the relations between the interactions and the outcomes. Our study thus provides first-hand insights into using ChatGPT to fulfill software engineering tasks with real-world developers and motivates the need for novel interaction mechanisms that help developers effectively work with large language models to achieve desired outcomes.<\/jats:p>","DOI":"10.1145\/3643758","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"699-721","source":"Crossref","is-referenced-by-count":29,"title":["Rocks Coding, Not Development: A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3240-343X","authenticated-orcid":false,"given":"Wei","family":"Wang","sequence":"first","affiliation":[{"name":"Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9393-6507","authenticated-orcid":false,"given":"Huilong","family":"Ning","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5767-2280","authenticated-orcid":false,"given":"Gaowei","family":"Zhang","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0136-8902","authenticated-orcid":false,"given":"Libo","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Melbourne, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1321-4035","authenticated-orcid":false,"given":"Yi","family":"Wang","sequence":"additional","affiliation":[{"name":"Beijing University of Posts and Telecommunications, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2020.2996587"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.21606\/drs.2020.282"},{"issue":"1","key":"e_1_3_1_5_2","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1109\/TSE.1986.6312975","article-title":"Experimentation in software engineering","volume":"12","author":"Basili Victor R","year":"1986","unstructured":"Victor R Basili, Richard W Selby, and David H Hutchens. 1986. Experimentation in software engineering. IEEE Transactions on Software Engineering, 12, 1 (1986), 733\u2013743. https:\/\/doi.org\/10.1109\/TSE.1986.6312975","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_6_2","article-title":"Generated faces in the wild: Quantitative comparison of stable diffusion, Midjourney and DALL-E 2","author":"Borji Ali","year":"2023","unstructured":"Ali Borji. 2023. Generated faces in the wild: Quantitative comparison of stable diffusion, Midjourney and DALL-E 2. arXiv, arXiv:2210.00586 cs.CV. https:\/\/arxiv.org\/abs\/2210.00586","journal-title":"arXiv, arXiv:2210.00586 cs.CV"},{"key":"e_1_3_1_7_2","first-page":"27","article-title":"Reliance and Automation for Human-AI Collaborative Data Labeling Conflict Resolution","volume":"321","author":"Brachman Michelle","year":"2022","unstructured":"Michelle Brachman, Zahra Ashktorab, Michael Desmond, Evelyn Duesterwald, Casey Dugan, Narendra Nath Joshi, Qian Pan, and Aabhas Sharma. 2022. Reliance and Automation for Human-AI Collaborative Data Labeling Conflict Resolution. Proceedings of the ACM on Human-Computer Interaction, 6, CSCW2, Article 321 (2022), 27 pages. https:\/\/doi.org\/10.1145\/3555212","journal-title":"Proceedings of the ACM on Human-Computer Interaction, 6"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.1987.1663532"},{"key":"e_1_3_1_9_2","article-title":"Evaluating Large Language Models Trained on Code","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv, https:\/\/arxiv.org\/abs\/2107.03374","journal-title":"arXiv"},{"key":"e_1_3_1_10_2","article-title":"Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling","author":"Chung Junyoung","year":"2014","unstructured":"Junyoung Chung, \u00c7aglar G\u00fcl\u00e7ehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR, abs\/1412.3555 (2014). http:\/\/arxiv.org\/abs\/1412.3555","journal-title":"CoRR"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376442"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9523-3"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10160-3"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568266"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3422622"},{"key":"e_1_3_1_17_2","article-title":"GraphCodeBERT: Pre-training Code Representations with Data Flow","author":"Guo Daya","year":"2021","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. Proceedings of the International Other on Learning Representations (ICLR\u201921). https:\/\/openreview.net\/forum?id=jLoC4ez43PZ","journal-title":"Proceedings of the International Other on Learning Representations (ICLR\u201921)"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1177\/154193120605000909"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-soc-071811-145443"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_8"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3304-4"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580919"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.4135\/9781483384733"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502030"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608128"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/THFE2.1960.4503259"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1057"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3480027"},{"key":"e_1_3_1_31_2","first-page":"31","volume-title":"Proceedings of the 2023 CHI Other on Human Factors in Computing Systems (Hamburg, Germany) (CHI\u201923)","author":"Liu Michael Xieyang","year":"2023","unstructured":"Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, Andrew D. Gordon. 2023. \u201cWhat It Wants Me To Say\u201d: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. Proceedings of the 2023 CHI Other on Human Factors in Computing Systems (Hamburg, Germany) (CHI\u201923). Association for Computing Machinery New York, NY, USA 598 31 https:\/\/doi.org\/10.1145\/3544548.3580817"},{"key":"e_1_3_1_32_2","first-page":"1045","volume-title":"Proceedings of the 11th Annual Other of the International Speech Communication Association (Interspeech\u201910) Makuhari, Chiba, Japan, September 26-30, 2010","author":"Mikolov Tomas","year":"2010","unstructured":"Tomas Mikolov, Martin Karafi\u00e1t, Lukas Burget, Jan Cernock\u00fd, Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the 11th Annual Other of the International Speech Communication Association (Interspeech\u201910) Makuhari, Chiba, Japan, September 26-30, 2010, Takao Kobayashi, Keikichi Hirose, and Satoshi Nakamura(Eds.). ISCA, 1045\u20131048 https:\/\/doi.org\/10.21437\/Interspeech.2010-343"},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","unstructured":"Hussein Mozannar Gagan Bansal Adam Fourney Eric Horvitz. 2023. Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. arXiv 2210.14306 [cs.SE]","DOI":"10.1145\/3613904.3641936"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2016.200"},{"key":"e_1_3_1_35_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray et al.. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022) 27730\u201327744 https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/b1efde53be364a73914f58805a001731-Paper-Conference.pdf","journal-title":"Advances in Neural Information Processing Systems 35 (2022)"},{"key":"e_1_3_1_36_2","first-page":"16","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI\u201921)","author":"Pater Jessica","year":"2021","unstructured":"Jessica Pater, Amanda Coupe, Rachel Pfafman, Chanda Phelan, Tammy Toscos, Maia Jacobs. 2021. Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the Field. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI\u201921). Association for Computing Machinery New York, NY, USA, Article 141, 16 pages. https:\/\/doi.org\/10.1145\/3411764.3445734"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3576915.3623157"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/325737.325845"},{"key":"e_1_3_1_39_2","first-page":"8748","volume-title":"Proceedings of the 38th International Conference on Machine Learning (ICML\u201921, Vol. 139)","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. 2021. Learning Transferable Visual Models from Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML\u201921, Vol. 139), Marina Meila, Tong Zhang (Eds.). PMLR 8748\u20138763 https:\/\/proceedings.mlr.press\/v139\/radford21a.html"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600211.3604712"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04898-2_616"},{"key":"e_1_3_1_42_2","volume-title":"Transformers for Natural Language Processing: Build, Train, and Fine-Tune Deep Neural Network Architectures for NLP with Python, PyTorch, TensorFlow, BERT, and GPT-3 (2nd ed.)","author":"Rothman Denis","year":"2022","unstructured":"Denis Rothman, Antonio Gulli. 2022. Transformers for Natural Language Processing: Build, Train, and Fine-Tune Deep Neural Network Architectures for NLP with Python, PyTorch, TensorFlow, BERT, and GPT-3 (2nd ed.) Packt Publishing Ltd. https:\/\/www.packtpub.com\/en-PT\/product\/transformers-for-natural-language-processing-second-edition-9781803247335"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.2307\/2287653"},{"key":"e_1_3_1_44_2","article-title":"Screening and Treatment of Subclinical Hypothyroidism or Hyperthyroidism","volume":"24","author":"Rugge Bruin","year":"2012","unstructured":"Bruin Rugge, Howard Balshem, Raj Sehgal, Rose Relevo, Paul Gorman, Mark Helfand. 2012. Screening and Treatment of Subclinical Hypothyroidism or Hyperthyroidism. Comparative Effectiveness Reviews 24 (2012). https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK83492\/","journal-title":"Comparative Effectiveness Reviews"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","unstructured":"Iflaah Salman Ayse Tosun Misirli Natalia Juristo. 2015. Are Students Representatives of Professionals in Software Engineering Experiments?. In Proceedings of the 37th International Other on Software Engineering - Volume 1 (ICSE\u201915) IEEE Press 666\u2013676 https:\/\/doi.org\/10.5555\/2818754.2818836 10.5555\/2818754.2818836","DOI":"10.5555\/2818754.2818836"},{"key":"e_1_3_1_46_2","first-page":"2205","volume-title":"Proceedings of the 32nd USENIX Security Symposium (USENIX Security\u201923)","author":"Sandoval Gustavo","year":"2023","unstructured":"Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, Brendan Dolan-Gavitt. 2023. Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants. In Proceedings of the 32nd USENIX Security Symposium (USENIX Security\u201923), USENIX Association, Anaheim, CA, 2205\u20132222 https:\/\/www.usenix.org\/conference\/usenixsecurity23\/presentation\/sandoval"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.97"},{"key":"e_1_3_1_48_2","article-title":"Unit Test Case Generation with Transformers","author":"Tufano Michele","year":"2020","unstructured":"Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, Neel Sundaresan. 2020. Unit Test Case Generation with Transformers. CoRR abs\/2009.05617 (2020). arXiv:2009.05617 https:\/\/arxiv.org\/abs\/2009.05617","journal-title":"CoRR abs\/2009.05617 (2020)"},{"key":"e_1_3_1_49_2","first-page":"191","article-title":"Programming with Unrestricted Natural Language","author":"Vadas David","year":"2005","unstructured":"David Vadas, James R. Curran. 2005. Programming with Unrestricted Natural Language. In Proceedings of the 2005 Australasian Language Technology Association Workshop (ATLA\u201905) Sydney, Australia, 191\u2013199 https:\/\/aclanthology.org\/U05-1027","journal-title":"Proceedings of the 2005 Australasian Language Technology Association Workshop (ATLA\u201905)"},{"key":"e_1_3_1_50_2","first-page":"7","volume-title":"Extended Abstracts of the 2022 CHI Other on Human Factors in Computing Systems (CHI EA\u201922)","author":"Vaithilingam Priyan","year":"2022","unstructured":"Priyan Vaithilingam, Tianyi Zhang, Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. Extended Abstracts of the 2022 CHI Other on Human Factors in Computing Systems (CHI EA\u201922) Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages. https:\/\/doi.org\/10.1145\/3491101.3519665"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-00288-7"},{"issue":"2","key":"e_1_3_1_52_2","first-page":"101","article-title":"A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong","volume":"25","author":"Vargha Andr\u00e1s","year":"2000","unstructured":"Andr\u00e1s Vargha, Harold D Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25 2 (2000) 101\u2013132 https:\/\/doi.org\/10.3102\/10769986025002101","journal-title":"Journal of Educational and Behavioral Statistics"},{"key":"e_1_3_1_53_2","first-page":"1","article-title":"Attention Is All You Need","volume":"31","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, Illia Polosukhin. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems 31 (2017), 1\u201315 https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_54_2","first-page":"1","volume-title":"Extended Abstracts of the 2020 CHI Other on Human Factors in Computing Systems (CHI EA\u201920)","author":"Wang Dakuo","year":"2020","unstructured":"Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, Qianying Wang. 2020. From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People. In Extended Abstracts of the 2020 CHI Other on Human Factors in Computing Systems (CHI EA\u201920) Association for Computing Machinery, New York, NY, USA 1\u20136 https:\/\/doi.org\/10.1145\/3334480.3381069"},{"key":"e_1_3_1_55_2","first-page":"18","volume-title":"Proceedings of the 2021 CHI Other on Human Factors in Computing Systems (CHI\u201921)","author":"Wang Dakuo","year":"2021","unstructured":"Dakuo Wang, Liuping Wang, Zhan Zhang, Ding Wang, Haiyi Zhu, Yvonne Gao, Xiangmin Fan, Feng Tian. 2021. \u201cBrilliant AI Doctor\u201d in Rural Clinics: Challenges in AI-Powered Clinical Decision Support System Deployment. In Proceedings of the 2021 CHI Other on Human Factors in Computing Systems (CHI\u201921) Association for Computing Machinery, New York, NY, USA Article 697, 18 pages. https:\/\/doi.org\/10.1145\/3411764.3445432"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2017.27"},{"key":"e_1_3_1_58_2","first-page":"833","volume-title":"Proceedings of the ACM 2012 Other on Computer Supported Cooperative Work (CSCW\u201912)","author":"Wang Yi-Chia","year":"2012","unstructured":"Yi-Chia Wang, Robert Kraut, John M. Levine. 2012. To stay or leave? The relationship of emotional and informational support to commitment in online health support groups. In Proceedings of the ACM 2012 Other on Computer Supported Cooperative Work (CSCW\u201912) Association for Computing Machinery, New York, NY, USA 833\u2013842 https:\/\/doi.org\/10.1145\/2145204.2145329"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387111"},{"key":"e_1_3_1_60_2","first-page":"402","volume-title":"26th International Conference on Intelligent User Interfaces College Station, TX, USA (IUI\u201921","author":"Weisz Justin D.","year":"2021","unstructured":"Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces College Station, TX, USA (IUI\u201921 Association for Computing Machinery, New York, NY, USA, 402\u2013412 https:\/\/doi.org\/10.1145\/3397481.3450656"},{"key":"e_1_3_1_61_2","first-page":"369","volume-title":"27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI\u201922)","author":"Weisz Justin D.","year":"2022","unstructured":"Justin D. Weisz, Michael Muller, Steven I. Ross, Fernando Martinez, Stephanie Houde, Mayank Agarwal, Kartik Talamadupula, John T. Richards. 2022. Better Together? An Evaluation of AI-Supported Code Translation. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI\u201922) Association for Computing Machinery, New York, NY, USA, 369\u2013391 https:\/\/doi.org\/10.1145\/3490099.3511157"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3570220"},{"key":"e_1_3_1_63_2","unstructured":"Jules White Quchen Fu Sam Hays Michael Sandborn Carlos Olea Henry Gilbert Ashraf Elnashar Jesse Spencer-Smith Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] https:\/\/arxiv.org\/abs\/2302.11382"},{"key":"e_1_3_1_64_2","unstructured":"Ben James Winer Donald R Brown Kenneth M Michels. 1971. Statistical Principles in Experimental Design. Vol. 2. McGraw-Hill New York NY USA"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517582"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3487569"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1041"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3156071"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534864"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643758","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643758","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T08:07:14Z","timestamp":1770192434000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643758"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":69,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3643758"],"URL":"https:\/\/doi.org\/10.1145\/3643758","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}