{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T22:35:39Z","timestamp":1776119739182,"version":"3.50.1"},"reference-count":109,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,11,29]],"date-time":"2023-11-29T00:00:00Z","timestamp":1701216000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.<\/jats:p>","DOI":"10.1145\/3617367","type":"journal-article","created":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T10:51:54Z","timestamp":1692787914000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":160,"title":["\u201cIt\u2019s Weird That it Knows What I Want\u201d: Usability and Interactions with Copilot for Novice Programmers"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2807-6042","authenticated-orcid":false,"given":"James","family":"Prather","sequence":"first","affiliation":[{"name":"Abilene Christian University, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5781-1136","authenticated-orcid":false,"given":"Brent N.","family":"Reeves","sequence":"additional","affiliation":[{"name":"Abilene Christian University, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5150-9806","authenticated-orcid":false,"given":"Paul","family":"Denny","sequence":"additional","affiliation":[{"name":"The University of Auckland, New Zealand"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1446-647X","authenticated-orcid":false,"given":"Brett A.","family":"Becker","sequence":"additional","affiliation":[{"name":"University College Dublin, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6829-9449","authenticated-orcid":false,"given":"Juho","family":"Leinonen","sequence":"additional","affiliation":[{"name":"The University of Auckland, New Zealand"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8269-2909","authenticated-orcid":false,"given":"Andrew","family":"Luxton-Reilly","sequence":"additional","affiliation":[{"name":"The University of Auckland, New Zealand"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3221-7015","authenticated-orcid":false,"given":"Garrett","family":"Powell","sequence":"additional","affiliation":[{"name":"Abilene Christian University, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4279-6284","authenticated-orcid":false,"given":"James","family":"Finnie-Ansley","sequence":"additional","affiliation":[{"name":"The University of Auckland, New Zealand"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5337-715X","authenticated-orcid":false,"given":"Eddie Antonio","family":"Santos","sequence":"additional","affiliation":[{"name":"University College Dublin, Ireland"}]}],"member":"320","published-online":{"date-parts":[[2023,11,29]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174156"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/1520340.1520420"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3371156"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432551"},{"key":"e_1_3_1_6_2","unstructured":"Desai Ankur and Deo Atul. 2022. Introducing Amazon CodeWhisperer the ML-powered Coding Companion. Retrieved from https:\/\/aws.amazon.com\/blogs\/machine-learning\/introducing-amazon-codewhisperer-the-ml-powered-coding-companion\/"},{"key":"e_1_3_1_7_2","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432430"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3586030"},{"key":"e_1_3_1_10_2","volume-title":"Extreme Programming Explained: Embrace Change","author":"Beck Kent","year":"2000","unstructured":"Kent Beck. 2000. Extreme Programming Explained: Embrace Change. Addison-Wesley Professional, Boston, Massachusetts."},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569759"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3344429.3372508"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159453"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287432"},{"key":"e_1_3_1_15_2","article-title":"A Typology of Ethical Risks in Language Technology with an eye towards where transparent documentation can help","author":"Bender Emily M.","year":"2019","unstructured":"Emily M. Bender. 2019. A Typology of Ethical Risks in Language Technology with an eye towards where transparent documentation can help. Future of Artificial Intelligence: Language, Ethics, Technology Workshop.","journal-title":"Future of Artificial Intelligence: Language, Ethics, Technology Workshop"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557079"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3589996"},{"key":"e_1_3_1_18_2","unstructured":"Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson Shyamal Buch Dallas Card Rodrigo Castellon Niladri Chatterji Annie Chen Kathleen Creel Jared Quincy Davis Dora Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren Gillespie Karan Goel Noah Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani Omar Khattab Pang Wei Koh Mark Krass Ranjay Krishna Rohith Kuditipudi Ananya Kumar Faisal Ladhak Mina Lee Tony Lee Jure Leskovec Isabelle Levent Xiang Lisa Li Xuechen Li Tengyu Ma Ali Malik Christopher D. Manning Suvir Mirchandani Eric Mitchell Zanele Munyikwa Suraj Nair Avanika Narayan Deepak Narayanan Ben Newman Allen Nie Juan Carlos Niebles Hamed Nilforoshan Julian Nyarko Giray Ogut Laurel Orr Isabel Papadimitriou Joon Sung Park Chris Piech Eva Portelance Christopher Potts Aditi Raghunathan Rob Reich Hongyu Ren Frieda Rong Yusuf Roohani Camilo Ruiz Jack Ryan Christopher R\u00e9 Dorsa Sadigh Shiori Sagawa Keshav Santhanam Andy Shih Krishnan Srinivasan Alex Tamkin Rohan Taori Armin W. Thomas Florian Tram\u00e8r Rose E. Wang William Wang Bohan Wu Jiajun Wu Yuhuai Wu Sang Michael Xie Michihiro Yasunaga Jiaxuan You Matei Zaharia Michael Zhang Tianyi Zhang Xikun Zhang Yuhui Zhang Lucia Zheng Kaitlyn Zhou and Percy Liang. 2022. On the Opportunities and Risks of Foundation Models. arXiv:2108.07258. Retrieved from https:\/\/arxiv.org\/abs\/2108.07258"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1080\/2159676X.2019.1628806"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1037\/qup0000196"},{"key":"e_1_3_1_22_2","first-page":"103","volume-title":"Proceedings of the Human-computer Interaction Handbook","author":"Brave Scott","year":"2007","unstructured":"Scott Brave and Clifford Nass. 2007. Emotion in human-computer interaction. In Proceedings of the Human-computer Interaction Handbook. CRC Press, Boca Raton, FL, 103\u2013118."},{"key":"e_1_3_1_23_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3084362"},{"key":"e_1_3_1_25_2","unstructured":"Bei Chen Fengji Zhang Anh Nguyen Daoguang Zan Zeqi Lin Jian-Guang Lou and Weizhu Chen. 2022. CodeT: Code Generation with Generated Tests. arXiv:2207.10397. Retrieved from https:\/\/arxiv.org\/abs\/2207.10397"},{"key":"e_1_3_1_26_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(83)90008-5"},{"key":"e_1_3_1_28_2","unstructured":"Code4Me. 2022. Code4Me. Retrieved May 24 2023 from https:\/\/code4me.me\/"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445257"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3160489.3160492"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.3354\/esep00195"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3478431.3499314"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/2325296.2325318"},{"key":"e_1_3_1_34_2","unstructured":"Paul Denny James Prather Brett A. Becker James Finnie-Ansley Arto Hellas Juho Leinonen Andrew Luxton-Reilly Brent N. Reeves Eddie Antonio Santos and Sami Sarsa. 2023. Computing Education in the Era of Generative AI. arXiv:2306.02608. Retrieved from https:\/\/arxiv.org\/abs\/2306.02608"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445696"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3441636.3442309"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2123433119"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3483843"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/5657.001.0001"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2021.3133805"},{"key":"e_1_3_1_41_2","unstructured":"FauxPilot. 2023. FauxPilot - An Open-source Alternative to GitHub Copilot server. Retrieved May 24 2023 from https:\/\/github.com\/fauxpilot\/fauxpilot"},{"key":"e_1_3_1_42_2","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155. Retrieved from https:\/\/arxiv.org\/abs\/2002.08155","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511861.3511863"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3576123.3576134"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3373165.3373177"},{"key":"e_1_3_1_46_2","unstructured":"GitHub. 2021. GitHub Copilot - Your AI Pair Programmer. Retrieved July 21 2022 from https:\/\/github.com\/features\/copilot\/"},{"key":"e_1_3_1_47_2","volume-title":"100 Questions (and Answers) About Qualitative Research","author":"Given Lisa M.","year":"2015","unstructured":"Lisa M. Given. 2015. 100 Questions (and Answers) About Qualitative Research. SAGE publications, Thousand Oaks, CA."},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.3102\/003465430298487"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3059009.3059065"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3300115.3309517"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3526113.3545659"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366882"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3293794"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231711"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/1839594.1839609"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2016911.2016917"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/2016911.2016934"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287378"},{"key":"e_1_3_1_59_2","doi-asserted-by":"crossref","unstructured":"Juho Leinonen Paul Denny Stephen MacNeil Sami Sarsa Seth Bernstein Joanne Kim Andrew Tran and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. arXiv:2304.03938. Retrieved from https:\/\/arxiv.org\/abs\/2304.03938","DOI":"10.1145\/3587102.3588785"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3478431.3499372"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569770"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376590"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/2960310.2960334"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858252"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3487050"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569785"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2021.12.259"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406279"},{"key":"e_1_3_1_71_2","volume-title":"Unlocking the Clubhouse: Women in Computing","author":"Margolis Jane","year":"2002","unstructured":"Jane Margolis and Allan Fisher. 2002. Unlocking the Clubhouse: Women in Computing. MIT press, Cambridge, MA, USA."},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3469885"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1207\/S15327019EB1103_2"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/1268784.1268831"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359174"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/563517.563353"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/2787622.2787733"},{"key":"e_1_3_1_78_2","unstructured":"OpenAI. 2022. DALL \\({\\cdot }\\) E 2. Retrieved May 24 2023 from https:\/\/openai.com\/product\/dall-e-2"},{"key":"e_1_3_1_79_2","unstructured":"OpenAI. 2022. Introducing ChatGPT. Retrieved May 24 2023 from https:\/\/openai.com\/blog\/chatgpt"},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-350772-3.50009-4"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP46214.2022.9833571"},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP46215.2023.00001"},{"key":"e_1_3_1_83_2","unstructured":"Hammond Pearce Benjamin Tan Prashanth Krishnamurthy Farshad Khorrami Ramesh Karri and Brendan Dolan-Gavitt. 2022. Pop Quiz! Can a Large Language Model Help With Reverse Engineering?arXiv:2202.01142. Retrieved from https:\/\/arxiv.org\/abs\/2202.01142"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3576123.3576130"},{"key":"e_1_3_1_85_2","unstructured":"Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590. Retrieved from https:\/\/arxiv.org\/abs\/2302.06590"},{"key":"e_1_3_1_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406263"},{"key":"e_1_3_1_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3501385.3543970"},{"key":"e_1_3_1_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287374"},{"key":"e_1_3_1_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230981"},{"key":"e_1_3_1_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3105726.3106169"},{"key":"e_1_3_1_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587102.3588805"},{"issue":"3","key":"e_1_3_1_92_2","first-page":"12","article-title":"Overview: Self-efficacy in STEM","volume":"1","author":"Rittmayer Ashley D.","year":"2008","unstructured":"Ashley D. Rittmayer and Margaret E. Beier. 2008. Overview: Self-efficacy in STEM. SWE-AWE CASEE Overviews 1, 3 (2008), 12.","journal-title":"SWE-AWE CASEE Overviews"},{"key":"e_1_3_1_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317590"},{"key":"e_1_3_1_94_2","unstructured":"Ryan J. Salva. 2022. Preview: Referencing Public Code in GitHub Copilot | The GitHub Blog. Retrieved May 24 2023 from https:\/\/github.blog\/2022-11-01-preview-referencing-public-code-in-github-copilot\/"},{"key":"e_1_3_1_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3501385.3543957"},{"key":"e_1_3_1_96_2","doi-asserted-by":"crossref","unstructured":"Jaromir Savelka Arav Agarwal Christopher Bogart Yifan Song and Majd Sakr. 2023. Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?arXiv:2303.09325. Retrieved from https:\/\/arxiv.org\/abs\/2303.09325","DOI":"10.1145\/3587102.3588792"},{"key":"e_1_3_1_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/3024906.3024910"},{"key":"e_1_3_1_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/6592.6594"},{"key":"e_1_3_1_99_2","doi-asserted-by":"publisher","DOI":"10.1016\/0959-4752(94)90003-5"},{"key":"e_1_3_1_100_2","unstructured":"Tabnine. 2023. AI Assistant for Software Developers | Tabnine. Retrieved May 24 2023 from https:\/\/www.tabnine.com\/"},{"key":"e_1_3_1_101_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-11647-6_127"},{"key":"e_1_3_1_102_2","unstructured":"Ross Taylor Marcin Kardas Guillem Cucurull Thomas Scialom Anthony Hartshorn Elvis Saravia Andrew Poulton Viktor Kerkez and Robert Stojnic. 2022. Galactica: A Large Language Model for Science. arXiv:2211.09085. Retrieved from https:\/\/arxiv.org\/abs\/2211.09085"},{"key":"e_1_3_1_103_2","unstructured":"The Joseph Saveri Law Firm and Matthew Butterick. 2022. GitHub Copilot Litigation. Retrieved May 24 2023 from https:\/\/githubcopilotlitigation.com\/"},{"key":"e_1_3_1_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_3_1_105_2","doi-asserted-by":"publisher","DOI":"10.1145\/3514094.3534150"},{"key":"e_1_3_1_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300831"},{"key":"e_1_3_1_107_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569830"},{"key":"e_1_3_1_108_2","doi-asserted-by":"publisher","DOI":"10.1080\/03075079.2020.1730313"},{"key":"e_1_3_1_109_2","unstructured":"Wojciech Zaremba Greg Brockman and OpenAI. 2021. OpenAI Codex. Retrieved from https:\/\/openai.com\/blog\/openai-codex\/"},{"key":"e_1_3_1_110_2","unstructured":"Sarah Zhang Reece Shuttleworth Zad Chin Pedro Lantigua Saisamrit Surbehera Gregory Hunter Derek Austin Yann Hicke Leonard Tang Sathwik Karnik Darnell Granberry and Iddo Drori. 2022. Automatically Answering and Generating Machine Learning Final Exams. arXiv:2206.05442. Retrieved from https:\/\/arxiv.org\/abs\/2206.05442"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3617367","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3617367","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:45:54Z","timestamp":1750178754000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3617367"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,29]]},"references-count":109,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3617367"],"URL":"https:\/\/doi.org\/10.1145\/3617367","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,29]]},"assertion":[{"value":"2023-01-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}