{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,18]],"date-time":"2026-10-18T22:28:53Z","timestamp":1792362533573,"version":"4.4.0"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"EICS","license":[{"start":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T00:00:00Z","timestamp":1718582400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,6,17]]},"abstract":"<jats:p>Large language models like GPT and Codex drastically alter many daily tasks, including programming, where they can rapidly generate code from natural language or informal specifications. Thus, they will change what it means to be a programmer and how programmers act during software development. This work explores how AI assistance for code generation impacts productivity. In our user study (N=24), we asked programmers to complete Python programming tasks supported by a) an auto-complete interface using GitHub Copilot, b) a conversational system using GPT-3, and c) traditionally with just the web browser. Aside from significantly increasing productivity metrics, participants displayed distinctive usage patterns and strategies, highlighting that the form of presentation and interaction affects how users engage with these systems. Our findings emphasize the benefits of AI-assisted coding and highlight the different design challenges for these systems.<\/jats:p>","DOI":"10.1145\/3661145","type":"journal-article","created":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T18:25:37Z","timestamp":1718648737000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":69,"title":["Significant Productivity Gains through Programming with Large Language Models"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6894-605X","authenticated-orcid":false,"given":"Thomas","family":"Weber","sequence":"first","affiliation":[{"name":"LMU Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4184-7763","authenticated-orcid":false,"given":"Maximilian","family":"Brandmaier","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3890-1990","authenticated-orcid":false,"given":"Albrecht","family":"Schmidt","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5462-8782","authenticated-orcid":false,"given":"Sven","family":"Mayer","sequence":"additional","affiliation":[{"name":"LMU Munich, Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,6,17]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2017.31"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1546"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.29"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2206.15000"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3582083"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. https:\/\/doi.org\/10.48550\/ARXIV.2005.14165","DOI":"10.48550\/ARXIV.2005.14165"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1595696.1595728"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/358886.358895"},{"key":"e_1_2_1_9_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pond\u00e9 de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR abs\/2107.03374 (2021) 35. arXiv:2107.03374 https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","unstructured":"Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra Adam Roberts Paul Barham Hyung Won Chung Charles Sutton Sebastian Gehrmann Parker Schuh Kensen Shi Sasha Tsvyashchenko Joshua Maynez Abhishek Rao Parker Barnes Yi Tay Noam Shazeer Vinodkumar Prabhakaran Emily Reif Nan Du Ben Hutchinson Reiner Pope James Bradbury Jacob Austin Michael Isard Guy Gur-Ari Pengcheng Yin Toju Duke Anselm Levskaya Sanjay Ghemawat Sunipa Dev Henryk Michalewski Xavier Garcia Vedant Misra Kevin Robinson Liam Fedus Denny Zhou Daphne Ippolito David Luan Hyeontaek Lim Barret Zoph Alexander Spiridonov Ryan Sepassi David Dohan Shivani Agrawal Mark Omernick Andrew M. Dai Thanumalayan Sankaranarayana Pillai Marie Pellat Aitor Lewkowycz Erica Moreira Rewon Child Oleksandr Polozov Katherine Lee Zongwei Zhou Xuezhi Wang Brennan Saeta Mark Diaz Orhan Firat Michele Catasta Jason Wei Kathy Meier-Hellstern Douglas Eck Jeff Dean Slav Petrov and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways. https:\/\/doi.org\/10.48550\/ARXIV.2204.02311","DOI":"10.48550\/ARXIV.2204.02311"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","unstructured":"Colin B. Clement Dawn Drain Jonathan Timcheck Alexey Svyatkovskiy and Neel Sundaresan. 2020. PyMT5: multi-mode translation of natural language and Python code with transformers. https:\/\/doi.org\/10.18653\/v1\/2020.emnlpmain.728 arXiv:2010.03150 [cs.LG]","DOI":"10.18653\/v1\/2020.emnlpmain.728"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.04.008"},{"key":"e_1_2_1_14_1","unstructured":"Margaret Efron. 2023. Curious about the new ChatGPT Code Interpreter plug-in? Here's how to get started. https:\/\/medium.com\/learning-data\/curious-about-the-new-chatgpt-code-interpreter-plug-in-heres-how-to-get-started-541fd998c017"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. https:\/\/doi.org\/10.48550\/ARXIV.2002.08155","DOI":"10.48550\/ARXIV.2002.08155"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3205873.3205875"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3454122.3454124"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-322-80058-9_19"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370024.2010.500139"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1142\/s0218194013500125"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3486607.3486749"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062455.1062491"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/p18-1031"},{"key":"e_1_2_1_24_1","unstructured":"Kevin Hurler. 2023. Stack Overflow Traffic Drops as Coders Opt for ChatGPT Help Instead. https:\/\/gizmodo.com\/stack-overflow-traffic-drops-as-coders-opt-for-chatgpt-1850427794"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501870"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196474"},{"key":"e_1_2_1_27_1","volume-title":"Research: Quantifying github copilot's impact on developer productivity and happiness.","author":"Kalliamvakou Eirini","year":"2022","unstructured":"Eirini Kalliamvakou. 2022. Research: Quantifying github copilot's impact on developer productivity and happiness. (2022). https:\/\/github.blog\/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness\/"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1181775.1181777"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_2_1_30_1","volume-title":"Yuyao Wang, and Lingming Zhang.","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE]"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635892"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","unstructured":"Arghavan Moradi Dakhel Vahid Majdinasab Amin Nikanjam Foutse Khomh Michel C. Desmarais and Zhen Ming (Jack) Jiang. 2023. GitHub Copilot AI pair programmer: Asset or Liability? 111734 pages. https:\/\/doi.org\/10.1016\/j.jss.2023.111734 arXiv:2206.15331 [cs.SE]","DOI":"10.1016\/j.jss.2023.111734"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","unstructured":"Hussein Mozannar Gagan Bansal Adam Fourney and Eric Horvitz. 2022. Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. https:\/\/doi.org\/10.48550\/ARXIV.2210.14306","DOI":"10.48550\/ARXIV.2210.14306"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2006.105"},{"key":"e_1_2_1_35_1","unstructured":"Adel Nehme. 2023. How to Use ChatGPT Code Interpreter. https:\/\/www.datacamp.com\/tutorial\/how-to-use-chat-gpt- code-interpreter"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09820-z"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.1992.242525"},{"key":"e_1_2_1_38_1","unstructured":"Paul Oman Jack Hagemeister and Dan Ash. 1992. A definition and taxonomy for software maintainability. Technical Report. Software Engineering Test Laboratory University of Idaho. 91--08 pages."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","unstructured":"Hammond Pearce Benjamin Tan Baleegh Ahmad Ramesh Karri and Brendan Dolan-Gavitt. 2021. Examining Zero-Shot Vulnerability Repair with Large Language Models. https:\/\/doi.org\/10.48550\/ARXIV.2112.02125","DOI":"10.48550\/ARXIV.2112.02125"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","unstructured":"Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. https:\/\/doi.org\/10.48550\/ARXIV.2302.06590","DOI":"10.48550\/ARXIV.2302.06590"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2010.12.001"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","unstructured":"Julian Aron Prenner and Romain Robbes. 2021. Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs. https:\/\/doi.org\/10.48550\/ARXIV.2111.03922","DOI":"10.48550\/ARXIV.2111.03922"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2744200"},{"key":"e_1_2_1_44_1","unstructured":"Python-World. 2022. Python-Mini-Projects. https:\/\/github.com\/Python-World\/python-mini-projects"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","unstructured":"Basit Qureshi. 2023. Exploring the Use of ChatGPT as a Tool for Learning and Assessment in Undergraduate Computer Science Curriculum: Opportunities and Challenges. https:\/\/doi.org\/10.48550\/ARXIV.2304.11214","DOI":"10.48550\/ARXIV.2304.11214"},{"key":"e_1_2_1_46_1","unstructured":"Alec Radford Karthik Narasimhan Tim Salimans and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical Report. OpenAI."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594321"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2008.42"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-010-0064-x"},{"key":"e_1_2_1_50_1","volume-title":"Research: Quantifying GitHub Copilot's impact on code quality.","author":"Rodriguez Mario","year":"2023","unstructured":"Mario Rodriguez. 2023. Research: Quantifying GitHub Copilot's impact on code quality. (2023). https:\/\/github.blog\/2023-10-10- research-quantifying-github-copilots-impact-on-code-quality\/"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3581641.3584037"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","unstructured":"Gustavo Sandoval Hammond Pearce Teo Nys Ramesh Karri Siddharth Garg and Brendan Dolan-Gavitt. 2022. Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants. https:\/\/doi.org\/10.48550\/ARXIV.2208.09727","DOI":"10.48550\/ARXIV.2208.09727"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2208.06213"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","unstructured":"Dominik Sobania Martin Briesch Carol Hanna and Justyna Petke. 2023. An Analysis of the Automatic Bug Fixing Performance of ChatGPT. https:\/\/doi.org\/10.48550\/ARXIV.2301.08653","DOI":"10.48550\/ARXIV.2301.08653"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512290.3528700"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2019.2944354"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330699"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.5220\/0009417801810192"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2009.05617"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. https:\/\/doi.org\/10.48550\/ARXIV.1706.03762","DOI":"10.48550\/ARXIV.1706.03762"},{"key":"e_1_2_1_62_1","unstructured":"w3resource. 2022. Python Projects. https:\/\/www.w3resource.com\/projects\/python\/index.php"},{"key":"e_1_2_1_63_1","first-page":"18","article-title":"The software maintainability index revisited","volume":"14","author":"Welker Kurt D","year":"2001","unstructured":"Kurt D Welker. 2001. The software maintainability index revisited. CrossTalk 14 (2001), 18--21.","journal-title":"CrossTalk"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1118178.1118215"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9634-5"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3487569"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3558489.3559072"},{"key":"e_1_2_1_68_1","unstructured":"Pengcheng Yin Wen-Ding Li Kefan Xiao Abhishek Rao Yeming Wen Kensen Shi Joshua Howland Paige Bailey Michele Catasta Henryk Michalewski Alex Polozov and Charles Sutton. 2022. Natural Language to Code Generation in Interactive Data Science Notebooks. arXiv:2212.09248 [cs.CL]"},{"key":"e_1_2_1_69_1","unstructured":"Shuyin Zhao. 2023. GitHub Copilot Chat beta now available for all individuals. (2023). https:\/\/github.blog\/2023-09-20-github-copilot-chat-beta-now-available-for-all-individuals\/"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534864"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3661145","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3661145","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T23:20:42Z","timestamp":1755904842000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3661145"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,17]]},"references-count":70,"journal-issue":{"issue":"EICS","published-print":{"date-parts":[[2024,6,17]]}},"alternative-id":["10.1145\/3661145"],"URL":"https:\/\/doi.org\/10.1145\/3661145","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,17]]},"assertion":[{"value":"2024-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}