{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T00:31:05Z","timestamp":1793147465767,"version":"4.5.0"},"publisher-location":"New York, NY, USA","reference-count":32,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,5,23]],"date-time":"2022-05-23T00:00:00Z","timestamp":1653264000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Canada Research Chairs Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,5,23]]},"DOI":"10.1145\/3524842.3528470","type":"proceedings-article","created":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T20:08:36Z","timestamp":1666037316000},"page":"1-5","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":286,"title":["An empirical evaluation of GitHub copilot's code suggestions"],"prefix":"10.1145","author":[{"given":"Nhan","family":"Nguyen","sequence":"first","affiliation":[{"name":"University of Alberta, Edmonton, AB, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Nadi","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, AB, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,10,17]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Romaana Aamir. 2021. GitHub copilot-bright future or an impending doom. https:\/\/code.likeagirl.io\/github-copilot-bright-future-or-an-impending-doom-df0f1674a50c  Romaana Aamir. 2021. GitHub copilot-bright future or an impending doom. https:\/\/code.likeagirl.io\/github-copilot-bright-future-or-an-impending-doom-df0f1674a50c"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3212695"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290353"},{"key":"e_1_3_2_1_4_1","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL]  Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL]"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3340458"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194164.3194186"},{"key":"e_1_3_2_1_7_1","unstructured":"G. Ann Campbell. 2021. Cognitive complexity - A new way of measuring understandability. https:\/\/www.sonarsource.com\/docs\/CognitiveComplexity.pdf  G. Ann Campbell. 2021. Cognitive complexity - A new way of measuring understandability. https:\/\/www.sonarsource.com\/docs\/CognitiveComplexity.pdf"},{"key":"e_1_3_2_1_8_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG]  Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG]"},{"key":"e_1_3_2_1_9_1","volume-title":"Readability and Understandability Scores for Snippet Assessment: an Exploratory Study. CoRR abs\/2108.09181","author":"de Carvalho Dantas Carlos Eduardo","year":"2021","unstructured":"Carlos Eduardo de Carvalho Dantas and Marcelo de Almeida Maia . 2021. Readability and Understandability Scores for Snippet Assessment: an Exploratory Study. CoRR abs\/2108.09181 ( 2021 ). arXiv:2108.09181 https:\/\/arxiv.org\/abs\/2108.09181 Carlos Eduardo de Carvalho Dantas and Marcelo de Almeida Maia. 2021. Readability and Understandability Scores for Snippet Assessment: an Exploratory Study. CoRR abs\/2108.09181 (2021). arXiv:2108.09181 https:\/\/arxiv.org\/abs\/2108.09181"},{"key":"e_1_3_2_1_10_1","unstructured":"fabasoad and Sachin131. 2016. Is there public API endpoints available for leet-code? https:\/\/leetcode.com\/discuss\/general-discussion\/1297705\/is-there-public-api-endpoints-available-for-leetcode  fabasoad and Sachin131. 2016. Is there public API endpoints available for leet-code? https:\/\/leetcode.com\/discuss\/general-discussion\/1297705\/is-there-public-api-endpoints-available-for-leetcode"},{"key":"e_1_3_2_1_11_1","unstructured":"GitHub. 2021. GitHub Copilot \u00b7 Your AI pair programmer. https:\/\/copilot.github.com\/  GitHub. 2021. GitHub Copilot \u00b7 Your AI pair programmer. https:\/\/copilot.github.com\/"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180167"},{"key":"e_1_3_2_1_13_1","unstructured":"HackerRank. [n.d.]. HackerRank for Work API. https:\/\/www.hackerrank.com\/work\/apidocs#  HackerRank. [n.d.]. HackerRank for Work API. https:\/\/www.hackerrank.com\/work\/apidocs#"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227135"},{"key":"e_1_3_2_1_15_1","unstructured":"LeetCode. [n.d.]. Integer Break. https:\/\/leetcode.com\/problems\/integer-break\/  LeetCode. [n.d.]. Integer Break. https:\/\/leetcode.com\/problems\/integer-break\/"},{"key":"e_1_3_2_1_16_1","unstructured":"LeetCode. [n.d.]. Longest Increasing Path in a Matrix. https:\/\/leetcode.com\/problems\/longest-increasing-path-in-a-matrix  LeetCode. [n.d.]. Longest Increasing Path in a Matrix. https:\/\/leetcode.com\/problems\/longest-increasing-path-in-a-matrix"},{"key":"e_1_3_2_1_17_1","unstructured":"LeetCode. 2019. Start your coding practice -. https:\/\/support.leetcode.com\/hc\/enus\/articles\/360012016874-Start-your-Coding-Practice  LeetCode. 2019. Start your coding practice -. https:\/\/support.leetcode.com\/hc\/enus\/articles\/360012016874-Start-your-Coding-Practice"},{"key":"e_1_3_2_1_18_1","unstructured":"LeetCode. 2021. The world's leading online programming learning platform. https:\/\/leetcode.com\/  LeetCode. 2021. The world's leading online programming learning platform. https:\/\/leetcode.com\/"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/578"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360578"},{"key":"e_1_3_2_1_21_1","unstructured":"Matthew MacDonald. 2021. GitHub copilot: Fatally flawed or the future of software development? https:\/\/medium.com\/young-coder\/github-copilot-fatally-flawed-or-the-future-of-software-development-390c30afbc97  Matthew MacDonald. 2021. GitHub copilot: Fatally flawed or the future of software development? https:\/\/medium.com\/young-coder\/github-copilot-fatally-flawed-or-the-future-of-software-development-390c30afbc97"},{"key":"e_1_3_2_1_22_1","unstructured":"Gerald M\u00fccke and G Ann Campbell. 2021. How to use cognitive complexity? https:\/\/community.sonarsource.com\/t\/how-to-use-cognitive-complexity\/1894\/7  Gerald M\u00fccke and G Ann Campbell. 2021. How to use cognitive complexity? https:\/\/community.sonarsource.com\/t\/how-to-use-cognitive-complexity\/1894\/7"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.18515141"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. arXiv:2108.09293 [cs.CR]  Hammond Pearce Baleegh Ahmad Benjamin Tan Brendan Dolan-Gavitt and Ramesh Karri. 2021. Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. arXiv:2108.09293 [cs.CR]","DOI":"10.1109\/SP46214.2022.9833571"},{"key":"e_1_3_2_1_25_1","volume-title":"Recommendation systems for software engineering","author":"Robillard Martin","year":"2009","unstructured":"Martin Robillard , Robert Walker , and Thomas Zimmermann . 2009. Recommendation systems for software engineering . IEEE software 27, 4 ( 2009 ), 80--86. Martin Robillard, Robert Walker, and Thomas Zimmermann. 2009. Recommendation systems for software engineering. IEEE software 27, 4 (2009), 80--86."},{"key":"e_1_3_2_1_26_1","unstructured":"Swapnil Rustagi and Jagga Jasoos. 2019. Access to CodeChef API. https:\/\/discuss.codechef.com\/t\/access-to-codechef-api\/27308  Swapnil Rustagi and Jagga Jasoos. 2019. Access to CodeChef API. https:\/\/discuss.codechef.com\/t\/access-to-codechef-api\/27308"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"crossref","unstructured":"Dominik Sobania Martin Briesch and Franz Rothlauf. 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv:2111.07875 [cs.SE]  Dominik Sobania Martin Briesch and Franz Rothlauf. 2021. Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming. arXiv:2111.07875 [cs.SE]","DOI":"10.1145\/3512290.3528700"},{"key":"e_1_3_2_1_28_1","unstructured":"SonarQube. 2021. Code quality and code security. https:\/\/www.sonarqube.org\/  SonarQube. 2021. Code quality and code security. https:\/\/www.sonarqube.org\/"},{"key":"e_1_3_2_1_29_1","unstructured":"SonarQube. 2021. Metric definitions. https:\/\/docs.sonarqube.org\/latest\/user-guide\/metric-definitions\/  SonarQube. 2021. Metric definitions. https:\/\/docs.sonarqube.org\/latest\/user-guide\/metric-definitions\/"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300864"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313632"},{"key":"e_1_3_2_1_32_1","unstructured":"Qihao Zhu and Wenjie Zhang. 2021. Code Generation Based on Deep Learning: a Brief Review. arXiv:2106.08253 [cs.SE]  Qihao Zhu and Wenjie Zhang. 2021. Code Generation Based on Deep Learning: a Brief Review. arXiv:2106.08253 [cs.SE]"}],"event":{"name":"MSR '22: 19th International Conference on Mining Software Repositories","location":"Pittsburgh Pennsylvania","acronym":"MSR '22","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the 19th International Conference on Mining Software Repositories"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3524842.3528470","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3524842.3528470","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T14:09:35Z","timestamp":1750169375000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3524842.3528470"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,23]]},"references-count":32,"alternative-id":["10.1145\/3524842.3528470","10.1145\/3524842"],"URL":"https:\/\/doi.org\/10.1145\/3524842.3528470","relation":{},"subject":[],"published":{"date-parts":[[2022,5,23]]},"assertion":[{"value":"2022-10-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}