{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T04:04:20Z","timestamp":1771905860530,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1011511","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2023,10,10]],"date-time":"2023-10-10T00:00:00Z","timestamp":1696896000000}}],"reference-count":74,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Computer programming is a fundamental tool for life scientists, allowing them to carry out essential research tasks. However, despite various educational efforts, learning to write code can be a challenging endeavor for students and researchers in life-sciences disciplines. Recent advances in artificial intelligence have made it possible to translate human-language prompts to functional code, raising questions about whether these technologies can aid (or replace) life scientists\u2019 efforts to write code. Using 184 programming exercises from an introductory-bioinformatics course, we evaluated the extent to which one such tool\u2014OpenAI\u2019s ChatGPT\u2014could successfully complete programming tasks. ChatGPT solved 139 (75.5%) of the exercises on its first attempt. For the remaining exercises, we provided natural-language feedback to the model, prompting it to try different approaches. Within 7 or fewer attempts, ChatGPT solved 179 (97.3%) of the exercises. These findings have implications for life-sciences education and research. Instructors may need to adapt their pedagogical approaches and assessment techniques to account for these new capabilities that are available to the general public. For some programming tasks, researchers may be able to work in collaboration with machine-learning models to produce functional code.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1011511","type":"journal-article","created":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T17:43:40Z","timestamp":1695923020000},"page":"e1011511","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":42,"title":["Evaluating a large language model\u2019s ability to solve programming exercises from an introductory bioinformatics course"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2001-5640","authenticated-orcid":true,"given":"Stephen R.","family":"Piccolo","sequence":"first","affiliation":[]},{"given":"Paul","family":"Denny","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8269-2909","authenticated-orcid":true,"given":"Andrew","family":"Luxton-Reilly","sequence":"additional","affiliation":[]},{"given":"Samuel H.","family":"Payne","sequence":"additional","affiliation":[]},{"given":"Perry G.","family":"Ridge","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2023,9,28]]},"reference":[{"issue":"1","key":"pcbi.1011511.ref001","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/bioinformatics\/1.1.1","volume":"1","author":"RJ Beynon","year":"1985","journal-title":"CABIOS editorial. Bioinformatics"},{"issue":"10","key":"pcbi.1011511.ref002","doi-asserted-by":"crossref","first-page":"e1005755","DOI":"10.1371\/journal.pcbi.1005755","article-title":"Unmet needs for analyzing biological big data: A survey of 704 NSF principal investigators","volume":"13","author":"L Barone","year":"2017","journal-title":"PLOS Computational Biology"},{"issue":"4","key":"pcbi.1011511.ref003","first-page":"6","article-title":"Teaching computing for everyone","volume":"21","author":"M. Guzdial","year":"2006","journal-title":"J Comput Sci Coll"},{"issue":"7638","key":"pcbi.1011511.ref004","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1038\/nj7638-563a","article-title":"Scientific computing: Code alert","volume":"541","author":"M. Baker","year":"2017","journal-title":"Nature"},{"key":"pcbi.1011511.ref005","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1007\/978-3-031-02216-6_3","volume-title":"Learner-centered design of computing education: Research on computing for everyone","author":"M. Guzdial","year":"2016"},{"issue":"4","key":"pcbi.1011511.ref006","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1093\/bib\/3.4.353","article-title":"Bioinformatics training in the USA","volume":"3","author":"MM Zatz","year":"2002","journal-title":"Brief Bioinform"},{"issue":"6","key":"pcbi.1011511.ref007","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1093\/bib\/bbq027","article-title":"Bioinformatics education in india","volume":"11","author":"U Kulkarni-Kale","year":"2010","journal-title":"Brief Bioinform"},{"issue":"3","key":"pcbi.1011511.ref008","doi-asserted-by":"crossref","first-page":"e1003496","DOI":"10.1371\/journal.pcbi.1003496","article-title":"Bioinformatics Curriculum Guidelines: Toward a Definition of Core Competencies","volume":"10","author":"L Welch","year":"2014","journal-title":"PLOS Computational Biology"},{"issue":"1","key":"pcbi.1011511.ref009","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1111\/nyas.13207","article-title":"A vision for collaborative training infrastructure for bioinformatics","volume":"1387","author":"JJ Williams","year":"2017","journal-title":"Ann N Y Acad Sci"},{"issue":"2","key":"pcbi.1011511.ref010","doi-asserted-by":"crossref","first-page":"e1005772","DOI":"10.1371\/journal.pcbi.1005772","article-title":"The development and application of bioinformatics core competencies to improve bioinformatics training and education","volume":"14","author":"N Mulder","year":"2018","journal-title":"PLOS Computational Biology"},{"key":"pcbi.1011511.ref011","first-page":"10","article-title":"Expanding Research Capacity in Sub-Saharan Africa Through Informatics, Bioinformatics, and Data Science Training Programs in Mali","author":"JG Shaffer","year":"2019","journal-title":"Front Genet"},{"issue":"2","key":"pcbi.1011511.ref012","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1093\/bib\/bbx100","article-title":"A global perspective on evolving bioinformatics and data science training needs","volume":"20","author":"TK Attwood","year":"2019","journal-title":"Brief Bioinform"},{"issue":"6","key":"pcbi.1011511.ref013","doi-asserted-by":"crossref","first-page":"e0196878","DOI":"10.1371\/journal.pone.0196878","article-title":"Bioinformatics core competencies for undergraduate life sciences education","volume":"13","author":"MAW Sayres","year":"2018","journal-title":"PLOS ONE"},{"key":"pcbi.1011511.ref014","first-page":"1","article-title":"A survey of the practice of computational science. In: State Pract Rep. New York, NY","author":"P Prabhu","year":"2011","journal-title":"USA: Association for Computing Machinery"},{"issue":"6","key":"pcbi.1011511.ref015","doi-asserted-by":"crossref","first-page":"e1004867","DOI":"10.1371\/journal.pcbi.1004867","article-title":"An Introduction to Programming for Bioscientists: A Python-Based Primer","volume":"12","author":"B Ekmekci","year":"2016","journal-title":"PLOS Computational Biology"},{"issue":"10","key":"pcbi.1011511.ref016","doi-asserted-by":"crossref","DOI":"10.18637\/jss.v059.i10","article-title":"Tidy Data","volume":"59","author":"H. Wickham","year":"2014","journal-title":"J Stat Softw"},{"key":"pcbi.1011511.ref017","doi-asserted-by":"crossref","DOI":"10.1002\/0471448354","volume-title":"Exploratory data mining and data cleaning","author":"T Dasu","year":"2003"},{"key":"pcbi.1011511.ref018","first-page":"852","volume-title":"Proc 49th ACM Tech Symp Comput Sci Educ","author":"Simon","year":"2018"},{"issue":"1","key":"pcbi.1011511.ref019","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1186\/1471-2105-9-82","article-title":"A comparison of common programming languages used in bioinformatics","volume":"9","author":"M Fourment","year":"2008","journal-title":"BMC Bioinformatics"},{"issue":"11","key":"pcbi.1011511.ref020","doi-asserted-by":"crossref","first-page":"1422","DOI":"10.1093\/bioinformatics\/btp163","article-title":"Biopython: Freely available Python tools for computational molecular biology and bioinformatics","volume":"25","author":"PJ Cock","year":"2009","journal-title":"Bioinformatics"},{"key":"pcbi.1011511.ref021","first-page":"6","article-title":"Data Structures for Statistical Computing in Python","author":"W. McKinney","year":"2010","journal-title":"Proc 9th Python Sci Conf."},{"issue":"2","key":"pcbi.1011511.ref022","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/MCSE.2011.37","article-title":"The NumPy Array: A Structure for Efficient Numerical Computation","volume":"13","author":"S van der Walt","year":"2011","journal-title":"Computing in Science & Engineering"},{"key":"pcbi.1011511.ref023","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"pcbi.1011511.ref024","volume-title":"Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems","author":"A. G\u00e9ron","year":"2017","edition":"1"},{"issue":"1","key":"pcbi.1011511.ref025","doi-asserted-by":"crossref","first-page":"37","DOI":"10.2190\/GUJT-JCBJ-Q6QU-Q9PL","article-title":"Conditions of learning in novice programmers","volume":"2","author":"DN Perkins","year":"1986","journal-title":"J Educ Comput Res"},{"issue":"2","key":"pcbi.1011511.ref026","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1145\/1089733.1089734","article-title":"Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers","volume":"37","author":"C Kelleher","year":"2005","journal-title":"ACM Comput Surv"},{"issue":"3","key":"pcbi.1011511.ref027","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1145\/1151954.1067453","article-title":"A study of the difficulties of novice programmers","volume":"37","author":"E Lahtinen","year":"2005","journal-title":"SIGCSE Bull"},{"key":"pcbi.1011511.ref028","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1145\/3293881.3295779","article-title":"Introductory programming: A systematic literature review","author":"A Luxton-Reilly","year":"2018","journal-title":"Proc Companion 23rd Annu ACM Conf Innov Technol Comput Sci Educ"},{"key":"pcbi.1011511.ref029","doi-asserted-by":"crossref","first-page":"538","DOI":"10.1145\/3287324.3287394","volume-title":"Proc 50th ACM Tech Symp Comput Sci Educ","author":"R Smith","year":"2019"},{"key":"pcbi.1011511.ref030","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1145\/3344429.3372508","volume-title":"Proc Work Group Rep Innov Technol Comput Sci Educ","author":"BA Becker","year":"2019"},{"key":"pcbi.1011511.ref031","first-page":"10","volume-title":"Proc 24th Australas Comput Educ Conf","author":"J Finnie-Ansley","year":"2022"},{"key":"pcbi.1011511.ref032","first-page":"97","volume-title":"Proc 25th Australas Comput Educ Conf","author":"J Finnie-Ansley","year":"2023"},{"key":"pcbi.1011511.ref033","article-title":"Evaluating Large Language Models Trained on Code","author":"M Chen","year":"2021","journal-title":"arXiv"},{"issue":"6624","key":"pcbi.1011511.ref034","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1126\/science.abq1158","article-title":"Competition-level code generation with AlphaCode","volume":"378","author":"Y Li","year":"2022","journal-title":"Science"},{"key":"pcbi.1011511.ref035","unstructured":"ChatGPT: Optimizing Language Models for Dialogue. OpenAI. [Cited 2023 September 19]. Available from https:\/\/openai.com\/blog\/chatgpt."},{"issue":"6632","key":"pcbi.1011511.ref036","doi-asserted-by":"crossref","first-page":"548","DOI":"10.1126\/science.adg4246","article-title":"Understanding the limits of AI coding","volume":"379","author":"J. Hendler","year":"2023","journal-title":"Science"},{"issue":"7947","key":"pcbi.1011511.ref037","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1038\/d41586-023-00288-7","article-title":"ChatGPT: Five priorities for research","volume":"614","author":"EAM van Dis","year":"2023","journal-title":"Nature"},{"issue":"6630","key":"pcbi.1011511.ref038","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1126\/science.adg7879","article-title":"ChatGPT is fun, but not an author","volume":"379","author":"HH Thorp","year":"2023","journal-title":"Science"},{"issue":"2","key":"pcbi.1011511.ref039","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"TH Kung","year":"2023","journal-title":"PLOS Digital Health"},{"key":"pcbi.1011511.ref040","article-title":"Is ChatGPT A Good Translator?","author":"W Jiao","year":"2023","journal-title":"A Preliminary Study"},{"issue":"0","key":"pcbi.1011511.ref041","article-title":"Generating scholarly content with ChatGPT: Ethical challenges for medical publishing","volume":"0","author":"M Liebrenz","year":"2023","journal-title":"The Lancet Digital Health"},{"key":"pcbi.1011511.ref042","article-title":"ChatGPT and Software Testing Education: Promises & Perils","author":"S Jalil","year":"2023","journal-title":"arXiv"},{"key":"pcbi.1011511.ref043","unstructured":"Elias J. Google is asking employees to test potential ChatGPT competitors, including a chatbot called \u2018Apprentice Bard\u2019. CNBC. https:\/\/www.cnbc.com\/2023\/01\/31\/google-testing-chatgpt-like-chatbot-apprentice-bard-with-employees.html; 2023."},{"key":"pcbi.1011511.ref044","article-title":"ChatGPT sets record for fastest-growing user base\u2014analyst note","author":"K. Hu","year":"2023"},{"key":"pcbi.1011511.ref045","author":"V. Arel-Bundock","year":"2023","journal-title":"Rdatasets: A collection of datasets originally distributed in various R packages"},{"issue":"3","key":"pcbi.1011511.ref046","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1214\/ss\/1032280214","article-title":"Bootstrap confidence intervals","volume":"11","author":"TJ DiCiccio","year":"1996","journal-title":"Stat Sci"},{"issue":"6202","key":"pcbi.1011511.ref047","first-page":"1369","article-title":"Genomic surveillance elucidates Ebola virus origin and transmission during the 2014","volume":"345","author":"SK Gire","year":"2014","journal-title":"outbreak. science"},{"issue":"D1","key":"pcbi.1011511.ref048","doi-asserted-by":"crossref","first-page":"D506","DOI":"10.1093\/nar\/gky1049","article-title":"UniProt: A worldwide hub of protein knowledge","volume":"47","author":"U. Consortium","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"pcbi.1011511.ref049","unstructured":"Random Name Generator Generated full names. Random Lists. Available from https:\/\/www.randomlists.com\/random-names."},{"issue":"10","key":"pcbi.1011511.ref050","doi-asserted-by":"crossref","first-page":"1108","DOI":"10.1158\/2159-8290.CD-13-0219","article-title":"Connecting genomic alterations to cancer biology with proteomics: The NCI Clinical Proteomic Tumor Analysis Consortium.","volume":"3","author":"MJ Ellis","year":"2013","journal-title":"Cancer Discov"},{"issue":"6","key":"pcbi.1011511.ref051","doi-asserted-by":"crossref","first-page":"2707","DOI":"10.1021\/pr501254j","article-title":"The CPTAC data portal: A resource for cancer proteomics research","volume":"14","author":"NJ Edwards","year":"2015","journal-title":"J Proteome Res"},{"issue":"4","key":"pcbi.1011511.ref052","doi-asserted-by":"crossref","first-page":"1902","DOI":"10.1021\/acs.jproteome.0c00919","article-title":"Simplified and unified access to cancer proteogenomic data","volume":"20","author":"CM Lindgren","year":"2021","journal-title":"J Proteome Res"},{"issue":"11","key":"pcbi.1011511.ref053","doi-asserted-by":"crossref","first-page":"1402","DOI":"10.1164\/rccm.201310-1917OC","article-title":"The mitochondrial cardiolipin remodeling enzyme lysocardiolipin acyltransferase is a novel target in pulmonary fibrosis","volume":"189","author":"LS Huang","year":"2014","journal-title":"Am J Respir Crit Care Med"},{"issue":"D1","key":"pcbi.1011511.ref054","doi-asserted-by":"crossref","first-page":"D991","DOI":"10.1093\/nar\/gks1193","article-title":"NCBI GEO: Archive for functional genomics data setsupdate","volume":"41","author":"T Barrett","year":"2012","journal-title":"Nucleic Acids Research"},{"key":"pcbi.1011511.ref055","unstructured":"White E. Programming for Biologists. Available from http:\/\/www.programmingforbiologists.org."},{"key":"pcbi.1011511.ref056","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/978-3-030-25943-3_21","article-title":"Working with excel files.","author":"J Hunt","year":"2019","journal-title":"Adv Guide Python 3 Program."},{"key":"pcbi.1011511.ref057","volume-title":"R: A language and environment for statistical computing.","author":"R Core Team","year":"2020"},{"issue":"43","key":"pcbi.1011511.ref058","doi-asserted-by":"crossref","first-page":"1686","DOI":"10.21105\/joss.01686","article-title":"Welcome to the tidyverse.","volume":"4","author":"H Wickham","year":"2019","journal-title":"J Open Source Softw"},{"key":"pcbi.1011511.ref059","article-title":"How Nature readers are using ChatGPT","author":"B. Owens","year":"2023","journal-title":"Nature"},{"key":"pcbi.1011511.ref060","article-title":"Program Synthesis with Large Language Models","author":"J Austin","year":"2021","journal-title":"arXiv"},{"issue":"4","key":"pcbi.1011511.ref061","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1145\/321186.321192","article-title":"Experiments with a Heuristic Compiler.","volume":"10","author":"HA Simon","year":"1963","journal-title":"J ACM"},{"issue":"3","key":"pcbi.1011511.ref062","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1145\/362566.362568","article-title":"Toward automatic program synthesis.","volume":"14","author":"Z Manna","year":"1971","journal-title":"Commun ACM."},{"key":"pcbi.1011511.ref063","article-title":"Codebert: A pre-trained model for programming and natural languages.","author":"Z Feng","year":"2020","journal-title":"ArXiv Prepr ArXiv200208155"},{"key":"pcbi.1011511.ref064","article-title":"PyMT5: Multi-mode translation of natural language and Python code with transformers","author":"CB Clement","year":"2020","journal-title":"ArXiv Prepr ArXiv201003150"},{"key":"pcbi.1011511.ref065","first-page":"1136","volume-title":"Proc 54th ACM Tech Symp Comput Sci Educ V 1","author":"P Denny","year":"2023"},{"issue":"2","key":"pcbi.1011511.ref066","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1145\/3555367","article-title":"The Premature Obituary of Programming.","volume":"66","author":"DM Yellin","year":"2023","journal-title":"Commun ACM."},{"key":"pcbi.1011511.ref067","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1017\/9781108654555.010","article-title":"Cognitive sciences for computing education.","author":"AV Robins","year":"2019","journal-title":"Camb Handb Comput Educ Res."},{"key":"pcbi.1011511.ref068","first-page":"4","volume-title":"Proc 23rd Annu ACM Conf Innov Technol Comput Sci Educ","author":"P Denny","year":"2018"},{"issue":"3","key":"pcbi.1011511.ref069","doi-asserted-by":"crossref","first-page":"698","DOI":"10.1002\/cae.22109","article-title":"Experiences from placing Stack Overflow at the core of an intermediate programming course.","volume":"27","author":"M L\u00f3pez-Nores","year":"2019","journal-title":"Comput Appl Eng Educ"},{"key":"pcbi.1011511.ref070","first-page":"49","volume-title":"In: Proc 25th Australas Comput Educ Conf.","author":"T Lehtinen","year":"2023"},{"key":"pcbi.1011511.ref071","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1145\/3160489.3160492","article-title":"Intelligent tutoring systems for programming education: A systematic review.","author":"T Crow","year":"2018","journal-title":"Proc 20th Australas Comput Educ Conf"},{"key":"pcbi.1011511.ref072","first-page":"931","volume-title":"Proc 54th ACM Tech Symp Comput Sci Educ V 1","author":"S MacNeil","year":"2023"},{"key":"pcbi.1011511.ref073","first-page":"27","volume-title":"Proc 2022 ACM Conf Int Comput Educ Res\u2014","author":"S Sarsa","year":"2022"},{"issue":"1","key":"pcbi.1011511.ref074","doi-asserted-by":"crossref","first-page":"153","DOI":"10.3102\/0034654307313795","article-title":"Focus on formative feedback.","volume":"78","author":"VJ Shute","year":"2008","journal-title":"Rev Educ Res"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1011511","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2023,10,10]],"date-time":"2023-10-10T00:00:00Z","timestamp":1696896000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011511","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,10]],"date-time":"2023-10-10T17:48:43Z","timestamp":1696960123000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1011511"}},"subtitle":[],"editor":[{"given":"Francis","family":"Ouellette","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"references-count":74,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,9,28]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1011511","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1011511","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,28]]}}}