{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T22:13:48Z","timestamp":1768774428697,"version":"3.49.0"},"reference-count":20,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T00:00:00Z","timestamp":1730937600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Introduction: With the growing development and adoption of artificial intelligence in healthcare and across other sectors of society, various user-friendly and engaging tools to support research have emerged, such as chatbots, notably ChatGPT. Objective: To investigate the performance of ChatGPT as an assistant to medical coders using the ICD-10-CM\/PCS. Methodology: We conducted a prospective exploratory study between 2023 and 2024 over 6 months. A total of 150 clinical cases coded using the ICD-10-CM\/PCS, extracted from technical coding books, were systematically randomized. All cases were translated into Portuguese (the native language of the authors) and English (the native language of the ICD-10-CM\/PCS). These clinical cases varied in complexity levels regarding the quantity of diagnoses and procedures, as well as the nature of the clinical information. Each case was input into the 2023 ChatGPT free version. The coding obtained from ChatGPT was analyzed by a senior medical auditor\/coder and compared with the expected results. Results: Regarding the correct codes, ChatGPT\u2019s performance was higher by approximately 29 percentage points between diagnoses and procedures, with greater proficiency in diagnostic codes. The accuracy rate for codes was similar across languages, with rates of 31.0% and 31.9%. The error rate in procedure codes was substantially higher than that in diagnostic codes by almost four times. For missing information, a higher incidence was observed in diagnoses compared to procedures of slightly more than double the comparative rates. Additionally, there was a statistically significant excess of codes not related to clinical information, which was higher in procedures and nearly the same value in both languages under study. Conclusion: Given the ease of access to these tools, this investigation serves as an awareness factor, demonstrating that ChatGPT can assist the medical coder in directed research. However, it does not replace their technical validation in this process. Therefore, further developments of this tool are necessary to increase the quality and reliability of the results.<\/jats:p>","DOI":"10.3390\/informatics11040084","type":"journal-article","created":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T12:12:13Z","timestamp":1730981533000},"page":"84","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Can ChatGPT Support Clinical Coding Using the ICD-10-CM\/PCS?"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-8496-3603","authenticated-orcid":false,"given":"Bernardo","family":"Nascimento Teixeira","sequence":"first","affiliation":[{"name":"Iscte-Instituto Universit\u00e1rio de Lisboa (ISCTE-IUL), Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"},{"name":"Instituto de Telecomunica\u00e7\u00f5es (IT), Av. Rovisco Pais, 1, 1049-001 Lisboa, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6123-7153","authenticated-orcid":false,"given":"Ana","family":"Leit\u00e3o","sequence":"additional","affiliation":[{"name":"Unidade Local de Sa\u00fade de Lisboa Ocidental, E.P.E (ULSLO), Estrada do Forte do Alto do Duque, 1449-005 Lisboa, Portugal"},{"name":"NOVA Medical School, Universidade Nova de Lisboa (NMS-UNL), Campo dos M\u00e1rtires da P\u00e1tria 130, 1169-056 Lisboa, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2902-7210","authenticated-orcid":false,"given":"Generosa","family":"Nascimento","sequence":"additional","affiliation":[{"name":"Iscte-Instituto Universit\u00e1rio de Lisboa (ISCTE-IUL), Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"},{"name":"Business Research Unit (BRU-Iscte), Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3784-8198","authenticated-orcid":false,"given":"Adalberto","family":"Campos-Fernandes","sequence":"additional","affiliation":[{"name":"NOVA National School of Public Health, Universidade Nova de Lisboa (ENSP-UNL), Av. Padre Cruz, 1600-407 Lisboa, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0374-4078","authenticated-orcid":false,"given":"Francisco","family":"Cercas","sequence":"additional","affiliation":[{"name":"Iscte-Instituto Universit\u00e1rio de Lisboa (ISCTE-IUL), Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"},{"name":"Instituto de Telecomunica\u00e7\u00f5es (IT), Av. Rovisco Pais, 1, 1049-001 Lisboa, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2024,11,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"372","DOI":"10.12968\/hmed.2008.69.7.30409","article-title":"The increasing importance of clinical coding","volume":"69","author":"Tatham","year":"2008","journal-title":"Br. J. Hosp. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1146\/annurev-publhealth-040218-044206","article-title":"The digitization of patient care: A review of the effects of electronic health records on health care quality and utilization","volume":"40","author":"Atasoy","year":"2019","journal-title":"Annu. Rev. Public Health"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"827","DOI":"10.1016\/j.healthpol.2018.05.014","article-title":"The impact of electronic health record systems on clinical documentation times: A systematic review","volume":"122","author":"Baumann","year":"2018","journal-title":"Health Policy"},{"key":"ref_4","unstructured":"Brynjolfsson, E., and McAfee, A. (2014). The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies, WW Norton & Company."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"47","DOI":"10.2147\/RMHP.S12985","article-title":"Benefits and drawbacks of electronic health record systems","volume":"4","author":"Menachemi","year":"2011","journal-title":"Risk Manag. Healthc. Policy"},{"key":"ref_6","unstructured":"(2021). INE 2023. Instituto Nacional de Estat\u00edstica\u2014Estat\u00edsticas da Sa\u00fade. Available online: https:\/\/www.ine.pt\/xurl\/pub\/11677508."},{"key":"ref_7","unstructured":"(2023, October 12). Pordata. Estat\u00edsticas Sobre Portugal e Europa. Popula\u00e7\u00e3o Residente: Censos. Available online: https:\/\/www.pordata.pt\/subtema\/portugal\/populacao+residente-558."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1038\/s41746-022-00705-7","article-title":"Automated clinical coding: What, why, and where we are?","volume":"5","author":"Dong","year":"2022","journal-title":"NPJ Digit. Med."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1038\/s41746-023-00768-0","article-title":"Automating the overburdened clinical coding system: Challenges and next steps","volume":"6","author":"Venkatesh","year":"2023","journal-title":"NPJ Digit. Med."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"232","DOI":"10.1177\/02683962231200411","article-title":"\u2018ChatGPT et al.\u2019: The ethics of using (generative) artificial intelligence in research and science","volume":"38","year":"2023","journal-title":"J. Inf. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1122","DOI":"10.1109\/JAS.2023.123618","article-title":"A brief overview of ChatGPT: The history, status quo and potential future development","volume":"10","author":"Wu","year":"2023","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_12","unstructured":"Kosinski, M. (2023). Theory of mind may have spontaneously emerged in large language models. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"102744","DOI":"10.1016\/j.dsx.2023.102744","article-title":"ChatGPT: Is this version good for healthcare and research?","volume":"17","author":"Vaishya","year":"2023","journal-title":"Diabetes Metab. Syndr. Clin. Res. Rev."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., and Wheless, L. (2023). Assessing the accuracy and reliability of AI-generated medical responses: An evaluation of the Chat-GPT model. Res. Square, preprint.","DOI":"10.21203\/rs.3.rs-2566942\/v1"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"Gilson","year":"2023","journal-title":"JMIR Med. Educ."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1080\/13645706.2019.1575882","article-title":"Introduction to artificial intelligence in medicine","volume":"28","author":"Mintz","year":"2019","journal-title":"Minimally Invasive Ther. Allied Technol."},{"key":"ref_17","unstructured":"Purdy, M., and Daugherty, P. (2016). Why artificial intelligence is the future of growth. Remarks at AI now: The social and economic implications of artificial intelligence technologies in the near term. Accenture, 1\u201372."},{"key":"ref_18","unstructured":"Schmidt, A., Willard, P., Krawzik, K., and Kenny, A. (2017). ICD-10-CM Professional for Hospitals. The Complete Official Code set. Optum 360 Coding, Optum 360\u00b0."},{"key":"ref_19","unstructured":"Sanmill\u00e1n, M., Cebri\u00e1n, R.N., Pato-Alonso, S., Asensio-Villahoz, P., Salido-Campos, C., Anso-Borda, I., Rodr\u00edguez-Martinez, G., Roces Fern\u00e1ndez, A., Guti\u00e9rrez Miras, A., and Echevarr\u00eda Echarri, L. (2023, July 13). Manual de Codificaci\u00f3n. CIE-10-ES Diagn\u00f3sticos. Unidad T\u00e9cnica de Codificaci\u00f3n CIE-10-ES. Ministerio de Sanidad, Servicios Sociales e Igualdad. Available online: https:\/\/www.sanidad.gob.es\/estadEstudios\/estadisticas\/normalizacion\/CIE10\/UT_MANUAL_DIAG_2016_prov1.pdf."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Patil, R., Heston, T.F., and Bhuse, V. (2024). Prompt Engineering in Healthcare. Electronics, 13.","DOI":"10.3390\/electronics13152961"}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/11\/4\/84\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:28:16Z","timestamp":1760113696000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/11\/4\/84"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,7]]},"references-count":20,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["informatics11040084"],"URL":"https:\/\/doi.org\/10.3390\/informatics11040084","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,7]]}}}