{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,11,7]],"date-time":"2026-11-07T15:21:03Z","timestamp":1794064863548,"version":"4.8.0"},"posted":{"date-parts":[[2024,10,17]]},"group-title":"Health Informatics","reference-count":29,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2024,10,17]],"date-time":"2024-10-17T00:00:00Z","timestamp":1729123200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.medrxiv.org\/about\/FAQ#license"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2024,10,17]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Accurate medical coding is essential for clinical and administrative purposes but complicated, time-consuming, and biased. This study compares Retrieval-Augmented Generation (RAG)-enhanced LLMs to provider-assigned codes in producing ICD-10-CM codes from emergency department (ED) clinical records.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Retrospective cohort study using 500 ED visits randomly selected from the Mount Sinai Health System between January and April 2024. The RAG system integrated past 1,038,066 ED visits data (2021-2023) into the LLMs\u2019 predictions to improve coding accuracy. Nine commercial and open-source LLMs were evaluated. The primary outcome was a head-to-head comparison of the ICD-10-CM codes generated by the RAG-enhanced LLMs and those assigned by the original providers. A panel of four physicians and two LLMs blindly reviewed the codes, comparing the RAG-enhanced LLM and provider-assigned codes on accuracy and specificity.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Findings<\/jats:title>\n                  <jats:p>RAG-enhanced LLMs demonstrated superior performance to provider coders in both the accuracy and specificity of code assignments. In a targeted evaluation of 200 cases where discrepancies existed between GPT-4 and provider-assigned codes, human reviewers favored GPT-4 for accuracy in 447 instances, compared to 277 instances where providers\u2019 codes were preferred (p&lt;0.001). Similarly, GPT-4 was selected for its superior specificity in 509 cases, whereas human coders were preferred in only 181 cases (p&lt;0.001). Smaller open-access models, such as Llama-3.1-70B, also demonstrated substantial scalability when enhanced with RAG, with 218 instances of accuracy preference compared to 90 for providers\u2019 codes. Furthermore, across all models, the exact match rate between LLM-generated and provider-assigned codes significantly improved following RAG integration, with Qwen-2-7B increasing from 0.8% to 17.6% and Gemma-2-9b-it improving from 7.2% to 26.4%.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Interpretation<\/jats:title>\n                  <jats:p>RAG-enhanced LLMs improve medical coding accuracy in EDs, suggesting clinical workflow applications. These findings show that generative AI can improve clinical outcomes and reduce administrative burdens.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Funding<\/jats:title>\n                  <jats:p>This work was supported in part through the computational and data resources and staff expertise provided by Scientific Computing and Data at the Icahn School of Medicine at Mount Sinai and supported by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under award number S10OD026880 and S10OD030463. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders played no role in study design, data collection, analysis and interpretation of data, or the writing of this manuscript.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Twitter Summary<\/jats:title>\n                  <jats:p>A study showed AI models with retrieval-augmented generation outperformed human doctors in ED diagnostic coding accuracy and specificity. Even smaller AI models perform favorably when using RAG. This suggests potential for reducing administrative burden in healthcare, improving coding efficiency, and enhancing clinical documentation.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2024.10.15.24315526","type":"posted-content","created":{"date-parts":[[2024,10,18]],"date-time":"2024-10-18T03:35:20Z","timestamp":1729222520000},"source":"Crossref","is-referenced-by-count":14,"title":["Assessing Retrieval-Augmented Large Language Model Performance in Emergency Department ICD-10-CM Coding Compared to Human Coders"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4567-3108","authenticated-orcid":false,"given":"Eyal","family":"Klang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3297-7022","authenticated-orcid":false,"given":"Idit","family":"Tessler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Donald U","family":"Apakama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ethan","family":"Abbott","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin S","family":"Glicksberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Monique","family":"Arnold","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akini","family":"Moses","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ankit","family":"Sakhuja","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Soroush","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander W","family":"Charney","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David L.","family":"Reich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jolion","family":"McGreevy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Gavin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brendan","family":"Carr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert","family":"Freeman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Girish N","family":"Nadkarni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2024102105450547000_2024.10.15.24315526v1.1","first-page":"263","article-title":"Applications of the international classification of diseases","volume":"43","year":"1990","journal-title":"World Health Stat Q"},{"key":"2024102105450547000_2024.10.15.24315526v1.2","first-page":"41","article-title":"The applications of Australian-coded ICD-10 and ICD-10-AM data in research: A scoping review of the literature","volume":"53","year":"2024","journal-title":"Health Inf Manag"},{"key":"2024102105450547000_2024.10.15.24315526v1.3","doi-asserted-by":"publisher","DOI":"10.1056\/aidbp2300040"},{"key":"2024102105450547000_2024.10.15.24315526v1.4","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1038\/s41746-022-00705-7","article-title":"Automated clinical coding: what, why, and where we are?","volume":"5","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2024102105450547000_2024.10.15.24315526v1.5","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2403.10822"},{"key":"2024102105450547000_2024.10.15.24315526v1.6","doi-asserted-by":"crossref","first-page":"445","DOI":"10.3390\/medicina60030445","article-title":"Integrating retrieval-augmented generation with large language models in nephrology: Advancing practical applications","volume":"60","year":"2024","journal-title":"Medicina"},{"key":"2024102105450547000_2024.10.15.24315526v1.7","doi-asserted-by":"publisher","DOI":"10.1056\/aioa2300068"},{"key":"2024102105450547000_2024.10.15.24315526v1.8","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2407.12849"},{"key":"2024102105450547000_2024.10.15.24315526v1.9","doi-asserted-by":"crossref","first-page":"1921","DOI":"10.1093\/jamia\/ocae103","article-title":"Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room","volume":"31","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2024102105450547000_2024.10.15.24315526v1.10","doi-asserted-by":"crossref","first-page":"8","DOI":"10.21037\/jmai-24-60","article-title":"Fine-tuning for accuracy: evaluation of Generative Pretrained Transformer (GPT) for automatic assignment of International Classification of Disease (ICD) codes to clinical documentation","volume":"7","year":"2024","journal-title":"J Med Artif Intell"},{"key":"2024102105450547000_2024.10.15.24315526v1.11","doi-asserted-by":"publisher","DOI":"10.1101\/2024.08.14.24312010"},{"key":"2024102105450547000_2024.10.15.24315526v1.12","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2019.2921572"},{"key":"2024102105450547000_2024.10.15.24315526v1.13","doi-asserted-by":"publisher","DOI":"10.1111\/j.1475-6773.2005.00444.x"},{"key":"2024102105450547000_2024.10.15.24315526v1.14","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.ijmedinf.2018.09.021","article-title":"Towards automated clinical coding","volume":"120","year":"2018","journal-title":"Int J Med Inform"},{"key":"2024102105450547000_2024.10.15.24315526v1.15","doi-asserted-by":"crossref","first-page":"763","DOI":"10.1016\/j.jopan.2018.07.001","article-title":"Machine Learning in Health Care","volume":"33","year":"2018","journal-title":"J Perianesth Nurs"},{"key":"2024102105450547000_2024.10.15.24315526v1.16","unstructured":"Topaz M , Shafran-Topaz L , Bowles KH . ICD-9 to ICD-10: evolution, revolution, and current debates in the United States. Perspect Health Inf Manag 2013; 10: 1d."},{"key":"2024102105450547000_2024.10.15.24315526v1.17","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-021-01614-0"},{"key":"2024102105450547000_2024.10.15.24315526v1.18","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1109\/TCBB.2018.2849968","article-title":"Natural language processing for EHR-based computational phenotyping","volume":"16","year":"2019","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2024102105450547000_2024.10.15.24315526v1.19","first-page":"983","article-title":"Using retrieval-augmented generation to capture molecularly-driven treatment relationships for precision oncology","volume":"316","year":"2024","journal-title":"Stud Health Technol Inform"},{"key":"2024102105450547000_2024.10.15.24315526v1.20","doi-asserted-by":"publisher","DOI":"10.1101\/2024.02.08.24302376"},{"key":"2024102105450547000_2024.10.15.24315526v1.21","doi-asserted-by":"publisher","DOI":"10.1016\/S0733-8627(03)00098-1"},{"key":"2024102105450547000_2024.10.15.24315526v1.22","doi-asserted-by":"crossref","first-page":"103479","DOI":"10.1016\/j.jbi.2020.103479","article-title":"Recent advances of HCI in decision-making tasks for optimized clinical workflows and precision medicine","volume":"108","year":"2020","journal-title":"J Biomed Inform"},{"key":"2024102105450547000_2024.10.15.24315526v1.23","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1038\/s41746-022-00611-y","article-title":"Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare","volume":"5","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2024102105450547000_2024.10.15.24315526v1.24","doi-asserted-by":"crossref","first-page":"862322","DOI":"10.3389\/fsurg.2022.862322","article-title":"Legal and ethical consideration in Artificial Intelligence in healthcare: Who takes responsibility?","volume":"9","year":"2022","journal-title":"Front Surg"},{"key":"2024102105450547000_2024.10.15.24315526v1.25","first-page":"427","article-title":"The black box problem revisited","volume":"32","year":"2024","journal-title":"Real and imaginary challenges for automated legal decision making. Artif Intell Law"},{"key":"2024102105450547000_2024.10.15.24315526v1.26","doi-asserted-by":"crossref","first-page":"115466","DOI":"10.1016\/j.psychres.2023.115466","article-title":"Ethical considerations for the use of artificial intelligence in medical decision-making capacity assessments","volume":"328","year":"2023","journal-title":"Psychiatry Res"},{"key":"2024102105450547000_2024.10.15.24315526v1.27","doi-asserted-by":"crossref","first-page":"8270","DOI":"10.1038\/s41467-024-52618-6","article-title":"Generalizability assessment of AI models across hospitals in a low-middle and high income country","volume":"15","year":"2024","journal-title":"Nat Commun"},{"key":"2024102105450547000_2024.10.15.24315526v1.28","doi-asserted-by":"crossref","first-page":"e220028","DOI":"10.1148\/ryai.220028","article-title":"Generalizability of Machine Learning models: Quantitative evaluation of three methodological pitfalls","volume":"5","year":"2023","journal-title":"Radiol Artif Intell"},{"key":"2024102105450547000_2024.10.15.24315526v1.29","doi-asserted-by":"crossref","unstructured":"Singh RP , Hom GL , Abramoff MD , Campbell JP , Chiang MF , AAO Task Force on Artificial Intelligence. Current challenges and barriers to real-world artificial intelligence adoption for the healthcare system, provider, and the patient. Transl Vis Sci Technol 2020; 9: 45.","DOI":"10.1167\/tvst.9.2.45"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2024.10.15.24315526","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:14:57Z","timestamp":1768508097000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2024.10.15.24315526"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,17]]},"references-count":29,"URL":"https:\/\/doi.org\/10.1101\/2024.10.15.24315526","relation":{},"subject":[],"published":{"date-parts":[[2024,10,17]]},"subtype":"preprint"}}