{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,3]],"date-time":"2026-11-03T13:42:36Z","timestamp":1793713356704,"version":"4.6.1"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T00:00:00Z","timestamp":1771977600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T00:00:00Z","timestamp":1773360000000},"content-version":"vor","delay-in-days":16,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000865","name":"Bill and Melinda Gates Foundation","doi-asserted-by":"publisher","award":["INV-062576"],"award-info":[{"award-number":["INV-062576"]}],"id":[{"id":"10.13039\/100000865","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000865","name":"Bill and Melinda Gates Foundation","doi-asserted-by":"publisher","award":["INV-062576"],"award-info":[{"award-number":["INV-062576"]}],"id":[{"id":"10.13039\/100000865","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Large Language Models (LLMs) hold the potential for clinical task-shifting by processing unstructured clinical text, enabling tasks such as clinical concept extraction and medical question answering from electronic health records. If implemented reliably, such approaches may benefit over-burdened healthcare systems, particularly in resource-limited settings and for traditionally overlooked populations, provided that local fine-tuning is supported by appropriate clinical and technical expertise. However, this powerful technology remains largely understudied in real-world contexts, particularly in the Global South. This study aims to assess whether openly available LLMs can be used reliably for processing medical notes in real-world settings in South Asia.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We used publicly available LLMs to parse de-identified clinical notes from a large electronic health records (EHR) database in Pakistan, containing hospital records for 8.2\u00a0million patients. ChatGPT (GPT-3.5) as a general-purpose LLM, and GatorTron (base), BioMegatron, BioBert and ClinicalBERT as medical LLMs were evaluated when applied to these data, after fine-tuning them with (a) publicly available clinical datasets namely Informatics for Integrating Biology &amp; the Bedside (I2B2) and National NLP Clinical Challenges (N2C2) for medical concept extraction (MCE) and emrQA for medical question answering (MQA), and (b) the local Pakistani de-identified EHR dataset, which includes inpatient Discharge Summaries (DS) and Subjective, Objective, Assessment, and Plan (SOAP) notes, as detailed in this paper. MCE models were applied to these clinical notes using both 3-label and 9-label formats, while MQA models were applied to medical questions. Internal and external validation performance was measured for (a) and (b) using F1 score, precision, recall, and accuracy for MCE and BLEU and ROUGE-L, which measure lexical and sequence similarity, for MQA.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>When clinical LLMs were not fine-tuned on the local EHR dataset, their performance during external validation on local data was notably poorer compared to internal validation on the dataset used for fine-tuning, with reductions of at least 15% in F1 scores for MCE and 35% in ROUGE-L and BLEU scores for MQA tasks. This suggests potential bias and highlights the inability of the medical LLMs to reliably handle the data distribution of the local population without further fine-tuning and adaptation. This trend persisted across two distinct natural language processing tasks: concept extraction and question answering, spanning a spectrum of task complexities. However, fine-tuning the LLMs with local EHR data significantly improved model performance across both tasks, yielding a 7.5% to 15% increase in the F1 score for MCE and a 27% to 53% increase in ROUGE-L and BLEU scores for MQA. Notably, ChatGPT, as a general-purpose LLM, stood out as an exception, demonstrating superior performance across all measured metrics on the local dataset compared to the publicly available dataset, with improvements ranging from 3% to 17% on the local EHR dataset, even without fine-tuning on the local data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Publicly available LLMs, predominantly trained on data from high-income regions, were found to be unreliable when applied in a real-world clinical setting in Pakistan. Fine-tuning them with local EHR data and regional clinical contexts improved their reliability, demonstrating a feasible adaptation strategy that is substantially less resource-intensive than training large language models from scratch. Close collaboration between local clinical and technical experts to curate and leverage more representative, inclusive, and unbiased medical datasets, can play a crucial role in further ensuring reliability of LLMs for resource-limited, overburdened settings, to be used in ways that are safe, fair, and beneficial for all.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12911-026-03366-8","type":"journal-article","created":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T15:37:45Z","timestamp":1772033865000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Evaluating large language models for clinical note processing: local fine-tuning and internal-external validation using electronic health records from South Asia"],"prefix":"10.1186","volume":"26","author":[{"given":"Seyed Alireza","family":"Hasheminasab","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faisal","family":"Jamil","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad Usman","family":"Afzal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali Haider","family":"Khan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sehrish","family":"Ilyas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ali","family":"Noor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Awais","family":"Touseef","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salma","family":"Abbas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hajira Nisar","family":"Cheema","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad Usman","family":"Shabbir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iqra","family":"Hameed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maleeha","family":"Ayub","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hamayal","family":"Masood","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amina","family":"Jafar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amir Mukhtar","family":"Khan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad Abid","family":"Nazir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad Asaad","family":"Jamil","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faisal","family":"Sultan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2845-5731","authenticated-orcid":false,"given":"Sara","family":"Khalid","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,2,25]]},"reference":[{"issue":"8","key":"3366_CR1","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","volume":"29","author":"AJ Thirunavukarasu","year":"2023","unstructured":"Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DS. W. Large Language models in medicine. Nat Med. 2023;29(8):1930\u201340.","journal-title":"Nat Med"},{"issue":"1","key":"3366_CR2","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","volume":"3","author":"J Clusmann","year":"2023","unstructured":"Clusmann J, et al. The future landscape of large Language models in medicine. Commun Med. 2023;3(1):141.","journal-title":"Commun Med"},{"issue":"6","key":"3366_CR3","first-page":"589","volume":"183","author":"JW Ayers","year":"2023","unstructured":"Ayers JW, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern. 2023;183(6):589\u201396.","journal-title":"JAMA Intern"},{"key":"3366_CR4","doi-asserted-by":"crossref","unstructured":"Kung TH et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023:2(2), e0000198.","DOI":"10.1371\/journal.pdig.0000198"},{"issue":"1","key":"3366_CR5","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","volume":"5","author":"X Yang","year":"2022","unstructured":"Yang X, et al. A large Language model for electronic health records. NPJ Digit Med. 2022;5(1):194.","journal-title":"NPJ Digit Med"},{"key":"3366_CR6","doi-asserted-by":"publisher","unstructured":"Shin HC. et al. BioMegatron: larger biomedical domain language model. 2020: ArXiv:2010.06060. https:\/\/doi.org\/10.48550\/arXiv.2010.06060.","DOI":"10.48550\/arXiv.2010.06060"},{"key":"3366_CR7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.03323","author":"E Alsentzer","year":"2019","unstructured":"Alsentzer E, et al. Publicly available clinical BERT embeddings. ArXiv:1904 03323. 2019. https:\/\/doi.org\/10.48550\/arXiv.1904.03323.","journal-title":"ArXiv:1904 03323"},{"issue":"4","key":"3366_CR8","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee J, et al. BioBERT: a pre-trained biomedical Language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234\u201340.","journal-title":"Bioinformatics"},{"issue":"1","key":"3366_CR9","first-page":"1","volume":"3","author":"Y Gu","year":"2021","unstructured":"Gu Y, et al. Domain-specific Language model pretraining for biomedical natural Language processing. ACM Trans Comput Healthc (HEALTH). 2021;3(1):1\u201323.","journal-title":"ACM Trans Comput Healthc (HEALTH)"},{"key":"3366_CR10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2001.08361","author":"J Kaplan","year":"2020","unstructured":"Kaplan J, et al. Scaling laws for neural Language models. ArXiv:2001 08361. 2020. https:\/\/doi.org\/10.48550\/arXiv.2001.08361.","journal-title":"ArXiv:2001 08361"},{"issue":"2","key":"3366_CR11","first-page":"181","volume":"18","author":"ST Rosenbloom","year":"2011","unstructured":"Rosenbloom ST, et al. Data from clinical notes: a perspective on the tension between structure and flexible Documentation. JAMIA. 2011;18(2):181\u20136.","journal-title":"JAMIA"},{"issue":"1","key":"3366_CR12","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1055\/s-0038-1638592","volume":"17","author":"SM Meystre","year":"2008","unstructured":"Meystre SM, et al. Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform. 2008;17(1):128\u201344.","journal-title":"Yearb Med Inform"},{"issue":"5","key":"3366_CR13","first-page":"552","volume":"18","author":"\u00d6 Uzuner","year":"2011","unstructured":"Uzuner \u00d6, et al. 2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text. JAMIA. 2011;18(5):552\u20136.","journal-title":"JAMIA"},{"issue":"1","key":"3366_CR14","first-page":"117","volume":"20","author":"G Hripcsak","year":"2013","unstructured":"Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. JAMIA. 2013;20(1):117\u201321.","journal-title":"JAMIA"},{"issue":"1","key":"3366_CR15","first-page":"3","volume":"27","author":"S Henry","year":"2020","unstructured":"Henry S, et al. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. JAMIA. 2020;27(1):3\u201312.","journal-title":"JAMIA"},{"key":"3366_CR16","unstructured":"Lin CY, Rouge. A package for automatic evaluation of summaries. Text Summarization Branches Out. 2004:74\u201381."},{"key":"3366_CR17","doi-asserted-by":"crossref","unstructured":"Papineni K et al. Bleu: a method for automatic evaluation of machine translation. Proc 40th Annu Meet Assoc Comput Linguist. 2002:311\u2013318.","DOI":"10.3115\/1073083.1073135"},{"key":"3366_CR18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1809.00732","author":"A Pampari","year":"2018","unstructured":"Pampari A, et al. Emrqa: A large corpus for question answering on electronic medical records. ArXiv:1809 00732. 2018. https:\/\/doi.org\/10.48550\/arXiv.1809.00732.","journal-title":"ArXiv:1809 00732"},{"issue":"13","key":"3366_CR19","doi-asserted-by":"publisher","first-page":"1317","DOI":"10.1001\/jama.2017.18391","volume":"319","author":"AL Beam","year":"2018","unstructured":"Beam AL, Kohane IS. Big data and machine learning in health care. JAMA. 2018;319(13):1317\u20138.","journal-title":"JAMA"},{"issue":"1","key":"3366_CR20","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1146\/annurev-biodatasci-092820-114757","volume":"4","author":"IY Chen","year":"2021","unstructured":"Chen IY, et al. Ethical machine learning in healthcare. Annual Rev Biomedical Data Sci. 2021;4(1):123\u201344.","journal-title":"Annual Rev Biomedical Data Sci"},{"issue":"3","key":"3366_CR21","doi-asserted-by":"publisher","first-page":"e0000022","DOI":"10.1371\/journal.pdig.0000022","volume":"1","author":"LA Celi","year":"2022","unstructured":"Celi LA, et al. Sources of bias in artificial intelligence that perpetuate healthcare disparities\u2014A global review. PLOS Digit Health. 2022;1(3):e0000022.","journal-title":"PLOS Digit Health"},{"issue":"12","key":"3366_CR22","doi-asserted-by":"publisher","first-page":"e0240376","DOI":"10.1371\/journal.pone.0240376","volume":"15","author":"I Straw","year":"2020","unstructured":"Straw I, Callison-Burch C. Artificial intelligence in mental health and the biases of Language based models. PLoS ONE. 2020;15(12):e0240376.","journal-title":"PLoS ONE"},{"issue":"13","key":"3366_CR23","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","volume":"388","author":"P Lee","year":"2023","unstructured":"Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. NEJM. 2023;388(13):1233\u20139.","journal-title":"NEJM"},{"issue":"1","key":"3366_CR24","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1186\/s13054-023-04393-x","volume":"27","author":"R Azamfirei","year":"2023","unstructured":"Azamfirei R, Kudchadkar SR, Fackler J. Large Language models and the perils of their hallucinations. Crit Care. 2023;27(1):120.","journal-title":"Crit Care"},{"issue":"7956","key":"3366_CR25","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","volume":"616","author":"M Moor","year":"2023","unstructured":"Moor M, et al. Foundation models for generalist medical artificial intelligence. Nature. 2023;616(7956):259\u201365.","journal-title":"Nature"},{"key":"3366_CR26","doi-asserted-by":"publisher","unstructured":"Shoeybi M et al. Megatron-lm: training multi-billion parameter language models using model parallelism. 2019: ArXiv:1909.08053. https:\/\/doi.org\/10.48550\/arXiv.1909.08053.","DOI":"10.48550\/arXiv.1909.08053"},{"key":"3366_CR27","doi-asserted-by":"publisher","unstructured":"Thoppilan R et al. Lamda: language models for dialog applications. 2022: ArXiv:2201.08239. https:\/\/doi.org\/10.48550\/arXiv.2201.08239.","DOI":"10.48550\/arXiv.2201.08239"},{"key":"3366_CR28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2210.02414","author":"A Zeng","year":"2022","unstructured":"Zeng A, et al. Glm-130b: an open bilingual pre-trained model. ArXiv:2210 02414. 2022. https:\/\/doi.org\/10.48550\/arXiv.2210.02414.","journal-title":"ArXiv:2210 02414"},{"key":"3366_CR29","doi-asserted-by":"publisher","unstructured":"Amatriain X et al. Transformer models: an introduction and catalog. 2023: ArXiv:2302.07730. https:\/\/doi.org\/10.48550\/arXiv.2302.07730.","DOI":"10.48550\/arXiv.2302.07730"},{"issue":"9","key":"3366_CR30","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1001\/jama.2023.14217","volume":"330","author":"NH Shah","year":"2023","unstructured":"Shah NH, Entwistle D, Pfeffer MA. Creation and adoption of large Language models in medicine. JAMA. 2023;330(9):866\u20139.","journal-title":"JAMA"},{"issue":"4","key":"3366_CR31","first-page":"643","volume":"30","author":"EP Junior","year":"2023","unstructured":"Junior EP, et al. Integrating real-world data from Brazil and Pakistan into the OMOP common data model and standardized health analytics framework to characterize COVID-19 in the global South. JAMIA. 2023;30(4):643\u201355.","journal-title":"JAMIA"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-026-03366-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-026-03366-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-026-03366-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T18:52:57Z","timestamp":1781635977000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s12911-026-03366-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,25]]},"references-count":31,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["3366"],"URL":"https:\/\/doi.org\/10.1186\/s12911-026-03366-8","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,25]]},"assertion":[{"value":"26 May 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 January 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The study was conducted in accordance with the ethical standards of the Institutional Review and Ethics Board (IRB) of Shaukat Khanum Memorial Cancer Hospital and Research Centre (SKMCH&RC) and the 1964 Helsinki Declaration and its later amendments. The ethics review number is EX-17-07-23-01. This study involved the secondary use of data from an existing electronic health records database. No direct human participation was required. The Institutional Review Board of SKMCH&RC approved the protocol for secondary use of this de-identified data. Therefore, the requirement for individual informed consent to participate was waived by the Institutional Review and Ethics Board of Shaukat Khanum Memorial Cancer Hospital and Research Centre (SKMCH&RC).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable. This study did not involve the use of any individual person\u2019s data in any form (including individual details, images, or videos), and the need for consent for publication was waived by the Institutional Review Board of SKMCH&RC.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Sara Khalid receives funding from the Wellcome Trust and the Gates Foundation outside of this work. All other authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"75"}}