{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T18:02:01Z","timestamp":1788976921044,"version":"build-2803163510"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T00:00:00Z","timestamp":1781827200000},"content-version":"vor","delay-in-days":45,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,5,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Emergency department (ED) returns for mental health (MH) conditions pose a substantial burden on healthcare systems. Traditional machine learning (ML) models have shown promise in predicting ED returns but often fall short in clinical interpretability. This study evaluates whether integrating Large Language Models (LLMs) with ML methods can enhance both predictive performance and interpretability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted a retrospective analysis of 42 464 ED visits involving 27 904 unique MH patients at an academic medical center in the deep south (2018-2022). Two primary outcomes were assessed: (1) 30-day ED return prediction, and (2) interpretability of model outputs. We employed LLaMA 3 (8B) for few-shot classification of chief complaints and social determinants of health (SDoH) features, including tobacco, alcohol, substance use, exercise, and home environment. These LLM-derived features were integrated into an XGBoost model. Model explanation was enhanced using a novel LLM-SHAP hybrid framework.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The LLM-based chief complaint classifier achieved 0.882 accuracy, 0.95 precision, 0.88 recall, and 0.86 F1-score, outperforming traditional models. SDoH classifiers reached F1-scores ranging from 0.67 to 0.96 across different categories. Incorporating LLM-extracted features into XGBoost improved AUC from 0.74 to 0.76 and AUC-PR from 0.58 to 0.61. The LLM-SHAP framework provided enhanced contextualized explanations for risk predictions, improving interpretability at both patient and population levels.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Integrating LLMs with traditional ML models modestly improved predictive performance but substantially enhanced interpretability. This hybrid approach offers a promising path toward more actionable and explainable clinical decision support tools in emergency psychiatry.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooag065","type":"journal-article","created":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T11:43:49Z","timestamp":1775735029000},"source":"Crossref","is-referenced-by-count":1,"title":["Explainable AI for mental health emergency returns: integrating large language models with predictive modeling"],"prefix":"10.1093","volume":"9","author":[{"given":"Abdulaziz","family":"Ahmed","sequence":"first","affiliation":[{"name":"Department of Health Services Administration, School of Health Professions, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]},{"name":"Department of Biomedical Informatics and Data Science, Heersink School of Medicine, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad","family":"Saleem","sequence":"additional","affiliation":[{"name":"Department of Health Services Administration, School of Health Professions, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5296-5380","authenticated-orcid":false,"given":"Mohammed","family":"Alzeen","sequence":"additional","affiliation":[{"name":"Department of Health Services Administration, School of Health Professions, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Badari","family":"Birur","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Neurobiology, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rachel E","family":"Fargason","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Neurobiology, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bradley G","family":"Burk","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Neurobiology, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]},{"name":"Department of Pharmacy, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmed","family":"Alhassan","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and Behavioral Neurobiology, University of Alabama at Birmingham , Birmingham, AL 35233,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammed Ali","family":"Al-Garadi","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, School of Medicine, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,6,19]]},"reference":[{"key":"2026061823100738000_ooag065-B1","author":"UHG","year":"2019"},{"key":"2026061823100738000_ooag065-B2","doi-asserted-by":"crossref","first-page":"1523","DOI":"10.1007\/s11845-020-02496-w","article-title":"The suburban-city divide: an evaluation of emergency department mental health presentations across two centres","volume":"190","author":"McLoughlin","year":"2021","journal-title":"Ir J Med Sci"},{"key":"2026061823100738000_ooag065-B3","doi-asserted-by":"publisher","first-page":"e20201174","DOI":"10.1542\/peds.2020-1174","article-title":"Pediatric mental health boarding","volume":"146","author":"McEnany","year":"2020","journal-title":"Pediatrics"},{"key":"2026061823100738000_ooag065-B4","author":"Shapiro"},{"key":"2026061823100738000_ooag065-B5","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.amepre.2024.02.001","article-title":"Social isolation, loneliness, and quality of life among Food-Insecure adults","volume":"67","author":"Park","year":"2024","journal-title":"Am J Prev Med"},{"key":"2026061823100738000_ooag065-B6","doi-asserted-by":"crossref","first-page":"1336","DOI":"10.1111\/acem.13812","article-title":"Psychiatric-related revisits to the emergency department following rapid expansion of community mental health services","volume":"26","author":"Singh","year":"2019","journal-title":"Acad Emerg Med"},{"key":"2026061823100738000_ooag065-B7","doi-asserted-by":"crossref","DOI":"10.15620\/cdc:112081","volume-title":"Emergency Department Visits among Adults with Mental Health Disorders: United States, 2017\u20132019","author":"Santo","year":"2021"},{"key":"2026061823100738000_ooag065-B8","author":"Rahman","year":"2023"},{"key":"2026061823100738000_ooag065-B9","doi-asserted-by":"crossref","first-page":"666","DOI":"10.1111\/acem.14201","article-title":"Emergency department-based interventions affecting social determinants of health in the United States: a scoping review","volume":"28","author":"Walter","year":"2021","journal-title":"Acad Emerg Med"},{"key":"2026061823100738000_ooag065-B10","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1186\/s12873-024-00939-6","article-title":"Machine learning models for predicting unscheduled return visits to an emergency department: a scoping review","volume":"24","author":"Lee","year":"2024","journal-title":"BMC Emerg Med"},{"key":"2026061823100738000_ooag065-B11","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/s10916-025-02210-2","article-title":"A Meta-Analysis of the diagnostic test accuracy of artificial intelligence for predicting emergency department revisits","volume":"49","author":"Kuo","year":"2025","journal-title":"J Med Syst"},{"key":"2026061823100738000_ooag065-B12","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1038\/s41746-025-01533-1","article-title":"Medical foundation large language models for comprehensive text analysis and beyond","volume":"8","author":"Xie","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026061823100738000_ooag065-B13","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1038\/s44387-025-00011-z","article-title":"Large language models for disease diagnosis: a scoping review","volume":"1","author":"Zhou","year":"2025","journal-title":"NPJ Artif Intell"},{"key":"2026061823100738000_ooag065-B14","author":"Agrawal","year":"2022"},{"key":"2026061823100738000_ooag065-B15","first-page":"319"},{"key":"2026061823100738000_ooag065-B16","doi-asserted-by":"crossref","first-page":"1921","DOI":"10.1093\/jamia\/ocae103","article-title":"Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room","volume":"31","author":"Glicksberg","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026061823100738000_ooag065-B17","doi-asserted-by":"crossref","first-page":"e65454","DOI":"10.2196\/65454","article-title":"Classifying unstructured text in electronic health records for mental health prediction models: large language model evaluation study","volume":"13","author":"Cardamone","year":"2025","journal-title":"JMIR Med Inform"},{"key":"2026061823100738000_ooag065-B18","doi-asserted-by":"crossref","first-page":"106584","DOI":"10.1016\/j.cmpb.2021.106584","article-title":"Explanation of machine learning models using shapley additive explanation and application for real data in hospital","volume":"214","author":"Nohara","year":"2022","journal-title":"Comput Methods Programs Biomed"},{"key":"2026061823100738000_ooag065-B19","doi-asserted-by":"crossref","first-page":"103316","DOI":"10.1016\/j.ajp.2022.103316","article-title":"An explainable predictive model for suicide attempt risk using an ensemble learning and Shapley additive explanations (SHAP) approach","volume":"79","author":"Nordin","year":"2023","journal-title":"Asian J Psychiatr"},{"key":"2026061823100738000_ooag065-B20","doi-asserted-by":"crossref","first-page":"6638","DOI":"10.3390\/app14156638","article-title":"XAI-based clinical decision support systems: a systematic review","volume":"14","author":"Kim","year":"2024","journal-title":"Appl Sci"},{"key":"2026061823100738000_ooag065-B21","first-page":"57","article-title":"Large language models in healthcare and medical domain: a review","volume":"11","author":"Nazi","year":"2024","journal-title":"MDPI"},{"key":"2026061823100738000_ooag065-B22","author":"Al-Garadi","year":"2025"},{"key":"2026061823100738000_ooag065-B23","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2026061823100738000_ooag065-B24","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med"},{"key":"2026061823100738000_ooag065-B25","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","article-title":"Scalable and accurate deep learning with electronic health records","volume":"1","author":"Rajkomar","year":"2018"},{"key":"2026061823100738000_ooag065-B26","doi-asserted-by":"crossref","first-page":"104657","DOI":"10.1016\/j.jbi.2024.104657","article-title":"Using a clinical narrative-aware pre-trained language model for predicting emergency department patient disposition and unscheduled return visits","volume":"155","author":"Chen","year":"2024","journal-title":"J Biomed Inform"},{"key":"2026061823100738000_ooag065-B27","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1007\/s12245-008-0027-1","article-title":"ED chief complaint categories for a medical student curriculum","volume":"1","author":"Kuykendal","year":"2008","journal-title":"Int J Emerg Med"},{"key":"2026061823100738000_ooag065-B28","author":"ICD-10 version, 2019","year":"2025"},{"key":"2026061823100738000_ooag065-B29","volume-title":"Davidson\u2019s Principles and Practice of Medicine e-Book","author":"Walker","year":"2013"},{"key":"2026061823100738000_ooag065-B30","author":"CDC","year":"2025"},{"key":"2026061823100738000_ooag065-B31","author":"Dubey","year":"2024"},{"key":"2026061823100738000_ooag065-B32","author":"Li","year":"2024"},{"key":"2026061823100738000_ooag065-B33","doi-asserted-by":"crossref","first-page":"21","DOI":"10.3389\/fnbot.2013.00021","article-title":"Gradient boosting machines, a tutorial","volume":"7","author":"Natekin","year":"2013","journal-title":"Front Neurorobot"},{"key":"2026061823100738000_ooag065-B34","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939785","article-title":"Xgboost: a scalable tree boosting system","author":"Chen","year":":"},{"key":"2026061823100738000_ooag065-B35","first-page":"148","article-title":"Experiments with a new boosting algorithm","author":"Freund","year":"1996"},{"key":"2026061823100738000_ooag065-B36","first-page":"1551","article-title":"Hyperparameter optimization: Comparing genetic algorithm against grid search and Bayesian optimization","author":"Alibrahim","year":"2021"},{"key":"2026061823100738000_ooag065-B37","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","article-title":"A systematic analysis of performance measures for classification tasks","volume":"45","author":"Sokolova","year":"2009","journal-title":"Inf Process Manag"},{"key":"2026061823100738000_ooag065-B38","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1109\/TKDE.2008.239","article-title":"Learning from imbalanced data","volume":"21","author":"He","year":"2009","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2026061823100738000_ooag065-B39","first-page":"243","article-title":"Machine learning with oversampling and undersampling techniques: overview study and experimental results","author":"Mohammed","year":"2020"},{"key":"2026061823100738000_ooag065-B40","first-page":"4765","article-title":"A unified approach to interpreting model predictions","volume":"30","author":"Scott","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026061823100738000_ooag065-B41","first-page":"333","article-title":"Evaluating the impact of social determinants on health prediction in the intensive care unit","author":"Yang","year":"2023"},{"key":"2026061823100738000_ooag065-B42","doi-asserted-by":"publisher","first-page":"567","DOI":"10.1186\/s12916-025-04340-3","article-title":"Navigating fairness aspects of clinical prediction models","volume":"23","author":"Chakradeo","year":"2025","journal-title":"BMC Med"},{"key":"2026061823100738000_ooag065-B43","doi-asserted-by":"publisher","first-page":"e98319","DOI":"10.7759\/cureus.98319","article-title":"Artificial Intelligence-driven risk stratification in chronic kidney disease progression: minimizing bias via race-specific algorithms","volume":"17","author":"Behmard","year":"2025","journal-title":"Cureus"},{"key":"2026061823100738000_ooag065-B44","doi-asserted-by":"publisher","first-page":"ooaf171","DOI":"10.1093\/jamiaopen\/ooaf171","article-title":"Evaluation and improvement of algorithmic fairness for COVID-19 severity classification using explainable artificial intelligence-based bias mitigation","volume":"9","author":"Nejadshamsi","year":"2026","journal-title":"JAMIA Open"},{"key":"2026061823100738000_ooag065-B45","first-page":"41","article-title":"Navigating fairness in healthcare: a comparative analysis of single-stage and multi-stage bias mitigation strategies","author":"Dehghani","year":"2025"},{"key":"2026061823100738000_ooag065-B46","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1370\/afm.2419","article-title":"Characteristics of case management in primary care associated with positive outcomes for frequent users of health care: a systematic review","volume":"17","author":"Hudon","year":"2019","journal-title":"Ann Fam Med"},{"key":"2026061823100738000_ooag065-B47","doi-asserted-by":"publisher","first-page":"e0192360","DOI":"10.1371\/journal.pone.0192360","article-title":"Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives","volume":"13","author":"Gehrmann","year":"2018","journal-title":"PLoS One"},{"key":"2026061823100738000_ooag065-B48","first-page":"110","article-title":"Hurtful words: quantifying biases in clinical contextual word embeddings","author":"Zhang","year":"2020"},{"key":"2026061823100738000_ooag065-B49","doi-asserted-by":"publisher","author":"Sun","year":"2021","DOI":"10.18653\/v1\/2021.naacl-main.77"},{"key":"2026061823100738000_ooag065-B50","doi-asserted-by":"publisher","author":"Alsentzer","year":"2019","DOI":"10.18653\/v1\/W19-1909"},{"key":"2026061823100738000_ooag065-B51","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.05342","author":"Huang","year":"2019","journal-title":"ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission"},{"key":"2026061823100738000_ooag065-B52","first-page":"359","article-title":"What clinicians want: contextualizing explainable machine learning for clinical end use","author":"Tonekaboni","year":"2019"},{"key":"2026061823100738000_ooag065-B53","first-page":"1135","author":"Ribeiro","year":"2016"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/3\/ooag065\/68556541\/ooag065.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/3\/ooag065\/68556541\/ooag065.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T03:10:20Z","timestamp":1781838620000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooag065\/8711433"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,5]]},"references-count":53,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,5,5]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooag065","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,6]]},"published":{"date-parts":[[2026,5,5]]},"article-number":"ooag065"}}