{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T10:33:32Z","timestamp":1778150012918,"version":"3.51.4"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"7969","license":[{"start":{"date-parts":[[2023,6,7]],"date-time":"2023-06-07T00:00:00Z","timestamp":1686096000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,6,7]],"date-time":"2023-06-07T00:00:00Z","timestamp":1686096000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nature"],"published-print":{"date-parts":[[2023,7,13]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Physicians make critical time-constrained decisions every day. Clinical predictive models can help physicians and administrators make decisions by forecasting clinical and operational events. Existing structured data-based clinical predictive models have limited use in everyday practice owing to complexity in data processing, as well as model development and deployment<jats:sup>1\u20133<\/jats:sup>. Here we show that unstructured clinical notes from the electronic health record can enable the training of clinical language models, which can be used as all-purpose clinical predictive engines with low-resistance development and deployment. Our approach leverages recent advances in natural language processing<jats:sup>4,5<\/jats:sup>to train a large language model for medical language (NYUTron) and subsequently fine-tune it across a wide range of clinical and operational predictive tasks. We evaluated our approach within our health system for five such tasks: 30-day all-cause readmission prediction, in-hospital mortality prediction, comorbidity index prediction, length of stay prediction, and insurance denial prediction. We show that NYUTron has an area under the curve (AUC) of 78.7\u201394.9%, with an improvement of 5.36\u201314.7% in the AUC compared with traditional models. We additionally demonstrate the benefits of pretraining with clinical text, the potential for increasing generalizability to different sites through fine-tuning and the full deployment of our system in a prospective, single-arm trial. These results show the potential for using clinical language models in medicine to read alongside physicians and provide guidance at the point of care.<\/jats:p>","DOI":"10.1038\/s41586-023-06160-y","type":"journal-article","created":{"date-parts":[[2023,6,7]],"date-time":"2023-06-07T16:01:58Z","timestamp":1686153718000},"page":"357-362","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":395,"title":["Health system-scale language models are all-purpose prediction engines"],"prefix":"10.1038","volume":"619","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2464-3281","authenticated-orcid":false,"given":"Lavender Yao","family":"Jiang","sequence":"first","affiliation":[]},{"given":"Xujin Chris","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Nima Pour","family":"Nejatian","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0389-1852","authenticated-orcid":false,"given":"Mustafa","family":"Nasir-Moin","sequence":"additional","affiliation":[]},{"given":"Duo","family":"Wang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0032-0664","authenticated-orcid":false,"given":"Anas","family":"Abidin","sequence":"additional","affiliation":[]},{"given":"Kevin","family":"Eaton","sequence":"additional","affiliation":[]},{"given":"Howard Antony","family":"Riina","sequence":"additional","affiliation":[]},{"given":"Ilya","family":"Laufer","sequence":"additional","affiliation":[]},{"given":"Paawan","family":"Punjabi","sequence":"additional","affiliation":[]},{"given":"Madeline","family":"Miceli","sequence":"additional","affiliation":[]},{"given":"Nora C.","family":"Kim","sequence":"additional","affiliation":[]},{"given":"Cordelia","family":"Orillac","sequence":"additional","affiliation":[]},{"given":"Zane","family":"Schnurman","sequence":"additional","affiliation":[]},{"given":"Christopher","family":"Livia","sequence":"additional","affiliation":[]},{"given":"Hannah","family":"Weiss","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4074-7497","authenticated-orcid":false,"given":"David","family":"Kurland","sequence":"additional","affiliation":[]},{"given":"Sean","family":"Neifert","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0003-3237-6037","authenticated-orcid":false,"given":"Yosef","family":"Dastagirzada","sequence":"additional","affiliation":[]},{"given":"Douglas","family":"Kondziolka","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0946-3493","authenticated-orcid":false,"given":"Alexander T. M.","family":"Cheung","sequence":"additional","affiliation":[]},{"given":"Grace","family":"Yang","sequence":"additional","affiliation":[]},{"given":"Ming","family":"Cao","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7362-3044","authenticated-orcid":false,"given":"Mona","family":"Flores","sequence":"additional","affiliation":[]},{"given":"Anthony B.","family":"Costa","sequence":"additional","affiliation":[]},{"given":"Yindalon","family":"Aphinyanaphongs","sequence":"additional","affiliation":[]},{"given":"Kyunghyun","family":"Cho","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1876-5963","authenticated-orcid":false,"given":"Eric Karl","family":"Oermann","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,6,7]]},"reference":[{"key":"6160_CR1","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","volume":"3","author":"M Roberts","year":"2021","unstructured":"Roberts, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intel. 3, 199\u2013217 (2021).","journal-title":"Nat. Mach. Intel."},{"key":"6160_CR2","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1426-2","volume":"17","author":"CJ Kelly","year":"2019","unstructured":"Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 195 (2019).","journal-title":"BMC Med."},{"key":"6160_CR3","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00385-9","volume":"4","author":"S Gaube","year":"2021","unstructured":"Gaube, S. et al. Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ Digit. Med. 4, 31 (2021).","journal-title":"NPJ Digit. Med."},{"key":"6160_CR4","unstructured":"Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. in Proc. 2019 NAACL: Human Language Technologies (eds Burstein, J., Doran, C. & Solorio, T.) 4171\u20134186 (Association for Computational Linguistics, 2019)."},{"key":"6160_CR5","unstructured":"Brown, T. B. et al. Language models are few-shot learners. in Proc. NeurIPS (eds Wallach, H. et al.) 1877\u20131901 (Neural Information Processing Systems, 2020)."},{"key":"6160_CR6","doi-asserted-by":"publisher","first-page":"2287","DOI":"10.1161\/01.CIR.0000145172.55640.93","volume":"110","author":"BF Gage","year":"2004","unstructured":"Gage, B. F. et al. Selecting patients with atrial fibrillation for anticoagulation: stroke risk stratification in patients taking aspirin. Circulation 110, 2287\u20132292 (2004).","journal-title":"Circulation"},{"key":"6160_CR7","first-page":"1","volume":"1","author":"CG Child","year":"1964","unstructured":"Child, C. G. & Turcotte, J. G. Surgery and portal hypertension. Major Prob. Clin. Surg. 1, 1\u201385 (1964).","journal-title":"Major Prob. Clin. Surg."},{"key":"6160_CR8","doi-asserted-by":"publisher","first-page":"646","DOI":"10.1002\/bjs.1800600817","volume":"60","author":"RNH Pugh","year":"2005","unstructured":"Pugh, R. N. H., Murray-Lyon, I. M., Dawson, J. L., Pietroni, M. C. & Williams, R. Transection of the oesophagus for bleeding oesophageal varices. Br. J. Surg. 60, 646\u2013649 (2005).","journal-title":"Br. J. Surg."},{"key":"6160_CR9","doi-asserted-by":"publisher","first-page":"1326","DOI":"10.1016\/S0140-6736(95)92535-X","volume":"345","author":"P Wells","year":"1995","unstructured":"Wells, P. et al. Accuracy of clinical assessment of deep-vein thrombosis. Lancet 345, 1326\u20131330 (1995).","journal-title":"Lancet"},{"key":"6160_CR10","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1038\/s41586-019-1390-1","volume":"572","author":"N Toma\u0161ev","year":"2019","unstructured":"Toma\u0161ev, N. et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature 572, 116\u2013119 (2019).","journal-title":"Nature"},{"key":"6160_CR11","first-page":"1184","volume":"39","author":"N Wu","year":"2020","unstructured":"Wu, N. et al. Deep neural networks improve radiologists\u2019 performance in breast cancer screening. IEEE TMI 39, 1184\u20131194 (2020).","journal-title":"IEEE TMI"},{"key":"6160_CR12","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1038\/s41591-018-0335-9","volume":"25","author":"H Liang","year":"2019","unstructured":"Liang, H. et al. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nat. Med. 25, 433\u2013438 (2019).","journal-title":"Nat. Med."},{"key":"6160_CR13","doi-asserted-by":"publisher","unstructured":"Kaplan, J. et al. Scaling laws for neural language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2001.08361 (2020).","DOI":"10.48550\/arXiv.2001.08361"},{"key":"6160_CR14","doi-asserted-by":"crossref","unstructured":"Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. in Proc. 2016 SIGKDD 785\u2013794 (Association for Computing Machinery, 2016).","DOI":"10.1145\/2939672.2939785"},{"key":"6160_CR15","doi-asserted-by":"publisher","first-page":"2957","DOI":"10.1001\/jama.1993.03510240069035","volume":"270","author":"J-RA Le Gall","year":"1993","unstructured":"Le Gall, J.-R. A. New simplified acute physiology score (SAPS II) based on a European\/North American multicenter study. J. Am. Med. Assoc. 270, 2957\u20132963 (1993).","journal-title":"J. Am. Med. Assoc."},{"key":"6160_CR16","doi-asserted-by":"publisher","first-page":"818","DOI":"10.1097\/00003246-198510000-00009","volume":"13","author":"WA Knaus","year":"1985","unstructured":"Knaus, W. A., Draper, E. A., Wagner, D. P. & Zimmerman, J. E. APACHE II: a severity of disease classification system. Crit. Care Med. 13, 818\u2013829 (1985).","journal-title":"Crit. Care Med."},{"key":"6160_CR17","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1016\/0021-9681(87)90171-8","volume":"40","author":"ME Charlson","year":"1987","unstructured":"Charlson, M. E., Pompei, P., Ales, K. L. & MacKenzie, C. R. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J. Chron. Dis. 40, 373\u2013383 (1987).","journal-title":"J. Chron. Dis."},{"key":"6160_CR18","doi-asserted-by":"crossref","unstructured":"Caetano, N., Laureano, R. M. S. & Cortez, P. A data-driven approach to predict hospital length of stay\u2014a Portuguese case study. in Proc. 2014 ICEIS (eds Hammoudi, S., Maciaszek, L. & Cordeiro, J.) 407\u2013414 (SCITEPRESS Digital Library, 2014).","DOI":"10.5220\/0004892204070414"},{"key":"6160_CR19","doi-asserted-by":"publisher","unstructured":"Johnson, M., Albizri, A. & Harfouche, A. Responsible artificial intelligence in healthcare: predicting and preventing insurance claim denials for economic and social wellbeing. Inf. Syst. Front. https:\/\/doi.org\/10.1007\/s10796-021-10137-5 (2021).","DOI":"10.1007\/s10796-021-10137-5"},{"key":"6160_CR20","first-page":"80","volume":"6","author":"C van Walraven","year":"2012","unstructured":"van Walraven, C., Wong, J. & Forster, A. J. LACE+ index: extension of a validated index to predict early death or urgent readmission after hospital discharge using administrative data. Open Med. 6, 80\u201390 (2012).","journal-title":"Open Med."},{"key":"6160_CR21","unstructured":"Center for Disease Control. What is C. diff? https:\/\/www.cdc.gov\/cdiff\/what-is.html (2022)."},{"key":"6160_CR22","doi-asserted-by":"publisher","unstructured":"Yang, G. et al. Language model classifier aligns better with physician word sensitivity than XGBoost on readmission prediction. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2211.07047 (2022).","DOI":"10.48550\/arXiv.2211.07047"},{"key":"6160_CR23","unstructured":"Perez, E., Kiela, D. & Cho, K. True few-shot learning with language models. in Proc. NeurIPS (eds Ranzato, M. et al.) 11054\u201311070 (Neural Information Processing Systems, 2021)."},{"key":"6160_CR24","doi-asserted-by":"publisher","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2212.13138 (2022).","DOI":"10.48550\/arXiv.2212.13138"},{"key":"6160_CR25","unstructured":"Bolton, E. et al. PubMedGPT 2.7B. Technical report. Stanford University Center for Research on Foundation Models https:\/\/crfm.stanford.edu\/2022\/12\/15\/pubmedgpt.html (2022)."},{"key":"6160_CR26","unstructured":"Hoffmann, J. et al. An empirical analysis of compute-optimal large language model training. in Proc. NeurIPS (eds Koyejo, S. et al.) 30016\u201330030 (Neural Information Processing Systems, 2022)."},{"key":"6160_CR27","unstructured":"Charlson, M. Charlson comorbidity index (CCI). MD+CALC https:\/\/www.mdcalc.com\/calc\/3917\/charlson-comorbidity-index-cci (2022)."},{"key":"6160_CR28","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/j.jbi.2013.07.004","volume":"46","author":"W Sun","year":"2013","unstructured":"Sun, W., Rumshisky, A., & Uzuner, O. Annotating temporal information in clinical narratives. J. Biomed. Inform. 46, 5\u201312 (2013).","journal-title":"J. Biomed. Inform."},{"key":"6160_CR29","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AEW Johnson","year":"2016","unstructured":"Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016).","journal-title":"Sci. Data"},{"key":"6160_CR30","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1503\/cmaj.091117","volume":"182","author":"C van Walraven","year":"2010","unstructured":"van Walraven, C. et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. Can. Med. Assoc. J. 182, 551\u2013557 (2010).","journal-title":"Can. Med. Assoc. J."},{"key":"6160_CR31","doi-asserted-by":"publisher","first-page":"1288","DOI":"10.1016\/j.jclinepi.2004.03.012","volume":"57","author":"V Sundararajan","year":"2004","unstructured":"Sundararajan, V. et al. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J. Clin. Epidemiol. 57, 1288\u20131294 (2004).","journal-title":"J. Clin. Epidemiol."},{"key":"6160_CR32","doi-asserted-by":"crossref","unstructured":"Bird, S. & Loper, E. NLTK: The Natural Language Toolkit. in Proc. 2004 ACL Interactive Poster and Demonstration Sessions 214\u2013217 (Association for Computational Linguistics, 2004).","DOI":"10.3115\/1219044.1219075"},{"key":"6160_CR33","unstructured":"Wolf, T. et al. Transformers: state-of-the-art natural language processing. in Proc. 2020 EMNLP (eds Webber, B., Cohn, T., He, Y. & Liu, Y.) 38\u201345 (Association for Computational Linguistics, 2020)."},{"key":"6160_CR34","doi-asserted-by":"crossref","unstructured":"Rajbhandari, S., Rasley, J., Ruwase, O. & He, Y. ZeRO: memory optimizations. Toward training trillion parameter models. in Proc. Int. Conf. High Performance Computing, Networking, Storage and Analysis 1\u201316 (IEEE Press, 2020).","DOI":"10.1109\/SC41405.2020.00024"},{"key":"6160_CR35","unstructured":"Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. ICLR https:\/\/openreview.net\/forum?id=Bkg6RiCqY7 (2019)."},{"key":"6160_CR36","unstructured":"Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. ICLR https:\/\/arxiv.org\/abs\/1412.6980 (2017)."},{"key":"6160_CR37","doi-asserted-by":"publisher","first-page":"21929","DOI":"10.2196\/21929","volume":"9","author":"M Ayaz","year":"2021","unstructured":"Ayaz, M., Pasha, M. F., Alzahrani, M. Y., Budiarto, R. & Stiawan, D. The Fast Health Interoperability Resources (FHIR) standard: systematic literature review of implementations, applications, challenges and opportunities. JMIR Med. Inform. 9, 21929 (2021).","journal-title":"JMIR Med. Inform."},{"key":"6160_CR38","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-Learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"6160_CR39","doi-asserted-by":"crossref","unstructured":"Zhu, Y. et al. Aligning books and movies: towards story-like visual explanations by watching movies and reading books. in Proc. 2015 ICCV (ed. O\u2019Conner, L.) 19\u201327 (IEEE, 2015).","DOI":"10.1109\/ICCV.2015.11"},{"key":"6160_CR40","unstructured":"Wikimedia Foundation. Wikimedia downloads. https:\/\/dumps.wikimedia.org\/ (2021)."},{"key":"6160_CR41","unstructured":"NCBI Literature Resources. Download PubMed data. https:\/\/pubmed.ncbi.nlm.nih.gov\/download\/ (2022)."},{"key":"6160_CR42","unstructured":"National Library of Medicine. PubMed Central: PMC article datasets. https:\/\/www.ncbi.nlm.nih.gov\/pmc\/tools\/textmining\/ (2022)."},{"key":"6160_CR43","doi-asserted-by":"crossref","unstructured":"Yang, X. et al. A large language model for electronic health records. NPJ Digit. Med. 5, 194 (2022).","DOI":"10.1038\/s41746-022-00742-2"},{"key":"6160_CR44","doi-asserted-by":"publisher","unstructured":"Shoeybi, M. et al. Megatron-LM: training multi-billion parameter language models using model parallelism. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1909.08053 (2020).","DOI":"10.48550\/arXiv.1909.08053"},{"key":"6160_CR45","doi-asserted-by":"publisher","unstructured":"Liaw, R. et al. Tune: a research platform for distributed model selection and training. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1807.05118 (2018).","DOI":"10.48550\/arXiv.1807.05118"},{"key":"6160_CR46","first-page":"28","volume":"34","author":"BL Welch","year":"1947","unstructured":"Welch, B. L. The generalization of Student\u2019s problem when several different population variances are involved. Biometrika 34, 28\u201335 (1947).","journal-title":"Biometrika"}],"container-title":["Nature"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41586-023-06160-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41586-023-06160-y","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41586-023-06160-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T22:31:15Z","timestamp":1729549875000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41586-023-06160-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,7]]},"references-count":46,"journal-issue":{"issue":"7969","published-print":{"date-parts":[[2023,7,13]]}},"alternative-id":["6160"],"URL":"https:\/\/doi.org\/10.1038\/s41586-023-06160-y","relation":{},"ISSN":["0028-0836","1476-4687"],"issn-type":[{"value":"0028-0836","type":"print"},{"value":"1476-4687","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,7]]},"assertion":[{"value":"14 October 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 May 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 June 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 July 2023","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"In the version of this article initially published, the graphs in Fig. 2b and Fig. 2c were interchanged, while there were additional patient icons shown in Fig. 4b; the correct figure versions are now shown in the HTML and PDF versions of the article.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"E.K.O. reports consulting with Sofinnova and Google, income from Merck & Co. and Mirati Therapeutics, and equity in Artisight. N.P.N., M.F. and A.B.C. are employed by NVIDIA. D.K. reports consulting with Elekta. K.C. is employed by Prescient Design, a Genentech accelerator, a subsidiary of Roche. There are no other potential conflicts of interest. The work presented herein was performed exclusively within the NYU Langone Health System.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}