{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T08:24:55Z","timestamp":1776414295044,"version":"3.51.2"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T00:00:00Z","timestamp":1701302400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T00:00:00Z","timestamp":1701302400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000080","name":"Anesthesia Patient Safety Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000080","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005292","name":"Brigham and Women\u2019s Hospital","doi-asserted-by":"publisher","award":["IGNITE"],"award-info":[{"award-number":["IGNITE"]}],"id":[{"id":"10.13039\/100005292","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["K08HL161326-01A1"],"award-info":[{"award-number":["K08HL161326-01A1"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["K08 HL146963-02"],"award-info":[{"award-number":["K08 HL146963-02"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"U.S. Department of Health & Human Services | NIH | National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["K08 HL146963-02S1"],"award-info":[{"award-number":["K08 HL146963-02S1"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Many areas of medicine would benefit from deeper, more accurate phenotyping, but there are limited approaches for phenotyping using clinical notes without substantial annotated data. Large language models (LLMs) have demonstrated immense potential to adapt to novel tasks with no additional training by specifying task-specific instructions. Here we report the performance of a publicly available LLM, Flan-T5, in phenotyping patients with postpartum hemorrhage (PPH) using discharge notes from electronic health records (\n                    <jats:italic>n<\/jats:italic>\n                    \u2009=\u2009271,081). The language model achieves strong performance in extracting 24 granular concepts associated with PPH. Identifying these granular concepts accurately allows the development of interpretable, complex phenotypes and subtypes. The Flan-T5 model achieves high fidelity in phenotyping PPH (positive predictive value of 0.95), identifying 47% more patients with this complication compared to the current standard of using claims codes. This LLM pipeline can be used reliably for subtyping PPH and outperforms a claims-based approach on the three most common PPH subtypes associated with uterine atony, abnormal placentation, and obstetric trauma. The advantage of this approach to subtyping is its interpretability, as each concept contributing to the subtype determination can be evaluated. Moreover, as definitions may change over time due to new guidelines, using granular concepts to create complex phenotypes enables prompt and efficient updating of the algorithm. Using this language modelling approach enables rapid phenotyping without the need for any manually annotated training data across multiple clinical use cases.\n                  <\/jats:p>","DOI":"10.1038\/s41746-023-00957-x","type":"journal-article","created":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T06:02:34Z","timestamp":1701324154000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":57,"title":["Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5370-1746","authenticated-orcid":false,"given":"Emily","family":"Alsentzer","sequence":"first","affiliation":[]},{"given":"Matthew J.","family":"Rasmussen","sequence":"additional","affiliation":[]},{"given":"Romy","family":"Fontoura","sequence":"additional","affiliation":[]},{"given":"Alexis L.","family":"Cull","sequence":"additional","affiliation":[]},{"given":"Brett","family":"Beaulieu-Jones","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3541-3724","authenticated-orcid":false,"given":"Kathryn J.","family":"Gray","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6268-1540","authenticated-orcid":false,"given":"David W.","family":"Bates","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7367-3394","authenticated-orcid":false,"given":"Vesela P.","family":"Kovacheva","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,11,30]]},"reference":[{"key":"957_CR1","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1093\/jamia\/ocv034","volume":"22","author":"S Yu","year":"2015","unstructured":"Yu, S. et al. Toward high-throughput phenotyping: unbiased automated feature extraction and selection from knowledge sources. J. Am. Med. Inform. Assoc.: JAMIA 22, 993\u20131000 (2015).","journal-title":"J. Am. Med. Inform. Assoc.: JAMIA"},{"key":"957_CR2","doi-asserted-by":"publisher","first-page":"e027689","DOI":"10.1161\/JAHA.122.027689","volume":"12","author":"R Nakamaru","year":"2023","unstructured":"Nakamaru, R. et al. Phenotyping of elderly patients with heart failure focused on noncardiac conditions: a latent class analysis from a multicenter registry of patients hospitalized with heart failure. J. Am. Heart Assoc. 12, e027689 (2023).","journal-title":"J. Am. Heart Assoc."},{"key":"957_CR3","doi-asserted-by":"publisher","first-page":"e24003","DOI":"10.2196\/24003","volume":"23","author":"WL Bennett","year":"2021","unstructured":"Bennett, W. L. et al. Patient recruitment into a multicenter clinical cohort linking electronic health records from 5 health systems: cross-sectional analysis. J. Med. Internet Res. 23, e24003 (2021).","journal-title":"J. Med. Internet Res."},{"key":"957_CR4","doi-asserted-by":"publisher","first-page":"e226","DOI":"10.1136\/amiajnl-2013-001926","volume":"20","author":"RL Richesson","year":"2013","unstructured":"Richesson, R. L. et al. Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory. J. Am. Med. Inform. Assoc.: JAMIA 20, e226\u2013e231 (2013).","journal-title":"J. Am. Med. Inform. Assoc.: JAMIA"},{"key":"957_CR5","first-page":"1160","volume":"2013","author":"AR Post","year":"2013","unstructured":"Post, A. R. et al. Temporal abstraction-based clinical phenotyping with Eureka! AMIA Ann. Symposium Proc. 2013, 1160\u20131169 (2013).","journal-title":"AMIA Ann. Symposium Proc."},{"key":"957_CR6","doi-asserted-by":"publisher","first-page":"1102","DOI":"10.1038\/nbt.2749","volume":"31","author":"JC Denny","year":"2013","unstructured":"Denny, J. C. et al. Systematic comparison of phenome-wide association study of electronic medical record data and genome-wide association study data. Nat. Biotechnol. 31, 1102\u20131110 (2013).","journal-title":"Nat. Biotechnol."},{"key":"957_CR7","doi-asserted-by":"publisher","first-page":"998","DOI":"10.1111\/trf.14498","volume":"58","author":"AJ Butwick","year":"2018","unstructured":"Butwick, A. J., Walsh, E. M., Kuzniewicz, M., Li, S. X. & Escobar, G. J. Accuracy of international classification of diseases, ninth revision, codes for postpartum hemorrhage among women undergoing cesarean delivery. Transfusion 58, 998\u20131005 (2018).","journal-title":"Transfusion"},{"key":"957_CR8","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1093\/jamia\/ocab181","volume":"29","author":"AB Zheutlin","year":"2022","unstructured":"Zheutlin, A. B. et al. A comprehensive digital phenotype for postpartum hemorrhage. J. Am. Med. Inform. Assoc.: JAMIA 29, 321\u2013328 (2022).","journal-title":"J. Am. Med. Inform. Assoc.: JAMIA"},{"key":"957_CR9","doi-asserted-by":"publisher","first-page":"1046","DOI":"10.1093\/jamia\/ocv202","volume":"23","author":"JC Kirby","year":"2016","unstructured":"Kirby, J. C. et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J. Am. Med. Inform. Assoc.: JAMIA 23, 1046\u20131052 (2016).","journal-title":"J. Am. Med. Inform. Assoc.: JAMIA"},{"key":"957_CR10","doi-asserted-by":"publisher","first-page":"h1885","DOI":"10.1136\/bmj.h1885","volume":"350","author":"KP Liao","year":"2015","unstructured":"Liao, K. P. et al. Development of phenotype algorithms using electronic medical records and incorporating natural language processing. BMJ 350, h1885 (2015).","journal-title":"BMJ"},{"key":"957_CR11","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1109\/TCBB.2018.2849968","volume":"16","author":"Z Zeng","year":"2019","unstructured":"Zeng, Z., Deng, Y., Li, X., Naumann, T. & Luo, Y. Natural language processing for EHR-based computational phenotyping. IEEE\/ACM Trans. Comput. Biol. Bioinform. 16, 139\u2013153 (2019).","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"957_CR12","unstructured":"Aramaki, E., Imai, T., Miyo, K. & Ohe, K. Patient status classification by using rule based sentence extraction and bm25 knn-based classifier. In i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data (2006)."},{"key":"957_CR13","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-019-0786-z","volume":"19","author":"H Sharma","year":"2019","unstructured":"Sharma, H. et al. Developing a portable natural language processing based phenotyping system. BMC Med. Inform. Decis. Making 19, 78 (2019).","journal-title":"BMC Med. Inform. Decis. Making"},{"key":"957_CR14","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-017-0418-4","volume":"17","author":"HS Chase","year":"2017","unstructured":"Chase, H. S., Mitrani, L. R., Lu, G. G. & Fulgieri, D. J. Early recognition of multiple sclerosis using natural language processing of the electronic health record. BMC Med. Inform. Decis. Making 17, 24 (2017).","journal-title":"BMC Med. Inform. Decis. Making"},{"key":"957_CR15","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1006\/jbin.2001.1000","volume":"34","author":"WW Chapman","year":"2001","unstructured":"Chapman, W. W., Fizman, M., Chapman, B. E. & Haug, P. J. A comparison of classification algorithms to automatically identify chest x-ray reports that support pneumonia. J. Biomed. Inform. 34, 4\u201314 (2001).","journal-title":"J. Biomed. Inform."},{"key":"957_CR16","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1093\/jamia\/ocac216","volume":"30","author":"S Yang","year":"2023","unstructured":"Yang, S., Varghese, P., Stephenson, E., Tu, K. & Gronsbell, J. Machine learning approaches for electronic health records phenotyping: a methodical review. J. Am. Med. Inform. Assoc. 30, 367\u2013381 (2023).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"957_CR17","doi-asserted-by":"publisher","first-page":"e0192360","DOI":"10.1371\/journal.pone.0192360","volume":"13","author":"S Gehrmann","year":"2018","unstructured":"Gehrmann, S. et al. Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives. PloS One 13, e0192360 (2018).","journal-title":"PloS One"},{"key":"957_CR18","unstructured":"Brown, T. B. et al. Language models are few-shot learners. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems. 1877\u20131901 (Curran Associates, Inc, 2020)."},{"key":"957_CR19","unstructured":"Ouyang, L. et al. Training language models to follow instructions with human feedback. In: Advances in Neural Information Processing Systems. 27730\u201327744 (Curran Associates, Inc, 2022)."},{"key":"957_CR20","unstructured":"Chung, H. W. et al. Scaling Instruction-Finetuned Language Models. Preprint at: http:\/\/arxiv.org\/abs\/2210.11416 (2022)."},{"key":"957_CR21","first-page":"972","volume":"2022","author":"S Sivarajkumar","year":"2022","unstructured":"Sivarajkumar, S. & Wang, Y. HealthPrompt: a zero-shot learning paradigm for clinical natural language processing. AMIA Ann. Symposium Proc. 2022, 972\u2013981 (2022).","journal-title":"AMIA Ann. Symposium Proc."},{"key":"957_CR22","unstructured":"Yao, Z., Cao, Y., Yang, Z., Deshpande, V. & Yu, H. Extracting biomedical factual knowledge using pretrained language model and electronic health record context. In AMIA Annual Symposium Proceedings, vol. 2022, 1188 (American Medical Informatics Association, 2022)."},{"key":"957_CR23","doi-asserted-by":"crossref","unstructured":"Agrawal, M., Hegselmann, S., Lang, H., Kim, Y. & Sontag, D. Large Language Models are Few-Shot Clinical Information Extractors. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 1998\u20132022 (Abu Dhabi, United Arab Emirates. Association for Computational Linguistics, 2022).","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"key":"957_CR24","doi-asserted-by":"crossref","unstructured":"McInerney, D. J., Young, G., van de Meent, J.-W. & Wallace, B. C. CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models. Preprint at: http:\/\/arxiv.org\/abs\/2302.12343 (2023).","DOI":"10.18653\/v1\/2023.findings-emnlp.568"},{"key":"957_CR25","unstructured":"Global Burden of Disease Collaborators. Global, regional, and national levels of maternal mortality, 1990\u20132015: a systematic analysis for the Global Burden of Disease Study. Lancet 388, 1775\u20131812 (2016)."},{"key":"957_CR26","doi-asserted-by":"crossref","unstructured":"American College of Obstetricians and Gynecologists. Practice Bulletin No. 183: Postpartum Hemorrhage. Obstetrics and Gynecology 130, e168\u2013e186 (2017).","DOI":"10.1097\/AOG.0000000000002351"},{"key":"957_CR27","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1097\/AOG.0000000000004972","volume":"141","author":"CM Corbetta-Rastelli","year":"2023","unstructured":"Corbetta-Rastelli, C. M. et al. Postpartum hemorrhage trends and outcomes in the United States, 2000\u20132019. Obstet. Gynecol. 141, 152\u2013161 (2023).","journal-title":"Obstet. Gynecol."},{"key":"957_CR28","doi-asserted-by":"publisher","first-page":"e0246119","DOI":"10.1371\/journal.pone.0246119","volume":"16","author":"D Walther","year":"2021","unstructured":"Walther, D. et al. Hospital discharge data is not accurate enough to monitor the incidence of postpartum hemorrhage. PloS One 16, e0246119 (2021).","journal-title":"PloS One"},{"key":"957_CR29","doi-asserted-by":"crossref","unstructured":"American College of Obstetricians and Gynecologists. ACOG Practice Bulletin: Clinical Management Guidelines for Obstetrician-Gynecologists Number 76, October 2006: postpartum hemorrhage. Obstetrics and Gynecology 108, 1039\u20131047 (2006).","DOI":"10.1097\/00006250-200610000-00046"},{"key":"957_CR30","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1002\/pds.4967","volume":"29","author":"M He","year":"2020","unstructured":"He, M. et al. Validation of algorithms to identify adverse perinatal outcomes in the Medicaid Analytic Extract database. Pharmacoepidemiol. Drug Safety 29, 419\u2013426 (2020).","journal-title":"Pharmacoepidemiol. Drug Safety"},{"key":"957_CR31","doi-asserted-by":"publisher","first-page":"103550","DOI":"10.1016\/j.ijoa.2022.103550","volume":"51","author":"HB Ende","year":"2022","unstructured":"Ende, H. B. et al. Development of clinical risk-prediction models for uterine atony following vaginal and cesarean delivery. Int. J. Obstet. Anesth. 51, 103550 (2022).","journal-title":"Int. J. Obstet. Anesth."},{"key":"957_CR32","unstructured":"Lehman, E. et al. Do we still need clinical language models? In: Proceedings of the Conference on Health, Inference, and Learning, PMLR 209, 578\u2013597, (2023)."},{"key":"957_CR33","first-page":"1044","volume":"2006","author":"R Nalichowski","year":"2006","unstructured":"Nalichowski, R., Keogh, D., Chueh, H. C. & Murphy, S. N. Calculating the benefits of a research patient data repository. AMIA Ann. Symposium Proc. 2006, 1044 (2006).","journal-title":"AMIA Ann. Symposium Proc."},{"key":"957_CR34","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1007\/s10995-007-0256-6","volume":"12","author":"EV Kuklina","year":"2008","unstructured":"Kuklina, E. V. et al. An enhanced method for identifying obstetric deliveries: implications for estimating maternal morbidity. Maternal Child Health J. 12, 469\u2013477 (2008).","journal-title":"Maternal Child Health J."},{"key":"957_CR35","unstructured":"Levy, A. & Agrawal, M. PRAnCER, Platform enabling Rapid Annotation for Clinical Entity Recognition. Original-date: 2020-10-25. https:\/\/github.com\/clinicalml\/prancer (2023)."},{"key":"957_CR36","unstructured":"Sylvain Gugger, T. W. P. S. Z. M. S. M., Lysandre Debut. Accelerate: training and inference at scale made simple, efficient and adaptable. https:\/\/github.com\/huggingface\/accelerate (2022)."},{"key":"957_CR37","doi-asserted-by":"publisher","first-page":"1317","DOI":"10.1097\/AOG.0000000000003559","volume":"134","author":"D Goffman","year":"2019","unstructured":"Goffman, D. et al. A framework for improving characterization of obstetric hemorrhage using informatics data. Obstet. Gynecol. 134, 1317\u20131325 (2019).","journal-title":"Obstet. Gynecol."},{"key":"957_CR38","first-page":"442","volume":"95","author":"A Evensen","year":"2017","unstructured":"Evensen, A., Anderson, J. M. & Fontaine, P. Postpartum hemorrhage: prevention and treatment. Am. Fam. Physician 95, 442\u2013449 (2017).","journal-title":"Am. Fam. Physician"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00957-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00957-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00957-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,26]],"date-time":"2023-12-26T20:31:50Z","timestamp":1703622710000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00957-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,30]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["957"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00957-x","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.05.31.23290753","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,30]]},"assertion":[{"value":"2 June 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 November 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"K.J.G. has served as a consultant to Illumina Inc., Aetion, Roche, and BillionToOne outside the scope of the submitted work. D.W.B. reports grants and personal fees from EarlySense, personal fees from CDI Negev, equity from Valera Health, equity from CLEW, equity from MDClone, personal fees and equity from AESOP Technology, personal fees and equity from FeelBetter, and grants from IBM Watson Health, outside the submitted work. V.P.K. reports consulting fees from Avania CRO unrelated to the current work.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"212"}}