{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T13:17:18Z","timestamp":1788182238294,"version":"build-2803163510"},"reference-count":33,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T00:00:00Z","timestamp":1765497600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"},{"start":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T00:00:00Z","timestamp":1765497600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/100005292","name":"Brigham and Women's Hospital","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005292","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["obgyn.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Pregnancy"],"published-print":{"date-parts":[[2026,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Postpartum hemorrhage (PPH) is a major cause of maternal morbidity and mortality. Timely prediction may prevent adverse maternal outcomes, and efforts are needed to develop accurate predictive tools. A high\u2010performing machine learning model to predict PPH using data from the US Consortium for Safe Labor (CSL) remains to be widely validated in contemporary clinical settings using electronic health record (EHR) data. Our goal was to evaluate the performance of the CSL PPH predictive model using EHR data across a large healthcare system in the Northeastern United States.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted a retrospective cohort study across eight hospitals in the Northeastern United States between May 2015 and May 2024. We used the same sociodemographic, clinical diagnoses, family history, laboratory, and vital signs available on labor and delivery admission in the EHR that were used to train the original CSL model. The binary outcome was PPH, defined as estimated blood loss of 1000\u00a0mL or more at delivery or blood transfusion within 24\u00a0h postpartum. We then refit a new model using the original features to assess whether model performance could be further improved in our study population using the best\u2010performing machine learning approach (extreme gradient boosting [XGBoost]) from the original CSL model. We evaluated model discrimination as measured using the area under the curve (AUC), feature importance, calibration, and decision analysis curves of both the original CSL model with external validation and the further refit model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Among 87,662 deliveries, the incidence of PPH was 7.7%. The original CSL model demonstrated modest discrimination for predicting PPH with an AUC of 0.60 (95% confidence interval [CI], 0.58\u20130.61). Refitting a new model with XGBoost resulted in improved discrimination with an AUC of 0.75 (95% CI, 0.74\u20130.76). Calibration analyses demonstrated that the refit model overestimated PPH risk across a range of predicted probabilities.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>A previously developed PPH predictive model had substantially reduced performance with external validation using contemporary EHR data across an eight\u2010hospital health system in the Northeastern United States, underscoring limited generalizability. These findings highlight the importance of external validation, local adaptation, and ongoing surveillance for assessing model performance before implementing predictive models clinically.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/pmf2.70200","type":"journal-article","created":{"date-parts":[[2025,12,13]],"date-time":"2025-12-13T06:43:37Z","timestamp":1765608217000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["External validation of a machine learning model to predict postpartum hemorrhage in a US northeastern healthcare system"],"prefix":"10.1002","volume":"2","author":[{"given":"Vesela P.","family":"Kovacheva","sequence":"first","affiliation":[{"name":"Department of Anesthesiology Perioperative and Pain Medicine Brigham and Women's Hospital, Harvard Medical School  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ricardo","family":"Kleinlein","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology Perioperative and Pain Medicine Brigham and Women's Hospital, Harvard Medical School  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nolan","family":"Wheeler","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology Perioperative and Pain Medicine Brigham and Women's Hospital, Harvard Medical School  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kartik K.","family":"Venkatesh","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology The Ohio State University  Columbus Ohio USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric","family":"Jelovsek","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology Duke University  Durham North Carolina USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David W.","family":"Bates","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care Brigham and Women's Hospital  Boston USA"},{"name":"Department of Health Care Policy and Management Harvard T. H. Chan School of Public Health  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathryn J.","family":"Gray","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology University of Washington  Seattle Washington USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,12,12]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.1097\/AOG.0000000000002351"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMra1513247"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1097\/AOG.0000000000004972"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1213\/ANE.0000000000006864"},{"key":"e_1_2_10_6_1","doi-asserted-by":"publisher","DOI":"10.1097\/AOG.0b013e3182941c78"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1097\/AOG.0000000000003759"},{"key":"e_1_2_10_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijoa.2022.103550"},{"key":"e_1_2_10_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajog.2022.01.026"},{"key":"e_1_2_10_10_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-60097-4"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajog.2010.06.058"},{"key":"e_1_2_10_12_1","doi-asserted-by":"publisher","DOI":"10.1055\/a-1745-1348"},{"key":"e_1_2_10_13_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-020-01766-9"},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2023.3259395"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.g7594"},{"key":"e_1_2_10_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10995-007-0256-6"},{"key":"e_1_2_10_17_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00957-x"},{"key":"e_1_2_10_18_1","first-page":"1","article-title":"An Efficient Explanation of Individual Classifications Using Game Theory","volume":"11","author":"Strumbelj E.","year":"2010","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_10_19_1","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000291"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/14767058.2021.1932805"},{"key":"e_1_2_10_21_1","doi-asserted-by":"publisher","DOI":"10.1055\/s-0039-1688823"},{"key":"e_1_2_10_22_1","doi-asserted-by":"publisher","DOI":"10.1213\/ANE.0000000000004560"},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.2196\/52059"},{"key":"e_1_2_10_24_1","unstructured":"The JointCommission.R3 Report: Provision of Care Treatment and Services Standards for Maternal Safety. Issue24. [Internet].2019. Available From:https:\/\/digitalassets.jointcommission.org\/api\/public\/content\/ed4a0ea369c44adca8bc48ba05b76b61?v=a0747800."},{"key":"e_1_2_10_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcjq.2018.05.007"},{"key":"e_1_2_10_26_1","doi-asserted-by":"publisher","DOI":"10.1097\/ALN.0000000000000682"},{"key":"e_1_2_10_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(24)02102-0"},{"key":"e_1_2_10_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(17)30638-4"},{"key":"e_1_2_10_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajog.2023.08.015"},{"key":"e_1_2_10_30_1","doi-asserted-by":"publisher","DOI":"10.1111\/trf.16216"},{"key":"e_1_2_10_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2024.111636"},{"key":"e_1_2_10_32_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.13685"},{"key":"e_1_2_10_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2024.111387"},{"key":"e_1_2_10_34_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocae209"}],"container-title":["Pregnancy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/pmf2.70200","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/pmf2.70200","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/pmf2.70200","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T12:26:35Z","timestamp":1788179195000},"score":1,"resource":{"primary":{"URL":"https:\/\/obgyn.onlinelibrary.wiley.com\/doi\/10.1002\/pmf2.70200"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,12]]},"references-count":33,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1]]}},"alternative-id":["10.1002\/pmf2.70200"],"URL":"https:\/\/doi.org\/10.1002\/pmf2.70200","archive":["Portico"],"relation":{},"ISSN":["2997-9684","2997-9684"],"issn-type":[{"value":"2997-9684","type":"print"},{"value":"2997-9684","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,12]]},"assertion":[{"value":"2025-08-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70200"}}