{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,3]],"date-time":"2026-11-03T23:40:16Z","timestamp":1793749216780,"version":"4.6.1"},"reference-count":35,"publisher":"American Medical Association (AMA)","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"Importance<\/jats:title>Risk-stratification tools are routinely used in obstetrics to assist care teams in assessing and communicating risk associated with delivery. Electronic health record data and machine learning methods may offer a novel opportunity to improve and automate risk assessment.<\/jats:p><\/jats:sec>Objective<\/jats:title>To compare the predictive performance of natural language processing (NLP) of clinician documentation with that of a previously validated tool to identify individuals at high risk for maternal morbidity.<\/jats:p><\/jats:sec>Design, Setting, and Participants<\/jats:title>This retrospective diagnostic study was conducted at Brigham and Women\u2019s Hospital and Massachusetts General Hospital, Boston, Massachusetts, and included individuals admitted for delivery at the former institution from July 1, 2016, to February 29, 2020. A subset of these encounters (admissions from February to December 2018) was part of a previous prospective validation study of the Obstetric Comorbidity Index (OB-CMI), a comorbidity-weighted score to stratify risk of severe maternal morbidity (SMM).<\/jats:p><\/jats:sec>Exposures<\/jats:title>Natural language processing of clinician documentation and OB-CMI scores.<\/jats:p><\/jats:sec>Main Outcomes and Measures<\/jats:title>Natural language processing of clinician-authored admission notes was used to predict SMM in individuals delivering at the same institution but not included in the prospective OB-CMI study. The NLP model was then compared with the OB-CMI in the subset with a known OB-CMI score. Model discrimination between the 2 approaches was compared using the DeLong test. Sensitivity and positive predictive value for the identification of individuals at highest risk were prioritized as the characteristics of interest.<\/jats:p><\/jats:sec>Results<\/jats:title>This study included 19\u202f794 individuals; 4034 (20.4%) were included in the original prospective validation study of the OB-CMI (testing set), and the remaining 15\u202f760 (79.6%) composed the training set. Mean (SD) age was 32.3 (5.2) years in the testing cohort and 32.2 (5.2) years in the training cohort. A total of 115 individuals in the testing cohort (2.9%) and 468 in the training cohort (3.0%) experienced SMM. The NLP model was built from a pruned vocabulary of 2783 unique words that occurred within the 15\u202f760 admission notes from individuals in the training set. The area under the receiver operating characteristic curve of the NLP-based model for the prediction of SMM was 0.76 (95% CI, 0.72-0.81) and was comparable with that of the OB-CMI model (0.74; 95% CI, 0.70-0.79) in the testing set (P<\/jats:italic>\u2009=\u2009.53). Sensitivity (NLP, 28.7%; OB-CMI, 24.4%) and positive predictive value (NLP, 19.4%; OB-CMI, 17.6%) were comparable between the NLP and OB-CMI high-risk designations for the prediction of SMM.<\/jats:p><\/jats:sec>Conclusions and Relevance<\/jats:title>In this study, the NLP method and a validated risk-stratification tool had a similar ability to identify patients at high risk of SMM. Future prospective research is needed to validate the NLP approach in clinical practice and determine whether it could augment or replace tools requiring manual user input.<\/jats:p><\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2022.34924","type":"journal-article","created":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T15:01:03Z","timestamp":1664982063000},"page":"e2234924","source":"Crossref","is-referenced-by-count":28,"title":["Comparison of Natural Language Processing of Clinical Notes With a Validated Risk-Stratification Tool to Predict Severe Maternal Morbidity"],"prefix":"10.1001","volume":"5","author":[{"given":"Mark A.","family":"Clapp","sequence":"first","affiliation":[{"name":"Department of Obstetrics and Gynecology, Massachusetts General Hospital, Boston"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ellen","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Brigham and Women\u2019s Hospital, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaitlyn E.","family":"James","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology, Massachusetts General Hospital, Boston"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roy H.","family":"Perlis","sequence":"additional","affiliation":[{"name":"Center for Quantitative Health, Massachusetts General Hospital, Boston"},{"name":"Department of Psychiatry, Massachusetts General Hospital, Boston"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anjali J.","family":"Kaimal","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology, Massachusetts General Hospital, Boston"},{"name":"Department of Population Medicine, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"suffix":"Jr","given":"Thomas H.","family":"McCoy","sequence":"additional","affiliation":[{"name":"Center for Quantitative Health, Massachusetts General Hospital, Boston"},{"name":"Department of Psychiatry, Massachusetts General Hospital, Boston"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah Rae","family":"Easter","sequence":"additional","affiliation":[{"name":"Department of Obstetrics and Gynecology, Brigham and Women\u2019s Hospital, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2022,10,5]]},"reference":[{"issue":"1","key":"zoi220992r1","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1089\/jwh.2013.4617","article-title":"Maternal mortality and morbidity in the United States: where are we now?","volume":"23","author":"Creanga","year":"2014","journal-title":"J Womens Health (Larchmt)"},{"issue":"3","key":"zoi220992r2","doi-asserted-by":"publisher","first-page":"B17","DOI":"10.1016\/j.ajog.2016.07.050","article-title":"Severe maternal morbidity: screening and review.","volume":"215","author":"Kilpatrick","year":"2016","journal-title":"Am J Obstet Gynecol"},{"issue":"5","key":"zoi220992r3","doi-asserted-by":"publisher","first-page":"1029","DOI":"10.1097\/AOG.0b013e31826d60c5","article-title":"Severe maternal morbidity among delivery and postpartum hospitalizations in the United States.","volume":"120","author":"Callaghan","year":"2012","journal-title":"Obstet Gynecol"},{"issue":"5","key":"zoi220992r4","doi-asserted-by":"publisher","first-page":"489.e1","DOI":"10.1016\/j.ajog.2020.02.020","article-title":"Pregnancy-related mortality in the United States, 2003-2016: age, race, and place of death.","volume":"222","author":"Burgess","year":"2020","journal-title":"Am J Obstet Gynecol"},{"issue":"10053","key":"zoi220992r5","doi-asserted-by":"publisher","first-page":"1775","DOI":"10.1016\/S0140-6736(16)31470-2","article-title":"Global, regional, and national levels of maternal mortality, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015.","volume":"388","author":"GBD 2015 Maternal Mortality Collaborators","year":"2016","journal-title":"Lancet"},{"issue":"3","key":"zoi220992r6","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/j.ajog.2014.12.030","article-title":"Levels of maternal care.","volume":"212","author":"Menard","year":"2015","journal-title":"Am J Obstet Gynecol"},{"issue":"3","key":"zoi220992r7","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1053\/j.semperi.2017.03.005","article-title":"Maternal early warning systems\u2014towards reducing preventable maternal mortality and severe maternal morbidity through improved clinical surveillance and responsiveness.","volume":"41","author":"Zuckerwise","year":"2017","journal-title":"Semin Perinatol"},{"issue":"2","key":"zoi220992r8","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1016\/j.ajog.2020.10.030","article-title":"Prediction of obstetrical and fetal complications using automated electronic health record data.","volume":"224","author":"Escobar","year":"2021","journal-title":"Am J Obstet Gynecol"},{"issue":"4","key":"zoi220992r9","doi-asserted-by":"publisher","first-page":"782","DOI":"10.1097\/AOG.0000000000000480","article-title":"The maternal early warning criteria: a proposal from the national partnership for maternal safety.","volume":"124","author":"Mhyre","year":"2014","journal-title":"Obstet Gynecol"},{"issue":"11","key":"zoi220992r10","doi-asserted-by":"publisher","first-page":"1106","DOI":"10.1055\/s-0039-1681097","article-title":"A validation study of maternal early warning systems: a retrospective cohort study.","volume":"36","author":"Blumenthal","year":"2019","journal-title":"Am J Perinatol"},{"issue":"4","key":"zoi220992r11","doi-asserted-by":"publisher","first-page":"527.e1","DOI":"10.1016\/j.ajog.2016.01.154","article-title":"Use of Maternal Early Warning Trigger tool reduces maternal morbidity.","volume":"214","author":"Shields","year":"2016","journal-title":"Am J Obstet Gynecol"},{"issue":"1","key":"zoi220992r12","doi-asserted-by":"publisher","first-page":"58.e1","DOI":"10.1016\/j.ajog.2016.09.103","article-title":"Performance of the Obstetric Early Warning Score in critically ill patients for the prediction of maternal death.","volume":"216","author":"Paternina-Caicedo","year":"2017","journal-title":"Am J Obstet Gynecol"},{"issue":"4","key":"zoi220992r13","doi-asserted-by":"publisher","first-page":"944","DOI":"10.1213\/ANE.0b013e3182a009c3","article-title":"Maternal sepsis mortality and morbidity during hospitalization for delivery: temporal trends and independent associations for severe sepsis.","volume":"117","author":"Bauer","year":"2013","journal-title":"Anesth Analg"},{"issue":"2","key":"zoi220992r15","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajogmf.2020.100305","article-title":"The utility of electronic health record data for identifying postpartum hemorrhage.","volume":"3","author":"Clapp","year":"2021","journal-title":"Am J Obstet Gynecol MFM"},{"issue":"11","key":"zoi220992r16","doi-asserted-by":"publisher","first-page":"2590","DOI":"10.1038\/s41372-021-01072-z","article-title":"Derivation and external validation of risk stratification models for severe maternal morbidity using prenatal encounter diagnosis codes.","volume":"41","author":"Clapp","year":"2021","journal-title":"J Perinatol"},{"issue":"5","key":"zoi220992r17","doi-asserted-by":"publisher","first-page":"957","DOI":"10.1097\/AOG.0b013e3182a603bb","article-title":"Development of a comorbidity index for use in obstetric patients.","volume":"122","author":"Bateman","year":"2013","journal-title":"Obstet Gynecol"},{"issue":"3","key":"zoi220992r18","doi-asserted-by":"publisher","first-page":"440","DOI":"10.1097\/AOG.0000000000004022","article-title":"An expanded obstetric comorbidity scoring system for predicting severe maternal morbidity.","volume":"136","author":"Leonard","year":"2020","journal-title":"Obstet Gynecol"},{"issue":"3","key":"zoi220992r19","doi-asserted-by":"publisher","first-page":"271.e1","DOI":"10.1016\/j.ajog.2019.06.025","article-title":"A comorbidity-based screening tool to predict severe maternal morbidity at the time of delivery.","volume":"221","author":"Easter","year":"2019","journal-title":"Am J Obstet Gynecol"},{"issue":"8","key":"zoi220992r21","doi-asserted-by":"publisher","first-page":"848","DOI":"10.1001\/jama.2011.1204","article-title":"Automated identification of postoperative complications within an electronic medical record using natural language processing.","volume":"306","author":"Murff","year":"2011","journal-title":"JAMA"},{"issue":"10","key":"zoi220992r22","doi-asserted-by":"publisher","first-page":"1064","DOI":"10.1001\/jamapsychiatry.2016.2172","article-title":"Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing.","volume":"73","author":"McCoy","year":"2016","journal-title":"JAMA Psychiatry"},{"key":"zoi220992r23","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1016\/j.tacc.2021.02.007","article-title":"Natural language processing in medicine: a review.","volume":"38","author":"Locke","year":"2021","journal-title":"Trends Anaesth Crit Care"},{"issue":"8","key":"zoi220992r24","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0136341","article-title":"Sentiment measured in hospital discharge notes is associated with readmission and mortality risk: an electronic health record study.","volume":"10","author":"McCoy","year":"2015","journal-title":"PLoS One"},{"issue":"5","key":"zoi220992r25","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1093\/jamia\/ocv180","article-title":"Extracting information from the text of electronic medical records to improve case detection: a systematic review.","volume":"23","author":"Ford","year":"2016","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"zoi220992r27","doi-asserted-by":"publisher","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"issue":"28","key":"zoi220992r28","doi-asserted-by":"publisher","first-page":"4455","DOI":"10.1002\/sim.v36.28","article-title":"Net reclassification index at event rate: properties and relationships.","volume":"36","author":"Pencina","year":"2017","journal-title":"Stat Med"},{"issue":"4","key":"zoi220992r34","doi-asserted-by":"publisher","DOI":"10.1056\/CAT.22.0097","article-title":"Moving past the promise of AI to real uses in health care delivery.","volume":"3","author":"Gordon","journal-title":"NEJM Catalyst Innovations in Health Care Delivery"},{"issue":"7","key":"zoi220992r35","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2018.4087","article-title":"Assessment of time-series machine learning methods for forecasting hospital discharge volume.","volume":"1","author":"McCoy","year":"2018","journal-title":"JAMA Netw Open"},{"key":"zoi220992r14","author":"Clapp"},{"key":"zoi220992r20","author":"Clapp"},{"key":"zoi220992r30","volume-title":"Stata Statistical Software","author":"StataCorp","year":"2019"},{"key":"zoi220992r26","unstructured":"Centers for Disease Control and Prevention. How does CDC identify severe maternal morbidity? Published December 26, 2019. Accessed April 21, 2020. https:\/\/www.cdc.gov\/reproductivehealth\/maternalinfanthealth\/smm\/severe-morbidity-ICD.htm"},{"key":"zoi220992r29","unstructured":"R Core Team. R: a language and environment for statistical computing. R Foundation for Statistical Computing. https:\/\/www.R-project.org"},{"key":"zoi220992r31","unstructured":"Association of Women\u2019s Health, Obstetric and Neonatal Nurses. Postpartum hemorrhage (PPH). Accessed April 6, 2022. https:\/\/www.awhonn.org\/postpartum-hemorrhage-pph\/"},{"key":"zoi220992r32","unstructured":"American College of Obstetricians and Gynecologists. Obstetric Hemorrhage. Accessed April 6, 2022. https:\/\/www.acog.org\/en\/community\/districts-and-sections\/district-ii\/programs-and-resources\/safe-motherhood-initiative\/obstetric-hemorrhage"},{"key":"zoi220992r33","unstructured":"Glover WJ, Li Z, Pachamanova D. The AI-enhanced future of health care administrative task management. NEJM Catalyst Innovations in Health Care Delivery. Published March 3, 2022. Accessed April 6, 2022. https:\/\/catalyst.nejm.org\/doi\/full\/10.1056\/CAT.21.0355"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2797033\/clapp_2022_oi_220992_1664303774.18643.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T15:01:07Z","timestamp":1664982067000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2797033"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,5]]},"references-count":35,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2022,10,3]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2022.34924","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,5]]}}}