{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T18:08:34Z","timestamp":1777918114666,"version":"3.51.4"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T00:00:00Z","timestamp":1732147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Columbia University Data Science Institute Seed Funds"},{"DOI":"10.13039\/100000936","name":"Gordon and Betty Moore Foundation","doi-asserted-by":"publisher","award":["GBMF9048"],"award-info":[{"award-number":["GBMF9048"]}],"id":[{"id":"10.13039\/100000936","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,2,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To identify stigmatizing language in obstetric clinical notes using natural language processing (NLP).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We analyzed electronic health records from birth admissions in the Northeast United States in 2017. We annotated 1771 clinical notes to generate the initial gold standard dataset. Annotators labeled for exemplars of 5 stigmatizing and 1 positive\/preferred language categories. We used a semantic similarity-based search approach to expand the initial dataset by adding additional exemplars, composing an enhanced dataset. We employed traditional classifiers (Support Vector Machine, Decision Trees, and Random Forest) and a transformer-based model, ClinicalBERT (Bidirectional Encoder Representations from Transformers) and BERT base. Models were trained and validated on initial and enhanced datasets and were tested on enhanced testing dataset.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>In the initial dataset, we annotated 963 exemplars as stigmatizing or positive\/preferred. The most frequently identified category was marginalized language\/identities (n\u2009=\u2009397, 41%), and the least frequent was questioning patient credibility (n\u2009=\u200951, 5%). After employing a semantic similarity-based search approach, 502 additional exemplars were added, increasing the number of low-frequency categories. All NLP models also showed improved performance, with Decision Trees demonstrating the greatest improvement (21%). ClinicalBERT outperformed other models, with the highest average F1-score of 0.78.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Clinical BERT seems to most effectively capture the nuanced and context-dependent stigmatizing language found in obstetric clinical notes, demonstrating its potential clinical applications for real-time monitoring and alerts to prevent usages of stigmatizing language use and reduce healthcare bias. Future research should explore stigmatizing language in diverse geographic locations and clinical settings to further contribute to high-quality and equitable perinatal care.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>ClinicalBERT effectively captures the nuanced stigmatizing language in obstetric clinical notes. Our semantic similarity-based search approach to rapidly extract additional exemplars enhanced the performances while reducing the need for labor-intensive annotation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae290","type":"journal-article","created":{"date-parts":[[2024,11,21]],"date-time":"2024-11-21T09:30:23Z","timestamp":1732181423000},"page":"308-317","source":"Crossref","is-referenced-by-count":19,"title":["Identifying stigmatizing and positive\/preferred language in obstetric clinical notes using natural language processing"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0864-7867","authenticated-orcid":false,"given":"Jihye Kim","family":"Scroggins","sequence":"first","affiliation":[{"name":"School of Nursing, Columbia University , New York, NY 10032,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-5504-4646","authenticated-orcid":false,"given":"Ismael I","family":"Hulchafo","sequence":"additional","affiliation":[{"name":"School of Nursing, Columbia University , New York, NY 10032,","place":["United States"]}]},{"given":"Sarah","family":"Harkins","sequence":"additional","affiliation":[{"name":"School of Nursing, Columbia University , New York, NY 10032,","place":["United States"]}]},{"given":"Danielle","family":"Scharp","sequence":"additional","affiliation":[{"name":"Icahn School of Medicine, Mount Sinai , NY 10029,","place":["United States"]}]},{"given":"Hans","family":"Moen","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aalto University , Espoo 02150,","place":["Finland"]}]},{"given":"Anahita","family":"Davoudi","sequence":"additional","affiliation":[{"name":"VNS Health , New York, NY 10017,","place":["United States"]}]},{"given":"Kenrick","family":"Cato","sequence":"additional","affiliation":[{"name":"School of Nursing, University of Pennsylvania , Philadelphia, PA 19104,","place":["United States"]}]},{"given":"Michele","family":"Tadiello","sequence":"additional","affiliation":[{"name":"Center for Community-Engaged Health Informatics and Data Science, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2358-9837","authenticated-orcid":false,"given":"Maxim","family":"Topaz","sequence":"additional","affiliation":[{"name":"School of Nursing, Columbia University , New York, NY 10032,","place":["United States"]}]},{"given":"Veronica","family":"Barcelona","sequence":"additional","affiliation":[{"name":"School of Nursing, Columbia University , New York, NY 10032,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2024,11,21]]},"reference":[{"key":"2025012119035162900_ocae290-B1","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1080\/01612840802694668","article-title":"Stigmatizing language with unintended meanings: \u201cpersons with mental illness\u201d or \u201cmentally ill persons\u201d?","volume":"30","author":"Shattell","year":"2009","journal-title":"Issues Ment Health Nurs."},{"key":"2025012119035162900_ocae290-B2","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1377\/hlthaff.2021.01423","article-title":"Negative patient descriptors: documenting racial bias in the electronic health record","volume":"41","author":"Sun","year":"2022","journal-title":"Health Aff (Millwood)"},{"key":"2025012119035162900_ocae290-B3","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1186\/s12910-017-0179-8","article-title":"Implicit bias in healthcare professionals: a systematic review","volume":"18","author":"FitzGerald","year":"2017","journal-title":"BMC Med Ethics"},{"key":"2025012119035162900_ocae290-B4","doi-asserted-by":"crossref","first-page":"e60","DOI":"10.2105\/AJPH.2015.302903","article-title":"Implicit racial\/ethnic bias among health care professionals and its influence on health care outcomes: a systematic review","volume":"105","author":"Hall","year":"2015","journal-title":"Am J Public Health."},{"key":"2025012119035162900_ocae290-B5","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1080\/08964289.2019.1588220","article-title":"Ubiquitous yet unclear: a systematic review of medical mistrust","volume":"45","author":"Benkert","year":"2019","journal-title":"Behav Med."},{"key":"2025012119035162900_ocae290-B6","author":"Martin","year":"2022"},{"issue":"6","key":"2025012119035162900_ocae290-B7","doi-asserted-by":"crossref","first-page":"e0303653","DOI":"10.1371\/journal.pone.0303653","article-title":"Identifying stigmatizing language in clinical documentation: A scoping review of emerging literature","volume":"19","author":"Barcelona","year":"2024","journal-title":"PLoS One"},{"key":"2025012119035162900_ocae290-B8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cosrev.2022.100511","article-title":"Neural natural language processing for unstructured data in electronic health records: a review","volume":"46","author":"Li","year":"2022","journal-title":"Comput Sci Rev"},{"key":"2025012119035162900_ocae290-B9","doi-asserted-by":"crossref","first-page":"102701","DOI":"10.1016\/j.artmed.2023.102701","article-title":"Natural language processing with machine learning methods to analyze unstructured patient-reported outcomes derived from electronic health records: a systematic review","volume":"146","author":"Sim","year":"2023","journal-title":"Artif Intell Med."},{"key":"2025012119035162900_ocae290-B10","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.tacc.2021.02.007","article-title":"Natural language processing in medicine: a review","volume":"38","author":"Locke","year":"2021","journal-title":"Trends in Anaesth Criti Care"},{"key":"2025012119035162900_ocae290-B11","doi-asserted-by":"crossref","first-page":"3713","DOI":"10.1007\/s11042-022-13428-4","article-title":"Natural language processing: state of the art, current trends and challenges","volume":"82","author":"Khurana","year":"2023","journal-title":"Multimed Tools Appl."},{"issue":"3","key":"2025012119035162900_ocae290-B12","doi-asserted-by":"crossref","first-page":"e12557","DOI":"10.1111\/nin.12557","article-title":"A qualitative analysis of stigmatizing language in birth admission clinical notes","volume":"30","author":"Barcelona","year":"2023","journal-title":"Nurs Inq"},{"issue":"3","key":"2025012119035162900_ocae290-B13","doi-asserted-by":"crossref","first-page":"578","DOI":"10.1007\/s10995-023-03857-4","article-title":"Using Natural Language Processing to Identify Stigmatizing Language in Labor and Birth Clinical Notes","volume":"28","author":"Barcelona","year":"2024","journal-title":"Matern Child Health J"},{"key":"2025012119035162900_ocae290-B14","doi-asserted-by":"crossref","first-page":"e2117052","DOI":"10.1001\/jamanetworkopen.2021.17052","article-title":"Physician use of stigmatizing language in patient medical records","volume":"4","author":"Park","year":"2021","journal-title":"JAMA Netw Open."},{"key":"2025012119035162900_ocae290-B15","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1109\/TKDE.2008.239","article-title":"Learning from imbalanced data","volume":"21","author":"He","year":"2009","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"2025012119035162900_ocae290-B16","author":"Hugging Face","year":"2024"},{"key":"2025012119035162900_ocae290-B17","first-page":"3982","author":"Reimers","year":"2019"},{"key":"2025012119035162900_ocae290-B18","author":"Johnson"},{"key":"2025012119035162900_ocae290-B19","first-page":"72","author":"Alsentzer"},{"key":"2025012119035162900_ocae290-B20","first-page":"4171","author":"Devlin","year":"2019"},{"key":"2025012119035162900_ocae290-B21","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/S0306-4573(02)00021-3","article-title":"An information-theoretic perspective of tf-idf measures","volume":"39","author":"Aizawa","year":"2003","journal-title":"Inf Process Manag"},{"key":"2025012119035162900_ocae290-B22","doi-asserted-by":"crossref","first-page":"2055207620914777","DOI":"10.1177\/2055207620914777","article-title":"Exploring feature selection and classification methods for predicting heart disease","volume":"6","author":"Spencer","year":"2020","journal-title":"Digit Health."},{"key":"2025012119035162900_ocae290-B23","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1136\/bjsports-2023-107496","article-title":"Adding confidence to our injury burden estimates: is bootstrapping the solution?","volume":"58","author":"Williams","year":"2024","journal-title":"Br J Sports Med."},{"key":"2025012119035162900_ocae290-B24","first-page":"201","author":"James","year":"2023"},{"key":"2025012119035162900_ocae290-B25","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1007\/s13253-012-0087-9","article-title":"Nonparametric bootstrap confidence intervals for variance components applied to interlaboratory comparisons","volume":"17","author":"Burch","year":"2012","journal-title":"JABES"},{"key":"2025012119035162900_ocae290-B26","doi-asserted-by":"crossref","first-page":"6086","DOI":"10.1038\/s41598-024-56706-x","article-title":"Evaluation metrics and statistical tests for machine learning","volume":"14","author":"Rainio","year":"2024","journal-title":"Sci Rep."},{"key":"2025012119035162900_ocae290-B27","author":"Zhang"},{"key":"2025012119035162900_ocae290-B28","first-page":"2623","author":"Akiba","year":"2019"},{"key":"2025012119035162900_ocae290-B29","first-page":"1","article-title":"A scoping review of methodological approaches to detect bias in the electronic health record","author":"Kelly","year":"2023","journal-title":"Stigma Health"},{"key":"2025012119035162900_ocae290-B30","first-page":"1","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2025012119035162900_ocae290-B31","first-page":"312","author":"Harrigan","year":"2023"},{"key":"2025012119035162900_ocae290-B32","first-page":"179","author":"Gillioz","year":"2020"},{"key":"2025012119035162900_ocae290-B33","author":"Park"},{"key":"2025012119035162900_ocae290-B34","first-page":"296","author":"Pati","year":"2022"},{"key":"2025012119035162900_ocae290-B35","first-page":"887","author":"Steck","year":"2024"},{"key":"2025012119035162900_ocae290-B36","doi-asserted-by":"crossref","first-page":"2571","DOI":"10.1007\/s11606-020-06432-7","article-title":"Words matter: what do patients find judgmental or offensive in outpatient notes?","volume":"36","author":"Fernandez","year":"2021","journal-title":"J Gen Intern Med."},{"key":"2025012119035162900_ocae290-B37","doi-asserted-by":"crossref","first-page":"e2144967","DOI":"10.1001\/jamanetworkopen.2021.44967","article-title":"Examination of stigmatizing language in the electronic health record","volume":"5","author":"Himmelstein","year":"2022","journal-title":"JAMA Netw Open."},{"key":"2025012119035162900_ocae290-B38","doi-asserted-by":"crossref","first-page":"e50428","DOI":"10.2196\/50428","article-title":"Examining linguistic differences in electronic health records for diverse patients with diabetes: natural language processing analysis","volume":"12","author":"Bilotta","year":"2024","journal-title":"JMIR Med Inform."},{"key":"2025012119035162900_ocae290-B39","doi-asserted-by":"crossref","first-page":"670","DOI":"10.1002\/jhm.13146","article-title":"Use of nonstigmatizing language is associated with improved outcomes in hospitalized people who inject drugs","volume":"18","author":"Carpenter","year":"2023","journal-title":"J Hosp Med."},{"key":"2025012119035162900_ocae290-B40","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"2025012119035162900_ocae290-B41","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1162\/coli.07-034-R2","article-title":"Inter-coder agreement for computational linguistics","volume":"34","author":"Artstein","year":"2008","journal-title":"Comput Linguist Assoc Comput Linguist"},{"key":"2025012119035162900_ocae290-B42","author":"Rolnick"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/2\/308\/60775509\/ocae290.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/2\/308\/60775509\/ocae290.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,21]],"date-time":"2025-01-21T19:04:07Z","timestamp":1737486247000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/2\/308\/7906099"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,21]]},"references-count":42,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,11,21]]},"published-print":{"date-parts":[[2025,2,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae290","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,2]]},"published":{"date-parts":[[2024,11,21]]}}}