{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T15:23:36Z","timestamp":1786116216299,"version":"build-2736575974"},"reference-count":30,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:00:00Z","timestamp":1785542400000},"content-version":"vor","delay-in-days":25,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000060","name":"National Institute of Allergy and Infectious Diseases","doi-asserted-by":"publisher","award":["R01AI172875"],"award-info":[{"award-number":["R01AI172875"]}],"id":[{"id":"10.13039\/100000060","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,7,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To evaluate the real-world performance of a transformer-based natural language processing (NLP) system for extracting Social Determinants of Health (SDoH) from clinical notes, using survey-based SDoH measures a reference comparators.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>This study was conducted at the University of Florida Health in adults with at least 2 clinical encounters in the prior year. A research survey was completed by 1001 participants; sampling targeted 50% Black patients to support subgroup analyses. Comparative analyses were restricted to the 414 participants who also had Epic SDoH survey data and clinical notes available for NLP extraction. We compared concepts extracted by the SOcial DeterminAnts (SODA) NLP pipeline against the independently administered research survey, which served as the primary reference standard, and against the structured Epic-embedded SDoH questionnaire. Nine domains were evaluated: abuse, alcohol use, drug use, education, financial constraints, housing, physical activity, social cohesion, and transportation. Sensitivity, specificity, positive predictive value, negative predictive value, and F1 scores were calculated by domain.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The NLP pipeline more consistently aligned with negative survey responses than with patient-reported social needs, although performance was lower in some domains, especially alcohol use and financial constraints. Sensitivity was higher only for alcohol use (55%); the lowest values were for abuse (5%), drug use (0%), and financial constraints (16%). These results cannot be attributed to SODA extraction alone. They reflect some combination of social information not being recorded in clinical notes, content that was recorded but not extracted, and mismatches between extracted concepts and survey definitions, and the present analysis cannot separate these contributions. The 2 surveys agreed only modestly with each other, so no single instrument provides a definitive ground truth.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The pipeline more consistently aligned with negative survey responses than with patient-reported social needs. Because the 2 surveys agreed only modestly, the reference itself is imperfect, and apparent NLP performance depends in part on which survey is used as the comparator. Apparent gaps in NLP performance reflect both how social risks are recorded in clinical notes and how patients disclose them across different survey settings, in addition to limits of the extraction pipeline. Improving documentation practices, integrating locally tuned large language models, and monitoring subgroup performance may all be needed to make SDoH identification tools reliably detect social needs across patient populations.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooag147","type":"journal-article","created":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T11:46:05Z","timestamp":1784288765000},"source":"Crossref","is-referenced-by-count":0,"title":["Real-world evaluation of a transformer-based natural language processing system for identifying social determinants of health from routine clinical documentation"],"prefix":"10.1093","volume":"9","author":[{"given":"Xiangren","family":"Wang","sequence":"first","affiliation":[{"name":"Department of Health Services Research, Management and Policy, University of Florida , Gainesville, FL 32610,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingchuan","family":"Guo","sequence":"additional","affiliation":[{"name":"Department of Pharmacy Practice, College of Pharmacy, Purdue University , West Lafayette, IN 47907,","place":["United States"]},{"name":"Regenstrief Institute Inc , Indianapolis, IN 46202,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0587-4105","authenticated-orcid":false,"given":"Yi","family":"Guo","sequence":"additional","affiliation":[{"name":"Department of Health Outcomes and Biomedical Informatics, University of Florida , Gainesville, FL 32610,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yao An","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Pharmaceutical Outcomes and Policy, University of Florida , Gainesville, FL 32611,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9021-5595","authenticated-orcid":false,"given":"Mattia","family":"Prosperi","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, University of Florida , Gainesville, FL 32603,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2238-5429","authenticated-orcid":false,"given":"Jiang","family":"Bian","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Health Data Science, Indiana University , Indianapolis, IN 46202,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3446-2700","authenticated-orcid":false,"given":"Noah","family":"Hammarlund","sequence":"additional","affiliation":[{"name":"Department of Health Services Research, Management and Policy, University of Florida , Gainesville, FL 32610,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,7,31]]},"reference":[{"key":"2026080604264425200_ooag147-B1","author":"Office of Disease Prevention and Health Promotion","year":"2025"},{"key":"2026080604264425200_ooag147-B2","author":"Centers for Disease Control and Prevention","year":"2025"},{"key":"2026080604264425200_ooag147-B3","doi-asserted-by":"publisher","first-page":"1370","DOI":"10.30574\/wjarr.2023.20.1.2018","article-title":"Social determinants of health: examining poverty, housing, and education in widening US healthcare access disparities","volume":"20","author":"Aidoo","year":"2023","journal-title":"World J Adv Res Rev"},{"key":"2026080604264425200_ooag147-B4","doi-asserted-by":"publisher","first-page":"qxae046","DOI":"10.1093\/haschl\/qxae046","article-title":"Social determinants of health Z-code documentation practices in mental health settings: a scoping review","volume":"2","author":"Hendricks-Sturrup","year":"2024","journal-title":"Health Aff Sch"},{"key":"2026080604264425200_ooag147-B5","doi-asserted-by":"publisher","first-page":"1771","DOI":"10.1001\/jama.2009.561","article-title":"Effect of a housing and case management program on emergency department visits and hospitalizations among chronically ill homeless adults: a randomized trial","volume":"301","author":"Sadowski","year":"2009","journal-title":"JAMA"},{"key":"2026080604264425200_ooag147-B6","doi-asserted-by":"publisher","first-page":"e23818","DOI":"10.1097\/MD.0000000000023818","article-title":"International classification of diseases, tenth revision, clinical modification social determinants of health codes are poorly used in electronic health records","volume":"99","author":"Guo","year":"2020","journal-title":"Medicine (Baltimore)"},{"key":"2026080604264425200_ooag147-B7","doi-asserted-by":"publisher","first-page":"e147","DOI":"10.1017\/cts.2024.571","article-title":"Realizing the potential of social determinants data in EHR systems: a scoping review of approaches for screening, linkage, extraction, analysis, and interventions","volume":"8","author":"Li","year":"2024","journal-title":"J Clin Transl Sci"},{"key":"2026080604264425200_ooag147-B8","doi-asserted-by":"publisher","first-page":"e60","DOI":"10.1017\/cts.2024.506","article-title":"Screening tools to address social determinants of health in the United States: a systematic review","volume":"8","author":"Neshan","year":"2024","journal-title":"J Clin Transl Sci"},{"key":"2026080604264425200_ooag147-B9","doi-asserted-by":"publisher","first-page":"e1911514","DOI":"10.1001\/jamanetworkopen.2019.11514","article-title":"Prevalence of screening for food insecurity, housing instability, utility needs, transportation needs, and interpersonal violence by US physician practices and hospitals","volume":"2","author":"Fraze","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2026080604264425200_ooag147-B10","doi-asserted-by":"publisher","first-page":"e2348646","DOI":"10.1001\/jamanetworkopen.2023.48646","article-title":"Screening for social determinants of health during primary care and emergency department encounters","volume":"6","author":"Vilendrer","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2026080604264425200_ooag147-B11","doi-asserted-by":"publisher","first-page":"342","DOI":"10.1186\/s12913-025-12494-2","article-title":"Impact of hospital and health system initiatives to address social determinants of health (SDOH) in the United States: a scoping review of the peer-reviewed literature","volume":"25","author":"Rangachari","year":"2025","journal-title":"BMC Health Serv Res"},{"key":"2026080604264425200_ooag147-B12","doi-asserted-by":"publisher","first-page":"104642","DOI":"10.1016\/j.jbi.2024.104642","article-title":"Identifying social determinants of health from clinical narratives: a study of performance, documentation ratio, and potential bias","volume":"153","author":"Yu","year":"2024","journal-title":"J Biomed Inform"},{"key":"2026080604264425200_ooag147-B13","doi-asserted-by":"publisher","first-page":"1438","DOI":"10.1093\/jamia\/ocad054","article-title":"Automatic extraction of social determinants of health from medical notes of chronic lower back pain patients","volume":"30","author":"Lituiev","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"2026080604264425200_ooag147-B14","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1093\/jamia\/ocx132","article-title":"CLAMP\u2014a toolkit for efficiently building customized clinical natural language processing pipelines","volume":"25","author":"Soysal","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2026080604264425200_ooag147-B15","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2026080604264425200_ooag147-B16","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1080\/14737167.2023.2275670","article-title":"Natural language processing\u2013relevance to patient outcomes and real-world evidence","volume":"24","author":"Stewart","year":"2024","journal-title":"Expert Rev Pharmacoecon Outcomes Res"},{"key":"2026080604264425200_ooag147-B17","first-page":"438","article-title":"Launching into clinical space with medspaCy: a new clinical text processing toolkit in python","volume":"2021","author":"Eyre","year":"2021","journal-title":"AMIA Annu Symp Proc"},{"key":"2026080604264425200_ooag147-B18","author":"UF Health","year":"2025"},{"key":"2026080604264425200_ooag147-B19","author":"UF Health","year":"2025"},{"key":"2026080604264425200_ooag147-B20","volume-title":"Social Determinants of Health Discussion Paper 2","author":"Solar","year":"2010"},{"key":"2026080604264425200_ooag147-B21","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1111\/1468-0009.12390","article-title":"Meanings and misunderstandings: a social determinants of health lexicon for health care systems","volume":"97","author":"Alderwick","year":"2019","journal-title":"Milbank Q"},{"key":"2026080604264425200_ooag147-B22","author":"PRAPARE","year":"2025"},{"key":"2026080604264425200_ooag147-B23","author":"National Academy of Medicine","year":"2025"},{"key":"2026080604264425200_ooag147-B24","first-page":"588","article-title":"Brief questions to identify patients with inadequate health literacy","volume":"36","author":"Chew","year":"2004","journal-title":"Fam Med"},{"key":"2026080604264425200_ooag147-B25","doi-asserted-by":"publisher","first-page":"1284","DOI":"10.1097\/01.MLR.0000093487.78664.3C","article-title":"The patient health questionnaire-2: validity of a two-item depression screener","volume":"41","author":"Kroenke","year":"2003","journal-title":"Med Care"},{"key":"2026080604264425200_ooag147-B26","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1186\/s12874-021-01419-2","article-title":"Response rate of patient reported outcomes: the delivery method matters","volume":"21","author":"Neve","year":"2021","journal-title":"BMC Med Res Methodol"},{"key":"2026080604264425200_ooag147-B27","doi-asserted-by":"publisher","first-page":"e0303653","DOI":"10.1371\/journal.pone.0303653","article-title":"Identifying stigmatizing language in clinical documentation: a scoping review of emerging literature","volume":"19","author":"Barcelona","year":"2024","journal-title":"PLoS One"},{"key":"2026080604264425200_ooag147-B28","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1007\/s10916-022-01837-9","article-title":"The impact of structured and standardized documentation on documentation quality: a multicenter, retrospective study","volume":"46","author":"Ebbers","year":"2022","journal-title":"J Med Syst"},{"key":"2026080604264425200_ooag147-B29","doi-asserted-by":"publisher","first-page":"1163","DOI":"10.1016\/j.amepre.2023.06.006","article-title":"Assessing area-level deprivation as a proxy for individual-level social risks","volume":"65","author":"Brown","year":"2023","journal-title":"Am J Prev Med"},{"key":"2026080604264425200_ooag147-B30","doi-asserted-by":"publisher","first-page":"1237","DOI":"10.3758\/s13428-014-0533-4","article-title":"Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis","volume":"47","author":"Gnambs","year":"2015","journal-title":"Behav Res Methods"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/4\/ooag147\/70422467\/ooag147.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/4\/ooag147\/70422467\/ooag147.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T08:26:48Z","timestamp":1786004808000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooag147\/8748658"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,7]]},"references-count":30,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,7,7]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooag147","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,8]]},"published":{"date-parts":[[2026,7,7]]},"article-number":"ooag147"}}