{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T16:47:04Z","timestamp":1786639624966,"version":"build-2736575974"},"reference-count":35,"publisher":"National Academy of Sciences","issue":"39","license":[{"start":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T00:00:00Z","timestamp":1726444800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2024,9,24]]},"abstract":"<jats:p>The assessment of social determinants of health (SDoH) within healthcare systems is crucial for comprehensive patient care and addressing health disparities. Current challenges arise from the limited inclusion of structured SDoH information within electronic health record (EHR) systems, often due to the lack of standardized diagnosis codes. This study delves into the transformative potential of large language models (LLM) to overcome these challenges. LLM-based classifiers\u2014using Bidirectional Encoder Representations from Transformers (BERT) and A Robustly Optimized BERT Pretraining Approach (RoBERTa)\u2014were developed for SDoH concepts, including homelessness, food insecurity, and domestic violence, using synthetic training datasets generated by generative pre-trained transformers combined with authentic clinical notes. Models were then validated on separate datasets: Medical Information Mart for Intensive Care-III and our institutional EHR data. When training the model with a combination of synthetic and authentic notes, validation on our institutional dataset yielded an area under the receiver operating characteristics curve of 0.78 for detecting homelessness, 0.72 for detecting food insecurity, and 0.83 for detecting domestic violence. This study underscores the potential of LLMs in extracting SDoH information from clinical text. Automated detection of SDoH may be instrumental for healthcare providers in identifying at-risk patients, guiding targeted interventions, and contributing to population health initiatives aimed at mitigating disparities.<\/jats:p>","DOI":"10.1073\/pnas.2320716121","type":"journal-article","created":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T15:34:47Z","timestamp":1726500887000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":24,"title":["On the development and validation of large language model-based classifiers for identifying social determinants of health"],"prefix":"10.1073","volume":"121","author":[{"given":"Rodney A.","family":"Gabriel","sequence":"first","affiliation":[{"name":"Division of Perioperative Informatics, Department of Anesthesiology, University of California, San Diego, La Jolla, CA 92037"},{"name":"Department of Biomedical Informatics, University of California, San Diego Health, La Jolla, CA 92037"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Onkar","family":"Litake","sequence":"additional","affiliation":[{"name":"Division of Perioperative Informatics, Department of Anesthesiology, University of California, San Diego, La Jolla, CA 92037"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1352-0617","authenticated-orcid":false,"given":"Sierra","family":"Simpson","sequence":"additional","affiliation":[{"name":"Division of Perioperative Informatics, Department of Anesthesiology, University of California, San Diego, La Jolla, CA 92037"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brittany N.","family":"Burton","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, University of California, Los Angeles, CA 90095"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7825-3142","authenticated-orcid":false,"given":"Ruth S.","family":"Waterman","sequence":"additional","affiliation":[{"name":"Division of Perioperative Informatics, Department of Anesthesiology, University of California, San Diego, La Jolla, CA 92037"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2276-9131","authenticated-orcid":false,"given":"Alvaro A.","family":"Macias","sequence":"additional","affiliation":[{"name":"Division of Perioperative Informatics, Department of Anesthesiology, University of California, San Diego, La Jolla, CA 92037"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2024,9,16]]},"reference":[{"key":"e_1_3_4_1_2","unstructured":"CMS finalizes physician payment rule that advances health equity (2023). https:\/\/www.cms.gov\/newsroom\/press-releases\/cms-finalizes-physician-payment-rule-advances-health-equity. Accessed 1 November 2023."},{"key":"e_1_3_4_2_2","unstructured":"Department of Health and Human Services Social determinants of health (2023). https:\/\/health.gov\/healthypeople\/priority-areas\/social-determinants-health. Accessed 1 November 2023."},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa143"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-022-00742-2"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1136\/rapm-2023-104637"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocab170"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1186\/s44247-023-00035-y"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1111\/1475-6773.14210"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1177\/00333549141291S206"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCOUTCOMES.109.880377"},{"key":"e_1_3_4_11_2","unstructured":"U.S. Health in International Perspective: Shorter Lives Poorer Health The National Academies Collection: Reports funded by National Institutes of Health S. H. Woolf L. Aron Eds. (2013)."},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0000000000002254"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.35"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.arth.2023.08.077"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00402-023-05045-z"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.amepre.2015.08.024"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.2105\/AJPH.2017.304000"},{"key":"e_1_3_4_18_2","unstructured":"A. Baciu Communities in Action: Pathways to Health Equity. Vol. The Root Causes of Health Inequity (National Academies Press Washington DC 2017)."},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.2196\/46159"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103984"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1055\/s-0040-1702214"},{"key":"e_1_3_4_22_2","first-page":"1","article-title":"A clinical text classification paradigm using weak supervision and deep representation","volume":"19","author":"Wang Y.","year":"2019","unstructured":"Y. Wang , A clinical text classification paradigm using weak supervision and deep representation. BMC Med. Inf. 19, 1 (2019), 10.1186\/s12911-018-0723-6.","journal-title":"BMC Med. Inf."},{"key":"e_1_3_4_23_2","first-page":"422","article-title":"Towards the inference of social and behavioral determinants of sexual health: Development of a gold-standard corpus with semi-supervised learning","volume":"2018","author":"Feller D. J.","year":"2018","unstructured":"D. J. Feller , Towards the inference of social and behavioral determinants of sexual health: Development of a gold-standard corpus with semi-supervised learning. AMIA Annu. Symp. Proc. 2018, 422\u2013429 (2018).","journal-title":"AMIA Annu. Symp. Proc."},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2020.103631"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaa069"},{"key":"e_1_3_4_26_2","unstructured":"J. Devlin M. W. Chang K. Lee K. Toutanova BERT: Pre-training of deep bidirectional transformers for langauge understanding. arXiv [Preprint] (2018). https:\/\/doi.org\/10.48550\/arXiv.1810.04805 (Accessed 1 November 2023)."},{"key":"e_1_3_4_27_2","unstructured":"Y. Liu RoBERTa: A robustly optimized BERT pretraining approach. arXiv [Preprint] (2019). https:\/\/doi.org\/10.48550\/arXiv.1907.11692 (Accessed 1 November 2023)."},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1111\/1468-0009.12390"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1002\/aet2.10056"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11524-007-9168-4"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1111\/jrh.12335"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2015.07.020"},{"key":"e_1_3_4_33_2","unstructured":"O. Litake R. Gabriel SDoHLLM. Github. https:\/\/github.com\/UCSDGabrielLab\/SDoHLLM. Deposited 7 May 2024."},{"key":"e_1_3_4_34_2","unstructured":"V. Kumar A. Stubbs S. Shaw \u00d6. Uzuner n2c2 NLP research data sets. DBMI Data Portal. https:\/\/portal.dbmi.hms.harvard.edu\/projects\/n2c2-nlp\/. Accessed 19 July 2023."},{"key":"e_1_3_4_35_2","unstructured":"A. Johnson T. Pollard R. Mark MIMIC-III Clinical database. PhysioNet. https:\/\/physionet.org\/content\/mimiciii\/1.4\/. Accessed 1 February 2023."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2320716121","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T15:36:41Z","timestamp":1726501001000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2320716121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,16]]},"references-count":35,"journal-issue":{"issue":"39","published-print":{"date-parts":[[2024,9,24]]}},"alternative-id":["10.1073\/pnas.2320716121"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2320716121","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,16]]},"assertion":[{"value":"2023-11-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2320716121"}}