{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T19:00:22Z","timestamp":1784919622145,"version":"3.55.0"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T00:00:00Z","timestamp":1728518400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T00:00:00Z","timestamp":1728518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Social and behavioral determinants of health (SBDH) are associated with a variety of health and utilization outcomes, yet these factors are not routinely documented in the structured fields of electronic health records (EHR). The objective of this study was to evaluate different machine learning approaches for detection of SBDH from the unstructured clinical notes in the EHR.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Latent Semantic Indexing (LSI) was applied to 2,083,180 clinical notes corresponding to 46,146 patients in the MIMIC-III dataset. Using LSI, patients were ranked based on conceptual relevance to a set of keywords (lexicons) pertaining to 15 different SBDH categories. For Generative Pretrained Transformer (GPT) models, API requests were made with a Python script to connect to the OpenAI services in Azure, using  and  models. Prediction of SBDH categories were performed using a logistic regression model that included age, gender, race and SBDH ICD-9 codes.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      LSI retrieved patients according to 15 SBDH domains, with an overall average PPV\n                      <jats:inline-formula>\n                        <jats:alternatives>\n                          <jats:tex-math>$$\\ge$$<\/jats:tex-math>\n                          <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                            <mml:mo>\u2265<\/mml:mo>\n                          <\/mml:math>\n                        <\/jats:alternatives>\n                      <\/jats:inline-formula>\n                      83%. Using manually curated gold standard (GS) sets for nine SBDH categories, the macro-F1 score of LSI (0.74) was better than ICD-9 (0.71) and GPT-3.5 (0.54), but lower than GPT-4 (0.80). Due to document size limitations, only a subset of the GS cases could be processed by GPT-3.5 (55.8%) and GPT-4 (94.2%), compared to LSI (100%). Using common GS subsets for nine different SBDH categories, the macro-F1 of ICD-9 combined with either LSI (mean 0.88, 95% CI 0.82-0.93), GPT-3.5 (0.86, 0.82-0.91) or GPT-4 (0.88, 0.83-0.94) was not significantly different. After including age, gender, race and ICD-9 in a logistic regression model, the AUC for prediction of six out of the nine SBDH categories was higher for LSI compared to GPT-4.0.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>These results demonstrate that the LSI approach performs comparable to more recent large language models, such as GPT-3.5 and GPT-4.0, when using the same set of documents. Importantly, LSI is robust, deterministic, and does not have document-size limitations or cost implications, which make it more amenable to real-world applications in health systems.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12911-024-02705-x","type":"journal-article","created":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T05:02:19Z","timestamp":1728536539000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Large-scale identification of social and behavioral determinants of health from clinical notes: comparison of Latent Semantic Indexing and Generative Pretrained Transformer (GPT) models"],"prefix":"10.1186","volume":"24","author":[{"given":"Sujoy","family":"Roy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shane","family":"Morrell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lili","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ramin","family":"Homayouni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,10,10]]},"reference":[{"issue":"11","key":"2705_CR1","doi-asserted-by":"publisher","first-page":"1764","DOI":"10.1093\/jamia\/ocaa143","volume":"27","author":"M Chen","year":"2020","unstructured":"Chen M, Tan X, Padman R. Social determinants of health in electronic health records and their impact on analysis and risk prediction: A systematic review. J Am Med Inform Assoc. 2020;27(11):1764\u201373. https:\/\/doi.org\/10.1093\/jamia\/ocaa143.","journal-title":"J Am Med Inform Assoc."},{"key":"2705_CR2","doi-asserted-by":"publisher","unstructured":"Tan M, Hatef E, Taghipour D, Vyas K, Kharrazi H, Gottlieb L, et\u00a0al. Including social and behavioral determinants in predictive models: Trends, challenges, and opportunities. JMIR Med Inform. 2020;8(9). https:\/\/doi.org\/10.2196\/18084.","DOI":"10.2196\/18084"},{"key":"2705_CR3","doi-asserted-by":"publisher","unstructured":"Guo Y, Chen Z, Xu K, George TJ, Wu Y, Hogan W, et\u00a0al. International Classification of Diseases, Tenth Revision, Clinical Modification social determinants of health codes are poorly used in electronic health records. Medicine (United States). 2020;99(52). https:\/\/doi.org\/10.1097\/MD.0000000000023818.","DOI":"10.1097\/MD.0000000000023818"},{"key":"2705_CR4","doi-asserted-by":"publisher","unstructured":"Andermann A. Screening for social determinants of health in clinical care: Moving from the margins to the mainstream. Public Health Rev. 2018;39(1). https:\/\/doi.org\/10.1186\/s40985-018-0094-7.","DOI":"10.1186\/s40985-018-0094-7"},{"key":"2705_CR5","doi-asserted-by":"publisher","unstructured":"Alpert J, Kim H, McDonnell C, Guo Y, George TJ, Bian J, et\u00a0al. Barriers and Facilitators of Obtaining Social Determinants of Health of Patients With Cancer Through the Electronic Health Record Using Natural Language Processing Technology: Qualitative Feasibility Study With Stakeholder Interviews. JMIR Formative Res. 2022;6(12). https:\/\/doi.org\/10.2196\/43059.","DOI":"10.2196\/43059"},{"issue":"2","key":"2705_CR6","doi-asserted-by":"publisher","first-page":"1110","DOI":"10.1111\/1475-6773.12670","volume":"53","author":"AS Navathe","year":"2018","unstructured":"Navathe AS, Zhong F, Lei VJ, Chang FY, Sordo M, Topaz M, et al. Hospital Readmission and Social Risk Factors Identified from Physician Notes. Health Serv Res. 2018;53(2):1110\u201336. https:\/\/doi.org\/10.1111\/1475-6773.12670.","journal-title":"Health Serv Res."},{"key":"2705_CR7","doi-asserted-by":"publisher","unstructured":"Hatef E, Rouhizadeh M, Tia I, Lasser E, Hill-Briggs F, Marsteller J, et\u00a0al. Assessing the availability of data on social and behavioral determinants in structured and unstructured electronic health records: A retrospective analysis of a multilevel health care system. J Med Internet Res. 2019;21(8). https:\/\/doi.org\/10.2196\/13802.","DOI":"10.2196\/13802"},{"issue":"12","key":"2705_CR8","doi-asserted-by":"publisher","first-page":"2716","DOI":"10.1093\/JAMIA\/OCAB170","volume":"28","author":"BG Patra","year":"2021","unstructured":"Patra BG, Sharma MM, Vekaria V, Adekkanattu P, Patterson OV, Glicksberg B, et al. Extracting social determinants of health from electronic health records using natural language processing: a systematic review. J Am Med Inform Assoc. 2021;28(12):2716\u201327. https:\/\/doi.org\/10.1093\/JAMIA\/OCAB170.","journal-title":"J Am Med Inform Assoc."},{"issue":"8","key":"2705_CR9","doi-asserted-by":"publisher","first-page":"1363","DOI":"10.1093\/JAMIA\/OCAD121","volume":"30","author":"K Lybarger","year":"2023","unstructured":"Lybarger K, Bear OJ, Yetisgen M, Uzuner O. Advancements in extracting social determinants of health information from narrative text. J Am Med Inform Assoc. 2023;30(8):1363\u20136. https:\/\/doi.org\/10.1093\/JAMIA\/OCAD121.","journal-title":"J Am Med Inform Assoc."},{"key":"2705_CR10","doi-asserted-by":"publisher","unstructured":"Allen KS, Hood DR, Cummins J, Kasturi S, Mendonca EA, Vest JR. Natural language processing-driven state machines to extract social factors from unstructured clinical documentation. JAMIA Open. 2023;6(2). https:\/\/doi.org\/10.1093\/JAMIAOPEN\/OOAD024.","DOI":"10.1093\/JAMIAOPEN\/OOAD024"},{"key":"2705_CR11","doi-asserted-by":"publisher","unstructured":"Mehta S, Lyles C, Rubinsky A, Kemper K, Auerbach J, Sarkar U, et\u00a0al. Social Determinants of Health Documentation in Structured and Unstructured Clinical Data of Patients With Diabetes: Comparative Analysis. JMIR Med Inform. 2023;11. https:\/\/doi.org\/10.2196\/46159.","DOI":"10.2196\/46159"},{"issue":"8","key":"2705_CR12","doi-asserted-by":"publisher","first-page":"1389","DOI":"10.1093\/JAMIA\/OCAD073","volume":"30","author":"K Lybarger","year":"2023","unstructured":"Lybarger K, Dobbins NJ, Long R, Singh A, Wedgeworth P, Uzuner O, et al. Leveraging natural language processing to augment structured social determinants of health data in the electronic health record. J Am Med Inform Assoc. 2023;30(8):1389\u201397. https:\/\/doi.org\/10.1093\/JAMIA\/OCAD073.","journal-title":"J Am Med Inform Assoc."},{"issue":"113","key":"2705_CR13","doi-asserted-by":"publisher","first-page":"103631","DOI":"10.1016\/j.jbi.2020.103631","volume":"2021","author":"K Lybarger","year":"2020","unstructured":"Lybarger K, Ostendorf M, Yetisgen M. Annotating social determinants of health using active learning, and characterizing determinants using neural event extraction. J Biomed Inform. 2020;2021(113):103631. https:\/\/doi.org\/10.1016\/j.jbi.2020.103631.","journal-title":"J Biomed Inform."},{"key":"2705_CR14","first-page":"1225","volume":"2021","author":"Z Yu","year":"2021","unstructured":"Yu Z, Yang X, Dang C, Wu S, Adekkanattu P, Pathak J, et al. A Study of Social and Behavioral Determinants of Health in Lung Cancer Patients Using Transformers-based Natural Language Processing Models. AMIA Ann Symp Proc. 2021;2021:1225.","journal-title":"AMIA Ann Symp Proc."},{"key":"2705_CR15","doi-asserted-by":"publisher","unstructured":"Yu Z, Yang X, Guo Y, Bian J, Wu Y. Assessing the Documentation of Social Determinants of Health for Lung Cancer Patients in Clinical Narratives. Front Public Health. 2022;10. https:\/\/doi.org\/10.3389\/FPUBH.2022.778463.","DOI":"10.3389\/FPUBH.2022.778463"},{"issue":"1","key":"2705_CR16","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1038\/s41746-023-00970-0","volume":"7","author":"M Guevara","year":"2024","unstructured":"Guevara M, Chen S, Thomas S, Chaunzwa TL, Franco I, Kann BH, et al. Large language models to identify social determinants of health in electronic health records. NPJ Digit Med. 2024;7(1):6.","journal-title":"NPJ Digit Med."},{"key":"2705_CR17","unstructured":"Fu Y, Ramachandran GK, Dobbins NJ, Park N, Leu M, Rosenberg AR, et al. Extracting social determinants of health from pediatric patient notes using large language models: novel corpus and methods. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).\u00a0Prague: International Committee for Computational Linguistics (ICCL); 2024. p. 7045\u201356."},{"key":"2705_CR18","doi-asserted-by":"crossref","unstructured":"Johnson AE, Pollard TJ, Shen L, Lehman LwH, Feng M, Ghassemi M, et\u00a0al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3(1):1\u20139.","DOI":"10.1038\/sdata.2016.35"},{"key":"2705_CR19","doi-asserted-by":"crossref","unstructured":"Zeimpekis D, Gallopoulos E. TMG: A MATLAB toolbox for generating term-document matrices from text collections. In: Grouping multidimensional data. Berlin: Springer; 2006. p. 187\u2013210.","DOI":"10.1007\/3-540-28349-8_7"},{"key":"2705_CR20","doi-asserted-by":"crossref","unstructured":"Salton G. The Smart document retrieval project. In: Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval.\u00a0Hoboken: Prentice Hall; 1991. p. 356\u20138.","DOI":"10.1145\/122860.122897"},{"issue":"1","key":"2705_CR21","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1093\/bioinformatics\/bth464","volume":"21","author":"R Homayouni","year":"2005","unstructured":"Homayouni R, Heinrich K, Wei L, Berry MW. Gene clustering by latent semantic indexing of MEDLINE abstracts. Bioinformatics. 2005;21(1):104\u201315.","journal-title":"Bioinformatics."},{"key":"2705_CR22","doi-asserted-by":"crossref","unstructured":"Heinrich KE, Berry MW, Homayouni R, et\u00a0al. Gene tree labeling using nonnegative matrix factorization on biomedical literature. Comput Intell Neurosci. 2008;2008(1):276535.","DOI":"10.1155\/2008\/276535"},{"key":"2705_CR23","doi-asserted-by":"crossref","unstructured":"Roy S, Heinrich K, Phan V, Berry MW, Homayouni R. Latent semantic indexing of PubMed abstracts for identification of transcription factor candidates from microarray derived gene sets. In: BMC bioinformatics. vol.\u00a012.\u00a0Heidelberg: Springer; 2011. pp. 1\u201313.","DOI":"10.1186\/1471-2105-12-S10-S19"},{"key":"2705_CR24","doi-asserted-by":"crossref","unstructured":"Roy S, Homayouni R, Berry MW, Puretskiy AA. Nonnegative tensor factorization of biomedical literature for analysis of genomic data. In: Data Mining for Service.\u00a0Berlin: Springer; 2014. pp. 97\u2013110.","DOI":"10.1007\/978-3-642-45252-9_7"},{"key":"2705_CR25","doi-asserted-by":"crossref","unstructured":"Roy S, Curry BC, Madahian B, Homayouni R. Prioritization, clustering and functional annotation of MicroRNAs using latent semantic indexing of MEDLINE abstracts. In: BMC bioinformatics. vol.\u00a017. BioMed Central; 2016. pp. 131\u201342.","DOI":"10.1186\/s12859-016-1223-2"},{"key":"2705_CR26","doi-asserted-by":"publisher","first-page":"48","DOI":"10.3389\/fbioe.2017.00048","volume":"5","author":"S Roy","year":"2017","unstructured":"Roy S, Yun D, Madahian B, Berry MW, Deng LY, Goldowitz D, et al. Navigating the functional landscape of transcription factors via non-negative tensor factorization analysis of MeDline abstracts. Front Bioeng Biotechnol. 2017;5:48.","journal-title":"Front Bioeng Biotechnol."},{"key":"2705_CR27","doi-asserted-by":"crossref","unstructured":"Roy S, Berry MW. Mining multimodal big data: tensor methods and applications. In: handbook of research on big data storage and visualization techniques.\u00a0Hershey, Pennsylvania USA: IGI Global; 2018. p. 674\u2013702.","DOI":"10.4018\/978-1-5225-3142-5.ch023"},{"key":"2705_CR28","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1186\/s12859-019-2621-z","volume":"20","author":"S Roy","year":"2019","unstructured":"Roy S, Zaman KI, Williams RW, Homayouni R. Evaluation of Sirtuin-3 probe quality and co-expressed genes using literature cohesion. BMC Bioinformatics. 2019;20:31\u201343.","journal-title":"BMC Bioinformatics."},{"key":"2705_CR29","doi-asserted-by":"publisher","unstructured":"Akbilgic O, Homayouni R, Heinrich K, Langham MR, Davis RL. Unstructured text in EMR improves prediction of death after surgery in children. Informatics. 2019;6(1). https:\/\/doi.org\/10.3390\/informatics6010004.","DOI":"10.3390\/informatics6010004"},{"issue":"9","key":"2705_CR30","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1097\/MLR.0000000000000764","volume":"55","author":"JM Torres","year":"2017","unstructured":"Torres JM, Lawlor J, Colvin JD, Sills MR, Bettenhausen JL, Davidson A, et al. ICD Social Codes: An underutilized resource for tracking social needs. Med Care. 2017;55(9):810\u20136. https:\/\/doi.org\/10.1097\/MLR.0000000000000764.","journal-title":"Med Care."},{"key":"2705_CR31","unstructured":"CMS. Chronic Conditions Data Warehouse. https:\/\/www2.ccwdata.org\/web\/guest\/home\/. Accessed 30 Oct 2023."},{"key":"2705_CR32","unstructured":"Microsoft. Microsoft Azure OpenAI. https:\/\/learn.microsoft.com\/en-us\/legal\/cognitive-services\/openai\/data-privacy. Accessed 10 Jan 2024."},{"issue":"5","key":"2705_CR33","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1001\/JAMA.2022.23631","volume":"329","author":"CA Harle","year":"2023","unstructured":"Harle CA, Wu W, Vest JR. Accuracy of Electronic Health Record Food Insecurity, Housing Instability, and Financial Strain Screening in Adult Primary Care. JAMA. 2023;329(5):423\u20134. https:\/\/doi.org\/10.1001\/JAMA.2022.23631.","journal-title":"JAMA."},{"issue":"6","key":"2705_CR34","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1097\/MLR.0000000000000364","volume":"53","author":"R Capp","year":"2015","unstructured":"Capp R, Camp-Binford M, Sobolewski S, Bulmer S, Kelley L. Do adult Medicaid enrollees prefer going to their primary care provider\u2019s clinic rather than emergency department (ED) for low acuity conditions? Med Care. 2015;53(6):530.","journal-title":"Med Care."},{"issue":"11","key":"2705_CR35","doi-asserted-by":"publisher","first-page":"e2345444","DOI":"10.1001\/jamanetworkopen.2023.45444","volume":"6","author":"AC Rudisill","year":"2023","unstructured":"Rudisill AC, Eicken MG, Gupta D, Macauda M, Self S, Kennedy AB, et al. Patient and Care Team Perspectives on Social Determinants of Health Screening in Primary Care: A Qualitative Study. JAMA Netw Open. 2023;6(11):e2345444\u2013e2345444.","journal-title":"JAMA Netw Open."},{"issue":"2","key":"2705_CR36","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1097\/QAI.0000000000001580","volume":"77","author":"DJ Feller","year":"2018","unstructured":"Feller DJ, Zucker J, Yin MT, Gordon P, Elhadad N. Using Clinical Notes and Natural Language Processing for Automated HIV Risk Assessment. J Acquir Immune Defic Syndr (1999). 2018;77(2):160\u20136. https:\/\/doi.org\/10.1097\/QAI.0000000000001580.","journal-title":"J Acquir Immune Defic Syndr (1999)."},{"issue":"1","key":"2705_CR37","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1055\/s-0040-1702214","volume":"11","author":"DJ Feller","year":"2020","unstructured":"Feller DJ, Bear OJ, Zucker J, Yin MT, Gordon P, Elhadad N. Detecting Social and Behavioral Determinants of Health with Structured and Free-Text Clinical Data. Appl Clin Inform. 2020;11(1):172\u201381. https:\/\/doi.org\/10.1055\/s-0040-1702214.","journal-title":"Appl Clin Inform."},{"issue":"1","key":"2705_CR38","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1093\/jamia\/ocx059","volume":"25","author":"CA Bejan","year":"2018","unstructured":"Bejan CA, Angiolillo J, Conway D, Nash R, Shirey-Rice JK, Lipworth L, et al. Mining 100 million notes to find homelessness and adverse childhood experiences: 2 case studies of rare and severe social determinants of health in electronic health records. J Am Med Inform Assoc. 2018;25(1):61\u201371. https:\/\/doi.org\/10.1093\/jamia\/ocx059.","journal-title":"J Am Med Inform Assoc."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02705-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-024-02705-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02705-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T17:02:53Z","timestamp":1728579773000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-024-02705-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,10]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["2705"],"URL":"https:\/\/doi.org\/10.1186\/s12911-024-02705-x","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.04.22.24306142","asserted-by":"object"}]},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,10]]},"assertion":[{"value":"21 March 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 September 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 October 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"R.H. and S.M. hold equity in Quire Inc.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"296"}}