{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T14:38:26Z","timestamp":1787063906181,"version":"build-2736575974"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2022,2,16]],"date-time":"2022-02-16T00:00:00Z","timestamp":1644969600000},"content-version":"vor","delay-in-days":40,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Johns Hopkins Institute for Clinical and Translational Research","award":["UL1 TR003098"],"award-info":[{"award-number":["UL1 TR003098"]}]},{"name":"National Center for Advancing Translational Sciences (NCATS) a component of the National Institutes of Health"},{"name":"NIH Roadmap for Medical Research"},{"name":"Johns Hopkins ICTR"},{"DOI":"10.13039\/100006108","name":"NCATS","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,1,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate whether a natural language processing (NLP) algorithm could be adapted to extract, with acceptable validity, markers of residential instability (ie, homelessness and housing insecurity) from electronic health records (EHRs) of 3 healthcare systems.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>We included patients 18 years and older who received care at 1 of 3 healthcare systems from 2016 through 2020 and had at least 1 free-text note in the EHR during this period. We conducted the study independently; the NLP algorithm logic and method of validity assessment were identical across sites. The approach to the development of the gold standard for assessment of validity differed across sites. Using the EntityRuler module of spaCy 2.3 Python toolkit, we created a rule-based NLP system made up of expert-developed patterns indicating residential instability at the lead site and enriched the NLP system using insight gained from its application at the other 2 sites. We adapted the algorithm at each site then validated the algorithm using a split-sample approach. We assessed the performance of the algorithm by measures of positive predictive value (precision), sensitivity (recall), and specificity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The NLP algorithm performed with moderate precision (0.45, 0.73, and 1.0) at 3 sites. The sensitivity and specificity of the NLP algorithm varied across 3 sites (sensitivity: 0.68, 0.85, and 0.96; specificity: 0.69, 0.89, and 1.0).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The performance of this NLP algorithm to identify residential instability in 3 different healthcare systems suggests the algorithm is generally valid and applicable in other healthcare systems with similar EHRs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The NLP approach developed in this project is adaptable and can be modified to extract types of social needs other than residential instability from EHRs across different healthcare systems.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooac006","type":"journal-article","created":{"date-parts":[[2022,2,16]],"date-time":"2022-02-16T13:32:19Z","timestamp":1645018339000},"source":"Crossref","is-referenced-by-count":29,"title":["Development and assessment of a natural language processing model to identify residential instability in electronic health records\u2019 unstructured data: a comparison of 3 integrated healthcare delivery systems"],"prefix":"10.1093","volume":"5","author":[{"given":"Elham","family":"Hatef","sequence":"first","affiliation":[{"name":"Center for Population Health Information Technology, Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masoud","family":"Rouhizadeh","sequence":"additional","affiliation":[{"name":"Institute for Clinical and Translational Research, Johns Hopkins Medical Institute, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claudia","family":"Nau","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Southern Caifornia, Pasadena, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fagen","family":"Xie","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Southern Caifornia, Pasadena, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Rouillard","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Mid-Atlantic States, Rockville, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahmoud","family":"Abu-Nasser","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Mid-Atlantic States, Rockville, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ariadna","family":"Padilla","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Southern Caifornia, Pasadena, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lindsay Joe","family":"Lyons","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Southern Caifornia, Pasadena, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hadi","family":"Kharrazi","sequence":"additional","affiliation":[{"name":"Center for Population Health Information Technology, Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"},{"name":"Department of Medicine Division of Health Sciences Informatics, Johns Hopkins School of Medicine, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan P","family":"Weiner","sequence":"additional","affiliation":[{"name":"Center for Population Health Information Technology, Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Douglas","family":"Roblin","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Mid-Atlantic States, Rockville, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,2,16]]},"reference":[{"key":"2022021918481291700_ooac006-B1","author":"Castrucci","year":"2019"},{"issue":"5","key":"2022021918481291700_ooac006-B2","first-page":"262","article-title":"Budget impact of a comprehensive nutrition-focused quality improvement program for malnourished hospitalized patients","volume":"10","author":"Sulo","year":"2017","journal-title":"Am Health Drug Benefits"},{"issue":"6","key":"2022021918481291700_ooac006-B3","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1089\/pop.2017.0199","article-title":"Expenditure reductions associated with a social service referral program","volume":"21","author":"Pruitt","year":"2018","journal-title":"Popul Health Manag"},{"key":"2022021918481291700_ooac006-B4","author":"The HI-5 Interventions | Health Impact in 5 Years | Health System Transformation | AD for Policy | CDC","year":"2021"},{"key":"2022021918481291700_ooac006-B5","author":"The de Beaumont Foundation and Kaiser Permanente","year":"2021"},{"key":"2022021918481291700_ooac006-B6","volume-title":"Capturing Social and Behavioral Domains and Measures in Electronic Health Records: Phase 2","author":"Institute of Medicine (US)","year":"2014"},{"issue":"2","key":"2022021918481291700_ooac006-B7","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1377\/hlthaff.21.2.60","article-title":"Socioeconomic disparities in health: pathways and policies","volume":"21","author":"Adler","year":"2002","journal-title":"Health Aff (Millwood)"},{"issue":"16","key":"2022021918481291700_ooac006-B8","doi-asserted-by":"crossref","first-page":"1750","DOI":"10.1001\/jama.2016.4226","article-title":"The association between income and life expectancy in the United States, 2001-2014","volume":"315","author":"Chetty","year":"2016","journal-title":"JAMA"},{"key":"2022021918481291700_ooac006-B9","author":"National Center for Chronic Disease Prevention and Health Promotion (US) Office on Smoking and Health","year":"1970"},{"issue":"12","key":"2022021918481291700_ooac006-B10","doi-asserted-by":"crossref","first-page":"1010","DOI":"10.1097\/MLR.0000000000000436","article-title":"Predictors of emergency department visits and inpatient admissions among homeless and unstably housed adolescents and young adults","volume":"53","author":"Mackelprang","year":"2015","journal-title":"Med Care"},{"issue":"1","key":"2022021918481291700_ooac006-B11","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1111\/acem.13060","article-title":"Effectiveness of interventions to decrease emergency department visits by adult frequent users: a systematic review","volume":"24","author":"Moe","year":"2017","journal-title":"Acad Emerg Med"},{"issue":"1","key":"2022021918481291700_ooac006-B12","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1377\/hlthaff.2015.0393","article-title":"Formerly homeless people had lower overall health care expenditures after moving into supportive housing","volume":"35","author":"Wright","year":"2016","journal-title":"Health Aff (Millwood)"},{"issue":"2","key":"2022021918481291700_ooac006-B13","doi-asserted-by":"crossref","first-page":"1110","DOI":"10.1111\/1475-6773.12670","article-title":"Hospital readmission and social risk factors identified from physician notes","volume":"53","author":"Navathe","year":"2018","journal-title":"Health Serv Res"},{"issue":"1S1","key":"2022021918481291700_ooac006-B14","doi-asserted-by":"crossref","first-page":"S40","DOI":"10.1016\/j.amepre.2016.09.029","article-title":"Street connectivity and obesity risk: evidence from electronic health records","volume":"52","author":"Leonardi","year":"2017","journal-title":"Am J Prev Med"},{"issue":"1","key":"2022021918481291700_ooac006-B15","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1186\/1471-2296-5-29","article-title":"Walkability and self-rated health in primary care patients","volume":"5","author":"Rohrer","year":"2004","journal-title":"BMC Fam Pract"},{"issue":"3","key":"2022021918481291700_ooac006-B16","doi-asserted-by":"crossref","first-page":"1600","DOI":"10.1111\/1475-6773.12730","article-title":"Food insecurity and health care expenditures in the United States, 2011-2013","volume":"53","author":"Berkowitz","year":"2018","journal-title":"Health Serv Res"},{"issue":"1","key":"2022021918481291700_ooac006-B17","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1111\/j.1525-1497.2005.00278.x","article-title":"Housing instability and food insecurity as barriers to health care among low-income Americans","volume":"21","author":"Kushel","year":"2006","journal-title":"J Gen Intern Med"},{"issue":"3","key":"2022021918481291700_ooac006-B18","doi-asserted-by":"crossref","first-page":"e13802","DOI":"10.2196\/13802","article-title":"Assessing the availability of data on social and behavioral determinants in structured and unstructured electronic health records: a retrospective analysis of a multilevel health care system","volume":"7","author":"Hatef","year":"2019","journal-title":"JMIR Med Inform"},{"issue":"9","key":"2022021918481291700_ooac006-B19","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1097\/MLR.0000000000001173","article-title":"Considerations for identifying social needs in health care systems: a commentary on the role of predictive models in supporting a comprehensive social needs strategy","volume":"57","author":"Nau","year":"2019","journal-title":"Med Care"},{"issue":"8","key":"2022021918481291700_ooac006-B20","doi-asserted-by":"crossref","first-page":"1499","DOI":"10.1111\/jgs.15411","article-title":"The value of unstructured electronic health record data in geriatric syndrome case identification","volume":"66","author":"Kharrazi","year":"2018","journal-title":"J Am Geriatr Soc"},{"issue":"1","key":"2022021918481291700_ooac006-B21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12877-017-0645-7","article-title":"Comparing clinician descriptions of frailty and geriatric syndromes using electronic health records: a retrospective cohort study","volume":"17","author":"Anzaldi","year":"2017","journal-title":"BMC Geriatr"},{"issue":"1","key":"2022021918481291700_ooac006-B22","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1186\/s13326-019-0198-0","article-title":"Moonstone: a novel natural language processing system for inferring social risk from clinical narratives","volume":"10","author":"Conway","year":"2019","journal-title":"J Biomed Semantics"},{"key":"2022021918481291700_ooac006-B23","first-page":"1456","article-title":"Identifying patients with significant problems related to social determinants of health with natural language processing","volume":"264","author":"Dorr","year":"2019","journal-title":"Stud Health Technol Inform"},{"key":"2022021918481291700_ooac006-B24","doi-asserted-by":"crossref","first-page":"2716","DOI":"10.1093\/jamia\/ocab170","article-title":"Extracting social determinants of health from electronic health records using natural language processing: a systematic review","author":"Patra","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2022021918481291700_ooac006-B25","author":"Arons","year":"2018"},{"key":"2022021918481291700_ooac006-B26","first-page":"221","article-title":"Enrich classifications in psychiatry with textual data: an ontology for psychiatry including social concepts","volume":"210","author":"Richard","year":"2015","journal-title":"Stud Health Technol Inform"},{"key":"2022021918481291700_ooac006-B27","author":"The United States Census Bureau","year":"2021"},{"key":"2022021918481291700_ooac006-B28","year":"2021"},{"key":"2022021918481291700_ooac006-B29","author":"National Association of Community Health Centers","year":"2021"},{"issue":"1","key":"2022021918481291700_ooac006-B30","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1056\/NEJMp1512532","article-title":"Accountable Health Communities\u2014addressing social needs through Medicare and Medicaid","volume":"374","author":"Alley","year":"2016","journal-title":"N Engl J Med"},{"key":"2022021918481291700_ooac006-B31","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"2022021918481291700_ooac006-B32","author":"Microsoft Excel","year":"2021"},{"key":"2022021918481291700_ooac006-B33","first-page":"63","author":"Loper","year":"2002"},{"issue":"1","key":"2022021918481291700_ooac006-B34","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1055\/s-0040-1702214","article-title":"Detecting social and behavioral determinants of health with structured and free-text clinical data","volume":"11","author":"Feller","year":"2020","journal-title":"Appl Clin Inform"},{"key":"2022021918481291700_ooac006-B35","author":"Honnibal","year":"2017"},{"issue":"1","key":"2022021918481291700_ooac006-B36","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1197\/jamia.M3378","article-title":"MedEx: a medication information extraction system for clinical narratives","volume":"17","author":"Xu","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"e2","key":"2022021918481291700_ooac006-B37","doi-asserted-by":"crossref","first-page":"e355-64","DOI":"10.1136\/amiajnl-2013-001946","article-title":"Validating a strategy for psychosocial phenotyping using a large corpus of clinical text","volume":"20","author":"Gundlapalli","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2022021918481291700_ooac006-B38","first-page":"270","article-title":"A broad-coverage natural language processing system","author":"Friedman","year":"2000","journal-title":"Proc AMIA Symp"},{"issue":"1","key":"2022021918481291700_ooac006-B39","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s12911-017-0418-4","article-title":"Early recognition of multiple sclerosis using natural language processing of the electronic health record","volume":"17","author":"Chase","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"2022021918481291700_ooac006-B40","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2022021918481291700_ooac006-B41","first-page":"537","article-title":"Using natural language processing on the free text of clinical documents to screen for evidence of homelessness among US veterans","volume":"2013","author":"Gundlapalli","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"key":"2022021918481291700_ooac006-B42","first-page":"589","article-title":"Extracting concepts related to homelessness from the free text of VA electronic medical records","volume":"2014","author":"Gundlapalli","year":"2014","journal-title":"AMIA Annu Symp Proc"},{"issue":"3-4","key":"2022021918481291700_ooac006-B43","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1017\/S1351324904003523","article-title":"UIMA: an architectural approach to unstructured information processing in the corporate research environment","volume":"10","author":"Ferrucci","year":"2004","journal-title":"Nat Lang Eng"},{"key":"2022021918481291700_ooac006-B44","first-page":"230","article-title":"Development and performance of text-mining algorithms to extract socioeconomic status from de-identified electronic health records","volume":"22","author":"Hollister","year":"2017","journal-title":"Pac Symp Biocomput"},{"issue":"1","key":"2022021918481291700_ooac006-B45","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1093\/jamia\/ocx059","article-title":"Mining 100 million notes to find homelessness and adverse childhood experiences: 2 case studies of rare and severe social determinants of health in electronic health records","volume":"25","author":"Bejan","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2022021918481291700_ooac006-B46","first-page":"1228","article-title":"v3NLP framework: tools to build applications for extracting concepts from clinical text","volume":"3","author":"Divita","year":"2016","journal-title":"J Electron Health Data Methods"},{"key":"2022021918481291700_ooac006-B47","first-page":"149","article-title":"Recognizing questions and answers in EMR templates using natural language processing","volume":"202","author":"Divita","year":"2014","journal-title":"Stud Health Technol Inform"},{"key":"2022021918481291700_ooac006-B48","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1186\/s12911-017-0468-7","article-title":"Entity recognition from clinical texts via recurrent neural network","volume":"17 (Suppl 2","author":"Liu","year":"2017","journal-title":"BMC Med Inform Decis Mak"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/5\/1\/ooac006\/42559715\/ooac006.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/5\/1\/ooac006\/42559715\/ooac006.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,19]],"date-time":"2022-02-19T13:48:49Z","timestamp":1645278529000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooac006\/6529555"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,7]]},"references-count":48,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1,7]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooac006","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,4,1]]},"published":{"date-parts":[[2022,1,7]]},"article-number":"ooac006"}}