{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T13:18:01Z","timestamp":1790774281506,"version":"4.1.0"},"reference-count":30,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T00:00:00Z","timestamp":1629331200000},"content-version":"vor","delay-in-days":49,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,7,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Clinical guidelines recommend annual eye examinations to detect diabetic retinopathy (DR) in patients with diabetes. However, timely DR detection remains a problem in medically underserved and under-resourced settings in the United States. Machine learning that identifies patients with latent\/undiagnosed DR could help to address this problem.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Using electronic health record data from 40 631 unique diabetic patients seen at Los Angeles County Department of Health Services healthcare facilities between January 1, 2015 and December 31, 2017, we compared ten machine learning environments, including five classifier models, for assessing the presence or absence of DR. We also used data from a distinct set of 9300 diabetic patients seen between January 1, 2018 and December 31, 2018 as an external validation set.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Following feature subset selection, the classifier with the best AUC on the external validation set was a deep neural network using majority class undersampling, with an AUC of 0.8, the sensitivity of 72.17%, and specificity of 74.2%.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>A deep neural network produced the best AUCs and sensitivity results on the test set and external validation set. Models are intended to be used to screen guideline noncompliant diabetic patients in an urban safety-net setting.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Machine learning on diabetic patients\u2019 routinely collected clinical data could help clinicians in safety-net settings to identify and target unscreened diabetic patients who potentially have undiagnosed DR.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooab066","type":"journal-article","created":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T03:31:27Z","timestamp":1629343887000},"source":"Crossref","is-referenced-by-count":27,"title":["Detecting diabetic retinopathy through machine learning on electronic health record data from an urban, safety net healthcare system"],"prefix":"10.1093","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1388-244X","authenticated-orcid":false,"given":"Omolola I","family":"Ogunyemi","sequence":"first","affiliation":[{"name":"Center for Biomedical Informatics, Charles R. Drew University of Medicine and Science, Los Angeles, California, USA"},{"name":"Department of Radiological Sciences, University of California, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meghal","family":"Gandhi","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics, Charles R. Drew University of Medicine and Science, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Lee","sequence":"additional","affiliation":[{"name":"Biostatistical Core, Charles R. Drew University of Medicine and Science, Los Angeles, California, USA"},{"name":"Department of Biostatistics, University of California Los Angeles Fielding School of Public Health, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Senait","family":"Teklehaimanot","sequence":"additional","affiliation":[{"name":"Biostatistical Core, Charles R. Drew University of Medicine and Science, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lauren Patty","family":"Daskivich","sequence":"additional","affiliation":[{"name":"Department of Surgery, Charles R. Drew University of Medicine and Science, Los Angeles, California, USA"},{"name":"Los Angeles County Department of Health Services, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Hindman","sequence":"additional","affiliation":[{"name":"Department of Family Medicine, Charles R. Drew University of Medicine and Science, Los Angeles, California, USA"},{"name":"Los Angeles County Department of Public Health, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Lopez","sequence":"additional","affiliation":[{"name":"Center for Biomedical Informatics, Charles R. Drew University of Medicine and Science, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ricky K","family":"Taira","sequence":"additional","affiliation":[{"name":"Department of Radiological Sciences, University of California, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,8,19]]},"reference":[{"key":"2021081907311648500_ooab066-B1","volume-title":"National Diabetes Statistics Report: Estimates of Diabetes and Its Burden in the United States, 2017","author":"Centers for Disease Control and Prevention","year":"2017"},{"key":"2021081907311648500_ooab066-B2","volume-title":"Vision Health Initiative Diabetic Retinopathy Fact Sheet","author":"Centers for Disease Control and Prevention","year":"2020"},{"issue":"3","key":"2021081907311648500_ooab066-B3","doi-asserted-by":"crossref","first-page":"556","DOI":"10.2337\/dc11-1909","article-title":"Global prevalence and major risk factors of diabetic retinopathy","volume":"35","author":"Yau","year":"2012","journal-title":"Diabetes Care"},{"issue":"12","key":"2021081907311648500_ooab066-B4","doi-asserted-by":"crossref","first-page":"1740","DOI":"10.1001\/archopht.126.12.1740","article-title":"Projection of diabetic retinopathy and other major eye diseases among people with diabetes mellitus: United States, 2005-2050","volume":"126","author":"Saaddine","year":"2008","journal-title":"Arch Ophthalmol"},{"key":"2021081907311648500_ooab066-B5","volume-title":"Vision Disorders in Diabetes. Diabetes in America","author":"Klein","year":"1995","edition":"2nd ed."},{"issue":"6","key":"2021081907311648500_ooab066-B6","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1001\/jama.2010.1111","article-title":"Prevalence of diabetic retinopathy in the United States, 2005-2008","volume":"304","author":"Zhang","year":"2010","journal-title":"JAMA"},{"issue":"4","key":"2021081907311648500_ooab066-B7","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1007\/s11892-012-0283-6","article-title":"Current epidemiology of diabetic retinopathy and diabetic macular edema","volume":"12","author":"Ding","year":"2012","journal-title":"Curr Diab Rep"},{"issue":"4","key":"2021081907311648500_ooab066-B8","doi-asserted-by":"crossref","first-page":"251","DOI":"10.3109\/09286586.2010.498661","article-title":"Prevalence and relationship between diabetic retinopathy and nephropathy, and its risk factors in the North-East of Spain, a population-based study","volume":"17","author":"Pedro","year":"2010","journal-title":"Ophthalmic Epidemiol"},{"issue":"4","key":"2021081907311648500_ooab066-B9","doi-asserted-by":"crossref","first-page":"140","DOI":"10.2337\/diaclin.27.4.140","article-title":"Diabetic retinopathy screening update","volume":"27","author":"Garg","year":"2009","journal-title":"Clin Diabetes"},{"issue":"12","key":"2021081907311648500_ooab066-B10","doi-asserted-by":"crossref","first-page":"1875","DOI":"10.2337\/diacare.15.12.1875","article-title":"Epidemiology of proliferative diabetic retinopathy","volume":"15","author":"Klein","year":"1992","journal-title":"Diabetes Care"},{"issue":"3","key":"2021081907311648500_ooab066-B11","doi-asserted-by":"crossref","first-page":"412","DOI":"10.2337\/dc16-2641","article-title":"Diabetic retinopathy: a position statement by the American Diabetes Association","volume":"40","author":"Solomon","year":"2017","journal-title":"Diabetes Care"},{"key":"2021081907311648500_ooab066-B12","first-page":"1082","article-title":"Teleretinal screening for diabetic retinopathy in six Los Angeles urban safety-net clinics: final study results","volume":"2013","author":"Ogunyemi","year":"2013","journal-title":"AMIA Annu Symp Proc"},{"issue":"5","key":"2021081907311648500_ooab066-B13","doi-asserted-by":"crossref","first-page":"642","DOI":"10.1001\/jamainternmed.2017.0204","article-title":"Implementation and evaluation of a large-scale teleretinal diabetic retinopathy screening program in the Los Angeles County Department of Health Services","volume":"177","author":"Daskivich","year":"2017","journal-title":"JAMA Intern Med"},{"issue":"4","key":"2021081907311648500_ooab066-B14","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1016\/j.oret.2018.12.003","article-title":"Accuracy of detection and grading of diabetic retinopathy and diabetic macular edema utilizing teleretinal screening","volume":"3","author":"Date","year":"2019","journal-title":"Ophthalmology Retina"},{"key":"2021081907311648500_ooab066-B15","volume-title":"Federally Qualified Health Centers and State Health Policy: A Primer for California","author":"Takach","year":"2009"},{"key":"2021081907311648500_ooab066-B16","volume-title":"About Los Angeles County Department of Health Services","author":"Los Angeles County Department of Health Services","year":"2020"},{"key":"2021081907311648500_ooab066-B17","volume-title":"My Health LA Annual Report to the Los Angeles County Board of Supervisors Fiscal Year 2018-19","author":"Los Angeles County Department of Health Services","year":"2019"},{"key":"2021081907311648500_ooab066-B18","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic Minority Over-sampling Technique","volume":"16","author":"Chawla","year":"2002","journal-title":"JAIR"},{"key":"2021081907311648500_ooab066-B19","first-page":"60","author":"Kuhn","year":"2018"},{"key":"2021081907311648500_ooab066-B20","volume-title":"R: A Language and Environment for Statistical Computing","author":"R Core Team","year":"2018"},{"key":"2021081907311648500_ooab066-B21","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"2021081907311648500_ooab066-B22","volume-title":"The Python Language Reference Manual","author":"van Rossum","year":"2011"},{"issue":"7","key":"2021081907311648500_ooab066-B23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v074.i07","article-title":"Imputation with the R Package VIM","volume":"74","author":"Kowarik","year":"2016","journal-title":"J Stat Soft"},{"key":"2021081907311648500_ooab066-B24","first-page":"162","article-title":"Evaluating predictive modeling's potential to improve teleretinal screening participation in urban safety net clinics","volume":"192","author":"Ogunyemi","year":"2013","journal-title":"Stud Health Technol Inform"},{"key":"2021081907311648500_ooab066-B25","first-page":"983","article-title":"Machine learning approaches for detecting diabetic retinopathy from clinical and public health records","volume":"2015","author":"Ogunyemi","year":"2015","journal-title":"AMIA Annu Symp Proc"},{"key":"2021081907311648500_ooab066-B26","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1186\/1472-6947-13-106","article-title":"Diabetic retinopathy risk prediction for fundus examination using sparse learning: a cross-sectional study","volume":"13","author":"Oh","year":"2013","journal-title":"BMC Med Inform Decis Mak"},{"key":"2021081907311648500_ooab066-B27","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1186\/s12859-018-2277-0","article-title":"Predicting diabetic retinopathy and identifying interpretable biomedical features using machine learning algorithms","volume":"19 (Suppl 9","author":"Tsao","year":"2018","journal-title":"BMC Bioinformatics"},{"issue":"2","key":"2021081907311648500_ooab066-B28","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1016\/j.diabres.2015.02.020","article-title":"A classification model for predicting eye disease in newly diagnosed people with type 2 diabetes","volume":"108","author":"Cichosz","year":"2015","journal-title":"Diabetes Res Clin Pract"},{"issue":"9","key":"2021081907311648500_ooab066-B29","doi-asserted-by":"crossref","DOI":"10.3390\/math8091620","article-title":"Deep neural network for predicting diabetic retinopathy from risk factors","volume":"8","author":"Alfian","year":"2020","journal-title":"Mathematics"},{"key":"2021081907311648500_ooab066-B30","first-page":"472","article-title":"Predictive models for diabetic retinopathy from non-image teleretinal screening data","volume":"2019","author":"Ogunyemi","year":"2019","journal-title":"AMIA Jt Summits Transl Sci Proc"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamiaopen\/article-pdf\/4\/3\/ooab066\/39805206\/ooab066.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamiaopen\/article-pdf\/4\/3\/ooab066\/39805206\/ooab066.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T03:31:59Z","timestamp":1629343919000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooab066\/6354638"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,1]]},"references-count":30,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2021,8,19]]},"published-print":{"date-parts":[[2021,7,31]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooab066","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,7,1]]},"published":{"date-parts":[[2021,7,1]]},"article-number":"ooab066"}}