{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T03:31:17Z","timestamp":1788838277844,"version":"build-2803163510"},"reference-count":57,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2021,8,1]],"date-time":"2021-08-01T00:00:00Z","timestamp":1627776000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["1K01DA041620"],"award-info":[{"award-number":["1K01DA041620"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["2UL1TR001425-05A1"],"award-info":[{"award-number":["2UL1TR001425-05A1"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["1R01HD103630"],"award-info":[{"award-number":["1R01HD103630"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006093","name":"Patient-Centered Outcomes Research Institute","doi-asserted-by":"publisher","award":["PCORI\/PCS-2018C1-11111"],"award-info":[{"award-number":["PCORI\/PCS-2018C1-11111"]}],"id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007172","name":"Cincinnati Children's Hospital Medical Center","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100007172","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,9,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Substance use screening in adolescence is unstandardized and often documented in clinical notes, rather than in structured electronic health records (EHRs). The objective of this study was to integrate logic rules with state-of-the-art natural language processing (NLP) and machine learning technologies to detect substance use information from both structured and unstructured EHR data.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Pediatric patients (10-20 years of age) with any encounter between July 1, 2012, and October 31, 2017, were included (n = 3890 patients; 19 478 encounters). EHR data were extracted at each encounter, manually reviewed for substance use (alcohol, tobacco, marijuana, opiate, any use), and coded as lifetime use, current use, or family use. Logic rules mapped structured EHR indicators to screening results. A knowledge-based NLP system and a deep learning model detected substance use information from unstructured clinical narratives. System performance was evaluated using positive predictive value, sensitivity, negative predictive value, specificity, and area under the receiver-operating characteristic curve (AUC).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The dataset included 17 235 structured indicators and 27 141 clinical narratives. Manual review of clinical narratives captured 94.0% of positive screening results, while structured EHR data captured 22.0%. Logic rules detected screening results from structured data with 1.0 and 0.99 for sensitivity and specificity, respectively. The knowledge-based system detected substance use information from clinical narratives with 0.86, 0.79, and 0.88 for AUC, sensitivity, and specificity, respectively. The deep learning model further improved detection capacity, achieving 0.88, 0.81, and 0.85 for AUC, sensitivity, and specificity, respectively. Finally, integrating predictions from structured and unstructured data achieved high detection capacity across all cases (0.96, 0.85, and 0.87 for AUC, sensitivity, and specificity, respectively).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>It is feasible to detect substance use screening and results among pediatric patients using logic rules, NLP, and machine learning technologies.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab116","type":"journal-article","created":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T03:15:57Z","timestamp":1622085357000},"page":"2116-2127","source":"Crossref","is-referenced-by-count":31,"title":["Automated detection of substance use information from electronic health records for a pediatric population"],"prefix":"10.1093","volume":"28","author":[{"given":"Yizhao","family":"Ni","sequence":"first","affiliation":[{"name":"Division of Biomedical Informatics, Cincinnati Children\u2019s Hospital Medical Center, Cincinnati, Ohio, USA"},{"name":"Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alycia","family":"Bachtel","sequence":"additional","affiliation":[{"name":"Division of Biomedical Informatics, Cincinnati Children\u2019s Hospital Medical Center, Cincinnati, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katie","family":"Nause","sequence":"additional","affiliation":[{"name":"Division of Psychology, Cincinnati Children\u2019s Hospital Medical Center, Cincinnati, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Beal","sequence":"additional","affiliation":[{"name":"Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, Ohio, USA"},{"name":"Division of Psychology, Cincinnati Children\u2019s Hospital Medical Center, Cincinnati, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,8,1]]},"reference":[{"issue":"2","key":"2021091817524888200_ocab116-B1","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1046\/j.1525-1381.1999.09256.x","article-title":"Mortality and morbidity attributable to use of addictive substances in the United States","volume":"111","author":"McGinnis","year":"1999","journal-title":"Proc Assoc Am Physicians"},{"key":"2021091817524888200_ocab116-B2","year":"2020"},{"key":"2021091817524888200_ocab116-B3","doi-asserted-by":"crossref","DOI":"10.3998\/2027.42\/150621","volume-title":"Monitoring the Future National Survey Results on Drug Use, 1975-2018: Overview Key Findings on Adolescent Drug Use","author":"Johnston","year":"2019"},{"key":"2021091817524888200_ocab116-B4","volume-title":"Key Substance Use and Mental Health Indicators in the United States: Results from the 2018 National Survey on Drug Use and Health (HHS Publication No. PEP19-5068, NSDUH Series H-54)","year":"2019"},{"issue":"4","key":"2021091817524888200_ocab116-B5","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1016\/j.addbeh.2011.11.011","article-title":"Early onset of drug and polysubstance use as predictors of injection drug use among adult drug users","volume":"37","author":"Trenz","year":"2012","journal-title":"Addict Behav"},{"issue":"1","key":"2021091817524888200_ocab116-B6","doi-asserted-by":"crossref","first-page":"e20161211","DOI":"10.1542\/peds.2016-1211","article-title":"Substance use screening, brief intervention, and referral to treatment","volume":"138","author":"Levy","year":"2016","journal-title":"Pediatrics"},{"issue":"1","key":"2021091817524888200_ocab116-B7","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1111\/j.1360-0443.2012.03876.x","article-title":"Common data elements for substance use disorders in electronic health records: The NIDA clinical trials network experience","volume":"108","author":"Ghitza","year":"2013","journal-title":"Addiction"},{"issue":"1","key":"2021091817524888200_ocab116-B8","first-page":"35","article-title":"Clinical workflow and substance use screening, brief intervention, and referral to treatment data in the electronic health records: a national drug abuse treatment clinical trials network study","volume":"7","author":"Wu","year":"2019","journal-title":"EGEMS (Wash DC)"},{"issue":"6","key":"2021091817524888200_ocab116-B9","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1097\/ADM.0000000000000340","article-title":"Screening adolescents for alcohol use","volume":"11","author":"Levy","year":"2017","journal-title":"J Addict Med"},{"issue":"1","key":"2021091817524888200_ocab116-B10","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1080\/08897077.2015.1015703","article-title":"Screening, brief intervention, and referral to treatment for adolescents: attitudes, perceptions, and practice of New York school-based health center providers","volume":"37","author":"Harris","year":"2016","journal-title":"Subst Abus"},{"issue":"15","key":"2021091817524888200_ocab116-B11","doi-asserted-by":"crossref","first-page":"1709","DOI":"10.1001\/jama.2010.1497","article-title":"Meaningful use of electronic health records: the road ahead","volume":"304","author":"Jha","year":"2010","journal-title":"JAMA"},{"issue":"1","key":"2021091817524888200_ocab116-B12","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1186\/s13011-020-00276-4","article-title":"A chart review of substance use screening and related documentation among adolescents in outpatient pediatric clinics: implications for practice","volume":"15","author":"Kuhns","year":"2020","journal-title":"Subst Abuse Treat Prev Policy"},{"key":"2021091817524888200_ocab116-B13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2147\/SAR.S22575","article-title":"Electronic health records: essential tools in integrating substance abuse treatment with primary care","volume":"3","author":"Tai","year":"2012","journal-title":"Subst Abuse Rehabil"},{"issue":"11","key":"2021091817524888200_ocab116-B14","doi-asserted-by":"crossref","first-page":"e153145","DOI":"10.1001\/jamapediatrics.2015.3145","article-title":"Implementation of screening, brief intervention, and referral to treatment for adolescents in pediatric primary care: a cluster randomized trial","volume":"169","author":"Sterling","year":"2015","journal-title":"JAMA Pediatr"},{"issue":"4","key":"2021091817524888200_ocab116-B15","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1016\/j.jadohealth.2017.10.016","article-title":"Pediatrician and behavioral clinician-delivered screening, brief intervention and referral to treatment: substance use and depression outcomes","volume":"62","author":"Sterling","year":"2018","journal-title":"J Adolesc Health"},{"issue":"5","key":"2021091817524888200_ocab116-B16","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1176\/appi.ps.201800427","article-title":"Barriers faced by physicians in screening for substance use disorders among adolescents","volume":"70","author":"Palmer","year":"2019","journal-title":"Psychiatr Serv"},{"issue":"8","key":"2021091817524888200_ocab116-B17","doi-asserted-by":"crossref","first-page":"1471","DOI":"10.1111\/add.14659","article-title":"Commentary on Winhusen et al. (2019): Substance use disorders, chronic diseases, and electronic health records-a paradigm for screening and intervention","volume":"114","author":"Mannelli","year":"2019","journal-title":"Addiction"},{"issue":"1","key":"2021091817524888200_ocab116-B18","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1186\/s12911-015-0160-8","article-title":"An end-to-end hybrid algorithm for automated medication discrepancy detection","volume":"15","author":"Li","year":"2015","journal-title":"BMC Med Inform Decis Mak"},{"key":"2021091817524888200_ocab116-B19","doi-asserted-by":"crossref","first-page":"104137","DOI":"10.1016\/j.ijmedinf.2020.104137","article-title":"Finding warning markers: leveraging natural language processing and machine learning technologies to detect risk of school violence","volume":"139","author":"Ni","year":"2020","journal-title":"Int J Med Inform"},{"issue":"1","key":"2021091817524888200_ocab116-B20","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1136\/amiajnl-2014-002887","article-title":"Automated clinical trial eligibility prescreening: Increasing the efficiency of patient identification for clinical trials in the emergency department","volume":"22","author":"Ni","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2021091817524888200_ocab116-B21","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1111\/sltb.12312","article-title":"A machine learning approach to identifying the thought markers of suicidal subjects: a prospective multicenter trial","volume":"47","author":"Pestian","year":"2017","journal-title":"Suicide Life Threat Behav"},{"key":"2021091817524888200_ocab116-B22","doi-asserted-by":"crossref","first-page":"1178222617713018","DOI":"10.1177\/1178222617713018","article-title":"Leveraging food and drug administration adverse event reports for the automated monitoring of electronic health records in a pediatric hospital","volume":"9","author":"Tang","year":"2017","journal-title":"Biomed Inform Insights"},{"key":"2021091817524888200_ocab116-B23","first-page":"2121","article-title":"Automated extraction of substance use information from clinical texts","volume":"2015","author":"Wang","year":"2015","journal-title":"AMIA Annu Symp Proc"},{"issue":"4","key":"2021091817524888200_ocab116-B24","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1016\/j.jpain.2015.01.011","article-title":"Automated prediction of risk for problem opioid use in a primary care setting","volume":"16","author":"Hylan","year":"2015","journal-title":"J Pain"},{"issue":"11","key":"2021091817524888200_ocab116-B25","doi-asserted-by":"crossref","first-page":"1272","DOI":"10.1093\/jamia\/ocz072","article-title":"Toward a clinical text encoder: pretraining for clinical natural language processing with applications to substance misuse","volume":"26","author":"Dligach","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"S3","key":"2021091817524888200_ocab116-B26","doi-asserted-by":"crossref","first-page":"S185","DOI":"10.2105\/AJPH.2019.305020","article-title":"Targeting youth to prevent later substance use disorder: an underutilized response to the us opioid crisis","volume":"109","author":"Compton","year":"2019","journal-title":"Am J Public Health"},{"key":"2021091817524888200_ocab116-B27","author":"Beal","year":"2017"},{"issue":"5","key":"2021091817524888200_ocab116-B28","doi-asserted-by":"crossref","first-page":"950","DOI":"10.1016\/j.jbi.2008.12.013","article-title":"Building a semantically annotated corpus of clinical texts","volume":"42","author":"Roberts","year":"2009","journal-title":"J Biomed Inform"},{"key":"2021091817524888200_ocab116-B29","year":"2020"},{"key":"2021091817524888200_ocab116-B30","year":"2021"},{"issue":"3","key":"2021091817524888200_ocab116-B31","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem Med (Zagreb)"},{"issue":"3","key":"2021091817524888200_ocab116-B32","doi-asserted-by":"crossref","first-page":"e14185","DOI":"10.2196\/14185","article-title":"A real-time automated patient screening system for clinical trials eligibility in an emergency department: design and evaluation","volume":"7","author":"Ni","year":"2019","journal-title":"JMIR Med Inform"},{"issue":"1","key":"2021091817524888200_ocab116-B33","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1186\/s12911-015-0149-3","article-title":"Increasing the efficiency of trial-patient matching: automated clinical trial eligibility pre-screening for pediatric oncology patients","volume":"15","author":"Ni","year":"2015","journal-title":"BMC Med Inform Decis Mak"},{"issue":"3","key":"2021091817524888200_ocab116-B34","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.cmpb.2011.01.002","article-title":"Systematized nomenclature of medicine clinical terms (SNOMED CT) to represent computed tomography procedures","volume":"101","author":"De Silva","year":"2011","journal-title":"Comput Methods Programs Biomed"},{"key":"2021091817524888200_ocab116-B35","year":"2016"},{"issue":"11","key":"2021091817524888200_ocab116-B36","doi-asserted-by":"crossref","first-page":"1163","DOI":"10.1093\/jamia\/ocz163","article-title":"Cohort selection for clinical trials: N2c2 2018 shared task track 1","volume":"26","author":"Stubbs","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2021091817524888200_ocab116-B37","first-page":"3111","volume-title":"Proceedings of the 26th International Conference on Neural Information Processing Systems","author":"Mikolov","year":"2013"},{"issue":"5\u20136","key":"2021091817524888200_ocab116-B38","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neural Netw"},{"key":"2021091817524888200_ocab116-B39","author":"Abadi"},{"key":"2021091817524888200_ocab116-B40","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1186\/1471-2105-7-91","article-title":"Bias in error estimation when using cross-validation for model selection","volume":"7","author":"Varma","year":"2006","journal-title":"BMC Bioinformatics"},{"issue":"6943","key":"2021091817524888200_ocab116-B41","doi-asserted-by":"crossref","first-page":"1552","DOI":"10.1136\/bmj.308.6943.1552","article-title":"Diagnostic tests. 1: Sensitivity and specificity","volume":"308","author":"Altman","year":"1994","journal-title":"BMJ"},{"issue":"6947","key":"2021091817524888200_ocab116-B42","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1136\/bmj.309.6947.102","article-title":"Diagnostic tests 2: predictive values","volume":"309","author":"Altman","year":"1994","journal-title":"BMJ"},{"key":"2021091817524888200_ocab116-B43","volume-title":"Mathematical Statistics and Data Analysis","author":"Rice","year":"2006","edition":"3rd ed"},{"issue":"1","key":"2021091817524888200_ocab116-B44","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1002\/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3","article-title":"Index for rating diagnostic tests","volume":"3","author":"Youden","year":"1950","journal-title":"Cancer"},{"key":"2021091817524888200_ocab116-B45","volume-title":"Handbook of Biological Statistics","author":"McDonald","year":"2014","edition":"3rd ed"},{"key":"2021091817524888200_ocab116-B46","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1186\/1471-2105-12-77","article-title":"Proc: An open-source package for r and s+ to analyze and compare ROC curves","volume":"12","author":"Robin","year":"2011","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"2021091817524888200_ocab116-B47","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/34.824819","article-title":"Statistical pattern recognition: a review","volume":"22","author":"Jain","year":"2000","journal-title":"IEEE Trans Pattern Anal Mach Intelligence"},{"key":"2021091817524888200_ocab116-B48","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.jsat.2020.01.007","article-title":"Adolescent SBIRT implementation: generalist vs. Specialist models of service delivery in primary care","volume":"111","author":"Mitchell","year":"2020","journal-title":"J Subst Abuse Treat"},{"key":"2021091817524888200_ocab116-B49","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.jsat.2015.10.006","article-title":"SBIRT-a: Adapting SBIRT to maximize developmental fit for adolescents in primary care","volume":"62","author":"Ozechowski","year":"2016","journal-title":"J Subst Abuse Treat"},{"issue":"6","key":"2021091817524888200_ocab116-B50","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1097\/ADM.0000000000000340","article-title":"Screening adolescents for alcohol use: tracking practice trends of Massachusetts pediatricians","volume":"11","author":"Levy","year":"2017","journal-title":"J Addict Med"},{"issue":"5","key":"2021091817524888200_ocab116-B51","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1016\/j.tig.2019.01.007","article-title":"Electronic health records are the next frontier for the genetics of substance use disorders","volume":"35","author":"Sanchez-Roige","year":"2019","journal-title":"Trends Genet"},{"key":"2021091817524888200_ocab116-B52","first-page":"38","author":"Cardie","year":"1992"},{"key":"2021091817524888200_ocab116-B53","volume-title":"Foundation of Statistical Natural Language Processing","author":"Manning","year":"1999"},{"issue":"3","key":"2021091817524888200_ocab116-B54","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1016\/j.eswa.2013.08.015","article-title":"Syntactic n-grams as machine learning features for natural language processing","volume":"41","author":"Sidorov","year":"2014","journal-title":"Expert Syst Appl"},{"key":"2021091817524888200_ocab116-B55","first-page":"45","article-title":"Support vector machine active learning with applications to text classification","volume":"2","author":"Tong","year":"2001","journal-title":"J Mach Learn Res"},{"issue":"2","key":"2021091817524888200_ocab116-B56","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1089\/jwh.2006.0070","article-title":"The effect of race on provider decisions to test for illicit drug use in the peripartum setting","volume":"16","author":"Kunins","year":"2007","journal-title":"J Womens Health"},{"issue":"1","key":"2021091817524888200_ocab116-B57","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s11414-011-9247-x","article-title":"Universal screening for alcohol and drug use and racial disparities in child protective services reporting","volume":"39","author":"Roberts","year":"2012","journal-title":"J Behav Health Serv Res"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/10\/2116\/40408909\/ocab116.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/10\/2116\/40408909\/ocab116.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,18]],"date-time":"2021-09-18T17:54:43Z","timestamp":1631987683000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/10\/2116\/6333356"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,1]]},"references-count":57,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,8,1]]},"published-print":{"date-parts":[[2021,9,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab116","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,10,1]]},"published":{"date-parts":[[2021,8,1]]}}}