{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T00:40:20Z","timestamp":1791333620156,"version":"4.3.1"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Despite restrictive opioid management guidelines, opioid use disorder (OUD) remains a major public health concern. Machine learning (ML) offers a promising avenue for identifying and alerting clinicians about OUD, thus supporting better clinical decision-making regarding treatment.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to assess the clinical validity of an ML application designed to identify and alert clinicians of different levels of OUD risk by comparing it to a structured review of medical records by clinicians.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n The ML application generated OUD risk alerts on outpatient data for 649,504 patients from 2 medical centers between 2010 and 2013. A random sample of 60 patients was selected from 3 OUD risk level categories (n=180). An OUD risk classification scheme and standardized data extraction tool were developed to evaluate the validity of the alerts. Clinicians independently conducted a systematic and structured review of medical records and reached a consensus on a patient\u2019s OUD risk level, which was then compared to the ML application\u2019s risk assignments.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A total of 78,587 patients without cancer with at least 1 opioid prescription were identified as follows: not high risk (n=50,405, 64.1%), high risk (n=16,636, 21.2%), and suspected OUD or OUD (n=11,546, 14.7%). The sample of 180 patients was representative of the total population in terms of age, sex, and race. The interrater reliability between the ML application and clinicians had a weighted kappa coefficient of 0.62 (95% CI 0.53-0.71), indicating good agreement. Combining the high risk and suspected OUD or OUD categories and using the review of medical records as a gold standard, the ML application had a corrected sensitivity of 56.6% (95% CI 48.7%-64.5%) and a corrected specificity of 94.2% (95% CI 90.3%-98.1%). The positive and negative predictive values were 93.3% (95% CI 88.2%-96.3%) and 60.0% (95% CI 50.4%-68.9%), respectively. Key themes for disagreements between the ML application and clinician reviews were identified.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n A systematic comparison was conducted between an ML application and clinicians for identifying OUD risk. The ML application generated clinically valid and useful alerts about patients\u2019 different OUD risk levels. ML applications hold promise for identifying patients at differing levels of OUD risk and will likely complement traditional rule-based approaches to generating alerts about opioid safety issues.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/53625","type":"journal-article","created":{"date-parts":[[2024,6,6]],"date-time":"2024-06-06T07:42:43Z","timestamp":1717659763000},"page":"e53625-e53625","source":"Crossref","is-referenced-by-count":12,"title":["A Machine Learning Application to Classify Patients at Differing Levels of Risk of Opioid Use Disorder: Clinician-Based Validation Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8268-0884","authenticated-orcid":false,"given":"Tewodros","family":"Eguale","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4060-0353","authenticated-orcid":false,"given":"Fran\u00e7ois","family":"Bastardot","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6146-4093","authenticated-orcid":false,"given":"Wenyu","family":"Song","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5955-0969","authenticated-orcid":false,"given":"Daniel","family":"Motta-Calderon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6548-0418","authenticated-orcid":false,"given":"Yasmin","family":"Elsobky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9635-801X","authenticated-orcid":false,"given":"Angela","family":"Rui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4667-2104","authenticated-orcid":false,"given":"Marlika","family":"Marceau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3089-1874","authenticated-orcid":false,"given":"Clark","family":"Davis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0121-7598","authenticated-orcid":false,"given":"Sandya","family":"Ganesan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-1518-1995","authenticated-orcid":false,"given":"Ava","family":"Alsubai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7941-3796","authenticated-orcid":false,"given":"Michele","family":"Matthews","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8735-2951","authenticated-orcid":false,"given":"Lynn A","family":"Volk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6268-1540","authenticated-orcid":false,"given":"David W","family":"Bates","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8503-5864","authenticated-orcid":false,"given":"Ronen","family":"Rozenblum","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2024,6,4]]},"reference":[{"key":"R1","unstructured":"2020 National Survey on Drug Use and Health (NSDUH): methodological summary and definitions. Substance Abuse and Mental Health Services Administration. 2021. URL: https:\/\/www.samhsa.gov\/data\/sites\/default\/files\/reports\/rpt35330\/2020NSDUHMethodSummDefs091721.pdf [Accessed 25-07-2023]"},{"issue":"7","key":"R2","doi-asserted-by":"publisher","first-page":"1325","DOI":"10.1111\/pme.12768","article-title":"The burden of undiagnosed opioid abuse among commercially insured individuals","volume":"16","author":"Kirson","journal-title":"Pain Med"},{"key":"R3","unstructured":"Drug overdose death rates. National Institute on Drug Abuse. 2023. URL: https:\/\/nida.nih.gov\/research-topics\/trends-statistics\/overdose-death-rates [Accessed 25-07-2023]"},{"issue":"20","key":"R4","doi-asserted-by":"publisher","first-page":"2040","DOI":"10.1001\/jama.2021.7464","article-title":"Massive costs of the US opioid epidemic in lives and dollars","volume":"325","author":"Kuehn","journal-title":"JAMA"},{"issue":"3","key":"R5","doi-asserted-by":"publisher","first-page":"375","DOI":"10.3122\/jabfm.2019.03.180306","article-title":"Sources and impact of time pressure on opioid management in the safety-net","volume":"32","author":"Satterwhite","journal-title":"J Am Board Fam Med"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1186\/s12875-015-0264-3","article-title":"Decision support for chronic pain care: how do primary care physicians decide when to prescribe opioids? A qualitative study","volume":"16","author":"Harle","journal-title":"BMC Fam Pract"},{"key":"R7","unstructured":"Artificial intelligence in health care: benefits and challenges of technologies to augment patient care. United States Government Accountability Office. 2022. URL: https:\/\/www.gao.gov\/assets\/720\/711471.pdf [Accessed 25-07-2023]"},{"issue":"2","key":"R8","first-page":"195","volume":"14","author":"McCoy","journal-title":"Ochsner J"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1093\/jamia\/ocaa319","article-title":"Recommendations for the safe, effective use of adaptive CDS in the US healthcare system: an AMIA position paper","volume":"28","author":"Petersen","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"R10","doi-asserted-by":"publisher","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","journal-title":"Lancet Digit Health"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0122-0","article-title":"Deep learning and alternative learning strategies for retrospective real-world clinical data","volume":"2","author":"Chen","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1038\/s41746-022-00595-9","article-title":"Machine learning and health need better values","volume":"5","author":"Ghassemi","journal-title":"NPJ Digit Med"},{"key":"R13","unstructured":"Technology: your safety layer within. MedAware. 2023. URL: https:\/\/www.medaware.com\/technology\/ [Accessed 04-11-2022]"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"e137","DOI":"10.1016\/S2589-7500(21)00229-6","article-title":"Key use cases for artificial intelligence to reduce the frequency of adverse drug events: a scoping review","volume":"4","author":"Syrowatka","journal-title":"Lancet Digit Health"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.wneu.2021.01.002","article-title":"Random forest-based prediction of outcome and mortality in patients with traumatic brain injury undergoing primary decompressive craniectomy","volume":"148","author":"Hanko","journal-title":"World Neurosurg"},{"issue":"4","key":"R16","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1093\/jamia\/ocaa336","article-title":"Predicting pressure injury using nursing assessment phenotypes and machine learning methods","volume":"28","author":"Song","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.jcjq.2019.09.008","article-title":"Using a machine learning system to identify and prevent medication prescribing errors: a clinical and cost analysis evaluation","volume":"46","author":"Rozenblum","journal-title":"Jt Comm J Qual Patient Saf"},{"issue":"2","key":"R18","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1093\/jamia\/ocw171","article-title":"Screening for medication errors using an outlier detection system","volume":"24","author":"Schiff","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"R19","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1093\/oxfordjournals.aje.a117323","article-title":"Efficient study designs to assess the accuracy of screening tests","volume":"140","author":"Irwig","journal-title":"Am J Epidemiol"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.15585\/mmwr.rr6501e1","article-title":"CDC guideline for prescribing opioids for chronic pain\u2014United States, 2016","volume":"65","author":"Dowell","journal-title":"MMWR Recomm Rep"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"American Psychiatric Association. Opioid use disorder. In: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. American Psychiatric Association; 2013:541-546. URL: https:\/\/psychiatryonline.org\/doi\/book\/10.1176\/appi.books.9780890425596 [Accessed 08-05-2024] [doi: 10.1176\/appi.books.9780890425596]","DOI":"10.1176\/appi.books.9780890425596"},{"issue":"5","key":"R22","doi-asserted-by":"publisher","first-page":"1741","DOI":"10.1213\/ANE.0000000000002496","article-title":"Risk factors for opioid-use disorder and overdose","volume":"125","author":"Webster","journal-title":"Anesth Analg"},{"issue":"5","key":"R23","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.2611","article-title":"Risk factors for opioid use disorders in adult postsurgical patients","volume":"10","author":"Burcher","journal-title":"Cureus"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1155\/2019\/7490801","article-title":"Risk factors and prevention strategies for postoperative opioid abuse","volume":"2019","author":"Zhao","journal-title":"Pain Res Manag"},{"issue":"1","key":"R25","doi-asserted-by":"crossref","first-page":"207","DOI":"10.2307\/2530820","volume":"39","author":"Begg","journal-title":"Biometrics"},{"issue":"4","key":"R26","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1056\/NEJMoa021659","article-title":"Effect of verification bias on screening for prostate cancer by measurement of prostate-specific antigen","volume":"349","author":"Punglia","journal-title":"N Engl J Med"},{"issue":"5","key":"R27","first-page":"360","volume":"37","author":"Viera","journal-title":"Fam Med"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1097\/ADM.0000000000001019","article-title":"Using machine learning to predict treatment adherence in patients on medication for opioid use disorder","volume":"17","author":"Burgess-Hull","journal-title":"J Addict Med"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"e455","DOI":"10.1016\/S2589-7500(22)00062-0","article-title":"Developing and validating a machine-learning algorithm to predict opioid overdose among medicaid beneficiaries in two US States: a prognostic modeling study","volume":"4","author":"Lo-Ciganic","journal-title":"Lancet Digit Health"},{"issue":"6","key":"R30","doi-asserted-by":"publisher","first-page":"e426","DOI":"10.1016\/S2589-7500(22)00041-3","article-title":"Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study","volume":"4","author":"Afshar","journal-title":"Lancet Digit Health"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","first-page":"4974","DOI":"10.1038\/s41598-022-09095-y","article-title":"Development and validation of a risk-score model for opioid overdose using a national claims database","volume":"12","author":"Heo","journal-title":"Sci Rep"},{"issue":"8","key":"R32","doi-asserted-by":"publisher","first-page":"1683","DOI":"10.1093\/jamia\/ocab043","article-title":"Identifying risk of opioid use disorder for patients taking opioid medications with deep learning","volume":"28","author":"Dong","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.1186\/s12889-021-11551-9","article-title":"Provider bias in prescribing opioid analgesics: a study of electronic medical records at a hospital emergency department","volume":"21","author":"Keister","journal-title":"BMC Public Health"},{"issue":"8","key":"R34","doi-asserted-by":"publisher","first-page":"899","DOI":"10.23736\/S0375-9393.19.13321-4","article-title":"Using opioid therapy for pain in clinically challenging situations: questions for clinicians","volume":"85","author":"Pergolizzi","journal-title":"Minerva Anestesiol"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1016\/j.cell.2020.03.022","article-title":"How machine learning will transform biomedicine","volume":"181","author":"Goecks","journal-title":"Cell"},{"issue":"2","key":"R36","doi-asserted-by":"publisher","first-page":"020303","DOI":"10.7189\/jogh.08.020303","article-title":"Artificial intelligence, machine learning and health systems","volume":"8","author":"Panch","journal-title":"J Glob Health"},{"issue":"1","key":"R37","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1038\/s41746-022-00703-9","article-title":"A digital physician peer to automatically detect erroneous prescriptions in radiotherapy","volume":"5","author":"Li","journal-title":"NPJ Digit Med"},{"issue":"6","key":"R38","doi-asserted-by":"publisher","first-page":"1204","DOI":"10.1093\/jamia\/ocx066","article-title":"Automatable algorithms to identify nonmedical opioid use using electronic data: a systematic review","volume":"24","author":"Canan","journal-title":"J Am Med Inform Assoc"},{"issue":"26","key":"R39","doi-asserted-by":"publisher","first-page":"2507","DOI":"10.1056\/NEJMp1702071","article-title":"Machine learning and prediction in medicine: beyond the peak of inflated expectations","volume":"376","author":"Chen","journal-title":"N Engl J Med"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1038\/s41746-021-00426-3","article-title":"Machine learning for patient risk stratification: standing on, or looking over, the shoulders of clinicians?","volume":"4","author":"Beaulieu-Jones","journal-title":"NPJ Digit Med"},{"issue":"3","key":"R41","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2019.0968","article-title":"Evaluation of machine-learning algorithms for predicting opioid overdose risk among medicare beneficiaries with opioid prescriptions","volume":"2","author":"Lo-Ciganic","journal-title":"JAMA Netw Open"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2021.108583","article-title":"Identifying individuals with opioid use disorder: validity of International Classification of Diseases diagnostic codes for opioid use, dependence and abuse","volume":"221","author":"Lagisetty","journal-title":"Drug Alcohol Depend"},{"issue":"1","key":"R43","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1186\/s13722-021-00229-7","article-title":"External validation of an opioid misuse machine learning classifier in hospitalized adult patients","volume":"16","author":"Afshar","journal-title":"Addict Sci Clin Pract"},{"issue":"12","key":"R44","doi-asserted-by":"publisher","first-page":"1675","DOI":"10.2105\/AJPH.2018.304673","article-title":"Estimated prevalence of opioid use disorder in Massachusetts, 2011-2015: a capture-recapture analysis","volume":"108","author":"Barocas","journal-title":"Am J Public Health"},{"issue":"5","key":"R45","doi-asserted-by":"publisher","first-page":"293","DOI":"10.7326\/M17-0865","article-title":"Prescription opioid use, misuse, and use disorders in U.S. adults: 2015 National Survey on Drug Use and Health","volume":"167","author":"Han","journal-title":"Ann Intern Med"},{"issue":"1","key":"R46","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1001\/archpsyc.1997.01830130077015","article-title":"Psychiatric and substance use comorbidity among treatment-seeking opioid abusers","volume":"54","author":"Brooner","journal-title":"Arch Gen Psychiatry"},{"issue":"9","key":"R47","doi-asserted-by":"publisher","first-page":"1561","DOI":"10.1093\/jamia\/ocad115","article-title":"Potential bias and lack of generalizability in electronic health record data: reflections on health equity from the National Institutes of Health Pragmatic Trials Collaboratory","volume":"30","author":"Boyd","journal-title":"J Am Med Inform Assoc"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2024\/1\/e53625","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:39:30Z","timestamp":1782841170000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2024\/1\/e53625"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,4]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/53625","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,4]]},"article-number":"v12i7e53625"}}