{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T16:11:59Z","timestamp":1787069519402,"version":"build-2736575974"},"reference-count":40,"publisher":"American Medical Association (AMA)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"Importance<\/jats:title>Machine learning approaches can assist opioid stewardship by identifying high-risk opioid prescribing for potential interventions.<\/jats:p><\/jats:sec>Objective<\/jats:title>To develop a machine learning model for deployment that can estimate the risk of adverse outcomes within 30 days of an opioid dispensation as a potential component of prescription drug monitoring programs using access to real-world data.<\/jats:p><\/jats:sec>Design, Setting, and Participants<\/jats:title>This prognostic study used population-level administrative health data to construct a machine learning model. This study took place in Alberta, Canada (from January 1, 2018, to December 31, 2019), and included all patients 18 years and older who received at least 1 opioid dispensation from a community pharmacy within the province.<\/jats:p><\/jats:sec>Exposures<\/jats:title>Each opioid dispensation served as the unit of analysis.<\/jats:p><\/jats:sec>Main Outcomes and Measures<\/jats:title>Opioid-related adverse outcomes were identified from administrative data sets. An XGBoost model was developed on 2018 data to estimate the risk of hospitalization, an emergency department visit, or mortality within 30 days of an opioid dispensation; validation on 2019 data was done to evaluate model performance. Model discrimination, calibration, and other relevant metrics are reported using daily and weekly predictions on both ranked predictions and predicted probability thresholds using all data from 2019.<\/jats:p><\/jats:sec>Results<\/jats:title>A total of 853\u202f324 participants represented 6\u202f181\u202f025 opioid dispensations, with 145\u202f016 outcome events reported (2.3%); 46.4% of the participants were men and 53.6% were women, with a mean (SD) age of 49.1 (15.6) years for men and 51.0 (18.0) years for women. Of the outcome events, 77\u202f326 (2.6% pretest probability) occurred within 30 days of a dispensation in the validation set (XGBoost C<\/jats:italic> statistic, 0.82 [95% CI, 0.81-0.82]). The top 0.1 percentile of estimated risk had a positive likelihood ratio (LR) of 28.7, which translated to a posttest probability of 43.1%. In our simulations, the weekly measured predictions had higher positive LRs in both the highest-risk dispensations and percentiles of estimated risk compared with predictions measured daily. Net benefit analysis showed that using machine learning prediction may not add additional benefit over the entire range of probability thresholds.<\/jats:p><\/jats:sec>Conclusions and Relevance<\/jats:title>These findings suggest that prescription drug monitoring programs can use machine learning classifiers to identify patients at risk of opioid-related adverse outcomes and intervene on high-risk ranked predictions. Better access to available administrative and clinical data could improve the prediction performance of machine learning classifiers and thus expand opioid stewardship efforts.<\/jats:p><\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2022.48559","type":"journal-article","created":{"date-parts":[[2022,12,27]],"date-time":"2022-12-27T16:00:45Z","timestamp":1672156845000},"page":"e2248559","source":"Crossref","is-referenced-by-count":20,"title":["Development and Validation of a Machine Learning Model to Estimate Risk of Adverse Outcomes Within 30 Days of Opioid Dispensation"],"prefix":"10.1001","volume":"5","author":[{"given":"Vishal","family":"Sharma","sequence":"first","affiliation":[{"name":"Li Ka Shing Center for Health Research Innovation, School of Public Health, University of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vinaykumar","family":"Kulkarni","sequence":"additional","affiliation":[{"name":"OKAKI, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ed","family":"Jess","sequence":"additional","affiliation":[{"name":"College of Physicians and Surgeons of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fizza","family":"Gilani","sequence":"additional","affiliation":[{"name":"College of Physicians and Surgeons of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dean","family":"Eurich","sequence":"additional","affiliation":[{"name":"Li Ka Shing Center for Health Research Innovation, School of Public Health, University of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scot H.","family":"Simpson","sequence":"additional","affiliation":[{"name":"Faculty of Pharmacy and Pharmaceutical Sciences, University of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Don","family":"Voaklander","sequence":"additional","affiliation":[{"name":"School of Public Health, University of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Semenchuk","sequence":"additional","affiliation":[{"name":"OKAKI, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Connor","family":"London","sequence":"additional","affiliation":[{"name":"OKAKI, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salim","family":"Samanani","sequence":"additional","affiliation":[{"name":"OKAKI, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2022,12,27]]},"reference":[{"issue":"6","key":"zoi221372r1","doi-asserted-by":"publisher","first-page":"224","DOI":"10.24095\/hpcdp.38.6.02","article-title":"The opioid crisis in Canada: a national perspective.","volume":"38","author":"Belzak","year":"2018","journal-title":"Health Promot Chronic Dis Prev Can"},{"issue":"18","key":"zoi221372r2","doi-asserted-by":"publisher","first-page":"E659","DOI":"10.1503\/cmaj.170363","article-title":"Guideline for opioid therapy and chronic noncancer pain.","volume":"189","author":"Busse","year":"2017","journal-title":"CMAJ"},{"issue":"6","key":"zoi221372r3","doi-asserted-by":"publisher","first-page":"256","DOI":"10.24095\/hpcdp.38.6.07","article-title":"Evaluating the early impacts of delisting high-strength opioids on patterns of prescribing in Ontario.","volume":"38","author":"Guan","year":"2018","journal-title":"Health Promot Chronic Dis Prev Can"},{"issue":"6","key":"zoi221372r4","doi-asserted-by":"publisher","first-page":"244","DOI":"10.24095\/hpcdp.38.6.04","article-title":"At-a-glance\u2014hospitalizations and emergency department visits due to opioid poisoning in Canada.","volume":"38","author":"O\u2019Connor","year":"2018","journal-title":"Health Promot Chronic Dis Prev Can"},{"issue":"18","key":"zoi221372r6","doi-asserted-by":"publisher","first-page":"1806","DOI":"10.1001\/jama.2019.16489","article-title":"How to read articles that use machine learning: users\u2019 guides to the medical literature.","volume":"322","author":"Liu","year":"2019","journal-title":"JAMA"},{"issue":"23","key":"zoi221372r8","doi-asserted-by":"publisher","first-page":"2283","DOI":"10.1001\/jama.2019.17831","article-title":"Machine learning, predictive analytics, and clinical practice: can the past inform the present?","volume":"322","author":"Peterson","year":"2019","journal-title":"JAMA"},{"issue":"3","key":"zoi221372r10","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2019.0968","article-title":"Evaluation of machine-learning algorithms for predicting opioid overdose risk among Medicare beneficiaries with opioid prescriptions.","volume":"2","author":"Lo-Ciganic","year":"2019","journal-title":"JAMA Netw Open"},{"issue":"10","key":"zoi221372r11","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2020-037860","article-title":"Predicting population health with machine learning: a scoping review.","volume":"10","author":"Morgenstern","year":"2020","journal-title":"BMJ Open"},{"issue":"3","key":"zoi221372r12","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2019-034568","article-title":"Reporting quality of studies using machine learning models for medical diagnosis: a systematic review.","volume":"10","author":"Yusuf","year":"2020","journal-title":"BMJ Open"},{"issue":"12","key":"zoi221372r13","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.5870","article-title":"Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view.","volume":"18","author":"Luo","year":"2016","journal-title":"J Med Internet Res"},{"issue":"5","key":"zoi221372r14","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2020-043964","article-title":"Safe opioid prescribing: a prognostic machine learning approach to predicting 30-day risk after an opioid dispensation in Alberta, Canada.","volume":"11","author":"Sharma","year":"2021","journal-title":"BMJ Open"},{"issue":"14","key":"zoi221372r17","doi-asserted-by":"publisher","first-page":"1377","DOI":"10.1001\/jama.2017.12126","article-title":"Discrimination and calibration of clinical prediction models: users\u2019 guides to the medical literature.","volume":"318","author":"Alba","year":"2017","journal-title":"JAMA"},{"issue":"29","key":"zoi221372r18","doi-asserted-by":"publisher","first-page":"1925","DOI":"10.1093\/eurheartj\/ehu207","article-title":"Towards better clinical prediction models: seven steps for development and an ABCD for validation.","volume":"35","author":"Steyerberg","year":"2014","journal-title":"Eur Heart J"},{"issue":"9","key":"zoi221372r19","doi-asserted-by":"publisher","first-page":"703","DOI":"10.1001\/jama.1994.03510330081039","article-title":"Users\u2019 guides to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients?","volume":"271","author":"Jaeschke","year":"1994","journal-title":"JAMA"},{"key":"zoi221372r25","doi-asserted-by":"publisher","first-page":"k3207","DOI":"10.1136\/bmj.k3207","article-title":"Contributions of prescribed and non-prescribed opioids to opioid related deaths: population based cohort study in Ontario, Canada.","volume":"362","author":"Gomes","year":"2018","journal-title":"BMJ"},{"issue":"6","key":"zoi221372r26","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-011060","article-title":"Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review.","volume":"6","author":"Zhou","year":"2016","journal-title":"BMJ Open"},{"issue":"2","key":"zoi221372r27","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1093\/oxfordjournals.pan.a004868","article-title":"Logistic regression in rare events data.","volume":"9","author":"King","year":"2001","journal-title":"Polit Anal"},{"issue":"15","key":"zoi221372r28","doi-asserted-by":"publisher","first-page":"1624","DOI":"10.1001\/jama.2016.1464","article-title":"CDC guideline for prescribing opioids for chronic pain\u2014United States, 2016.","volume":"315","author":"Dowell","year":"2016","journal-title":"JAMA"},{"issue":"3","key":"zoi221372r31","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0118432","article-title":"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets.","volume":"10","author":"Saito","year":"2015","journal-title":"PLoS One"},{"issue":"3","key":"zoi221372r32","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2019.0348","article-title":"Assessment of machine learning vs standard prediction rules for predicting hospital readmissions.","volume":"2","author":"Morgan","year":"2019","journal-title":"JAMA Netw Open"},{"issue":"5","key":"zoi221372r33","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.11315","article-title":"Development and validation of a machine learning model using administrative health data to predict onset of type 2 diabetes.","volume":"4","author":"Ravaut","year":"2021","journal-title":"JAMA Netw Open"},{"issue":"5","key":"zoi221372r34","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1001\/jama.1994.03510290071040","article-title":"Users\u2019 guides to the medical literature. III. How to use an article about a diagnostic test. A. Are the results of the study valid?","volume":"271","author":"Jaeschke","year":"1994","journal-title":"JAMA"},{"key":"zoi221372r35","doi-asserted-by":"publisher","first-page":"i6","DOI":"10.1136\/bmj.i6","article-title":"Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.","volume":"352","author":"Vickers","year":"2016","journal-title":"BMJ"},{"issue":"4","key":"zoi221372r36","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1001\/jama.2015.37","article-title":"Decision curve analysis.","volume":"313","author":"Fitzgerald","year":"2015","journal-title":"JAMA"},{"issue":"1","key":"zoi221372r40","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1146\/publhealth.2018.39.issue-1","article-title":"Big data in public health: terminology, machine learning, and privacy.","volume":"39","author":"Mooney","year":"2018","journal-title":"Annu Rev Public Health"},{"key":"zoi221372r16","author":"Chen","year":"2016"},{"key":"zoi221372r38","author":"Lundberg","year":"2017"},{"key":"zoi221372r5","unstructured":"College of Physicians and Surgeons of Alberta. Tracked Prescription Program Alberta. 2021. Accessed December 17, 2021. https:\/\/www.tppalberta.ca\/"},{"key":"zoi221372r7","unstructured":"Reznick? RHK, Horsely? T, Hassani? MS. Task force report on artificial intelligence and emerging digital technologies. 2020. Accessed November 14, 2021. https:\/\/www.royalcollege.ca\/rcsite\/health-policy\/initiatives\/ai-task-force-e"},{"key":"zoi221372r9","unstructured":"Alberta Machine Intelligence Institute. Machine learning process lifecycle. 2019. Accessed July 8, 2021. https:\/\/www.amii.ca\/latest-from-amii\/ml-process-lifecycle-part-1\/"},{"key":"zoi221372r15","unstructured":"XGBoost. Python API reference. Accessed August 10, 2021. https:\/\/xgboost.readthedocs.io\/en\/latest\/python\/python_api.html#module-xgboost.sklearn"},{"key":"zoi221372r20","unstructured":"Alberta College of Pharmacy. Standards of practice for pharmacists and pharmacy technicians. 2019. Accessed September 4, 2019. https:\/\/abpharmacy.ca\/"},{"key":"zoi221372r21","unstructured":"Government of Alberta. Pharmaceutical Information Network (PIN). 2022. Accessed October 10, 2022. https:\/\/www.albertanetcare.ca\/learningcentre\/pharmaceutical-information-network.htm?"},{"key":"zoi221372r22","unstructured":"World Health Organization. International Statistical Classification of Diseases and Related Health Problems, 10th Revision. 2019. Accessed June 14, 2021. https:\/\/icd.who.int\/browse10\/2019\/en#\/"},{"key":"zoi221372r23","unstructured":"Canadian Institute for Health Information. Canadian Institute for Health Information website. 2019. Accessed July 23, 2019. https:\/\/www.cihi.ca\/en"},{"key":"zoi221372r24","unstructured":"World Health Organization. International language for drug utilization research, ATC\/DDD. 2020. Accessed June 7, 2020. https:\/\/www.whocc.no\/"},{"key":"zoi221372r29","unstructured":"Government of Canada. Forward Sortation Area\u2014definition. 2015. Accessed April 7, 2020. https:\/\/ised-isde.canada.ca\/site\/office-superintendent-bankruptcy\/en\/statistics-and-research\/forward-sortation-area-fsa-and-north-american-industry-classification-naics-reports\/forward-sortation-area-definition"},{"key":"zoi221372r30","unstructured":"College of Physicians and Surgeons of Alberta. OME and DDD conversion factors. 2017. Accessed July 8, 2021. https:\/\/cpsa.ca\/prescribing-tools-resources\/"},{"key":"zoi221372r37","unstructured":"Buitinck? L, Louppe? G, Blondel? M, . API design for machine learning software: experiences from the scikit-learn project.? arXiv. Preprint posted online September 1, 2013. doi:10.48550\/arXiv.1309.0238"},{"key":"zoi221372r39","unstructured":"Pandas Development Team. pandas-dev\/pandas. October 18, 2019. Accessed July 8, 2021. https:\/\/github.com\/pandas-dev\/pandas"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2799944\/sharma_2022_oi_221372_1671208873.89443.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,27]],"date-time":"2022-12-27T16:00:53Z","timestamp":1672156853000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2799944"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,27]]},"references-count":40,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2022,12,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2022.48559","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,27]]}}}