{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T19:12:27Z","timestamp":1779390747086,"version":"3.53.1"},"reference-count":44,"publisher":"BMJ","issue":"5","license":[{"start":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T00:00:00Z","timestamp":1779321600000},"content-version":"unspecified","delay-in-days":20,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2026,4,21]]},"published-print":{"date-parts":[[2026,5]]},"abstract":"\n Objective<\/jats:title>\n To develop machine-learning (ML) models during the COVID-19 pandemic and adjacent time periods to evaluate the impact of data drift on model performance.<\/jats:p>\n <\/jats:sec>\n \n Design<\/jats:title>\n This prognostic study used population-level administrative health data to develop ML prediction models.<\/jats:p>\n <\/jats:sec>\n \n Setting<\/jats:title>\n Alberta, Canada during 2019\u20132023.<\/jats:p>\n <\/jats:sec>\n \n Participants<\/jats:title>\n All patients over 18 who received at least one opioid dispensation from a community pharmacy within the province of Alberta between 2019\u20132023.<\/jats:p>\n <\/jats:sec>\n \n Exposure<\/jats:title>\n Each opioid dispensation served as the unit-of-analysis.<\/jats:p>\n <\/jats:sec>\n \n Main outcomes\/measures<\/jats:title>\n Opioid-related outcomes were identified from linked health administrative datasets. Light Gradient Boosting-machine models were developed on pre-pandemic, pandemic and endemic data and temporally validated on 2023 data (pre-pandemic model was also validated on 2020\u20132021 data) to predict the risk of emergency department visit, hospitalisation or mortality within 30-days of an opioid dispensation. We described key feature distributions across the study time period and changes in model prediction performance on the validation sets using relevant metrics.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Among 1.2\u2009million study participants representing over 13\u2009million opioid dispensations, there were 59\u2009809 (2.1%), 134\u2009402 (2.4%) and 62\u2009143 (2.3%) events reported in the pre-pandemic (2019), pandemic (2020 and 2021) and endemic (2022) time periods, respectively (estimated 2023 validation set pre-test probability of 2.8%). Notable differences in key features were observed in the 2020\u20132021\u2009model relative to other years. In the 2023 validation set, discrimination performance was highest for the pre-pandemic and endemic models compared with the pandemic model (0.81, 0.83, 0.74, respectively). A similar trend regarding changes from pre-test to post-test probabilities in higher categories of predicted risk (23%, 40%, 16%) was observed. 2020\u20132021 had the lowest discrimination performance (0.71) and uninformative post-test probabilities (<10%).<\/jats:p>\n <\/jats:sec>\n \n Conclusion<\/jats:title>\n COVID-19 pandemic health data contributed to significant ML drift. Although ML approaches allow for quick re-training to mitigate drift, health regulators should approach ML prediction with caution when using pandemic-times data.<\/jats:p>\n <\/jats:sec>","DOI":"10.1136\/bmjopen-2025-103318","type":"journal-article","created":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T18:35:19Z","timestamp":1779388519000},"page":"e103318","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":0,"title":["Impact of COVID-19-related data drift on machine-learning prognostic models predicting 30-day opioid-related emergency department visits, hospitalisation or mortality: a population-level administrative data study in Alberta, Canada"],"prefix":"10.1136","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7907-1183","authenticated-orcid":false,"given":"Vishal","family":"Sharma","sequence":"first","affiliation":[{"name":"School of Public Health, University of Alberta, Li Ka Shing Centre for Health Research Innovation, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiting","family":"Li","sequence":"additional","affiliation":[{"name":"Okaki Health Intelligence Inc, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tanya","family":"Joon","sequence":"additional","affiliation":[{"name":"Okaki Health Intelligence Inc, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cerina","family":"Dubois","sequence":"additional","affiliation":[{"name":"School of Public Health, University of Alberta, Li Ka Shing Centre for Health Research Innovation, Edmonton, Alberta, Canada"},{"name":"Department of Mental Health, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Darren","family":"Lau","sequence":"additional","affiliation":[{"name":"Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ed","family":"Jess","sequence":"additional","affiliation":[{"name":"College of Physicians and Surgeons of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2851-265X","authenticated-orcid":false,"given":"Cliff","family":"Lindeman","sequence":"additional","affiliation":[{"name":"College of Physicians and Surgeons of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4701-7873","authenticated-orcid":false,"given":"Nicole Allison","family":"Kain","sequence":"additional","affiliation":[{"name":"College of Physicians and Surgeons of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Ye","sequence":"additional","affiliation":[{"name":"School of Public Health, University of Alberta, Li Ka Shing Centre for Health Research Innovation, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Semenchuk","sequence":"additional","affiliation":[{"name":"Okaki Health Intelligence Inc, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2197-0463","authenticated-orcid":false,"given":"Dean T","family":"Eurich","sequence":"additional","affiliation":[{"name":"School of Public Health, University of Alberta, Li Ka Shing Centre for Health Research Innovation, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salim","family":"Samanani","sequence":"additional","affiliation":[{"name":"Okaki Health Intelligence Inc, Calgary, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,5,21]]},"reference":[{"key":"2026052111350995000_16.5.e103318.1","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2020-037860","article-title":"Predicting population health with machine learning: a scoping review","volume":"10","author":"Morgenstern","year":"2020","journal-title":"BMJ Open"},{"key":"2026052111350995000_16.5.e103318.2","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2019.0968","article-title":"Evaluation of machine-learning algorithms for predicting opioid overdose risk among medicare beneficiaries with opioid prescriptions","volume":"2","author":"Lo-Ciganic","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2026052111350995000_16.5.e103318.3","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-024-01312-4","article-title":"A systematic review of machine learning applications in predicting opioid associated adverse events","volume":"8","author":"Ram\u00edrez Medina","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026052111350995000_16.5.e103318.4","doi-asserted-by":"crossref","first-page":"100437","DOI":"10.1016\/j.lana.2023.100437","article-title":"The continuous opioid death crisis in Canada: changing characteristics and implications for path options forward","volume":"19","author":"Fischer","year":"2023","journal-title":"The Lancet Regional Health - Americas"},{"key":"2026052111350995000_16.5.e103318.5","unstructured":"Public Health Agency of Canada . Federal, provincial, and territorial special advisory committee on toxic drug poisonings. opioid- and stimulant-related harms in Canada. opioid- and stimulant-related harms in Canada. Available: https:\/\/health-infobase.canada.ca\/substance-related-harms\/opioids-stimulants\/#a9 [Accessed 30 Jan 2025]."},{"key":"2026052111350995000_16.5.e103318.6","unstructured":"College of Physicians and Surgeons of Alberta . Tracked prescription program. 2021. Available: https:\/\/www.tppalberta.ca"},{"key":"2026052111350995000_16.5.e103318.7","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2019.16489"},{"key":"2026052111350995000_16.5.e103318.8","unstructured":"Alberta Machine Intelligence Institute . Machine learning process lifecycle. 2019."},{"key":"2026052111350995000_16.5.e103318.9","unstructured":"Ali M . Understanding data drift and model drift: drift detection in python. Available: https:\/\/www.datacamp.com\/tutorial\/understanding-data-drift-model-drift [accessed 25 Jan 2025]."},{"key":"2026052111350995000_16.5.e103318.10","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2020-045343","article-title":"Impact of COVID-19 pandemic on utilisation of healthcare services: a systematic review","volume":"11","author":"Moynihan","year":"2021","journal-title":"BMJ Open"},{"key":"2026052111350995000_16.5.e103318.11","unstructured":"Canadian Institute for Health Information . Impact of COVID-19 on Canada\u2019s health care systems. 2022."},{"key":"2026052111350995000_16.5.e103318.12","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2022.48559","article-title":"Development and validation of a machine learning model to estimate risk of adverse outcomes within 30 days of opioid dispensation","volume":"5","author":"Sharma","year":"2022","journal-title":"JAMA Netw Open"},{"key":"2026052111350995000_16.5.e103318.13","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2020-043964","article-title":"Safe opioid prescribing: a prognostic machine learning approach to predicting 30-day risk after an opioid dispensation in Alberta, Canada","volume":"11","author":"Sharma","year":"2021","journal-title":"BMJ Open"},{"key":"2026052111350995000_16.5.e103318.14","doi-asserted-by":"publisher","DOI":"10.1503\/cmaj.231156"},{"key":"2026052111350995000_16.5.e103318.15","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0290646","article-title":"The impact of the early COVID-19 pandemic on healthcare system resource use and costs in two provinces in Canada: an interrupted time series analysis","volume":"18","author":"Zeitouny","year":"2023","journal-title":"PLoS One"},{"key":"2026052111350995000_16.5.e103318.16","unstructured":"Equator network . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod): the tripod statement. Available: https:\/\/www.equator-network.org\/reporting-guidelines\/tripod-statement [Accessed Feb 2020]."},{"key":"2026052111350995000_16.5.e103318.17","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2019-034568","article-title":"Reporting quality of studies using machine learning models for medical diagnosis: a systematic review","volume":"10","author":"Yusuf","year":"2020","journal-title":"BMJ Open"},{"key":"2026052111350995000_16.5.e103318.18","unstructured":"Alberta college of pharmacy. Available: https:\/\/abpharmacy.ca [Accessed Sep 2019]."},{"key":"2026052111350995000_16.5.e103318.19","unstructured":"World health organization . Classification of diseases (ICD), Available: https:\/\/www.who.int\/classifications\/icd\/icdonlineversions\/en [Accessed Jun 2020]."},{"key":"2026052111350995000_16.5.e103318.20","unstructured":"Canadian Institute for Health Information , 2019. Available: https:\/\/www.cihi.ca\/en"},{"key":"2026052111350995000_16.5.e103318.21","unstructured":"World Health Organization . International language for drug utilization research, atc\/ddd. Available: https:\/\/www.whocc.no [Accessed Jun 2020]."},{"key":"2026052111350995000_16.5.e103318.22","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.k3207","article-title":"Contributions of prescribed and non-prescribed opioids to opioid related deaths: population based cohort study in Ontario, Canada","volume":"362","author":"Gomes","year":"2018","journal-title":"BMJ"},{"key":"2026052111350995000_16.5.e103318.23","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2016-011060","article-title":"Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review","volume":"6","author":"Zhou","year":"2016","journal-title":"BMJ Open"},{"key":"2026052111350995000_16.5.e103318.24","unstructured":"Brownlee J . A gentle introduction to imbalanced classification, Available: https:\/\/machinelearningmastery.com\/what-is-imbalanced-classification [Accessed Jan 2021]."},{"key":"2026052111350995000_16.5.e103318.25","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordjournals.pan.a004868"},{"key":"2026052111350995000_16.5.e103318.26","doi-asserted-by":"crossref","unstructured":"Dowell D . CDC guideline for prescribing opioids for chronic pain. 2016. doi:10.1001\/jama.2016.9242","DOI":"10.1001\/jama.2016.9242"},{"key":"2026052111350995000_16.5.e103318.27","doi-asserted-by":"publisher","DOI":"10.1503\/cmaj.170363"},{"key":"2026052111350995000_16.5.e103318.28","unstructured":"Government of Canada . Forward sortation area\u2014definition. Available: https:\/\/www.ic.gc.ca\/eic\/site\/bsf-osb.nsf\/eng\/br03396.html [Accessed Apr 2020]."},{"key":"2026052111350995000_16.5.e103318.29","unstructured":"College of Physicians and Surgeons of Alberta . OME and DDD conversion factors. 2020 Available: http:\/\/www.cpsa.ca\/wp-content\/uploads\/2017\/06\/OME-and-DDD-Conversion-Factors.pdf"},{"key":"2026052111350995000_16.5.e103318.30","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehw302"},{"key":"2026052111350995000_16.5.e103318.31","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2018.17163"},{"key":"2026052111350995000_16.5.e103318.32","unstructured":"Molnar C . Interpretable machine learning. a guide for making black box models explainable. 2019."},{"key":"2026052111350995000_16.5.e103318.33","unstructured":"Lundberg SM , Lee S-I . A Unified Approach to Interpreting Model Predictions. Paper Presented at: Advances in Neural Information Processing Systems. 2017."},{"key":"2026052111350995000_16.5.e103318.34","unstructured":"Lin A . Examining distributional shifts by using population stability index (psi) for model validation and diagnosis 2018. Available: https:\/\/github.com\/mwburke\/population-stability-index [accessed Aug 2025]."},{"key":"2026052111350995000_16.5.e103318.35","doi-asserted-by":"crossref","DOI":"10.2196\/jmir.5870","article-title":"Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view","volume":"18","author":"Luo","year":"2016","journal-title":"J Med Internet Res"},{"key":"2026052111350995000_16.5.e103318.36","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.12126"},{"key":"2026052111350995000_16.5.e103318.37","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1994.03510330081039"},{"key":"2026052111350995000_16.5.e103318.38","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehu207"},{"key":"2026052111350995000_16.5.e103318.39","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0118432","article-title":"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets","volume":"10","author":"Saito","year":"2015","journal-title":"PLoS ONE"},{"key":"2026052111350995000_16.5.e103318.40","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1994.03510290071040"},{"key":"2026052111350995000_16.5.e103318.41","unstructured":"Pedregosa F , et al . Scikit-learn: machine learning in python. Available: https:\/\/scikit-learn.org\/stable [accessed 15 Aug 2015]."},{"key":"2026052111350995000_16.5.e103318.42","doi-asserted-by":"publisher","DOI":"10.1109\/lsp.2014.2337313"},{"key":"2026052111350995000_16.5.e103318.43","doi-asserted-by":"publisher","DOI":"10.17269\/s41997-022-00702-z"},{"key":"2026052111350995000_16.5.e103318.44","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.ijregi.2022.08.011","article-title":"Excess deaths during the COVID-19 pandemic in Alberta, Canada","volume":"5","author":"Beesoon","year":"2022","journal-title":"IJID Reg"}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2025-103318","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T18:35:29Z","timestamp":1779388529000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2025-103318"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5]]},"references-count":44,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,5,21]]},"published-print":{"date-parts":[[2026,5]]}},"alternative-id":["10.1136\/bmjopen-2025-103318"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2025-103318","relation":{},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5]]}}}