{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T12:05:33Z","timestamp":1790078733725,"version":"4.0.1"},"reference-count":51,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T00:00:00Z","timestamp":1789603200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Clin. Med.","JCM"],"abstract":"<jats:p>Background\/Objectives: Traditional machine learning metrics often fail to capture the clinical consequences of classification errors, particularly in high-stakes screening applications. When applying task-specific AI tools to human lives, clinicians cannot rely on simple, headline metrics and marketing materials. Impressive accuracy figures can hide dangerous algorithmic shortcuts that collapse when encountering actual patients. This study addresses the gap between statistical performance and clinical utility by evaluating classifiers for prescription opioid misuse using a clinically oriented framework. Methods: Three ensemble models, namely, Cost-Sensitive Bagged Trees (CSBT-Untuned and CSBT-Tuned) and Random Undersampling Boosting (RUSBoost), were trained on National Survey on Drug Use and Health data and assessed using composite metrics integrating clinical priorities and asymmetric error costs. Results: The results demonstrate that while CSBT-Untuned achieved the highest raw accuracy of 86.7%, it missed 61.5% of positive cases. Conversely, the threshold-optimized CSBT-Tuned model achieved enhanced minority class detection and numerically higher Clinical Discriminative Performance Scores under sensitivity-priority scenarios, though its performance remained statistically comparable to RUSBoost given overlapping confidence intervals. Learning curve analysis confirmed stable convergence for the cost-sensitive bagging approach. Conclusions: Before accepting AI tools in patient care, physicians must demand an evaluation report similarly detailed to this manuscript, utilizing this framework as guidance for what an evidence report should demonstrate prior to clinical deployment.<\/jats:p>","DOI":"10.3390\/jcm15187237","type":"journal-article","created":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T11:09:35Z","timestamp":1790075375000},"page":"7237","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Integrating Clinical Priorities into the Technical Validation of Machine Learning Classifiers: A Generalizable Framework Demonstrated on Opioid Misuse Screening"],"prefix":"10.3390","volume":"15","author":[{"given":"Jacob","family":"Washton","sequence":"first","affiliation":[{"name":"Laboratory of Algorithmic Medicine, Department of Osteopathic Manipulative Medicine, College of Osteopathic Medicine, New York Institute of Technology, Old Westbury, NY 11568, USA"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2154-3728","authenticated-orcid":false,"given":"Milan","family":"Toma","sequence":"additional","affiliation":[{"name":"Laboratory of Algorithmic Medicine, Department of Osteopathic Manipulative Medicine, College of Osteopathic Medicine, New York Institute of Technology, Old Westbury, NY 11568, USA"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Toma, M., and Yusupov, D. (2026). Perspectives on the Limits and Clinical Alignment of Medical AI from Population Statistics to Individual Care. Bioengineering, 13.","DOI":"10.3390\/bioengineering13091034"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Ghanem, M., Ghaith, A.K., El-Hajj, V.G., Bhandarkar, A., de Giorgio, A., Elmi-Terander, A., and Bydon, M. (2023). Limitations in Evaluating Machine Learning Models for Imbalanced Binary Outcome Classification in Spine Surgery: A Systematic Review. Brain Sci., 13.","DOI":"10.3390\/brainsci13121723"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tarek, Z., Shams, M.Y., Towfek, S.K., Alkahtani, H.K., Ibrahim, A., Abdelhamid, A.A., Eid, M.M., Khodadadi, N., Abualigah, L., and Khafaga, D.S. (2023). An Optimized Model Based on Deep Learning and Gated Recurrent Unit for COVID-19 Death Prediction. Biomimetics, 8.","DOI":"10.3390\/biomimetics8070552"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Alkhammash, E.H., Hadjouni, M., and Elshewey, A.M. (2022). A Hybrid Ensemble Stacking Model for Gender Voice Recognition Approach. Electronics, 11.","DOI":"10.3390\/electronics11111750"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"765","DOI":"10.32604\/csse.2023.034324","article-title":"Weight Prediction Using the Hybrid Stacked-LSTM Food Selection Model","volume":"46","author":"Elshewey","year":"2023","journal-title":"Comput. Syst. Sci. Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1755","DOI":"10.1016\/j.jacc.2010.02.016","article-title":"ACCF\/AHA 2010 Position Statement on Composite Measures for Healthcare Performance Assessment","volume":"55","author":"Peterson","year":"2010","journal-title":"J. Am. Coll. Cardiol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1007\/s11760-025-04545-2","article-title":"Enhancing crop yield prediction based on dove optimization algorithm and gradient boosting model","volume":"19","author":"Elshewey","year":"2025","journal-title":"Signal Image Video Process."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"14372","DOI":"10.1038\/s41598-025-97401-9","article-title":"An enhanced CNN with ResNet50 and LSTM deep learning forecasting model for climate change decision making","volume":"15","author":"Elshewey","year":"2025","journal-title":"Sci. Rep."},{"key":"ref_9","first-page":"101344","article-title":"DNN-XGBoost framework for CO2 emissions prediction based on binary whale optimization and grey wolf optimization algorithms","volume":"50","author":"Elshewey","year":"2026","journal-title":"Sustain. Comput. Inform. Syst."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1111\/add.16133","article-title":"Outcome class imbalance and rare events: An underappreciated complication for overdose risk prediction modeling","volume":"118","author":"Cartus","year":"2023","journal-title":"Addiction"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"105886","DOI":"10.1016\/j.ypmed.2019.105886","article-title":"Using machine learning to predict opioid misuse among U.S. adolescents","volume":"130","author":"Han","year":"2020","journal-title":"Prev. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"e190968","DOI":"10.1001\/jamanetworkopen.2019.0968","article-title":"Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries with Opioid Prescriptions","volume":"2","author":"Huang","year":"2019","journal-title":"JAMA Netw. Open"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1105","DOI":"10.1080\/08870446.2019.1622013","article-title":"Opioid use and misuse: Health impact, prevalence, correlates and interventions","volume":"34","author":"Bolshakova","year":"2019","journal-title":"Psychol. Health"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"108350","DOI":"10.1016\/j.drugalcdep.2020.108350","article-title":"The economic burden of opioid use disorder and fatal opioid overdose in the United States, 2017","volume":"218","author":"Florence","year":"2021","journal-title":"Drug Alcohol Depend."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"e193365","DOI":"10.1001\/jamanetworkopen.2019.3365","article-title":"Strategies to Identify Patient Risks of Prescription Opioid Addiction When Initiating Opioids for Pain: A Systematic Review","volume":"2","author":"Klimas","year":"2019","journal-title":"JAMA Netw. Open"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"622","DOI":"10.1176\/appi.ps.202200034","article-title":"Use of Predictive Analytics to Identify Unhealthy Opioid Use and Guide Intervention","volume":"74","author":"Valdes","year":"2023","journal-title":"Psychiatr. Serv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.15585\/mmwr.rr7103a1","article-title":"CDC Clinical Practice Guideline for Prescribing Opioids for Pain\u2014United States, 2022","volume":"71","author":"Dowell","year":"2022","journal-title":"MMWR Recomm. Rep."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Picco, L., Jung, M., Cangadis-Douglass, H., Lam, T., and Nielsen, S. (2023). Identifying Prescription-Opioid-Related Risks Using Prescription Drug Monitoring Programs\u2019 Algorithms and Clinical Screening Tools. Pharmacy, 11.","DOI":"10.3390\/pharmacy11050164"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.ajem.2026.02.037","article-title":"Machine learning models to detect opioid misuse in emergency department patients at triage","volume":"104","author":"Chhablani","year":"2026","journal-title":"Am. J. Emerg. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1097\/PCC.0000000000002942","article-title":"Evaluation of Machine Learning Models for Clinical Prediction Problems","volume":"23","author":"Bennett","year":"2022","journal-title":"Pediatr. Crit. Care Med."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Park, C., Son, M., Kim, J., Kim, B., Ahn, Y., and Kwon, N. (2025). TOPSIS and AHP-Based Multi-Criteria Decision-Making Approach for Evaluating Redevelopment in Old Residential Projects. Sustainability, 17.","DOI":"10.3390\/su17157072"},{"key":"ref_22","first-page":"1","article-title":"Learning dynamics vs. aggregate metrics: Clinical applicability of machine learning survival prediction in eye cancer","volume":"4","author":"Shin","year":"2026","journal-title":"Acad. Biol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/TSMCA.2009.2029559","article-title":"RUSBoost: A Hybrid Approach to Alleviating Class Imbalance","volume":"40","author":"Seiffert","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part Syst. Hum."},{"key":"ref_24","unstructured":"Toma, M. (2025). AI-Assisted Medical Diagnostics: A Clinical Guide to Next-Generation Diagnostics, Dawning Research Press. Available online: https:\/\/openlibrary.org\/works\/OL44048041W\/."},{"key":"ref_25","unstructured":"Toma, M. (2026). Diagnosing AI: Evaluation of AI in Clinical Practice, Dawning Research Press. Available online: https:\/\/openlibrary.org\/works\/OL61375300M\/."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"7799","DOI":"10.1109\/TPAMI.2022.3220744","article-title":"The Shape of Learning Curves: A Review","volume":"45","author":"Viering","year":"2023","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Montesinos L\u00f3pez, O.A., Montesinos L\u00f3pez, A., and Crossa, J. (2022). Overfitting, Model Tuning, and Evaluation of Prediction Performance. Multivariate Statistical Machine Learning Methods for Genomic Prediction, Springer International Publishing.","DOI":"10.1007\/978-3-030-89010-0"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Varoquaux, G., and Colliot, O. (2023). Evaluating Machine Learning Models and Their Diagnostic Value. Machine Learning for Brain Disorders, Springer.","DOI":"10.1007\/978-1-0716-3195-9_20"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"23218","DOI":"10.1021\/acsomega.3c03722","article-title":"Interpretation of Machine Learning Models for Data Sets with Many Features Using Feature Importance","volume":"8","author":"Kaneko","year":"2023","journal-title":"ACS Omega"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Carvalho, D.V., Pereira, E.M., and Cardoso, J.S. (2019). Machine Learning Interpretability: A Survey on Methods and Metrics. Electronics, 8.","DOI":"10.3390\/electronics8080832"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.21037\/jhmhp-2026-0023","article-title":"A cost-effectiveness analysis of AI-assisted OCT classification for age-related macular degeneration","volume":"10","author":"Daly","year":"2026","journal-title":"J. Hosp. Manag. Health Policy"},{"key":"ref_32","unstructured":"Clemen, R.T., and Reilly, T. (2013). Making Hard Decisions with DecisionTools, South-Western College Publishing. [3rd ed.]."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e48763","DOI":"10.2196\/48763","article-title":"Consolidated Reporting Guidelines for Prognostic and Diagnostic Machine Learning Modeling Studies: Development and Validation","volume":"25","author":"Klement","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"e034568","DOI":"10.1136\/bmjopen-2019-034568","article-title":"Reporting quality of studies using machine learning models for medical diagnosis: A systematic review","volume":"10","author":"Yusuf","year":"2020","journal-title":"BMJ Open"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"6556","DOI":"10.1161\/CIRCOUTCOMES.120.006556","article-title":"Recommendations for Reporting Machine Learning Analyses in Clinical Research","volume":"13","author":"Stevens","year":"2020","journal-title":"Circ. Cardiovasc. Qual. Outcomes"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"5979","DOI":"10.1038\/s41598-022-09954-8","article-title":"On evaluation metrics for medical applications of artificial intelligence","volume":"12","author":"Hicks","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2200302","DOI":"10.1002\/bimj.202200302","article-title":"Stability of clinical prediction models developed using statistical or machine learning methods","volume":"65","author":"Riley","year":"2023","journal-title":"Biom. J."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1109\/TKDE.2008.239","article-title":"Learning from Imbalanced Data","volume":"21","author":"He","year":"2009","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Valverde-Albacete, F.J., and Pel\u00e1ez-Moreno, C. (2014). 100% Classification Accuracy Considered Harmful: The Normalized Information Transfer Factor Explains the Accuracy Paradox. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0084217"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"e078378","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD + AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","year":"2024","journal-title":"BMJ"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"100128","DOI":"10.1016\/j.dadr.2022.100128","article-title":"Specific polysubstance use patterns predict relapse among patients entering opioid use disorder treatment","volume":"5","author":"Pan","year":"2022","journal-title":"Drug Alcohol Depend. Rep."},{"key":"ref_42","first-page":"1","article-title":"Prevalence of alcohol use and depressive\/anxiety symptoms among patients of opioid agonist treatment programs in Ukraine","volume":"1","author":"Klymchuk","year":"2024","journal-title":"Acad. Ment. Health Well-Being"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"293","DOI":"10.7326\/M17-0865","article-title":"Prescription Opioid Use, Misuse, and Use Disorders in U.S. Adults: 2015 National Survey on Drug Use and Health","volume":"167","author":"Han","year":"2017","journal-title":"Ann. Intern. Med."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1080\/15504263.2021.1979349","article-title":"Co-Occurring Opioid Use and Depressive Disorders: Patient Characteristics and Co-Occurring Health Conditions","volume":"17","author":"Tormohlen","year":"2021","journal-title":"J. Dual Diagn."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"567","DOI":"10.15288\/jsad.2018.79.567","article-title":"Marital Status as a Partial Mediator of the Associations Between Young Adult Substance Use and Subsequent Substance Use Disorder: Application of Causal Inference Methods","volume":"79","author":"Jang","year":"2018","journal-title":"J. Stud. Alcohol Drugs"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1186\/s13722-023-00362-5","article-title":"Screening for opioid use disorder and co-occurring depression and post-traumatic stress disorder in primary care in New Mexico","volume":"18","author":"Dopp","year":"2023","journal-title":"Addict. Sci. Clin. Pract."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"176381","DOI":"10.1109\/ACCESS.2025.3618851","article-title":"Validity of Feature Importance in Low-Performing Machine Learning for Tabular Biomedical Data","volume":"13","author":"Lee","year":"2025","journal-title":"IEEE Access"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Alqudah, A.M., and Moussavi, Z. (2026). CUES: A Multiplicative Composite Metric for Evaluating Clinical Prediction Models Theory, Inference, and Properties. Mathematics, 14.","DOI":"10.3390\/math14030398"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"100916","DOI":"10.1016\/j.landig.2025.100916","article-title":"Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: Overview and guidance","volume":"7","author":"Collins","year":"2025","journal-title":"Lancet Digit. Health"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"7526","DOI":"10.1161\/CIRCOUTCOMES.120.007526","article-title":"Performance Metrics for the Comparative Analysis of Clinical Risk Prediction Models Employing Machine Learning","volume":"14","author":"Huang","year":"2021","journal-title":"Circ. Cardiovasc. Qual. Outcomes"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"932","DOI":"10.1007\/s00330-014-3487-0","article-title":"Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach","volume":"25","author":"Halligan","year":"2015","journal-title":"Eur. Radiol."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/18\/7237\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T11:24:30Z","timestamp":1790076270000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/18\/7237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,17]]},"references-count":51,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["jcm15187237"],"URL":"https:\/\/doi.org\/10.3390\/jcm15187237","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,17]]}}}