{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T21:30:31Z","timestamp":1777498231599,"version":"3.51.4"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2024,2,27]],"date-time":"2024-02-27T00:00:00Z","timestamp":1708992000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000092","name":"National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM013362"],"award-info":[{"award-number":["R01LM013362"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,4,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Model development included 41\u00a0929 patients, 14.6% with POU. The external validation included 31\u00a0932 (UK), 23\u00a0100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD\u2009=\u20090.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P\u2009&amp;lt;\u2009.05).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae028","type":"journal-article","created":{"date-parts":[[2024,2,27]],"date-time":"2024-02-27T20:21:07Z","timestamp":1709065267000},"page":"1051-1061","source":"Crossref","is-referenced-by-count":10,"title":["Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the\u00a0OHDSI network"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4429-5341","authenticated-orcid":false,"given":"Behzad","family":"Naderalvojoud","sequence":"first","affiliation":[{"name":"Department of Medicine, Stanford University , Stanford, CA 94305, United States"}]},{"given":"Catherine M","family":"Curtin","sequence":"additional","affiliation":[{"name":"Department of Surgery, Veterans Affairs Palo Alto Health Care System , Palo Alto, CA 94304, United States"}]},{"given":"Chen","family":"Yanover","sequence":"additional","affiliation":[{"name":"KI Research Institute , Kfar Malal, 4592000, Israel"}]},{"given":"Tal","family":"El-Hay","sequence":"additional","affiliation":[{"name":"KI Research Institute , Kfar Malal, 4592000, Israel"}]},{"given":"Byungjin","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Ajou University Graduate School of Medicine , Suwon, 16499, Korea"}]},{"given":"Rae Woong","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Ajou University Graduate School of Medicine , Suwon, 16499, Korea"}]},{"given":"Javier Gracia","family":"Tabuenca","sequence":"additional","affiliation":[{"name":"Institute for Molecular Medicine Finland (FIMM), HiLIFE, University of Helsinki , Helsinki, 00014, Finland"}]},{"given":"Mary Pat","family":"Reeve","sequence":"additional","affiliation":[{"name":"Institute for Molecular Medicine Finland (FIMM), HiLIFE, University of Helsinki , Helsinki, 00014, Finland"}]},{"given":"Thomas","family":"Falconer","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, NY 10032, United States"}]},{"given":"Keith","family":"Humphreys","sequence":"additional","affiliation":[{"name":"Department of Psychiatry and the Behavioral Sciences, Stanford University , Stanford, CA 94305, United States"},{"name":"Center for Innovation to Implementation, Veterans Affairs Palo Alto Health Care System , Palo Alto, CA 94304, United States"}]},{"given":"Steven M","family":"Asch","sequence":"additional","affiliation":[{"name":"Department of Medicine, Stanford University , Stanford, CA 94305, United States"},{"name":"Center for Innovation to Implementation, Veterans Affairs Palo Alto Health Care System , Palo Alto, CA 94304, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6553-3455","authenticated-orcid":false,"given":"Tina","family":"Hernandez-Boussard","sequence":"additional","affiliation":[{"name":"Department of Medicine, Stanford University , Stanford, CA 94305, United States"}]}],"member":"286","published-online":{"date-parts":[[2024,2,27]]},"reference":[{"issue":"6","key":"2024041923354279100_ocae028-B1","doi-asserted-by":"crossref","first-page":"e210097","DOI":"10.1148\/ryai.2021210097","article-title":"Toward generalizability in the deployment of artificial intelligence in radiology: role of computation stress testing to overcome underspecification","volume":"3","author":"Eche","year":"2021","journal-title":"Radiol Artif Intell"},{"issue":"1","key":"2024041923354279100_ocae028-B2","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1038\/s41746-023-00868-x","article-title":"Challenges of implementing computer-aided diagnostic models for neuroimages in a clinical setting","volume":"6","author":"Leming","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"10","key":"2024041923354279100_ocae028-B3","doi-asserted-by":"crossref","first-page":"e29301","DOI":"10.2196\/29301","article-title":"Adoption of machine learning systems for medical diagnostics in clinics: qualitative interview study","volume":"23","author":"Pumplun","year":"2021","journal-title":"J Med Internet Res"},{"issue":"8","key":"2024041923354279100_ocae028-B4","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern Med"},{"issue":"3","key":"2024041923354279100_ocae028-B5","doi-asserted-by":"crossref","first-page":"e007988","DOI":"10.1161\/CIRCEP.119.007988","article-title":"Assessing and mitigating bias in medical artificial intelligence: the effects of race and ethnicity on a deep learning model for ECG analysis","volume":"13","author":"Noseworthy","year":"2020","journal-title":"Circ Arrhythm Electrophysiol"},{"issue":"12","key":"2024041923354279100_ocae028-B6","doi-asserted-by":"crossref","first-page":"2176","DOI":"10.1038\/s41591-021-01595-0","article-title":"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations","volume":"27","author":"Seyyed-Kalantari","year":"2021","journal-title":"Nat Med"},{"issue":"1","key":"2024041923354279100_ocae028-B7","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1038\/s41746-023-00832-9","article-title":"Perspectives on validation of clinical predictive algorithms","volume":"6","author":"de Hond","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"7","key":"2024041923354279100_ocae028-B8","doi-asserted-by":"crossref","first-page":"1119","DOI":"10.1016\/j.spinee.2022.02.009","article-title":"A machine learning algorithm for predicting prolonged postoperative opioid prescription after lumbar disc herniation surgery. An external validation study using 1,316 patients from a Taiwanese cohort","volume":"22","author":"Yen","year":"2022","journal-title":"Spine J"},{"issue":"3","key":"2024041923354279100_ocae028-B9","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1158\/1055-9965.EPI-22-0873","article-title":"Machine learning and real-world data to predict lung cancer risk in routine care","volume":"32","author":"Chandran","year":"2023","journal-title":"Cancer Epidemiol Biomarkers Prev"},{"issue":"1","key":"2024041923354279100_ocae028-B10","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1186\/s12874-022-01793-5","article-title":"External validation of existing dementia prediction models on observational health data","volume":"22","author":"John","year":"2022","journal-title":"BMC Med Res Methodol"},{"key":"2024041923354279100_ocae028-B11","doi-asserted-by":"crossref","first-page":"106394","DOI":"10.1016\/j.cmpb.2021.106394","article-title":"A standardized analytics pipeline for reliable and rapid development and validation of prediction models using observational health data","volume":"211","author":"Khalid","year":"2021","journal-title":"Comput Methods Programs Biomed"},{"issue":"5","key":"2024041923354279100_ocae028-B12","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1007\/s40264-022-01161-8","article-title":"Using iterative pairwise external validation to contextualize prediction model performance: a use case predicting 1-year heart failure risk in patients with diabetes across five data sources","volume":"45","author":"Williams","year":"2022","journal-title":"Drug Saf"},{"issue":"6","key":"2024041923354279100_ocae028-B13","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1093\/jamia\/ocaa277","article-title":"Machine-learning model to predict the cause of death using a stacking ensemble method for observational data","volume":"28","author":"Kim","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2024041923354279100_ocae028-B14","doi-asserted-by":"crossref","first-page":"e0228632","DOI":"10.1371\/journal.pone.0228632","article-title":"Wisdom of the CROUD: development and validation of a patient-level prediction model for opioid use disorder using population-level claims data","volume":"15","author":"Reps","year":"2020","journal-title":"PLoS One"},{"issue":"12","key":"2024041923354279100_ocae028-B15","doi-asserted-by":"crossref","first-page":"e0226255","DOI":"10.1371\/journal.pone.0226255","article-title":"Inferring disease severity in rheumatoid arthritis using predictive modeling in administrative claims databases","volume":"14","author":"Chandran","year":"2019","journal-title":"PLoS One"},{"issue":"1","key":"2024041923354279100_ocae028-B16","doi-asserted-by":"crossref","first-page":"e0226718","DOI":"10.1371\/journal.pone.0226718","article-title":"Development and validation of a prognostic model predicting symptomatic hemorrhagic transformation in acute ischemic stroke at scale in the OHDSI network","volume":"15","author":"Wang","year":"2020","journal-title":"PLoS One"},{"issue":"1","key":"2024041923354279100_ocae028-B17","doi-asserted-by":"crossref","first-page":"642","DOI":"10.1038\/s41398-021-01760-6","article-title":"Predictors of diagnostic transition from major depressive disorder to bipolar disorder: a retrospective observational network study","volume":"11","author":"Nestsiarovich","year":"2021","journal-title":"Transl Psychiatry"},{"issue":"4","key":"2024041923354279100_ocae028-B18","doi-asserted-by":"crossref","first-page":"e21547","DOI":"10.2196\/21547","article-title":"Implementation of the COVID-19 vulnerability index across an international network of health care data sets: collaborative external validation study","volume":"9","author":"Reps","year":"2021","journal-title":"JMIR Med Inform"},{"issue":"4","key":"2024041923354279100_ocae028-B19","doi-asserted-by":"crossref","first-page":"288","DOI":"10.3390\/jpm10040288","article-title":"Prediction of major depressive disorder following beta-blocker therapy in patients with cardiovascular diseases","volume":"10","author":"Jin","year":"2020","journal-title":"J Pers Med"},{"issue":"9","key":"2024041923354279100_ocae028-B20","doi-asserted-by":"crossref","first-page":"3068","DOI":"10.1007\/s00167-021-06799-y","article-title":"90-Day all-cause mortality can be predicted following a total knee replacement: an international, network study to develop and validate a prediction model","volume":"30","author":"Williams","year":"2022","journal-title":"Knee Surg Sports Traumatol Arthrosc"},{"issue":"3","key":"2024041923354279100_ocae028-B21","doi-asserted-by":"crossref","first-page":"e33044","DOI":"10.2196\/33044","article-title":"A roadmap for boosting model generalizability for predicting hospital encounters for asthma","volume":"10","author":"Luo","year":"2022","journal-title":"JMIR Med Inform"},{"issue":"1","key":"2024041923354279100_ocae028-B22","doi-asserted-by":"crossref","first-page":"e130","DOI":"10.1017\/cts.2023.548","article-title":"Electronic health record data quality variability across a multistate clinical research network","volume":"7","author":"Mohamed","year":"2023","journal-title":"J Clin Transl Sci"},{"issue":"1","key":"2024041923354279100_ocae028-B23","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1038\/s41597-021-01110-7","article-title":"Peeking into a black box, the fairness and generalizability of a MIMIC-III benchmarking model","volume":"9","author":"R\u00f6\u00f6sli","year":"2022","journal-title":"Sci Data"},{"key":"2024041923354279100_ocae028-B24","author":"Naderalvojoud","year":"2023"},{"issue":"8","key":"2024041923354279100_ocae028-B25","doi-asserted-by":"crossref","first-page":"e0287697","DOI":"10.1371\/journal.pone.0287697","article-title":"Postoperative opioid prescribing patients with diabetes: Opportunities for personalized pain management","volume":"18","author":"Zammit","year":"2023","journal-title":"PLoS One"},{"issue":"4","key":"2024041923354279100_ocae028-B26","doi-asserted-by":"crossref","first-page":"390","DOI":"10.7861\/clinmedicine.5-4-390","article-title":"Pain symptoms in depression: definition and clinical significance","volume":"5","author":"Katona","year":"2005","journal-title":"Clin Med (Lond)"},{"issue":"3","key":"2024041923354279100_ocae028-B27","doi-asserted-by":"crossref","first-page":"1047","DOI":"10.1016\/j.jpainsymman.2017.11.005","article-title":"Challenges to safe and effective pain management in patients with super obesity: case report and literature review","volume":"55","author":"Kramer","year":"2018","journal-title":"J Pain Symptom Manage"},{"issue":"1","key":"2024041923354279100_ocae028-B28","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1186\/s12911-021-01408-x","article-title":"An empirical analysis of dealing with patients who are lost to follow-up when developing prognostic models using a cohort design","volume":"21","author":"Reps","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"issue":"2","key":"2024041923354279100_ocae028-B29","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1213\/ANE.0000000000005527","article-title":"Prediction of prolonged opioid use after surgery in adolescents: insights from machine learning","volume":"133","author":"Ward","year":"2021","journal-title":"Anesth Analg"},{"issue":"10","key":"2024041923354279100_ocae028-B30","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1016\/j.jhsa.2016.07.113","article-title":"Risk of prolonged opioid use among opioid-na\u00efve patients following common hand surgery procedures","volume":"41","author":"Johnson","year":"2016","journal-title":"J Hand Surg Am"},{"key":"2024041923354279100_ocae028-B31","first-page":"325","author":"Naderalvojoud","year":"2015"},{"issue":"8","key":"2024041923354279100_ocae028-B32","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1093\/jamia\/ocy032","article-title":"Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data","volume":"25","author":"Reps","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2024041923354279100_ocae028-B33","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.jclinepi.2007.04.018","article-title":"Updating methods improved the performance of a clinical prediction model in new patients","volume":"61","author":"Janssen","year":"2008","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2024041923354279100_ocae028-B34","doi-asserted-by":"crossref","first-page":"e220028","DOI":"10.1148\/ryai.220028","article-title":"Generalizability of machine learning models: quantitative evaluation of three methodological pitfalls","volume":"5","author":"Maleki","year":"2022","journal-title":"Radiol Artif Intell"},{"issue":"12","key":"2024041923354279100_ocae028-B35","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1038\/s42256-022-00563-8","article-title":"Generalizability of an acute kidney injury prediction model across health systems","volume":"4","author":"Cao","year":"2022","journal-title":"Nat Mach Intell"},{"issue":"1","key":"2024041923354279100_ocae028-B36","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1038\/s41746-022-00614-9","article-title":"Machine learning generalizability across healthcare settings: insights from multi-site COVID-19 screening","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"1","key":"2024041923354279100_ocae028-B37","doi-asserted-by":"crossref","first-page":"4423","DOI":"10.1038\/s41467-021-24698-1","article-title":"The impact of site-specific digital histology signatures on deep learning model accuracy and bias","volume":"12","author":"Howard","year":"2021","journal-title":"Nat Commun"},{"issue":"3","key":"2024041923354279100_ocae028-B38","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1016\/j.mayocpiqo.2022.03.003","article-title":"Assessing the generalizability of a clinical machine learning model across multiple emergency departments","volume":"6","author":"Ryu","year":"2022","journal-title":"Mayo Clin Proc Innov Qual Outcomes"},{"issue":"2","key":"2024041923354279100_ocae028-B39","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1080\/10618600.2020.1820345","article-title":"Boosting random forests to reduce bias; one-step boosted forest and its variance estimate","volume":"30","author":"Ghosal","year":"2020","journal-title":"J Comput Graph Stat"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/5\/1051\/57286309\/ocae028.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/5\/1051\/57286309\/ocae028.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,19]],"date-time":"2024-04-19T23:36:33Z","timestamp":1713569793000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/5\/1051\/7614966"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,27]]},"references-count":39,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,2,27]]},"published-print":{"date-parts":[[2024,4,19]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae028","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,5,1]]},"published":{"date-parts":[[2024,2,27]]}}}