{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T13:08:29Z","timestamp":1781788109379,"version":"3.54.5"},"reference-count":24,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,5,25]],"date-time":"2022-05-25T00:00:00Z","timestamp":1653436800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,25]],"date-time":"2022-05-25T00:00:00Z","timestamp":1653436800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100010767","name":"Innovative Medicines Initiative","doi-asserted-by":"publisher","award":["806968"],"award-info":[{"award-number":["806968"]}],"id":[{"id":"10.13039\/501100010767","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010767","name":"Innovative Medicines Initiative","doi-asserted-by":"publisher","award":["806968"],"award-info":[{"award-number":["806968"]}],"id":[{"id":"10.13039\/501100010767","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Prognostic models that are accurate could help aid medical decision making. Large observational databases often contain temporal medical data for large and diverse populations of patients. It may be possible to learn prognostic models using the large observational data. Often the performance of a prognostic model undesirably worsens when transported to a different database (or into a clinical setting). In this study we investigate different ensemble approaches that combine prognostic models independently developed using different databases (a simple federated learning approach) to determine whether ensembles that combine models developed across databases can improve model transportability (perform better in new data than single database models)?<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>For a given prediction question we independently trained five single database models each using a different observational healthcare database. We then developed and investigated numerous ensemble models (fusion, stacking and mixture of experts) that combined the different database models. Performance of each model was investigated via discrimination and calibration using a leave one dataset out technique, i.e., hold out one database to use for validation and use the remaining four datasets for model development. The internal validation of a model developed using the hold out database was calculated and presented as the \u2018internal benchmark\u2019 for comparison.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>In this study the fusion ensembles generally outperformed the single database models when transported to a previously unseen database and the performances were more consistent across unseen databases. Stacking ensembles performed poorly in terms of discrimination when the labels in the unseen database were limited. Calibration was consistently poor when both ensembles and single database models were applied to previously unseen databases.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>A simple federated learning approach that implements ensemble techniques to combine models independently developed across different databases for the same prediction question may improve the discriminative performance in new data (new database or clinical setting) but will need to be recalibrated using the new data. This could help medical decision making by improving prognostic model performance.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-022-01879-6","type":"journal-article","created":{"date-parts":[[2022,5,25]],"date-time":"2022-05-25T13:08:21Z","timestamp":1653484101000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Learning patient-level prediction models across multiple healthcare databases: evaluation of ensembles for increasing model transportability"],"prefix":"10.1186","volume":"22","author":[{"given":"Jenna Marie","family":"Reps","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ross D.","family":"Williams","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martijn J.","family":"Schuemie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick B.","family":"Ryan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter R.","family":"Rijnbeek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,5,25]]},"reference":[{"issue":"10","key":"1879_CR1","doi-asserted-by":"publisher","first-page":"e1288","DOI":"10.1016\/S2214-109X(19)30365-1","volume":"7","author":"F Farzadfar","year":"2019","unstructured":"Farzadfar F. Cardiovascular disease risk prediction models: challenges and perspectives. Lancet Glob Health. 2019;7(10):e1288\u20139.","journal-title":"Lancet Glob Health"},{"issue":"8","key":"1879_CR2","doi-asserted-by":"publisher","first-page":"969","DOI":"10.1093\/jamia\/ocy032","volume":"25","author":"JM Reps","year":"2018","unstructured":"Reps JM, Schuemie MJ, Suchard MA, et al. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc. 2018;25(8):969\u201375.","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"1879_CR3","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1016\/j.jclinepi.2014.06.018","volume":"68","author":"TP Debray","year":"2015","unstructured":"Debray TP, Vergouwe Y, Koffijberg H, et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol. 2015;68(3):279\u201389.","journal-title":"J Clin Epidemiol"},{"issue":"8","key":"1879_CR4","doi-asserted-by":"publisher","first-page":"971","DOI":"10.1093\/aje\/kwq223","volume":"172","author":"Y Vergouwe","year":"2010","unstructured":"Vergouwe Y, Moons KG, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol. 2010;172(8):971\u201380.","journal-title":"Am J Epidemiol"},{"issue":"3","key":"1879_CR5","doi-asserted-by":"publisher","first-page":"459","DOI":"10.1016\/j.radonc.2016.10.002","volume":"121","author":"A Jochems","year":"2016","unstructured":"Jochems A, Deist TM, Van Soest J, Eble M, Bulens P, Coucke P, Dries W, Lambin P, Dekker A. Distributed learning: developing a predictive model based on data from multiple hospitals without data leaving the hospital\u2013a real life proof of concept. Radiother Oncol. 2016;121(3):459\u201367.","journal-title":"Radiother Oncol"},{"issue":"1","key":"1879_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-020-61297-4","volume":"10","author":"M Bogowicz","year":"2020","unstructured":"Bogowicz M, Jochems A, Deist TM, Tanadini-Lang S, Huang SH, Chan B, Waldron JN, Bratman S, O\u2019Sullivan B, Riesterer O, Studer G. Privacy-preserving distributed learning of radiomics to predict overall survival and HPV status in head and neck cancer. Sci Rep. 2020;10(1):1\u201310.","journal-title":"Sci Rep"},{"key":"1879_CR7","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1016\/j.radonc.2019.11.019","volume":"144","author":"TM Deist","year":"2020","unstructured":"Deist TM, Dankers FJ, Ojha P, Marshall MS, Janssen T, Faivre-Finn C, Masciocchi C, Valentini V, Wang J, Chen J, Zhang Z. Distributed learning on 20,000+ lung cancer patients\u2013The Personal Health Train. Radiother Oncol. 2020;144:189\u2013200.","journal-title":"Radiother Oncol"},{"key":"1879_CR8","doi-asserted-by":"crossref","unstructured":"Luo C, Islam MN, Sheils NE, Reps JM, Buresh J, Duan R, Tong JM, Edmondson M, Schuemie MJ, Chen Y. Lossless distributed linear mixed model with application to integration of heterogeneous healthcare data.\u00a0medRxiv. 2020.","DOI":"10.1101\/2020.11.16.20230730"},{"issue":"1","key":"1879_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-020-69250-1","volume":"10","author":"MJ Sheller","year":"2020","unstructured":"Sheller MJ, Edwards B, Reina GA, Martin J, Pati S, Kotrotsou A, Milchenko M, Xu W, Marcus D, Colen RR, Bakas S. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Sci Rep. 2020;10(1):1\u20132.","journal-title":"Sci Rep"},{"issue":"1","key":"1879_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-020-00323-1","volume":"3","author":"N Rieke","year":"2020","unstructured":"Rieke N, Hancox J, Li W, Milletari F, Roth HR, Albarqouni S, Bakas S, Galtier MN, Landman BA, Maier-Hein K, Ourselin S. The future of digital health with federated learning. NPJ Digit Med. 2020;3(1):1\u20137.","journal-title":"NPJ Digit Med"},{"issue":"3","key":"1879_CR11","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1109\/MSP.2020.2975749","volume":"37","author":"T Li","year":"2020","unstructured":"Li T, Sahu AK, Talwalkar A, Smith V. Federated learning: Challenges, methods, and future directions. IEEE Signal Process Mag. 2020;37(3):50\u201360.","journal-title":"IEEE Signal Process Mag"},{"key":"1879_CR12","unstructured":"McMahan B, Moore E, Ramage D, Hampson S, y Arcas BA. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics 2017;1273\u20131282."},{"key":"1879_CR13","doi-asserted-by":"crossref","unstructured":"Tsoumakas G, Partalas I, Vlahavas I. A taxonomy and short review of ensemble selection. In\u00a0Workshop on Supervised and Unsupervised Ensemble Methods and Their Applications\u00a02008;1\u20136.","DOI":"10.1007\/978-3-642-03999-7_1"},{"issue":"1\u20132","key":"1879_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10462-009-9124-7","volume":"33","author":"L Rokach","year":"2010","unstructured":"Rokach L. Ensemble-based classifiers. Artif Intell Rev. 2010;33(1\u20132):1\u201339. https:\/\/doi.org\/10.1007\/s10462-009-9124-7.","journal-title":"Artif Intell Rev"},{"key":"1879_CR15","doi-asserted-by":"crossref","unstructured":"Fumera G, Roli F. Performance analysis and comparison of linear combiners for classifier fusion. In\u00a0Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR) Springer, Berlin, Heidelberg.\u00a02002:424\u2013432.","DOI":"10.1007\/3-540-70659-3_44"},{"issue":"2","key":"1879_CR16","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/S0893-6080(05)80023-1","volume":"5","author":"DH Wolpert","year":"1992","unstructured":"Wolpert DH. Stacked generalization. Neural Netw. 1992;5(2):241\u201359.","journal-title":"Neural Netw"},{"issue":"2","key":"1879_CR17","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1007\/s10462-012-9338-y","volume":"42","author":"S Masoudnia","year":"2014","unstructured":"Masoudnia S, Ebrahimpour R. Mixture of experts: a literature survey. Artif Intell Rev. 2014;42(2):275\u201393.","journal-title":"Artif Intell Rev"},{"key":"1879_CR18","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L. Random forests. Mach Learn. 2001;45:5\u201332.","journal-title":"Mach Learn"},{"key":"1879_CR19","doi-asserted-by":"crossref","unstructured":"Freund Y, Schapire RE. Experiments with a new boosting algorithm. Machine learning: proceedings of the thirteenth international conference 1996:325\u2013332","DOI":"10.1145\/238061.238163"},{"key":"1879_CR20","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1093\/jamia\/ocu023","volume":"223","author":"EA Voss","year":"2015","unstructured":"Voss EA, Makadia R, Matcho A, et al. Feasibility and utility of applications of the common data model to multiple, disparate observational health databases. J Am Med Inform Assoc. 2015;223:553\u201364.","journal-title":"J Am Med Inform Assoc"},{"key":"1879_CR21","first-page":"10","volume":"231","author":"MA Suchard","year":"2013","unstructured":"Suchard MA, Simpson SE, Zorych I, et al. Massive parallelization of serial inference algorithms for complex generalized linear models. ACM Transact Model Comput Simul. 2013;231:10\u201332.","journal-title":"ACM Transact Model Comput Simul"},{"key":"1879_CR22","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1016\/j.jclinepi.2015.12.005","volume":"74","author":"B Van Calster","year":"2016","unstructured":"Van Calster B, Nieboer D, Vergouwe Y, et al. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167\u201376.","journal-title":"J Clin Epidemiol"},{"key":"1879_CR23","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1186\/s40537-021-00501-2","volume":"8","author":"JM Reps","year":"2021","unstructured":"Reps JM, Ryan PB, Rijnbeek PR, et al. Design matters in patient-level prediction: evaluation of a cohort vs. case\u2013control design when developing predictive models in observational healthcare datasets. J Big Data. 2021;8:108.","journal-title":"J Big Data"},{"key":"1879_CR24","first-page":"574","volume":"216","author":"G Hripcsak","year":"2015","unstructured":"Hripcsak G, Duke JD, Shah NH, et al. Observational Health Data Sciences and Informatics (OHDSI): opportunities for observational researchers. Stud Health Technol Inform. 2015;216:574.","journal-title":"Stud Health Technol Inform"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01879-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-022-01879-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01879-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T22:01:53Z","timestamp":1727301713000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-022-01879-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,25]]},"references-count":24,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["1879"],"URL":"https:\/\/doi.org\/10.1186\/s12911-022-01879-6","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,25]]},"assertion":[{"value":"30 December 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 May 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 May 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All patient data included in this study were deidentified. The New England Institutional Review Board determined that studies conducted in Optum\/IBM data are exempt from study-specific IRB review, as these studies do not qualify as human subjects research. No experiments were conducted on humans in this study. The research methods were conducted in accordance with appropriate guidelines.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"JMR, MJS, PBR are employees of Janssen Research and Development and shareholders of Johnson and Johnson. PRR and RDW work for a research group who received unconditional research grants from Boehringer-Ingelheim, GSK, Janssen Research & Development, Novartis, Pfizer, Yamanouchi, Servier. None of these grants result in a conflict of interest to the content of this paper.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"142"}}