{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T04:36:15Z","timestamp":1774154175417,"version":"3.50.1"},"reference-count":28,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2020,11,13]],"date-time":"2020-11-13T00:00:00Z","timestamp":1605225600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Medical Research Council and Health Data Research UK","award":["MR\/S004149\/1"],"award-info":[{"award-number":["MR\/S004149\/1"]}]},{"name":"Industrial Strategy Challenge","award":["MC_PC_18029"],"award-info":[{"award-number":["MC_PC_18029"]}]},{"name":"Wellcome Institutional Translation Partnership Award","award":["PIII054"],"award-info":[{"award-number":["PIII054"]}]},{"DOI":"10.13039\/501100000265","name":"Medical Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Health Data Research UK","award":["MR\/S003991\/1"],"award-info":[{"award-number":["MR\/S003991\/1"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["81700006"],"award-info":[{"award-number":["81700006"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"UKRI Innovation Fellowship","award":["MR\/S00310X\/1"],"award-info":[{"award-number":["MR\/S00310X\/1"]}]},{"DOI":"10.13039\/501100000272","name":"National Institute for Health Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]},{"name":"LifeArc STOPCOVID award"},{"name":"NIHR Birmingham Experimental Cancer Medical Centre"},{"name":"NIHR Birmingham Surgical Reconstruction and Microbiology Research Centre"},{"name":"Nanocommons H2020-EU","award":["731032"],"award-info":[{"award-number":["731032"]}]},{"DOI":"10.13039\/501100018952","name":"NIHR Birmingham Biomedical Research Centre","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100018952","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Medical Research Council Health Data Research UK","award":["HDRUK\/CFC\/01"],"award-info":[{"award-number":["HDRUK\/CFC\/01"]}]},{"name":"NIHR Biomedical Research Centre"},{"DOI":"10.13039\/100009362","name":"South London and Maudsley NHS Foundation Trust","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009362","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009360","name":"King\u2019s College London","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009360","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Health Data Research UK"},{"name":"BigData@Heart Consortium"},{"name":"Innovative Medicines Initiative-2 Joint Undertaking","award":["116074"],"award-info":[{"award-number":["116074"]}]},{"name":"National Institute for Health Research University College London Hospitals Biomedical Research Centre"},{"name":"UK Research and Innovation London Medical Imaging and Artificial Intelligence Centre"},{"name":"Value Based Healthcare"},{"name":"NIHR Applied Research Collaboration South London"},{"DOI":"10.13039\/100010872","name":"King\u2019s College Hospital NHS Foundation Trust","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100010872","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,3,18]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>Risk prediction models are widely used to inform evidence-based clinical decision making. However, few models developed from single cohorts can perform consistently well at population level where diverse prognoses exist (such as the SARS-CoV-2 [severe acute respiratory syndrome coronavirus 2] pandemic). This study aims at tackling this challenge by synergizing prediction models from the literature using ensemble learning.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>In this study, we selected and reimplemented 7 prediction models for COVID-19 (coronavirus disease 2019) that were derived from diverse cohorts and used different implementation techniques. A novel ensemble learning framework was proposed to synergize them for realizing personalized predictions for individual patients. Four diverse international cohorts (2 from the United Kingdom and 2 from China; N\u2009=\u20095394) were used to validate all 8 models on discrimination, calibration, and clinical usefulness.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Results showed that individual prediction models could perform well on some cohorts while poorly on others. Conversely, the ensemble model achieved the best performances consistently on all metrics quantifying discrimination, calibration, and clinical usefulness. Performance disparities were observed in cohorts from the 2 countries: all models achieved better performances on the China cohorts.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>When individual models were learned from complementary cohorts, the synergized model had the potential to achieve better performances than any individual model. Results indicate that blood parameters and physiological measurements might have better predictive powers when collected early, which remains to be confirmed by further studies.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Combining a diverse set of individual prediction models, the ensemble method can synergize a robust and well-performing model by choosing the most competent ones for individual patients.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocaa295","type":"journal-article","created":{"date-parts":[[2020,11,12]],"date-time":"2020-11-12T12:10:12Z","timestamp":1605183012000},"page":"791-800","source":"Crossref","is-referenced-by-count":7,"title":["Ensemble learning for poor prognosis predictions: A case study on SARS-CoV-2"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0213-5668","authenticated-orcid":false,"given":"Honghan","family":"Wu","sequence":"first","affiliation":[{"name":"Institute of Health Informatics, University College London, London, United Kingdom"},{"name":"Health Data Research UK, University College London, London, United Kingdom"}]},{"given":"Huayu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Centre for Medical Informatics, Usher Institute, University of Edinburgh, Edinburgh, United Kingdom"}]},{"given":"Andreas","family":"Karwath","sequence":"additional","affiliation":[{"name":"Institute of Cancer and Genomic Sciences, University of Birmingham, Birmingham, United Kingdom"},{"name":"Health Data Research UK, University of Birmingham, Birmingham, United Kingdom"}]},{"given":"Zina","family":"Ibrahim","sequence":"additional","affiliation":[{"name":"Health Data Research UK, University College London, London, United Kingdom"},{"name":"Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King\u2019s College London, London, United Kingdom"}]},{"given":"Ting","family":"Shi","sequence":"additional","affiliation":[{"name":"Centre for Global Health, Usher Institute, University of Edinburgh, Edinburgh, United Kingdom"}]},{"given":"Xin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Pulmonary and Critical Care Medicine, People\u2019s Liberation Army Joint Logistic Support Force 920th Hospital, Kunming, China"}]},{"given":"Kun","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Pulmonary and Critical Care Medicine, Shanghai East Hospital, Tongji University, Shanghai, China"}]},{"given":"Jiaxing","family":"Sun","sequence":"additional","affiliation":[{"name":"Department of Pulmonary and Critical Care Medicine, Shanghai East Hospital, Tongji University, Shanghai, China"}]},{"given":"Kevin","family":"Dhaliwal","sequence":"additional","affiliation":[{"name":"Centre for Inflammation Research, Queens Medical Research Institute, University of Edinburgh, Edinburgh, United Kingdom"}]},{"given":"Daniel","family":"Bean","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King\u2019s College London, London, United Kingdom"}]},{"given":"Victor Roth","family":"Cardoso","sequence":"additional","affiliation":[{"name":"Institute of Cancer and Genomic Sciences, University of Birmingham, Birmingham, United Kingdom"},{"name":"Health Data Research UK, University of Birmingham, Birmingham, United Kingdom"}]},{"given":"Kezhi","family":"Li","sequence":"additional","affiliation":[{"name":"Institute of Health Informatics, University College London, London, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6899-8319","authenticated-orcid":false,"given":"James T","family":"Teo","sequence":"additional","affiliation":[{"name":"Department of Stroke and Neurology, King\u2019s College Hospital NHS Foundation Trust, London, United Kingdom"}]},{"given":"Amitava","family":"Banerjee","sequence":"additional","affiliation":[{"name":"Institute of Health Informatics, University College London, London, United Kingdom"}]},{"given":"Fang","family":"Gao-Smith","sequence":"additional","affiliation":[{"name":"Department of Intensive Care Medicine, Queen Elizabeth Hospital Birmingham, Birmingham, United Kingdom"},{"name":"Birmingham Acute Care Research, University of Birmingham, Birmingham, United Kingdom"}]},{"given":"Tony","family":"Whitehouse","sequence":"additional","affiliation":[{"name":"Department of Intensive Care Medicine, Queen Elizabeth Hospital Birmingham, Birmingham, United Kingdom"},{"name":"Birmingham Acute Care Research, University of Birmingham, Birmingham, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4125-8804","authenticated-orcid":false,"given":"Tonny","family":"Veenith","sequence":"additional","affiliation":[{"name":"Department of Intensive Care Medicine, Queen Elizabeth Hospital Birmingham, Birmingham, United Kingdom"},{"name":"Birmingham Acute Care Research, University of Birmingham, Birmingham, United Kingdom"}]},{"given":"Georgios V","family":"Gkoutos","sequence":"additional","affiliation":[{"name":"Institute of Cancer and Genomic Sciences, University of Birmingham, Birmingham, United Kingdom"},{"name":"Health Data Research UK, University of Birmingham, Birmingham, United Kingdom"},{"name":"Institute of Translational Medicine, University Hospitals Birmingham NHS Foundation Trust, Birmingham, United Kingdom"}]},{"given":"Xiaodong","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Pulmonary and Critical Care Medicine, Shanghai East Hospital, Tongji University, Shanghai, China"},{"name":"Department of Pulmonary and Critical Care Medicine, Taikang Tongji Hospital, Wuhan, China"}]},{"given":"Richard","family":"Dobson","sequence":"additional","affiliation":[{"name":"Institute of Health Informatics, University College London, London, United Kingdom"},{"name":"Health Data Research UK, University College London, London, United Kingdom"},{"name":"Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King\u2019s College London, London, United Kingdom"}]},{"given":"Bruce","family":"Guthrie","sequence":"additional","affiliation":[{"name":"Centre for Population Health Sciences, Usher Institute, University of Edinburgh, Edinburgh, United Kingdom"}]}],"member":"286","published-online":{"date-parts":[[2020,11,13]]},"reference":[{"issue":"15","key":"2021031906045059900_ocaa295-B1","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1001\/jama.2011.1515","article-title":"Risk prediction models for hospital readmission: a systematic review","volume":"306","author":"Kansagara","year":"2011","journal-title":"JAMA"},{"key":"2021031906045059900_ocaa295-B2","doi-asserted-by":"crossref","first-page":"e3318","DOI":"10.1136\/bmj.e3318","article-title":"Comparisons of established risk prediction models for cardiovascular disease: systematic review","volume":"344","author":"Siontis","year":"2012","journal-title":"BMJ"},{"issue":"1","key":"2021031906045059900_ocaa295-B3","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/jamia\/ocw042","article-title":"Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review","volume":"24","author":"Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2021031906045059900_ocaa295-B4","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1097\/CCM.0000000000001571","article-title":"Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards","volume":"44","author":"Churpek","year":"2016","journal-title":"Crit Care Med"},{"issue":"4","key":"2021031906045059900_ocaa295-B5","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1080\/23744235.2016.1253860","article-title":"Failure of CRP decline within three days of hospitalization is associated with poor prognosis of community-acquired pneumonia","volume":"49","author":"Andersen","year":"2017","journal-title":"Infect Dis"},{"issue":"5","key":"2021031906045059900_ocaa295-B6","doi-asserted-by":"crossref","first-page":"510","DOI":"10.1001\/jama.2020.11787","article-title":"Excess deaths from COVID-19 and other causes, March-April 2020","volume":"324","author":"Woolf","year":"2020","journal-title":"JAMA"},{"issue":"10238","key":"2021031906045059900_ocaa295-B7","doi-asserted-by":"crossref","first-page":"1715","DOI":"10.1016\/S0140-6736(20)30854-0","article-title":"Estimating excess 1-year mortality associated with the COVID-19 pandemic according to underlying conditions and age: a population-based cohort study","volume":"395","author":"Banerjee","year":"2020","journal-title":"Lancet"},{"key":"2021031906045059900_ocaa295-B8","author":"Lai","year":"2020"},{"key":"2021031906045059900_ocaa295-B9","author":"Banerjee","year":"2020"},{"issue":"6","key":"2021031906045059900_ocaa295-B10","doi-asserted-by":"crossref","first-page":"1393","DOI":"10.1093\/cid\/ciaa414","article-title":"Prediction for progression risk in patients with COVID-19 pneumonia: the CALL score","volume":"71","author":"Ji","year":"2020","journal-title":"Clin Infect Dis"},{"key":"2021031906045059900_ocaa295-B11","author":"Carr","year":"2020"},{"key":"2021031906045059900_ocaa295-B12","article-title":"Risk prediction for poor outcome and death in hospital in-patients with COVID-19: Derivation in Wuhan, China and external validation in London, UK","author":"Zhang","journal-title":"medRxiv"},{"key":"2021031906045059900_ocaa295-B13","doi-asserted-by":"crossref","first-page":"m1328","DOI":"10.1136\/bmj.m1328","article-title":"Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal","volume":"369","author":"Wynants","year":"2020","journal-title":"BMJ"},{"issue":"6","key":"2021031906045059900_ocaa295-B14","doi-asserted-by":"crossref","first-page":"2003498","DOI":"10.1183\/13993003.03498-2020","article-title":"Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: An observational cohort study","volume":"56","author":"Gupta","year":"2020","journal-title":"Eur Respir J"},{"key":"2021031906045059900_ocaa295-B15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.inffus.2018.11.008","article-title":"Ensembles for feature selection: a review and future trends","volume":"52","author":"Bol\u00f3n-Canedo","year":"2019","journal-title":"Inf Fusion"},{"issue":"10232","key":"2021031906045059900_ocaa295-B16","doi-asserted-by":"crossref","first-page":"1305","DOI":"10.1016\/S0140-6736(20)30744-3","article-title":"Fangcang shelter hospitals: a novel concept for responding to public health emergencies","volume":"395","author":"Chen","year":"2020","journal-title":"Lancet"},{"issue":"1","key":"2021031906045059900_ocaa295-B17","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1186\/s13054-020-2833-7","article-title":"Host susceptibility to severe COVID-19 and establishment of a host risk score: findings of 487 cases outside Wuhan","volume":"24","author":"Shi","year":"2020","journal-title":"Crit Care"},{"key":"2021031906045059900_ocaa295-B18","article-title":"A tool to early predict severe corona virus disease 2019 (COVID-19): a multicenter study using the risk nomogram in Wuhan and Guangdong, China","author":"Gong","year":"]","journal-title":"Clin Infect Dis 2020 [E-pub ahead of print]"},{"key":"2021031906045059900_ocaa295-B19","article-title":"ACP risk grade: a simple mortality index for patients with confirmed or suspected severe acute respiratory syndrome coronavirus 2 disease (COVID-19) during the early stage of outbreak in Wuhan, China","author":"Lu","year":"2020","journal-title":"medRxiv"},{"key":"2021031906045059900_ocaa295-B20","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1038\/s42256-020-0180-7","article-title":"An interpretable mortality prediction model for COVID-19 patients","volume":"2","author":"Yan","year":"2020","journal-title":"Nat Mach Intell"},{"key":"2021031906045059900_ocaa295-B21","author":"Xie","year":"2020"},{"key":"2021031906045059900_ocaa295-B22","article-title":"Development and validation of a survival calculator for hospitalized patients with COVID-19","author":"Levy","year":"2020","journal-title":"medRxiv"},{"key":"2021031906045059900_ocaa295-B23","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4419-9326-7","volume-title":"Ensemble Machine Learning: Methods and Applications","author":"Zhang","year":"2012"},{"issue":"2","key":"2021031906045059900_ocaa295-B24","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/S0893-6080(05)80023-1","article-title":"Stacked generalization","volume":"5","author":"Wolpert","year":"1992","journal-title":"Neural Netw"},{"issue":"2","key":"2021031906045059900_ocaa295-B25","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach Learn"},{"key":"2021031906045059900_ocaa295-B26","doi-asserted-by":"crossref","DOI":"10.1002\/9781118914564","volume-title":"Combining Pattern Classifiers: Methods and Algorithms","author":"Kuncheva","year":"2014"},{"key":"2021031906045059900_ocaa295-B27","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1007\/978-3-642-41136-6_5","volume-title":"Empirical Inference","author":"Schapire","year":"2013"},{"issue":"2","key":"2021031906045059900_ocaa295-B28","doi-asserted-by":"crossref","first-page":"122","DOI":"10.7326\/M13-1522","article-title":"Net reclassification improvement: computation, interpretation, and controversies","volume":"160","author":"Leening","year":"2014","journal-title":"Ann Intern Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocaa295\/34298119\/ocaa295.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/4\/791\/36642157\/ocaa295.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/4\/791\/36642157\/ocaa295.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,12]],"date-time":"2023-10-12T05:03:40Z","timestamp":1697087020000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/4\/791\/5981140"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,13]]},"references-count":28,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,11,13]]},"published-print":{"date-parts":[[2021,3,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa295","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,4,1]]},"published":{"date-parts":[[2020,11,13]]}}}