{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T12:36:58Z","timestamp":1783600618630,"version":"3.55.0"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2025,3,21]],"date-time":"2025-03-21T00:00:00Z","timestamp":1742515200000},"content-version":"vor","delay-in-days":20,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Medical Research Future Fund","award":["2008996"],"award-info":[{"award-number":["2008996"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,3,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The complementary information found in different modalities of patient data can aid in more accurate modelling of a patient\u2019s disease state and a better understanding of the underlying biological processes of a disease. However, the analysis of multi-modal, multi-omics data presents many challenges. In this work, we compare the performance of a variety of ensemble machine learning (ML) algorithms that are capable of late integration of multi-class data from different modalities. The ensemble methods and their variations tested were (i) a voting ensemble, with hard and soft vote, (ii) a meta learner, and (iii) a multi-modal AdaBoost model using hard vote, soft vote, and meta learner to integrate the modalities on each boosting round, the PB-MVBoost model and a novel application of a mixture of expert\u2019s model. These were compared to simple concatenation. We examine these methods using data from an in-house study on hepatocellular carcinoma, plus validation datasets on studies from breast cancer and irritable bowel disease. We develop models that achieve an area under the receiver operating curve of up to 0.85 and find that two boosted methods, PB-MVBoost and AdaBoost with soft vote were the best performing models. We also examine the stability of features selected and the size of the clinical signature. Our work shows that integrating complementary omics and data modalities with effective ensemble ML models enhances accuracy in multi-class clinical outcome predictions and produces more stable predictive features than individual modalities or simple concatenation. We provide recommendations for the integration of multi-modal multi-class data.<\/jats:p>","DOI":"10.1093\/bib\/bbaf116","type":"journal-article","created":{"date-parts":[[2025,3,23]],"date-time":"2025-03-23T08:50:36Z","timestamp":1742719836000},"source":"Crossref","is-referenced-by-count":21,"title":["Benchmarking ensemble machine learning algorithms for multi-class, multi-omics data integration in clinical outcome prediction"],"prefix":"10.1093","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5705-0602","authenticated-orcid":false,"given":"Annette","family":"Spooner","sequence":"first","affiliation":[{"name":"School of Computer Science and Engineering, University of New South Wales , High St, Kensington, NSW 2052 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad Karimi","family":"Moridani","sequence":"additional","affiliation":[{"name":"School of Biotechnology and Biomolecular Sciences, University of New South Wales , NSW 2052 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Barbra","family":"Toplis","sequence":"additional","affiliation":[{"name":"St George and Sutherland Clinical Campuses, University of New South Wales , Short St, Kogarah, NSW 2217 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Behary","sequence":"additional","affiliation":[{"name":"St George and Sutherland Clinical Campuses, University of New South Wales , Short St, Kogarah, NSW 2217 ,","place":["Australia"]},{"name":"Department of Gastroenterology and Hepatology, St George Hospital , Gray St, Kogarah, NSW 2217 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Azadeh","family":"Safarchi","sequence":"additional","affiliation":[{"name":"Health and Biosecurity, Microbiome for One System Health, Commonwealth Scientific and Industrial Research Organisation , 160 Hawkesbury Rd, Westmead, NSW 2145 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salim","family":"Maher","sequence":"additional","affiliation":[{"name":"St George and Sutherland Clinical Campuses, University of New South Wales , Short St, Kogarah, NSW 2217 ,","place":["Australia"]},{"name":"Department of Gastroenterology and Hepatology, St George Hospital , Gray St, Kogarah, NSW 2217 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7521-2417","authenticated-orcid":false,"given":"Fatemeh","family":"Vafaee","sequence":"additional","affiliation":[{"name":"School of Biotechnology and Biomolecular Sciences, University of New South Wales , NSW 2052 ,","place":["Australia"]},{"name":"UNSW Data Science Hub, University of New South Wales , High St, Kensington, NSW 2052 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amany","family":"Zekry","sequence":"additional","affiliation":[{"name":"St George and Sutherland Clinical Campuses, University of New South Wales , Short St, Kogarah, NSW 2217 ,","place":["Australia"]},{"name":"Department of Gastroenterology and Hepatology, St George Hospital , Gray St, Kogarah, NSW 2217 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arcot","family":"Sowmya","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, University of New South Wales , High St, Kensington, NSW 2052 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,3,21]]},"reference":[{"key":"2025032119021709800_ref1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/fgene.2020.610798","article-title":"State of the field in multi-omics research: from computational needs to data mining and sharing","volume":"11","author":"Krassowski","year":"2020","journal-title":"Front Genet"},{"key":"2025032119021709800_ref2","doi-asserted-by":"publisher","first-page":"e1011224","DOI":"10.1371\/journal.pcbi.1011224","article-title":"Ten quick tips for avoiding pitfalls in multi-omics data integration analyses","volume":"19","author":"Chicco","year":"2023","journal-title":"PLoS Comput Biol"},{"key":"2025032119021709800_ref3","doi-asserted-by":"publisher","first-page":"3735","DOI":"10.1016\/j.csbj.2021.06.030","article-title":"Integration strategies of multi-omics data for machine learning analysis","volume":"19","author":"Picard","year":"2021","journal-title":"Comput Struct Biotechnol J"},{"key":"2025032119021709800_ref4","doi-asserted-by":"publisher","first-page":"4908","DOI":"10.1093\/bioinformatics\/btac631","article-title":"Multi-omic integration by machine learning (MIMaL)","volume":"38","author":"Dickinson","year":"2022","journal-title":"Bioinformatics"},{"key":"2025032119021709800_ref5","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.tig.2022.08.005","article-title":"Insights from multi-omics integration in complex disease primary tissues","volume":"39","author":"Kreitmaier","year":"2023","journal-title":"Trends Genet"},{"key":"2025032119021709800_ref6","doi-asserted-by":"publisher","DOI":"10.1142\/s2811032322500047","article-title":"Multimodal learning for multi-omics: a survey","volume":"01","author":"Tabakhi","year":"2023","journal-title":"World Sci Annu Rev Artif Intell"},{"key":"2025032119021709800_ref7","doi-asserted-by":"publisher","first-page":"151456","DOI":"10.1016\/j.semperi.2021.151456","article-title":"Multi \u2018omic data integration: a review of concepts, considerations, and approaches","volume":"45","author":"Santiago-Rodriguez","year":"2021","journal-title":"Semin Perinatol"},{"key":"2025032119021709800_ref8","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-815480-9.00013-X","volume-title":"Multiview Learning in Biomedical Applications","author":"Serra","year":"2018"},{"key":"2025032119021709800_ref9","doi-asserted-by":"publisher","first-page":"6149","DOI":"10.1016\/j.csbj.2022.11.008","article-title":"Multi-modal intermediate integrative methods in neuropsychiatric disorders: a review","volume":"20","author":"Wang","year":"2022","journal-title":"Comput Struct Biotechnol J"},{"key":"2025032119021709800_ref10","doi-asserted-by":"publisher","first-page":"1761","DOI":"10.1016\/j.patcog.2011.01.017","article-title":"An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes","volume":"44","author":"Galar","year":"2011","journal-title":"Pattern Recognit"},{"key":"2025032119021709800_ref11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/fmolb.2022.962743","article-title":"Multi-omics analysis: paving the path toward achieving precision medicine in cancer treatment and immuno-oncology","volume":"9","author":"Raufaste-Cazavieille","year":"2022","journal-title":"Front Mol Biosci"},{"key":"2025032119021709800_ref12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.36001\/ijphm.2024.v15i2.3850","article-title":"A comprehensive review of machine learning techniques for multi-omics data integration: challenges and applications in precision oncology","volume":"23","author":"Acharya","year":"2024","journal-title":"Brief Funct Genomics"},{"key":"2025032119021709800_ref13","doi-asserted-by":"publisher","first-page":"623","DOI":"10.1038\/s41586-021-04278-5","article-title":"Multi-omic machine learning predictor of breast cancer therapy response","volume":"601","author":"Sammut","year":"2022","journal-title":"Nature"},{"key":"2025032119021709800_ref14","doi-asserted-by":"publisher","first-page":"2023","DOI":"10.1016\/j.compbiomed.2022.106373","article-title":"Differential diagnosis of systemic lupus erythematosus and Sj\u00f6gren\u2019s syndrome using machine learning and multi-omics data","volume":"152","author":"Martorell-Marug\u00e1n","year":"2022","journal-title":"Comput Biol Med"},{"key":"2025032119021709800_ref15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0263248","article-title":"Combining explainable machine learning, demographic and multi-omic data to inform precision medicine strategies for inflammatory bowel disease","volume":"17","author":"Gardiner","year":"2022","journal-title":"PLoS One"},{"key":"2025032119021709800_ref16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12864-021-07524-2","article-title":"Deep learning assisted multi-omics integration for survival and drug-response prediction in breast cancer","volume":"22","author":"Malik","year":"2021","journal-title":"BMC Genomics"},{"key":"2025032119021709800_ref17","doi-asserted-by":"publisher","first-page":"i501","DOI":"10.1093\/bioinformatics\/btz318","article-title":"MOLI: multi-omics late integration with deep neural networks for drug response prediction","volume":"35","author":"Sharifi-Noghabi","year":"2019","journal-title":"Bioinformatics"},{"key":"2025032119021709800_ref18","doi-asserted-by":"publisher","DOI":"10.3390\/jpm12040601","article-title":"Machine-learning-based late fusion on multi-omics and multi-scale data for non-small-cell lung cancer diagnosis","volume":"12","author":"Carrillo-Perez","year":"2022","journal-title":"J Pers Med"},{"key":"2025032119021709800_ref19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-022-00712-8","article-title":"Multimodal machine learning in precision health: a scoping review","volume":"5","author":"Kline","year":"2022","journal-title":"npj Digit Med"},{"key":"2025032119021709800_ref20","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1016\/j.inffus.2017.02.007","article-title":"Multi-view learning overview: recent progress and new challenges","volume":"38","author":"Zhao","year":"2017","journal-title":"Inf Fusion"},{"key":"2025032119021709800_ref21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-020-20422-7","article-title":"Gut microbiota impact on the peripheral immune response in non-alcoholic fatty liver disease related hepatocellular carcinoma","volume":"12","author":"Behary","year":"2021","journal-title":"Nat Commun"},{"key":"2025032119021709800_ref22","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1006\/jcss.1997.1504","article-title":"A decision-theoretic generalization of on-line learning and an application to boosting","volume":"55","author":"Freund","year":"1995","journal-title":"Comput Learn theory"},{"key":"2025032119021709800_ref23","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1016\/j.neucom.2019.04.072","article-title":"Multiview boosting by controlling the diversity and the accuracy of view-specific voters","volume":"358","author":"Goyal","year":"2019","journal-title":"Neurocomputing"},{"key":"2025032119021709800_ref24","doi-asserted-by":"publisher","first-page":"700","DOI":"10.1038\/s41591-023-02217-7","article-title":"Gut microbial metabolism of 5-ASA diminishes its clinical efficacy in inflammatory bowel disease","volume":"29","author":"Mehta","year":"2023","journal-title":"Nat Med"},{"key":"2025032119021709800_ref25","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1038\/s41564-018-0306-4","article-title":"Gut microbiome structure and metabolic activity in inflammatory bowel disease","volume":"4","author":"Franzosa","year":"2019","journal-title":"Nat Microbiol"},{"key":"2025032119021709800_ref26","doi-asserted-by":"publisher","first-page":"1436","DOI":"10.1016\/j.cell.2020.10.036","article-title":"Proteogenomic landscape of breast cancer tumorigenesis and targeted therapy","volume":"183","author":"Krug","year":"2020","journal-title":"Cell"},{"key":"2025032119021709800_ref27","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J Artif Intell Res"},{"key":"2025032119021709800_ref28","doi-asserted-by":"publisher","first-page":"13","DOI":"10.30812\/matrik.v22i1.1803","article-title":"The application of repeated SMOTE for multi class classification on imbalanced data","volume":"22","author":"Rachmatullah","year":"2022","journal-title":"Tek Inform dan Rekayasa Komput"},{"key":"2025032119021709800_ref29","first-page":"2007","article-title":"MICE: multivariate imputation by chained equations","volume":"1","author":"Buuren","year":"2007","journal-title":"R Packag version"},{"key":"2025032119021709800_ref30","doi-asserted-by":"publisher","first-page":"271","DOI":"10.3233\/FI-2010-288","article-title":"Boruta\u2014a system for feature selection","volume":"101","author":"Kursa","year":"2010","journal-title":"Fundam Informaticae"},{"key":"2025032119021709800_ref31","first-page":"1","article-title":"Meta-learning in healthcare: a survey","volume-title":"SN Computer Science","author":"Rafiei"},{"key":"2025032119021709800_ref32","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1002\/nbm.1810","article-title":"The strength of weak learnability (extended abstract)","volume":"5","author":"Schapire","year":"1990","journal-title":"Mach Learn"},{"key":"2025032119021709800_ref33","doi-asserted-by":"publisher","first-page":"349","DOI":"10.4310\/sii.2009.v2.n3.a8","article-title":"Multi-class AdaBoost","volume":"2","author":"Zhu","year":"2009","journal-title":"Stat Interface"},{"key":"2025032119021709800_ref34","doi-asserted-by":"publisher","first-page":"3236","DOI":"10.1007\/978-3-319-24612-3_301724","article-title":"An algorithm on multi-view AdaBoost","volume":"6443","author":"Xu","year":"2010","journal-title":"Neural Inf Process\u2014Theory Algorithms"},{"key":"2025032119021709800_ref35","first-page":"2851","article-title":"Multi-view AdaBoost for multilingual subjectivity analysis","author":"Xiao","year":"2012","journal-title":"24th International Conference on Computing Linguistics\u2014Proceedings of theCOLING 2012 Technical Papers"},{"key":"2025032119021709800_ref36","doi-asserted-by":"crossref","DOI":"10.1101\/2021.04.15.440095","article-title":"MVD-fuse: detection of white matter degeneration via multi-view learning of diffusion microstructure","author":"Fadnavis","year":"2021"},{"key":"2025032119021709800_ref37","doi-asserted-by":"publisher","first-page":"1177","DOI":"10.1109\/TNNLS.2012.2200299","article-title":"Twenty years of mixture of experts","volume":"23","author":"Yuksel","year":"2012","journal-title":"IEEE Trans Neural Networks Learn Syst"},{"key":"2025032119021709800_ref38","doi-asserted-by":"publisher","first-page":"100071","DOI":"10.1016\/j.crmeth.2021.100071","article-title":"A mixture-of-experts deep generative model for integrated analysis of single-cell multiomics data","volume":"1","author":"Minoura","year":"2021","journal-title":"Cell Reports Methods"},{"key":"2025032119021709800_ref39","doi-asserted-by":"publisher","DOI":"10.1145\/3136625","article-title":"Feature selection: a data perspective","volume":"50","author":"Li","year":"2018","journal-title":"ACM Comput Surv"},{"key":"2025032119021709800_ref40","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1007\/bf00993473","article-title":"Technical note: bias and the quantification of stability","volume":"20","author":"Turney","year":"1995","journal-title":"Mach Learn"},{"key":"2025032119021709800_ref41","doi-asserted-by":"publisher","first-page":"1921","DOI":"10.1109\/TPAMI.2010.34","article-title":"Evaluating stability and comparing output of feature selectors that optimize feature subset cardinality","volume":"32","author":"Somol","year":"2010","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2025032119021709800_ref42","doi-asserted-by":"publisher","first-page":"01","DOI":"10.5121\/ijdkp.2015.5201","article-title":"A review on evaluation metrics for data classification evaluations","volume":"5","author":"Hossin","year":"2015","journal-title":"Int J Data Min Knowl Manag Process"},{"key":"2025032119021709800_ref43","doi-asserted-by":"publisher","author":"Katana","year":"2010","DOI":"10.26190\/669X-A286"},{"key":"2025032119021709800_ref44","volume-title":"R: A language and environment for statistical computing","author":"R Core Team","year":"2019"},{"key":"2025032119021709800_ref45","first-page":"5938","article-title":"Mlr: machine learning in R","volume":"17","author":"Bischl","year":"2016","journal-title":"J Mach Learn Res"},{"key":"2025032119021709800_ref46","doi-asserted-by":"publisher","first-page":"273","DOI":"10.32614\/RJ-2021-048","article-title":"A unifying framework for parallel and distributed processing in R using futures","volume":"13","author":"Bengtsson","year":"2021","journal-title":"R J"},{"key":"2025032119021709800_ref47","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1023\/A:1024068626366","article-title":"Inference for the generalization error","volume":"52","author":"Nadeau","year":"2003","journal-title":"Mach Learn"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/2\/bbaf116\/62519410\/bbaf116.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/2\/bbaf116\/62519410\/bbaf116.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,23]],"date-time":"2025-03-23T08:51:12Z","timestamp":1742719872000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbaf116\/8089936"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3]]},"references-count":47,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,3,4]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbaf116","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,3]]},"published":{"date-parts":[[2025,3]]},"article-number":"bbaf116"}}