{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T15:38:52Z","timestamp":1784821132796,"version":"3.55.0"},"reference-count":43,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T00:00:00Z","timestamp":1727136000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>This paper proposes a novel model-agnostic method for weighting the outputs of base classifiers in machine learning (ML) ensembles. Our approach uses class-based weight coefficients assigned to every output class in each learner in the ensemble. This is particularly useful when the base classifiers have highly variable performance across classes. Our method generates a dense set of coefficients for the models in our ensemble by considering the model performance on each class. We compare our novel method to the commonly used ensemble approaches like voting and weighted averages. In addition, we compare our approach to class-specific soft voting (CSSV), which was also designed to address variable performance but generates a sparse set of weights by solving a linear system. We choose to illustrate the power of this approach by applying it to an ensemble of extreme learning machines (ELMs), which are well suited for this approach due to their stochastic, highly varying performance across classes. We illustrate the superiority of our approach by comparing its performance to that of simple majority voting, weighted majority voting, and class-specific soft voting using ten popular open-source multiclass classification datasets.<\/jats:p>","DOI":"10.3390\/make6040106","type":"journal-article","created":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T09:16:10Z","timestamp":1727428570000},"page":"2149-2160","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Ensemble Learning with Highly Variable Class-Based Performance"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-2346-272X","authenticated-orcid":false,"given":"Brandon","family":"Warner","sequence":"first","affiliation":[{"name":"Verseon International Corporation, Fremont, CA 94538, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5456-8916","authenticated-orcid":false,"given":"Edward","family":"Ratner","sequence":"additional","affiliation":[{"name":"Verseon International Corporation, Fremont, CA 94538, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9935-3476","authenticated-orcid":false,"given":"Kallin","family":"Carlous-Khan","sequence":"additional","affiliation":[{"name":"Verseon International Corporation, Fremont, CA 94538, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Douglas","sequence":"additional","affiliation":[{"name":"Verseon International Corporation, Fremont, CA 94538, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amaury","family":"Lendasse","sequence":"additional","affiliation":[{"name":"Department of Engineering Management and Systems Engineering, Missouri University of Science and Technology, Rolla, MO 65409, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,9,24]]},"reference":[{"key":"ref_1","unstructured":"Freund, Y., and Schapire, R.E. (1996, January 3\u20136). Experiments with a new boosting algorithm. Proceedings of the 13th International Conference on Machine Learning ICML, Bari, Italy."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_4","unstructured":"Ho, T.K. (1995, January 14\u201316). Random decision forests. Proceedings of the 3rd Internal Conference on Document Analysis and Recognition, Montreal, QC, Canada."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1016\/S0167-9473(01)00065-2","article-title":"Stochastic gradient boosting","volume":"38","author":"Friedman","year":"2002","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"597","DOI":"10.1142\/S0219622006002258","article-title":"10 challenging problems in data mining research","volume":"5","author":"Yang","year":"2006","journal-title":"Int. J. Inf. Technol. Decis. Mak."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1080\/095400996116802","article-title":"Actively searching for an effective neural network ensemble","volume":"8","author":"Opitz","year":"1996","journal-title":"Connect. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.ins.2013.02.030","article-title":"A time series forest for classification and feature extraction","volume":"239","author":"Deng","year":"2013","journal-title":"Inf. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"584","DOI":"10.1016\/j.ijar.2011.12.011","article-title":"The impact of diversity on the accuracy of evidential classifier ensembles","volume":"53","author":"Bi","year":"2012","journal-title":"Int. J. Approx. Reason."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"5626","DOI":"10.1109\/TNNLS.2021.3071122","article-title":"Class-imbalanced deep learning via a class-balanced ensemble","volume":"33","author":"Chen","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_11","first-page":"144","article-title":"A novel classifier method based on class weighting in huge dataset","volume":"Volume 8","author":"Parvin","year":"2011","journal-title":"8th Inernational Symposium on Neural Networks, ISNN 2011, Guilin, China, 29 May\u20131 June 2011"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1016\/j.inffus.2004.04.008","article-title":"Classifier selection for majority voting","volume":"6","author":"Ruta","year":"2005","journal-title":"Inf. Fusion"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Rojarath, A., Songpan, W., and Pong-inwong, C. (2016, January 26\u201328). Improved ensemble learning for classification techniques based on majority voting. Proceedings of the 2016 7th IEEE International Conference on Software Engineering and Service Science (ICSESS), Beijing, China.","DOI":"10.1109\/ICSESS.2016.7883026"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1007\/s10115-012-0586-6","article-title":"A weighted voting framework for classifiers ensembles","volume":"38","author":"Kuncheva","year":"2014","journal-title":"Knowl. Inf. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Dogan, A., and Birant, D. (2019, January 11\u201315). A weighted majority voting ensemble approach for classification. Proceedings of the 4th International Conference on Computer Science and Engineering (UBMK), Samsun, T\u00fcrkiye.","DOI":"10.1109\/UBMK.2019.8907028"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1016\/j.neucom.2014.02.072","article-title":"Class-specific soft voting based multiple extreme learning machines ensemble","volume":"149","author":"Cao","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_17","unstructured":"Warner, B., Ratner, E., and Lendasse, A. (2021). Edammo\u2019s extreme AutoML technology\u2014Benchmarks and analysis. International Conference on Extreme Learning Machine, Springer."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Khan, K., Ratner, E., Ludwig, R., and Lendasse, A. (2020, January 19\u201324). Feature bagging and extreme learning machines: Machine learning with severe memory constraints. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9207673"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1016\/j.neucom.2005.12.126","article-title":"Extreme learning machine: Theory and applications","volume":"70","author":"Huang","year":"2006","journal-title":"Neurocomputing"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1109\/TSMCB.2011.2168604","article-title":"Extreme learning machine for regression and multiclass classification","volume":"42","author":"Huang","year":"2009","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3391","DOI":"10.1016\/j.neucom.2009.02.013","article-title":"Ensemble of online sequential extreme learning machine","volume":"72","author":"Lan","year":"2009","journal-title":"Neurocomputing"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"754","DOI":"10.1109\/LSP.2010.2053356","article-title":"Ensemble based extreme learning machine","volume":"17","author":"Liu","year":"2010","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_23","unstructured":"Wang, H., He, Q., Shang, T., Zhuang, F., and Shi, Z. (2014, January 8\u201310). Extreme learning machine ensemble classifier for large-scale data. Proceedings of the ELM 2014, Singapore."},{"key":"ref_24","first-page":"106","article-title":"Short-term load forecasting method based on ensemble improved extreme learning machine","volume":"43","author":"Cheng","year":"2009","journal-title":"J. Xian Jiaotong Univ."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1016\/j.neucom.2015.04.105","article-title":"Parallel ensemble of online sequential extreme learning machine based on MapReduce","volume":"174","author":"Huang","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1145\/1327452.1327492","article-title":"MapReduce: Simplified data processing on large clusters","volume":"51","author":"Dean","year":"2008","journal-title":"Commun. ACM"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wang, G., and Li, P. (2010, January 20\u201322). Dynamic Adaboost ensemble extreme learning machine. Proceedings of the 2010 3rd International Conference on Advanced Computer Theory and Engineering (ICACTE), Chengdu, China.","DOI":"10.1109\/ICACTE.2010.5579726"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4526","DOI":"10.1016\/j.jfranklin.2016.08.024","article-title":"Ensemble extreme learning machine and sparse representation classification","volume":"353","author":"Cao","year":"2016","journal-title":"J. Frankl. Inst."},{"key":"ref_29","first-page":"19","article-title":"Sparse representation for signal classification","volume":"Volume 19","author":"Huang","year":"2006","journal-title":"Advances in Neural Information Processing Systems 19, Proceedings of the 2006 Conference, Vancouver, BC, Canada, 4\u20137 December 2006"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1016\/j.ins.2011.09.015","article-title":"Voting based extreme learning machine","volume":"185","author":"Cao","year":"2012","journal-title":"Inf. Sci."},{"key":"ref_31","first-page":"245","article-title":"An analysis of ensemble pruning techniques based on ordered aggregation","volume":"31","author":"Suarez","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1016\/j.patcog.2010.07.021","article-title":"Sparse ensembles using weighted combination methods based on linear programming","volume":"44","author":"Zhang","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_33","unstructured":"Grove, A.J., and Schuurmans, D. (1998, January 26\u201330). Boosting is the limit: Maximizing the margin of learned ensembles. Proceedings of the AAI\/IAAI 98, Madison, WI, USA."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chen, H., Tino, P., and Yao, X. (2006, January 18\u201322). A probabilistic ensemble pruning algorithm. Proceedings of the Sixth IEEE Internal Conference on Data Mining-Workshops (ICDMW\u201906), Hong Kong, China.","DOI":"10.1109\/ICDMW.2006.18"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Fletcher, S., and Islam, M.Z. (2018). Comparing sets of patters with the Jaccard Index. Australas. J. Inf. Syst., 22.","DOI":"10.3127\/ajis.v22i0.1538"},{"key":"ref_36","unstructured":"Siegler, R. (1994). Balance Scale. UCI Machine Learning Repository, UC Irvine."},{"key":"ref_37","unstructured":"Lim, T.S. (1997). Contraceptive Method Choice. UCI Machine Learning Repository, UC Irvine."},{"key":"ref_38","unstructured":"Alcock, R. (1999). Synthetic Control Chart Time Series. UCI Machine Learning Repository, UC Irvine."},{"key":"ref_39","unstructured":"Bohanec, M. (1997). Car evaluation. UCI Machine Learning Repository, UC Irvine."},{"key":"ref_40","unstructured":"Reyes-Ortiz, J., Anguita, D., Ghio, A., Oneto, L., and Parra, X. (2012). Human Actvity Recognition Using Smartphones. UCI Machine Learning Repository, UC Irvine."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Realinho, V., Martins, M., Machado, J., and Baptista, L. (2021). Predict Students\u2019 Dropout and Academic Success. UCI Machine Learning Repository, UC Irvine.","DOI":"10.3390\/data7110146"},{"key":"ref_42","unstructured":"Ciarelli, P., and Oliveira, E. (2012). CNAE-9. UCI Machine Learning Repository, UC Irvine."},{"key":"ref_43","unstructured":"Fisher, R.A. (1988). Iris. UCI Machine Learning Repository, UC Irvine."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/4\/106\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:01:38Z","timestamp":1760112098000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/4\/106"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,24]]},"references-count":43,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["make6040106"],"URL":"https:\/\/doi.org\/10.3390\/make6040106","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,24]]}}}