{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T14:45:36Z","timestamp":1781621136533,"version":"3.54.5"},"reference-count":32,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2019,9,5]],"date-time":"2019-09-05T00:00:00Z","timestamp":1567641600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The reliability of complex or safety critical systems is of increasing importance in several application fields. In many cases, decisions evaluating situations or conditions are made. To ensure the high accuracy of these decisions, the assignments from different classifiers can be fused to one final decision to improve the decision performance in terms of given measures like accuracy or false alarm rate. Recent research results show that fusion methods not always outperform individual classifiers trained and optimized for a specific situation. Nevertheless fusion helps to ensure reliability and redundancy by combining the advantages of individual classifiers, even if some classifiers are not performing well for specific situations. Especially in unexpected (untrained) situations, fusion of more than one classifier allows to get a suitable decision, because of different behavior of classifiers in this case. Nevertheless, there are several examples, where fusion not always improves the overall accuracy of a decision. In this contribution fusion options are discussed to overcome the problem to overcome the aforementioned problem and to define influencing factors on overall fusion accuracy. As a results requirements for good or guaranteed or possibly increased fusion performance and also suggestions denoting those options not leading to any kind of improvement are given. For illustrating the effects a practical example based on three characteristics of fusion methods (type of classifier output, use of these outputs and necessity of training) and four data properties (number of classes, number of samples, entropy of classes and entropy of attributes) are considered and analyzed with 15 different benchmark data sets, which are classified with eight classification methods. The classification results are fused using seven fusion methods. From the discussion of the results it can be concluded, which fusion method performs best\/worst for all data sets as well as which fusion method characteristic or data property has more or less positive\/negative influence on the fusion performance in comparison to the best base classifier.Using this information, suitable fusion methods can be selected or data sets can be adapted to improve the reliability of decisions made in complex or safety critical systems.<\/jats:p>","DOI":"10.3390\/e21090866","type":"journal-article","created":{"date-parts":[[2019,9,6]],"date-time":"2019-09-06T02:59:22Z","timestamp":1567738762000},"page":"866","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Does Classifier Fusion Improve the Overall Performance? Numerical Analysis of Data and Fusion Method Characteristics Influencing Classifier Fusion Performance"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5726-8185","authenticated-orcid":false,"given":"Sandra","family":"Rothe","sequence":"first","affiliation":[{"name":"Chair of Dynamics and Control, Faculty of Engineering, University of Duisburg-Essen, 47057 Duisburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3572-2524","authenticated-orcid":false,"given":"Bastian","family":"Kudszus","sequence":"additional","affiliation":[{"name":"Chair of Dynamics and Control, Faculty of Engineering, University of Duisburg-Essen, 47057 Duisburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8299-101X","authenticated-orcid":false,"given":"Dirk","family":"S\u00f6ffker","sequence":"additional","affiliation":[{"name":"Chair of Dynamics and Control, Faculty of Engineering, University of Duisburg-Essen, 47057 Duisburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2019,9,5]]},"reference":[{"key":"ref_1","first-page":"1","article-title":"An overview of classifier fusion methods","volume":"7","author":"Ruta","year":"2000","journal-title":"Comput. Inf. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1109\/34.273716","article-title":"Decision combination in multiple classifier systems","volume":"16","author":"Ho","year":"1994","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1016\/j.inffus.2018.12.001","article-title":"A survey on data fusion in internet of things: Towards secure and privacy-preserving fusion","volume":"51","author":"Ding","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Azzedin, F., and Ghaleb, M. (2019). Internet-of-Things and Information Fusion: Trust Perspective Survey. Sensors, 19.","DOI":"10.3390\/s19081929"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.asoc.2004.12.002","article-title":"On learning algorithm selection for classification","volume":"6","author":"Ali","year":"2006","journal-title":"Appl. Soft Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"3665","DOI":"10.1016\/j.patcog.2014.05.003","article-title":"Dynamic selection of classifiers\u2014A comprehensive review","volume":"47","author":"Britto","year":"2014","journal-title":"Pattern Recognit."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1016\/0004-3702(87)90040-3","article-title":"Implementing dempster\u2019s rule for hierarchical evidence","volume":"33","author":"Shafer","year":"1987","journal-title":"Artif. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Huang, Y.S., and Suen, C.Y. (1993, January 15\u201317). The behavior-knowledge space method for combination of multiple classifiers. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New York, NY, USA.","DOI":"10.1109\/CVPR.1993.1626170"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Suen, C.Y., and Lam, L. (2000, January 21\u201323). Multiple classifier combination methodologies for different output levels. Proceedings of the Multiple Classifier Systems. First International Workshop MCS 2000, Cagliari, Italy.","DOI":"10.1007\/3-540-45014-9_5"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1016\/S0031-3203(99)00223-X","article-title":"Decision templates for multiple classifier fusion: An experimental comparison","volume":"34","author":"Kuncheva","year":"2001","journal-title":"Pattern Recognit."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kuncheva, L.I. (2004). Combining Pattern Classifiers: Methods and Algorithms, John Wiley & Sons.","DOI":"10.1002\/0471660264"},{"key":"ref_12","unstructured":"Michie, D., Spiegelhalter, D.J., and Taylor, C.C. (1994). Machine Learning, Neural and Statistical Classification, Ellis Horwood Ltd."},{"key":"ref_13","unstructured":"Bezdek, J.C., Keller, J., Krisnapuram, R., and Pal, N. (2006). Fuzzy Models and Algorithms for Pattern Recognition and Image Processing, Springer Science & Business Media."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"906","DOI":"10.1109\/TFUZZ.2010.2052258","article-title":"Comparing fuzzy, probabilistic, and possibilistic partitions","volume":"18","author":"Anderson","year":"2010","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_15","first-page":"611","article-title":"Classifier conditional posterior probabilities","volume":"Volume 1451","author":"Amin","year":"1998","journal-title":"Advances in Pattern Recognition, Proceedings of Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR)"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1109\/21.155943","article-title":"Methods of combining multiple classifiers and their applications to handwriting recognition","volume":"22","author":"Xu","year":"1992","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_17","unstructured":"Duin, R.P.W. (2002, January 11\u201315). The combining classifier: To train or not to train?. Proceedings of the Object Recognition Supported by User Interaction for Service Robots, Quebec City, QC, Canada."},{"key":"ref_18","unstructured":"Suen, C.Y. (1990, January 2\u20133). Recognition of totally unconstrained handwritten numerals based on the concept of multiple experts. Proceedings of the 1st International Workshop on Frontiers in Handwriting Recognition, Montreal, QC, Canada."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"87","DOI":"10.23919\/SAIEE.2007.9488132","article-title":"Data characteristics that determine classifier performance","volume":"98","author":"Walt","year":"2006","journal-title":"SAIEE Afr. Res. J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1177\/1475921714566530","article-title":"The probability of detection for structural health monitoring systems: Repeated measures data","volume":"14","author":"Greenwell","year":"2015","journal-title":"Struct. Health Monit."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1109\/21.364825","article-title":"Combining multiple neural networks by fuzzy integral and robust classification","volume":"25","author":"Cho","year":"1995","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/1656274.1656278","article-title":"The weka data mining software","volume":"11","author":"Hall","year":"2009","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_23","unstructured":"Zheng, A. (2015). Evaluating Machine Learning Models, O\u2019Reilly Media, Inc."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1007\/s10618-016-0483-9","article-title":"The great time series classification bake off: A review and experimental evaluation of recent algorithmic advances","volume":"31","author":"Bagnall","year":"2017","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yeung, D.Y., Kwok, J.T., Fred, A., Roli, F., and de Ridder, D. (2006). Comparison of classifier fusion methods for classification in pattern recognition tasks. Structural, Syntactic, and Statistical Pattern Recognition, Springer. SSPR \/SPR 2006. Lecture Notes in Computer Science.","DOI":"10.1007\/11815921"},{"key":"ref_26","unstructured":"Bouckaert, R.R., Frank, E., Hall, M., Kirkby, R., Reutemann, P., Seewald, A., and Scuse, D. (2013). Waikato Environment for Knowledge Analysis (WEKA) Manual for Version 3-7-8 (accessed: 2018-05-20), The University of Waikato."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Bouckaert, R.R., and Frank, E. (2004). Evaluating the replicability of significance tests for comparing learning algorithms. Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Springer.","DOI":"10.1007\/978-3-540-24775-3_3"},{"key":"ref_28","unstructured":"Kohavi, R. (1995, January 20\u201325). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence IJCAI\u201995, Montreal, QC, Canada."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1109\/TPAMI.2009.187","article-title":"Sensitivity analysis of k-fold cross validation in prediction error estimation","volume":"32","author":"Rodriguez","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-7-91","article-title":"Bias in error estimation when using cross-validation for model selection","volume":"7","author":"Varma","year":"2006","journal-title":"BMC Bioinform."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/1758-2946-6-10","article-title":"Cross-validation pitfalls when selecting and assessing regression and classification models","volume":"6","author":"Krstajic","year":"2014","journal-title":"J. Cheminform."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2839","DOI":"10.1016\/j.patcog.2015.03.009","article-title":"Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation","volume":"48","author":"Wong","year":"2015","journal-title":"Pattern Recognit."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/9\/866\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:17:12Z","timestamp":1760188632000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/9\/866"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,5]]},"references-count":32,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2019,9]]}},"alternative-id":["e21090866"],"URL":"https:\/\/doi.org\/10.3390\/e21090866","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9,5]]}}}