{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,13]],"date-time":"2025-11-13T01:53:52Z","timestamp":1762998832384,"version":"3.41.2"},"reference-count":51,"publisher":"Emerald","issue":"4","license":[{"start":{"date-parts":[[2010,11,22]],"date-time":"2010-11-22T00:00:00Z","timestamp":1290384000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2010,11,22]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>Multiple classifier systems have been used widely in computing, communications, and informatics. Combining multiple classifier systems (MCS) has been shown to outperform a single classifier system. It has been demonstrated that improvement in ensemble performance depends on either the diversity among or the performance of individual systems. A variety of diversity measures and ensemble methods have been proposed and studied. However, it remains a challenging problem to estimate the ensemble performance in terms of the performance of and the diversity among individual systems. The purpose of this paper is to study the general problem of estimating ensemble performance for various combination methods using the concept of a performance distribution pattern (PDP).<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>In particular, the paper establishes upper and lower bounds for majority voting ensemble performance with disagreement diversity measure <jats:italic>Dis<\/jats:italic>, weighted majority voting performance in terms of weighted average performance and weighted disagreement diversity, and plurality voting ensemble performance with entropy diversity measure <jats:italic>D<\/jats:italic>.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>Bounds for these three cases are shown to be tight using the PDP for the input set.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>As a consequence of the authors' previous results on diversity equivalence, the results of majority voting ensemble performance can be extended to several other diversity measures. Moreover, the paper showed in the case of majority voting ensemble performance that when the average of individual systems performance <jats:italic>P<\/jats:italic> is big enough, the ensemble performance <jats:italic>P<\/jats:italic><jats:sup>m<\/jats:sup> resulting from a maximum (information\u2010theoretic) entropy PDP is an increasing function with respect to the disagreement diversity <jats:italic>Dis<\/jats:italic>. Eight experiments using data sets from various application domains are conducted to demonstrate the complexity, richness, and diverseness of the problem in estimating the ensemble performance.<\/jats:p><\/jats:sec>","DOI":"10.1108\/17427371011097604","type":"journal-article","created":{"date-parts":[[2010,12,18]],"date-time":"2010-12-18T07:06:05Z","timestamp":1292655965000},"page":"373-403","source":"Crossref","is-referenced-by-count":7,"title":["Performance evaluation of classifier ensembles in terms of diversity and performance of individual systems"],"prefix":"10.1108","volume":"6","author":[{"given":"Yun\u2010Sheng","family":"Chung","sequence":"first","affiliation":[]},{"given":"D. Frank","family":"Hsu","sequence":"additional","affiliation":[]},{"given":"Chun\u2010Yi","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Chun\u2010Yi","family":"Tang","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2022030920235571900_b21","doi-asserted-by":"crossref","unstructured":"Aksela, M. and Laaksonen, J. (2006), \u201cUsing diversity of errors for selecting members of a committee classifier\u201d, Pattern Recognition, Vol. 39, pp. 608\u201023.","DOI":"10.1016\/j.patcog.2005.08.017"},{"key":"key2022030920235571900_b43","doi-asserted-by":"crossref","unstructured":"Bertsimas, D. and Popescu, I. (2005), \u201cOptimal inequalities in probability theory: a convex optimization approach\u201d, SIAM Journal on Optimization, Vol. 15 No. 3, pp. 780\u2010804.","DOI":"10.1137\/S1052623401399903"},{"key":"key2022030920235571900_b49","unstructured":"Blake, C.L., Newman, D.J., Hettich, S. and Merz, C. (1998), \u201cUCI repository of machine learning databases\u201d, available at: www.ics.uci.edu\/\u223clearn\/MLRepository.html"},{"key":"key2022030920235571900_b50","doi-asserted-by":"crossref","unstructured":"Breiman, L. (1998), \u201cArcing classifiers\u201d, The Annals of Statistics, Vol. 26 No. 3, pp. 801\u2010849.","DOI":"10.1214\/aos\/1024691079"},{"key":"key2022030920235571900_b11","doi-asserted-by":"crossref","unstructured":"Breiman, L. (2001), \u201cRandom forests\u201d, Machine Learning, Vol. 45, pp. 5\u201032.","DOI":"10.1023\/A:1010933404324"},{"key":"key2022030920235571900_b26","doi-asserted-by":"crossref","unstructured":"Brown, G., Wyatt, J., Harris, R. and Yao, X. (2005), \u201cDiversity creation methods: a survey and categorisation\u201d, Information Fusion, Vol. 6, pp. 5\u201020.","DOI":"10.1016\/j.inffus.2004.04.004"},{"key":"key2022030920235571900_b39","unstructured":"Chung, Y.\u2010S. (2007), \u201cEnsemble performance in terms of diversity and performance of individual classifier systems (2007)\u201d, PhD thesis, Department of Computer Science, National Tsing Hua University, Hsinchu."},{"key":"key2022030920235571900_b44","doi-asserted-by":"crossref","unstructured":"Chung, Y.\u2010S., Hsu, D.F. and Tang, C.Y. (2007), \u201cOn the diversity\u2010performance relationship for majority voting in classifier ensembles\u201d, in Kittler, J., Roli, F. and Haindl, M. (Eds), Lecture Notes in Computer Science: Multiple Classifier Systems, Vol. 4472, Springer, New York, NY, pp. 407\u201020.","DOI":"10.1007\/978-3-540-72523-7_41"},{"key":"key2022030920235571900_b38","doi-asserted-by":"crossref","unstructured":"Chung, Y.\u2010S., Hsu, D.F. and Tang, C.Y. (2008), \u201cOn the relationships among various diversity measures in multiple classifier systems\u201d, The International Symposium on Parallel Architectures, Algorithms, and Networks (I\u2010SPAN 2008), pp. 184\u201090.","DOI":"10.1109\/I-SPAN.2008.46"},{"key":"key2022030920235571900_b46","doi-asserted-by":"crossref","unstructured":"Cover, T.M. and Thomas, J.A. (1991), Elements of Information Theory, John Wiley & Sons, New York, NY.","DOI":"10.1002\/0471200611"},{"key":"key2022030920235571900_b32","doi-asserted-by":"crossref","unstructured":"Demirekler, M. and Altin\u00e7ay, H. (2002), \u201cPlurality voting\u2010based multiple classifier systems: statistically independent with respect to dependent classifier sets\u201d, Pattern Recognition, Vol. 35 No. 11, pp. 2365\u201079.","DOI":"10.1016\/S0031-3203(01)00227-8"},{"key":"key2022030920235571900_b42","doi-asserted-by":"crossref","unstructured":"Devroye, L., Gy\u00f6r, L. and Lugosi, G. (1996), A Probabilistic Theory of Pattern Recognition, Springer\u2010Verlag, Berlin.","DOI":"10.1007\/978-1-4612-0711-5"},{"key":"key2022030920235571900_b13","doi-asserted-by":"crossref","unstructured":"Dietterich, T.G. (2000a), \u201cEnsemble methods in machine learning\u201d, in Kittler, J. and Roli, F. (Eds),, Lecture Notes in Computer Science: Multiple Classifier Systems, Vol. 1857, Springer, New York, NY, pp. 1\u201015.","DOI":"10.1007\/3-540-45014-9_1"},{"key":"key2022030920235571900_b45","unstructured":"Dietterich, T.G. (2000b), \u201cAn experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting and randomization\u201d, Machine Learning, Vol. 40, pp. 139\u201057."},{"key":"key2022030920235571900_b8","doi-asserted-by":"crossref","unstructured":"Fumera, G. and Roli, F. (2005), \u201cA theoretical and experimental analysis of linear combiners and multiple classifier systems\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27 No. 6, pp. 942\u201056.","DOI":"10.1109\/TPAMI.2005.109"},{"key":"key2022030920235571900_b10","doi-asserted-by":"crossref","unstructured":"Hansen, L. and Salamon, P. (1990), \u201cNeural network ensembles\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 12 No. 10, pp. 993\u20101001.","DOI":"10.1109\/34.58871"},{"key":"key2022030920235571900_b35","unstructured":"Ho, T.K. (1992), \u201cA theory of multiple classifier systems and its application to visual word recognition\u201d, PhD thesis, State University of New York at Buffalo, Buffalo, NY."},{"key":"key2022030920235571900_b2","doi-asserted-by":"crossref","unstructured":"Ho, T.K. (1998), \u201cThe random space method for constructing decision forests\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, pp. 832\u2010844.","DOI":"10.1109\/34.709601"},{"key":"key2022030920235571900_b3","doi-asserted-by":"crossref","unstructured":"Ho, T.K. (2002), \u201cMultiple classifier combination: lessons and next steps\u201d, in Bunke, H. and Kandel, A. (Eds), Hybrid Methods in Pattern Recognition, World Scientific, Hackensack, NJ, pp. 171\u201098.","DOI":"10.1142\/9789812778147_0007"},{"key":"key2022030920235571900_b1","doi-asserted-by":"crossref","unstructured":"Ho, T.K., Hull, J.J. and Srihari, S.N. (1991), \u201cDecision combination in multiple classifier systems\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 16, pp. 66\u201075.","DOI":"10.1109\/34.273716"},{"key":"key2022030920235571900_b15","doi-asserted-by":"crossref","unstructured":"Hsu, D.F. and Taksa, I. (2005), \u201cComparing rank and score combination methods for data fusion in information retrieval\u201d, Information Retrieval, Vol. 8 No. 3, pp. 449\u201080.","DOI":"10.1007\/s10791-005-6994-4"},{"key":"key2022030920235571900_b14","doi-asserted-by":"crossref","unstructured":"Hsu, D.F., Chung, Y.S. and Kristal, B.S. (2006), \u201cCombinatorial fusion analysis: methods and practices of combining multiple scoring systems\u201d, in Hsu, H.\u2010H. (Ed.), Advanced Data Mining Technologies in Bioinformatics, Idea Group, Hershey, PA, pp. 32\u201062.","DOI":"10.4018\/978-1-59140-863-5.ch003"},{"key":"key2022030920235571900_b51","unstructured":"Kendall, M. and Gibbons, J.D. (1990), Rank Correlation Methods, Edward Arnold, London."},{"key":"key2022030920235571900_b7","doi-asserted-by":"crossref","unstructured":"Kittler, J. and Alkoot, F. (2003), \u201cSum versus vote fusion in multiple classifier systems\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25 No. 1, pp. 110\u201015.","DOI":"10.1109\/TPAMI.2003.1159950"},{"key":"key2022030920235571900_b6","doi-asserted-by":"crossref","unstructured":"Kittler, J., Hatef, M., Duin, R.P. and Matas, J. (1998), \u201cOn combining classifiers\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20 No. 3, pp. 226\u201039.","DOI":"10.1109\/34.667881"},{"key":"key2022030920235571900_b19","unstructured":"Krogh, A. and Vedelsby, J. (1995), \u201cNeural network ensembles, cross validation, and active learning\u201d, in Tesauro, G., Touretzky, D.S. and Leen, T.K. (Eds), Advances in Neural Information Processing Systems, Vol. 7, MIT Press, Cambridge, MA, pp. 231\u201038."},{"key":"key2022030920235571900_b18","doi-asserted-by":"crossref","unstructured":"Kuncheva, L. (2003), \u201cThat elusive diversity in classifier ensembles\u201d, in L\u00f3pez, F.J.P., Campilho, A.C., de la Blanca, N.P. and Sanfeliu, A. (Eds), Lecture Notes in Computer Science: IbPRIA, Vol. 2652, Springer, New York, NY, pp. 1126\u201038.","DOI":"10.1007\/978-3-540-44871-6_130"},{"key":"key2022030920235571900_b48","doi-asserted-by":"crossref","unstructured":"Kuncheva, L.I. (2000), Fuzzy Classifier Design. Studies in Fuzziness and Soft Computing, Springer\u2010Verlag, Berlin.","DOI":"10.1007\/978-3-7908-1850-5"},{"key":"key2022030920235571900_b4","doi-asserted-by":"crossref","unstructured":"Kuncheva, L.I. (2004), Combining Pattern Classifiers: Methods and Algorithms, John Wiley & Sons, Hoboken, NJ.","DOI":"10.1002\/0471660264"},{"key":"key2022030920235571900_b34","doi-asserted-by":"crossref","unstructured":"Kuncheva, L.I. (2005), \u201cDiversity in multiple classifier systems\u201d, Information Fusion, Vol. 6 No. 1, pp. 3\u20104.","DOI":"10.1016\/j.inffus.2004.04.009"},{"key":"key2022030920235571900_b17","doi-asserted-by":"crossref","unstructured":"Kuncheva, L.I. and Whitaker, C.J. (2003), \u201cMeasures of diversity in classifier ensembles and their relationship with the ensemble accuracy\u201d, Machine Learning, Vol. 51, pp. 181\u2010207.","DOI":"10.1023\/A:1022859003006"},{"key":"key2022030920235571900_b30","doi-asserted-by":"crossref","unstructured":"Kuncheva, L.I., Whitaker, C.J., Shipp, C.A. and Duin, R.P.W. (2003), \u201cLimits on the majority vote accuracy in classifier fusion\u201d, Pattern Analysis and Applications, Vol. 6, pp. 22\u201031.","DOI":"10.1007\/s10044-002-0173-7"},{"key":"key2022030920235571900_b41","unstructured":"Lacasse, A., Laviolette, F., Marchand, M., Germain, P. and Usunier, N. (2006), \u201cPAC\u2010Bayes bounds for the risk of the majority vote and the variance of the Gibbs classifier\u201d, in Sch\u0151lkopf, B., Platt, J. and Hoffman, T.  (Eds), Advances in Neural Information Processing Systems, 19, MIT Press, Cambridge, MA, pp. 449\u201056."},{"key":"key2022030920235571900_b37","doi-asserted-by":"crossref","unstructured":"Lam, L. and Suen, C.Y. (1997), \u201cApplication of majority voting to pattern recognition: an analysis of its behaviour and performance\u201d, IEEE Transactions on Systems, Man, and Cybernetics, Vol. 27 No. 5, pp. 533\u201068.","DOI":"10.1109\/3468.618255"},{"key":"key2022030920235571900_b47","doi-asserted-by":"crossref","unstructured":"McAllester, D. (1999), \u201cSome PAC\u2010Bayesian theorems\u201d, Machine Learning, Vol. 37, pp. 355\u201063.","DOI":"10.1023\/A:1007618624809"},{"key":"key2022030920235571900_b29","unstructured":"Matan, O. (1996), \u201cOn voting ensembles of classifiers\u201d, in Proceedings of the American Association for Artificial Intelligence, AAAI\u201096, Integrating Multiple Learned Models Workshop, pp. 84\u20108."},{"key":"key2022030920235571900_b9","doi-asserted-by":"crossref","unstructured":"Narasimhamurthy, A. (2005a), \u201cTheoretical bounds of majority voting performance for a binary classification problem\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27 No. 12, pp. 1988\u201095.","DOI":"10.1109\/TPAMI.2005.249"},{"key":"key2022030920235571900_b25","doi-asserted-by":"crossref","unstructured":"Narasimhamurthy, A. (2005b), \u201cEvaluation of diversity measures for binary classifier ensembles\u201d, LNCS: Multiple Classifier Systems, Vol. 3541, Springer, New York, NY, pp. 267\u201077.","DOI":"10.1007\/11494683_27"},{"key":"key2022030920235571900_b27","doi-asserted-by":"crossref","unstructured":"Ng, K.B. and Kantor, P.B. (2000), \u201cPredicting the effectiveness of na\u00efve data fusion on the basis of system characteristics\u201d, Journal of the American Society for Information Science, Vol. 51 No. 13, pp. 1177\u201089.","DOI":"10.1002\/1097-4571(2000)9999:9999<::AID-ASI1030>3.0.CO;2-E"},{"key":"key2022030920235571900_b22","unstructured":"Partridge, D. and Krzanowski, W. (2003), Refining Multiple Classifier System Diversity, Technical Report, Computer Science Department, University of Exeter, Exeter."},{"key":"key2022030920235571900_b5","doi-asserted-by":"crossref","unstructured":"Rodriguez, J.J., Kuncheva, L.I. and Alonso, C.J. (2006), \u201cRotation forest: a new classifier ensemble method\u201d, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28 No. 10, pp. 1619\u201030.","DOI":"10.1109\/TPAMI.2006.211"},{"key":"key2022030920235571900_b33","doi-asserted-by":"crossref","unstructured":"Roli, F. and Kittler, J. (2002), \u201cEditorial: Fusion of multiple classifiers\u201d, Information Fusion, Vol. 3 No. 4, p. 243.","DOI":"10.1016\/S1566-2535(02)00094-5"},{"key":"key2022030920235571900_b23","unstructured":"Ruta, D. (2003), \u201cClassifier diversity in combined pattern recognition systems\u201d, PhD thesis, University of Paisley, Paisley."},{"key":"key2022030920235571900_b31","doi-asserted-by":"crossref","unstructured":"Ruta, D. and Gabrys, B. (2002), \u201cA theoretical analysis of the majority voting errors for multiple classifier systems\u201d, Pattern Analysis and Applications, Vol. 5, pp. 333\u201050.","DOI":"10.1007\/s100440200030"},{"key":"key2022030920235571900_b12","doi-asserted-by":"crossref","unstructured":"Sharkey, A., (Ed.) (1999), Combining Artificial Neural Nets, Springer\u2010Verlag, Berlin.","DOI":"10.1007\/978-1-4471-0793-4"},{"key":"key2022030920235571900_b24","doi-asserted-by":"crossref","unstructured":"Shipp, C. and Kuncheva, L. (2002), \u201cRelationships between combination methods and measures of diversity in combining classifiers\u201d, Information Fusion, Vol. 3 No. 2, pp. 135\u201048.","DOI":"10.1016\/S1566-2535(02)00051-9"},{"key":"key2022030920235571900_b20","doi-asserted-by":"crossref","unstructured":"Tumer, K. and Ghosh, J. (1999), \u201cLinear and order statistics combiners for pattern classification\u201d, in Sharkey, A. (Ed.), Combining Artificial Neural Nets, Springer\u2010Verlag, Berlin, pp. 127\u201062.","DOI":"10.1007\/978-1-4471-0793-4_6"},{"key":"key2022030920235571900_b36","doi-asserted-by":"crossref","unstructured":"Xu, L., Krzyzak, A. and Suen, C.Y. (1992), \u201cMethods for combining multiple classifiers and their applications to handwritten recognition\u201d, IEEE Transactions on Systems, Man, and Cybernetics, Vol. 22 No. 3, pp. 418\u201035.","DOI":"10.1109\/21.155943"},{"key":"key2022030920235571900_b16","doi-asserted-by":"crossref","unstructured":"Yang, J.M., Chen, Y.F., Shen, T.W., Kristal, B.S. and Hsu, D.F. (2005), \u201cConsensus scoring for improving enrichment in virtual screening\u201d, Journal of Chemical Information and Modeling, Vol. 45, pp. 1134\u201046.","DOI":"10.1021\/ci050034w"},{"key":"key2022030920235571900_b40","doi-asserted-by":"crossref","unstructured":"Yule, G.U. (1900), \u201cOn the association of attributes in statistics\u201d, Philosophy Trans., A, Vol. 194, pp. 257\u201019.","DOI":"10.1098\/rsta.1900.0019"},{"key":"key2022030920235571900_b28","doi-asserted-by":"crossref","unstructured":"Zenobi, G. and Cunningham, P. (2001), \u201cUsing diversity in preparing ensembles of classifiers based on different feature subsets to minimize generalization error\u201d, LNCS: European Conference on Machine Learning, Vol. 2167, Springer, New York, NY, pp. 576\u201087.","DOI":"10.1007\/3-540-44795-4_49"}],"container-title":["International Journal of Pervasive Computing and Communications"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/17427371011097604","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/17427371011097604\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/17427371011097604\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:24:17Z","timestamp":1753403057000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/ijpcc\/article\/6\/4\/373-403\/161951"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,11,22]]},"references-count":51,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2010,11,22]]}},"alternative-id":["10.1108\/17427371011097604"],"URL":"https:\/\/doi.org\/10.1108\/17427371011097604","relation":{},"ISSN":["1742-7371"],"issn-type":[{"type":"print","value":"1742-7371"}],"subject":[],"published":{"date-parts":[[2010,11,22]]}}}