{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:32:30Z","timestamp":1740123150476,"version":"3.37.3"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T00:00:00Z","timestamp":1637193600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T00:00:00Z","timestamp":1637193600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004895","name":"European Social Fund","doi-asserted-by":"publisher","award":["EFOP-3.6.2-16-2017-00015"],"award-info":[{"award-number":["EFOP-3.6.2-16-2017-00015"]}],"id":[{"id":"10.13039\/501100004895","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100009232","name":"University of Debrecen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100009232","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2022,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Ensemble-based methods are highly popular approaches that increase the accuracy of a decision by aggregating the opinions of individual voters. The common point is to maximize accuracy; however, a natural limitation occurs if incremental costs are also assigned to the individual voters. Consequently, we investigate creating ensembles under an additional constraint on the total cost of the members. This task can be formulated as a knapsack problem, where the energy is the ensemble accuracy formed by some aggregation rules. However, the generally applied aggregation rules lead to a nonseparable energy function, which takes the common solution tools\u2014such as dynamic programming\u2014out of action. We introduce a novel stochastic approach that considers the energy as the joint probability function of the member accuracies. This type of knowledge can be efficiently incorporated in a stochastic search process as a stopping rule, since we have the information on the expected accuracy or, alternatively, the probability of finding more accurate ensembles. Experimental analyses of the created ensembles of pattern classifiers and object detectors confirm the efficiency of our approach over other pruning ones. Moreover, we propose a novel stochastic search method that better fits the energy, which can be incorporated in other stochastic strategies as well.<\/jats:p>","DOI":"10.1007\/s10994-021-06109-0","type":"journal-article","created":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T18:02:53Z","timestamp":1637258573000},"page":"1551-1595","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A stochastic approach to handle resource constraints as knapsack problems in ensemble pruning"],"prefix":"10.1007","volume":"111","author":[{"given":"Andr\u00e1s","family":"Hajdu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gy\u00f6rgy","family":"Terdik","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3383-0134","authenticated-orcid":false,"given":"Attila","family":"Tiba","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Henrietta","family":"Tom\u00e1n","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,11,18]]},"reference":[{"issue":"6","key":"6109_CR1","doi-asserted-by":"publisher","first-page":"1720","DOI":"10.1109\/TBME.2012.2193126","volume":"59","author":"B Antal","year":"2012","unstructured":"Antal, B., & Hajdu, A. (2012). An ensemble-based system for microaneurysm detection and diabetic retinopathy grading. IEEE Transactions on Biomedical Engineering, 59(6), 1720\u20131726. https:\/\/doi.org\/10.1109\/TBME.2012.2193126.","journal-title":"IEEE Transactions on Biomedical Engineering"},{"key":"6109_CR2","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.knosys.2013.12.023","volume":"60","author":"B Antal","year":"2014","unstructured":"Antal, B., & Hajdu, A. (2014). An ensemble-based system for automatic screening of diabetic retinopathy. Knowledge-Based Systems, 60, 20\u201327. https:\/\/doi.org\/10.1016\/j.knosys.2013.12.023.","journal-title":"Knowledge-Based Systems"},{"issue":"1","key":"6109_CR3","first-page":"2653","volume":"18","author":"A Benavoli","year":"2017","unstructured":"Benavoli, A., et al. (2017). Time for a change: A tutorial for comparing multiple classifiers through Bayesian analysis. The Journal of Machine Learning Research, 18(1), 2653\u20132688.","journal-title":"The Journal of Machine Learning Research"},{"key":"6109_CR4","doi-asserted-by":"publisher","unstructured":"Bucilu, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model compression. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining, KDD\u201906 (pp. 535\u2013541). Association for Computing Machinery, New York, NY, USA https:\/\/doi.org\/10.1145\/1150402.1150464","DOI":"10.1145\/1150402.1150464"},{"key":"6109_CR5","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1016\/j.patrec.2016.01.029","volume":"74","author":"GD Cavalcanti","year":"2016","unstructured":"Cavalcanti, G. D., Oliveira, L. S., Moura, T. J., & Carvalho, G. V. (2016). Combining diversity measures for ensemble pruning. Pattern Recognition Letters, 74, 38\u201345. https:\/\/doi.org\/10.1016\/j.patrec.2016.01.029.","journal-title":"Pattern Recognition Letters"},{"issue":"2","key":"6109_CR6","doi-asserted-by":"publisher","first-page":"380","DOI":"10.1109\/21.364825","volume":"25","author":"SB Cho","year":"1995","unstructured":"Cho, S. B., & Kim, J. H. (1995). Combining multiple neural networks by fuzzy integral for robust classification. IEEE Transactions on Systems, Man, and Cybernetics, 25(2), 380\u2013384. https:\/\/doi.org\/10.1109\/21.364825.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"issue":"2","key":"6109_CR7","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1109\/4235.996017","volume":"6","author":"K Deb","year":"2002","unstructured":"Deb, K., Pratap, A., Agarwal, S., & Meyarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation, 6(2), 182\u2013197.","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"6109_CR8","unstructured":"Dheeru, D., & Karra Taniskidou, E. (2017) UCI machine learning repository."},{"key":"6109_CR9","doi-asserted-by":"crossref","unstructured":"Du, K., & Swamy, M. (2016). Search and optimization by metaheuristics: Techniques and algorithms inspired by nature. Springer","DOI":"10.1007\/978-3-319-41192-7"},{"key":"6109_CR10","unstructured":"Dzahini, K. (2020). Expected complexity analysis of stochastic direct-search. Les Cahiers du GERAD. GERAD HEC Montr\u00e9al. https:\/\/books.google.hu\/books?id=PKuvzQEACAAJ"},{"key":"6109_CR11","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1023\/A:1007662407062","volume":"37","author":"Y Freund","year":"1999","unstructured":"Freund, Y., & Schapire, R. E. (1999). Large margin classification using the perceptron algorithm. Machine Learning, 37, 277\u2013296.","journal-title":"Machine Learning"},{"key":"6109_CR12","volume-title":"Genetic algorithms in search, optimization and machine learning","author":"D Goldberg","year":"1989","unstructured":"Goldberg, D. (1989). Genetic algorithms in search, optimization and machine learning. Addison-Wesley."},{"issue":"11","key":"6109_CR13","doi-asserted-by":"publisher","first-page":"4182","DOI":"10.1109\/TIP.2013.2271116","volume":"22","author":"A Hajdu","year":"2013","unstructured":"Hajdu, A., Hajdu, L., J\u00f3n\u00e1s, A., Kov\u00e1cs, L., & Tom\u00e1n, H. (2013). Generalizing the majority voting scheme to spatially constrained voting. IEEE Transactions on Image Processing, 22(11), 4182\u20134194. https:\/\/doi.org\/10.1109\/TIP.2013.2271116.","journal-title":"IEEE Transactions on Image Processing"},{"key":"6109_CR14","doi-asserted-by":"crossref","unstructured":"Hajdu, A., Hajdu, L., Kov\u00e1cs, L., & Tom\u00e1n, H. (2013). Diversity measures for majority voting in the spatial domain. In J.S. Pan, M.M. Polycarpou, M.\u00a0Wo\u017aniak, A.C.P.L.F. de\u00a0Carvalho, H.\u00a0Quinti\u00e1n, E.\u00a0Corchado (Eds.), Hybrid artificial intelligent systems (pp. 314\u2013323). Springer Berlin Heidelberg.","DOI":"10.1007\/978-3-642-40846-5_32"},{"key":"6109_CR15","doi-asserted-by":"publisher","unstructured":"Hajdu, A., Tom\u00e1n, H., Kov\u00e1cs, L., & Hajdu, L. (2016). Composing ensembles by a stochastic approach under execution time constraint. In 2016 23rd international conference on pattern recognition (ICPR) (pp. 222\u2013227). https:\/\/doi.org\/10.1109\/ICPR.2016.7899637","DOI":"10.1109\/ICPR.2016.7899637"},{"issue":"10","key":"6109_CR16","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1109\/34.58871","volume":"12","author":"LK Hansen","year":"1990","unstructured":"Hansen, L. K., & Salamon, P. (1990). Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10), 993\u20131001. https:\/\/doi.org\/10.1109\/34.58871.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"6109_CR17","doi-asserted-by":"crossref","unstructured":"Harangi, B., Baran, A., & Hajdu, A. (2018). Classification of skin lesions using an ensemble of deep neural networks. In 40th annual international conference of the IEEE engineering in medicine and biology society, EMBC 2018, Honolulu, HI, USA, July 18\u201321, 2018 (pp. 2575\u20132578).","DOI":"10.1109\/EMBC.2018.8512800"},{"issue":"2","key":"6109_CR18","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1109\/TPAMI.2008.204","volume":"31","author":"D Hern\u00e1ndez-Lobato","year":"2009","unstructured":"Hern\u00e1ndez-Lobato, D., Martinez-Munoz, G., & Suarez, A. (2009). Statistical instance-based pruning in ensembles of independent classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2), 364\u2013369.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"6109_CR19","unstructured":"Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network."},{"issue":"1","key":"6109_CR20","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1109\/34.273716","volume":"16","author":"TK Ho","year":"1994","unstructured":"Ho, T. K., Hull, J. J., & Srihari, S. N. (1994). Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(1), 66\u201375. https:\/\/doi.org\/10.1109\/34.273716.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"1","key":"6109_CR21","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/34.368145","volume":"17","author":"YS Huang","year":"1995","unstructured":"Huang, Y. S., & Suen, C. Y. (1995). A method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(1), 90\u201394. https:\/\/doi.org\/10.1109\/34.368145.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"4","key":"6109_CR22","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1016\/0167-6377(90)90067-F","volume":"9","author":"T Klastorin","year":"1990","unstructured":"Klastorin, T. (1990). On a discrete nonlinear and nonseparable knapsack problem. Operations Research Letters, 9(4), 233\u2013237. https:\/\/doi.org\/10.1016\/0167-6377(90)90067-F.","journal-title":"Operations Research Letters"},{"key":"6109_CR23","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1016\/B978-1-55860-377-6.50046-3","volume-title":"Machine learning proceedings 1995","author":"EB Kong","year":"1995","unstructured":"Kong, E. B., & Dietterich, T. G. (1995). Error-correcting output coding corrects bias and variance. In A. Prieditis & S. Russell (Eds.), Machine learning proceedings 1995 (pp. 313\u2013321). Morgan Kaufmann. https:\/\/doi.org\/10.1016\/B978-1-55860-377-6.50046-3"},{"key":"6109_CR24","doi-asserted-by":"publisher","DOI":"10.1002\/0471660264","volume-title":"Combining pattern classifiers: Methods and algorithms","author":"LI Kuncheva","year":"2004","unstructured":"Kuncheva, L. I. (2004). Combining pattern classifiers: Methods and algorithms. Wiley-Interscience."},{"issue":"6","key":"6109_CR25","doi-asserted-by":"publisher","first-page":"652385","DOI":"10.1155\/2013\/652385","volume":"9","author":"M Kurz","year":"2013","unstructured":"Kurz, M., H\u00f6lzl, G., & Ferscha, A. (2013). Enabling dynamic sensor configuration and cooperation in opportunistic activity recognition systems. International Journal of Distributed Sensor Networks, 9(6), 652385. https:\/\/doi.org\/10.1155\/2013\/652385.","journal-title":"International Journal of Distributed Sensor Networks"},{"issue":"5","key":"6109_CR26","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1109\/3468.618255","volume":"27","author":"L Lam","year":"1997","unstructured":"Lam, L., & Suen, S. Y. (1997). Application of majority voting to pattern recognition: An analysis of its behavior and performance. IEEE Transactions on Systems, Man, and Cybernetics, 27(5), 553\u2013568. https:\/\/doi.org\/10.1109\/3468.618255.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"6109_CR27","doi-asserted-by":"crossref","unstructured":"Larochelle, H., & Bengio, Y. (2008). Classification using discriminative restricted Boltzmann machines. In Proceedings of the 25th international conference on machine learning (ICML) (pp. 536\u2013543).","DOI":"10.1145\/1390156.1390224"},{"key":"6109_CR28","volume-title":"Knapsack problems: Algorithms and computer implementations","author":"S Martello","year":"1990","unstructured":"Martello, S., & Toth, P. (1990). Knapsack problems: Algorithms and computer implementations. Wiley."},{"key":"6109_CR29","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1016\/j.patrec.2006.06.018","volume":"28","author":"G Martinez-Munoz","year":"2007","unstructured":"Martinez-Munoz, G., & Suarez, A. (2007). Using boosting to prune bagging ensembles. Pattern Recognition Letters, 28, 156\u2013165. https:\/\/doi.org\/10.1016\/j.patrec.2006.06.018.","journal-title":"Pattern Recognition Letters"},{"key":"6109_CR30","doi-asserted-by":"publisher","first-page":"652","DOI":"10.1016\/j.asoc.2015.09.009","volume":"37","author":"R Mousavi","year":"2015","unstructured":"Mousavi, R., & Eftekhari, M. (2015). A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Applied Soft Computing, 37, 652\u2013666. https:\/\/doi.org\/10.1016\/j.asoc.2015.09.009.","journal-title":"Applied Soft Computing"},{"key":"6109_CR31","unstructured":"Pisinger, D. (1995). Algorithms for knapsack problems."},{"key":"6109_CR32","first-page":"81","volume":"1","author":"JR Quinlan","year":"1986","unstructured":"Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1, 81\u2013106.","journal-title":"Machine Learning"},{"issue":"1","key":"6109_CR33","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1007\/s10107-009-0274-9","volume":"126","author":"TC Sharkey","year":"2011","unstructured":"Sharkey, T. C., Romeijn, H. E., & Geunes, J. (2011). A class of nonlinear nonseparable continuous knapsack and multiple-choice knapsack problems. Mathematical Programming, 126(1), 69\u201396. https:\/\/doi.org\/10.1007\/s10107-009-0274-9.","journal-title":"Mathematical Programming"},{"key":"6109_CR34","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1007\/978-3-642-12127-2_11","volume-title":"Multiple classifier systems","author":"V Soto","year":"2010","unstructured":"Soto, V., Mart\u00ednez-Mu\u00f1oz, G., Hern\u00e1ndez-Lobato, D., & Su\u00e1rez, A. (2010). A double pruning algorithm for classification ensembles. In N. El Gayar, J. Kittler, & F. Roli (Eds.), Multiple classifier systems (pp. 104\u2013113). Springer Berlin Heidelberg."},{"key":"6109_CR35","first-page":"24","volume-title":"Modified Levenberg\u2013Marquardt method for neural networks training","author":"AA Suratgar","year":"2005","unstructured":"Suratgar, A. A., Tavakoli, M. B., & Hoseinabadi, A. (2005). Modified Levenberg\u2013Marquardt method for neural networks training (pp. 24\u201348). World Academy of Science, Engineering and Technology."},{"key":"6109_CR36","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1\u20139).","DOI":"10.1109\/CVPR.2015.7298594"},{"issue":"3","key":"6109_CR37","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1016\/0167-7152(84)90008-7","volume":"2","author":"J Tang","year":"1984","unstructured":"Tang, J., & Gupta, A. K. (1984). On the distribution of the product of independent beta random variables. Statistics & Probability Letters, 2(3), 165\u2013168.","journal-title":"Statistics & Probability Letters"},{"issue":"2","key":"6109_CR38","doi-asserted-by":"publisher","first-page":"189","DOI":"10.3166\/ejc.13.189-203","volume":"13","author":"R Tempo","year":"2007","unstructured":"Tempo, R., & Ishii, H. (2007). Monte Carlo and Las Vegas randomized algorithms for systems and control*: An introduction. European Journal of Control, 13(2), 189\u2013203. https:\/\/doi.org\/10.3166\/ejc.13.189-203.","journal-title":"European Journal of Control"},{"key":"6109_CR39","doi-asserted-by":"crossref","unstructured":"Tiba, A., Hajdu, A., Terdik, G., & Toman, H. (2019). Optimizing majority voting based systems under a resource constraint for multiclass problems. eprint arXiv:1904.04360","DOI":"10.1007\/978-3-030-27550-1_67"},{"key":"6109_CR40","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1093\/comjnl\/bxp032","volume":"53","author":"S Timotheou","year":"2010","unstructured":"Timotheou, S. (2010). The random neural network: A survey. The Computer Journal, 53, 251\u2013267.","journal-title":"The Computer Journal"},{"key":"6109_CR41","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511921735","volume-title":"The design of approximation algorithms","author":"DP Williamson","year":"2011","unstructured":"Williamson, D. P., & Shmoys, D. B. (2011). The design of approximation algorithms (1st ed.). Cambridge University Press.","edition":"1"},{"issue":"3","key":"6109_CR42","doi-asserted-by":"publisher","first-page":"418","DOI":"10.1109\/21.155943","volume":"22","author":"L Xu","year":"1992","unstructured":"Xu, L., Krzyzak, A., & Suen, C. Y. (1992). Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Transactions on Systems, Man, and Cybernetics, 22(3), 418\u2013435. https:\/\/doi.org\/10.1109\/21.155943.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"6109_CR43","doi-asserted-by":"publisher","DOI":"10.1201\/b12207","volume-title":"Ensemble methods: Foundations and algorithms","author":"ZH Zhou","year":"2012","unstructured":"Zhou, Z. H. (2012). Ensemble methods: Foundations and algorithms (1st ed.). Chapman & Hall\/CRC.","edition":"1"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06109-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-021-06109-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-021-06109-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,17]],"date-time":"2022-05-17T15:38:57Z","timestamp":1652801937000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-021-06109-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,18]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,4]]}},"alternative-id":["6109"],"URL":"https:\/\/doi.org\/10.1007\/s10994-021-06109-0","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"type":"print","value":"0885-6125"},{"type":"electronic","value":"1573-0565"}],"subject":[],"published":{"date-parts":[[2021,11,18]]},"assertion":[{"value":"22 November 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 September 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 October 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 November 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}