{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T19:24:31Z","timestamp":1757618671305,"version":"3.44.0"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,7,11]],"date-time":"2025-07-11T00:00:00Z","timestamp":1752192000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,11]],"date-time":"2025-07-11T00:00:00Z","timestamp":1752192000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Neuroevolution combined with Novelty Search to promote behavioural diversity is capable of constructing high-performing ensembles for classification. However, using gradient descent to train evolved architectures during the search can be computationally prohibitive. We have proposed a method to overcome this limitation by using a <jats:italic>surrogate<\/jats:italic> model which estimates the behavioural distance between two neural network architectures, required to calculate novelty scores. This has demonstrated a speedup of 10 times over previous work, significantly improving on previous reported results on three benchmark datasets from Computer Vision\u2014CIFAR-10, CIFAR-100, and SVHN. This method makes an explicit search for diversity considerably more tractable <jats:italic>for the same bounded resources<\/jats:italic>. Here we investigate a range of search methods that span the full spectrum of favouring accuracy, diversity, or different combinations of both. Surprisingly, we show that multiple unique combinations between a diversity metric and accuracy give rise to similar results. This enables us to posit the existence of a diversity-accuracy duality in ensembles of classifiers, which suggests that there might not be a need to find a trade-off between the two.<\/jats:p>","DOI":"10.1007\/s42979-025-04056-4","type":"journal-article","created":{"date-parts":[[2025,7,11]],"date-time":"2025-07-11T11:33:39Z","timestamp":1752233619000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Improving Novelty Search with a Surrogate Model and Accuracy Objectives to Build High-Performing Ensembles of Classifiers"],"prefix":"10.1007","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8616-8482","authenticated-orcid":false,"given":"Rui P.","family":"Cardoso","sequence":"first","affiliation":[]},{"given":"Emma","family":"Hart","sequence":"additional","affiliation":[]},{"given":"Jeremy V.","family":"Pitt","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,7,11]]},"reference":[{"key":"4056_CR1","doi-asserted-by":"crossref","unstructured":"Dietterich TG, Ensemble methods in machine learning. In: International workshop on multiple classifier systems. Berlin: Springer; 2000. p. 1\u201315.","DOI":"10.1007\/3-540-45014-9_1"},{"key":"4056_CR2","doi-asserted-by":"crossref","unstructured":"Cardoso RP, Hart E, Kurka DB, Pitt JV. Using novelty search to explicitly create diversity in ensembles of classifiers. In: Proceedings of the Genetic and Evolutionary Computation Conference. GECCO \u201921. New York: Association for Computing Machinery; 2021. p. 849\u2013857.","DOI":"10.1145\/3449639.3459308"},{"key":"4056_CR3","doi-asserted-by":"publisher","first-page":"418","DOI":"10.1007\/978-3-031-02462-7_27","volume-title":"Applications of evolutionary computation","author":"RP Cardoso","year":"2022","unstructured":"Cardoso RP, Hart E, Kurka DB, Pitt J. Augmenting novelty search with a surrogate model to engineer meta-diversity in ensembles of classifiers. In: Jim\u00e9nez Laredo JL, Hidalgo JI, Babaagba KO, editors. Applications of evolutionary computation. Cham: Springer; 2022. p. 418\u201334."},{"key":"4056_CR4","unstructured":"Siems J, Zimmer L, Zela A, Lukasik J, Keuper M, Hutter F. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. CoRR https:\/\/arxiv.org\/abs\/2008.097772008.09777; 2020."},{"key":"4056_CR5","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1162\/EVCO_a_00025","volume":"19","author":"J Lehman","year":"2011","unstructured":"Lehman J, Stanley KO. Abandoning objectives: evolution through the search for novelty alone. Evol Comput. 2011;19:189\u2013223.","journal-title":"Evol Comput."},{"key":"4056_CR6","unstructured":"Zagoruyko S, Komodakis N. Wide residual networks. In: Wilson RC, Hancock ER, Smith WAP, editors. Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press; 2016. http:\/\/www.bmva.org\/bmvc\/2016\/papers\/paper087\/index.html"},{"key":"4056_CR7","doi-asserted-by":"publisher","unstructured":"Cardoso RP, Hart E, Kurka DB, Pitt JV. The diversity-accuracy duality in ensembles of classifiers. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion. GECCO \u201922. New York: Association for Computing Machinery; 2022. p. 627\u2013630. https:\/\/doi.org\/10.1145\/3520304.3528914.","DOI":"10.1145\/3520304.3528914"},{"key":"4056_CR8","doi-asserted-by":"publisher","unstructured":"Gu S, Jin Y. Generating diverse and accurate classifier ensembles using multi-objective optimization. In: 2014 IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM); 2014. p. 9\u201315 . https:\/\/doi.org\/10.1109\/MCDM.2014.7007182","DOI":"10.1109\/MCDM.2014.7007182"},{"issue":"1\u20133","key":"4056_CR9","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1007\/s10994-014-5477-5","volume":"101","author":"S \u00d6z\u00f6\u011f\u00fcr-Aky\u00fcz","year":"2015","unstructured":"\u00d6z\u00f6\u011f\u00fcr-Aky\u00fcz S, Windeatt T, Smith R. Pruning of error correcting output codes by optimization of accuracy-diversity trade off. Mach Learn. 2015;101(1\u20133):253\u201369. https:\/\/doi.org\/10.1007\/s10994-014-5477-5.","journal-title":"Mach Learn."},{"key":"4056_CR10","doi-asserted-by":"publisher","unstructured":"Zhu X, Zhong J, Zhuo L. Optimization of the trade-off by artificially re-sampling for ensemble learning. In: Third International Conference on Natural Computation (ICNC 2007), vol. 5; 2007. p. 49\u201353 https:\/\/doi.org\/10.1109\/ICNC.2007.527","DOI":"10.1109\/ICNC.2007.527"},{"key":"4056_CR11","unstructured":"Krizhevsky A. Learning Multiple Layers of Features from Tiny Images. ... Science Department, University of Toronto, Tech. ... (2009) arXiv: 1011.1669v3"},{"key":"4056_CR12","unstructured":"Netzer Y, Wang T. Reading digits in natural images with unsupervised feature learning. Nips; 2011."},{"issue":"1","key":"4056_CR13","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1007\/s12065-007-0002-4","volume":"1","author":"D Floreano","year":"2008","unstructured":"Floreano D, D\u00fcrr P, Mattiussi C. Neuroevolution: from architectures to learning. Evol Intell. 2008;1(1):47\u201362. https:\/\/doi.org\/10.1007\/s12065-007-0002-4.","journal-title":"Evol Intell."},{"key":"4056_CR14","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1016\/j.ins.2021.03.002","volume":"562","author":"H Tong","year":"2021","unstructured":"Tong H, Huang C, Minku LL, Yao X. Surrogate models in evolutionary single-objective optimization: a new taxonomy and experimental study. Inf Sci. 2021;562:414\u201337.","journal-title":"Inf Sci."},{"key":"4056_CR15","doi-asserted-by":"crossref","unstructured":"Ruan X, Li K, Derbel B, Liefooghe A Surrogate assisted evolutionary algorithm for medium scale multi-objective optimisation problems. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference; 2020. p. 560\u2013568.","DOI":"10.1145\/3377930.3390191"},{"issue":"1","key":"4056_CR16","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1109\/TSMCC.2005.855506","volume":"37","author":"Z Zhou","year":"2006","unstructured":"Zhou Z, Ong YS, Nair PB, Keane AJ, Lum KY. Combining global and local surrogate models to accelerate evolutionary optimization. IEEE Trans Syst Man Cybern Part C (Appl Rev). 2006;37(1):66\u201376.","journal-title":"IEEE Trans Syst Man Cybern Part C (Appl Rev)"},{"key":"4056_CR17","doi-asserted-by":"crossref","unstructured":"Gaier A, Asteroth A, Mouret J-B. Data-efficient neuroevolution with kernel-based surrogate models. In: Proceedings of the genetic and evolutionary computation conference; 2018. p. 85\u201392.","DOI":"10.1145\/3205455.3205510"},{"issue":"2","key":"4056_CR18","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1162\/106365602320169811","volume":"10","author":"KO Stanley","year":"2002","unstructured":"Stanley KO, Miikkulainen R. Evolving neural networks through augmenting topologies. Evol Comput. 2002;10(2):99\u2013127.","journal-title":"Evol Comput."},{"key":"4056_CR19","unstructured":"Deng B, Yan J, Lin D. Peephole: Predicting network performance before training. CoRR https:\/\/arxiv.org\/abs\/1712.033511712.03351; 2017."},{"key":"4056_CR20","doi-asserted-by":"crossref","unstructured":"Stork J, Zaefferer M, Bartz-Beielstein T. Improving neuroevolution efficiency by surrogate model-based optimization with phenotypic distance kernels. In: International conference on the applications of evolutionary computation (part of EvoStar). Berlin: Springer; 2019. p. 504\u2013519.","DOI":"10.1007\/978-3-030-16692-2_34"},{"key":"4056_CR21","doi-asserted-by":"publisher","unstructured":"Chil\u00e8s J-P, Desassis N. In: Daya Sagar BS, Cheng Q, Agterberg F, editors. Fifty years of kriging. Cham: Springer; 2018. p. 589\u2013612. https:\/\/doi.org\/10.1007\/978-3-319-78999-6_29.","DOI":"10.1007\/978-3-319-78999-6_29"},{"key":"4056_CR22","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1109\/TEVC.2019.2924461","volume":"24","author":"Y Sun","year":"2019","unstructured":"Sun Y, Wang H, Xue B, Jin Y, Yen GG, Zhang M. Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor. IEEE Trans Evol Comput. 2019;24:350\u201364.","journal-title":"IEEE Trans Evol Comput."},{"key":"4056_CR23","doi-asserted-by":"crossref","unstructured":"Hagg A, Zaefferer M, Stork J, Gaier A. Prediction of neural network performance by phenotypic modeling. In: Proceedings of the genetic and evolutionary computation conference companion. GECCO \u201919. New York: Association for Computing Machinery; 2019. p. 1576\u20131582","DOI":"10.1145\/3319619.3326815"},{"key":"4056_CR24","unstructured":"Krogh A, Vedelsby J. Neural network ensembles, cross validation and active learning. In: Proceedings of the 7th international conference on neural information processing systems. NIPS\u201994. Cambridge: MIT Press; 1994. p. 231\u2013238."},{"key":"4056_CR25","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1007\/11399346_19","volume":"16","author":"A Chandra","year":"2006","unstructured":"Chandra A, Chen H, Yao X. Trade-off between diversity and accuracy in ensemble generation. Stud Comput Intell. 2006;16:429\u201364. https:\/\/doi.org\/10.1007\/11399346_19.","journal-title":"Stud Comput Intell."},{"issue":"2","key":"4056_CR26","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1007\/BF00116037","volume":"5","author":"RE Schapire","year":"1990","unstructured":"Schapire RE. The strength of weak learnability. Mach Learn. 1990;5(2):197\u2013227. https:\/\/doi.org\/10.1007\/BF00116037.","journal-title":"Mach Learn."},{"issue":"2","key":"4056_CR27","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1007\/BF00058655","volume":"24","author":"L Breiman","year":"1996","unstructured":"Breiman L. Bagging predictors. Mach Learn. 1996;24(2):123\u201340. https:\/\/doi.org\/10.1007\/BF00058655.","journal-title":"Mach Learn."},{"issue":"3","key":"4056_CR28","doi-asserted-by":"publisher","first-page":"368","DOI":"10.1109\/TEVC.2012.2199119","volume":"17","author":"U Bhowan","year":"2012","unstructured":"Bhowan U, Johnston M, Zhang M, Yao X. Evolving diverse ensembles using genetic programming for classification with unbalanced data. IEEE Trans Evol Comput. 2012;17(3):368\u201386.","journal-title":"IEEE Trans Evol Comput."},{"issue":"6","key":"4056_CR29","doi-asserted-by":"publisher","first-page":"893","DOI":"10.1109\/TEVC.2013.2293393","volume":"18","author":"U Bhowan","year":"2013","unstructured":"Bhowan U, Johnston M, Zhang M, Yao X. Reusing genetic programming for ensemble selection in classification of unbalanced data. IEEE Trans Evol Comput. 2013;18(6):893\u2013908.","journal-title":"IEEE Trans Evol Comput."},{"key":"4056_CR30","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1016\/j.neucom.2017.03.055","volume":"247","author":"W Sheng","year":"2017","unstructured":"Sheng W, Shan P, Chen S, Liu Y, Alsaadi FE. A niching evolutionary algorithm with adaptive negative correlation learning for neural network ensemble. Neurocomputing. 2017;247:173\u201382. https:\/\/doi.org\/10.1016\/j.neucom.2017.03.055.","journal-title":"Neurocomputing."},{"issue":"1","key":"4056_CR31","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1162\/EVCO_a_00203","volume":"26","author":"E Hart","year":"2018","unstructured":"Hart E, Sim K. On constructing ensembles for combinatorial optimisation. Evol Comput. 2018;26(1):67\u201387. https:\/\/doi.org\/10.1162\/EVCO_a_00203.","journal-title":"Evol Comput."},{"issue":"4","key":"4056_CR32","doi-asserted-by":"publisher","first-page":"710","DOI":"10.1007\/s10489-013-0507-8","volume":"40","author":"A Tsakonas","year":"2014","unstructured":"Tsakonas A. An analysis of accuracy-diversity trade-off for hybrid combined system with multiobjective predictor selection. Appl Intell. 2014;40(4):710\u201323. https:\/\/doi.org\/10.1007\/s10489-013-0507-8.","journal-title":"Appl Intell."},{"key":"4056_CR33","doi-asserted-by":"crossref","unstructured":"Cardoso RP, Hart E, Pitt JV. Diversity-driven wide learning for training distributed classification models. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion. GECCO \u201920. New York: Association for Computing Machinery; 2020. p. 119\u2013120","DOI":"10.1145\/3377929.3390012"},{"key":"4056_CR34","doi-asserted-by":"crossref","unstructured":"Cardoso RP, Hart E, Kurka DB, Pitt J. WILDA: wide learning of diverse architectures for classification of large datasets. In: Castillo PA, Laredo JLJ, editors. Applications of Evolutionary Computation\u201424th International Conference, EvoApplications 2021, Held as Part of EvoStar 2021, Virtual Event, April 7-9, 2021, Proceedings. Lecture Notes in Computer Science, vol. 12694. Berlin: Springer; 2021. p. 649\u2013664.","DOI":"10.1007\/978-3-030-72699-7_41"},{"key":"4056_CR35","unstructured":"Mouret J, Clune J. Illuminating search spaces by mapping elites. CoRR https:\/\/arxiv.org\/abs\/1504.049091504.04909; 2015."},{"key":"4056_CR36","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1016\/j.patrec.2021.03.008","volume":"146","author":"A Mukhriya","year":"2021","unstructured":"Mukhriya A, Kumar R. Building outlier detection ensembles by selective parameterization of heterogeneous methods. Pattern Recogn Lett. 2021;146:126\u201333. https:\/\/doi.org\/10.1016\/j.patrec.2021.03.008.","journal-title":"Pattern Recogn Lett."},{"key":"4056_CR37","first-page":"1621","volume":"6","author":"G Brown","year":"2005","unstructured":"Brown G, Wyatt JL, Ti\u00f1o P. Managing diversity in regression ensembles. J Mach Learn Res. 2005;6:1621\u201350.","journal-title":"J Mach Learn Res."},{"key":"4056_CR38","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.isprsjprs.2017.04.017","volume":"129","author":"A Mellor","year":"2017","unstructured":"Mellor A, Boukir S. Exploring diversity in ensemble classification: applications in large area land cover mapping. ISPRS J Photogrammetry Remote Sens. 2017;129:151\u201361. https:\/\/doi.org\/10.1016\/j.isprsjprs.2017.04.017.","journal-title":"ISPRS J Photogrammetry Remote Sens."},{"key":"4056_CR39","doi-asserted-by":"publisher","unstructured":"Wang S, Yao X. Diversity analysis on imbalanced data sets by using ensemble models. In: Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2009, Part of the IEEE Symposium Series on Computational Intelligence 2009, Nashville, TN, USA, March 30, 2009\u2014April 2, 2009. IEEE; 2009. p. 324\u2013331. https:\/\/doi.org\/10.1109\/CIDM.2009.4938667.","DOI":"10.1109\/CIDM.2009.4938667"},{"key":"4056_CR40","doi-asserted-by":"publisher","unstructured":"Trawinski K, Quirin A, Cord\u00f3n O. On the combination of accuracy and diversity measures for genetic selection of bagging fuzzy rule-based multiclassification systems. In: Ninth International Conference on Intelligent Systems Design and Applications, ISDA 2009, Pisa, Italy , November 30-December 2, 2009. IEEE Computer Society; 2009. p. 121\u2013127. https:\/\/doi.org\/10.1109\/ISDA.2009.123.","DOI":"10.1109\/ISDA.2009.123"},{"key":"4056_CR41","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016. p. 770\u2013778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"4056_CR42","unstructured":"Paszke A, Gross S, Chintala S, Chanan G, Yang E, Facebook ZD, Research AI, Lin Z, Desmaison A, Antiga L, Srl O, Lerer A. Automatic differentiation in PyTorch. In: Advances in neural information processing systems, vol. 32; 2019."},{"key":"4056_CR43","unstructured":"Van Krevelen R. Error diversity in classification ensembles. PhD thesis; 2005."},{"key":"4056_CR44","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1023\/A:1022859003006","volume":"51","author":"LI Kuncheva","year":"2003","unstructured":"Kuncheva LI, Whitaker CJ. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn. 2003;51:181\u2013207.","journal-title":"Mach Learn."},{"issue":"3","key":"4056_CR45","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1007\/s11047-009-9124-1","volume":"9","author":"R Pasti","year":"2010","unstructured":"Pasti R, De Castro LN, Coelho GP, Von Zuben FJ. Neural network ensembles: immune-inspired approaches to the diversity of components. Nat Comput. 2010;9(3):625\u201353.","journal-title":"Nat Comput."},{"issue":"1","key":"4056_CR46","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L. Random forests. Mach Learn. 2001;45(1):5\u201332.","journal-title":"Mach Learn."},{"key":"4056_CR47","doi-asserted-by":"crossref","unstructured":"Gomes J, Mariano P, Christensen AL. Devising effective novelty search algorithms: a comprehensive empirical study. In: GECCO 2015\u2014proceedings of the 2015 genetic and evolutionary computation conference; 2015.","DOI":"10.1145\/2739480.2754736"},{"key":"4056_CR48","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/S0893-6080(05)80023-1","volume":"5","author":"DH Wolpert","year":"1992","unstructured":"Wolpert DH. Stacked generalization. Neural Netw. 1992;5:241\u201359.","journal-title":"Neural Netw."},{"key":"4056_CR49","doi-asserted-by":"crossref","unstructured":"Lehman J, Stanley KO. Evolving a diversity of virtual creatures through novelty search and local competition. In: Proceedings of the 13th annual conference on genetic and evolutionary computation. ACM; 2011. p. 211\u2013218.","DOI":"10.1145\/2001576.2001606"}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-025-04056-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-025-04056-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-025-04056-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,7]],"date-time":"2025-09-07T05:24:49Z","timestamp":1757222689000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-025-04056-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,11]]},"references-count":49,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["4056"],"URL":"https:\/\/doi.org\/10.1007\/s42979-025-04056-4","relation":{},"ISSN":["2661-8907"],"issn-type":[{"type":"electronic","value":"2661-8907"}],"subject":[],"published":{"date-parts":[[2025,7,11]]},"assertion":[{"value":"2 September 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 May 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 July 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Research Involving Human and\/or Animals"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed Consent"}}],"article-number":"631"}}