{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:03:59Z","timestamp":1772906639371,"version":"3.50.1"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,1,16]],"date-time":"2024-01-16T00:00:00Z","timestamp":1705363200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF CISE","award":["1564097, 2038029, 2302720, and 2312758"],"award-info":[{"award-number":["1564097, 2038029, 2302720, and 2312758"]}]},{"name":"IBM Faculty Award, and a CISCO Edge AI"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,2,29]]},"abstract":"<jats:p>\n            Deep neural network ensembles combine the wisdom of multiple deep neural networks to improve the generalizability and robustness over individual networks. It has gained increasing popularity to study and apply deep ensemble techniques in the deep learning community. Some mission-critical applications utilize a large number of deep neural networks to form deep ensembles to achieve desired accuracy and resilience, which introduces high time and space costs for ensemble execution. However, it still remains a critical challenge whether a small subset of the entire deep ensemble can achieve the same or better generalizability and how to effectively identify these small deep ensembles for improving the space and time efficiency of ensemble execution. This article presents a novel deep ensemble pruning approach, which can efficiently identify smaller deep ensembles and provide higher ensemble accuracy than the entire deep ensemble of a large number of member networks. Our hierarchical ensemble pruning approach (HQ) leverages three novel ensemble pruning techniques. First, we show that the focal ensemble diversity metrics can accurately capture the complementary capacity of the member networks of an ensemble team, which can guide ensemble pruning. Second, we design a focal ensemble diversity based hierarchical pruning approach, which will iteratively find high quality deep ensembles with low cost and high accuracy. Third, we develop a focal diversity consensus method to integrate multiple focal diversity metrics to refine ensemble pruning results, where smaller deep ensembles can be effectively identified to offer high accuracy, high robustness and high ensemble execution efficiency. Evaluated using popular benchmark datasets, we demonstrate that the proposed hierarchical ensemble pruning approach can effectively identify high quality deep ensembles with better classification generalizability while being more time and space efficient in ensemble decision making. We have released the source codes on GitHub at\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"url\" xlink:href=\"https:\/\/github.com\/git-disl\/HQ-Ensemble\">https:\/\/github.com\/git-disl\/HQ-Ensemble<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3633286","type":"journal-article","created":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T12:14:57Z","timestamp":1700223297000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Hierarchical Pruning of Deep Ensembles with Focal Diversity"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8761-5486","authenticated-orcid":false,"given":"Yanzhao","family":"Wu","sequence":"first","affiliation":[{"name":"Florida International University, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5917-2577","authenticated-orcid":false,"given":"Ka-Ho","family":"Chow","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9177-114X","authenticated-orcid":false,"given":"Wenqi","family":"Wei","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4138-3082","authenticated-orcid":false,"given":"Ling","family":"Liu","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,1,16]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2021.108135"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2004.04.005"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2019.2945116"},{"issue":"2","key":"e_1_3_1_5_2","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman Leo","year":"1996","unstructured":"Leo Breiman. 1996. Bagging predictors. Machine Learning 24, 2 (1996), 123\u2013140.","journal-title":"Machine Learning"},{"key":"e_1_3_1_6_2","first-page":"5","volume-title":"Proceedings of the Machine Learning","author":"Breiman Leo","year":"2001","unstructured":"Leo Breiman. 2001. Random forests. In Proceedings of the Machine Learning. 5\u201332."},{"issue":"3","key":"e_1_3_1_7_2","first-page":"801","article-title":"Arcing classifier (with discussion and a rejoinder by the author)","volume":"26","author":"Breiman Leo","year":"1998","unstructured":"Leo Breiman. 1998. Arcing classifier (with discussion and a rejoinder by the author). The Annals of Statistics 26, 3 (1998), 801\u2013849.","journal-title":"The Annals of Statistics"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015432"},{"key":"e_1_3_1_9_2","doi-asserted-by":"crossref","first-page":"2703","DOI":"10.1145\/3447548.3467121","volume-title":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.","author":"Chow Ka-Ho","year":"2021","unstructured":"Ka-Ho Chow and Ling Liu. 2021. Robust object detection fusion against deception. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.Association for Computing Machinery, New York, NY, 2703\u20132713. DOI:10.1145\/3447548.3467121"},{"key":"e_1_3_1_10_2","doi-asserted-by":"crossref","first-page":"1282","DOI":"10.1109\/BigData47090.2019.9006090","volume-title":"Proceedings of the 2019 IEEE International Conference on Big Data","author":"Chow Ka-Ho","year":"2019","unstructured":"Ka-Ho Chow, Wenqi Wei, Yanzhao Wu, and Ling Liu. 2019. Denoising and verification cross-layer ensemble against black-box adversarial attacks. In Proceedings of the 2019 IEEE International Conference on Big Data. 1282\u20131291. DOI:10.1109\/BigData47090.2019.9006090"},{"key":"e_1_3_1_11_2","unstructured":"Stanislav Fort Huiyi Hu and Balaji Lakshminarayanan. 2020. Deep Ensembles: A Loss Landscape Perspective. arXiv:1912.02757. Retrieved from https:\/\/arxiv.org\/abs\/1912.02757"},{"key":"e_1_3_1_12_2","doi-asserted-by":"crossref","first-page":"1614","DOI":"10.1145\/3394486.3403212","volume-title":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.","author":"Hong Shenda","year":"2020","unstructured":"Shenda Hong, Yanbo Xu, Alind Khare, Satria Priambada, Kevin Maher, Alaa Aljiffry, Jimeng Sun, and Alexey Tumanov. 2020. HOLMES: Health online model ensemble serving for deep learning models in intensive care units. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.Association for Computing Machinery, New York, NY, 1614\u20131624. DOI:10.1145\/3394486.3403212"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10845-020-01687-7"},{"key":"e_1_3_1_14_2","article-title":"The relative performance of ensemble methods with deep convolutional neural networks for image classification","volume":"45","author":"Ju Cheng","year":"2018","unstructured":"Cheng Ju, Aur\u00e9lien Bibaut, and Mark Laan. 2018. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics 45, 15(2018), 2800\u20132818.","journal-title":"Journal of Applied Statistics"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10845-019-01502-y"},{"key":"e_1_3_1_16_2","first-page":"275","volume-title":"Proceedings of the 13th International Conference on International Conference on Machine Learning.","author":"Kohavi Ron","year":"1996","unstructured":"Ron Kohavi and David Wolpert. 1996. Bias plus variance decomposition for zero-one loss functions. In Proceedings of the 13th International Conference on International Conference on Machine Learning.Morgan Kaufmann Publishers Inc., San Francisco, CA, 275\u2013283."},{"key":"e_1_3_1_17_2","volume-title":"Learning Multiple Layers of Features from Tiny Images","author":"Krizhevsky Alex","year":"2009","unstructured":"Alex Krizhevsky and Geoffrey Hinton. 2009. Learning Multiple Layers of Features from Tiny Images. Technical Report. University of Toronto."},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022859003006"},{"key":"e_1_3_1_19_2","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1109\/IJCNN.2001.939461","volume-title":"IJCNN\u201901. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222)","author":"Lazarevic A.","year":"2001","unstructured":"A. Lazarevic and Z. Obradovic. 2001. Effective pruning of neural network classifier ensembles. In IJCNN\u201901. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222)796\u2013801."},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_3_1_21_2","first-page":"274","volume-title":"Proceedings of the 2019 IEEE 16th International Conference on Mobile Ad Hoc and Sensor Systems","author":"Liu L.","year":"2019","unstructured":"L. Liu, W. Wei, K. Chow, M. Loper, E. Gursoy, S. Truex, and Y. Wu. 2019. Deep neural network ensembles against deception: Ensemble diversity, accuracy and robustness. In Proceedings of the 2019 IEEE 16th International Conference on Mobile Ad Hoc and Sensor Systems. 274\u2013282."},{"key":"e_1_3_1_22_2","volume-title":"Learning to Rank for Information Retrieval","author":"Liu Tie-Yan","year":"2011","unstructured":"Tie-Yan Liu. 2011. Learning to Rank for Information Retrieval. Springer."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2019.2947145"},{"key":"e_1_3_1_24_2","first-page":"496","volume-title":"Proceedings of the 20th International Conference on International Conference on Machine Learning.","author":"Lu Qing","year":"2003","unstructured":"Qing Lu and Lise Getoor. 2003. Link-based classification. In Proceedings of the 20th International Conference on International Conference on Machine Learning.AAAI Press, 496\u2013503."},{"issue":"2","key":"e_1_3_1_25_2","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1109\/TPAMI.2008.78","article-title":"An analysis of ensemble pruning techniques based on ordered aggregation","volume":"31","author":"Mart\u00ednez-Mu\u00f1oz G.","year":"2009","unstructured":"G. Mart\u00ednez-Mu\u00f1oz, D. Hern\u00e1ndez-Lobato, and A. Su\u00e1rez. 2009. An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2 (2009), 245\u2013259.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"3","key":"e_1_3_1_26_2","first-page":"276\u2014282","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"McHugh Mary L.","year":"2012","unstructured":"Mary L. McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276\u2014282. Retrieved from https:\/\/europepmc.org\/articles\/PMC3900052","journal-title":"Biochemia medica"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVT.2019.2933232"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(97)00023-2"},{"issue":"4","key":"e_1_3_1_29_2","first-page":"301","article-title":"Analysis of different norms and corresponding Lipschitz constants for global optimization","volume":"12","author":"Paulavi\u010dius Remigijus","year":"2006","unstructured":"Remigijus Paulavi\u010dius and Julius \u017dilinskas. 2006. Analysis of different norms and corresponding Lipschitz constants for global optimization. Ukio Technologinis ir Ekonominis Vystymas 12, 4 (2006), 301\u2013306.","journal-title":"Ukio Technologinis ir Ekonominis Vystymas"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_1_31_2","first-page":"120","volume-title":"Proceedings of the American Association for Arti Intelligence, AAAI-96, Integrating Multiple Learned Models Workshop","author":"Skalak David B.","year":"1996","unstructured":"David B. Skalak. 1996. The sources of increased accuracy for two proposed boosting algorithms. In Proceedings of the American Association for Arti Intelligence, AAAI-96, Integrating Multiple Learned Models Workshop. 120\u2013125."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-006-9449-2"},{"key":"e_1_3_1_33_2","first-page":"1","volume-title":"An Ensemble Pruning Primer","author":"Tsoumakas Grigorios","year":"2009","unstructured":"Grigorios Tsoumakas, Ioannis Partalas, and Ioannis Vlahavas. 2009. An Ensemble Pruning Primer. Springer, Berlin, 1\u201313."},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1080\/095400996116839"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2020.3024660"},{"key":"e_1_3_1_36_2","first-page":"456","volume-title":"Proceedings of the 2020 International Conference on Computing, Networking and Communications","author":"Wei W.","year":"2020","unstructured":"W. Wei, L. Liu, M. Loper, K. Chow, E. Gursoy, S. Truex, and Y. Wu. 2020. Cross-layer strategic ensemble defense against adversarial examples. In Proceedings of the 2020 International Conference on Computing, Networking and Communications. 456\u2013460."},{"key":"e_1_3_1_37_2","volume-title":"Proceedings of the 2023 IEEE International Conference on Data Mining","author":"Wu Yanzhao","year":"2023","unstructured":"Yanzhao Wu, Ka-Ho Chow, Wenqi Wei, and Ling Liu. 2023. Exploring model learning heterogeneity for boosting ensemble robustness. In Proceedings of the 2023 IEEE International Conference on Data Mining."},{"key":"e_1_3_1_38_2","first-page":"1433","volume-title":"Proceedings of the 2021 IEEE International Conference on Data Mining","author":"Wu Yanzhao","year":"2021","unstructured":"Yanzhao Wu and Ling Liu. 2021. Boosting deep ensemble performance with hierarchical pruning. In Proceedings of the 2021 IEEE International Conference on Data Mining. 1433\u20131438. DOI:10.1109\/ICDM51629.2021.00184"},{"key":"e_1_3_1_39_2","first-page":"208","volume-title":"Proceedings of the 2020 IEEE 2nd International Conference on Cognitive Machine Intelligence","author":"Wu Yanzhao","year":"2020","unstructured":"Yanzhao Wu, Ling Liu, Zhongwei Xie, Juhyun Bae, Ka-Ho Chow, and Wenqi Wei. 2020. Promoting high diversity ensemble learning with ensemblebench. In Proceedings of the 2020 IEEE 2nd International Conference on Cognitive Machine Intelligence. 208\u2013217. DOI:10.1109\/CogMI50398.2020.00034"},{"key":"e_1_3_1_40_2","first-page":"16464\u201316472","volume-title":"Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wu Yanzhao","year":"2021","unstructured":"Yanzhao Wu, Ling Liu, Zhongwei Xie, Ka-Ho Chow, and Wenqi Wei. 2021. Boosting ensemble accuracy by revisiting ensemble diversity metrics. In Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 16464\u201316472. DOI:10.1109\/CVPR46437.2021.01620"},{"key":"e_1_3_1_41_2","unstructured":"Xu-Cheng Yin Chun Yang and Hong-Wei Hao. 2014. Learning to Diversify via Weighted Kernels for Classifier Ensemble. arXiv:1406.1167. Retrieved from https:\/\/arxiv.org\/abs\/1406.1167"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3633286","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3633286","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:01Z","timestamp":1750287241000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3633286"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,16]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2,29]]}},"alternative-id":["10.1145\/3633286"],"URL":"https:\/\/doi.org\/10.1145\/3633286","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,16]]},"assertion":[{"value":"2022-07-18","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-10-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}