{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,27]],"date-time":"2025-11-27T02:58:50Z","timestamp":1764212330461,"version":"3.37.3"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T00:00:00Z","timestamp":1708560000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T00:00:00Z","timestamp":1708560000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Hochschule f\u00fcr Angewandte Wissenschaften Hamburg (HAW Hamburg)"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Optim Theory Appl"],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In recent years, an interest appeared in integrating various optimization algorithms in machine learning. We study the potential of ensemble learning in classification tasks and how to efficiently decompose the underlying optimization problem. Ensemble learning has become popular for machine learning applications and it is particularly interesting from an optimization perspective due to its resemblance to column generation. The challenge for learning is not only to obtain a good fit for the training data set, but also good generalization, such that the classifier is generally applicable. Deep networks have the drawback that they require a lot of computational effort to get to an accurate classification. Ensemble learning can combine various weak learners, which individually require less computational time. We consider binary classification problems studying a three-phase algorithm. After initializing a set of base learners refined by a bootstrapping approach, base learners are generated using the solution of an linear programming (LP) master problem and then solving a machine learning sub-problem regarding a reduced data set, which can be viewed as a so-called pricing problem. We theoretically show that the algorithm computes an optimal ensemble model in the convex hull of a given model space. The implementation of the algorithm is part of an ensemble learning framework called <jats:sc>decolearn<\/jats:sc>. Numerical experiments with CIFAR-10 data set show that the base learners are diverse and that both the training and generalization error are reduced after several iterations.\n<\/jats:p>","DOI":"10.1007\/s10957-024-02391-9","type":"journal-article","created":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T12:18:20Z","timestamp":1708604300000},"page":"1794-1819","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["On Optimizing Ensemble Models using Column Generation"],"prefix":"10.1007","volume":"203","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1068-9729","authenticated-orcid":false,"given":"Vanya","family":"Aziz","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ouyang","family":"Wu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ivo","family":"Nowak","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eligius M. T.","family":"Hendrix","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jan","family":"Kronqvist","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,2,22]]},"reference":[{"key":"2391_CR1","doi-asserted-by":"crossref","unstructured":"Aakerberg, A., Nasrollahi, K., Heder, T.: Improving a deep learning based rgb-d object recognition model by ensemble learning. In: 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1\u20136. IEEE (2017)","DOI":"10.1109\/IPTA.2017.8310101"},{"key":"2391_CR2","doi-asserted-by":"crossref","unstructured":"Abdollahi, A., Pradhan, B., Alamri, A.M.: An ensemble architecture of deep convolutional SEGNET and UNET networks for building semantic segmentation from high-resolution aerial images. Geocarto International pp. 1\u201316 (2020)","DOI":"10.1080\/10106049.2020.1856199"},{"issue":"2","key":"2391_CR3","doi-asserted-by":"publisher","first-page":"648","DOI":"10.1016\/j.ejor.2021.04.016","volume":"295","author":"S Ben\u00edtez-Pe\u00f1a","year":"2021","unstructured":"Ben\u00edtez-Pe\u00f1a, S., Carrizosa, E., Guerrero, V., Jim\u00e9nez-Gamero, M.D., Mart\u00edn-Barrag\u00e1n, B., Molero-R\u00edo, C., Ram\u00edrez-Cobo, P., Romero Morales, D., Sillero-Denamiel, M.R.: On sparse ensemble methods: an application to short-term predictions of the evolution of COVID-19. Eur. J. Oper. Res. 295(2), 648\u2013663 (2021)","journal-title":"Eur. J. Oper. Res."},{"key":"2391_CR4","unstructured":"Berrada, L., Zisserman, A., Kumar, M.P.: Deep Frank\u2013Wolfe for neural network optimization. (2018). arXiv:1811.07591"},{"issue":"9","key":"2391_CR5","doi-asserted-by":"publisher","first-page":"9059","DOI":"10.1109\/TCYB.2021.3053165","volume":"52","author":"Y Bian","year":"2022","unstructured":"Bian, Y., Chen, H.: When does diversity help generalization in classification ensembles? IEEE Trans. Cyber. 52(9), 9059\u20139075 (2022)","journal-title":"IEEE Trans. Cyber."},{"key":"2391_CR6","volume-title":"Classification and Regression Trees","author":"L Breiman","year":"1984","unstructured":"Breiman, L., Friedman, J., Stone, C.J., Olshen, R.: Classification and Regression Trees. Chapman and Hall\/CRC, Boca Raton (1984)"},{"issue":"1","key":"2391_CR7","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1023\/A:1012470815092","volume":"46","author":"A Demiriz","year":"2002","unstructured":"Demiriz, A., Bennett, K.P., Shawe-Taylor, J.: Linear programming boosting via column generation. Mach. Learn. 46(1), 225\u2013254 (2002)","journal-title":"Mach. Learn."},{"key":"2391_CR8","doi-asserted-by":"crossref","unstructured":"Dietterich, T.G.: Ensemble methods in machine learning. In: Multiple Classifier Systems, pp. 1\u201315. Springer Berlin Heidelberg, Berlin, Heidelberg (2000)","DOI":"10.1007\/3-540-45014-9_1"},{"issue":"4","key":"2391_CR9","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1007\/BF00344251","volume":"36","author":"K Fukushima","year":"1980","unstructured":"Fukushima, K.: Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36(4), 193\u2013202 (1980)","journal-title":"Biol. Cybern."},{"issue":"4","key":"2391_CR10","first-page":"429","volume":"8","author":"MHA Hijazi","year":"2019","unstructured":"Hijazi, M.H.A., Hwa, S.K.T., Bade, A., Yaakob, R., Jeffree, M.S.: Ensemble deep learning for tuberculosis detection using chest x-ray and canny edge detected images. IAES Int. J. AI 8(4), 429 (2019)","journal-title":"IAES Int. J. AI"},{"key":"2391_CR11","doi-asserted-by":"publisher","first-page":"61797","DOI":"10.1109\/ACCESS.2019.2915985","volume":"7","author":"H Jiang","year":"2019","unstructured":"Jiang, H., Qiu, X., Chen, J., Liu, X., Miao, X., Zhuang, S.: Insulator fault detection in aerial images based on ensemble learning with multi-level perception. IEEE Access 7, 61797\u201361810 (2019)","journal-title":"IEEE Access"},{"key":"2391_CR12","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/3897.001.0001","volume-title":"An Introduction to Computational Learning Theory","author":"MJ Kearns","year":"1994","unstructured":"Kearns, M.J., Vazirani, U.V.: An Introduction to Computational Learning Theory. MIT Press, Cambridge, MA, USA (1994)"},{"key":"2391_CR13","unstructured":"Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images. http:\/\/api.semanticscholar.org\/CorpusID:18268744 (2009)"},{"key":"2391_CR14","unstructured":"Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural Inform. Process. Syst. 30 (2017)"},{"issue":"4","key":"2391_CR15","doi-asserted-by":"publisher","first-page":"2833","DOI":"10.1109\/TII.2020.3008010","volume":"17","author":"Y Li","year":"2020","unstructured":"Li, Y., Song, Y., Jia, L., Gao, S., Li, Q., Qiu, M.: Intelligent fault diagnosis by fusing domain adversarial training and maximum mean discrepancy via ensemble learning. IEEE Trans. Industr. Inf. 17(4), 2833\u20132841 (2020)","journal-title":"IEEE Trans. Industr. Inf."},{"key":"2391_CR16","doi-asserted-by":"crossref","unstructured":"Lin, Y.Y., Liu, T.L., Fuh, C.S.: Local ensemble kernel learning for object category recognition. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1\u20138. IEEE (2007)","DOI":"10.1109\/CVPR.2007.383084"},{"key":"2391_CR17","doi-asserted-by":"crossref","unstructured":"Malisiewicz, T., Gupta, A., Efros, A.A.: Ensemble of exemplar-svms for object detection and beyond. In: 2011 International Conference on Computer Vision, pp. 89\u201396. IEEE (2011)","DOI":"10.1109\/ICCV.2011.6126229"},{"key":"2391_CR18","first-page":"317","volume":"24","author":"P Muts","year":"2023","unstructured":"Muts, P., Bruche, S., Nowak, I., Wu, O., Hendrix, E.M., Tsatsaronis, G.: A column generation algorithm for solving energy system planning problems. Optim. Eng. 24, 317\u2013351 (2023)","journal-title":"Optim. Eng."},{"key":"2391_CR19","doi-asserted-by":"crossref","unstructured":"Rokach, L.: Ensemble-based classifiers. AI Rev. 33, 1\u201339 (2010)","DOI":"10.1007\/s10462-009-9124-7"},{"key":"2391_CR20","doi-asserted-by":"crossref","unstructured":"Ruck, D.W., Rogers, S., Kabrisky, M., Mills, J.: Target recognition: Conventional and neural network approaches. In: International 1989 Joint Conference on Neural Networks, pp. 608\u2013vol. IEEE (1989)","DOI":"10.1109\/IJCNN.1989.118433"},{"issue":"4","key":"2391_CR21","volume":"8","author":"O Sagi","year":"2018","unstructured":"Sagi, O., Rokach, L.: Ensemble learning: a survey. Wiley Interdisciplin. Rev.: Data Min. Knowled. Discov. 8(4), e1249 (2018)","journal-title":"Wiley Interdisciplin. Rev.: Data Min. Knowled. Discov."},{"key":"2391_CR22","unstructured":"Simonyan, Z.: Very Deep Convolutional Networks for Large-Scale Image Recogniton. arXiv preprint, arXiv:1409.1556, (2014)"},{"key":"2391_CR23","doi-asserted-by":"publisher","DOI":"10.1016\/j.compchemeng.2021.107343","volume":"151","author":"A Thebelt","year":"2021","unstructured":"Thebelt, A., Kronqvist, J., Mistry, M., Lee, R.M., Sudermann-Merx, N., Misener, R.: Entmoot: a framework for optimization over ensemble tree models. Comput. Chem. Eng. 151, 107343 (2021)","journal-title":"Comput. Chem. Eng."},{"issue":"4","key":"2391_CR24","first-page":"515","volume":"17","author":"G T\u00fcys\u00fczo\u01e7lu","year":"2020","unstructured":"T\u00fcys\u00fczo\u01e7lu, G., Birant, D.: Enhanced bagging (Ebagging): a novel approach for ensemble learning. Int. Arab J. Inform. Technol. 17(4), 515\u2013528 (2020)","journal-title":"Int. Arab J. Inform. Technol."},{"key":"2391_CR25","doi-asserted-by":"publisher","first-page":"125662","DOI":"10.1109\/ACCESS.2019.2925566","volume":"7","author":"Y Wang","year":"2019","unstructured":"Wang, Y., Zhu, K., Sun, M., Deng, Y.: An ensemble learning approach for fault diagnosis in self-organizing heterogeneous networks. IEEE Access 7, 125662\u2013125675 (2019)","journal-title":"IEEE Access"},{"key":"2391_CR26","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2019.107098","volume":"99","author":"J Xu","year":"2020","unstructured":"Xu, J., Wang, W., Wang, H., Guo, J.: Multi-model ensemble with rich spatial information for object detection. Pattern Recognit. 99, 107098 (2020)","journal-title":"Pattern Recognit."},{"key":"2391_CR27","doi-asserted-by":"publisher","DOI":"10.1201\/b12207","volume-title":"Ensemble Methods: Foundations and Algorithms","author":"ZH Zhou","year":"2012","unstructured":"Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms. CRC Press, Boca Raton (2012)"}],"container-title":["Journal of Optimization Theory and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10957-024-02391-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10957-024-02391-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10957-024-02391-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,11]],"date-time":"2024-11-11T10:14:08Z","timestamp":1731320048000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10957-024-02391-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,22]]},"references-count":27,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,11]]}},"alternative-id":["2391"],"URL":"https:\/\/doi.org\/10.1007\/s10957-024-02391-9","relation":{},"ISSN":["0022-3239","1573-2878"],"issn-type":[{"type":"print","value":"0022-3239"},{"type":"electronic","value":"1573-2878"}],"subject":[],"published":{"date-parts":[[2024,2,22]]},"assertion":[{"value":"4 August 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 January 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 February 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}