{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:46:41Z","timestamp":1760147201640,"version":"build-2065373602"},"reference-count":46,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,18]],"date-time":"2023-01-18T00:00:00Z","timestamp":1674000000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Importance sampling, a variant of online sampling, is often used in neural network training to improve the learning process, and, in particular, the convergence speed of the model. We study, here, the performance of a set of batch selection algorithms, namely, online sampling algorithms that process small parts of the dataset at each iteration. Convergence is accelerated through the creation of a bias towards the learning of hard samples. We first consider the baseline algorithm and investigate its performance in terms of convergence speed and generalization efficiency. The latter, however, is limited in case of poor balancing of data sets. To alleviate this shortcoming, we propose two variations of the algorithm that achieve better generalization and also manage to not undermine the convergence speed boost offered by the original algorithm. Various data transformation techniques were tested in conjunction with the proposed scheme to develop an overall training method of the model and to ensure robustness in different training environments. An experimental framework was constructed using three naturally imbalanced datasets and one artificially imbalanced one. The results assess the advantage in convergence of the extended algorithm over the vanilla one, but, mostly, show better generalization performance in imbalanced data environments.<\/jats:p>","DOI":"10.3390\/a16020065","type":"journal-article","created":{"date-parts":[[2023,1,19]],"date-time":"2023-01-19T05:06:14Z","timestamp":1674104774000},"page":"65","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Online Batch Selection for Enhanced Generalization in Imbalanced Datasets"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6153-7712","authenticated-orcid":false,"given":"George","family":"Ioannou","sequence":"first","affiliation":[{"name":"Artificial Intelligence & Learning Systems Laboratory, School of Electrical & Computer Engineering, National Technical University of Athens, 157 80 Zografou, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3611-8292","authenticated-orcid":false,"given":"Georgios","family":"Alexandridis","sequence":"additional","affiliation":[{"name":"Artificial Intelligence & Learning Systems Laboratory, School of Electrical & Computer Engineering, National Technical University of Athens, 157 80 Zografou, Greece"}]},{"given":"Andreas","family":"Stafylopatis","sequence":"additional","affiliation":[{"name":"Artificial Intelligence & Learning Systems Laboratory, School of Electrical & Computer Engineering, National Technical University of Athens, 157 80 Zografou, Greece"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"429","DOI":"10.3233\/IDA-2002-6504","article-title":"The class imbalance problem: A systematic study","volume":"6","author":"Japkowicz","year":"2002","journal-title":"Intell. Data Anal."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Guo, X., Yin, Y., Dong, C., Yang, G., and Zhou, G. (2008, January 18\u201320). On the Class Imbalance Problem. Proceedings of the 2008 Fourth International Conference on Natural Computation, Jinan, China.","DOI":"10.1109\/ICNC.2008.871"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ioannou, G., Tagaris, T., and Stafylopatis, A. (2019, January 26\u201328). Improving the Convergence Speed of Deep Neural Networks with Biased Sampling. Proceedings of the 3rd International Conference on Advances in Artificial Intelligence, ICAAI 2019, Istanbul, Turkey.","DOI":"10.1145\/3369114.3369116"},{"key":"ref_4","first-page":"321","article-title":"SMOTE: Synthetic Minority over-Sampling Technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Int. Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"238","DOI":"10.2307\/1403797","article-title":"Discriminatory Analysis. Nonparametric Discrimination: Consistency Properties","volume":"57","author":"Fix","year":"1989","journal-title":"Int. Stat. Rev.\/Rev. Int. Stat."},{"key":"ref_6","first-page":"878","article-title":"Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning","volume":"Volume 3644","author":"Huang","year":"2005","journal-title":"Advances in Intelligent Computing, Proceedings of the International Conference on Intelligent Computing, ICIC 2005, Hefei, China, 23\u201326 August 2005, Proceedings, Part I"},{"key":"ref_7","unstructured":"Last, F., Douzas, G., and Ba\u00e7\u00e3o, F. (2017). Oversampling for Imbalanced Learning Based on K-Means and SMOTE. arXiv."},{"key":"ref_8","unstructured":"He, H., Bai, Y., Garcia, E.A., and Li, S. (2008, January 1\u20136). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China."},{"key":"ref_9","unstructured":"Drummond, C., and Holte, R. (2003, January 21). C4.5, Class Imbalance, and Cost Sensitivity: Why Under-Sampling beats OverSampling. Proceedings of the ICML\u201903 Workshop on Learning from Imbalanced Datasets, Washington, DC, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1109\/TSMC.1972.4309137","article-title":"Asymptotic Properties of Nearest Neighbor Rules Using Edited Data","volume":"2","author":"Wilson","year":"1972","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"(1976). Two Modifications of CNN. IEEE Trans. Syst. Man Cybern., SMC-6, 769\u2013772.","DOI":"10.1109\/TSMC.1976.4309452"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5718","DOI":"10.1016\/j.eswa.2008.06.108","article-title":"Cluster-based under-sampling approaches for imbalanced data distributions","volume":"36","author":"Yen","year":"2009","journal-title":"Expert Syst. Appl."},{"key":"ref_13","unstructured":"Batista, G.E., Bazzan, A.L., and Monard, M.C. (2003, January 3\u20135). Balancing Training Data for Automated Annotation of Keywords: A Case Study. Proceedings of the II Brazilian Workshop on Bioinformatics, Maca\u00e9, RJ, Brazil."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1145\/1007730.1007735","article-title":"A study of the behavior of several methods for balancing machine learning training data","volume":"6","author":"Batista","year":"2004","journal-title":"SIGKDD Explor."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Shamsudin, H., Yusof, U.K., Jayalakshmi, A., and Akmal Khalid, M.N. (2020, January 9\u201311). Combining oversampling and undersampling techniques for imbalanced classification: A comparative study using credit card fraudulent transaction dataset. Proceedings of the 2020 IEEE 16th International Conference on Control Automation (ICCA), Singapore.","DOI":"10.1109\/ICCA51439.2020.9264517"},{"key":"ref_16","unstructured":"Elkan, C. (2001, January 4\u201310). The Foundations of Cost-Sensitive Learning. Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, IJCAI 2001, Seattle, WA, USA."},{"key":"ref_17","unstructured":"Kukar, M., and Kononenko, I. (1998, January 23\u201328). Cost-Sensitive Learning with Neural Networks. Proceedings of the 13th European Conference on Artificial Intelligence, Brighton, UK."},{"key":"ref_18","first-page":"41","article-title":"Curriculum learning","volume":"Volume 382","author":"Danyluk","year":"2009","journal-title":"Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009"},{"key":"ref_19","unstructured":"Graves, A., Bellemare, M.G., Menick, J., Munos, R., and Kavukcuoglu, K. (2017, January 6\u201311). Automated Curriculum Learning for Neural Networks. Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Shrivastava, A., Gupta, A., and Girshick, R.B. (2016, January 27\u201330). Training Region-Based Object Detectors with Online Hard Example Mining. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.89"},{"key":"ref_21","unstructured":"Loshchilov, I., and Hutter, F. (2015). Online Batch Selection for Faster Training of Neural Networks. arXiv."},{"key":"ref_22","unstructured":"Katharopoulos, A., and Fleuret, F. (2018, January 10\u201315). Not All Samples Are Created Equal: Deep Learning with Importance Sampling. Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm\u00e4ssan, Stockholm, Sweden."},{"key":"ref_23","unstructured":"Bouchard, G., Trouillon, T., Perez, J., and Gaidon, A. (2015). Accelerating Stochastic Gradient Descent via Online Learning to Sample. arXiv."},{"key":"ref_24","unstructured":"Zhao, P., and Zhang, T. (2014). Stochastic Optimization with Importance Sampling. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1109\/TNN.2007.912312","article-title":"Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model","volume":"19","author":"Bengio","year":"2008","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_26","unstructured":"Alain, G., Lamb, A., Sankar, C., Courville, A.C., and Bengio, Y. (2015). Variance Reduction in SGD by Distributed Importance Sampling. arXiv."},{"key":"ref_27","unstructured":"Chang, H., Learned-Miller, E.G., and McCallum, A. (2017, January 4\u20139). Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA."},{"key":"ref_28","unstructured":"Roh, Y., Lee, K., Whang, S.E., and Suh, C. (2021, January 3\u20137). FairBatch: Batch Selection for Model Fairness. Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Joseph, K.J., Vamshi Teja, R., Singh, K., and Balasubramanian, V.N. (2019, January 10\u201316). Submodular Batch Selection for Training Deep Neural Networks. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China.","DOI":"10.24963\/ijcai.2019\/372"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"106759","DOI":"10.1016\/j.asoc.2020.106759","article-title":"Deep transfer with minority data augmentation for imbalanced breast cancer dataset","volume":"97","author":"Saini","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_31","first-page":"68","article-title":"Improving Imbalanced Learning by Pre-finetuning with Data Augmentation","volume":"183","author":"Shi","year":"2022","journal-title":"Proc. Mach. Learn. Res."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"056017","DOI":"10.1088\/1741-2552\/abb5be","article-title":"EEG data augmentation: Towards class imbalance problem in sleep staging tasks","volume":"17","author":"Fan","year":"2020","journal-title":"J. Neural Eng."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/j.solener.2021.05.095","article-title":"Resampling and data augmentation for short-term PV output prediction based on an imbalanced sky images dataset using convolutional neural networks","volume":"224","author":"Nie","year":"2021","journal-title":"Sol. Energy"},{"key":"ref_34","first-page":"107","article-title":"SMOTEBoost: Improving Prediction of the Minority Class in Boosting","volume":"Volume 2838","author":"Lavrac","year":"2003","journal-title":"Knowledge Discovery in Databases: PKDD 2003, Proceedings of the 7th European Conference on Principles and Practice of Knowledge Discovery in Databases, Cavtat-Dubrovnik, Croatia, 22\u201326 September 2003, Proceedings"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1109\/TSMCA.2009.2029559","article-title":"RUSBoost: A Hybrid Approach to Alleviating Class Imbalance","volume":"40","author":"Seiffert","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part A"},{"key":"ref_36","first-page":"14","article-title":"Hybrid Sampling with Bagging for Class Imbalance Learning","volume":"Volume 9651","author":"Bailey","year":"2016","journal-title":"Advances in Knowledge Discovery and Data Mining, Proceedings of the 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, 19\u201322 April 2016, Proceedings, Part I"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1109\/TSMCB.2008.2007853","article-title":"Exploratory Undersampling for Class-Imbalance Learning","volume":"39","author":"Liu","year":"2009","journal-title":"IEEE Trans. Syst. Man Cybern. Part B"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"983","DOI":"10.1002\/(SICI)1097-024X(199708)27:8<983::AID-SPE117>3.0.CO;2-#","article-title":"Introspective Sorting and Selection Algorithms","volume":"27","author":"Musser","year":"1997","journal-title":"Softw. Pract. Exp."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on Image Data Augmentation for Deep Learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Cubuk, E.D., Zoph, B., Man\u00e9, D., Vasudevan, V., and Le, Q.V. (2019, January 16\u201320). AutoAugment: Learning Augmentation Strategies From Data. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00020"},{"key":"ref_42","unstructured":"Dua, D., and Graff, C. (2023, January 10). UCI Machine Learning Repository. Available online: http:\/\/archive.ics.uci."},{"key":"ref_43","unstructured":"LeCun, Y., and Cortes, C. (2023, January 10). MNIST Handwritten Digit Database. Available online: http:\/\/yann.lecun."},{"key":"ref_44","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA."},{"key":"ref_45","unstructured":"Nair, V., and Hinton, G.E. (2010, January 21\u201324). Rectified Linear Units Improve Restricted Boltzmann Machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.neunet.2018.07.011","article-title":"A systematic study of the class imbalance problem in convolutional neural networks","volume":"106","author":"Buda","year":"2018","journal-title":"Neural Netw."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/2\/65\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:09:24Z","timestamp":1760119764000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/2\/65"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,18]]},"references-count":46,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["a16020065"],"URL":"https:\/\/doi.org\/10.3390\/a16020065","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2023,1,18]]}}}