{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T02:30:09Z","timestamp":1782873009169,"version":"3.54.5"},"reference-count":57,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,2,20]],"date-time":"2023-02-20T00:00:00Z","timestamp":1676851200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Nowadays, the solution to many practical problems relies on machine learning tools. However, compiling the appropriate training data set for real-world classification problems is challenging because collecting the right amount of data for each class is often difficult or even impossible. In such cases, we can easily face the problem of imbalanced learning. There are many methods in the literature for solving the imbalanced learning problem, so it has become a serious question how to compare the performance of the imbalanced learning methods. Inadequate validation techniques can provide misleading results (e.g., due to data shift), which leads to the development of methods designed for imbalanced data sets, such as stratified cross-validation (SCV) and distribution optimally balanced SCV (DOB-SCV). Previous studies have shown that higher classification performance scores (AUC) can be achieved on imbalanced data sets using DOB-SCV instead of SCV. We investigated the effect of the oversamplers on this difference. The study was conducted on 420 data sets, involving several sampling methods and the DTree, kNN, SVM, and MLP classifiers. We point out that DOB-SCV often provides a little higher F1 and AUC values for classification combined with sampling. However, the results also prove that the selection of the sampler\u2013classifier pair is more important for the classification performance than the choice between the DOB-SCV and the SCV techniques.<\/jats:p>","DOI":"10.3390\/s23042333","type":"journal-article","created":{"date-parts":[[2023,2,20]],"date-time":"2023-02-20T04:28:57Z","timestamp":1676867337000},"page":"2333","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":140,"title":["A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4121-4745","authenticated-orcid":false,"given":"Szilvia","family":"Szeghalmy","sequence":"first","affiliation":[{"name":"Faculty of Informatics, University of Debrecen, H-4028 Debrecen, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6893-3067","authenticated-orcid":false,"given":"Attila","family":"Fazekas","sequence":"additional","affiliation":[{"name":"Faculty of Informatics, University of Debrecen, H-4028 Debrecen, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"4139","DOI":"10.1007\/s11042-022-13434-6","article-title":"An efficient fraud detection framework with credit card imbalanced data in financial services","volume":"82","author":"Hemdan","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1080\/0952813X.2021.1907795","article-title":"Credit card fraud detection under extreme imbalanced data: A comparative study of data-level algorithms","volume":"34","author":"Singh","year":"2022","journal-title":"J. Exp. Theor. Artif. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1111\/coin.12452","article-title":"A comprehensive data-level investigation of cancer diagnosis on imbalanced data","volume":"38","author":"Gupta","year":"2022","journal-title":"Comput. Intell."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1504\/IJBRA.2020.109103","article-title":"A study of data pre-processing techniques for imbalanced biomedical data classification","volume":"16","author":"Liu","year":"2020","journal-title":"Int. J. Bioinform. Res. Appl."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"115492","DOI":"10.1016\/j.eswa.2021.115492","article-title":"A minority oversampling approach for fault detection with heterogeneous imbalanced data","volume":"184","author":"Liu","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"47491","DOI":"10.1109\/ACCESS.2021.3068316","article-title":"Effects of data augmentation method borderline-SMOTE on emotion recognition of EEG signals based on convolutional neural network","volume":"9","author":"Chen","year":"2021","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6914","DOI":"10.1049\/joe.2019.0557","article-title":"From Kinect skeleton data to hand gesture recognition with radar","volume":"2019","author":"Li","year":"2019","journal-title":"J. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"109363","DOI":"10.1016\/j.asoc.2022.109363","article-title":"A survey on unsupervised learning for wearable sensor-based activity recognition","volume":"127","author":"Ige","year":"2022","journal-title":"Appl. Soft Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"59192","DOI":"10.1109\/ACCESS.2018.2873502","article-title":"Sensor-based datasets for human activity recognition\u2014A systematic review of literature","volume":"6","author":"Quero","year":"2018","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Link, J., Perst, T., Stoeve, M., and Eskofier, B.M. (2022). Wearable sensors for activity recognition in ultimate frisbee using convolutional neural networks and transfer learning. Sensors, 22.","DOI":"10.3390\/s22072560"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Guglielmo, G., Blom, P.M., Klincewicz, M., \u010cule, B., and Spronck, P. (2022, January 21\u201324). Face in the game: Using facial action units to track expertise in competitive video game play. Proceedings of the 2022 IEEE Conference on Games (CoG), Beijing, China.","DOI":"10.1109\/CoG51982.2022.9893599"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1927","DOI":"10.1016\/j.neucom.2015.09.064","article-title":"Adaptive weighted imbalance learning with application to abnormal activity recognition","volume":"173","author":"Xingyu","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Zhang, J., Li, J., and Wang, W. (2021). A class-imbalanced deep learning fall detection algorithm using wearable sensors. Sensors, 21.","DOI":"10.3390\/s21196511"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"113026","DOI":"10.1016\/j.eswa.2019.113026","article-title":"Understanding the apparent superiority of over-sampling through an analysis of local information for class-imbalanced data","volume":"158","author":"Florencia","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez, A., Garc\u00eda, S., Galar, M., Prati, R.C., Krawczyk, B., and Herrera, F. (2018). Learning from Imbalanced Data Sets, Springer.","DOI":"10.1007\/978-3-319-98074-4"},{"key":"ref_16","unstructured":"Quinonero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N.D. (2022). Dataset Shift in Machine Learning, MIT Press."},{"key":"ref_17","unstructured":"Kohavi, R. (1995, January 20\u201325). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI\u201995), Montreal, QC, Canada."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Sammut, C., and Webb, G.I. (2011). Encyclopedia of Machine Learning, Springer.","DOI":"10.1007\/978-0-387-30164-8"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ins.2013.09.038","article-title":"On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed","volume":"257","author":"Herrera","year":"2014","journal-title":"Inf. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1304","DOI":"10.1109\/TNNLS.2012.2199516","article-title":"Study on the impact of partition-induced dataset shift on k-fold cross-validation","volume":"23","author":"Herrera","year":"2012","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1109\/TPAMI.2009.187","article-title":"Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation","volume":"32","author":"Rodriguez","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/095281300146272","article-title":"Distribution-balanced stratified cross-validation for accuracy estimation","volume":"12","author":"Zeng","year":"2000","journal-title":"J. Exp. Theor. Artif. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1109\/TIT.1967.1053964","article-title":"Nearest neighbor pattern classification","volume":"13","author":"Cover","year":"1967","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhou, Z.H. (2021). Machine Learning, Springer.","DOI":"10.1007\/978-981-15-1967-3"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1961189.1961199","article-title":"LIBSVM: A library for support vector machines","volume":"2","author":"Chang","year":"2011","journal-title":"ACM Trans. Intell. Syst. Technol. (TIST)"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/0925-2312(91)90023-5","article-title":"Multilayer perceptrons for classification and regression","volume":"2","author":"Murtagh","year":"1991","journal-title":"Neurocomputing"},{"key":"ref_27","unstructured":"Quinlan, J.R. (2014). C4.5: Programs for Machine Learning, Morgan Kaufmann."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"105662","DOI":"10.1016\/j.asoc.2019.105662","article-title":"An empirical comparison and evaluation of minority oversampling techniques on a large number of imbalanced datasets","volume":"83","year":"2019","journal-title":"Appl. Soft Comput."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1145\/1007730.1007735","article-title":"A study of the behavior of several methods for balancing machine learning training data","volume":"6","author":"Batista","year":"2004","journal-title":"ACM Sigkdd Explor. Newsl."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.ins.2014.08.051","article-title":"SMOTE\u2013IPF: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering","volume":"291","author":"Luengo","year":"2015","journal-title":"Inf. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Lee, J., Kim, N.R., and Lee, J.H. (2015, January 8\u201310). An over-sampling technique with rejection for imbalanced class learning. Proceedings of the 9th International Conference on Ubiquitous Information Management and Communication, Bali, Indonesia.","DOI":"10.1145\/2701126.2701181"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1515\/amcs-2017-0050","article-title":"CCR: A combined cleaning and resampling algorithm for imbalanced data classification","volume":"27","author":"Koziarski","year":"2017","journal-title":"Int. J. Appl. Math. Comput. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhou, B., Yang, C., Guo, H., and Hu, J. (2013, January 4\u20139). A quasi-linear SVM combined with assembled SMOTE for imbalanced data classification. Proceedings of the 2013 International Joint Conference on Neural Networks (IJCNN), Dallas, TX, USA.","DOI":"10.1109\/IJCNN.2013.6707035"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Barua, S., Islam, M., and Murase, K. (2013, January 14\u201317). ProWSyn: Proximity weighted synthetic oversampling technique for imbalanced data set learning. Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Gold Coast, Australia.","DOI":"10.1007\/978-3-642-37456-2_27"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Cao, Q., and Wang, S. (2011, January 26\u201327). Applying over-sampling technique based on data density and cost-sensitive SVM to imbalanced learning. Proceedings of the 2011 International Conference on Information Management, Innovation Management and Industrial Engineering, Shenzhen, China.","DOI":"10.1109\/ICIII.2011.276"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Nakamura, M., Kajiwara, Y., Otsuka, A., and Kimura, H. (2013). Lvq-smote\u2014learning vector quantization based synthetic minority over-sampling technique for biomedical data. Biodata Min., 6.","DOI":"10.1186\/1756-0381-6-16"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1016\/j.neucom.2019.06.100","article-title":"Smote-variants: A python implementation of 85 minority oversampling techniques","volume":"366","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_39","unstructured":"Ester, M., Kriegel, H.P., Sander, J., and Xu, X. (1996, January 2\u20134). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the KDD\u201996: Second International Conference on Knowledge Discovery and Data Mining, Portland, OR, USA."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Szeghalmy, S., and Fazekas, A. (2022). A Highly Adaptive Oversampling Approach to Address the Issue of Data Imbalance. Computers, 11.","DOI":"10.3390\/computers11050073"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2378","DOI":"10.1016\/j.fss.2007.12.023","article-title":"A study of the behaviour of linguistic fuzzy rule based classification systems in the framework of imbalanced data-sets","volume":"159","author":"Herrera","year":"2008","journal-title":"Fuzzy Sets Syst."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1016\/j.ijar.2008.11.004","article-title":"Hierarchical fuzzy rule based classification systems with genetic rule selection for imbalanced data-sets","volume":"50","author":"Herrera","year":"2009","journal-title":"Int. J. Approx. Reason."},{"key":"ref_43","unstructured":"(2022, December 18). Abalone. UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/abalone."},{"key":"ref_44","unstructured":"Nakai, K. (2022, December 18). Ecoli. UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/ecoli."},{"key":"ref_45","unstructured":"Ilter, N., and Guvenir, H. (2022, December 18). Dermatology. UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/dermatology."},{"key":"ref_46","unstructured":"(2022, December 18). Car Evaluation. UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/car+evaluation."},{"key":"ref_47","unstructured":"Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, J. (2022, December 18). Wine Quality. UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/wine+quality."},{"key":"ref_48","unstructured":"(2022, December 18). Statlog (Vehicle Silhouettes). UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/Statlog+%28Vehicle+Silhouettes%29."},{"key":"ref_49","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1145\/1882471.1882479","article-title":"Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement","volume":"12","author":"Forman","year":"2010","journal-title":"ACM Sigkdd Explor. Newsl."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Wardhani, N.W.S., Rochayani, M.Y., Iriany, A., Sulistyono, A.D., and Lestantyo, P. (2019, January 23\u201324). Cross-validation metrics for evaluating classification performance on imbalanced data. Proceedings of the 2019 international conference on computer, control, informatics and its applications (IC3INA), Tangerang, Indonesia.","DOI":"10.1109\/IC3INA48034.2019.8949568"},{"key":"ref_52","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","year":"2006","journal-title":"J. Mach. Learn. Res."},{"key":"ref_53","unstructured":"Nemenyi, P. (1963). Distribution-Free Multiple Comparisons, Princeton University."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Weaver, K.F., Morales, V., Dunn, S.L., Godde, K., and Weaver, P.F. (2017). An Introduction to Statistical Analysis in Research: With Applications in the Biological and Life Sciences, Wiley.","DOI":"10.1002\/9781119454205"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Gu, Q., Zhu, L., and Cai, Z. (2009, January 23\u201325). Evaluation measures of the classification performance of imbalanced data sets. Proceedings of the International Symposium on Intelligence Computation and Applications, Huangshi, China.","DOI":"10.1007\/978-3-642-04962-0_53"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"100071","DOI":"10.1016\/j.dajour.2022.100071","article-title":"A comparative analysis of K-Nearest Neighbour, Genetic, Support Vector Machine, Decision Tree, and Long Short Term Memory algorithms in machine learning","volume":"3","author":"Bansal","year":"2022","journal-title":"Decis. Anal. J."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Abdualgalil, B., and Abraham, S. (2020, January 24\u201325). Applications of machine learning algorithms and performance comparison: A review. Proceedings of the 2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE), Vellore, India.","DOI":"10.1109\/ic-ETITE47903.2020.490"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/2333\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:37:05Z","timestamp":1760121425000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/2333"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,20]]},"references-count":57,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["s23042333"],"URL":"https:\/\/doi.org\/10.3390\/s23042333","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,20]]}}}