{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:47:26Z","timestamp":1760240846309,"version":"build-2065373602"},"reference-count":67,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T00:00:00Z","timestamp":1570665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Operational Program Human Resources Development, Education and Lifelong Learning","award":["80537"],"award-info":[{"award-number":["80537"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>One of the major aspects affecting the performance of the classification algorithms is the amount of labeled data which is available during the training phase. It is widely accepted that the labeling procedure of vast amounts of data is both expensive and time-consuming since it requires the employment of human expertise. For a wide variety of scientific fields, unlabeled examples are easy to collect but hard to handle in a useful manner, thus improving the contained information for a subject dataset. In this context, a variety of learning methods have been studied in the literature aiming to efficiently utilize the vast amounts of unlabeled data during the learning process. The most common approaches tackle problems of this kind by individually applying active learning or semi-supervised learning methods. In this work, a combination of active learning and semi-supervised learning methods is proposed, under a common self-training scheme, in order to efficiently utilize the available unlabeled data. The effective and robust metrics of the entropy and the distribution of probabilities of the unlabeled set, to select the most sufficient unlabeled examples for the augmentation of the initial labeled set, are used. The superiority of the proposed scheme is validated by comparing it against the base approaches of supervised, semi-supervised, and active learning in the wide range of fifty-five benchmark datasets.<\/jats:p>","DOI":"10.3390\/e21100988","type":"journal-article","created":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T03:07:11Z","timestamp":1570763231000},"page":"988","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Combination of Active Learning and Semi-Supervised Learning under a Self-Training Scheme"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7687-2380","authenticated-orcid":false,"given":"Nikos","family":"Fazakis","sequence":"first","affiliation":[{"name":"Wired Communications Lab, Department of Electrical and Computer Engineering, University of Patras, 26504 Achaia, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vasileios G.","family":"Kanas","sequence":"additional","affiliation":[{"name":"Wired Communications Lab, Department of Electrical and Computer Engineering, University of Patras, 26504 Achaia, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christos K.","family":"Aridas","sequence":"additional","affiliation":[{"name":"Educational Software Development Lab, Department of Mathematics, University of Patras, 26504 Achaia, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5307-6186","authenticated-orcid":false,"given":"Stamatis","family":"Karlos","sequence":"additional","affiliation":[{"name":"Educational Software Development Lab, Department of Mathematics, University of Patras, 26504 Achaia, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2247-3082","authenticated-orcid":false,"given":"Sotiris","family":"Kotsiantis","sequence":"additional","affiliation":[{"name":"Educational Software Development Lab, Department of Mathematics, University of Patras, 26504 Achaia, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Rosenberg, C., Hebert, M., and Schneiderman, H. (2005, January 5\u20137). Semi-supervised self-training of object detection models. Proceedings of the Seventh IEEE Workshop on Applications of Computer Vision (WACV 2005), Breckenridge, CO, USA.","DOI":"10.1109\/ACVMOT.2005.107"},{"key":"ref_2","first-page":"651","article-title":"Speech Recognition Combining MFCCs and Image Features","volume":"Volume 9811 LNCS","author":"Karlos","year":"2016","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tsukada, M., Washio, T., and Motoda, H. (2001). Automatic Web-Page Classification by Using Machine Learning Methods. Web Intelligence: Research and Development, Springer.","DOI":"10.1007\/3-540-45490-X_36"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1186\/s13040-016-0116-2","article-title":"MISSEL: A method to identify a large number of small species-specific genomic subsequences and its application to viruses classification","volume":"9","author":"Fiscon","year":"2016","journal-title":"BioData Min."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.cmpb.2017.03.006","article-title":"A novel method and software for automatically classifying Alzheimer\u2019s disease patients by magnetic resonance imaging analysis","volume":"143","author":"Previtali","year":"2017","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/j.bdr.2018.02.005","article-title":"Classification of Large DNA Methylation Datasets for Identifying Cancer Drivers","volume":"13","author":"Celli","year":"2018","journal-title":"Big Data Res."},{"key":"ref_7","unstructured":"Settles, B. Active Learning Literature Survey."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/s10115-013-0706-y","article-title":"Self-labeled techniques for semi-supervised learning: Taxonomy, software and empirical study","volume":"42","author":"Triguero","year":"2015","journal-title":"Knowl. Inf. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1850024","DOI":"10.1142\/S0218213018500240","article-title":"Omni-Ensemble Learning (OEL): Utilizing Over-Bagging, Static and Dynamic Ensemble Selection Approaches for Software Defect Prediction","volume":"27","author":"Mousavi","year":"2018","journal-title":"Int. J. Artif. Intell. Tools"},{"key":"ref_10","first-page":"1","article-title":"A Comparison Study on Rule Extraction from Neural Network Ensembles, Boosted Shallow Trees, and SVMs","volume":"2018","author":"Bologna","year":"2018","journal-title":"Appl. Comput. Intell. Soft Comput."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.ins.2015.04.003","article-title":"Combination of active learning and self-training for cross-lingual sentiment classification with density analysis of unlabelled samples","volume":"317","author":"Hajmohammadi","year":"2015","journal-title":"Inf. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ahsan, M.N.I., Nahian, T., Kafi, A.A., Hossain, M.I., and Shah, F.M. (2016, January 13\u201315). Review spam detection using active learning. Proceedings of the IEEE 2016 7th IEEE Annual Information Technology, Electronics and Mobile Communication Conference, Vancouver, BC, Canada.","DOI":"10.1109\/IEMCON.2016.7746279"},{"key":"ref_13","unstructured":"Xu, J., Fumera, G., Roli, F., and Zhou, Z. (2009, January 16\u201317). Training spamassassin with active semi-supervised learning. Proceedings of the 6th Conference on Email and Anti-Spam (CEAS\u201909), Mountain View, CA, USA."},{"key":"ref_14","unstructured":"Dua, D., and Graff, C. (2019, October 09). UCI Machine Learning Repository. Available online: https:\/\/archive.ics.uci.edu\/ml\/citation_policy.html."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sourati, J., Akcakaya, M., Dy, J., Leen, T., and Erdogmus, D. (2016). Classification Active Learning Based on Mutual Information. Entropy, 18.","DOI":"10.3390\/e18020051"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1936","DOI":"10.1109\/TPAMI.2014.2307881","article-title":"Active Learning by Querying Informative and Representative Examples","volume":"36","author":"Huang","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lewis, D.D., and Gale, W.A. (1994, January 3\u20136). A Sequential Algorithm for Training Text Classifiers. Proceedings of the ACM SIGIR Forum, Dublin, Ireland.","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1109\/TSA.2005.848882","article-title":"Active learning: Theory and applications to automatic speech recognition","volume":"13","author":"Riccardi","year":"2005","journal-title":"IEEE Trans. Speech Audio Process."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, Z., and Schuller, B. (2012, January 9\u201313). Active Learning by Sparse Instance Tracking and Classifier Confidence in Acoustic Emotion Recognition. Proceedings of the Interspeech 2012, Portland, OR, USA.","DOI":"10.21437\/Interspeech.2012-117"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Roma, G., Janer, J., and Herrera, P. (2012, January 5\u20138). Active learning of custom sound taxonomies in unstructured audio data. Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, Hong Kong, China.","DOI":"10.1145\/2324796.2324872"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1845","DOI":"10.1016\/S0167-8655(03)00008-4","article-title":"Learning with progressive transductive support vector machine","volume":"24","author":"Chen","year":"2003","journal-title":"Pattern Recognit. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1109\/TIT.2007.911294","article-title":"Graph-based semi-supervised learning and spectral kernel design","volume":"54","author":"Johnson","year":"2008","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2322","DOI":"10.1109\/TIT.2018.2879897","article-title":"A Sampling Theory Perspective of Graph-Based Semi-Supervised Learning","volume":"65","author":"Anis","year":"2019","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1109\/TPAMI.2007.70765","article-title":"Graph-based semisupervised learning","volume":"30","author":"Culp","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Blum, A., and Mitchell, T. (1998, January 24\u201326). Combining Labeled and Unlabeled Data with Co-Training. Proceedings of the Eleventh Annual Conference on Computational Learning Theory, Madison, WI, USA.","DOI":"10.1145\/279943.279962"},{"key":"ref_27","unstructured":"McCallum, A.K., Nigam, K., McCallumzy, A.K., and Nigamy, K. (1998, January 24\u201327). Employing EM and pool-based active learning for text classification. Proceedings of the Fifteenth International Conference on Machine Learning, Madison, WI, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.specom.2004.08.002","article-title":"Combining active and semi-supervised learning for spoken language understanding","volume":"45","author":"Tur","year":"2005","journal-title":"Speech Commun."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Tomanek, K., and Hahn, U. (2009, January 2\u20137). Semi-supervised active learning for sequence labeling. Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2, Singapore.","DOI":"10.3115\/1690219.1690291"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Han, W., Coutinho, E., Ruan, H., Li, H., Schuller, B., Yu, X., and Zhu, X. (2016). Semi-supervised active learning for sound classification in hybrid learning environments. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0162075"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"13009","DOI":"10.1038\/s41598-018-31395-5","article-title":"A novel logistic regression model combining semi-supervised learning and active learning for disease classification","volume":"8","author":"Chai","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"762","DOI":"10.1109\/TMI.2015.2494582","article-title":"Interactive Cell Segmentation Based on Active and Semi-Supervised Learning","volume":"35","author":"Su","year":"2016","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.cogsys.2017.05.006","article-title":"Active and semi-supervised learning for object detection with imperfect data","volume":"45","author":"Rhee","year":"2017","journal-title":"Cogn. Syst. Res."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yang, Y., and Loog, M. (2016, January 4\u20138). Active learning using uncertainty information. Proceedings of the International Conference on Pattern Recognition, Cancun, Mexico.","DOI":"10.1109\/ICPR.2016.7900034"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"711","DOI":"10.3233\/JIFS-152641","article-title":"Self-trained Rotation Forest for semi-supervised learning","volume":"32","author":"Fazakis","year":"2017","journal-title":"J. Intell. Fuzzy Syst."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1016\/j.patcog.2018.06.004","article-title":"A benchmark and comparison of active learning for logistic regression","volume":"83","author":"Yang","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1080\/02331887808801414","article-title":"Cross-validation: A review","volume":"9","author":"Stone","year":"1978","journal-title":"Ser. Stat."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1007\/BF00993309","article-title":"C4.5: Programs for Machine Learning by J. Ross Quinlan. Morgan Kaufmann Publishers, Inc., 1993","volume":"16","author":"Salzberg","year":"1994","journal-title":"Mach. Learn."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1007\/BF00153759","article-title":"Instance-Based Learning Algorithms","volume":"6","author":"Aha","year":"1991","journal-title":"Mach. Learn."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"191","DOI":"10.2307\/2347628","article-title":"Van Ridge Estimators in Logistic Regression","volume":"41","author":"Houwelingen","year":"1992","journal-title":"Appl. Stat."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1007\/s10994-005-0466-3","article-title":"Logistic model trees","volume":"59","author":"Landwehr","year":"2005","journal-title":"Mach. Learn."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1214\/aos\/1016218223","article-title":"Additive logistic regression: A statistical view of boosting","volume":"28","author":"Friedman","year":"2000","journal-title":"Ann. Stat."},{"key":"ref_44","first-page":"771","article-title":"A Short Introduction to Boosting","volume":"14","author":"Schapire","year":"1999","journal-title":"J. Jpn. Soc. Artif. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random Forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1613\/jair.614","article-title":"Popular Ensemble Methods: An Empirical Study","volume":"11","author":"Opitz","year":"1999","journal-title":"J. Artif. Intell. Res."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1619","DOI":"10.1109\/TPAMI.2006.211","article-title":"Rotation forest: A new classifier ensemble method","volume":"28","author":"Rodriguez","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_48","unstructured":"Han, J., Kamber, M., and Pei, J. (2011). Data Mining: Concepts and Techniques, Elsevier."},{"key":"ref_49","unstructured":"Chen, T., and Guestrin, C. (2019, October 09). XGBoost: Reliable Large-scale Tree Boosting System. Available online: http:\/\/learningsys.org\/papers\/LearningSys_2015_paper_32.pdf."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Ferreira, A.J., and Figueiredo, M.A.T. (2012). Boosting algorithms: A review of methods, theory, and applications. Ensemble Machine Learning: Methods and Applications, Springer.","DOI":"10.1007\/978-1-4419-9326-7_2"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1080\/01621459.1937.10503522","article-title":"The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance","volume":"32","author":"Friedman","year":"1937","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_52","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm","year":"1979","journal-title":"Scand. J. Stat."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"758","DOI":"10.1016\/j.patrec.2019.07.022","article-title":"A multi-scheme semi-supervised regression approach","volume":"125","author":"Fazakis","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Culotta, A., and McCallum, A. (2005, January 9\u201313). Reducing labeling effort for structured prediction tasks. Proceedings of the National Conference on Artificial Intelligence, Pittsburgh, PA, USA.","DOI":"10.21236\/ADA440382"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Scheffer, T., Decomain, C., and Wrobel, S. (2001). Active hidden markov models for information extraction. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer.","DOI":"10.1007\/3-540-44816-0_31"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"426","DOI":"10.1016\/j.neucom.2014.06.042","article-title":"Active learning via query synthesis and nearest neighbour search","volume":"147","author":"Wang","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"711","DOI":"10.3233\/JIFS-181237","article-title":"Graph-based semisupervised and manifold learning for image retrieval with SVM-based relevant feedback","volume":"37","author":"Huu","year":"2019","journal-title":"J. Intell. Fuzzy Syst."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1109\/LSP.2011.2162583","article-title":"An efficient switching median filter based on local outlier factor","volume":"18","author":"Wang","year":"2011","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Liu, F.T., Ting, K.M., and Zhou, Z.-H. (2008, January 15\u201319). Isolation forest. Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy.","DOI":"10.1109\/ICDM.2008.17"},{"key":"ref_60","unstructured":"Tang, J., Alelyani, S., and Liu, H. (2014). Feature selection for classification: A review. Data Classification: Algorithms and Applications, CRC Press."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Hulten, G., Spencer, L., and Domingos, P. (2001, January 26\u201329). Mining time-changing data streams. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and data Mining KDD \u201901, San Francisco, CA, USA.","DOI":"10.1145\/502512.502529"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s10107-010-0420-4","article-title":"Pegasos: Primal estimated sub-gradient solver for SVM","volume":"127","author":"Singer","year":"2011","journal-title":"Math. Program."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.neucom.2016.12.038","article-title":"A survey of deep neural network architectures and their applications","volume":"234","author":"Liu","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"1550033","DOI":"10.1142\/S0218213015500335","article-title":"A Neural Network Ensemble Classifier for Effective Intrusion Detection Using Fuzzy Clustering and Radial Basis Function Networks","volume":"25","author":"Amini","year":"2016","journal-title":"Int. J. Artif. Intell. Tools"},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Elreedy, D., Atiya, A.F., and Shaheen, S.I. (2019). A Novel Active Learning Regression Framework for Balancing the Exploration-Exploitation Trade-Off. Entropy, 21.","DOI":"10.3390\/e21070651"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Fazakis, N., Kostopoulos, G., Karlos, S., Kotsiantis, S., and Sgarbas, K. (2020). An Active Learning Ensemble Method for Regression Tasks. Intell. Data Anal., 24.","DOI":"10.3233\/IDA-194608"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/1656274.1656278","article-title":"The WEKA data mining software","volume":"11","author":"Hall","year":"2009","journal-title":"ACM SIGKDD Explor. Newsl."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/10\/988\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:25:00Z","timestamp":1760189100000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/10\/988"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,10]]},"references-count":67,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["e21100988"],"URL":"https:\/\/doi.org\/10.3390\/e21100988","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2019,10,10]]}}}