{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:00:34Z","timestamp":1760230834499,"version":"build-2065373602"},"reference-count":58,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T00:00:00Z","timestamp":1660608000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62176084","62176083","PA2021GDSK0093","PA2022GDSK0068"],"award-info":[{"award-number":["62176084","62176083","PA2021GDSK0093","PA2022GDSK0068"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities of China","doi-asserted-by":"publisher","award":["62176084","62176083","PA2021GDSK0093","PA2022GDSK0068"],"award-info":[{"award-number":["62176084","62176083","PA2021GDSK0093","PA2022GDSK0068"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>A high-quality annotated text corpus is vital when training a deep learning model. However, it is insurmountable to acquire absolute abundant label-balanced data because of the huge labor and time costs needed in the labeling stage. To alleviate this situation, a novel active learning (AL) method is proposed in this paper, which is designed to scratch samples to construct multi-class and multi-label Chinese emotional text corpora. This work shrewdly leverages the superiorities, i.e., less learning time and generating parameters randomly possessed by extreme learning machines (ELMs), to initially measure textual emotion features. In addition, we designed a novel combined query strategy called an asymmetric sampler (which simultaneously considers uncertainty and representativeness) to verify and extract ideal samples. Furthermore, this model progressively modulates state-of-the-art prescriptions through cross-entropy, Kullback\u2013Leibler, and Earth Mover\u2019s distance. Finally, through stepwise-assessing the experimental results, the updated corpora present more enriched label distributions and have a higher weight of correlative emotional information. Likewise, in emotion classification experiments by ELM, the precision, recall, and F1 scores obtained 7.17%, 6.31%, and 6.71% improvements, respectively. Extensive emotion classification experiments were conducted by two widely used classifiers\u2014SVM and LR\u2014and their results also prove our method\u2019s effectiveness in scratch emotional texts through comparisons.<\/jats:p>","DOI":"10.3390\/sym14081698","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T03:15:27Z","timestamp":1660706127000},"page":"1698","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["ELM-Based Active Learning via Asymmetric Samplers: Constructing a Multi-Class Text Corpus for Emotion Classification"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1514-9265","authenticated-orcid":false,"given":"Xuefeng","family":"Shi","sequence":"first","affiliation":[{"name":"School of Computer and Information, Hefei University of Technology, Hefei 230601, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2122-0240","authenticated-orcid":false,"given":"Min","family":"Hu","sequence":"additional","affiliation":[{"name":"School of Computer and Information, Hefei University of Technology, Hefei 230601, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4860-9184","authenticated-orcid":false,"given":"Fuji","family":"Ren","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 610056, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0783-5487","authenticated-orcid":false,"given":"Piao","family":"Shi","sequence":"additional","affiliation":[{"name":"School of Computer and Information, Hefei University of Technology, Hefei 230601, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9750-7032","authenticated-orcid":false,"given":"Xiao","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Computer and Information, Hefei University of Technology, Hefei 230601, China"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,16]]},"reference":[{"key":"ref_1","unstructured":"Deng, J., and Ren, F. (2020). Multi-label Emotion Detection via Emotion-Specified Feature Extraction and Emotion Correlation Learning. IEEE Trans. Affect. Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"65988","DOI":"10.1109\/ACCESS.2021.3074953","article-title":"Tweet-based bot detection using big data analytics","volume":"9","author":"Derhab","year":"2021","journal-title":"IEEE Access"},{"key":"ref_3","unstructured":"Feng, S., Tan, Z., Li, R., and Luo, M. (March, January 22). Heterogeneity-aware twitter bot detection with relational graph transformers. Proceedings of the AAAI Conference on Artificial Intelligence, online."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"107074","DOI":"10.1016\/j.knosys.2021.107074","article-title":"Efficient on-the-fly Web bot detection","volume":"223","author":"Suchacka","year":"2021","journal-title":"Knowl.-Based Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"68917","DOI":"10.1109\/ACCESS.2021.3074819","article-title":"Automated machine learning approaches for emergency response and coordination via social media in the aftermath of a disaster: A review","volume":"9","author":"Dwarakanath","year":"2021","journal-title":"IEEE Access"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"45137","DOI":"10.1109\/ACCESS.2021.3066365","article-title":"Artificial intelligence and internet of things enabled disease diagnosis model for smart healthcare systems","volume":"9","author":"Mansour","year":"2021","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1900","DOI":"10.1109\/TCAD.2018.2864251","article-title":"Data efficient lithography modeling with transfer learning and active data selection","volume":"38","author":"Lin","year":"2018","journal-title":"IEEE Trans.-Comput.-Aided Des. Integr. Circuits Syst."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Yan, Y., and Huang, S.J. (2018, January 13\u201319). Cost-Effective Active Learning for Hierarchical Multi-Label Classification. Proceedings of the IJCAI, Stockholm, Sweden.","DOI":"10.24963\/ijcai.2018\/411"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Yoo, D., and Kweon, I.S. (2019, January 16\u201317). Learning loss for active learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00018"},{"key":"ref_10","unstructured":"Le, Q., and Mikolov, T. (2014, January 21\u201326). Distributed representations of sentences and documents. Proceedings of the International Conference on Machine Learning, Beijing, China."},{"key":"ref_11","first-page":"1499","article-title":"MII: A novel text classification model combining deep active learning with BERT","volume":"63","author":"Zhang","year":"2020","journal-title":"Comput. Mater. Contin."},{"key":"ref_12","first-page":"649","article-title":"Character-level convolutional networks for text classification","volume":"28","author":"Zhang","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_13","first-page":"1","article-title":"Deep active learning for classifying cancer pathology reports","volume":"22","author":"Gao","year":"2021","journal-title":"BMC Bioinform."},{"key":"ref_14","unstructured":"Dor, L.E., Halfon, A., Gera, A., Shnarch, E., Dankin, L., Choshen, L., Danilevsky, M., Aharonov, R., Katz, Y., and Slonim, N. (2020, January 1). Active learning for BERT: An empirical study. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), online."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"14637","DOI":"10.1007\/s11042-020-10371-0","article-title":"Q-learning and LSTM based deep active learning strategy for malware defense in industrial IoT applications","volume":"80","author":"Khowaja","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1016\/j.neucom.2005.12.126","article-title":"Extreme learning machine: Theory and applications","volume":"70","author":"Huang","year":"2006","journal-title":"Neurocomputing"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s10994-021-06003-9","article-title":"How to measure uncertainty in uncertainty sampling for active learning","volume":"111","author":"Nguyen","year":"2022","journal-title":"Mach. Learn."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Huang, S.J., and Zhou, Z.H. (2013, January 7\u201310). Active query driven by uncertainty and diversity for incremental multi-label learning. Proceedings of the 2013 IEEE 13th International Conference on Data Mining, Dallas, TX, USA.","DOI":"10.1109\/ICDM.2013.74"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/TCYB.2015.2496974","article-title":"Exploring representativeness and informativeness for active learning","volume":"47","author":"Du","year":"2015","journal-title":"IEEE Trans. Cybern."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.patrec.2020.11.019","article-title":"Pool-based unsupervised active learning for regression using iterative representativeness-diversity maximization (iRDM)","volume":"142","author":"Liu","year":"2021","journal-title":"Pattern Recognit. Lett."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kang, X., Wu, Y., and Ren, F. (2018, January 18\u201320). Progressively improving supervised emotion classification through active learning. Proceedings of the International Conference on Multi-disciplinary Trends in Artificial Intelligence, Hanoi, Vietnam.","DOI":"10.1007\/978-3-030-03014-8_4"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yao, L., Huang, H., Wang, K.W., Chen, S.H., and Xiong, Q. (2020). Fine-grained mechanical Chinese named entity recognition based on ALBERT-AttBiLSTM-CRF and transfer learning. Symmetry, 12.","DOI":"10.3390\/sym12121986"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"106960","DOI":"10.1016\/j.compeleceng.2020.106960","article-title":"Prediction of COVID-19-pneumonia based on selected deep features and one class kernel extreme learning machine","volume":"90","author":"Khan","year":"2021","journal-title":"Comput. Electr. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3045","DOI":"10.1109\/TNNLS.2016.2607757","article-title":"Evolutionary cost-sensitive extreme learning machine","volume":"28","author":"Zhang","year":"2016","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1109\/MCOM.2017.1600425CM","article-title":"Smart health solution integrating IoT and cloud: A case study of voice pathology monitoring","volume":"55","author":"Muhammad","year":"2017","journal-title":"IEEE Commun. Mag."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1109\/MCOM.2018.1700790","article-title":"Edge computing with cloud for voice disorder assessment and treatment","volume":"56","author":"Muhammad","year":"2018","journal-title":"IEEE Commun. Mag."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1016\/j.neucom.2014.01.064","article-title":"An ELM-based model for affective analogical reasoning","volume":"149","author":"Cambria","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/MCI.2016.2572540","article-title":"Statistical learning theory and ELM for big social data analysis","volume":"11","author":"Oneto","year":"2016","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1016\/j.neucom.2012.02.041","article-title":"EEG-based vigilance estimation using extreme learning machines","volume":"102","author":"Shi","year":"2013","journal-title":"Neurocomputing"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1007\/s11517-015-1351-2","article-title":"Hierarchical multi-class SVM with ELM kernel for epileptic EEG signal classification","volume":"54","author":"Murugavel","year":"2016","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2885","DOI":"10.1109\/TCYB.2015.2492468","article-title":"Extreme learning machine with subnetwork hidden nodes for regression and classification","volume":"46","author":"Yang","year":"2015","journal-title":"IEEE Trans. Cybern."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.neucom.2015.03.110","article-title":"Deep extreme learning machines: Supervised autoencoding architecture for classification","volume":"174","author":"Tissera","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.neunet.2015.09.003","article-title":"A fast SVD-Hidden-nodes based extreme learning machine for large-scale data Analytics","volume":"77","author":"Deng","year":"2016","journal-title":"Neural Netw."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.neucom.2013.05.051","article-title":"Boosting weighted ELM for imbalanced learning","volume":"128","author":"Li","year":"2014","journal-title":"Neurocomputing"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"097296","DOI":"10.1117\/1.JRS.9.097296","article-title":"Optimizing extreme learning machine for hyperspectral image classification","volume":"9","author":"Li","year":"2015","journal-title":"J. Appl. Remote. Sens."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Huang, S.J., Chen, J.L., Mu, X., and Zhou, Z.H. (2017, January 20). Cost-Effective Active Learning from Diverse Labelers. Proceedings of the IJCAI, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/261"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Neutatz, F., Mahdavi, M., and Abedjan, Z. (2019, January 3\u20137). ED2: A case for active learning in error detection. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.","DOI":"10.1145\/3357384.3358129"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Cai, J.J., Tang, J., Chen, Q.G., Hu, Y., Wang, X., and Huang, S.J. (2019, January 11\u201312). Multi-View Active Learning for Video Recommendation. Proceedings of the IJCAI, Macao, China.","DOI":"10.24963\/ijcai.2019\/284"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"105140","DOI":"10.1016\/j.knosys.2019.105140","article-title":"Active learning through label error statistical methods","volume":"189","author":"Wang","year":"2020","journal-title":"Knowl.-Based Syst."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"616","DOI":"10.5465\/amr.2010.0456","article-title":"Venturing for others with heart and head: How compassion encourages social entrepreneurship","volume":"37","author":"Miller","year":"2012","journal-title":"Acad. Manag. Rev."},{"key":"ref_41","unstructured":"Plutchik, R., and Kellerman, H. (2013). Theories of Emotion, Academic Press."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1111\/j.1467-8640.2012.00460.x","article-title":"Crowdsourcing a word-emotion association lexicon","volume":"29","author":"Mohammad","year":"2013","journal-title":"Comput. Intell."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1016\/j.csl.2010.02.002","article-title":"A blog emotion corpus for emotional expression analysis in Chinese","volume":"24","author":"Quan","year":"2010","journal-title":"Comput. Speech Lang."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/s10799-012-0138-5","article-title":"Linguistic-based emotion analysis and recognition for measuring consumer satisfaction: An application of affective computing","volume":"13","author":"Ren","year":"2012","journal-title":"Inf. Technol. Manag."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1016\/j.csl.2012.07.012","article-title":"Employing hierarchical Bayesian networks in simple and complex emotion topic analysis","volume":"27","author":"Ren","year":"2013","journal-title":"Comput. Speech Lang."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1016\/j.csl.2013.04.010","article-title":"Automatically annotating a five-billion-word corpus of Japanese blogs for sentiment and affect analysis","volume":"28","author":"Ptaszynski","year":"2014","journal-title":"Comput. Speech Lang."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1080\/09540091.2013.851172","article-title":"Sentiment analysis of Chinese microblogging based on sentiment ontology: A case study of \u20187.23 Wenzhou Train Collision\u2019","volume":"25","author":"Shi","year":"2013","journal-title":"Connect. Sci."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"231","DOI":"10.2501\/IJMR-2014-014","article-title":"Sentiment analysis: A market-relevant and reliable measure of public feeling?","volume":"56","author":"Gunter","year":"2014","journal-title":"Int. J. Mark. Res."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.neucom.2012.01.040","article-title":"Parallel extreme learning machine for regression based on MapReduce","volume":"102","author":"He","year":"2013","journal-title":"Neurocomputing"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Tan, Y., Yang, L., Hu, Q., and Du, Z. (2019, January 3\u20137). Batch mode active learning for semantic segmentation based on multi-clue sample selection. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.","DOI":"10.1145\/3357384.3357968"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Guo, Y., Ding, G., Gao, Y., and Han, J. (2017, January 4\u20139). Active learning with cross-class similarity transfer. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10712"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Jamshidpour, N., Aria, E.H., Safari, A., and Homayouni, S. (2019, January 24\u201326). Adaptive Self-Learned Active Learning Framework for Hyperspectral Classification. Proceedings of the 2019 10th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), Amsterdam, The Netherlands.","DOI":"10.1109\/WHISPERS.2019.8921298"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1023\/A:1026543900054","article-title":"The earth mover\u2019s distance as a metric for image retrieval","volume":"40","author":"Rubner","year":"2000","journal-title":"Int. J. Comput. Vis."},{"key":"ref_54","unstructured":"Kang, X., Shi, X., Wu, Y., and Ren, F. (2020). Active learning with complementary sampling for instructing class-biased multi-label text emotion classification. IEEE Trans. Affect. Comput., 1."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Li, Y., Lv, Y., Wang, S., Liang, J., Li, J., and Li, X. (2019). Cooperative hybrid semi-supervised learning for text sentiment classification. Symmetry, 11.","DOI":"10.3390\/sym11020133"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Sarker, I.H., Abushark, Y.B., Alsolami, F., and Khan, A.I. (2020). Intrudtree: A machine learning based cyber security intrusion detection model. Symmetry, 12.","DOI":"10.20944\/preprints202004.0481.v1"},{"key":"ref_57","unstructured":"Zhu, S., Li, S., Chen, Y., and Zhou, G. (2016, January 11\u201316). Corpus fusion for emotion classification. Proceedings of the COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Iosifidis, A., Tefas, A., and Pitas, I. (2014, January 4\u20139). Minimum variance extreme learning machine for human action recognition. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.","DOI":"10.1109\/ICASSP.2014.6854640"}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/8\/1698\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:09:15Z","timestamp":1760141355000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/8\/1698"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,16]]},"references-count":58,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["sym14081698"],"URL":"https:\/\/doi.org\/10.3390\/sym14081698","relation":{},"ISSN":["2073-8994"],"issn-type":[{"type":"electronic","value":"2073-8994"}],"subject":[],"published":{"date-parts":[[2022,8,16]]}}}