{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,22]],"date-time":"2025-12-22T18:36:47Z","timestamp":1766428607353,"version":"build-2065373602"},"reference-count":48,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,4,23]],"date-time":"2022-04-23T00:00:00Z","timestamp":1650672000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>At present, short text classification is a hot topic in the area of natural language processing. Due to the sparseness and irregularity of short text, the task of short text classification still faces great challenges. In this paper, we propose a new classification model from the aspects of short text representation, global feature extraction and local feature extraction. We use convolutional networks to extract shallow features from short text vectorization, and introduce a multi-level semantic extraction framework. It uses BiLSTM as the encoding layer while the attention mechanism and normalization are used as the interaction layer. Finally, we concatenate the convolution feature vector and semantic results of the semantic framework. After several rounds of feature integration, the framework improves the quality of the feature representation. Combined with the capsule network, we obtain high-level local information by dynamic routing and then squash them. In addition, we explore the optimal depth of semantic feature extraction for short text based on a multi-level semantic framework. We utilized four benchmark datasets to demonstrate that our model provides comparable results. The experimental results show that the accuracy of SUBJ, TREC, MR and ProcCons are 93.8%, 91.94%, 82.81% and 98.43%, respectively, which verifies that our model has greatly improves classification accuracy and model robustness.<\/jats:p>","DOI":"10.3390\/e24050590","type":"journal-article","created":{"date-parts":[[2022,4,23]],"date-time":"2022-04-23T08:14:06Z","timestamp":1650701646000},"page":"590","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Investigating Multi-Level Semantic Extraction with Squash Capsules for Short Text Classification"],"prefix":"10.3390","volume":"24","author":[{"given":"Jing","family":"Li","sequence":"first","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Beijing Key Laboratory of Knowledge Engineering for Materials Science, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3456-5259","authenticated-orcid":false,"given":"Dezheng","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Beijing Key Laboratory of Knowledge Engineering for Materials Science, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aziguli","family":"Wulamu","sequence":"additional","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China"},{"name":"Beijing Key Laboratory of Knowledge Engineering for Materials Science, University of Science and Technology Beijing, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Shi, T., Kang, K., Choo, J., and Reddy, C.K. (2018, January 23\u201327). Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. Proceedings of the 2018 World Wide Web Conference, Lyon, France.","DOI":"10.1145\/3178876.3186009"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1427","DOI":"10.1109\/TKDE.2020.2992485","article-title":"Short text topic modeling techniques, applications, and performance: A survey","volume":"34","author":"Qiang","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1016\/j.neucom.2020.01.064","article-title":"Siamese capsule networks with global and local features for text classification","volume":"390","author":"Wu","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Tian, Y.W., and Zhang, Z. (2021, January 1\u20134). Text classification model based on BERT-capsule with integrated deep learning. Proceedings of the 2021 IEEE 16th Conference on Industrial Electronics and Applications, Chengdu, China.","DOI":"10.1109\/ICIEA51954.2021.9516041"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.neucom.2019.08.080","article-title":"Incorporating context-relevant concepts into convolutional neural networks for short text classification","volume":"386","author":"Xu","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"166578","DOI":"10.1109\/ACCESS.2019.2953918","article-title":"Improving term weighting schemes for short text classification in vector space model","volume":"7","author":"Samant","year":"2017","journal-title":"IEEE Access"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"92120","DOI":"10.1109\/ACCESS.2020.2994450","article-title":"A hybrid classification method via character embedding in Chinese short text with few words","volume":"8","author":"Zhu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hu, Y.B., Li, Y., Yang, T., and Pan, Q. (2018, January 18\u201321). Short text classification with a convolutional neural networks based method. Proceedings of the 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore.","DOI":"10.1109\/ICARCV.2018.8581332"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Edo-Osagie, O., Lake, I., Edeghere, O., and Iglesia, B.D.L. (2019, January 12\u201314). Attention-based recurrent neural networks (RNNs) for short text classification: An application in public health monitoring. Proceedings of the International Work-Conference on Artificial Neural Networks, Gran Canaria, Spain.","DOI":"10.1007\/978-3-030-20521-8_73"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"38706","DOI":"10.1109\/ACCESS.2021.3063234","article-title":"Neural network-based tree translation for knowledge base construction","volume":"9","author":"Zhang","year":"2021","journal-title":"IEEE Access"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3450352","article-title":"HGAT: Heterogeneous graph attention networks for semi-supervised short text classification","volume":"39","author":"Yang","year":"2021","journal-title":"ACM Trans. Inf. Syst."},{"key":"ref_12","first-page":"1","article-title":"Multiple weak supervision for short text classification","volume":"1","author":"Chen","year":"2022","journal-title":"Appl. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"101268","DOI":"10.1016\/j.csl.2021.101268","article-title":"Combining context-relevant features with multi-stage attention network for short text classification","volume":"71","author":"Liu","year":"2022","journal-title":"Comput. Speech Lang."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"101104","DOI":"10.1016\/j.csl.2020.101104","article-title":"tax2vec: Constructing interpretable features from taxonomies for short text classification","volume":"65","author":"Martinc","year":"2021","journal-title":"Comput. Speech Lang."},{"key":"ref_15","first-page":"53","article-title":"Label oriented hierarchical attention neural network for short text classification","volume":"5","author":"Feng","year":"2022","journal-title":"Acad. J. Eng. Technol. Sci."},{"key":"ref_16","first-page":"1","article-title":"Mutual attention inception network for remote sensing visual question answering","volume":"60","author":"Zheng","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ye, Y., Ren, X., Zhu, B., Tang, T., Tan, X., Gui, Y., and Yao, Q. (2022). An adaptive attention fusion mechanism convolutional network for object detection in remote sensing images. Remote Sens., 14.","DOI":"10.3390\/rs14030516"},{"key":"ref_18","unstructured":"Sabour, S., Frosst, N., and Hinton, G.E. (2017, January 4\u20139). Dynamic routing between capsules. Proceedings of the 2017 Conference, Advances in Neural Information Processing Systems, Los Angeles, CA, USA."},{"key":"ref_19","unstructured":"Zhao, W., Ye, J.B., Yang, M., Lei, Z.Y., Zhang, S.F., and Zhao, Z. (October, January 4). Investigating capsule networks with dynamic routing for text classification. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"e831","DOI":"10.7717\/peerj-cs.831","article-title":"Attention enhanced capsule network for text classification by encoding syntactic dependency trees with graph convolutional neural network","volume":"7","author":"Jia","year":"2022","journal-title":"PeerJ Comput. Sci."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s42979-021-00963-4","article-title":"A novel BGCapsule network for text classification","volume":"3","author":"Gangwar","year":"2022","journal-title":"SN Comput. Sci."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Du, C.N., Sun, H.F., Wang, J.Y., Qi, Q., Liao, J.X., Xu, T., and Liu, M. (2019, January 3\u20137). Capsule network with interactive attention for aspect-level sentiment classification. Proceedings of the 2019 conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Hong Kong, China.","DOI":"10.18653\/v1\/D19-1551"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zheng, W.S., Zheng, Z.B., Wan, H., and Chen, C. (2019, January 10\u201316). Dynamically route hierarchical structure representation to attentive capsule for text classification. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, Macau, China.","DOI":"10.24963\/ijcai.2019\/759"},{"key":"ref_24","unstructured":"Chen, Z., and Qian, T.Y. (August, January 28). Transfer capsule network for aspect level sentiment classification. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"39321","DOI":"10.1109\/ACCESS.2019.2906398","article-title":"A novel capsule based hybrid neural network for sentiment classification","volume":"7","author":"Du","year":"2019","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"58284","DOI":"10.1109\/ACCESS.2018.2874623","article-title":"Cross-domain sentiment classification by capsule network with semantic rules","volume":"6","author":"Zhang","year":"2018","journal-title":"IEEE Access"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1839","DOI":"10.1007\/s00521-019-04620-z","article-title":"Deep refinement: Capsule network with attention mechanism-based system for text classification","volume":"32","author":"Jain","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1016\/j.neucom.2019.10.033","article-title":"Text classification using capsules","volume":"376","author":"Kim","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"111036","DOI":"10.1016\/j.jss.2021.111036","article-title":"SeCNN: A semantic CNN parser for code comment generation","volume":"181","author":"Zheng","year":"2021","journal-title":"J. Syst. Softw."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.isprsjprs.2020.12.010","article-title":"Review on Convolutional Neural Networks (CNN) in vegetation remote sensing","volume":"173","author":"Kattenborn","year":"2021","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Rahman, S., and Chakraborty, P. (2020, January 4\u20135). Bangla document classification using deep recurrent neural network with BiLSTM. Proceedings of the International Conference on Machine Intelligence and Data Science Applications, Dehradun, India.","DOI":"10.1007\/978-981-33-4087-9_43"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1016\/j.ins.2020.10.021","article-title":"Fusion of heterogeneous attention mechanisms in multi-view convolutional neural network for text classification","volume":"548","author":"Liang","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_33","first-page":"1295","article-title":"Capsule network\u2014A survey","volume":"34","author":"Mensah","year":"2022","journal-title":"J. King Saud Univ.-Comput. Inf. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Pang, B., and Lee, L. (2004, January 21\u201326). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the Association for Computational Linguistics, Barcelona, Spain.","DOI":"10.3115\/1218955.1218990"},{"key":"ref_35","unstructured":"Li, X., and Roth, D. (September,, January 24). Learning question classifiers. Proceedings of the 19th International Conference on Computational Linguistics, Taipei, Taiwan."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Pang, B., Lee, L., and Vaithyanathan, S. (2002, January 6\u20137). Thumbs up? Sentiment classification using machine learning techniques. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Philadelphia, PA, USA.","DOI":"10.3115\/1118693.1118704"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Kim, Y. (2014, January 25\u201329). Convolutional neural networks for sentence classification. Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing, Doha, Qatar.","DOI":"10.3115\/v1\/D14-1181"},{"key":"ref_38","unstructured":"Qian, Q., Huang, M., Lei, J., and Zhu, X. (August, January 30). Linguistically regularized lstms for sentiment classification. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, QC, Canada."},{"key":"ref_39","unstructured":"Miyato, T., Dai, A.M., and Goodfellow, L. (2017, January 24\u201326). Adversarial training methods for semi-supervised text classification. Proceedings of the International Conference on Learning Representations, Toulon, France."},{"key":"ref_40","unstructured":"Shen, T., Zhou, T.Y., Long, G.D., Jiang, J., and Zhang, C.Q. (May, January 30). Bi-directional block self-attention for fast and memory-efficient sequence modeling. Proceedings of the International Conference and Learning Representations, Vancouver, QC, Canada."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3052770","article-title":"Encoding syntactic knowledge in neural networks for sentiment classification","volume":"35","author":"Huang","year":"2017","journal-title":"ACM Trans. Inf. Syst."},{"key":"ref_42","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, U., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Li, X., Li, Z., and Xie, H. (2021, January 2\u20139). Merging statistical feature via adaptive gate for improved text classification. Proceedings of the AAAI Conference on Artificial Intelligence, Shenzhen, China.","DOI":"10.1609\/aaai.v35i15.17569"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Yu, X., Cui, Z., Wu, S., Wen, Z., and Wang, L. (2020). Every document owns its structure: Inductive text classification via graph neural networks. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.31"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Lu, Z., Du, P., and Nie, J.Y. (2020, January 14\u201317). VGCN-BERT: Augmenting BERT with graph embedding for text classification. Proceedings of the 42nd European Conference on IR Research, Lisbon, Portugal.","DOI":"10.1007\/978-3-030-45439-5_25"},{"key":"ref_46","unstructured":"Yang, T., Hu, L., Shi, C., Ji, H., Li, X., and Nie, L. (2019, January 3\u20137). Heterogeneous graph attention networks for semi-supervised short text classification. Proceedings of the Empirical Methods in Natural Language Processing, Hong Kong, China."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"11947","DOI":"10.1109\/ACCESS.2022.3146303","article-title":"Graph convolutional network based on multi-head pooling for short text classification","volume":"10","author":"Zhao","year":"2022","journal-title":"IEEE Access"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"6865287","DOI":"10.1155\/2021\/6865287","article-title":"Short text paraphrase identification model based on RDN-MESIM","volume":"2021","author":"Li","year":"2021","journal-title":"Comput. Intell. Neurosci."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/5\/590\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:59:19Z","timestamp":1760137159000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/5\/590"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,23]]},"references-count":48,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["e24050590"],"URL":"https:\/\/doi.org\/10.3390\/e24050590","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2022,4,23]]}}}