{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T06:57:48Z","timestamp":1777705068056,"version":"3.51.4"},"reference-count":13,"publisher":"SAGE Publications","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IFS"],"published-print":{"date-parts":[[2024,1,10]]},"abstract":"<jats:p>Text classification is an important tasks in natural language processing. Multilayer attention networks have achieved excellent performance in text classification tasks, but they also face challenges such as high temporal and spatial complexity levels and low-rank bottleneck problems. This paper incorporates spatial attention into a neural network architecture that utilizes fewer encoder layers. The proposed model aims to enhance the spatial information of semantic features while addressing the high temporal and spatial demands of traditional multilayer attention networks. This approach utilizes spatial attention to selectively weigh the relevance of the spatial locations in the input feature maps, thereby enabling the model to focus on the most informative regions while ignoring the less important regions. By incorporating spatial attention into a shallower encoder network, the proposed model achieves improved performance on spatially oriented tasks while reducing the computational overhead associated with deeper attention-based models. To alleviate the low-rank bottleneck problem of multihead attention, this paper proposes a variable multihead attention mechanism, which changes the number of attention heads in a layer-by-layer manner with the encoder, achieving a balance between expression power and computational efficiency. We use two Chinese text classification datasets and an English sentiment classification dataset to verify the effectiveness of the proposed model.<\/jats:p>","DOI":"10.3233\/jifs-231368","type":"journal-article","created":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T11:01:30Z","timestamp":1700564490000},"page":"1443-1454","source":"Crossref","is-referenced-by-count":5,"title":["BVMHA: Text classification model with variable multihead hybrid attention based on BERT"],"prefix":"10.1177","volume":"46","author":[{"given":"Bo","family":"Peng","sequence":"first","affiliation":[{"name":"School of Software, XinJiang University, Urumqi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tao","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Software, XinJiang University, Urumqi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kundong","family":"Han","sequence":"additional","affiliation":[{"name":"School of Software, XinJiang University, Urumqi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhe","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Software, XinJiang University, Urumqi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuquan","family":"Ma","sequence":"additional","affiliation":[{"name":"Zhuzhou Times Electronic Technology Co. Ltd., Zhuzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mengnan","family":"Ma","sequence":"additional","affiliation":[{"name":"School of Software, XinJiang University, Urumqi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/JIFS-231368_ref1","doi-asserted-by":"crossref","first-page":"117","DOI":"10.28945\/4066","article-title":"Text classificationtechniques: A literature review","volume":"13","author":"Muthuraman Thangaraj","year":"2018","journal-title":"Interdisciplinary Journal ofInformation, Knowledge, and Management"},{"issue":"1","key":"10.3233\/JIFS-231368_ref2","first-page":"325","article-title":"A comparative review on deep learning models fortext classification","volume":"19","author":"Muhammad Zulqarnain","year":"2020","journal-title":"Indones. J. Electr. Eng. Comput. Sci"},{"key":"10.3233\/JIFS-231368_ref3","first-page":"41","article-title":"An empirical study of the naive bayes classifier","volume":"3","author":"Irina Rish","year":"2001","journal-title":"In IJCAI 2001 Workshop on Empirical Methods in ArtificialIntelligence"},{"issue":"4","key":"10.3233\/JIFS-231368_ref5","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/5254.708428","article-title":"Support vector machines","volume":"13","author":"Marti Hearst","year":"1998","journal-title":"IEEE IntelligentSystems and their Applications"},{"issue":"8","key":"10.3233\/JIFS-231368_ref7","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Sepp Hochreiter","year":"1997","journal-title":"Neural Computation"},{"issue":"10","key":"10.3233\/JIFS-231368_ref8","doi-asserted-by":"crossref","first-page":"2451","DOI":"10.1162\/089976600300015015","article-title":"Learning toforget: Continual prediction with lstm","volume":"12","author":"Felix Gers","year":"2000","journal-title":"Neural Computation"},{"key":"10.3233\/JIFS-231368_ref9","doi-asserted-by":"crossref","unstructured":"Jiuxiang Gu , Zhenhua Wang , Jason Kuen , Lianyang Ma , Amir Shahroudy , Bing Shuai , Ting Liu , Xingxing Wang, , Gang Wang , Jianfei Cai, et al. Recent advances in convolutional neural networks, Pattern Recognition, 77 (2018), 354\u2013377.","DOI":"10.1016\/j.patcog.2017.10.013"},{"key":"10.3233\/JIFS-231368_ref15","unstructured":"Ashish Vaswani , Noam Shazeer, , Niki Parmar, , Jakob Uszkoreit, , LlionJones, , Aidan Gomez, N , \u0141ukasz Kaiser , Illia Polosukhin, , Attention is all you need, Advances in Neural InformationProcessing Systems 30 (2017)."},{"key":"10.3233\/JIFS-231368_ref20","unstructured":"Zhilin Yang , Zihang Dai , Yiming Yang , Jaime Carbonell , Russ RSalakhutdinov and Quoc Le V , Xlnet: Generalized autoregressivepretraining for language understanding, Advances in NeuralInformation Processing Systems 32 (2019)."},{"key":"10.3233\/JIFS-231368_ref22","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.spc.2018.10.001","article-title":"Value-supply chain analysis (vsca) of crude palmoil production in brazil, focusing on economic, environmental andsocial sustainability","volume":"17","author":"Mohan Munasinghe","year":"2019","journal-title":"Sustainable Production and Consumption"},{"issue":"9","key":"10.3233\/JIFS-231368_ref23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: A systematicsurvey of prompting methods in natural language processing","volume":"55","author":"Pengfei Liu","year":"2023","journal-title":"ACMComputing Surveys"},{"issue":"2020","key":"10.3233\/JIFS-231368_ref25","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Tom Brown,","journal-title":"Advances in Neural Information Processing Systems"},{"key":"10.3233\/JIFS-231368_ref30","doi-asserted-by":"crossref","first-page":"8968","DOI":"10.1609\/aaai.v34i05.6428","article-title":"Ernie 2.0: A continual pretraining framework forlanguage understanding. In","volume":"34","author":"Yu Sun","year":"2020","journal-title":"Proceedings of the AAAI conferenceon artificial intelligence"}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/JIFS-231368","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:42:49Z","timestamp":1777455769000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/JIFS-231368"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,10]]},"references-count":13,"journal-issue":{"issue":"1"},"URL":"https:\/\/doi.org\/10.3233\/jifs-231368","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,10]]}}}