{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T10:00:05Z","timestamp":1771668005724,"version":"3.50.1"},"reference-count":47,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,7,8]],"date-time":"2023-07-08T00:00:00Z","timestamp":1688774400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>The study focuses on news category prediction and investigates the performance of sentence embedding of four transformer models (BERT, RoBERTa, MPNet, and T5) and their variants as feature vectors when combined with Softmax and Random Forest using two accessible news datasets from Kaggle. The data are stratified into train and test sets to ensure equal representation of each category. Word embeddings are generated using transformer models, with the last hidden layer selected as the embedding. Mean pooling calculates a single vector representation called sentence embedding, capturing the overall meaning of the news article. The performance of Softmax and Random Forest, as well as the soft voting of both, is evaluated using evaluation measures such as accuracy, F1 score, precision, and recall. The study also contributes by evaluating the performance of Softmax and Random Forest individually. The macro-average F1 score is calculated to compare the performance of different transformer embeddings in the same experimental settings. The experiments reveal that MPNet versions v1 and v3 achieve the highest F1 score of 97.7% when combined with Random Forest, while T5 Large embedding achieves the highest F1 score of 98.2% when used with Softmax regression. MPNet v1 performs exceptionally well when used in the voting classifier, obtaining an impressive F1 score of 98.6%. In conclusion, the experiments validate the superiority of certain transformer models, such as MPNet v1, MPNet v3, and DistilRoBERTa, when used to calculate sentence embeddings within the Random Forest framework. The results also highlight the promising performance of T5 Large and RoBERTa Large in voting of Softmax regression and Random Forest. The voting classifier, employing transformer embeddings and ensemble learning techniques, consistently outperforms other baselines and individual algorithms. These findings emphasize the effectiveness of the voting classifier with transformer embeddings in achieving accurate and reliable predictions for news category classification tasks.<\/jats:p>","DOI":"10.3390\/computers12070137","type":"journal-article","created":{"date-parts":[[2023,7,10]],"date-time":"2023-07-10T00:45:37Z","timestamp":1688949937000},"page":"137","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Unifying Sentence Transformer Embedding and Softmax Voting Ensemble for Accurate News Category Prediction"],"prefix":"10.3390","volume":"12","author":[{"given":"Saima","family":"Khosa","sequence":"first","affiliation":[{"name":"Department of Information Security, The Islamia University of Bahawalpur, Bahawalpur 63100, Pakistan"},{"name":"Department of Information Technology, Khwaja Fareed University of Engineering and Information Technology, Rahim Yar Khan 64200, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5822-4005","authenticated-orcid":false,"given":"Arif","family":"Mehmood","sequence":"additional","affiliation":[{"name":"Department of Information Security, The Islamia University of Bahawalpur, Bahawalpur 63100, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6286-2544","authenticated-orcid":false,"given":"Muhammad","family":"Rizwan","sequence":"additional","affiliation":[{"name":"Department of Information Technology, Khwaja Fareed University of Engineering and Information Technology, Rahim Yar Khan 64200, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,8]]},"reference":[{"key":"ref_1","unstructured":"Karaman, Y., Akdeniz, F., Sava\u015f, B.K., and Becerikli, Y. (2022, January 19\u201321). A Comparative Analysis of SVM, LSTM and CNN-RNN Models for the BBC News Classification. Proceedings of the 7th International Conference on Smart City Applications, Castelo Branco, Portugal."},{"key":"ref_2","unstructured":"Gupta, A., Chugh, D., and Katarya, R. (2022). Sustainable Advanced Computing: Select Proceedings of ICSAC 2021, Springer."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ding, H., Yang, J., Deng, Y., Zhang, H., and Roth, D. (2023). Towards open-domain topic classification. arXiv.","DOI":"10.18653\/v1\/2022.naacl-demo.10"},{"key":"ref_4","first-page":"91","article-title":"Recommendation of effectiveness of YouTube video contents by qualitative sentiment analysis of its comments and replies","volume":"71","author":"Nawaz","year":"2019","journal-title":"Pak. J. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"121413","DOI":"10.1016\/j.techfore.2021.121413","article-title":"Deep learning for patent landscaping using transformer and graph embedding","volume":"175","author":"Choi","year":"2022","journal-title":"Technol. Forecast. Soc. Chang."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mars, M. (2022). From word embeddings to pre-trained language models: A state-of-the-art walkthrough. Appl. Sci., 12.","DOI":"10.3390\/app12178805"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1007\/s42452-020-2266-6","article-title":"Multi-category news classification using Support Vector Machine based classifiers","volume":"2","author":"Saigal","year":"2020","journal-title":"SN Appl. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ugwuoke, U.C., Aminu, E.F., and Ekundayo, A. (2022). Performing Data Augmentation Experiment to Enhance Model Accuracy: A Case Study of BBC News\u2019 Data, Elsevier.","DOI":"10.2139\/ssrn.4333014"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e1440","DOI":"10.1002\/widm.1440","article-title":"A novel methodology for Arabic news classification","volume":"12","author":"Alfonse","year":"2022","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"104076","DOI":"10.1016\/j.dib.2019.104076","article-title":"Sanad: Single-label arabic news articles dataset for automatic text categorization","volume":"25","author":"Einea","year":"2019","journal-title":"Data Brief"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1016\/j.future.2021.01.015","article-title":"Tracking sentiment towards news entities from Arabic news on social media","volume":"118","author":"Shahbaz","year":"2021","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Jang, B., Kim, I., and Kim, J.W. (2019). Word2vec convolutional neural networks for classification of news articles and tweets. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0220976"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2291","DOI":"10.1080\/09540091.2022.2117274","article-title":"WTL-CNN: A news text classification method of convolutional neural network based on weighted word embedding","volume":"34","author":"Zhao","year":"2022","journal-title":"Connect. Sci."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1197534","DOI":"10.1155\/2022\/1197534","article-title":"News Text Classification Method Based on the GRU_CNN Model","volume":"2022","author":"Deng","year":"2022","journal-title":"Int. Trans. Electr. Energy Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.53106\/199115992022083304001","article-title":"Chinese News Text Classification and Its Application Based on Combined-Convolutional Neural Network","volume":"33","author":"Liu","year":"2022","journal-title":"J. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3774607","DOI":"10.1155\/2021\/3774607","article-title":"Automated Amharic news categorization using deep learning models","volume":"2021","author":"Endalie","year":"2021","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_17","unstructured":"Ahmad, I., AlQurashi, F., and Mehmood, R. (2022). Machine and Deep Learning Methods with Manual and Automatic Labelling for News Classification in Bangla Language. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1215","DOI":"10.1166\/jmihi.2019.2735","article-title":"Social media news classification in healthcare communication","volume":"9","author":"Majeed","year":"2019","journal-title":"J. Med. Imaging Health Inform."},{"key":"ref_19","first-page":"363","article-title":"Vietnamese News Articles Classification Using Neural Networks","volume":"12","author":"Vinh","year":"2021","journal-title":"J. Adv. Inf. Technol. (JAIT)"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Rahman, M.M., Khan, M.A.Z., and Biswas, A.A. (2021, January 27\u201329). Bangla news classification using graph convolutional networks. Proceedings of the 2021 International Conference on Computer Communication and Informatics (ICCCI), Coimbatore, India.","DOI":"10.1109\/ICCCI50826.2021.9402567"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chowdhury, P., Eumi, E.M., Sarkar, O., and Ahamed, M.F. (2021, January 23\u201325). Bangla news classification using GloVe vectorization, LSTM, and CNN. Proceedings of the International Conference on Big Data, IoT, and Machine Learning: BIM2021, Cox\u2019s Bazar, Bangladesh.","DOI":"10.1007\/978-981-16-6636-0_54"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Amin, R., Sworna, N.S., and Hossain, N. (2020, January 5\u20137). Multiclass classification for bangla news tags with parallel cnn using word level data augmentation. Proceedings of the 2020 IEEE Region 10 Symposium (TENSYMP), Dhaka, Bangladesh.","DOI":"10.1109\/TENSYMP50017.2020.9230981"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"129176","DOI":"10.1109\/ACCESS.2022.3223049","article-title":"Depression Classification From Tweets Using Small Deep Transfer Learning Language Models","volume":"10","author":"Rizwan","year":"2022","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chandana, N., Sreelekha, A., Rasi, K., Sreeja, J., and Prassanna, P.L. (2021, January 4\u20136). BCC NEWS Classification Comparison between Na\u00efve Bayes, Support Vector Machine, Recurrent Neural Network. Proceedings of the 2021 Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV), Tirunelveli, India.","DOI":"10.1109\/ICICV50876.2021.9388461"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"8064579","DOI":"10.1155\/2021\/8064579","article-title":"News text classification method and simulation based on the hybrid deep learning model","volume":"2021","author":"Sun","year":"2021","journal-title":"Complexity"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Nerg\u0131z, G., Safali, Y., Avaro\u011flu, E., and Erdo\u011fan, S. (2019, January 21\u201322). Classification of Turkish news content by deep learning based LSTM using Fasttext model. Proceedings of the 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), Malatya, Turkey.","DOI":"10.1109\/IDAP.2019.8875949"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Dogru, H.B., Tilki, S., Jamil, A., and Hameed, A.A. (2021, January 6\u20137). Deep learning-based classification of news texts using doc2vec model. Proceedings of the 2021 1st International Conference on Artificial Intelligence and Data Analytics (CAIDA), Riyadh, Saudi Arabia.","DOI":"10.1109\/CAIDA51941.2021.9425290"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1508150","DOI":"10.1155\/2021\/1508150","article-title":"Research on news text classification based on deep learning convolutional neural network","volume":"2021","author":"Zhu","year":"2021","journal-title":"Wirel. Commun. Mob. Comput."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Ahmed, F., Akther, N., Hasan, M., Chowdhury, K., and Mukta, M.S.H. (2021, January 24\u201326). Word embedding based news classification by using CNN. Proceedings of the 2021 International Conference on Software Engineering & Computer Systems and 4th International Conference on Computational Science and InformationManagement (ICSECS-ICOCSIM), Pekan, Malaysia.","DOI":"10.1109\/ICSECS52883.2021.00117"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.733"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Huang, J., Tang, D., Zhong, W., Lu, S., Shou, L., Gong, M., Jiang, D., and Duan, N. (2021). Whiteningbert: An easy unsupervised sentence embedding approach. arXiv.","DOI":"10.18653\/v1\/2021.findings-emnlp.23"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Jiang, T., Jiao, J., Huang, S., Zhang, Z., Wang, D., Zhuang, F., Wei, F., Huang, H., Deng, D., and Zhang, Q. (2022). Promptbert: Improving bert sentence embeddings with prompts. arXiv.","DOI":"10.18653\/v1\/2022.emnlp-main.603"},{"key":"ref_33","unstructured":"(2023, May 24). Sentence-Transformers\/All-Mpnet-Base-v1\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/all-mpnet-base-v1."},{"key":"ref_34","unstructured":"(2023, May 24). Sentence-Transformers\/All-Mpnet-Base-v2\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/all-mpnet-base-v2."},{"key":"ref_35","unstructured":"(2023, May 24). Flax-Sentence-Embeddings\/All-Datasets-v3-Mpnet-Base\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/flax-sentence-embeddings\/all_datasets_v3_mpnet-base."},{"key":"ref_36","unstructured":"(2023, May 24). Flax-Sentence-Embeddings\/All-Datasets-v4-Mpnet-Base\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/flax-sentence-embeddings\/all_datasets_v4_mpnet-base."},{"key":"ref_37","unstructured":"(2023, May 24). Sentence-Transformers\/Msmarco-Distilbert-Base-Tas-b\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/msmarco-distilbert-base-tas-b."},{"key":"ref_38","unstructured":"(2023, May 24). Bongsoo\/Moco-SentencebertV2.0\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/bongsoo\/moco-sentencebertV2.0."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"102756","DOI":"10.1016\/j.ipm.2021.102756","article-title":"An ensemble model for classifying idioms and literal texts using BERT and RoBERTa","volume":"59","author":"Briskilal","year":"2022","journal-title":"Inf. Process. Manag."},{"key":"ref_40","unstructured":"(2023, May 24). Sentence-Transformers\/All-Roberta-Large-v1\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/all-roberta-large-v1."},{"key":"ref_41","unstructured":"(2023, May 24). Sentence-Transformers\/All-Distilroberta-v1\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/all-distilroberta-v1."},{"key":"ref_42","unstructured":"(2023, May 24). Sentence-Transformers\/gtr-t5-Base\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/gtr-t5-base."},{"key":"ref_43","unstructured":"(2023, May 24). Sentence-Transformers\/gtr-t5-large\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/gtr-t5-large."},{"key":"ref_44","unstructured":"(2023, May 24). Sentence-Transformers\/gtr-t5-xl\u00b7Hugging Face\u2014Huggingface.co. Available online: https:\/\/huggingface.co\/sentence-transformers\/gtr-t5-xl."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Abhishek, K. (2022, January 13\u201314). News Article Classification using a Transfer Learning Approach. Proceedings of the 2022 10th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), Noida, India.","DOI":"10.1109\/ICRITO56286.2022.9965142"},{"key":"ref_46","first-page":"5485","article-title":"Multi-Label News Category Text Classification","volume":"13","author":"Patil","year":"2022","journal-title":"J. Algebr. Stat."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"102791","DOI":"10.1016\/j.cose.2022.102791","article-title":"Tamp-X: Attacking explainable natural language classifiers through tampered activations","volume":"120","author":"Ali","year":"2022","journal-title":"Comput. Secur."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/7\/137\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:09:09Z","timestamp":1760126949000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/7\/137"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,8]]},"references-count":47,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["computers12070137"],"URL":"https:\/\/doi.org\/10.3390\/computers12070137","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,8]]}}}