{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T21:40:40Z","timestamp":1791841240387,"version":"4.3.4"},"reference-count":28,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,1,28]],"date-time":"2022-01-28T00:00:00Z","timestamp":1643328000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The task of summarization can be categorized into two methods, extractive and abstractive. Extractive summarization selects the salient sentences from the original document to form a summary while abstractive summarization interprets the original document and generates the summary in its own words. The task of generating a summary, whether extractive or abstractive, has been studied with different approaches in the literature, including statistical-, graph-, and deep learning-based approaches. Deep learning has achieved promising performances in comparison to the classical approaches, and with the advancement of different neural architectures such as the attention network (commonly known as the transformer), there are potential areas of improvement for the summarization task. The introduction of transformer architecture and its encoder model \u201cBERT\u201d produced an improved performance in downstream tasks in NLP. BERT is a bidirectional encoder representation from a transformer modeled as a stack of encoders. There are different sizes for BERT, such as BERT-base with 12 encoders and BERT-larger with 24 encoders, but we focus on the BERT-base for the purpose of this study. The objective of this paper is to produce a study on the performance of variants of BERT-based models on text summarization through a series of experiments, and propose \u201cSqueezeBERTSum\u201d, a trained summarization model fine-tuned with the SqueezeBERT encoder variant, which achieved competitive ROUGE scores retaining the BERTSum baseline model performance by 98%, with 49% fewer trainable parameters.<\/jats:p>","DOI":"10.3390\/info13020067","type":"journal-article","created":{"date-parts":[[2022,1,29]],"date-time":"2022-01-29T05:26:17Z","timestamp":1643433977000},"page":"67","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":90,"title":["Performance Study on Extractive Text Summarization Using BERT Models"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1256-7245","authenticated-orcid":false,"given":"Shehab","family":"Abdel-Salam","sequence":"first","affiliation":[{"name":"Computer Science Department, The American University in Cairo, AUC Avenue, Cairo 11835, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8109-1845","authenticated-orcid":false,"given":"Ahmed","family":"Rafea","sequence":"additional","affiliation":[{"name":"Computer Science Department, The American University in Cairo, AUC Avenue, Cairo 11835, Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10462-016-9475-9","article-title":"Recent automatic text summarization techniques: A survey","volume":"47","author":"Gambhir","year":"2016","journal-title":"Artif. Intell. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Nallapati, R., Zhai, F., and Zhou, B. (2017, January 4\u20139). Summarunner: A Recurrent Neural Network-Based Sequence Model for Extractive Summarization of Documents. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10958"},{"key":"ref_3","unstructured":"Hovy, E., and Lin, C.Y. (1999). Automated Text Summarization in SUMMARIST, MIT Press."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1147\/rd.22.0159","article-title":"The Automatic Creation of Literature Abstracts","volume":"2","author":"Luhn","year":"1958","journal-title":"IBM J. Res. Dev."},{"key":"ref_5","unstructured":"Mihalcea, R., and Tarau, P. (2004, January 1). TextRank: Bringing Order into Texts. Proceedings of the Empirical Methods on Natural Language Processing, Barcelona, Spain."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1613\/jair.1523","article-title":"LexRank: Graph-based Lexical Centrality as Salience in Text Summarization","volume":"22","author":"Erkan","year":"2004","journal-title":"J. Artif. Intell. Res."},{"key":"ref_7","unstructured":"Steinberger, J., and Jezek, K. (2004, January 23\u201324). Using Latent Semantic Analysis in Text Summarization and Summary Evaluation. Proceedings of the 7th International Conference on Information System Implementation and Modeling, New York, NY, USA."},{"key":"ref_8","unstructured":"Nenkova, A., and Vanderwende, L. (2005). The Impact of Frequency on Summarization. Technical Report, Microsoft Research (MSR-TR-2005-101), Columbia University."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Cheng, J., and Lapata, M. (2016, January 7\u201312). Neural Summarization by Extracting Sentences and Words. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.","DOI":"10.18653\/v1\/P16-1046"},{"key":"ref_10","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"102264","DOI":"10.1016\/j.ipm.2020.102264","article-title":"EdgeSumm: Graph-based framework for automatic text summarization","volume":"57","author":"Salama","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_12","unstructured":"Lin, C.Y. (2004, January 25\u201326). Rouge: A Package for Automatic Evaluation Of Summaries. Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Liu, Y., and Lapata, M. (2019). Text Summarization with Pretrained Encoders. arXiv.","DOI":"10.18653\/v1\/D19-1387"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., and Huang, X. (2021, August 09). Extractive Summarization as Text Matching. ACL Anthology. Available online: https:\/\/www.aclweb.org\/anthology\/2020.acl-main.552\/.","DOI":"10.18653\/v1\/2020.acl-main.552"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.eswa.2019.03.045","article-title":"SummCoder: An unsupervised framework for extractive text summarization based on deep auto-encoders","volume":"129","author":"Joshi","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sun, S., Cheng, Y., Gan, Z., and Liu, J. (2019). Patient knowledge distillation for bert model compression. arXiv.","DOI":"10.18653\/v1\/D19-1441"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Iandola, F.N., Shaw, A.E., Krishna, R., and Keutzer, K.W. (2020). SqueezeBERT: What Can Computer Vision Teach NLP about Efficient Neural Networks?. arXiv.","DOI":"10.18653\/v1\/2020.sustainlp-1.17"},{"key":"ref_18","unstructured":"VSanh, i., Debut, L., Chaumond, J., and Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1910.01108v4.pdf."},{"key":"ref_19","unstructured":"Dong, Y., Shen, Y., Crawford, E., van Hoof, H., and Cheung, J.C.K. (November, January 31). Banditsum: Extractive summarization as a contextual bandit. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Brussels, Belgium."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Bucila, C., Caruana, R., and Niculescu-Mizil, A. (2006, January 20\u201323). Model compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Philadelphia, PA, USA.","DOI":"10.1145\/1150402.1150464"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18\u201322). ShuffleNet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the 2018 Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UH, USA.","DOI":"10.1109\/CVPR.2018.00716"},{"key":"ref_22","unstructured":"Hinton, G.E., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv."},{"key":"ref_23","unstructured":"Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., and Keutzer, K. (2016). SqueezeNet: Alexnet-level accuracy with 50\u00d7 fewer parameters and <0.5 mb model size. arXiv."},{"key":"ref_24","unstructured":"Howard, G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_25","unstructured":"Tan, M., and Le, Q.V. (2019, January 10\u201315). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2021, November 06). HuggingFace\u2019s Transformers: State-of-the-Art Natural Language Processing. arXiv.org. Available online: https:\/\/arxiv.org\/abs\/1910.03771.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. (2019). Q8BERT: Quantized 8bit bert. arXiv.","DOI":"10.1109\/EMC2-NIPS53020.2019.00016"},{"key":"ref_28","unstructured":"Sanh, V., Wolf, T., and Rush, A.M. (2020). Movement pruning: Adaptive sparsity by fine-tuning. arXiv."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/2\/67\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:09:41Z","timestamp":1760134181000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/2\/67"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,28]]},"references-count":28,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["info13020067"],"URL":"https:\/\/doi.org\/10.3390\/info13020067","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,28]]}}}