{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T16:10:38Z","timestamp":1775837438072,"version":"3.50.1"},"reference-count":60,"publisher":"Cambridge University Press (CUP)","issue":"5","license":[{"start":{"date-parts":[[2022,5,13]],"date-time":"2022-05-13T00:00:00Z","timestamp":1652400000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The tremendous amount of increase in the number of documents available on the Web has turned finding the relevant piece of information into a challenging, tedious, and time-consuming activity. Accordingly, automatic text summarization has become an important field of study by gaining significant attention from the researchers. Lately, with the advances in deep learning, neural abstractive text summarization with sequence-to-sequence (Seq2Seq) models has gained popularity. There have been many improvements in these models such as the use of pretrained language models (e.g., GPT, BERT, and XLM) and pretrained Seq2Seq models (e.g., BART and T5). These improvements have addressed certain shortcomings in neural summarization and have improved upon challenges such as saliency, fluency, and semantics which enable generating higher quality summaries. Unfortunately, these research attempts were mostly limited to the English language. Monolingual BERT models and multilingual pretrained Seq2Seq models have been released recently providing the opportunity to utilize such state-of-the-art models in low-resource languages such as Turkish. In this study, we make use of pretrained Seq2Seq models and obtain state-of-the-art results on the two large-scale Turkish datasets, TR-News and MLSum, for the text summarization task. Then, we utilize the title information in the datasets and establish hard baselines for the title generation task on both datasets. We show that the input to the models has a substantial amount of importance for the success of such tasks. Additionally, we provide extensive analysis of the models including cross-dataset evaluations, various text generation options, and the effect of preprocessing in ROUGE evaluations for Turkish. It is shown that the monolingual BERT models outperform the multilingual BERT models on all tasks across all the datasets. Lastly, qualitative evaluations of the generated summaries and titles of the models are provided.<\/jats:p>","DOI":"10.1017\/s1351324922000195","type":"journal-article","created":{"date-parts":[[2022,5,13]],"date-time":"2022-05-13T09:25:02Z","timestamp":1652433902000},"page":"1275-1304","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":22,"title":["Turkish abstractive text summarization using pretrained sequence-to-sequence models"],"prefix":"10.1017","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8549-6380","authenticated-orcid":false,"given":"Batuhan","family":"Baykara","sequence":"first","affiliation":[]},{"given":"Tunga","family":"G\u00fcng\u00f6r","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2022,5,13]]},"reference":[{"key":"S1351324922000195_ref17","doi-asserted-by":"publisher","DOI":"10.1109\/SIU.2019.8806510"},{"key":"S1351324922000195_ref40","unstructured":"Radford, A. , Narasimhan, K. , Salimans, T. and Sutskever, I. (2018). Improving language understanding by generative pre-training."},{"key":"S1351324922000195_ref25","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00343"},{"key":"S1351324922000195_ref37","doi-asserted-by":"publisher","DOI":"10.1109\/ISCIS.2008.4717908"},{"key":"S1351324922000195_ref5","doi-asserted-by":"crossref","unstructured":"\u00c7\u0131\u011f\u0131r, C. , Kutlu, M. and \u00c7i\u00e7ekli, \u0130. (2009). Generic text summarization for Turkish. In ISCIS. IEEE, pp. 224\u2013229.","DOI":"10.1109\/ISCIS.2009.5291848"},{"key":"S1351324922000195_ref21","unstructured":"Kuratov, Y. and Arkhipov, M. (2019). Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language. CoRR, abs\/1905.07213."},{"key":"S1351324922000195_ref29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K16-1028"},{"key":"S1351324922000195_ref34","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-014-9267-2"},{"key":"S1351324922000195_ref2","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-021-09568-y"},{"key":"S1351324922000195_ref15","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"S1351324922000195_ref6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.598"},{"key":"S1351324922000195_ref39","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.217"},{"key":"S1351324922000195_ref18","doi-asserted-by":"publisher","DOI":"10.1109\/SIU49456.2020.9302096"},{"key":"S1351324922000195_ref45","unstructured":"Rust, P. , Pfeiffer, J. , Vulic, I. , Ruder, S. and Gurevych, I. (2020). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. CoRR, abs\/2012.15613."},{"key":"S1351324922000195_ref49","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1162"},{"key":"S1351324922000195_ref46","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.3770924"},{"key":"S1351324922000195_ref51","unstructured":"Song, K. , Tan, X. , Qin, T. , Lu, J. and Liu, T.-Y. (2019). MASS: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pp. 5926\u20135936."},{"key":"S1351324922000195_ref47","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.647"},{"key":"S1351324922000195_ref54","unstructured":"Virtanen, A. , Kanerva, J. , Ilo, R. , Luoma, J. , Luotolahti, J. , Salakoski, T. , Ginter, F. and Pyysalo, S. (2019). Multilingual is not enough: BERT for Finnish. CoRR, abs\/1912.07076."},{"key":"S1351324922000195_ref57","unstructured":"Wu, Y. , Schuster, M. , Chen, Z. , Le, Q.V. , Norouzi, M. , Macherey, W. , Krikun, M. , Cao, Y. , Gao, Q. , Macherey, K. , Klingner, J. , Shah, A. , Johnson, M. , Liu, X. , \u0141ukasz Kaiser, Gouws S. , Kato, Y. , Kudo, T. , Kazawa, H. , Stevens, K. , Kurian, G. , Patil, N. , Wang, W. , Young, C. , Smith, J. , Riesa, J. , Rudnick, A. , Vinyals, O. , Corrado, G. , Hughes, M. and Dean, J. (2016). Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs\/1609.08144."},{"key":"S1351324922000195_ref9","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019. Association for Computational Linguistics, pp. 4171\u20134186."},{"key":"S1351324922000195_ref43","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00313"},{"key":"S1351324922000195_ref35","unstructured":"\u00d6zsoy, M.G. , \u00c7i\u00e7ekli, \u0130. and Alpaslan, F.N. (2010). Text summarization of Turkish texts using latent semantic analysis. In Proceedings of the 23rd International Conference on Computational Linguistics, COLING\u201910, USA. Association for Computational Linguistics, pp. 869\u2013876."},{"key":"S1351324922000195_ref20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1007"},{"key":"S1351324922000195_ref59","unstructured":"Yang, Z. , Dai, Z. , Yang, Y. , Carbonell, J. , Salakhutdinov, R.R. and Le, Q.V. (2019). XLNet: Generalized autoregressive pretraining for language understanding. In Wallach H., Larochelle H., Beygelzimer A., d\u2019 Alch\u00e9-Buc F., Fox E. and Garnett R. (eds), Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc.,"},{"key":"S1351324922000195_ref28","unstructured":"Mihalcea, R. and Tarau, P. (2004). TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain, July 2004. Association for Computational Linguistics, pp. 404\u2013411."},{"key":"S1351324922000195_ref41","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"S1351324922000195_ref7","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1012"},{"key":"S1351324922000195_ref16","unstructured":"Hu, J. , Ruder, S. , Siddhant, A. , Neubig, G. , Firat, O. and Johnson, M. (2020) XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Daume, H. III and Singh A. (eds), Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 119, 13\u201318 July 2020. PMLR, pp. 4411\u20134421."},{"key":"S1351324922000195_ref3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.414"},{"key":"S1351324922000195_ref36","unstructured":"Paulus, R. , Xiong, C. and Socher, R. (2018). A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30\u2013May 3, 2018, Conference Track Proceedings. OpenReview.net."},{"key":"S1351324922000195_ref32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1158"},{"key":"S1351324922000195_ref52","doi-asserted-by":"publisher","DOI":"10.1002\/9781119004752"},{"key":"S1351324922000195_ref53","volume-title":"In Advances in Neural Information Processing Systems","volume":"30","author":"Vaswani","year":"2017"},{"key":"S1351324922000195_ref58","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.41"},{"key":"S1351324922000195_ref55","unstructured":"Wenzek, G. , Lachaux, M.-A. , Conneau, V , Chaudhary, V. , Guzm\u00e1n, F. , Joulin, A. and Grave, E. (2020). CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, May 2020. European Language Resources Association, pp. 4003\u20134012."},{"key":"S1351324922000195_ref4","doi-asserted-by":"crossref","unstructured":"\u00c7eliky\u0131lmaz, A. , Bosselut, A. , He, X. and Choi, Y. (2018). Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, June 2018. Association for Computational Linguistics, pp. 1662\u20131675.","DOI":"10.18653\/v1\/N18-1150"},{"key":"S1351324922000195_ref48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1099"},{"key":"S1351324922000195_ref10","unstructured":"Dong, L. , Yang, N. , Wang, W. , Wei, F. , Liu, X. , Wang, Y. , Gao, J. , Zhou, M. and Hon, H.-W. (2019). Unified language model pre-training for natural language understanding and generation. In Wallach H., Larochelle H., Beygelzimer A., d\u2019 Alch\u00e9-Buc F., Fox E. and Garnett R. (eds), Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc."},{"key":"S1351324922000195_ref30","doi-asserted-by":"crossref","unstructured":"Nallapati, R. , Zhai, F. and Zhou, B. (2017). SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI\u201917. AAAI Press, pp. 3075\u20133081.","DOI":"10.1609\/aaai.v31i1.10958"},{"key":"S1351324922000195_ref22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"S1351324922000195_ref19","unstructured":"Kingma, D.P. and Ba, J. (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7\u20139 May 2015, Conference Track Proceedings."},{"key":"S1351324922000195_ref44","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1044"},{"key":"S1351324922000195_ref14","unstructured":"Hermann, K.M. , Ko\u010disk\u00fd, T. , Grefenstette, E. , Espeholt, L. , Kay, W. , Suleyman, M. and Blunsom, P. (2015). Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS\u201915, Cambridge, MA, USA. MIT Press, pp. 1693\u20131701."},{"key":"S1351324922000195_ref56","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"S1351324922000195_ref33","unstructured":"Ng, A. , Ngiam, J. , Foo, C.Y. and Mai, Y. (2014). Deep learning. In CS229 Lecture Notes, pp. 1\u201330."},{"key":"S1351324922000195_ref50","unstructured":"Shazeer, N. and Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In Dy, J. and Krause, A. (eds), Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 80, 10\u201315 July 2018. PMLR, pp. 4596\u20134604."},{"key":"S1351324922000195_ref24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1387"},{"key":"S1351324922000195_ref38","unstructured":"Polignano, M. , Basile, P. , de Gemmis, M. , Semeraro, G. and Basile, V. (2019). AlBERTo: Italian BERT language understanding model for NLP challenging tasks based on Tweets. In Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019), vol. 2481. CEUR."},{"key":"S1351324922000195_ref23","unstructured":"Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, July 2004. Association for Computational Linguistics, pp. 74\u201381."},{"key":"S1351324922000195_ref13","article-title":"Efficient feature integration with Wikipedia-based semantic feature extraction for Turkish text summarization","volume":"21","author":"G\u00fcran","year":"2013","journal-title":"Turkish Journal of Electrical Engineering and Computer Sciences"},{"key":"S1351324922000195_ref27","doi-asserted-by":"publisher","DOI":"10.1147\/rd.22.0159"},{"key":"S1351324922000195_ref8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"S1351324922000195_ref1","unstructured":"Altan, Z. (2004). A Turkish automatic text summarization system. In IASTED International Conference on AIA."},{"key":"S1351324922000195_ref12","doi-asserted-by":"publisher","DOI":"10.1109\/INISTA.2011.5946121"},{"key":"S1351324922000195_ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1206"},{"key":"S1351324922000195_ref26","unstructured":"Liu, Y. , Ott, M. , Goyal, N. , Du, J. , Joshi, M. , Chen, D. , Levy, O. , Lewis, M. , Zettlemoyer, L. and Stoyanov, V. (2019). Roberta: A Robustly Optimized BERT Pretraining Approach. CoRR, abs\/1907.11692."},{"key":"S1351324922000195_ref11","doi-asserted-by":"publisher","DOI":"10.1145\/321510.321519"},{"key":"S1351324922000195_ref60","unstructured":"Zhang, J. , Zhao, Y. , Saleh, M. and Liu, P. (2020). PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Daume H. III and Singh A. (eds), Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 119, 13\u201318 July 2020. PMLR, pp. 11328\u201311339."},{"key":"S1351324922000195_ref42","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324922000195","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,11]],"date-time":"2023-09-11T02:07:13Z","timestamp":1694398033000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324922000195\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,13]]},"references-count":60,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["S1351324922000195"],"URL":"https:\/\/doi.org\/10.1017\/s1351324922000195","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,13]]},"assertion":[{"value":"\u00a9 The Author(s), 2022. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}