{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,2]],"date-time":"2025-11-02T10:40:40Z","timestamp":1762080040893,"version":"build-2065373602"},"reference-count":40,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,12,11]],"date-time":"2022-12-11T00:00:00Z","timestamp":1670716800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the \u201cInnovative Solution for Optimizing User Productivity through Multi-Modal Monitoring of Activity and Profiles \u2013 OPTIMIZE\u201d\/\u201cSolu\u021bie Inovativ\u0103 de Optimizare a Productivit\u0103\u021bii Utilizatorilor prin Monitorizarea Multi-Modala a Activit\u0103\u021bii \u0219i a Profilelor\u2014OPTIMIZE\u201d project","award":["366\/390042\/27.09.2021"],"award-info":[{"award-number":["366\/390042\/27.09.2021"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Significant progress has been achieved in text generation due to recent developments in neural architectures; nevertheless, this task remains challenging, especially for low-resource languages. This study is centered on developing a model for abstractive summarization in Romanian. A corresponding dataset for summarization is introduced, followed by multiple models based on the Romanian GPT-2, on top of which control tokens were considered to specify characteristics for the generated text, namely: counts of sentences and words, token ratio, and n-gram overlap. These are special tokens defined in the prompt received by the model to indicate traits for the text to be generated. The initial model without any control tokens was assessed using BERTScore (F1 = 73.43%) and ROUGE (ROUGE-L accuracy = 34.67%). Control tokens improved the overall BERTScore to 75.42% using &lt;LexOverlap&gt;, while the model was influenced more by the second token specified in the prompt when performing various combinations of tokens. Six raters performed human evaluations of 45 generated summaries with different models and decoding methods. The generated texts were all grammatically correct and consistent in most cases, while the evaluations were promising in terms of main idea coverage, details, and cohesion. Paraphrasing still requires improvements as the models mostly repeat information from the reference text. In addition, we showcase an exploratory analysis of the generated summaries using one or two specific control tokens.<\/jats:p>","DOI":"10.3390\/a15120472","type":"journal-article","created":{"date-parts":[[2022,12,12]],"date-time":"2022-12-12T04:05:22Z","timestamp":1670817922000},"page":"472","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["RoSummary: Control Tokens for Romanian News Summarization"],"prefix":"10.3390","volume":"15","author":[{"given":"Mihai Alexandru","family":"Niculescu","sequence":"first","affiliation":[{"name":"Computer Science & Engineering Department, University Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, Romania"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0380-6814","authenticated-orcid":false,"given":"Stefan","family":"Ruseti","sequence":"additional","affiliation":[{"name":"Computer Science & Engineering Department, University Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, Romania"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4815-9227","authenticated-orcid":false,"given":"Mihai","family":"Dascalu","sequence":"additional","affiliation":[{"name":"Research Technology, 19D Soseaua Virtutii, 060782 Bucharest, Romania"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,11]]},"reference":[{"key":"ref_1","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_2","first-page":"2","article-title":"Better language models and their implications","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_3","unstructured":"Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv."},{"key":"ref_4","unstructured":"Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. (2020). GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. arXiv."},{"key":"ref_5","unstructured":"Wang, B., and Komatsuzaki, A. (2022, October 14). GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. Available online: https:\/\/github.com\/kingoflolz\/mesh-transformer-jax."},{"key":"ref_6","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Masala, M., Ruseti, S., and Dascalu, M. (2020, January 8\u201313). Robert\u2013a romanian bert model. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.","DOI":"10.18653\/v1\/2020.coling-main.581"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Dumitrescu, S.D., Avram, A.M., and Pyysalo, S. (2020). The birth of Romanian BERT. arXiv.","DOI":"10.18653\/v1\/2020.findings-emnlp.387"},{"key":"ref_9","unstructured":"Avram, A.M., Catrina, D., Cercel, D.C., Dasc\u0103lu, M., Rebedea, T., P\u0103i\u015f, V., and Tufi\u015f, D. (2021). Distilling the Knowledge of Romanian BERTs Using Multiple Teachers. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Niculescu, M.A., Ruseti, S., and Dascalu, M. (2021, January 1\u20133). RoGPT2: Romanian GPT2 for Text Generation. Proceedings of the 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), Washington, DC, USA.","DOI":"10.1109\/ICTAI52525.2021.00183"},{"key":"ref_11","unstructured":"Dumitrescu, S.D., Rebeja, P., Lorincz, B., Gaman, M., Avram, A., Ilie, M., Pruteanu, A., Stan, A., Rosia, L., and Iacobescu, C. (2021, January 7\u201310). Liro: Benchmark and leaderboard for Romanian language tasks. Proceedings of the Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), Online."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Nallapati, R., Zhou, B., Nogueira dos santos, C., Gulcehre, C., and Xiang, B. (2016). Abstractive text summarization using sequence-to-sequence RNNs and beyond. arXiv.","DOI":"10.18653\/v1\/K16-1028"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Scialom, T., Dray, P.A., Lamprier, S., Piwowarski, B., and Staiano, J. (2020). MLSUM: The multilingual summarization corpus. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.647"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Narayan, S., Cohen, S.B., and Lapata, M. (2018). Don\u2019t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv.","DOI":"10.18653\/v1\/D18-1206"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"V\u00f6lske, M., Potthast, M., Syed, S., and Stein, B. (2017, January 7). Tl; dr: Mining Reddit to learn automatic summarization. Proceedings of the Workshop on New Frontiers in Summarization, Copenhagen, Denmark.","DOI":"10.18653\/v1\/W17-4508"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Cioaca, V., Dascalu, M., and McNamara, D.S. (2020, January 1\u20134). Extractive Summarization using Cohesion Network Analysis and Submodular Set Functions. Proceedings of the 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2020), Timisoara, Romania.","DOI":"10.1109\/SYNASC51798.2020.00035"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Dutulescu, A., Ruseti, S., and Dascalu, M. (2022, January 12\u201315). Unsupervised Extractive Summarization with BERT. Proceedings of the 24th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2022), Linz, Austria.","DOI":"10.1109\/SYNASC57785.2022.00032"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., and Huang, X.J. (2020, January 6\u20138). Extractive Summarization as Text Matching. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.552"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Liu, Y., and Lapata, M. (2019, January 3\u20137). Text Summarization with Pretrained Encoders. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1387"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020, January 6\u20138). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Liu, Y., Liu, P., Radev, D., and Neubig, G. (2022, January 22\u201327). BRIO: Bringing Order to Abstractive Summarization. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.","DOI":"10.18653\/v1\/2022.acl-long.207"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., and Zhou, M. (2020, January 16\u201320). ProphetNet: Predicting Future N-gram for Sequence-to-SequencePre-training. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Online.","DOI":"10.18653\/v1\/2020.findings-emnlp.217"},{"key":"ref_23","first-page":"11328","article-title":"Pegasus: Pre-training with extracted gap-sentences for abstractive summarization","volume":"119","author":"Zhang","year":"2020","journal-title":"Proc. Int. Conf. Mach. Learn."},{"key":"ref_24","unstructured":"Martin, L., Fan, A., de la Clergerie, \u00c9., Bordes, A., and Sagot, B. (2020). MUSS: Multilingual unsupervised sentence simplification by mining paraphrases. arXiv."},{"key":"ref_25","unstructured":"Clive, J., Cao, K., and Rei, M. (2021). Control prefixes for text generation. arXiv."},{"key":"ref_26","unstructured":"Kieuvongngam, V., Tan, B., and Niu, Y. (2020). Automatic text summarization of COVID-19 medical research articles using BERT and GPT-2. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, Philadelphia, PE, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_28","first-page":"2005","article-title":"Recall-oriented understudy for gisting evaluation (rouge)","volume":"20","author":"Lin","year":"2005","journal-title":"Retrieved August"},{"key":"ref_29","unstructured":"Banerjee, S., and Lavie, A. (2005, January 25). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, Ann Arbor, MI, USA."},{"key":"ref_30","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with BERT. arXiv."},{"key":"ref_31","first-page":"27263","article-title":"Bartscore: Evaluating generated text as text generation","volume":"34","author":"Yuan","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Sellam, T., Das, D., and Parikh, A.P. (2020). BLEURT: Learning robust metrics for text generation. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.704"},{"key":"ref_33","unstructured":"Celikyilmaz, A., Clark, E., and Gao, J. (2020). Evaluation of text generation: A survey. arXiv."},{"key":"ref_34","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Freitag, M., and Al-Onaizan, Y. (2017). Beam search strategies for neural machine translation. arXiv.","DOI":"10.18653\/v1\/W17-3207"},{"key":"ref_36","unstructured":"Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019). The curious case of neural text degeneration. arXiv."},{"key":"ref_37","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_38","unstructured":"Taylor, L. (2022, October 14). Assessing Reading-Into-Writing Skills for an Academic Context: Some Theoretical and Practical Considerations. Available online: https:\/\/github.com\/kingoflolz\/mesh-transformer-jax."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1097\/TLD.0b013e3181ff5a88","article-title":"Summarizing expository texts","volume":"30","author":"Westby","year":"2010","journal-title":"Top. Lang. Disord."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J. Chiropr. Med."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/12\/472\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:38:07Z","timestamp":1760146687000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/12\/472"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,11]]},"references-count":40,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["a15120472"],"URL":"https:\/\/doi.org\/10.3390\/a15120472","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2022,12,11]]}}}