{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T16:50:27Z","timestamp":1792687827897,"version":"4.5.0"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,3,25]],"date-time":"2023-03-25T00:00:00Z","timestamp":1679702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>Tokenization is an important text preprocessing step to prepare input tokens for deep language models. WordPiece and BPE are de facto methods employed by important models, such as BERT and GPT. However, the impact of tokenization can be different for morphologically rich languages, such as Turkic languages, in which many words can be generated by adding prefixes and suffixes. We compare five tokenizers at different granularity levels, that is, their outputs vary from the smallest pieces of characters to the surface form of words, including a Morphological-level tokenizer. We train these tokenizers and pretrain medium-sized language models using the RoBERTa pretraining procedure on the Turkish split of the OSCAR corpus. We then fine-tune our models on six downstream tasks. Our experiments, supported by statistical tests, reveal that the morphological-level tokenizer delivers a challenging performance with de facto tokenizers. Furthermore, we find that increasing the vocabulary size improves the performance of Morphological- and Word-level tokenizers more than that of de facto tokenizers. The ratio of the number of vocabulary parameters to the total number of model parameters can be empirically chosen as 20% for de facto tokenizers and 40% for other tokenizers to obtain a reasonable trade-off between model size and performance.<\/jats:p>","DOI":"10.1145\/3578707","type":"journal-article","created":{"date-parts":[[2023,3,25]],"date-time":"2023-03-25T07:43:09Z","timestamp":1679730189000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":60,"title":["Impact of Tokenization on Language Models: An Analysis for Turkish"],"prefix":"10.1145","volume":"22","author":[{"given":"Cagri","family":"Toraman","sequence":"first","affiliation":[{"name":"Aselsan Research Center, Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eyup Halit","family":"Yilmaz","sequence":"additional","affiliation":[{"name":"Aselsan Research Center, Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Furkan","family":"\u015eahi\u0307nu\u00e7","sequence":"additional","affiliation":[{"name":"Aselsan Research Center, Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oguzhan","family":"Ozcelik","sequence":"additional","affiliation":[{"name":"Aselsan Research Center, Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,3,25]]},"reference":[{"key":"e_1_3_2_2_1","first-page":"114","volume-title":"Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects","author":"Ahmadi Sina","year":"2020","unstructured":"Sina Ahmadi. 2020. A tokenization system for the Kurdish language. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects. International Committee on Computational Linguistics (ICCL), Barcelona, Spain (Online), 114\u2013127."},{"key":"e_1_3_2_3_1","first-page":"1","article-title":"Zemberek, an open source NLP framework for Turkic languages","volume":"10","author":"Ak\u0131n Ahmet Afsin","year":"2007","unstructured":"Ahmet Afsin Ak\u0131n and Mehmet D\u00fcndar Ak\u0131n. 2007. Zemberek, an open source NLP framework for Turkic languages. Structure 10 (2007), 1\u20135.","journal-title":"Structure"},{"key":"e_1_3_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3498370"},{"key":"e_1_3_2_5_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.gem-1.3"},{"key":"e_1_3_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"e_1_3_2_7_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.414"},{"key":"e_1_3_2_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.662"},{"key":"e_1_3_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3383200"},{"issue":"3","key":"e_1_3_2_10_1","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1002\/asi.20750","article-title":"Information retrieval on Turkish texts","volume":"59","author":"Can Fazli","year":"2008","unstructured":"Fazli Can, Seyit Kocberber, Erman Balcik, Cihan Kaynak, H. Cagdas Ocalan, and Onur M. Vursavas. 2008. Information retrieval on Turkish texts. Journal of the American Society for Information Science and Technology 59, 3 (2008), 407\u2013421.","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1322391.1322394"},{"key":"e_1_3_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2502069.2502078"},{"key":"e_1_3_2_13_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_3_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3325885"},{"key":"e_1_3_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3276773"},{"key":"e_1_3_2_16_1","first-page":"204","volume-title":"Proceedings of Machine Translation Summit XVII: Research Track","author":"Ding Shuoyang","year":"2019","unstructured":"Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019b. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII: Research Track. European Association for Machine Translation, Dublin, Ireland, 204\u2013213."},{"key":"e_1_3_2_17_1","article-title":"Crowdsourced phrase-based tokenization for low-resourced neural machine translation: The case of Fon language","author":"Dossou Bonaventure F. P.","year":"2021","unstructured":"Bonaventure F. P. Dossou and Chris C. Emezue. 2021. Crowdsourced phrase-based tokenization for low-resourced neural machine translation: The case of Fon language. arXiv preprint arXiv:2103.08052 (2021).","journal-title":"arXiv preprint arXiv:2103.08052"},{"key":"e_1_3_2_18_1","unstructured":"ElectricityMap. 2022. Climate Impact by Area. Retrieved March 16 2022 from https:\/\/app.electricitymap.org\/map."},{"key":"e_1_3_2_19_1","unstructured":"Chengyue Gong Di He Xu Tan Tao Qin Liwei Wang and Tie-Yan Liu. 2018. FRAGE: Frequency-agnostic word representation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (Montreal Canada) (NIPS\u201918) . Curran Associates Inc. Red Hook NY 1341\u20131352."},{"key":"e_1_3_2_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-90165-7_3"},{"issue":"248","key":"e_1_3_2_21_1","first-page":"1","article-title":"Towards the systematic reporting of the energy and carbon footprints of machine learning","volume":"21","author":"Henderson Peter","year":"2020","unstructured":"Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020. Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research 21, 248 (2020), 1\u201343.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.21"},{"key":"e_1_3_2_23_1","unstructured":"Huggingface. 2021. Oscar Dataset Huggingface. Retrieved March 16 2022 from https:\/\/huggingface.co\/datasets\/oscar."},{"key":"e_1_3_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3464378"},{"issue":"9","key":"e_1_3_2_25_1","doi-asserted-by":"crossref","first-page":"094037","DOI":"10.1088\/1748-9326\/ac1d0b","article-title":"The social cost of carbon dioxide under climate-economy feedbacks and temperature variability","volume":"16","author":"Kikstra J. S.","year":"2021","unstructured":"J. S. Kikstra, P. Waidelich, J. Rising, D. Yumashev, C. Hope, and C. M. Brierley. 2021. The social cost of carbon dioxide under climate-economy feedbacks and temperature variability. Environmental Research Letters 16, 9 (2021), 094037.","journal-title":"Environmental Research Letters"},{"key":"e_1_3_2_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1007"},{"key":"e_1_3_2_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-2012"},{"key":"e_1_3_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1929908.1929912"},{"key":"e_1_3_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448216"},{"key":"e_1_3_2_30_1","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019).","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"e_1_3_2_31_1","unstructured":"Loodos. 2020. Retrieved March 16 2022 from Turkish Language Models. https:\/\/github.com\/Loodos\/turkish-language-models."},{"key":"e_1_3_2_32_1","unstructured":"Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations (ICLR\u201919) New Orleans LA."},{"key":"e_1_3_2_33_1","article-title":"WangchanBERTa: Pretraining transformer-based Thai language models","author":"Lowphansirikul Lalita","year":"2021","unstructured":"Lalita Lowphansirikul, Charin Polpanumas, Nawat Jantrakulchai, and Sarana Nutanong. 2021. WangchanBERTa: Pretraining transformer-based Thai language models. arXiv preprint arXiv:2101.09635 (2021).","journal-title":"arXiv preprint arXiv:2101.09635"},{"key":"e_1_3_2_34_1","first-page":"12","volume-title":"Proceedings of the 2nd International Workshop on Software Knowledge 2011, in conjunction with 3rd International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management","author":"Jimenez Alejandro Metke","year":"2011","unstructured":"Alejandro Metke Jimenez, Kerry Raymond, and Ian MacColl. 2011. Information extraction from web services: A comparison of tokenisation algorithms. In Proceedings of the 2nd International Workshop on Software Knowledge 2011, in conjunction with 3rd International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management. Scitepress, 12\u201323."},{"key":"e_1_3_2_35_1","volume-title":"1st International Conference on Learning Representations (ICLR\u201913), Workshop Track Proceedings","author":"Mikolov Tom\u00e1s","year":"2013","unstructured":"Tom\u00e1s Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations (ICLR\u201913), Workshop Track Proceedings. Scottsdale, AZ."},{"key":"e_1_3_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.insights-1.1"},{"key":"e_1_3_2_38_1","doi-asserted-by":"publisher","DOI":"10.14618\/ids-pub-9021"},{"key":"e_1_3_2_39_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-industry.13"},{"key":"e_1_3_2_40_1","first-page":"133","volume-title":"Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing","author":"Park Kyubyong","year":"2020","unstructured":"Kyubyong Park, Joohong Lee, Seongbo Jang, and Dawoon Jung. 2020. An empirical study of tokenization strategies for various Korean NLP tasks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, Suzhou, China, 133\u2013142."},{"key":"e_1_3_2_41_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.170"},{"issue":"8","key":"e_1_3_2_43_1","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et\u00a0al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_2_44_1","volume-title":"3rd Workshop on Very Large Corpora","author":"Ramshaw Lance","year":"1995","unstructured":"Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In 3rd Workshop on Very Large Corpora. https:\/\/aclanthology.org\/W95-0107."},{"key":"e_1_3_2_45_1","unstructured":"Ministry of Environment Urbanization Climate Change Republic of Turkey. 2020. Primary Energy and GHG Emissions Coefficients of Electricity. Retrieved March 16 2022 from https:\/\/meslekihizmetler.csb.gov.tr\/elektrik-enerjisinin-birincil-enerji-ve-sera-gazi-salimi-katsayilari-2021-yilindan-itibaren-kullanilmak-uzere-guncellenmistir-duyuru-411795."},{"key":"e_1_3_2_46_1","article-title":"DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter","author":"Sanh Victor","year":"2019","unstructured":"Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019).","journal-title":"arXiv preprint arXiv:1910.01108"},{"key":"e_1_3_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2012.6289079"},{"key":"e_1_3_2_48_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.3770924"},{"key":"e_1_3_2_49_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1162"},{"key":"e_1_3_2_50_1","unstructured":"Nakatani Shuyo. 2010. Language Detection Library for Java. Retrieved March 16 2022 from http:\/\/code.google.com\/p\/language-detection\/"},{"key":"e_1_3_2_51_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/E14-2006"},{"key":"e_1_3_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357895"},{"key":"e_1_3_2_53_1","article-title":"Charformer: Fast character transformers via gradient-based subword tokenization","author":"Tay Yi","year":"2021","unstructured":"Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672 (2021).","journal-title":"arXiv preprint arXiv:2106.12672"},{"key":"e_1_3_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459104.3459170"},{"key":"e_1_3_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/INISTA.2011.5946096"},{"key":"e_1_3_2_56_1","first-page":"2215","volume-title":"Proceedings of the 13th Conference on Language Resources and Evaluation (LREC\u201922) (Marseille, France)","author":"Toraman Cagri","year":"2022","unstructured":"Cagri Toraman, Furkan \u015eahinu\u00e7, and Eyup Halit Yilmaz. 2022. Large-scale hate speech detection with cross-domain transfer. In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC\u201922) (Marseille, France). 2215\u20132225."},{"key":"e_1_3_2_57_1","doi-asserted-by":"publisher","DOI":"10.1017\/S135132490200284X"},{"key":"e_1_3_2_58_1","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/978-3-319-45925-7_4","volume-title":"International Conference on Statistical Language and Speech Processing","author":"\u00dcst\u00fcn Ahmet","year":"2016","unstructured":"Ahmet \u00dcst\u00fcn and Burcu Can. 2016. Unsupervised morphological segmentation using neural word embeddings. In International Conference on Statistical Language and Speech Processing. Springer, 43\u201353."},{"key":"e_1_3_2_59_1","doi-asserted-by":"crossref","unstructured":"Ahmet \u00dcst\u00fcn Murathan Kurfal\u0131 and Burcu Can. 2018. Characters or morphemes: How to represent words?. In Proceedings of The 3rd Workshop on Representation Learning for NLP (Melbourne Australia) . 144\u2013153.","DOI":"10.18653\/v1\/W18-3019"},{"key":"e_1_3_2_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCP51029.2020.9266140"},{"key":"e_1_3_2_61_1","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems (CA, USA)","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (CA, USA). 5998\u20136008."},{"key":"e_1_3_2_62_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.571"},{"key":"e_1_3_2_63_1","article-title":"ByT5: Towards a token-free future with pre-trained byte-to-byte models","author":"Xue Linting","year":"2021","unstructured":"Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, et\u00a0al. 2021. ByT5: Towards a token-free future with pre-trained byte-to-byte models. arXiv preprint arXiv:2105.13626 (2021).","journal-title":"arXiv preprint arXiv:2105.13626"},{"key":"e_1_3_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441667"},{"key":"e_1_3_2_65_1","unstructured":"Reyyan Yeniterzi. 2011. Exploiting morphology in Turkish named entity recognition system. In Proceedings of the ACL 2011 Student Session Association for Computational Linguistics Portland OR 105\u2013110."},{"key":"e_1_3_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3282443"},{"issue":"3","key":"e_1_3_2_67_1","doi-asserted-by":"crossref","first-page":"133","DOI":"10.3390\/info12030133","article-title":"Pre-training on mixed data for low-resource neural machine translation","volume":"12","author":"Zhang Wenbo","year":"2021","unstructured":"Wenbo Zhang, Xiao Li, Yating Yang, and Rui Dong. 2021b. Pre-training on mixed data for low-resource neural machine translation. Information 12, 3 (2021), 133.","journal-title":"Information"},{"key":"e_1_3_2_68_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.37"}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578707","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3578707","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T14:08:50Z","timestamp":1750169330000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3578707"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,25]]},"references-count":67,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3578707"],"URL":"https:\/\/doi.org\/10.1145\/3578707","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"value":"2375-4699","type":"print"},{"value":"2375-4702","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,25]]},"assertion":[{"value":"2022-04-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-23","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}