{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T15:05:13Z","timestamp":1784819113467,"version":"3.55.0"},"reference-count":27,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T00:00:00Z","timestamp":1784764800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"FUN4DATE","award":["PID2022-136684OB-C22"],"award-info":[{"award-number":["PID2022-136684OB-C22"]}]},{"name":"Spanish Agencia Estatal de Investigacion","award":["10.13039\/501100011033"],"award-info":[{"award-number":["10.13039\/501100011033"]}]},{"name":"Chips Act Joint Undertaking project SMARTY","award":["101140087"],"award-info":[{"award-number":["101140087"]}]},{"name":"OpenAI API research access program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2026,10,31]]},"abstract":"<jats:p>Large Language Models (LLMs) have achieved unprecedented performance on many complex tasks, being able, for example, to answer questions on almost any topic. However, they struggle with other simple tasks, such as counting the occurrences of letters in a word, as illustrated by the inability of many LLMs to count the number of \u201cr\u201d letters in \u201cstrawberry.\u201d Several works have studied this problem and linked it to the tokenization used by LLMs, the intrinsic limitations of the attention mechanism, or to the lack of character-level training data. In this paper, we conduct an experimental study to evaluate when LLMs fail to count letters. In more detail, we study the relations between LLM errors when counting letters, including: (1) the frequency of the word and its components in the training dataset and (2) the complexity of the counting operation. We present a comprehensive analysis of the errors of LLMs when counting letter occurrences by evaluating a representative group of models over a large number of words. The results show a number of consistent trends in the models evaluated: (1) models are capable of recognizing the letters but not counting them; (2) the frequency of the word and tokens in the word does not have a significant impact on the LLM errors; (3) there is a positive correlation of letter frequency with errors; more frequent letters tend to have more counting errors; (4) the errors show a strong correlation with the number of letters or tokens in a word; and (5) the strongest correlation occurs with the number of letters with counts larger than one, with most models being unable to correctly count words in which letters appear more than twice. These results suggest that the problems of LLMs to counting letters are not related to the frequency of words or tokens in the training data but to the complexity of the counting operation. However, further studies are needed to build a better understanding of the limitations of LLMs in count the letters in a word.<\/jats:p>","DOI":"10.1145\/3818606","type":"journal-article","created":{"date-parts":[[2026,5,25]],"date-time":"2026-05-25T13:39:56Z","timestamp":1779716396000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["When Do Large Language Models (LLMs) Struggle to Count Letters?"],"prefix":"10.1145","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-9571-7250","authenticated-orcid":false,"given":"Tairan","family":"Fu","sequence":"first","affiliation":[{"name":"Politecnico di Milano, Milan, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9522-1643","authenticated-orcid":false,"given":"Raquel","family":"Ferrando","sequence":"additional","affiliation":[{"name":"Universidad Politecnica de Madrid, Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5304-0626","authenticated-orcid":false,"given":"Javier","family":"Conde","sequence":"additional","affiliation":[{"name":"Universidad Politecnica de Madrid, Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0513-2550","authenticated-orcid":false,"given":"Carlos","family":"Arriaga","sequence":"additional","affiliation":[{"name":"Universidad Politecnica de Madrid, Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2273-1341","authenticated-orcid":false,"given":"Pedro","family":"Reviriego","sequence":"additional","affiliation":[{"name":"Universidad Politecnica de Madrid, Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,23]]},"reference":[{"issue":"2","key":"e_1_3_2_2_2","first-page":"127","article-title":"Alphabetic vs. non-alphabetic writing: Linguistic fit and natural tendencies","volume":"23","author":"Baroni Antonio","year":"2011","unstructured":"Antonio Baroni, et al. 2011. Alphabetic vs. non-alphabetic writing: Linguistic fit and natural tendencies. Rivista di Linguistica 23, 2 (2011), 127\u2013159.","journal-title":"Rivista di Linguistica"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781003116325"},{"key":"e_1_3_2_4_2","volume-title":"Labyrinths: Selected Stories and Other Writings","author":"Borges Jorge Luis","year":"1962","unstructured":"Jorge Luis Borges. 1962. Funes, the memorious. In Labyrinths: Selected Stories and Other Writings. New Directions."},{"key":"e_1_3_2_5_2","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The Llama 3 herd of models. arXiv:2407.21783. Retrieved from https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.177"},{"key":"e_1_3_2_7_2","unstructured":"Albert Q. Jiang Alexandre Sablayrolles Arthur Mensch Chris Bamford Devendra Singh Chaplot Diego de las Casas Florian Bressand Gianna Lengyel Guillaume Lample Lucile Saulnier et al. 2023. Mistral 7B. arXiv:2310.06825. Retrieved from https:\/\/arxiv.org\/abs\/2310.06825"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.17077\/0743-2747.1259"},{"key":"e_1_3_2_9_2","unstructured":"Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Dawn Song and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. arXiv:2103.03874. Retrieved from https:\/\/arxiv.org\/abs\/2103.03874"},{"key":"e_1_3_2_10_2","unstructured":"Shervin Minaee Tomas Mikolov Narjes Nikzad Meysam Chenaghlu Richard Socher Xavier Amatriain and Jianfeng Gao. 2024. Large language models: A survey. arXiv:2402.06196. Retrieved from https:\/\/arxiv.org\/abs\/2402.06196"},{"key":"e_1_3_2_11_2","unstructured":"Benjamin Minixhofer Ivan Vuli\u0107 and Edoardo Maria Ponti. 2025. Universal cross-tokenizer distillation via approximate likelihood matching. arXiv:2503.20083. Retrieved from https:\/\/arxiv.org\/abs\/2503.20083"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203726631"},{"key":"e_1_3_2_13_2","first-page":"219","volume-title":"Beautiful Data: The Stories Behind Elegant Data Solutions","author":"Norvig Peter","year":"2009","unstructured":"Peter Norvig. 2009. Natural language corpus data. In Beautiful Data: The Stories Behind Elegant Data Solutions. Toby Segaran and Jeff Hammerbacher (Eds.), O\u2019Reilly Media, 219\u2013242."},{"key":"e_1_3_2_14_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-1608"},{"key":"e_1_3_2_16_2","unstructured":"Abrar Rahman Garry Bowlin Binit Mohanty and Sean McGunigal. 2024. Towards linguistically-aware and language-independent tokenization for large language models (LLMs). arXiv:2410.03568. Retrieved from https:\/\/arxiv.org\/abs\/2410.03568"},{"key":"e_1_3_2_17_2","unstructured":"Nived Rajaraman Jiantao Jiao and Kannan Ramchandran. 2024. Toward a theory of tokenization in LLMs. arXiv:2404.08335. Retrieved from https:\/\/arxiv.org\/abs\/2404.08335"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1162"},{"key":"e_1_3_2_19_2","unstructured":"Andrew Shin and Kunitake Kaneko. 2024. Large language models lack understanding of character composition of words. arXiv:2405.11357. Retrieved from https:\/\/arxiv.org\/abs\/2405.11357"},{"key":"e_1_3_2_20_2","unstructured":"Gemma Team Morgane Riviere Shreya Pathak Pier Giuseppe Sessa Cassidy Hardin Surya Bhupatiraju L\u00e9onard Hussenot Thomas Mesnard Bobak Shahriari Alexandre Ram\u00e9 et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv:2408.00118. Retrieved from https:\/\/arxiv.org\/abs\/2408.00118"},{"key":"e_1_3_2_21_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2023. Attention is all you need. arXiv:1706.03762. Retrieved from https:\/\/arxiv.org\/abs\/1706.03762"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"Yubo Wang Xueguang Ma Ge Zhang Yuansheng Ni Abhranil Chandra Shiguang Guo Weiming Ren Aaran Arulraj Xuan He Ziyan Jiang et al. 2024. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. arXiv:2406.01574. Retrieved from https:\/\/arxiv.org\/abs\/2406.01574","DOI":"10.52202\/079017-3018"},{"key":"e_1_3_2_23_2","unstructured":"Nan Xu and Xuezhe Ma. 2024. LLM the genius paradox: A linguistic and math expert\u2019s struggle with simple word-based counting problems. arXiv:2410.14166. Retrieved from https:\/\/arxiv.org\/abs\/2410.14166"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00461"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649506"},{"key":"e_1_3_2_26_2","unstructured":"Gilad Yehudai Haim Kaplan Asma Ghandeharioun Mor Geva and Amir Globerson. 2024. When can transformers count to n? arXiv:2407.15160. Retrieved from https:\/\/arxiv.org\/abs\/2407.15160"},{"key":"e_1_3_2_27_2","unstructured":"Zheng Yuan Hongyi Yuan Chuanqi Tan Wei Wang and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? arXiv:2304.02015. Retrieved from https:\/\/arxiv.org\/abs\/2304.02015"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1472"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3818606","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T14:34:30Z","timestamp":1784817270000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3818606"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,23]]},"references-count":27,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,10,31]]}},"alternative-id":["10.1145\/3818606"],"URL":"https:\/\/doi.org\/10.1145\/3818606","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,23]]},"assertion":[{"value":"2024-12-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-07","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}