{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T04:13:36Z","timestamp":1773288816930,"version":"3.50.1"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"6","funder":[{"name":"Spanish Agencia Estatal de Investigacion","award":["10.13039\/501100011033"],"award-info":[{"award-number":["10.13039\/501100011033"]}]},{"name":"Spanish project ENTRUDIT","award":["TED2021-130118B-I00"],"award-info":[{"award-number":["TED2021-130118B-I00"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2025,12,31]]},"abstract":"<jats:p>The performance of conversational Large Language Models (LLMs) in general, and of ChatGPT in particular, is currently being evaluated on many different tasks, from logical reasoning or math to answering questions on a myriad of topics. Instead, much less attention is being devoted to the study of the linguistic features of the texts generated by these LLMs. This is surprising since LLMs are models for language, and understanding how they use the language is important. Indeed, conversational LLMs are poised to have a significant impact on the evolution of languages as they may eventually dominate the creation of new text. This means that for example, if conversational LLMs do not use a word it may become less and less frequent and eventually stop being used altogether. Therefore, evaluating the linguistic features of the text they produce and how those depend on the model parameters is the first step toward understanding the potential impact of conversational LLMs on the evolution of languages. In this article, we consider the evaluation of the lexical diversity of the text generated by LLMs in English and how it depends on the model parameters. A methodology is presented and used to conduct a comprehensive evaluation of lexical diversity using ChatGPT as a case study. The results show how lexical diversity depends on the version of ChatGPT and some of its parameters, such as the presence penalty, or the role assigned to the model. The dataset and tools used in our analysis are released under open licenses with the goal of drawing much-needed attention to the evaluation of the linguistic features of LLM-generated\u00a0text.<\/jats:p>","DOI":"10.1145\/3696459","type":"journal-article","created":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T12:41:46Z","timestamp":1726836106000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9125-6225","authenticated-orcid":false,"given":"Gonzalo","family":"Mart\u00ednez","sequence":"first","affiliation":[{"name":"Universidad Carlos III de Madrid, Legan\u00e9, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9551-4308","authenticated-orcid":false,"given":"Jos\u00e9","family":"Alberto Hern\u00e1ndez","sequence":"additional","affiliation":[{"name":"Universidad Carlos III de Madrid, Legan\u00e9, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5304-0626","authenticated-orcid":false,"given":"Javier","family":"Conde","sequence":"additional","affiliation":[{"name":"Universidad Polit\u00e9cnica de Madrid, Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2273-1341","authenticated-orcid":false,"given":"Pedro","family":"Reviriego","sequence":"additional","affiliation":[{"name":"Universidad Polit\u00e9cnica de Madrid, Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4129-4626","authenticated-orcid":false,"given":"Elena","family":"Merino-Gomez","sequence":"additional","affiliation":[{"name":"Universidad de Valladolid, Valladolid, Spain"}]}],"member":"320","published-online":{"date-parts":[[2025,11,24]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","unstructured":"Quentin Bertrand Avishek Joey Bose Alexandre Duplessis Marco Jiralerspong and Gauthier Gidel. 2023. On the stability of iterative retraining of generative models on their own data. arXiv:2310.00429 [cs.LG]. Retrieved from 10.48550\/arXiv.2310.00429","DOI":"10.48550\/arXiv.2310.00429"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","unstructured":"Martin Briesch Dominik Sobania and Franz Rothlauf. 2023. Large language models suffer from their own output: An analysis of the self-consuming training loop. arXiv:2311.16822 [cs.LG]. Retrieved from 10.48550\/arXiv.2311.1682","DOI":"10.48550\/arXiv.2311.1682"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","unstructured":"Michael A. Covington and Joe D. McFall. 2010. Cutting the gordian knot: The moving-average type\u2013token ratio (MATTR). Journal of Quantitative Linguistics 17 2 (2010) 94\u2013100. DOI: 10.1080\/09296171003643098","DOI":"10.1080\/09296171003643098"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","unstructured":"Albert Q. Jiang Alexandre Sablayrolles Arthur Mensch Chris Bamford Devendra Singh Chaplot Diego de las Casas Florian Bressand Gianna Lengyel Guillaume Lample Lucile Saulnier L\u00e9lio Renard Lavaud Marie-Anne Lachaux Pierre Stock Teven Le Scao Thibaut Lavril Thomas Wang Timoth\u00e9e Lacroix and William El Sayed. 2023. Mistral 7B. arXiv:2310.06825 [cs.CL]. Retrieved from 10.48550\/arXiv.2310.06825","DOI":"10.48550\/arXiv.2310.06825"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","unstructured":"Aarohi Srivastava et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615 [cs.CL]. Retrieved from 10.48550\/arXiv.2206.04615","DOI":"10.48550\/arXiv.2206.04615"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Punit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov and Thomas Scialom. 2023. LLaMa 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 [cs.CL]. Retrieved from 10.48550\/arXiv.2307.09288","DOI":"10.48550\/arXiv.2307.09288"},{"key":"e_1_3_2_8_2","unstructured":"Angela Fan Yacine Jernite Ethan Perez David Grangier Jason Weston and Michael Auli. 2019. ELI5: Long form question answering. arXiv:1907.09190."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","unstructured":"Annette Gerstenberg. 2015. A Sociolinguistic Perspective on Vocabulary Richness in a Seven-Year Comparison of Older Adults John Benjamins Publishing Company 109\u2013127. DOI: 10.1075\/impact.37.06ger","DOI":"10.1075\/impact.37.06ger"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2018.05.012"},{"key":"e_1_3_2_11_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hendrycks Dan","unstructured":"Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=d7KBjmI3GmQ"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","unstructured":"Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang Dawn Song and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. arXiv:2103.03874 [cs.LG]. Retrieved from 10.48550\/arXiv.2103.03874","DOI":"10.48550\/arXiv.2103.03874"},{"key":"e_1_3_2_13_2","volume-title":"Proceedings of the Workshop on Evaluating Evaluation of Ai Systems at AAAI","author":"Hernandez-Orallo Jose","year":"2020","unstructured":"Jose Hernandez-Orallo. 2020. AI evaluation: On broken yardsticks and measurement scales. In Proceedings of the Workshop on Evaluating Evaluation of Ai Systems at AAAI."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","unstructured":"Ari Holtzman Jan Buys Li Du Maxwell Forbes and Yejin Choi. 2020. The curious case of neural text degeneration. arXiv:1904.09751 [cs.CL]. Retrieved from 10.48550\/arXiv.1904.09751","DOI":"10.48550\/arXiv.1904.09751"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511667268.008"},{"key":"e_1_3_2_16_2","volume-title":"Defining and Measuring Lexical Diversity","author":"Jarvis Scott","year":"2013","unstructured":"Scott Jarvis and M. Daller. 2013. Defining and Measuring Lexical Diversity. Vocabulary knowledge: Human ratings and automated measures."},{"key":"e_1_3_2_17_2","volume-title":"Lexical Diversity and Lexical Density in Speech and Writing","author":"Johansson Victoria","year":"2008","unstructured":"Victoria Johansson. 2008. Lexical Diversity and Lexical Density in Speech and Writing. WorkingPaper. The information about affiliations in this record was updated in December 2015. The record was previously connected to the following departments: Linguistics and Phonetics (015010003)."},{"key":"e_1_3_2_18_2","volume-title":"The History of English in a Social Context: A Contribution to Historical Sociolinguistics","author":"Kastovsky Dieter","year":"2011","unstructured":"Dieter Kastovsky and Arthur Mettinger. 2011. The History of English in a Social Context: A Contribution to Historical Sociolinguistics. Vol. 129. Walter de Gruyter."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1515\/cllt-2016-0056"},{"key":"e_1_3_2_20_2","volume-title":"Sociolinguistic Patterns","author":"Labov W.","year":"1972","unstructured":"W. Labov. 1972. Sociolinguistic Patterns. University of Pennsylvania Press."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","unstructured":"Theresa Lillis and Carolyn McKinney. 2013. The sociolinguistics of writing in a global context: Objects lenses consequences. Journal of Sociolinguistics 17 4 (2013) 415\u2013439. DOI: 10.1111\/josl.12046","DOI":"10.1111\/josl.12046"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-57963-9_5"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1177\/0265532207080767"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.42.2.381"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","unstructured":"Alberto Mu\u00f1oz-Ortiz Carlos G\u00f3mez-Rodr\u00edguez and David Vilares. 2023. Contrasting linguistic patterns in human and LLM-generated text. arXiv:2308.09067. Retrieved from 10.1007\/s10462-024-10903-2","DOI":"10.1007\/s10462-024-10903-2"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774 [cs.CL]. Retrieved from 10.48550\/arXiv.2303.08774","DOI":"10.48550\/arXiv.2303.08774"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","unstructured":"Pedro Reviriego Javier Conde Elena Merino-G\u00f3mez Gonzalo Mart\u00ednez and Jos\u00e9 Alberto Hern\u00e1ndez. 2023. Playing with words: Comparing the vocabulary and lexical richness of ChatGPT and humans. arXiv:2308.07462 [cs.CL]. Retrieved from 10.48550\/arXiv.2308.07462","DOI":"10.48550\/arXiv.2308.07462"},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","unstructured":"Christelle Robert and Liliana Rico Duarte. 2015. Semantic richness and aging: The effect of number of features in the lexical decision task. Journal of Psycholinguistic Research 45 (2015) 359\u2013365.","DOI":"10.1007\/s10936-015-9352-8"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","unstructured":"Lucas Shen. 2022. LexicalRichness: A small module to compute textual lexical richness. DOI: 10.5281\/zenodo.6607007","DOI":"10.5281\/zenodo.6607007"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.lingua.2021.103155"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1093\/llc\/16.3.251"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","unstructured":"Gemini Team. 2023. Gemini: A family of highly capable multimodal models. arXiv:2312.11805 [cs.CL]. Retrieved from 10.48550\/arXiv.2312.11805","DOI":"10.48550\/arXiv.2312.11805"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","unstructured":"Juan Manuel Toro. 2023. Emergence of a phonological bias in ChatGPT. arXiv:2305.15929. Retrieved from 10.48550\/arXiv.2305.15929","DOI":"10.48550\/arXiv.2305.15929"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1001749303137"},{"key":"e_1_3_2_35_2","author":"Wardhaugh Ronald","year":"2014","unstructured":"Ronald Wardhaugh and Janet Fuller. 2014. An Introduction to Sociolinguistics (7th. ed.). Wiley.","journal-title":"An Introduction to Sociolinguistics"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1472"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","unstructured":"Kyrie Zhixuan Zhou and Madelyn Rose Sanfilippo. 2023. Public perceptions of gender bias in large language models: Cases of chatgpt and ernie. arXiv:2309.09120. Retrieved from 10.48550\/arXiv.2309.09120","DOI":"10.48550\/arXiv.2309.09120"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3696459","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,24]],"date-time":"2025-11-24T15:09:31Z","timestamp":1763996971000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3696459"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,24]]},"references-count":36,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1145\/3696459"],"URL":"https:\/\/doi.org\/10.1145\/3696459","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,24]]},"assertion":[{"value":"2024-02-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-06","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}