{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T23:37:54Z","timestamp":1761176274267,"version":"build-2065373602"},"reference-count":0,"publisher":"IOS Press","isbn-type":[{"value":"9781643686318","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T00:00:00Z","timestamp":1761004800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,10,21]]},"abstract":"<jats:p>The undeniable revolution brought forth by Large Language Models (LLMs) stems from the amazing fluency of the texts they generate, mastering language with seemingly human-like finesse. This fluency raises a key scientific question: How much lexical knowledge do LLMs actually capture in order to produce such fluent language? To address this, we present ALF, a freely available, analogical dataset endowed with rich lexicographic information grounded in Meaning-Text Theory for the French language. It comprises 2600 fine-grained lexical analogies with which we evaluate the lexical ability of five off-the-shelf LLMs, namely ChatGPT-4o mini, Llama3.0-8B, Llama3.1-8B, Qwen2.5-14B, and Mistral7B. Their performance spans from 45% for Mistral, through about 55% for the ChatGPT and Llama models, and up to nearly 60% for Qwen2.5-14B, thus qualifying ALF as a challenging dataset. Experimenting with larger models (OpenAI o1, Llama3.0\/3.1-70B, and Qwen2.5-32B) yields rather limited returns considering the drastic increase in computational cost. We further identify certain types of analogies and prompting methods that reveal performance disparities.<\/jats:p>","DOI":"10.3233\/faia251331","type":"book-chapter","created":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T09:58:26Z","timestamp":1761127106000},"source":"Crossref","is-referenced-by-count":0,"title":["ALF: A Fine-Grained French Analogical Dataset for Evaluating Lexical Knowledge of Large Language Models"],"prefix":"10.3233","author":[{"given":"Alexander","family":"Petrov","sequence":"first","affiliation":[{"name":"RALI, D\u00e9partement d\u2019informatique et de recherche op\u00e9rationelle, Universit\u00e9 de Montr\u00e9al"}]},{"given":"Antoine","family":"Venant","sequence":"additional","affiliation":[{"name":"OLST, D\u00e9partement de linguistique et de traduction, Universit\u00e9 de Montr\u00e9al"}]},{"given":"Fran\u00e7ois","family":"Lareau","sequence":"additional","affiliation":[{"name":"OLST, D\u00e9partement de linguistique et de traduction, Universit\u00e9 de Montr\u00e9al"}]},{"given":"Yves","family":"Lepage","sequence":"additional","affiliation":[{"name":"EBMT\/NLP, Faculty of Science and Engineering, Waseda University"}]},{"given":"Philippe","family":"Langlais","sequence":"additional","affiliation":[{"name":"RALI, D\u00e9partement d\u2019informatique et de recherche op\u00e9rationelle, Universit\u00e9 de Montr\u00e9al"}]}],"member":"7437","container-title":["Frontiers in Artificial Intelligence and Applications","ECAI 2025"],"original-title":[],"link":[{"URL":"https:\/\/ebooks.iospress.nl\/pdf\/doi\/10.3233\/FAIA251331","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T09:58:26Z","timestamp":1761127106000},"score":1,"resource":{"primary":{"URL":"https:\/\/ebooks.iospress.nl\/doi\/10.3233\/FAIA251331"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,21]]},"ISBN":["9781643686318"],"references-count":0,"URL":"https:\/\/doi.org\/10.3233\/faia251331","relation":{},"ISSN":["0922-6389","1879-8314"],"issn-type":[{"value":"0922-6389","type":"print"},{"value":"1879-8314","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,21]]}}}