{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T10:42:51Z","timestamp":1753440171475,"version":"3.37.3"},"reference-count":40,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2024,1,15]],"date-time":"2024-01-15T00:00:00Z","timestamp":1705276800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Young Academy Groningen"},{"name":"Dutch National Research School in Classical Studies"},{"name":"Dutch ministry of Education, Culture and Science","award":["024.003.012"],"award-info":[{"award-number":["024.003.012"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,4,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The last years have seen the application of Natural Language Processing, in particular, language models, to the study of the Semantics of ancient Greek, but only a little work has been done to create gold data for the evaluation of such models. In this contribution we introduce AGREE, the first benchmark for intrinsic evaluation of semantic models of ancient Greek created from expert judgements. In the absence of native speakers, eliciting expert judgements to create a gold standard is a way to leverage a competence that is the closest to that of natives. Moreover, this method allows for collecting data in a uniform way and giving precise instructions to participants. Human judgements about word relatedness were collected via two questionnaires: in the first, experts provided related lemmas to some proposed seeds, while in the second, they assigned relatedness judgements to pairs of lemmas. AGREE was built from a selection of the collected data.<\/jats:p>","DOI":"10.1093\/llc\/fqad087","type":"journal-article","created":{"date-parts":[[2024,1,15]],"date-time":"2024-01-15T14:33:49Z","timestamp":1705329229000},"page":"373-392","source":"Crossref","is-referenced-by-count":2,"title":["AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek"],"prefix":"10.1093","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3041-3477","authenticated-orcid":false,"given":"Silvia","family":"Stopponi","sequence":"first","affiliation":[{"name":"Centre for Language and Cognition Groningen, University of Groningen , Postbus 716, 9700 AS Groningen, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saskia","family":"Peels-Matthey","sequence":"additional","affiliation":[{"name":"Centre for Language and Cognition Groningen, University of Groningen , Postbus 716, 9700 AS Groningen, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Malvina","family":"Nissim","sequence":"additional","affiliation":[{"name":"Centre for Language and Cognition Groningen, University of Groningen , Postbus 716, 9700 AS Groningen, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2024,1,15]]},"reference":[{"first-page":"19","year":"2009","author":"Agirre","key":"2024040210380646500_fqad087-B1"},{"year":"2019","author":"Basile","key":"2024040210380646500_fqad087-B2"},{"year":"2020","author":"Bianchi","key":"2024040210380646500_fqad087-B3"},{"year":"2009","author":"Boschetti","key":"2024040210380646500_fqad087-B4"},{"year":"2016","author":"Boschetti","key":"2024040210380646500_fqad087-B5"},{"first-page":"136","year":"2012","author":"Bruni","key":"2024040210380646500_fqad087-B6"},{"first-page":"130","year":"2020","author":"Cassotti","key":"2024040210380646500_fqad087-B7"},{"issue":"3","key":"2024040210380646500_fqad087-B8","doi-asserted-by":"crossref","first-page":"987","DOI":"10.3758\/s13428-018-1115-7","article-title":"The \u201cSmall World of Words\u201d English Word Association Norms for Over 12,000 Cue Words\u2019","volume":"51","author":"De Deyne","year":"2019","journal-title":"Behavior Research Methods"},{"first-page":"6326","year":"2019","author":"Di Carlo","key":"2024040210380646500_fqad087-B9"},{"first-page":"31","year":"2013","author":"Dinu","key":"2024040210380646500_fqad087-B10"},{"first-page":"3819","year":"2018","author":"Ercan","key":"2024040210380646500_fqad087-B11"},{"first-page":"406","year":"2001","author":"Finkelstein","key":"2024040210380646500_fqad087-B12"},{"key":"2024040210380646500_fqad087-B13","doi-asserted-by":"crossref","first-page":"E3635","DOI":"10.1073\/pnas.1720347115","article-title":"Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes\u2019,","volume":"115","author":"Garg","year":"2018","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024040210380646500_fqad087-B14","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1075\/cilt.177.10gib","article-title":"Psycholinguistic Perspectives on Polysemy\u2019","author":"Gibbs","year":"2001","journal-title":"Polysemy in Cognitive Linguistics: Selected papers from the International Cognitive Linguistics Conference, Amsterdam, 1997"},{"first-page":"33","year":"2013","author":"Graham","key":"2024040210380646500_fqad087-B15"},{"first-page":"2116","year":"2016","author":"Hamilton","key":"2024040210380646500_fqad087-B16"},{"key":"2024040210380646500_fqad087-B17","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1080\/00437956.1954.11659520","article-title":"Distributional Structure\u2019,","volume":"10","author":"Harris","year":"1954","journal-title":"WORD"},{"key":"2024040210380646500_fqad087-B18","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1111\/cogs.12076","article-title":"A Quantitative Empirical Analysis of the Abstract\/Concrete Distinction\u2019,","volume":"38","author":"Hill","year":"2014","journal-title":"Cognitive Science"},{"key":"2024040210380646500_fqad087-B19","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1162\/COLI_a_00237","article-title":"Simlex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation\u2019","volume":"41","author":"Hill","year":"2015","journal-title":"Computational Linguistics"},{"year":"2020","author":"Keersmaekers","key":"2024040210380646500_fqad087-B20"},{"key":"2024040210380646500_fqad087-B21","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1177\/0003122419877135","article-title":"The Geometry of Culture: Analyzing the Meanings of Class through Word Embeddings\u2019","volume":"84","author":"Kozlowski","year":"2019","journal-title":"American Sociological Review"},{"key":"2024040210380646500_fqad087-B22","doi-asserted-by":"crossref","first-page":"1269","DOI":"10.1007\/s10579-021-09575-z","article-title":"A Comparative Evaluation and Analysis of Three Generations of Distributional Semantic Models\u2019","volume":"56","author":"Lenci","year":"2022","journal-title":"Language Resources and Evaluation"},{"first-page":"3861","year":"2022","author":"Lugli","key":"2024040210380646500_fqad087-B23"},{"key":"2024040210380646500_fqad087-B24","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1093\/llc\/fqz036","article-title":"A Computational Approach to Lexical Polysemy in Ancient Greek\u2019","volume":"34","author":"McGillivray","year":"2019","journal-title":"Digital Scholarship in the Humanities"},{"key":"2024040210380646500_fqad087-B25","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1515\/joll-2022-2007","article-title":"A New Corpus Annotation Framework for Latin Diachronic Lexical Semantics\u2019","volume":"21","author":"McGillivray","year":"2022","journal-title":"Journal of Latin Linguistics"},{"year":"2013","author":"Mikolov","key":"2024040210380646500_fqad087-B26"},{"key":"2024040210380646500_fqad087-B27","doi-asserted-by":"crossref","first-page":"402","DOI":"10.3758\/BF03195588","article-title":"The University of South Florida Free Association, Rhyme, and Word Fragment Norms\u2019","volume":"36","author":"Nelson","year":"2004","journal-title":"Behavior Research Methods, Instruments, & Computers"},{"key":"2024040210380646500_fqad087-B28","first-page":"287","volume-title":"Computational Approaches to Semantic Change","author":"Perrone","year":"2021"},{"first-page":"56","year":"2019","author":"Perrone","key":"2024040210380646500_fqad087-B29"},{"key":"2024040210380646500_fqad087-B30","first-page":"63","article-title":"Vector Space Models of Ancient Greek Word Meaning, and A Case Study on Homer\u2019","volume":"60","author":"Rodda","year":"2019","journal-title":"Traitement Automatique des Langues (TAL)"},{"key":"2024040210380646500_fqad087-B31","doi-asserted-by":"crossref","first-page":"11","DOI":"10.4000\/ijcol.421","article-title":"Panta Rei: Tracking Semantic Change with Distributional Semantics in Ancient Greek\u2019","volume":"3","author":"Rodda","year":"2017","journal-title":"Italian Journal of Computational Linguistics"},{"key":"2024040210380646500_fqad087-B32","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1145\/365628.365657","article-title":"Contextual Correlates of Synonymy\u2019","volume":"8","author":"Rubenstein","year":"1965","journal-title":"Communications of the ACM"},{"key":"2024040210380646500_fqad087-B33","first-page":"33","article-title":"The Distributional Hypothesis\u2019","volume":"20","author":"Sahlgren","year":"2008","journal-title":"Italian Journal of Disability Studies"},{"year":"2020","author":"Schlechtweg","key":"2024040210380646500_fqad087-B34"},{"year":"2021","author":"Schlechtweg","key":"2024040210380646500_fqad087-B35"},{"first-page":"298","year":"2015","author":"Schnabel","key":"2024040210380646500_fqad087-B36"},{"key":"2024040210380646500_fqad087-B37","doi-asserted-by":"crossref","first-page":"29","DOI":"10.4000\/ijcol.624","article-title":"Building and Comparing Lemma Embeddings for Latin. Classical Latin versus Thomas Aquinas\u2019","volume":"6","author":"Sprugnoli","year":"2020","journal-title":"Italian Journal of Computational Linguistics"},{"first-page":"176","year":"2020","author":"Stringham","key":"2024040210380646500_fqad087-B38"},{"first-page":"137","year":"2019","author":"Tauber","key":"2024040210380646500_fqad087-B39"},{"key":"2024040210380646500_fqad087-B40","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1163\/24523666-01000013","article-title":"The Diorisis Ancient Greek Corpus\u2019","volume":"3","author":"Vatri","year":"2018","journal-title":"Research Data Journal for the Humanities and Social Sciences"}],"container-title":["Digital Scholarship in the Humanities"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/dsh\/article-pdf\/39\/1\/373\/57134494\/fqad087.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/dsh\/article-pdf\/39\/1\/373\/57134494\/fqad087.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,2]],"date-time":"2024-04-02T13:56:50Z","timestamp":1712066210000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/dsh\/article\/39\/1\/373\/7542408"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,15]]},"references-count":40,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1,15]]},"published-print":{"date-parts":[[2024,4,2]]}},"URL":"https:\/\/doi.org\/10.1093\/llc\/fqad087","relation":{},"ISSN":["2055-7671","2055-768X"],"issn-type":[{"type":"print","value":"2055-7671"},{"type":"electronic","value":"2055-768X"}],"subject":[],"published-other":{"date-parts":[[2024,4,1]]},"published":{"date-parts":[[2024,1,15]]}}}