{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,17]],"date-time":"2026-10-17T03:00:01Z","timestamp":1792206001126,"version":"4.4.0"},"reference-count":67,"publisher":"MIT Press","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2015,12]]},"abstract":"<jats:p>We present SimLex-999, a gold standard resource for evaluating distributional semantic models that improves on existing resources in several important ways. First, in contrast to gold standards such as WordSim-353 and MEN, it explicitly quantifies similarity rather than association or relatedness so that pairs of entities that are associated but not actually similar (Freud, psychology) have a low rating. We show that, via this focus on similarity, SimLex-999 incentivizes the development of models with a different, and arguably wider, range of applications than those which reflect conceptual association. Second, SimLex-999 contains a range of concrete and abstract adjective, noun, and verb pairs, together with an independent rating of concreteness and (free) association strength for each pair. This diversity enables fine-grained analyses of the performance of models on concepts of different types, and consequently greater insight into how architectures can be improved. Further, unlike existing gold standard evaluations, for which automatic approaches have reached or surpassed the inter-annotator agreement ceiling, state-of-the-art models perform well below this ceiling on SimLex-999. There is therefore plenty of scope for SimLex-999 to quantify future improvements to distributional semantic models, guiding the development of the next generation of representation-learning architectures.<\/jats:p>","DOI":"10.1162\/coli_a_00237","type":"journal-article","created":{"date-parts":[[2015,12,10]],"date-time":"2015-12-10T19:06:29Z","timestamp":1449774389000},"page":"665-695","source":"Crossref","is-referenced-by-count":484,"title":["SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation"],"prefix":"10.1162","volume":"41","author":[{"given":"Felix","family":"Hill","sequence":"first","affiliation":[{"name":"University of Cambridge"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roi","family":"Reichart","sequence":"additional","affiliation":[{"name":"Technion, Israel Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anna","family":"Korhonen","sequence":"additional","affiliation":[{"name":"University of Cambridge"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.3115\/1620754.1620758"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1037\/a0016261"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Bansal, Mohit, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of ACL, Baltimore, MD.","DOI":"10.3115\/v1\/P14-2131"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Baroni, Marco, Georgiana Dinu, and Germ\u00e1n Kruszewski. 2014. Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of ACL, Baltimore, MD.","DOI":"10.3115\/v1\/P14-1023"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00016"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/S1364-6613(02)00029-3"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-2402"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1162\/153244303322533223"},{"key":"R10","unstructured":"Bernardi, Raffaella, Georgiana Dinu, Marco Marelli, and Marco Baroni. 2013. A relatedness benchmark to test the role of determiners in compositional distributional semantics. In Proceedings of ACL, Sofia."},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Biemann, Chris. 2005. Ontology learning from text: A survey of methods. LDV Forum, 20(2):75\u201393.","DOI":"10.21248\/jlcl.20.2005.76"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.3115\/1225403.1225421"},{"key":"R13","unstructured":"Bruni, Elia, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012a. Distributional semantics in technicolor. In Proceedings of ACL, Jeju Island."},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1145\/2393347.2396422"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2006.32.1.13"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1648"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390177"},{"key":"R19","unstructured":"Cunningham, Hamish. 2005. Information extraction, automatic. Encyclopedia of language and linguistics, pages 665\u2013677."},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th International Conference on World Wide Web, pages 406\u2013414, Hong Kong.","DOI":"10.1145\/371920.372094"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.2307\/1128738"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780195170009.003.0022"},{"key":"R25","unstructured":"Gershman, Anatole, Yulia Tsvetkov, Leonid Boytsov, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proceedings of ACL, Baltimore, MD."},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1007\/BF02163027"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.114.2.211"},{"key":"R28","unstructured":"Haghighi, Aria, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL 2008, Columbus, OH."},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Hassan, Samer and Rada Mihalcea. 2011. Semantic relatedness using salient semantic analysis. In AAAI, San Francisco, CA.","DOI":"10.1609\/aaai.v25i1.7971"},{"key":"R30","unstructured":"Hatzivassiloglou, Vasileios, Judith L. Klavans, Melissa L. Holcombe, Regina Barzilay, Min-Yen Kan, and Kathleen McKeown. 2001. Simfinder: A flexible clustering tool for summarization. In NAACL Workshop on Automatic Summarization, Pittsburgh, PA."},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613730"},{"key":"R32","unstructured":"Hill, Felix, Douwe Kiela, and Anna Korhonen. 2013. Concreteness and corpora: A theoretical and practical analysis. CMCL 2013, page 75, Sofia."},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.1111\/cogs.12076"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00183"},{"key":"R35","unstructured":"Huang, Eric H., Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of ACL, pages 873\u2013882, Jeju Island."},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-1503"},{"key":"R37","doi-asserted-by":"crossref","unstructured":"Kiela, Douwe, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving multi-modal representations using image dispersion: Why less is sometimes more. In Proceedings of ACL, Baltimore, MD.","DOI":"10.3115\/v1\/P14-2135"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.104.2.211"},{"key":"R39","doi-asserted-by":"crossref","unstructured":"Leech, Geoffrey, Roger Garside, and Michael Bryant. 1994. Claws4: The tagging of the British National Corpus. In Proceedings of COLING, pages 622\u2013628, Kyoto.","DOI":"10.3115\/991886.991996"},{"key":"R40","unstructured":"Levy, Omer and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL, volume 2."},{"key":"R41","doi-asserted-by":"crossref","unstructured":"Levy, Omer, Steffen Remus, Chris Biemann, and Idol Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? Proceedings of NAACL, Denver, CO.","DOI":"10.3115\/v1\/N15-1098"},{"key":"R42","unstructured":"Lewis, David D., Yiming Yang, Tony G. Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361\u2013397."},{"key":"R44","unstructured":"Li, Mu, Yang Zhang, Muhua Zhu, and Ming Zhou. 2006. Exploring distributional similarity based models for query spelling correction. In Proceedings of ALC, pages 1025\u20131032."},{"key":"R45","unstructured":"Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. CoNLL-2013, page 104, Sofia."},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.1037\/0278-7393.23.1.54"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.3115\/1699510.1699560"},{"key":"R49","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2009.05.004"},{"key":"R50","unstructured":"Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, AZ."},{"key":"R51","unstructured":"Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111\u20133119, Lake Tahoe, NV."},{"key":"R52","doi-asserted-by":"publisher","DOI":"10.1145\/1459352.1459355"},{"key":"R53","doi-asserted-by":"publisher","DOI":"10.3758\/BF03195588"},{"key":"R54","unstructured":"Pad\u00f3, Sebastian, Ulrike Pad\u00f3, and Katrin Erk. 2007. Flexible, corpus-based modelling of human plausibility judgements. In Proceedings of EMNLP-CoNLL, pages 400\u2013409, Prague."},{"key":"R55","doi-asserted-by":"publisher","DOI":"10.1037\/h0084295"},{"key":"R56","doi-asserted-by":"publisher","DOI":"10.3115\/1614025.1614037"},{"key":"R57","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367510"},{"key":"R58","unstructured":"Plaut, David C. 1995. Semantic and associative priming in a distributed attractor network. In Proceedings of CogSci, volume 17, pages 37\u201342, Pittsburgh, PA."},{"key":"R59","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.41.3.647"},{"key":"R60","unstructured":"Reisinger, Joseph and Raymond Mooney. 2010a. A mixture model with sharing for lexical semantics. In Proceedings of EMNLP, pages 1173\u20131182, Cambridge, MA."},{"key":"R61","unstructured":"Reisinger, Joseph and Raymond J. Mooney. 2010b. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109\u2013117, Los Angeles, CA."},{"key":"R62","unstructured":"Resnik, Philip. 1995. Using information content to evaluate semantic similarity in a taxonomy. In Proceedings of IJCAI."},{"key":"R63","doi-asserted-by":"publisher","DOI":"10.1002\/9781444324044.ch11"},{"key":"R64","doi-asserted-by":"publisher","DOI":"10.1037\/0096-1523.2.4.491"},{"key":"R65","unstructured":"Rose, Tony, Mark Stevenson, and Miles Whitehead. 2002. The Reuters corpus volume 1\u2014from yesterday's news to tomorrow's language resources. In LREC, volume 2, pages 827\u2013832, Las Palmas."},{"key":"R66","doi-asserted-by":"publisher","DOI":"10.1145\/365628.365657"},{"key":"R69","unstructured":"Turian, Joseph, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384\u2013394, Uppsala."},{"key":"R70","doi-asserted-by":"publisher","DOI":"10.1613\/jair.3640"},{"key":"R71","doi-asserted-by":"publisher","DOI":"10.1613\/jair.2934"},{"key":"R72","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.84.4.327"},{"key":"R73","unstructured":"Wiebe, Janyce. 2000. Learning subjective adjectives from corpora. In AAAI\/IAAI, pages 735\u2013740, Austin, TX."},{"key":"R74","doi-asserted-by":"crossref","unstructured":"Williams, Gbolahan K. and Sarabjot Singh Anand. 2009. Predicting the polarity SimLex-999: Evaluating Semantic Models strength of adjectives using Wordnet. In ICWSM, San Jose, CA.","DOI":"10.1609\/icwsm.v3i1.13995"},{"key":"R75","doi-asserted-by":"crossref","unstructured":"Wu, Zhibiao and Martha Palmer. 1994. Verbs, semantics and lexical selection. In Proceedings of ACL, pages 133\u2013138, Las Cruces, NM.","DOI":"10.3115\/981732.981751"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00237","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T03:36:41Z","timestamp":1692157001000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/41\/4\/665-695\/1517"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,12]]},"references-count":67,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2015,12]]}},"alternative-id":["10.1162\/COLI_a_00237"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00237","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,12]]}}}