{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T17:58:03Z","timestamp":1775066283304,"version":"3.50.1"},"reference-count":105,"publisher":"Wiley","issue":"5","license":[{"start":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T00:00:00Z","timestamp":1684108800000},"content-version":"vor","delay-in-days":14,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Cognitive Science"],"published-print":{"date-parts":[[2023,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Distributional semantic models (DSMs) are a primary method for distilling semantic information from corpora. However, a key question remains: What types of semantic relations among words do DSMs detect? Prior work typically has addressed this question using limited human data that are restricted to semantic similarity and\/or general semantic relatedness. We tested eight DSMs that are popular in current cognitive and psycholinguistic research (positive pointwise mutual information; global vectors; and three variations each of Skip\u2010gram and continuous bag of words (CBOW) using word, context, and mean embeddings) on a theoretically motivated, rich set of semantic relations involving words from multiple syntactic classes and spanning the abstract\u2013concrete continuum (19 sets of ratings). We found that, overall, the DSMs are best at capturing overall semantic similarity and also can capture verb\u2013noun thematic role relations and noun\u2013noun event\u2010based relations that play important roles in sentence comprehension. Interestingly, Skip\u2010gram and CBOW performed the best in terms of capturing similarity, whereas GloVe dominated the thematic role and event\u2010based relations. We discuss the theoretical and practical implications of our results, make recommendations for users of these models, and demonstrate significant differences in model performance on event\u2010based relations.<\/jats:p>","DOI":"10.1111\/cogs.13291","type":"journal-article","created":{"date-parts":[[2023,5,15]],"date-time":"2023-05-15T07:37:00Z","timestamp":1684136220000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Investigating the Extent to which Distributional Semantic Models Capture a Broad Range of Semantic Relations"],"prefix":"10.1111","volume":"47","author":[{"given":"Kevin S.","family":"Brown","sequence":"first","affiliation":[{"name":"Department of Pharmaceutical Sciences Oregon State University"},{"name":"School of Chemical, Biological, and Environmental Engineering Oregon State University"}]},{"given":"Eiling","family":"Yee","sequence":"additional","affiliation":[{"name":"Department of Psychological Sciences University of Connecticut"}]},{"given":"Gitte","family":"Joergensen","sequence":"additional","affiliation":[{"name":"Department of Psychological Sciences University of Connecticut"}]},{"given":"Melissa","family":"Troyer","sequence":"additional","affiliation":[{"name":"Department of Psychology University of Western Ontario"}]},{"given":"Elliot","family":"Saltzman","sequence":"additional","affiliation":[{"name":"Department of Physical Therapy Boston University"}]},{"given":"Jay","family":"Rueckl","sequence":"additional","affiliation":[{"name":"Department of Psychological Sciences University of Connecticut"}]},{"given":"James S.","family":"Magnuson","sequence":"additional","affiliation":[{"name":"Department of Psychological Sciences University of Connecticut"},{"name":"BCBL, Basque Center on Cognition, Brain, &amp; Language"},{"name":"Ikerbasque, Basque Foundation for Science"}]},{"given":"Ken","family":"McRae","sequence":"additional","affiliation":[{"name":"Department of Psychology University of Western Ontario"}]}],"member":"311","published-online":{"date-parts":[[2023,5,15]]},"reference":[{"key":"e_1_2_10_2_1","volume-title":"The Hitchhiker's guide to the galaxy","author":"Adams D.","year":"1979"},{"key":"e_1_2_10_3_1","doi-asserted-by":"crossref","unstructured":"Agirre E. Alfonseca E. Hall K. Kravalova J. Pasca M. &Soroa A.(2009).A study on similarity and relatedness using distributional and wordnet\u2010based approaches.Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics NAACL \u201909 Boulder CO (pp.19\u201327).","DOI":"10.3115\/1620754.1620758"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0016261"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1177\/17470218221090483"},{"key":"e_1_2_10_6_1","unstructured":"Asr F. T. Zinkov R. &Jones M.(2018).Querying word embeddings for similarity and relatedness.Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume 1: Long Papers New Orleans LA(pp.675\u2013684)."},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00016"},{"key":"e_1_2_10_8_1","doi-asserted-by":"crossref","unstructured":"Baroni M. Dinu G. &Kruszewski G.(2014).Don't count predict! A systematic comparison of context\u2010counting vs. context\u2010predicting semantic vectors.Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics; Volume 1: Long Papers Baltimore MD(pp.238\u2013247).","DOI":"10.3115\/v1\/P14-1023"},{"key":"e_1_2_10_9_1","doi-asserted-by":"crossref","unstructured":"Bommasani R. Davis K. &Cardie C.(2020).Interpreting pretrained contextualized representations via reductions to static embeddings.Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics Online (pp.4758\u20134781).","DOI":"10.18653\/v1\/2020.acl-main.431"},{"key":"e_1_2_10_10_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-013-0403-5"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-019-01243-z"},{"key":"e_1_2_10_12_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03193020"},{"key":"e_1_2_10_13_1","first-page":"263","article-title":"Thematic roles and language comprehension","volume":"21","author":"Carlson G. N.","year":"1988","journal-title":"Syntax and Semantics"},{"key":"e_1_2_10_14_1","doi-asserted-by":"crossref","unstructured":"Chronis G. &Erk K.(2020).When is a bishop not like a rook? When it's like a rabbi! Multi\u2010prototype BERT embeddings for estimating semantic relationships.Proceedings of the 24th Conference on Computational Natural Language Learning Online (pp.227\u2013244).","DOI":"10.18653\/v1\/2020.conll-1.17"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-5371(69)80069-1"},{"key":"e_1_2_10_16_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0034626"},{"key":"e_1_2_10_17_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-012-0260-7"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-018-1115-7"},{"key":"e_1_2_10_19_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0000_9"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1366728922000049"},{"key":"e_1_2_10_21_1","unstructured":"Devlin J. Chang M.\u2010W. Lee K. &Toutanova K.(2019)BERT: Pre\u2010training of deep bidrectional transformere for language understanding. arXiv 1810.04805."},{"key":"e_1_2_10_22_1","doi-asserted-by":"publisher","DOI":"10.1080\/02643294.2019.1656604"},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00017"},{"key":"e_1_2_10_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-385527-5.00008-5"},{"key":"e_1_2_10_25_1","doi-asserted-by":"crossref","unstructured":"Ethayarajh K.(2019).How contextual are contextualized word representations? Comparing the geometry of BERT ELMo and GPT\u20102 embeddings.Proceedings of the 29th Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing Hong Kong China (pp.55\u201365).","DOI":"10.18653\/v1\/D19-1006"},{"key":"e_1_2_10_26_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00298"},{"key":"e_1_2_10_27_1","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00237"},{"key":"e_1_2_10_28_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2108091119"},{"key":"e_1_2_10_29_1","doi-asserted-by":"publisher","DOI":"10.1006\/jmla.2000.2728"},{"key":"e_1_2_10_30_1","doi-asserted-by":"crossref","unstructured":"Finkelstein L. Gabrilovich E. Matias Y. Rivlin E. Solan Z. Wolfman G. &Ruppin E.(2002).Placing search in context: The concept revisited.Proceedings of the 10th International Conference on World Wide Web Hong Kong Hong Kong(pp.406\u2013414).","DOI":"10.1145\/371920.372094"},{"key":"e_1_2_10_31_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511620669.010"},{"key":"e_1_2_10_32_1","doi-asserted-by":"crossref","unstructured":"Gerz D. Vuli I. Hill F. Reichart R. &Korhonen A.(2016).SimVerb\u20103500: A large\u2010scale evaluation set of verb similarity.Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing Austin TX(pp.2173\u20132182).","DOI":"10.18653\/v1\/D16-1235"},{"key":"e_1_2_10_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuropsychologia.2013.11.010"},{"key":"e_1_2_10_34_1","doi-asserted-by":"publisher","DOI":"10.1044\/2020_JSLHR-20-00283"},{"key":"e_1_2_10_35_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-020-01352-0"},{"key":"e_1_2_10_36_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.114.2.211"},{"key":"e_1_2_10_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2009.01.009"},{"key":"e_1_2_10_38_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9280.2009.02365.x"},{"key":"e_1_2_10_39_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-012-0304-z"},{"key":"e_1_2_10_40_1","doi-asserted-by":"publisher","DOI":"10.1111\/cogs.12730"},{"key":"e_1_2_10_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogpsych.2021.101441"},{"key":"e_1_2_10_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2006.07.003"},{"key":"e_1_2_10_43_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-015-0679-8"},{"key":"e_1_2_10_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-019-09452-w"},{"key":"e_1_2_10_45_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03196852"},{"key":"e_1_2_10_46_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btr709"},{"key":"e_1_2_10_47_1","doi-asserted-by":"publisher","DOI":"10.1111\/tops.12548"},{"key":"e_1_2_10_48_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03196761"},{"key":"e_1_2_10_49_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.104.2.211"},{"key":"e_1_2_10_50_1","doi-asserted-by":"publisher","DOI":"10.1037\/rev0000297"},{"key":"e_1_2_10_51_1","unstructured":"Lapesa G. &Evert S.(2013).Evaluating neighbor rank and distance measures as predictors of semantic priming.Proceedings of the ACL Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2013) Minneapolis MN(pp.66\u201374."},{"key":"e_1_2_10_52_1","doi-asserted-by":"crossref","unstructured":"Lapesa G. Evert S. &Schulte im Walde S.(2014).Contrasting syntagmatic and paradigmatic relations: Insights from distributional semantic models.Proceedings of the Third Joint Conference on Lexical and Computational Semantics (SEM 2014) Dublin Ireland(pp.160\u2013170.","DOI":"10.3115\/v1\/S14-1020"},{"key":"e_1_2_10_53_1","doi-asserted-by":"crossref","unstructured":"Lazaridou A. Pham N. T. &Baroni M.(2015).Combining language and vision with a multimodal Skip\u2010gram model. arXiv:1501.02598.","DOI":"10.3115\/v1\/N15-1016"},{"key":"e_1_2_10_54_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-linguistics-030514-125254"},{"key":"e_1_2_10_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-021-09575-z"},{"key":"e_1_2_10_56_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00134"},{"key":"e_1_2_10_57_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1910148116"},{"key":"e_1_2_10_58_1","doi-asserted-by":"crossref","unstructured":"Liu N. F. Gardner M. Belinkov Y. Peters M. E. &Smith N. A.(2019).Linguistic knowledge and transferability of contextual representations.Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume 1 (Long and Short Papers) Minneapolis MN (pp.1073\u20131094.","DOI":"10.18653\/v1\/N19-1112"},{"key":"e_1_2_10_59_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03204766"},{"key":"e_1_2_10_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2016.04.001"},{"key":"e_1_2_10_61_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.57.102904.190143"},{"key":"e_1_2_10_62_1","doi-asserted-by":"publisher","DOI":"10.1177\/09637214221078325"},{"key":"e_1_2_10_63_1","doi-asserted-by":"publisher","DOI":"10.4324\/9780203338001"},{"key":"e_1_2_10_64_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03192726"},{"key":"e_1_2_10_65_1","doi-asserted-by":"publisher","DOI":"10.1037\/13493-002"},{"key":"e_1_2_10_66_1","doi-asserted-by":"publisher","DOI":"10.1006\/jmla.1997.2543"},{"key":"e_1_2_10_67_1","unstructured":"Mikolov T. Chen K. Corrado G. &Dean J.(2013a).Efficient estimation of word representations in vector space. arXiv:1301.3781."},{"key":"e_1_2_10_68_1","unstructured":"Mikholov T. Sutskever I. Chen K. Corrado G. &Dean J.(2013b).Distributed representations of words and phrases and their compositionality.Advances in Neural Information Processing Systems 28(2013) Lake Tahoe NV."},{"key":"e_1_2_10_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"e_1_2_10_70_1","doi-asserted-by":"publisher","DOI":"10.1037\/0278-7393.34.1.65"},{"key":"e_1_2_10_71_1","unstructured":"Mohajer M. Englmeier K.\u2010H. &Schmid V. J.(2010).A comparison of Gap statistic with and with\u2010out logarithm function. Technical Report No. 096 Department of Statistics University of Munich."},{"key":"e_1_2_10_72_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2013.00128"},{"key":"e_1_2_10_73_1","unstructured":"Musz E. Yee E. &Thompson\u2010Schill S. L.(2012).Mapping the similarity space of concepts in sensorimotor cortex.Poster presented at the 2012 Meeting of the Cognitive Neuroscience Society Chicago IL:."},{"key":"e_1_2_10_74_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03195588"},{"key":"e_1_2_10_75_1","doi-asserted-by":"publisher","DOI":"10.1037\/xlm0001208"},{"key":"e_1_2_10_76_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2022.841466"},{"key":"e_1_2_10_77_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2013.00838"},{"key":"e_1_2_10_78_1","doi-asserted-by":"crossref","unstructured":"Pennington J. Socher R. &Manning C. D.(2014).Glove: Global vectors for word representation.Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) Doha Qatar(pp.1532\u20131543).","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_2_10_79_1","doi-asserted-by":"publisher","DOI":"10.3758\/PBR.15.1.161"},{"key":"e_1_2_10_80_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-016-0720-6"},{"key":"e_1_2_10_81_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2022.08.010"},{"key":"e_1_2_10_82_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2014.03.010"},{"key":"e_1_2_10_83_1","doi-asserted-by":"publisher","DOI":"10.1080\/23273798.2021.2022171"},{"key":"e_1_2_10_84_1","doi-asserted-by":"crossref","unstructured":"Rapp R.(2002).The computation of word associations: Comparing syntagmatic and paradigmatic approaches.Proceedings of the 19th International Conference on Computational Linguistics New Delhi India (pp.1\u20137).","DOI":"10.3115\/1072228.1072235"},{"key":"e_1_2_10_85_1","unstructured":"Rehurek R. &Sojka P.(2011).Gensim\u2013python framework for vector space modelling. NLP Centre Faculty of Informatics Masaryk University Brno Czech Republic 3(2)."},{"key":"e_1_2_10_86_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.111.1.205"},{"key":"e_1_2_10_87_1","doi-asserted-by":"publisher","DOI":"10.1111\/cogs.12690"},{"key":"e_1_2_10_88_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.274.5294.1926"},{"key":"e_1_2_10_89_1","unstructured":"Sahlgren M.(2006).The word\u2010space model: Using distributional analysis to represent syntagmatic and paradigmatic relations between words in high dimensional vector spaces. [PhD dissertation] University of Stockholm."},{"key":"e_1_2_10_90_1","doi-asserted-by":"crossref","unstructured":"Santus E. Chersoni E. Lenci A. &Blache P.(2017).Measuring thematic fit with distributional feature overlap.Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing Copenhagen Denmark(pp.659\u2013669).","DOI":"10.18653\/v1\/D17-1068"},{"key":"e_1_2_10_91_1","doi-asserted-by":"crossref","unstructured":"Sayeed A. Greenberg C. &Demberg V.(2016).Thematic fit evaluation: An aspect of selectional preferences.Proceedings of ACL Workshop for Evaluating Vector Space Representations for NLP Berlin Germany.","DOI":"10.18653\/v1\/W16-2518"},{"key":"e_1_2_10_92_1","first-page":"1409","article-title":"A statistical method for evaluating systematic relationships","volume":"38","author":"Sokal R.","year":"1958","journal-title":"University of Kansas Science Bulletin"},{"key":"e_1_2_10_93_1","doi-asserted-by":"publisher","DOI":"10.1038\/srep46730"},{"key":"e_1_2_10_94_1","doi-asserted-by":"publisher","DOI":"10.1177\/107769905303000401"},{"key":"e_1_2_10_95_1","unstructured":"Tenney I. Xia P. Chen B. Wang A. Poliak A. McCoy R. T. Kim N. Van Durme B. Bowman S. R. Das D. &Pavlick E.(2019).What do you learn from context? Probing for sentence structure in contextualized word representations.International Conference on Learning Representations 2019 New Orleans LA."},{"key":"e_1_2_10_96_1","doi-asserted-by":"crossref","unstructured":"Tilk O. Demberg V. Sayeed A. Klakow D. &Thater S.(2016).Event participant modelling with neural networks.Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing Austic TX (pp.171\u2013182).","DOI":"10.18653\/v1\/D16-1017"},{"key":"e_1_2_10_97_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2020.104111"},{"key":"e_1_2_10_98_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0136277"},{"key":"e_1_2_10_99_1","unstructured":"Vasmani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A. N. Kaiser L. &Polosukhin I.(2017).Attention is all you need.National Information Processing Systems NIPS 2017 Long Beach CA."},{"key":"e_1_2_10_100_1","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.40.1.183"},{"key":"e_1_2_10_101_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-95627-x"},{"key":"e_1_2_10_102_1","doi-asserted-by":"publisher","DOI":"10.1080\/23273798.2022.2069278"},{"key":"e_1_2_10_103_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611430691"},{"key":"e_1_2_10_104_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2009.12.036"},{"key":"e_1_2_10_105_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0022840"},{"key":"e_1_2_10_106_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-010419-051101"}],"container-title":["Cognitive Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/cogs.13291","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,20]],"date-time":"2023-08-20T08:18:57Z","timestamp":1692519537000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/cogs.13291"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5]]},"references-count":105,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,5]]}},"alternative-id":["10.1111\/cogs.13291"],"URL":"https:\/\/doi.org\/10.1111\/cogs.13291","archive":["Portico"],"relation":{},"ISSN":["0364-0213","1551-6709"],"issn-type":[{"value":"0364-0213","type":"print"},{"value":"1551-6709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5]]},"assertion":[{"value":"2021-11-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-07","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e13291"}}