{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T16:45:35Z","timestamp":1773247535459,"version":"3.50.1"},"reference-count":58,"publisher":"MIT Press","issue":"1","license":[{"start":{"date-parts":[[2024,4,24]],"date-time":"2024-04-24T00:00:00Z","timestamp":1713916800000},"content-version":"vor","delay-in-days":114,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Despite the success of Transformer-based language models in a wide variety of natural language processing tasks, our understanding of how these models process a given input in order to represent task-relevant information remains incomplete. In this work, we focus on semantic composition and examine how Transformer-based language models represent semantic information related to the meaning of English noun-noun compounds. We probe Transformer-based language models for their knowledge of the thematic relations that link the head nouns and modifier words of compounds (e.g., KITCHEN CHAIR: a chair located in a kitchen). Firstly, using a dataset featuring groups of compounds with shared lexical or semantic features, we find that token representations of six Transformer-based language models distinguish between pairs of compounds based on whether they use the same thematic relation. Secondly, we utilize fine-grained vector representations of compound semantics derived from human annotations, and find that token vectors from several models elicit a strong signal of the semantic relations used in the compounds. In a novel \u201ccompositional probe\u201d setting, where we compare the semantic relation signal in mean-pooled token vectors of compounds to mean-pooled token vectors when the two constituent words appear in separate sentences, we find that the Transformer-based language models that best represent the semantics of noun-noun compounds also do so substantially better than in the control condition where the two constituent works are processed separately. Overall, our results shed light on the ability of Transformer-based language models to support compositional semantic processes in representing the meaning of noun-noun compounds.<\/jats:p>","DOI":"10.1162\/coli_a_00495","type":"journal-article","created":{"date-parts":[[2023,11,15]],"date-time":"2023-11-15T19:25:57Z","timestamp":1700076357000},"page":"49-81","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":3,"title":["How Is a \u201cKitchen Chair\u201d like a \u201cFarm\n                    Horse\u201d? Exploring the Representation of Noun-Noun Compound Semantics in\n                    Transformer-based Language Models"],"prefix":"10.1162","volume":"50","author":[{"given":"Mark","family":"Ormerod","sequence":"first","affiliation":[{"name":"Queen\u2019s University Belfast. mormerod01@qub.ac.uk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jes\u00fas Mart\u00ednez","family":"del Rinc\u00f3n","sequence":"additional","affiliation":[{"name":"Queen\u2019s University Belfast"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Barry","family":"Devereux","sequence":"additional","affiliation":[{"name":"Queen\u2019s University Belfast"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,3,1]]},"reference":[{"key":"2024042419445055500_bib1","doi-asserted-by":"publisher","first-page":"191","DOI":"10.18653\/v1\/W19-4820","article-title":"Blackbox meets blackbox: Representational\n                        similarity & stability analysis of neural language models and\n                        brains","volume-title":"Proceedings of the 2019 ACL Workshop\n                        BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Abnar","year":"2019"},{"key":"2024042419445055500_bib2","article-title":"Proceedings of the Third\n                            BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for\n                            NLP","author":"Alishahi","year":"2020"},{"issue":"18","key":"2024042419445055500_bib3","doi-asserted-by":"publisher","first-page":"4100","DOI":"10.1523\/JNEUROSCI.1152-20.2021","article-title":"Deep artificial neural networks reveal a\n                        distributed cortical network encoding propositional sentence-level\n                        meaning","volume":"41","author":"Anderson","year":"2021","journal-title":"Journal of Neuroscience"},{"issue":"1791","key":"2024042419445055500_bib4","doi-asserted-by":"publisher","first-page":"20190307","DOI":"10.1098\/rstb.2019.0307","article-title":"Linguistic generalization and\n                        compositionality in modern artificial neural networks","volume":"375","author":"Baroni","year":"2020","journal-title":"Philosophical Transactions of the Royal Society B"},{"key":"2024042419445055500_bib5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-acl.169","article-title":"From chocolate bunny to chocolate\n                        crocodile: Do Language models understand noun\n                        compounds?","author":"Coil","year":"2023","journal-title":"arXiv preprint arXiv:2305.10568"},{"key":"2024042419445055500_bib6","doi-asserted-by":"publisher","first-page":"619","DOI":"10.18653\/v1\/2021.emnlp-main.49","article-title":"The devil is in the detail: Simple tricks\n                        improve systematic generalization of transformers","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural\n                        Language Processing","author":"Csord\u00e1s","year":"2021"},{"issue":"3\u20134","key":"2024042419445055500_bib7","doi-asserted-by":"publisher","first-page":"489","DOI":"10.1007\/s10462-005-9007-5","article-title":"Investigating the relations used in\n                        conceptual combination","volume":"24","author":"Devereux","year":"2005","journal-title":"Artificial Intelligence\n                        Review"},{"key":"2024042419445055500_bib8","article-title":"Modelling the interpretation and\n                        interpretation ease of noun-noun compounds using a relation space approach\n                        to compound meaning","volume-title":"28th Annual Conference of\n                        the Cognitive Science Society","author":"Devereux","year":"2006"},{"key":"2024042419445055500_bib9","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1007\/978-3-642-31863-4_8","article-title":"Learning to interpret novel noun-noun\n                        compounds: Evidence from category learning experiments","volume-title":"Cognitive Aspects of Computational Language Acquisition","author":"Devereux","year":"2012"},{"key":"2024042419445055500_bib10","article-title":"BERT: Pre-training of deep bidirectional\n                        transformers for language understanding","author":"Devlin","year":"2018","journal-title":"CoRR\n                        abs\/1810.04805"},{"key":"2024042419445055500_bib11","doi-asserted-by":"publisher","first-page":"810","DOI":"10.2307\/412913","article-title":"On the creation and use of English\n                        compound nouns","author":"Downing","year":"1977","journal-title":"Language"},{"key":"2024042419445055500_bib12","doi-asserted-by":"publisher","DOI":"10.1201\/9781420011814","volume-title":"Randomization Tests","author":"Edgington","year":"2007"},{"key":"2024042419445055500_bib13","first-page":"226","article-title":"A density-based algorithm for discovering\n                        clusters in large spatial databases with noise","volume-title":"The International Conference on Knowledge Discovery and Data\n                        Mining","author":"Ester","year":"1996"},{"key":"2024042419445055500_bib14","doi-asserted-by":"publisher","first-page":"1082","DOI":"10.1037\/0278-7393.30.5.1082","article-title":"The importance of being nonalignable: A\n                        critical test of the structural alignment theory of\n                        similarity","volume":"30","author":"Estes","year":"2004","journal-title":"Journal of Experimental Psychology:\n                        Learning, Memory and Cognition"},{"key":"2024042419445055500_bib15","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1162\/tacl_a_00298","article-title":"What BERT is not: Lessons from a new suite\n                        of psycholinguistic diagnostics for language models","volume":"8","author":"Ettinger","year":"2020","journal-title":"Transactions of the Association for Computational\n                        Linguistics"},{"key":"2024042419445055500_bib16","doi-asserted-by":"publisher","first-page":"1488","DOI":"10.18653\/v1\/D18-1178","article-title":"Transfer and multi-task learning for\n                        noun\u2013noun compound interpretation","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural\n                        Language Processing","author":"Fares","year":"2018"},{"key":"2024042419445055500_bib17","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1037\/\/0278-7393.23.1.71","article-title":"Influence of thematic relations on the\n                        comprehension of modifier-noun combinations","volume":"23","author":"Gagn\u00e9","year":"1997","journal-title":"Journal\n                        of Experimental Psychology: Learning, Memory, and Cognition"},{"issue":"1","key":"2024042419445055500_bib18","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1037\/\/0278-7393.27.1.236","article-title":"Relation and lexical priming during the\n                        interpretation of noun\u2013noun combinations","volume":"27","author":"Gagn\u00e9","year":"2001","journal-title":"Journal of Experimental Psychology: Learning, Memory, and\n                        Cognition"},{"key":"2024042419445055500_bib19","doi-asserted-by":"publisher","first-page":"529","DOI":"10.18653\/v1\/D19-1050","article-title":"Linking artificial and human neural representations of\n                        language","volume-title":"Proceedings of the 2019 Conference on\n                        Empirical Methods in Natural Language Processing and the 9th International\n                        Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Gauthier","year":"2019"},{"key":"2024042419445055500_bib20","doi-asserted-by":"publisher","first-page":"68","DOI":"10.3115\/1596431.1596441","article-title":"Support vector machines applied to the\n                        classification of semantic relations in nominalized noun\n                        phrases","volume-title":"Proceedings of the Computational Lexical\n                        Semantics Workshop at HLT-NAACL 2004","author":"Girju","year":"2004"},{"key":"2024042419445055500_bib21","doi-asserted-by":"publisher","first-page":"4","DOI":"10.3389\/neuro.06.004.2008","article-title":"Representational similarity\n                        analysis\u2013connecting the branches of systems\n                        neuroscience","volume":"2","author":"Kriegeskorte","year":"2008","journal-title":"Frontiers in Systems\n                        Neuroscience"},{"key":"2024042419445055500_bib22","article-title":"Cross-lingual language model\n                        pretraining","author":"Lample","year":"2019","journal-title":"arXiv preprint\n                        arXiv:1901.07291"},{"issue":"3","key":"2024042419445055500_bib23","first-page":"205","article-title":"The grammar of English nominalizations","volume":"26","author":"Lees","year":"1960","journal-title":"International Journal of American Linguistics"},{"key":"2024042419445055500_bib24","volume-title":"The Syntax and Semantics of Complex\n                    Nominals","author":"Levi","year":"1978"},{"key":"2024042419445055500_bib25","doi-asserted-by":"publisher","first-page":"717","DOI":"10.18653\/v1\/2022.findings-emnlp.50","article-title":"Systematicity in GPT-3\u2019s interpretation\n                        of novel English noun compounds","volume-title":"Findings of the\n                        Association for Computational Linguistics: EMNLP 2022","author":"Li","year":"2022"},{"issue":"1","key":"2024042419445055500_bib26","doi-asserted-by":"publisher","first-page":"e99\u2013e108","DOI":"10.1353\/lan.2019.0015","article-title":"What can linguistics and deep learning\n                        contribute to each other? Response to Pater","volume":"95","author":"Linzen","year":"2019","journal-title":"Language"},{"key":"2024042419445055500_bib27","article-title":"RoBERTa: A robustly optimized BERT\n                        pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv preprint\n                        arXiv:1907.11692 [cs]"},{"key":"2024042419445055500_bib28","doi-asserted-by":"publisher","first-page":"212","DOI":"10.3389\/fpsyg.2010.00212","article-title":"Embodied conceptual\n                        combination","volume":"1","author":"Lynott","year":"2010","journal-title":"Frontiers in Psychology"},{"key":"2024042419445055500_bib29","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1037\/0278-7393.33.4.811","article-title":"A re-analysis of the CARIN theory of\n                        conceptual combination","volume":"33","author":"Maguire","year":"2007","journal-title":"Journal of Experimental\n                        Psychology, Learning Memory and Cognition"},{"key":"2024042419445055500_bib30","doi-asserted-by":"publisher","first-page":"217","DOI":"10.18653\/v1\/2020.blackboxnlp-1.21","article-title":"BERTs of a feather do not generalize\n                        together: Large variability in generalization across models with similar\n                        test set performance","volume-title":"Proceedings of the Third\n                        BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for\n                        NLP","author":"McCoy","year":"2020"},{"key":"2024042419445055500_bib31","unstructured":"Merity,\n                                Stephen,\n                                CaimingXiong,\n                                JamesBradbury, and\n                                RichardSocher.\n                        2016. Wikitext-103.\n                        Technical report,\n                    Salesforce."},{"issue":"8","key":"2024042419445055500_bib32","doi-asserted-by":"publisher","first-page":"1388","DOI":"10.1111\/j.1551-6709.2010.01106.x","article-title":"Composition in distributional models of\n                        semantics","volume":"34","author":"Mitchell","year":"2010","journal-title":"Cognitive Science"},{"issue":"5880","key":"2024042419445055500_bib33","doi-asserted-by":"publisher","first-page":"1191","DOI":"10.1126\/science.1152876","article-title":"Predicting human brain activity associated with the meanings\n                        of Nouns","volume":"320","author":"Mitchell","year":"2008","journal-title":"Science"},{"issue":"4","key":"2024042419445055500_bib34","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1016\/0364-0213(88)90012-2","article-title":"Comprehending complex\n                    concepts","volume":"12","author":"Murphy","year":"1988","journal-title":"Cognitive Science"},{"key":"2024042419445055500_bib35","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1602.001.0001","volume-title":"The Big Book of Concepts","author":"Murphy","year":"2002"},{"key":"2024042419445055500_bib36","article-title":"Characterizing intrinsic compositionality\n                        in transformers with tree projections","author":"Murty","year":"2022","journal-title":"arXiv\n                        preprint arXiv:2211.01288"},{"key":"2024042419445055500_bib37","article-title":"Paraphrasing verbs for noun compound\n                        interpretation","author":"Nakov","year":"2019","journal-title":"arXiv preprint\n                        arXiv:1911.08762"},{"issue":"4","key":"2024042419445055500_bib38","doi-asserted-by":"publisher","first-page":"e1003553","DOI":"10.1371\/journal.pcbi.1003553","article-title":"A toolbox for representational similarity\n                        analysis","volume":"10","author":"Nili","year":"2014","journal-title":"PLoS Computational Biology"},{"key":"2024042419445055500_bib39","doi-asserted-by":"publisher","first-page":"3591","DOI":"10.18653\/v1\/2022.acl-long.251","article-title":"Making transformers solve compositional\n                        tasks","volume-title":"Proceedings of the 60th Annual Meeting of\n                        the Association for Computational Linguistics (Volume 1: Long\n                        Papers)","author":"Ontanon","year":"2022"},{"issue":"8","key":"2024042419445055500_bib40","first-page":"9","article-title":"Language models are unsupervised multitask\n                        learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2024042419445055500_bib41","first-page":"210","article-title":"An empirical study on compositionality in\n                        compound nouns","volume-title":"Proceedings of 5th International\n                        Joint Conference on Natural Language Processing","author":"Reddy","year":"2011"},{"key":"2024042419445055500_bib42","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1162\/tacl_a_00349","article-title":"A primer in BERTology: What we know about\n                        how BERT works","volume":"8","author":"Rogers","year":"2021","journal-title":"Transactions of the Association for\n                        Computational Linguistics"},{"key":"2024042419445055500_bib43","article-title":"DistilBERT, a distilled version of BERT: Smaller, faster,\n                        cheaper and lighter","author":"Sanh","year":"2020","journal-title":"arXiv preprint\n                        arXiv:1910.01108"},{"key":"2024042419445055500_bib44","article-title":"The multiBERTs: BERT reproductions for robustness\n                        analysis","author":"Sellam","year":"2021","journal-title":"arXiv preprint\n                    arXiv:2106.16163"},{"key":"2024042419445055500_bib45","doi-asserted-by":"publisher","first-page":"1200","DOI":"10.18653\/v1\/P18-1111","article-title":"Paraphrase to explicate: Revealing implicit\n                        noun-compound relations","volume-title":"Proceedings of the 56th\n                        Annual Meeting of the Association for Computational Linguistics (Volume 1:\n                        Long Papers)","author":"Shwartz","year":"2018"},{"key":"2024042419445055500_bib46","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1162\/tacl_a_00277","article-title":"Still a Pain in the neck: Evaluating text\n                        representations on lexical composition","volume":"7","author":"Shwartz","year":"2019","journal-title":"Transactions\n                        of the Association for Computational Linguistics"},{"key":"2024042419445055500_bib47","doi-asserted-by":"publisher","first-page":"4593","DOI":"10.18653\/v1\/P19-1452","article-title":"BERT rediscovers the classical NLP\n                        pipeline","volume-title":"Proceedings of the 57th Annual Meeting\n                        of the Association for Computational Linguistics","author":"Tenney","year":"2019"},{"key":"2024042419445055500_bib48","first-page":"678","article-title":"A taxonomy, dataset, and classifier for automatic noun\n                        compound interpretation","volume-title":"Proceedings of the 48th\n                        Annual Meeting of the Association for Computational Linguistics","author":"Tratz","year":"2010"},{"issue":"6","key":"2024042419445055500_bib49","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1007\/BF01067911","article-title":"Frequency effects in the processing of\n                        lexicalized and novel nominal compounds","volume":"17","author":"van Jaarsveld","year":"1988","journal-title":"Journal of\n                        Psycholinguistic Research"},{"key":"2024042419445055500_bib50","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"Advances In Neural Information Processing\n                    Systems"},{"key":"2024042419445055500_bib51","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1007\/978-3-319-45977-6_11","article-title":"How does the left anterior temporal lobe\n                        contribute to conceptual combination? Interdisciplinary\n                        perspectives","volume-title":"Compositionality and Concepts in\n                        Linguistics and Psychology","author":"Westerlund","year":"2017"},{"key":"2024042419445055500_bib52","doi-asserted-by":"publisher","first-page":"1832","DOI":"10.3389\/fpsyg.2016.01832","article-title":"Degrees of freedom in planning, running,\n                        analyzing, and reporting psychological studies: A checklist to avoid\n                        p-hacking","author":"Wicherts","year":"2016","journal-title":"Frontiers in Psychology"},{"key":"2024042419445055500_bib53","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language\n                        processing","volume-title":"Proceedings of the 2020 Conference on\n                        Empirical Methods in Natural Language Processing: System\n                        Demonstrations","author":"Wolf","year":"2020"},{"key":"2024042419445055500_bib54","doi-asserted-by":"publisher","first-page":"2017","DOI":"10.18653\/v1\/D16-1213","article-title":"BrainBench: A brain-image test suite for\n                        distributional semantic models","volume-title":"Proceedings of\n                        the 2016 Conference on Empirical Methods in Natural Language\n                        Processing","author":"Xu","year":"2016"},{"key":"2024042419445055500_bib55","article-title":"XLNet: Generalized autoregressive pretraining for language\n                        understanding","volume":"32","author":"Yang","year":"2019","journal-title":"Advances In Neural Information\n                        Processing Systems"},{"key":"2024042419445055500_bib56","doi-asserted-by":"publisher","first-page":"4896","DOI":"10.18653\/v1\/2020.emnlp-main.397","article-title":"Assessing phrasal representation and\n                        composition in transformers","volume-title":"Proceedings of the\n                        2020 Conference on Empirical Methods in Natural Language Processing\n                        (EMNLP)","author":"Yu","year":"2020"},{"key":"2024042419445055500_bib57","doi-asserted-by":"publisher","first-page":"2279","DOI":"10.18653\/v1\/2021.findings-acl.201","article-title":"On the interplay between fine-tuning and\n                        composition in transformers","volume-title":"Findings of the\n                        Association for Computational Linguistics: ACL-IJCNLP 2021","author":"Yu","year":"2021"},{"key":"2024042419445055500_bib58","doi-asserted-by":"publisher","first-page":"57","DOI":"10.3115\/1613704.1613712","article-title":"Co-occurrence contexts for noun compound\n                        interpretation","volume-title":"Proceedings of the Workshop on a\n                        Broader Perspective on Multiword Expressions","author":"\u00d3 S\u00e9aghdha","year":"2007"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/1\/49\/2367163\/coli_a_00495.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/50\/1\/49\/2367163\/coli_a_00495.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,24]],"date-time":"2024-04-24T19:45:19Z","timestamp":1713987919000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/50\/1\/49\/118133\/How-Is-a-Kitchen-Chair-like-a-Farm-Horse-Exploring"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":58,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,3,1]]},"published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00495","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}