{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T10:38:27Z","timestamp":1790419107122,"version":"4.1.0"},"reference-count":75,"publisher":"MIT Press","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2016,12]]},"abstract":"<jats:p>This article introduces RELPRON, a large data set of subject and object relative clauses, for the evaluation of methods in compositional distributional semantics. RELPRON targets an intermediate level of grammatical complexity between content-word pairs and full sentences. The task involves matching terms, such as \u201cwisdom,\u201d with representative properties, such as \u201cquality that experience teaches.\u201d A unique feature of RELPRON is that it is built from attested properties, but without the need for them to appear in relative clause format in the source corpus. The article also presents some initial experiments on RELPRON, using a variety of composition methods including simple baselines, arithmetic operators on vectors, and finally, more complex methods in which argument-taking words are represented as tensors. The latter methods are based on the Categorial framework, which is described in detail. The results show that vector addition is difficult to beat\u2014in line with the existing literature\u2014but that an implementation of the Categorial framework based on the Practical Lexical Function model is able to match the performance of vector addition. The article finishes with an in-depth analysis of RELPRON, showing how results vary across subject and object relative clauses, across different head nouns, and how the methods perform on the subtasks necessary for capturing relative clause semantics, as well as providing a qualitative analysis highlighting some of the more common errors. Our hope is that the competitive results presented here, in which the best systems are on average ranking one out of every two properties correctly for a given term, will inspire new approaches to the RELPRON ranking task and other tasks based on linguistically interesting constructions.<\/jats:p>","DOI":"10.1162\/coli_a_00263","type":"journal-article","created":{"date-parts":[[2016,9,28]],"date-time":"2016-09-28T14:20:41Z","timestamp":1475072441000},"page":"661-701","source":"Crossref","is-referenced-by-count":18,"title":["RELPRON: A Relative Clause Evaluation Data Set for Compositional Distributional Semantics"],"prefix":"10.1162","volume":"42","author":[{"given":"Laura","family":"Rimell","sequence":"first","affiliation":[{"name":"University of Cambridge Computer Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean","family":"Maillard","sequence":"additional","affiliation":[{"name":"University of Cambridge Computer Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tamara","family":"Polajnar","sequence":"additional","affiliation":[{"name":"University of Cambridge Computer Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen","family":"Clark","sequence":"additional","affiliation":[{"name":"University of Cambridge Computer Laboratory"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S15-2045"},{"key":"R2","unstructured":"Agirre, Eneko, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A pilot on Semantic Textual Similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM 2012), pages 385\u2013393, Montr\u00e9al."},{"key":"R3","unstructured":"Agirre, Eneko, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and WeiWei Guo. 2013. *SEM 2013 Shared Task: Semantic Textual Similarity. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*SEM 2013), pages 32\u201343, Atlanta, GA."},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Baroni, Marco, Raffaella Bernardi, and Roberto Zamparelli. 2014. Frege in space: A program for compositional distributional semantics. Linguistic Issues in Language Technologies, 9:5\u2013110.","DOI":"10.33011\/lilt.v9i.1321"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1023"},{"key":"R6","unstructured":"Baroni, Marco and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1183\u20131193, Boston, MA."},{"key":"R7","unstructured":"Bender, Emily M., Dan Flickinger, Stephan Oepen, and Yi Zhang. 2011. Parser evaluation over local and non-local deep dependencies in a large corpus. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 397\u2013408, Edinburgh, UK."},{"key":"R8","unstructured":"Bernardi, Raffaella, Georgiana Dinu, Marco Marelli, and Marco Baroni. 2013. A relatedness benchmark to test the role of determiners in compositional distributional semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), pages 53\u201357, Sofia, Bulgaria."},{"key":"R9","unstructured":"Biemann, Chris and Eugenie Giesbrecht. 2011. Distributional semantics and compositionality 2011: Shared task description and results. In Proceedings of the Workshop on Distributional Semantics and Compositionality (DiSCo 2011), pages 21\u201328, Portland, OR."},{"key":"R10","unstructured":"Boleda, Gemma, Marco Baroni, Nghia The Pham, and Louise McNally. 2013. Intensionality was only alleged: On adjective-noun composition in distributional semantics. In Proceedings of the Tenth International Conference on Computational Semantics (IWCS 2013), pages 35\u201346, Potsdam."},{"key":"R11","unstructured":"Boleda, Gemma, Eva Maria Vecchi, Miquel Cornudella, and Louise McNally. 2012. First order vs. higher order modification in distributional semantics. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 1223\u20131233, Jeju Island."},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-013-0403-5"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1082"},{"key":"R15","unstructured":"Cheung, Jackie C. K. and Gerald Penn. 2012. Evaluating distributional models of semantics for syntactically invariant inference. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2012), pages 33\u201343, Avignon."},{"key":"R18","unstructured":"Clark, Stephen, Bob Coecke, and Mehrnoosh Sadrzadeh. 2008. A compositional distributional model of meaning. In Proceedings of the Second Symposium on Quantum Interaction (QI 2008), pages 133\u2013140, Oxford."},{"key":"R19","unstructured":"Clark, Stephen, Bob Coecke, and Mehrnoosh Sadrzadeh. 2013. The Frobenius anatomy of relative pronouns. In Proceedings of the 13th Meeting on the Mathematics of Language (MoL 2013), pages 41\u201351, Sofia."},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2007.33.4.493"},{"key":"R21","unstructured":"Clark, Stephen and Stephen Pulman. 2007. Combining symbolic and distributional models of meaning. In Proceedings of the AAAI Spring Symposium on Quantum Interaction, pages 52\u201355, Stanford, CA."},{"key":"R22","unstructured":"Coecke, Bob, Mehrnoosh Sadrzadeh, and Stephen Clark. 2010. Mathematical foundations for a compositional distributional model of meaning. In Linguistic Analysis 36: A Festschrift for Joachim Lambek, pages 345\u2013384."},{"key":"R23","unstructured":"Curran, James R. 2004. From Distributional to Semantic Similarity. Ph.D. thesis, University of Edinburgh."},{"key":"R24","doi-asserted-by":"crossref","unstructured":"Dowty, David R., Robert E. Wall, and Stanley Peters. 1981. Introduction to Montague Semantics. Dordrecht.","DOI":"10.1007\/978-94-009-9065-4"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0277(88)90031-5"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Fried, Daniel, Tamara Polajnar, and Stephen Clark. 2015. Low-rank tensors for verbs in compositional distributional semantics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), pages 731\u2013736, Beijing.","DOI":"10.3115\/v1\/P15-2120"},{"key":"R27","unstructured":"Garrette, Dan, Katrin Erk, and Raymond Mooney. 2011. Integrating logical representations with probabilistic information using Markov Logic. In Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011), pages 105\u2013114, Oxford."},{"key":"R28","unstructured":"Grefenstette, Edward. 2013. Category-Theoretic Quantitative Compositional Distributional Models of Natural Language Semantics. Ph.D. thesis, University of Oxford."},{"key":"R29","unstructured":"Grefenstette, Edward, Georgiana Dinu, Yao-Zhong Zhang, Mehrnoosh Sadrzadeh, and Marco Baroni. 2013. Multi-step regression learning for compositional distributional semantics. In Proceedings of the Tenth International Conference on Computational Semantics (IWCS 2013), pages 131\u2013142, Potsdam."},{"key":"R30","unstructured":"Grefenstette, Edward and Mehrnoosh Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1394\u20131404, Edinburgh."},{"key":"R31","unstructured":"Grefenstette, Edward, Mehrnoosh Sadrzadeh, Stephen Clark, Bob Coecke, and Stephen Pulman. 2011. Concrete sentence spaces for compositional distributional models of meaning. In Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011), pages 125\u2013134, Oxford, UK."},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S15-1017"},{"key":"R35","unstructured":"Hermann, Karl Moritz, Edward Grefenstette, and Phil Blunsom. 2013. \u201cNot not bad\u201d is not \u201cbad\u201d: A distributional account of negation. In Proceedings of the ACL Workshop on Continuous Vector Space Models and their Compositionality (CVSC 2013), pages 74\u201382, Sofia."},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2007.33.3.355"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1970.10488634"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/S14-2003"},{"key":"R39","doi-asserted-by":"crossref","unstructured":"Kartsaklis, Dimitri and Mehrnoosh Sadrzadeh. 2013. Prior disambiguation of word tensors for constructing sentence vectors. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1590\u20131601, Seattle, WA.","DOI":"10.18653\/v1\/D13-1166"},{"key":"R40","doi-asserted-by":"crossref","unstructured":"Kartsaklis, Dimitri and Mehrnoosh Sadrzadeh. 2014. A study of entanglement in a categorical framework of natural language. In Proceedings of the 11th Workshop on Quantum Physics and Logic (QPL 2014), pages 249\u2013261, Kyoto.","DOI":"10.4204\/EPTCS.172.17"},{"key":"R41","unstructured":"Kartsaklis, Dimitri, Mehrnoosh Sadrzadeh, and Stephen Pulman. 2012. A unified sentence space for categorical distributional-compositional semantics: Theory and experiments. In Proceedings of the 24th International Conference on Computational Linguistics (COLING), pages 549\u2013558, Mumbai."},{"key":"R42","unstructured":"Kiros, Ryan, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In Proceedings of the 29th Annual Conference on Neural Information Processing Systems (NIPS 2015), pages 3294\u20133302, Montr\u00e9al."},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1137\/07070111X"},{"key":"R46","unstructured":"Krishnamurthy, Jayant and Tom M. Mitchell. 2013. Vector space semantic parsing: A framework for compositional vector space models. In Proceedings of the ACL Workshop on Continuous Vector Space Models and their Compositionality (CVSC 2013), pages 1\u201310, Sofia."},{"key":"R48","doi-asserted-by":"crossref","unstructured":"Lazaridou, Angeliki, Eva Maria Vecchi, and Marco Baroni. 2013. Fish transporters and miracle homes: How compositional distributional semantics can help NP parsing. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1908\u20131913, Seattle, WA.","DOI":"10.18653\/v1\/D13-1196"},{"key":"R49","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00134"},{"key":"R50","unstructured":"Levy, Omer and Yoav Goldberg. 2014. Neural word embeddings as implicit matrix factorization. In Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NIPS 2014), pages 2177\u20132185, Montr\u00e9al."},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00219"},{"key":"R52","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K15-1035"},{"key":"R53","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-1406"},{"key":"R56","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/S14-2001"},{"key":"R57","unstructured":"Mikolov, T., W. Yih, and G. Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 746\u2013751, Atlanta, GA."},{"key":"R58","unstructured":"Mikolov, Tomas, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems (NIPS 2013), pages 3111\u20133119, Lake Tahoe, NV."},{"key":"R59","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1079"},{"key":"R60","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"R61","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324901002728"},{"key":"R62","unstructured":"Mitchell, Jeff and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL 2008), pages 236\u2013244, Columbus, OH."},{"key":"R63","doi-asserted-by":"publisher","DOI":"10.1111\/j.1551-6709.2010.01106.x"},{"key":"R64","unstructured":"Montague, Richard. 1970. English as a formal language. In B. Visentini , editor, Linguaggi nella Societ\u00e0 et nella Technica. Edizioni di Communita, Milan, pages 188\u2013221. Reprinted in Richmond H. Thomason (ed.), 1974,Formal Philosophy: Selected Papers of Richard Montague. Yale University Press."},{"key":"R65","unstructured":"Pad\u00f3, Sebastian and Yves Peirsman. 2011. Introduction. In Proceedings of the EMNLP Workshop on Geometrical Models for Natural Language Semantics (GEMS 2011), page iii, Edinburgh."},{"key":"R66","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1009"},{"key":"R67","unstructured":"Pham, Nghia, Raffaella Bernardi, Yao Zhong Zhang, and Marco Baroni. 2013. Sentence paraphrase detection: When determiners and word order make the difference. In Proceedings of the IWCS 2013 Towards a Formal Distributional Semantics Workshop, pages 21\u201329, Potsdam."},{"key":"R68","unstructured":"Plate, Tony A. 1991. Holographic reduced representations: Convolution algebra for compositional distributed representations. In Proceedings of the 12th International Joint Conference on Artificial Intelligence (IJCAI 1991), pages 30\u201335, Sydney."},{"key":"R69","doi-asserted-by":"publisher","DOI":"10.1111\/1468-0394.00125"},{"key":"R70","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/E14-1025"},{"key":"R71","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1111"},{"key":"R72","unstructured":"Polajnar, Tamara, Laura Rimell, and Stephen Clark. 2014. Evaluation of simple distributional compositional operations on longer texts. In Proceedings of the 9th Language Resources and Evaluation Conference (LREC 2014), pages 4440\u20134443, Reykjavik."},{"key":"R73","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-2701"},{"key":"R74","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(90)90005-K"},{"key":"R75","unstructured":"Reddy, Siva, Diana McCarthy, and Suresh Manandhar. 2011. An empirical study on compositionality in compound nouns. In Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP 2011), pages 210\u2013218, Chiang Mai."},{"key":"R76","doi-asserted-by":"publisher","DOI":"10.3115\/1699571.1699619"},{"key":"R77","doi-asserted-by":"publisher","DOI":"10.1093\/logcom\/ext044"},{"key":"R78","unstructured":"Sch\u00fctze, Hinrich. 1998. Automatic word sense discrimination. Computational Linguistics, 24(1):97\u2013124."},{"key":"R79","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(90)90007-M"},{"key":"R81","unstructured":"Socher, Richard, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix\u2013vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 1201\u20131211, Jeju Island."},{"key":"R83","doi-asserted-by":"crossref","unstructured":"Socher, Richard, Alex Perelygin, Jean Wu, Jason Chuang, Chris Manning, Andrew Ng, and Chris Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1631\u20131642, Seattle, WA.","DOI":"10.18653\/v1\/D13-1170"},{"key":"R85","doi-asserted-by":"publisher","DOI":"10.1613\/jair.2934"},{"key":"R86","doi-asserted-by":"crossref","unstructured":"Utsumi, Akira. 2012. Extending and evaluating a multiplicative model for semantic composition in a distributional semantic model. In Proceedings of the 11th International Conference on Cognitive Modeling (ICCM 2012), pages 243\u2013248, Berlin.","DOI":"10.1037\/e557102013-065"},{"key":"R87","unstructured":"Vecchi, Eva Maria, Marco Baroni, and Roberto Zamparelli. 2011. (Linear) maps of the impossible: Capturing semantic anomalies in distributional space. In Proceedings of the Workshop on Distributional Semantics and Compositionality (DiSCo 2011), pages 1\u20139, Portland, OR."},{"key":"R88","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00215"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00263","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,10]],"date-time":"2025-06-10T21:47:03Z","timestamp":1749592023000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/42\/4\/661-701\/1555"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,12]]},"references-count":75,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,12]]}},"alternative-id":["10.1162\/COLI_a_00263"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00263","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,12]]}}}