{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T09:59:41Z","timestamp":1775815181431,"version":"3.50.1"},"reference-count":112,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2022,8,9]],"date-time":"2022-08-09T00:00:00Z","timestamp":1660003200000},"content-version":"vor","delay-in-days":220,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>In the context of text representation, Compositional Distributional Semantics models aim to fuse the Distributional Hypothesis and the Principle of Compositionality. Text embedding is based on co-ocurrence distributions and the representations are in turn combined by compositional functions taking into account the text structure. However, the theoretical basis of compositional functions is still an open issue. In this article we define and study the notion of Information Theory\u2013based Compositional Distributional Semantics (ICDS): (i) We first establish formal properties for embedding, composition, and similarity functions based on Shannon\u2019s Information Theory; (ii) we analyze the existing approaches under this prism, checking whether or not they comply with the established desirable properties; (iii) we propose two parameterizable composition and similarity functions that generalize traditional approaches while fulfilling the formal properties; and finally (iv) we perform an empirical study on several textual similarity datasets that include sentences with a high and low lexical overlap, and on the similarity between words and their description. Our theoretical analysis and empirical results show that fulfilling formal properties affects positively the accuracy of text representation models in terms of correspondence (isometry) between the embedding and meaning spaces.<\/jats:p>","DOI":"10.1162\/coli_a_00454","type":"journal-article","created":{"date-parts":[[2022,8,9]],"date-time":"2022-08-09T14:35:55Z","timestamp":1660055755000},"page":"907-948","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Information Theory\u2013based Compositional Distributional\n                    Semantics"],"prefix":"10.1162","volume":"48","author":[{"given":"Enrique","family":"Amig\u00f3","sequence":"first","affiliation":[{"name":"Universidad Nacional de Educaci\u00f3n a Distancia (UNED). enrique@lsi.uned.es"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alejandro","family":"Ariza-Casabona","sequence":"additional","affiliation":[{"name":"CLiC - UBICS, Universitat de Barcelona. alejandro.ariza14@ub.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Victor","family":"Fresno","sequence":"additional","affiliation":[{"name":"Universidad Nacional de Educaci\u00f3n a Distancia (UNED). vfresno@lsi.uned.es"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M. Ant\u00f2nia","family":"Mart\u00ed","sequence":"additional","affiliation":[{"name":"CLiC - UBICS, Universitat de Barcelona. amarti@ub.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2022,12,1]]},"reference":[{"key":"2022120516072502000_bib1","doi-asserted-by":"publisher","first-page":"16","DOI":"10.3115\/992628.992635","article-title":"Word sense disambiguation using conceptual\n                        density","volume-title":"COLING 1996 Vol. 1: The 16th\n                        International Conference on Computational Linguistics","author":"Agirre","year":"1996"},{"key":"2022120516072502000_bib2","unstructured":"Almuhareb,\n                                Abdulrahman\n          .\n                        2006. Attributes in lexical\n                            acquisition. Ph.D. thesis,\n                        University of Essex, Colchester,\n                    UK."},{"key":"2022120516072502000_bib3","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1007\/s10791-020-09375-z","article-title":"On the foundations of similarity in\n                        information access","volume":"23","author":"Amig\u00f3","year":"2020","journal-title":"Information Retrieval\n                        Journal"},{"key":"2022120516072502000_bib4","doi-asserted-by":"publisher","DOI":"10.31219\/osf.io\/3b27t","article-title":"An empirical study on similarity\n                        functions: Parameter estimation for the information contrast\n                        model","author":"Amig\u00f3","year":"2022"},{"key":"2022120516072502000_bib5","first-page":"47","article-title":"Semantic parsing as machine\n                        translation","volume-title":"Proceedings of the 51st ACL (Vol. 2:\n                        Short Papers)","author":"Andreas","year":"2013"},{"key":"2022120516072502000_bib6","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1162\/tacl_a_00106","article-title":"A latent variable model approach to\n                        PMI-based word embeddings","volume":"4","author":"Arora","year":"2016","journal-title":"Transactions of the\n                        Association for Computational Linguistics"},{"key":"2022120516072502000_bib7","article-title":"A simple but tough-to-beat baseline for sentence\n                        embeddings","volume-title":"International Conference on Learning\n                        Representations","author":"Arora","year":"2017"},{"key":"2022120516072502000_bib8","doi-asserted-by":"publisher","first-page":"238","DOI":"10.3115\/v1\/P14-1023","article-title":"Don\u2019t count, predict! A systematic\n                        comparison of context-counting vs. context-predicting semantic\n                        vectors","volume-title":"Proceedings of the 52nd Annual Meeting\n                        of the ACL (Volume 1: Long Papers)","author":"Baroni","year":"2014"},{"key":"2022120516072502000_bib9","article-title":"Esslli workshop on distributional lexical\n                        semantics bridging the gap between semantic theory and computational\n                        simulations","author":"Baroni","year":"2008"},{"issue":"4","key":"2022120516072502000_bib10","doi-asserted-by":"publisher","first-page":"673","DOI":"10.1162\/coli_a_00016","article-title":"Distributional memory: A general framework for\n                        corpus-based semantics","volume":"36","author":"Baroni","year":"2010","journal-title":"Computational\n                        Linguistics"},{"key":"2022120516072502000_bib11","doi-asserted-by":"publisher","first-page":"5185","DOI":"10.18653\/v1\/2020.acl-main.463","article-title":"Climbing towards NLU: On meaning, form,\n                        and understanding in the age of data","volume-title":"Proceedings\n                        of the 58th Annual Meeting of the Association for Computational\n                        Linguistics","author":"Bender","year":"2020"},{"key":"2022120516072502000_bib12","first-page":"1137","article-title":"A neural probabilistic language\n                        model","volume":"3","author":"Bengio","year":"2003","journal-title":"Journal of Machine Learning\n                    Research"},{"key":"2022120516072502000_bib13","first-page":"546","article-title":"A comparison of vector-based\n                        representations for semantic composition","volume-title":"Proceedings of the 2012 Joint Conference on Empirical Methods in\n                        Natural Language Processing and Computational Natural Language\n                        Learning","author":"Blacoe","year":"2012"},{"key":"2022120516072502000_bib14","first-page":"993","article-title":"Latent Dirichlet\n                        allocation","volume":"3","author":"Blei","year":"2003","journal-title":"Journal of Machine Learning\n                        Research"},{"issue":"1","key":"2022120516072502000_bib15","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1146\/annurev-linguistics-011619-030303","article-title":"Distributional semantics and linguistic\n                        theory","volume":"6","author":"Boleda","year":"2020","journal-title":"Annual Review of Linguistics"},{"key":"2022120516072502000_bib16","first-page":"2","article-title":"Distributional semantic features as semantic\n                        primitives\u2014or not","volume-title":"AAAI Spring Symposium\n                        Series","author":"Boleda","year":"2015"},{"key":"2022120516072502000_bib17","doi-asserted-by":"publisher","first-page":"632","DOI":"10.18653\/v1\/D15-1075","article-title":"A large annotated corpus for learning\n                        natural language inference","volume-title":"Proceedings of the\n                        2015 Conference on EMNLP","author":"Bowman","year":"2015"},{"issue":"2","key":"2022120516072502000_bib18","first-page":"263","article-title":"The mathematics of statistical machine\n                        translation: Parameter estimation","volume":"19","author":"Brown","year":"1993","journal-title":"Computational\n                        Linguistics"},{"key":"2022120516072502000_bib19","first-page":"1877","article-title":"Language models are few-shot\n                        learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing\n                        Systems"},{"key":"2022120516072502000_bib20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1613\/jair.4135","article-title":"Multimodal distributional\n                        semantics","volume":"49","author":"Bruni","year":"2014","journal-title":"Journal of Artificial Intelligence\n                        Research"},{"issue":"1","key":"2022120516072502000_bib21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1613\/jair.4135","article-title":"Multimodal distributional\n                        semantics","volume":"49","author":"Bruni","year":"2014","journal-title":"Journal of Artificial Intelligence\n                        Research"},{"key":"2022120516072502000_bib22","article-title":"Isotropy in the contextual embedding\n                        space: Clusters and manifolds","volume-title":"International\n                        Conference on Learning Representations","author":"Cai","year":"2021"},{"key":"2022120516072502000_bib23","doi-asserted-by":"publisher","first-page":"43","DOI":"10.18653\/v1\/W16-2508","article-title":"Find the word that does not belong: A\n                        framework for an intrinsic evaluation of word vector\n                        representations","volume-title":"Proceedings of the 1st Workshop\n                        on Evaluating Vector-Space Representations for NLP","author":"Camacho-Collados","year":"2016"},{"key":"2022120516072502000_bib24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/S17-2001","article-title":"SemEval-2017 task 1: Semantic textual\n                        similarity multilingual and crosslingual focused evaluation","volume-title":"Proceedings of the 11th International Workshop on Semantic Evaluation\n                        (SemEval-2017)","author":"Cer","year":"2017"},{"key":"2022120516072502000_bib25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-2029","article-title":"Universal sentence\n                    encoder","author":"Cer","year":"2018","journal-title":"CoRR"},{"key":"2022120516072502000_bib26","article-title":"ELECTRA: Pre-training text encoders as\n                        discriminators rather than generators","volume-title":"International Conference on Learning\n                    Representations","author":"Clark","year":"2020"},{"key":"2022120516072502000_bib27","first-page":"52","article-title":"Combining symbolic and distributional\n                        models of meaning","volume-title":"AAAI Spring Symposium: Quantum\n                        Interaction","author":"Clark","year":"2007"},{"key":"2022120516072502000_bib28","article-title":"Mathematical foundations for a compositional\n                        distributional model of meaning","author":"Coecke","year":"2010","journal-title":"CoRR"},{"key":"2022120516072502000_bib29","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1145\/1390156.1390177","article-title":"A unified architecture for natural\n                        language processing: Deep neural networks with multitask\n                        learning","volume-title":"Machine Learning, Proceedings of the\n                        ICML","author":"Collobert","year":"2008"},{"key":"2022120516072502000_bib30","doi-asserted-by":"publisher","first-page":"591","DOI":"10.2307\/2941118","article-title":"Systematicity","volume":"93","author":"Cummins","year":"1996","journal-title":"The\n                        Journal of Philosophy"},{"key":"2022120516072502000_bib31","doi-asserted-by":"publisher","first-page":"91","DOI":"10.18653\/v1\/W19-0408","article-title":"Words are vectors, dependencies are\n                        matrices: Learning word embeddings from dependency graphs","volume-title":"Proceedings of the 13th International Conference on Computational\n                        Semantics - Long Papers","author":"Czarnowska","year":"2019"},{"key":"2022120516072502000_bib32","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1002\/(SICI)1097-4571(199009)41:6&lt;391::AID-ASI1&gt;3.0.CO;2-9","article-title":"Indexing by latent semantic\n                        analysis","volume":"41","author":"Deerwester","year":"1990","journal-title":"Journal of the American Society for\n                        Information Science"},{"key":"2022120516072502000_bib33","doi-asserted-by":"publisher","first-page":"2191","DOI":"10.18653\/v1\/2020.acl-main.198","article-title":"Stolen probability: A structural weakness\n                        of neural language models","volume-title":"Proceedings of the\n                        58th Annual Meeting of the Association for Computational\n                        Linguistics","author":"Demeter","year":"2020"},{"key":"2022120516072502000_bib34","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional\n                        transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of\n                        the ACL: Human Language Technologies, Volume 1","author":"Devlin","year":"2019"},{"key":"2022120516072502000_bib35","doi-asserted-by":"publisher","first-page":"104","DOI":"10.3115\/1693756.1693769","article-title":"Supporting inferences in semantic space:\n                        Representing words as regions","volume-title":"Proceedings of the\n                        Eighth International Conference on Computational Semantics","author":"Erk","year":"2009"},{"key":"2022120516072502000_bib36","doi-asserted-by":"publisher","first-page":"91","DOI":"10.18653\/v1\/W18-3012","article-title":"Unsupervised random walk sentence\n                        embeddings: A strong but simple baseline","volume-title":"Proceedings of the Third Workshop on Representation Learning for\n                        NLP","author":"Ethayarajh","year":"2018"},{"key":"2022120516072502000_bib37","doi-asserted-by":"publisher","first-page":"55","DOI":"10.18653\/v1\/D19-1006","article-title":"How contextual are contextualized word\n                        representations? Comparing the geometry of BERT, ELMo, and GPT-2\n                        embeddings","volume-title":"Proceedings of the 2019\n                        EMNLP-IJCNLP","author":"Ethayarajh","year":"2019"},{"key":"2022120516072502000_bib38","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/7287.001.0001","volume-title":"WordNet: An Electronic Lexical Database","author":"Fellbaum","year":"1998"},{"key":"2022120516072502000_bib39","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1145\/371920.372094","article-title":"Placing search in context: The concept\n                        revisited","volume-title":"Proceedings of the 10th International\n                        Conference on World Wide Web","author":"Finkelstein","year":"2001"},{"key":"2022120516072502000_bib40","volume-title":"Papers in Linguistics, 1934\u20131951","author":"Firth","year":"1957"},{"key":"2022120516072502000_bib41","doi-asserted-by":"crossref","first-page":"4522","DOI":"10.1098\/rspb.2012.1741","article-title":"How hierarchical is language\n                        use?","volume":"279","author":"Frank","year":"2012","journal-title":"Proceedings. Biological Sciences \/ The Royal Society"},{"key":"2022120516072502000_bib42","article-title":"Representation degeneration problem in training natural\n                        language generation models","volume-title":"Proceedings of\n                        ICLR","author":"Gao","year":"2019"},{"key":"2022120516072502000_bib43","doi-asserted-by":"publisher","first-page":"2173","DOI":"10.18653\/v1\/D16-1235","article-title":"SimVerb-3500: A large-scale evaluation set\n                        of verb similarity","volume-title":"EMNLP","author":"Gerz","year":"2016"},{"key":"2022120516072502000_bib44","doi-asserted-by":"publisher","first-page":"36","DOI":"10.18653\/v1\/W16-2507","article-title":"Intrinsic evaluations of word embeddings: What\n                        can we do better?","volume-title":"Proceedings of the 1st Workshop\n                        on Evaluating Vector-Space Representations for NLP","author":"Gladkova","year":"2016"},{"key":"2022120516072502000_bib45","doi-asserted-by":"publisher","first-page":"1958","DOI":"10.18653\/v1\/2020.acl-main.177","article-title":"Probing linguistic\n                        systematicity","volume-title":"Proceedings of the 58th Annual\n                        Meeting of the ACL","author":"Goodwin","year":"2020"},{"key":"2022120516072502000_bib46","first-page":"1394","article-title":"Experimental support for a categorical\n                        compositional distributional model of meaning","volume-title":"Proceedings of the EMNLP \u201911","author":"Grefenstette","year":"2011"},{"issue":"2\u20133","key":"2022120516072502000_bib47","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1080\/00437956.1954.11659520","article-title":"Distributional structure","volume":"10","author":"Harris","year":"1954","journal-title":"Word"},{"issue":"4","key":"2022120516072502000_bib48","doi-asserted-by":"publisher","first-page":"665","DOI":"10.1162\/COLI_a_00237","article-title":"SimLex-999: Evaluating semantic models\n                        with (genuine) similarity estimation","volume":"41","author":"Hill","year":"2015","journal-title":"Computational\n                        Linguistics"},{"issue":"8","key":"2022120516072502000_bib49","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2022120516072502000_bib50","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1613\/jair.1.11674","article-title":"Compositionality decomposed: How do neural\n                        networks generalize?","volume":"67","author":"Hupkes","year":"2020","journal-title":"JAIR"},{"issue":"3","key":"2022120516072502000_bib51","doi-asserted-by":"publisher","first-page":"111","DOI":"10.5840\/jphil2004101321","article-title":"On the systematicity of language and\n                        thought","volume":"101","author":"Johnson","year":"2004","journal-title":"Journal of Philosophy"},{"key":"2022120516072502000_bib52","first-page":"549","article-title":"A unified sentence space for categorical\n                        distributional-compositional semantics: Theory and\n                        experiments","volume-title":"COLING (Posters)","author":"Kartsaklis","year":"2012"},{"key":"2022120516072502000_bib53","doi-asserted-by":"publisher","first-page":"941","DOI":"10.18653\/v1\/P16-1089","article-title":"Siamese CBOW: Optimizing word embeddings\n                        for sentence representations","volume-title":"Proceedings of the\n                        54th Annual Meeting of the Association for Computational Linguistics (Volume\n                        1: Long Papers)","author":"Kenter","year":"2016"},{"key":"2022120516072502000_bib54","first-page":"3294","article-title":"Skip-thought vectors","volume-title":"Advances in Neural Information Processing Systems 28","author":"Kiros","year":"2015"},{"issue":"2","key":"2022120516072502000_bib55","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1037\/0033-295X.104.2.211","article-title":"A solution to Plato\u2019s problem: The\n                        latent semantic analysis theory of acquisition, induction, and\n                        representation of knowledge","volume":"104","author":"Landauer","year":"1997","journal-title":"Psychological\n                        Review"},{"key":"2022120516072502000_bib56","first-page":"1188","article-title":"Distributed representations of sentences\n                        and documents","volume-title":"International Conference on\n                        Machine Learning","author":"Le","year":"2014"},{"issue":"1","key":"2022120516072502000_bib57","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1146\/annurev-linguistics-030514-125254","article-title":"Distributional models of word\n                        meaning","volume":"4","author":"Lenci","year":"2018","journal-title":"Annual Review of Linguistics"},{"key":"2022120516072502000_bib58","first-page":"2177","article-title":"Neural word embedding as implicit matrix\n                        factorization","author":"Levy","year":"2014","journal-title":"Advances in Neural Information\n                        Processing Systems 27"},{"key":"2022120516072502000_bib59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence\n                        pre-training for natural language generation, translation, and\n                        comprehension","author":"Lewis","year":"2019","journal-title":"CoRR"},{"key":"2022120516072502000_bib60","doi-asserted-by":"publisher","first-page":"9119","DOI":"10.18653\/v1\/2020.emnlp-main.733","article-title":"On the sentence embeddings from pre-trained language\n                        models","volume-title":"2020 Conference on EMNLP","author":"Li","year":"2020"},{"key":"2022120516072502000_bib61","first-page":"296","article-title":"An information-theoretic definition of\n                        similarity","volume-title":"Proceedings of the Fifteenth\n                        International Conference on Machine Learning","author":"Lin","year":"1998"},{"key":"2022120516072502000_bib62","first-page":"216","article-title":"A SICK cure for the evaluation of\n                        compositional distributional semantic models","volume-title":"Proceedings of the LREC\u201914","author":"Marelli","year":"2014"},{"key":"2022120516072502000_bib63","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1007\/978-3-030-34974-5_14","article-title":"Compositionality and contextuality: The\n                        symbolic and statistical theories of meaning","volume-title":"Modeling and Using Context - 11th International and\n                        Interdisciplinary Conference","author":"Maruyama","year":"2019"},{"key":"2022120516072502000_bib64","first-page":"6294","article-title":"Learned in translation: Contextualized\n                        word vectors","volume-title":"Advances in Neural Information\n                        Processing Systems","author":"McCann","year":"2017"},{"key":"2022120516072502000_bib65","first-page":"52","article-title":"Advances in pre-training distributed word\n                        representations","volume-title":"11th LREC 2018","author":"Mikolov","year":"2018"},{"key":"2022120516072502000_bib66","first-page":"3111","article-title":"Distributed representations of words and phrases and their\n                        compositionality","volume-title":"Advances in Neural Information\n                        Processing Systems","author":"Mikolov","year":"2013"},{"key":"2022120516072502000_bib67","doi-asserted-by":"publisher","first-page":"240","DOI":"10.3115\/1075812.1075866","article-title":"Using a semantic concordance for sense\n                        identification","volume-title":"Human Language Technology:\n                        Proceedings","author":"Miller","year":"1994"},{"key":"2022120516072502000_bib68","doi-asserted-by":"publisher","first-page":"2873","DOI":"10.18653\/v1\/D17-1308","article-title":"The strange geometry of skip-gram with\n                        negative sampling","volume-title":"Proceedings of the 2017\n                        Conference on EMNLP","author":"Mimno","year":"2017"},{"key":"2022120516072502000_bib69","first-page":"236","article-title":"Vector-based models of semantic\n                        composition","author":"Mitchell","year":"2008","journal-title":"Proceedings of ACL-08: HLT"},{"issue":"8","key":"2022120516072502000_bib70","doi-asserted-by":"publisher","first-page":"1388","DOI":"10.1111\/j.1551-6709.2010.01106.x","article-title":"Composition in distributional models of\n                        semantics","volume":"34","author":"Mitchell","year":"2010","journal-title":"Cognitive Science"},{"key":"2022120516072502000_bib71","article-title":"All-but-the-top: Simple and effective\n                        postprocessing for word representations","volume-title":"International Conference on Learning\n                    Representations","author":"Mu","year":"2018"},{"issue":"2","key":"2022120516072502000_bib72","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1459352.1459355","article-title":"Word sense disambiguation: A\n                        survey","volume":"41","author":"Navigli","year":"2009","journal-title":"ACM Computing Surveys (CSUR)"},{"issue":"2","key":"2022120516072502000_bib73","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1162\/coli.2007.33.2.161","article-title":"Dependency-based construction of semantic\n                        space models","volume":"33","author":"Pad\u00f3","year":"2007","journal-title":"Computational Linguistics"},{"key":"2022120516072502000_bib74","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: Global vectors for word\n                        representation","volume-title":"Proceedings of the 2014\n                        EMNLP","author":"Pennington","year":"2014"},{"key":"2022120516072502000_bib75","article-title":"Evaluation of sentence embeddings in\n                        downstream and linguistic probing tasks","volume":"abs\/1806.06259","author":"Perone","year":"2018","journal-title":"CoRR"},{"key":"2022120516072502000_bib76","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word\n                        representations","volume-title":"2018 NAACL: Human Language\n                        Technologies, Volume 1 (Long Papers)","author":"Peters","year":"2018"},{"key":"2022120516072502000_bib77","doi-asserted-by":"publisher","first-page":"4609","DOI":"10.18653\/v1\/2020.acl-main.420","article-title":"Information-theoretic probing for\n                        linguistic structure","volume-title":"58th Annual Meeting of the\n                        ACL","author":"Pimentel","year":"2020"},{"key":"2022120516072502000_bib78","doi-asserted-by":"publisher","first-page":"7027","DOI":"10.18653\/v1\/2020.acl-main.628","article-title":"Sentence meta-embeddings for unsupervised\n                        semantic textual similarity","volume-title":"Proceedings of the\n                        58th Meeting of the ACL","author":"Poerner","year":"2020"},{"key":"2022120516072502000_bib79","first-page":"4440","article-title":"Evaluation of simple distributional\n                        compositional operations on longer texts","volume-title":"Proceedings of the (LREC\u201914)","author":"Polajnar","year":"2014"},{"key":"2022120516072502000_bib80","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1007\/978-3-030-01716-3_18","article-title":"Revisiting correlations between intrinsic and extrinsic\n                        evaluations of word embeddings","volume-title":"Chinese\n                        Computational Linguistics and Natural Language Processing Based on Naturally\n                        Annotated Big Data","author":"Qiu","year":"2018"},{"issue":"8","key":"2022120516072502000_bib81","first-page":"9","article-title":"Language models are unsupervised multitask\n                        learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"issue":"140","key":"2022120516072502000_bib82","first-page":"1","article-title":"Exploring the limits of transfer learning with\n                        a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of\n                        Machine Learning Research"},{"key":"2022120516072502000_bib83","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-BERT: Sentence embeddings using\n                        Siamese BERT-networks","volume-title":"Proceedings of the 2019\n                        EMNLP-IJCNLP","author":"Reimers","year":"2019"},{"issue":"4","key":"2022120516072502000_bib84","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1162\/COLI_a_00263","article-title":"RELPRON: A relative clause evaluation data set\n                        for compositional distributional semantics","volume":"42","author":"Rimell","year":"2016","journal-title":"Computational Linguistics"},{"issue":"5","key":"2022120516072502000_bib85","doi-asserted-by":"publisher","first-page":"503","DOI":"10.1108\/00220410410560582","article-title":"Understanding inverse document frequency:\n                        On theoretical arguments for IDF","volume":"60","author":"Robertson","year":"2004","journal-title":"Journal of\n                        Documentation"},{"key":"2022120516072502000_bib86","doi-asserted-by":"publisher","first-page":"135","DOI":"10.18653\/v1\/S17-1017","article-title":"The (too many) problems of analogical reasoning with word\n                        vectors","volume-title":"Proceedings of the 6th Joint Conference\n                        on Lexical and Computational Semantics (*SEM 2017)","author":"Rogers","year":"2017"},{"issue":"6","key":"2022120516072502000_bib87","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1145\/364955.364990","article-title":"The SMART automatic document retrieval systems\u2014an\n                        illustration","volume":"8","author":"Salton","year":"1965","journal-title":"Communications of the ACM"},{"key":"2022120516072502000_bib88","doi-asserted-by":"publisher","DOI":"10.3233\/AIC-210084","article-title":"Neuro-symbolic artificial intelligence:\n                        Current trends","author":"Sarker","year":"2021","journal-title":"arXiv preprint\n                        arXiv:2105.05330"},{"key":"2022120516072502000_bib89","doi-asserted-by":"publisher","first-page":"298","DOI":"10.18653\/v1\/D15-1036","article-title":"Evaluation methods for unsupervised word\n                        embeddings","volume-title":"Proceedings of the 2015 Conference on\n                        Empirical Methods in Natural Language Processing","author":"Schnabel","year":"2015"},{"key":"2022120516072502000_bib90","first-page":"1089","article-title":"An intrinsic information content metric for\n                        semantic similarity in Wordnet","volume-title":"ECAI\u201904:\n                        Proceedings of the 16th European Conference on Artificial\n                        Intelligence","author":"Seco","year":"2004"},{"key":"2022120516072502000_bib91","first-page":"216","article-title":"A corpus-based probabilistic grammar with\n                        only two non-terminals","volume-title":"Proceedings of the 4th\n                        International Workshop on Parsing Technologies","author":"Sekine","year":"1995"},{"key":"2022120516072502000_bib92","article-title":"Basic reasoning with tensor product\n                        representations","author":"Smolensky","year":"2016","journal-title":"arXiv preprint\n                        arXiv:1601.02745"},{"key":"2022120516072502000_bib93","first-page":"1201","article-title":"Semantic compositionality through recursive matrix-vector\n                        spaces","volume-title":"Proceedings of the EMNLP-CoNLL\n                        \u201912","author":"Socher","year":"2012"},{"key":"2022120516072502000_bib94","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1037\/0033-2909.87.2.245","article-title":"Tests for comparing elements of a correlation\n                        matrix","volume":"87","author":"Steiger","year":"1980","journal-title":"Psychological Bulletin"},{"key":"2022120516072502000_bib95","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1162\/tacl_a_00342","article-title":"oLMpics\u2014On what language model\n                        pre-training captures","volume":"8","author":"Talmor","year":"2020","journal-title":"Transactions of the\n                        ACL"},{"key":"2022120516072502000_bib96","doi-asserted-by":"publisher","first-page":"2049","DOI":"10.18653\/v1\/D15-1243","article-title":"Evaluation of word vector representations by subspace\n                        alignment","volume-title":"Proceedings of the 2015 Conference on\n                        Empirical Methods in Natural Language Processing","author":"Tsvetkov","year":"2015"},{"key":"2022120516072502000_bib97","article-title":"Empirical evaluation of four tensor\n                        decomposition algorithms","author":"Turney","year":"2007","journal-title":"CoRR"},{"key":"2022120516072502000_bib98","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1037\/0033-295X.84.4.327","article-title":"Features of similarity","volume":"84","author":"Tversky","year":"1977","journal-title":"Psychological Review"},{"key":"2022120516072502000_bib99","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems 30","author":"Vaswani","year":"2017"},{"key":"2022120516072502000_bib100","doi-asserted-by":"publisher","first-page":"e19","DOI":"10.1017\/ATSIP.2019.12","article-title":"Evaluating word embedding models: Methods\n                        and experimental results","volume":"8","author":"Wang","year":"2019","journal-title":"APSIPA Transactions on\n                        Signal and Information Processing"},{"key":"2022120516072502000_bib101","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1162\/tacl_a_00143","article-title":"From paraphrase database to compositional\n                        paraphrase model and back","volume":"3","author":"Wieting","year":"2015","journal-title":"Transactions of the\n                        Association for Computational Linguistics"},{"key":"2022120516072502000_bib102","doi-asserted-by":"publisher","first-page":"451","DOI":"10.18653\/v1\/P18-1042","article-title":"ParaNMT-50M: Pushing the limits of\n                        paraphrastic sentence embeddings with millions of machine\n                        translations","volume-title":"Proceedings of the 56th ACL (Vol.\n                        1: Long Papers)","author":"Wieting","year":"2018"},{"issue":"2","key":"2022120516072502000_bib103","first-page":"59","article-title":"On-line semantic analysis of English\n                        texts","volume":"11","author":"Wilks","year":"1968","journal-title":"Mechanical Translation and Computational\n                        Linguistics"},{"key":"2022120516072502000_bib104","volume-title":"Philosophical Investigations","author":"Wittgenstein","year":"1953"},{"key":"2022120516072502000_bib105","doi-asserted-by":"publisher","first-page":"4638","DOI":"10.18653\/v1\/2020.acl-main.422","article-title":"Similarity analysis of contextual word\n                        representation models","volume-title":"Proceedings of the 58th\n                        Annual Meeting of the Association for Computational Linguistics","author":"Wu","year":"2020"},{"key":"2022120516072502000_bib106","first-page":"32","article-title":"XLNet: Generalized autoregressive pretraining for language\n                        understanding","author":"Yang","year":"2019","journal-title":"Advances in Neural Information\n                        Processing Systems"},{"key":"2022120516072502000_bib107","article-title":"Learning and evaluating general linguistic\n                        intelligence","author":"Yogatama","year":"2019","journal-title":"CoRR"},{"key":"2022120516072502000_bib108","first-page":"1263","article-title":"Estimating linear models for compositional\n                        distributional semantics","volume-title":"Proceedings of the\n                        COLING 2010","author":"Zanzotto","year":"2010"},{"issue":"3","key":"2022120516072502000_bib109","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1561\/1500000008","article-title":"Statistical language models for\n                        information retrieval a critical review","volume":"2","author":"Zhai","year":"2008","journal-title":"Foundations\n                        and Trends in Information Retrieval"},{"key":"2022120516072502000_bib110","unstructured":"Zhang,\n                                Jiannan\n          .\n                        2014. A Generic Evaluation of a Categorical\n                        Compositional-distributional Model of Meaning. Ph.D. thesis,\n                        University of Oxford."},{"key":"2022120516072502000_bib111","doi-asserted-by":"publisher","first-page":"1601","DOI":"10.18653\/v1\/2020.emnlp-main.124","article-title":"An unsupervised sentence embedding method by mutual\n                        information maximization","volume-title":"Proceedings of the 2020\n                        Conference on EMNLP","author":"Zhang","year":"2020"},{"key":"2022120516072502000_bib112","doi-asserted-by":"publisher","first-page":"8361","DOI":"10.18653\/v1\/2020.acl-main.741","article-title":"Estimating mutual information between\n                        dense word embeddings","volume-title":"Proceedings of the 58th\n                        Annual Meeting of the Association for Computational Linguistics","author":"Zhelezniak","year":"2020"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/48\/4\/907\/2061987\/coli_a_00454.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/48\/4\/907\/2061987\/coli_a_00454.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,5]],"date-time":"2022-12-05T16:08:14Z","timestamp":1670256494000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/48\/4\/907\/112556\/Information-Theory-based-Compositional"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":112,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12,1]]},"published-print":{"date-parts":[[2022,12,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00454","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}