{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,13]],"date-time":"2025-05-13T22:00:04Z","timestamp":1747173604944,"version":"3.40.5"},"reference-count":55,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2019,7,31]],"date-time":"2019-07-31T00:00:00Z","timestamp":1564531200000},"content-version":"unspecified","delay-in-days":30,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2019,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Current approaches to learning semantic representations of sentences often use prior word-level knowledge. The current study aims to leverage visual information in order to capture sentence level semantics without the need for word embeddings. We use a multimodal sentence encoder trained on a corpus of images with matching text captions to produce visually grounded sentence embeddings. Deep Neural Networks are trained to map the two modalities to a common embedding space such that for an image the corresponding caption can be retrieved and vice versa. We show that our model achieves results comparable to the current state of the art on two popular image-caption retrieval benchmark datasets: Microsoft Common Objects in Context (MSCOCO) and Flickr8k. We evaluate the semantic content of the resulting sentence embeddings using the data from the Semantic Textual Similarity (STS) benchmark task and show that the multimodal embeddings correlate well with human semantic similarity judgements. The system achieves state-of-the-art results on several of these benchmarks, which shows that a system trained solely on multimodal data, without assuming any word representations, is able to capture sentence level semantics. Importantly, this result shows that we do not need prior knowledge of lexical level semantics in order to model sentence level semantics. These findings demonstrate the importance of visual information in semantics.<\/jats:p>","DOI":"10.1017\/s1351324919000196","type":"journal-article","created":{"date-parts":[[2019,7,31]],"date-time":"2019-07-31T07:33:09Z","timestamp":1564558389000},"page":"451-466","source":"Crossref","is-referenced-by-count":3,"title":["Learning semantic sentence representations from visually grounded language without lexical knowledge"],"prefix":"10.1017","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5829-5214","authenticated-orcid":false,"given":"Danny","family":"Merkx","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7026-711X","authenticated-orcid":false,"given":"Stefan L.","family":"Frank","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2019,7,31]]},"reference":[{"volume-title":"ASRU","year":"2015","author":"Harwath","key":"S1351324919000196_ref26"},{"key":"S1351324919000196_ref25","article-title":"LSTM: A search space odyssey","volume":"28","author":"Greff","year":"2017","journal-title":"Transactions on Neural Networks and Learning Systems"},{"key":"S1351324919000196_ref24","unstructured":"Faghri F. , Fleet D.J. , Kiros R. and Fidler S. (2017). VSE++: improved visual-semantic embeddings. 1\u201313. CoRR abs\/1707.05612."},{"volume-title":"CoNLL","year":"2018","author":"Derby","key":"S1351324919000196_ref21"},{"key":"S1351324919000196_ref18","first-page":"4806","volume-title":"ICJAI","author":"De Deyne","year":"2017"},{"volume-title":"AAAI","year":"2017","author":"Collell","key":"S1351324919000196_ref15"},{"key":"S1351324919000196_ref10","doi-asserted-by":"publisher","DOI":"10.2307\/1165959"},{"volume-title":"SemEval","year":"2014","author":"Agirre","key":"S1351324919000196_ref2"},{"volume-title":"EMNLP","year":"2015","author":"Bowman","key":"S1351324919000196_ref9"},{"key":"S1351324919000196_ref13","first-page":"613","volume-title":"Proceedings of the 55th Annual Meeting of the ACL","author":"Chrupa\u0142a","year":"2017"},{"volume-title":"ICDMW","year":"2015","author":"Boom","key":"S1351324919000196_ref8"},{"key":"S1351324919000196_ref7","article-title":"SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment","volume":"50","author":"Bentivogli","year":"2016","journal-title":"LRE"},{"key":"S1351324919000196_ref19","article-title":"Indexing by latent semantic analysis","volume":"41","author":"Deerwester","year":"1990","journal-title":"Journal of the Association for Information Science"},{"volume-title":"SemEval","year":"2015","author":"Agirre","key":"S1351324919000196_ref1"},{"volume-title":"SemEval","year":"2012","author":"Agirre","key":"S1351324919000196_ref5"},{"key":"S1351324919000196_ref30","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"S1351324919000196_ref34","first-page":"664","volume-title":"CVPR","author":"Karpathy","year":"2015"},{"key":"S1351324919000196_ref27","unstructured":"Harwath D. , Torralba A. and Glass J. (2016). Unsupervised learning of spoken language with visual context. In NIPS."},{"volume-title":"EMNLP","year":"2014","author":"Pennington","key":"S1351324919000196_ref46"},{"key":"S1351324919000196_ref23","unstructured":"Everingham M. , Van Gool L. , Williams C.K.I. , Winn J. and Zisserman A. (2008). The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results."},{"key":"S1351324919000196_ref32","first-page":"1","volume-title":"ICLR","author":"Huang","year":"2017"},{"key":"S1351324919000196_ref33","unstructured":"Jaakkola T. and Haussler D. (1999). Exploiting generative models in discriminative classifiers. In NIPS."},{"key":"S1351324919000196_ref12","unstructured":"Chen X. , Fang H. , Lin T.-Y. , Vedantam R. , Gupta S. , Dollar P. and Zitnick C. L. (2015). Microsoft COCO Captions: Data Collection and Evaluation Server. 1\u20137. arXiv: 1504. 00325."},{"volume-title":"CVPR","year":"2009","author":"Deng","key":"S1351324919000196_ref20"},{"key":"S1351324919000196_ref41","doi-asserted-by":"publisher","DOI":"10.1017\/S0305000903005592"},{"key":"S1351324919000196_ref16","first-page":"1699","volume-title":"LREC","author":"Conneau","year":"2018"},{"key":"S1351324919000196_ref36","first-page":"1","volume-title":"ICLR","author":"Kingma","year":"2015"},{"key":"S1351324919000196_ref6","first-page":"1","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Bahdanau","year":"2015"},{"volume-title":"ASRU","year":"2017","author":"Leidal","key":"S1351324919000196_ref40"},{"key":"S1351324919000196_ref28","first-page":"770","volume-title":"CVPR","author":"He","year":"2016"},{"volume-title":"NAACL-HLT","year":"2018","author":"Kiela","key":"S1351324919000196_ref35"},{"key":"S1351324919000196_ref37","unstructured":"Kiros R. , Zhu Y. , Salakhutdinov R.R. , Zemel R. , Urtasun R. , Torralba A. and Fidler S. (2015). Skip-thought vectors. In NIPS."},{"volume-title":"EMNLP","year":"2017","author":"Conneau","key":"S1351324919000196_ref17"},{"key":"S1351324919000196_ref39","volume-title":"International Conference on Machine Learning","volume":"32","author":"Le","year":"2014"},{"key":"S1351324919000196_ref31","doi-asserted-by":"publisher","DOI":"10.1613\/jair.3994"},{"volume-title":"SemEval","year":"2016","author":"Agirre","key":"S1351324919000196_ref3"},{"volume-title":"ICCV","year":"2015","author":"Ma","key":"S1351324919000196_ref42"},{"key":"S1351324919000196_ref11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S17-2001"},{"key":"S1351324919000196_ref44","unstructured":"Mikolov T. , Chen K. , Corrado G. and Dean J. (2013). Efficient Estimation of Word Representations in Vector Space. 1\u201312. arXiv: 1301.3781."},{"key":"S1351324919000196_ref45","unstructured":"Patel R.N. , Pimpale P.B. and Sasikumar M. (2016). Recurrent Neural Network based Part-of-Speech Tagger for Code-Mixed Social Media Text. 1\u20137. arXiv: 1611.04989."},{"key":"S1351324919000196_ref47","article-title":"Reanalysing rote-learned phrases: individual differences in the transition to multi-word speech","volume":"20","author":"Pine","year":"1993","journal-title":"Journal of Child Language"},{"volume-title":"NAACL-HLT","year":"2018","author":"Qi","key":"S1351324919000196_ref48"},{"key":"S1351324919000196_ref49","doi-asserted-by":"publisher","DOI":"10.1145\/365628.365657"},{"key":"S1351324919000196_ref52","first-page":"1","volume-title":"ICLR","author":"Vendrov","year":"2016"},{"volume-title":"RepL4NLP","year":"2018","author":"Yang","key":"S1351324919000196_ref54"},{"key":"S1351324919000196_ref55","first-page":"19","volume-title":"ICCV","author":"Zhu","year":"2015"},{"volume-title":"SemEval","year":"2013","author":"Agirre","key":"S1351324919000196_ref4"},{"key":"S1351324919000196_ref51","article-title":"First steps toward a usage-based theory of language acquisition","volume":"11","author":"Tomasello","year":"2000","journal-title":"Cognitive Linguistics"},{"volume-title":"WACV","year":"2017","author":"Smith","key":"S1351324919000196_ref50"},{"volume-title":"CVPR","year":"2015","author":"Klein","key":"S1351324919000196_ref38"},{"key":"S1351324919000196_ref53","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2017.11.020"},{"key":"S1351324919000196_ref43","article-title":"The role of image representations in vision to language tasks","volume":"24","author":"Madhyastha","year":"2018","journal-title":"NLE"},{"key":"S1351324919000196_ref22","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2018.2832602"},{"volume-title":"NAACL HLT","year":"2016","author":"Hill","key":"S1351324919000196_ref29"},{"key":"S1351324919000196_ref14","first-page":"1","volume-title":"Paper presented at the NIPS Workshop on Deep Learning","author":"Chung","year":"2014"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324919000196","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,10,15]],"date-time":"2019-10-15T04:26:59Z","timestamp":1571113619000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324919000196\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7]]},"references-count":55,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,7]]}},"alternative-id":["S1351324919000196"],"URL":"https:\/\/doi.org\/10.1017\/s1351324919000196","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"type":"print","value":"1351-3249"},{"type":"electronic","value":"1469-8110"}],"subject":[],"published":{"date-parts":[[2019,7]]}}}