{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T21:50:55Z","timestamp":1790718655809,"version":"4.1.0"},"reference-count":50,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2021,7,13]],"date-time":"2021-07-13T00:00:00Z","timestamp":1626134400000},"content-version":"vor","delay-in-days":193,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Recently, multimodal transformer models have gained popularity because their performance on downstream tasks suggests they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors that can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality-specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers.<\/jats:p>","DOI":"10.1162\/tacl_a_00385","type":"journal-article","created":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T19:04:56Z","timestamp":1626375896000},"page":"570-585","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":80,"title":["Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers"],"prefix":"10.1162","volume":"9","author":[{"given":"Lisa Anne","family":"Hendricks","sequence":"first","affiliation":[{"name":"DeepMind, United Kingdom. lmh@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Mellor","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom. johnme@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rosalia","family":"Schneider","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom. rgschneider@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean-Baptiste","family":"Alayrac","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom. jalayrac@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aida","family":"Nematzadeh","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom. nematzadeh@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,7,8]]},"reference":[{"key":"2021071519044966400_bib1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.586","article-title":"Words aren\u2019t enough, their order matters: On the robustness of grounding visual referring expressions","author":"Akula","year":"2020","journal-title":"arXiv preprint arXiv:2005.01655"},{"key":"2021071519044966400_bib2","doi-asserted-by":"publisher","first-page":"6077","DOI":"10.1109\/CVPR.2018.00636","article-title":"Bottom-up and top-down attention for image captioning and visual question answering","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Anderson","year":"2018"},{"key":"2021071519044966400_bib3","doi-asserted-by":"publisher","first-page":"2425","DOI":"10.1109\/ICCV.2015.279","article-title":"Vqa: Visual question answering","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Antol","year":"2015"},{"key":"2021071519044966400_bib4","article-title":"Language models are few-shot learners","author":"Brown","year":"2020","journal-title":"arXiv preprint arXiv:2005.14165"},{"key":"2021071519044966400_bib5","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58539-6_34","article-title":"Behind the scene: Revealing the secrets of pre-trained vision-and-language models","author":"Cao","year":"2020","journal-title":"arXiv preprint arXiv:2005.07310"},{"key":"2021071519044966400_bib6","article-title":"Microsoft coco captions: Data collection and evaluation server","author":"Chen","year":"2015","journal-title":"arXiv preprint arXiv:1504.00325"},{"key":"2021071519044966400_bib7","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58577-8_7","article-title":"UNITER: Universal image-text representation learning","volume-title":"European Conference on Computer Vision (ECCV)","author":"Chen","year":"2020"},{"key":"2021071519044966400_bib8","doi-asserted-by":"publisher","DOI":"10.1145\/3240508.3240712","article-title":"Webly supervised joint embedding for cross- modal image-text retrieval","volume-title":"ACM International Conference on Multimedia","author":"Chowdhury","year":"2018"},{"key":"2021071519044966400_bib9","article-title":"Bert: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2019","journal-title":"North American Chapter of the Association for Computational Linguistics (NAACL)"},{"key":"2021071519044966400_bib10","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1007\/3-540-47979-1_7","article-title":"Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary","volume-title":"European Conference on Computer Vision (ECCV)","author":"Duygulu","year":"2002"},{"key":"2021071519044966400_bib11","article-title":"Vse++: Improving visual-semantic embeddings with hard negatives","author":"Faghri","year":"2017","journal-title":"arXiv preprint arXiv:1707.05612"},{"key":"2021071519044966400_bib12","article-title":"Devise: A deep visual-semantic embedding model","volume-title":"NIPS","author":"Frome","year":"2013"},{"key":"2021071519044966400_bib13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1303","article-title":"Image pivoting for learning multilingual multimodal representations","author":"Gella","year":"2017","journal-title":"arXiv preprint arXiv:1707.07601"},{"key":"2021071519044966400_bib14","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-013-0658-4","article-title":"A multi-view embedding space for modeling internet images, tags, and their semantics","author":"Gong","year":"2014","journal-title":"IJCV"},{"key":"2021071519044966400_bib15","doi-asserted-by":"publisher","first-page":"6904","DOI":"10.1109\/CVPR.2017.670","article-title":"Making the V in VQA matter: Elevating the role of image understanding in visual question answering","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Goyal","year":"2017"},{"key":"2021071519044966400_bib16","doi-asserted-by":"publisher","first-page":"2733","DOI":"10.18653\/v1\/D19-1275","article-title":"Designing and interpreting probes with control tasks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Hewitt","year":"2019"},{"key":"2021071519044966400_bib17","article-title":"Pixel-bert: Aligning image pixels with text by deep multi-modal transformers","author":"Huang","year":"2020","journal-title":"arXiv preprint arXiv:2004.00849"},{"key":"2021071519044966400_bib18","doi-asserted-by":"publisher","first-page":"3128","DOI":"10.1109\/CVPR.2015.7298932","article-title":"Deep visual-semantic alignments for generating image descriptions","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Karpathy","year":"2015"},{"key":"2021071519044966400_bib19","article-title":"Unifying visual-semantic embeddings with multimodal neural language models","author":"Kiros","year":"2014","journal-title":"arXiv preprint arXiv:1411.2539"},{"issue":"1","key":"2021071519044966400_bib20","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","article-title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","volume":"123","author":"Krishna","year":"2017","journal-title":"International Journal of Computer Vision"},{"key":"2021071519044966400_bib21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-2012","article-title":"Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing","author":"Kudo","year":"2018","journal-title":"arXiv preprint arXiv:1808.06226"},{"key":"2021071519044966400_bib22","article-title":"The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale","author":"Kuznetsova","year":"2018","journal-title":"arXiv preprint arXiv: 1811.00982"},{"key":"2021071519044966400_bib23","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6795","article-title":"Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training","volume-title":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI","author":"Li","year":"2020"},{"key":"2021071519044966400_bib24","article-title":"Visualbert: A simple and performant baseline for vision and language","author":"Li","year":"2019","journal-title":"arXiv preprint arXiv:1908.03557"},{"key":"2021071519044966400_bib25","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-58577-8_8","article-title":"Oscar: Object-semantics aligned pre-training for vision-language tasks","volume-title":"European Conference on Computer Vision","author":"Li","year":"2020"},{"key":"2021071519044966400_bib26","first-page":"13","article-title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks","volume-title":"Advances in Neural Information Processing Systems","author":"Jiasen","year":"2019"},{"key":"2021071519044966400_bib27","first-page":"289","article-title":"Hierarchical question-image co-attention for visual question answering","volume-title":"Advances in Neural Information Processing Systems","author":"Jiasen","year":"2016"},{"key":"2021071519044966400_bib28","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00990","article-title":"End-to-end learning of visual Representations from uncurated instructional videos","volume-title":"Computer Vision and Pattern Recognition","author":"Miech","year":"2020"},{"key":"2021071519044966400_bib29","article-title":"Learning a text-video embedding from incomplete and heterogeneous data","author":"Miech","year":"2018","journal-title":"arXiv preprint arXiv:1804.02516"},{"key":"2021071519044966400_bib30","doi-asserted-by":"publisher","first-page":"2630","DOI":"10.1109\/ICCV.2019.00272","article-title":"Howto100m: Learning a text-video embedding by watching hundred million narrated video clips","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Miech","year":"2019"},{"key":"2021071519044966400_bib31","article-title":"Representation learning with contrastive predictive coding","volume-title":"arXiv preprint arXiv: 1807.03748","author":"Oord","year":"2018"},{"key":"2021071519044966400_bib32","first-page":"1143","article-title":"Im2text: Describing images using 1 million captioned photographs","volume-title":"Advances in Neural Information Processing Systems","author":"Ordonez","year":"2011"},{"key":"2021071519044966400_bib33","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58558-7_38","article-title":"Connecting vision and language with localized narratives","author":"Pont-Tuset","year":"2019","journal-title":"arXiv preprint arXiv: 1912.03098"},{"key":"2021071519044966400_bib34","first-page":"91","article-title":"Faster r-cnn: Towards real-time object detection with region proposal networks","volume-title":"Advances in Neural Information Processing Systems","author":"Ren","year":"2015"},{"key":"2021071519044966400_bib35","doi-asserted-by":"publisher","first-page":"2556","DOI":"10.18653\/v1\/P18-1238","article-title":"Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sharma","year":"2018"},{"key":"2021071519044966400_bib36","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01086","article-title":"Visual grounding in video for unsupervised word translation","volume-title":"Computer Vision and Pattern Recognition","author":"Sigurdsson","year":"2020"},{"key":"2021071519044966400_bib37","article-title":"Are we pretraining it right? digging deeper into visio-linguistic pretraining","author":"Singh","year":"2020","journal-title":"arXiv preprint arXiv:2004.08744"},{"key":"2021071519044966400_bib38","doi-asserted-by":"publisher","first-page":"7464","DOI":"10.1109\/ICCV.2019.00756","article-title":"Videobert: A joint model for video and language representation learning","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Sun","year":"2019"},{"key":"2021071519044966400_bib39","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1514","article-title":"Lxmert: Learning cross-modality encoder representations from transformers","volume-title":"Empirical Methods in Natural Language Processing","author":"Tan","year":"2019"},{"key":"2021071519044966400_bib40","doi-asserted-by":"publisher","first-page":"4223","DOI":"10.1109\/CVPR.2018.00444","article-title":"Tips and tricks for visual question answering: Learnings from the 2017 challenge","volume-title":"Proceedings of the IEEE Conference on Computer vision and Pattern Recognition","author":"Teney","year":"2018"},{"key":"2021071519044966400_bib41","article-title":"Contrastive multiview coding","author":"Tian","year":"2019","journal-title":"arXiv preprint arXiv:1906.05849"},{"key":"2021071519044966400_bib42","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2021071519044966400_bib43","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.14","article-title":"Information-theoretic probing with minimum description length","author":"Voita","year":"2020","journal-title":"arXiv preprint arXiv:2003.12298"},{"key":"2021071519044966400_bib44","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.541","article-title":"Learning deep structure-preserving image- text embeddings","volume-title":"CVPR","author":"Wang","year":"2016"},{"key":"2021071519044966400_bib45","article-title":"Wsabie: Scaling up to large vocabulary image annotation","volume-title":"IJCAI","author":"Weston","year":"2011"},{"key":"2021071519044966400_bib46","first-page":"5753","article-title":"Xlnet: Generalized autoregressive pretraining for language understanding","volume-title":"Advances in Neural Information Processing Systems","author":"Yang","year":"2019"},{"key":"2021071519044966400_bib47","article-title":"Learning and evaluating general linguistic intelligence","author":"Yogatama","year":"2019","journal-title":"arXiv preprint arXiv:1901.11373"},{"key":"2021071519044966400_bib48","article-title":"Large batch optimization for deep learning: Training bert in 76 minutes","author":"You","year":"2019","journal-title":"arXiv preprint arXiv:1904.00962"},{"key":"2021071519044966400_bib49","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1162\/tacl_a_00166","article-title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","volume":"2","author":"Young","year":"2014","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021071519044966400_bib50","doi-asserted-by":"publisher","first-page":"359","DOI":"10.18653\/v1\/W18-5448","article-title":"Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Zhang","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00385\/1929720\/tacl_a_00385.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00385\/1929720\/tacl_a_00385.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,4]],"date-time":"2023-01-04T13:31:26Z","timestamp":1672839086000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00385\/102847\/Decoupling-the-Role-of-Data-Attention-and-Losses"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":50,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00385","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}