{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,2]],"date-time":"2025-07-02T20:40:50Z","timestamp":1751488850823,"version":"3.37.3"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,4,4]],"date-time":"2021-04-04T00:00:00Z","timestamp":1617494400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,4,4]],"date-time":"2021-04-04T00:00:00Z","timestamp":1617494400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["678017"],"award-info":[{"award-number":["678017"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Machine Translation"],"published-print":{"date-parts":[[2021,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We propose multimodal machine translation (MMT) approaches that exploit the correspondences between words and image regions. In contrast to existing work, our referential grounding method considers <jats:italic>objects<\/jats:italic> as the visual unit for grounding, rather than whole images or abstract image regions, and performs visual grounding in the <jats:italic>source<\/jats:italic> language, rather than at the decoding stage via attention. We explore two referential grounding approaches: (i) implicit grounding, where the model jointly learns how to ground the source language in the visual representation and to translate; and (ii) explicit grounding, where grounding is performed independent of the translation model, and is subsequently used to guide machine translation. We performed experiments on the Multi30K dataset for three language pairs: English\u2013German, English\u2013French and English\u2013Czech. Our referential grounding models outperform existing MMT models according to automatic and human evaluation metrics.<\/jats:p>","DOI":"10.1007\/s10590-021-09259-z","type":"journal-article","created":{"date-parts":[[2021,4,4]],"date-time":"2021-04-04T10:02:37Z","timestamp":1617530557000},"page":"145-165","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Read, spot and translate"],"prefix":"10.1007","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5495-3128","authenticated-orcid":false,"given":"Lucia","family":"Specia","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Josiah","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sun Jae","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alissa","family":"Ostapenko","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pranava","family":"Madhyastha","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,4,4]]},"reference":[{"key":"9259_CR1","unstructured":"Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd International conference on learning representations, ICLR 2015, San Diego, CA. arXiv:1409.0473"},{"key":"9259_CR2","doi-asserted-by":"crossref","unstructured":"Barrault L, Bougares F, Specia L, Lala C, Elliott D, Frank S (2018) Findings of the third shared task on multimodal machine translation. In: Proceedings of the third conference on machine translation: shared task papers. Association for Computational Linguistics, Brussels, pp 304\u2013323","DOI":"10.18653\/v1\/W18-6402"},{"key":"9259_CR3","doi-asserted-by":"crossref","unstructured":"Bojanowski P, Grave E, Joulin A, Mikolov T (2017) Enriching word vectors with subword information. Trans Assoc Comput Linguist 5:135\u2013146. https:\/\/transacl.org\/ojs\/index.php\/tacl\/article\/view\/999","DOI":"10.1162\/tacl_a_00051"},{"key":"9259_CR4","doi-asserted-by":"crossref","unstructured":"Caglayan O, Aransa W, Bardet A, Garc\u00eda-Mart\u00ednez M, Bougares F, Barrault L, Masana M, Herranz L, van de Weijer J (2017) LIUM-CVC submissions for WMT17 multimodal translation task. In: Proceedings of the second conference on machine translation, Copenhagen, Denmark, pp 432\u2013439","DOI":"10.18653\/v1\/W17-4746"},{"key":"9259_CR5","doi-asserted-by":"crossref","unstructured":"Calixto I, Liu Q (2017) Incorporating global visual features into attention-based neural machine translation. In: Proceedings of the 2017 conference on empirical methods in natural language processing, Copenhagen, Denmark, pp 992\u20131003","DOI":"10.18653\/v1\/D17-1105"},{"key":"9259_CR6","doi-asserted-by":"crossref","unstructured":"Calixto I, Liu Q, Campbell N (2017) Doubly-attentive decoder for multi-modal neural machine translation. In: Proceedings of the 55th annual meeting of the Association for Computational Linguistics. Long Papers, vol 1, Vancouver, Canada, pp 1913\u20131924","DOI":"10.18653\/v1\/P17-1175"},{"key":"9259_CR7","doi-asserted-by":"crossref","unstructured":"Cho K, van Merri\u00ebnboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: encoder\u2013decoder approaches. In: Proceedings of SSST-8, eighth workshop on syntax, semantics and structure in statistical translation, Doha, Qatar, pp 103\u2013111","DOI":"10.3115\/v1\/W14-4012"},{"key":"9259_CR8","doi-asserted-by":"crossref","unstructured":"Deena S, Ng RW, Madhyashtha P, Specia L, Hain T (2017) Exploring the use of acoustic embeddings in neural machine translation. In: Proceedings of IEEE automatic speech recognition and understanding workshop. IEEE, Okinawa. http:\/\/eprints.whiterose.ac.uk\/121515\/","DOI":"10.1109\/ASRU.2017.8268971"},{"key":"9259_CR9","doi-asserted-by":"crossref","unstructured":"Delbrouck JB, Dupont S (2017) An empirical study on the effectiveness of images in multimodal neural machine translation. In: Proceedings of the 2017 conference on empirical methods in natural language processing, Copenhagen, Denmark, pp 910\u2013919","DOI":"10.18653\/v1\/D17-1095"},{"key":"9259_CR10","unstructured":"Elliott D, K\u00e1d\u00e1r A (2017) Imagination improves multimodal translation. In: Proceedings of the eighth international joint conference on natural language processing. Long Papers, vol 1, Taipei, Taiwan, pp 130\u2013141"},{"key":"9259_CR11","unstructured":"Elliott D, Frank S, Hasler E (2015) Multi-language image description with neural sequence models. CoRR abs\/1510.04709. arXiv:1510.04709"},{"key":"9259_CR12","doi-asserted-by":"crossref","unstructured":"Elliott D, Frank S, Sima\u2019an K, Specia L (2016) Multi30K: multilingual English\u2013German image descriptions. In: Proceedings of the 5th workshop on vision and language, Berlin, Germany, pp 70\u201374","DOI":"10.18653\/v1\/W16-3210"},{"key":"9259_CR13","doi-asserted-by":"crossref","unstructured":"Elliott D, Frank S, Barrault L, Bougares F, Specia L (2017) Findings of the second shared task on multimodal machine translation and multilingual image description. In: Proceedings of the second conference on machine translation, Copenhagen, Denmark, pp 215\u2013233","DOI":"10.18653\/v1\/W17-4718"},{"key":"9259_CR14","doi-asserted-by":"crossref","unstructured":"Gr\u00f6nroos SA, Huet B, Kurimo M, Laaksonen J, Merialdo B, Pham P, Sj\u00f6berg M, Sulubacak U, Tiedemann J, Troncy R, V\u00e1zquez R (2018) The MeMAD submission to the WMT18 multimodal translation task. In: Proceedings of the third conference on machine translation: shared task papers, Brussels, Belgium, pp 603\u2013611","DOI":"10.18653\/v1\/W18-6439"},{"key":"9259_CR15","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"9259_CR16","doi-asserted-by":"crossref","unstructured":"Helcl J, Libovick\u00fd J, Varis D (2018) CUNI system for the WMT18 multimodal translation task. In: Proceedings of the third conference on machine translation: shared task papers, Brussels, Belgium, pp 616\u2013623","DOI":"10.18653\/v1\/W18-6441"},{"key":"9259_CR17","doi-asserted-by":"crossref","unstructured":"Hitschler J, Schamoni S, Riezler S (2016) Multimodal pivots for image caption translation. In: Proceedings of the 54th annual meeting of the Association for Computational Linguistics. Long Papers, vol 1, Berlin, Germany, pp 2399\u20132409","DOI":"10.18653\/v1\/P16-1227"},{"issue":"3\/4","key":"9259_CR18","doi-asserted-by":"publisher","first-page":"321","DOI":"10.2307\/2333955","volume":"28","author":"H Hotelling","year":"1936","unstructured":"Hotelling H (1936) Relations between two sets of variates. Biometrika 28(3\/4), 321\u2013377","journal-title":"Biometrika"},{"key":"9259_CR19","doi-asserted-by":"crossref","unstructured":"Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, Song Y, Guadarrama S, Murphy K (2017) Speed\/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), IEEE, Honolulu, HI, pp 3296\u20133297","DOI":"10.1109\/CVPR.2017.351"},{"key":"9259_CR20","doi-asserted-by":"crossref","unstructured":"Huang PY, Liu F, Shiang SR, Oh J, Dyer C (2016) Attention-based multimodal neural machine translation. In: Proceedings of the first conference on machine translation, Berlin, Germany, pp 639\u2013645","DOI":"10.18653\/v1\/W16-2360"},{"key":"9259_CR21","unstructured":"Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980"},{"key":"9259_CR22","unstructured":"Krasin I, Duerig T, Alldrin N, Ferrari V, Abu-El-Haija S, Kuznetsova A, Rom H, Uijlings J, Popov S, Veit A, Belongie S, Gomes V, Gupta A, Sun C, Chechik G, Cai D, Feng Z, Narayanan D, Murphy K (2017) OpenImages: a public dataset for large-scale multi-label and multi-class image classification. https:\/\/github.com\/openimages"},{"key":"9259_CR23","unstructured":"Lala C, Specia L (2018) Multimodal lexical translation. In: Proceedings of the language resources and evaluation conference (LREC), Miyazaki, Japan, pp 3810\u20133817"},{"key":"9259_CR24","doi-asserted-by":"crossref","unstructured":"Lavie A, Agarwal A (2007) METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In: Proceedings of the second workshop on statistical machine translation, Prague, Czech Republic, StatMT \u201907, pp 228\u2013231","DOI":"10.3115\/1626355.1626389"},{"key":"9259_CR25","doi-asserted-by":"crossref","unstructured":"Libovick\u00fd J, Helcl J (2017) Attention strategies for multi-source sequence-to-sequence learning. In: Proceedings of the 55th annual meeting of the Association for Computational Linguistics. Short Papers, vol 2, Vancouver, Canada, pp 196\u2013202","DOI":"10.18653\/v1\/P17-2031"},{"key":"9259_CR26","unstructured":"Lu J, Yang J, Batra D, Parikh D (2016) Hierarchical question-image co-attention for visual question answering. In: Proceedings of the 30th international conference on neural information processing systems, Barcelona, Spain, pp 289\u2013297"},{"key":"9259_CR27","doi-asserted-by":"crossref","unstructured":"Madhyastha P, Wang J, Specia L (2017) Sheffield MultiMT: using object posterior predictions for multimodal machine translation. In: Proceedings of the second conference on machine translation. Association for Computational Linguistics, Copenhagen, pp 470\u2013476","DOI":"10.18653\/v1\/W17-4752"},{"key":"9259_CR28","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems 26, Lake Tahoe, NV, pp 3111\u20133119"},{"key":"9259_CR29","doi-asserted-by":"crossref","unstructured":"Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on Association for Computational Linguistics, Philadelphia, PA, ACL \u201902, pp 311\u2013318","DOI":"10.3115\/1073083.1073135"},{"key":"9259_CR30","doi-asserted-by":"crossref","unstructured":"Plummer BA, Wang L, Cervantes CM, Caicedo JC, Hockenmaier J, Lazebnik S (2015) Flickr30k Entities: collecting region-to-phrase correspondences for richer image-to-sentence models. In: Proceedings of the IEEE international conference on computer vision (ICCV), Santiago, Chile, pp 2641\u20132649","DOI":"10.1109\/ICCV.2015.303"},{"key":"9259_CR31","doi-asserted-by":"crossref","unstructured":"Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, pp 6517\u20136525","DOI":"10.1109\/CVPR.2017.690"},{"key":"9259_CR32","unstructured":"Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems 28, Montr\u00e9al, Canada, pp 91\u201399"},{"key":"9259_CR33","doi-asserted-by":"crossref","unstructured":"Rohrbach A, Rohrbach M, Hu R, Darrell T, Schiele B (2016) Grounding of textual phrases in images by reconstruction. In: Proceedings of the European conference on computer vision (ECCV), Amsterdam, The Netherlands, pp 817\u2013834","DOI":"10.1007\/978-3-319-46448-0_49"},{"key":"9259_CR34","doi-asserted-by":"crossref","unstructured":"Sennrich R, Haddow B (2016) Linguistic input features improve neural machine translation. In: Proceedings of the first conference on machine translation. Research Papers, vol 1, Berlin, Germany, pp 83\u201391","DOI":"10.18653\/v1\/W16-2209"},{"key":"9259_CR35","doi-asserted-by":"crossref","unstructured":"Sennrich R, Haddow B, Birch A (2016) Neural machine translation of rare words with subword units. In: Proceedings of the 54th annual meeting of the Association for Computational Linguistics. Long Papers, vol 1, Berlin, Germany, pp 1715\u20131725","DOI":"10.18653\/v1\/P16-1162"},{"key":"9259_CR36","doi-asserted-by":"crossref","unstructured":"Shah K, Wang J, Specia L (2016) SHEF-Multimodal: grounding machine translation on images. In: First conference on machine translation. Shared Task Papers, vol 2, Berlin, Germany, pp 657\u2013662","DOI":"10.18653\/v1\/W16-2363"},{"key":"9259_CR37","doi-asserted-by":"crossref","unstructured":"Specia L, Frank S, Sima\u2019an K, Elliott D (2016) A shared task on multimodal machine translation and crosslingual image description. In: Proceedings of the first conference on machine translation. Shared Task Papers, vol 2, Berlin, Germany, pp 543\u2013553","DOI":"10.18653\/v1\/W16-2346"},{"key":"9259_CR38","doi-asserted-by":"crossref","unstructured":"Wang J, Specia L (2019) Phrase localization without paired training examples. In: Proceedings of the IEEE\/CVF international conference on computer vision, Seoul, South Korea, pp 4662\u20134671","DOI":"10.1109\/ICCV.2019.00476"},{"key":"9259_CR39","unstructured":"Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the 32nd international conference on machine learning, Lille, France, proceedings of machine learning research, vol 37, pp 2048\u20132057"},{"key":"9259_CR40","doi-asserted-by":"crossref","unstructured":"Young P, Lai A, Hodosh M, Hockenmaier J (2014) From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Trans Assoc Comput Linguist 2:67\u201378. https:\/\/transacl.org\/ojs\/index.php\/tacl\/article\/view\/229","DOI":"10.1162\/tacl_a_00166"}],"container-title":["Machine Translation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10590-021-09259-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10590-021-09259-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10590-021-09259-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,22]],"date-time":"2021-07-22T16:27:48Z","timestamp":1626971268000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10590-021-09259-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,4]]},"references-count":40,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,6]]}},"alternative-id":["9259"],"URL":"https:\/\/doi.org\/10.1007\/s10590-021-09259-z","relation":{},"ISSN":["0922-6567","1573-0573"],"issn-type":[{"type":"print","value":"0922-6567"},{"type":"electronic","value":"1573-0573"}],"subject":[],"published":{"date-parts":[[2021,4,4]]},"assertion":[{"value":"19 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 February 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 April 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}