{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T13:56:31Z","timestamp":1792072591444,"version":"4.4.0"},"reference-count":45,"publisher":"Cambridge University Press (CUP)","issue":"3","license":[{"start":{"date-parts":[[2018,4,23]],"date-time":"2018-04-23T00:00:00Z","timestamp":1524441600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2018,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>When a recurrent neural network (RNN) language model is used for caption generation, the image information can be fed to the neural network either by directly incorporating it in the RNN \u2013 conditioning the language model by \u2018injecting\u2019 image features \u2013 or in a layer following the RNN \u2013 conditioning the language model by \u2018merging\u2019 image features. While both options are attested in the literature, there is as yet no systematic comparison between the two. In this paper, we empirically show that it is not especially detrimental to performance whether one architecture is used or another. The merge architecture does have practical advantages, as conditioning by merging allows the RNN\u2019s hidden state vector to shrink in size by up to four times. Our results suggest that the visual and linguistic modalities for caption generation need not be jointly encoded by the RNN as that yields large, memory-intensive models with few tangible advantages in performance; rather, the multimodal integration should be delayed to a subsequent stage.<\/jats:p>","DOI":"10.1017\/s1351324918000098","type":"journal-article","created":{"date-parts":[[2018,4,23]],"date-time":"2018-04-23T08:42:33Z","timestamp":1524472953000},"page":"467-489","source":"Crossref","is-referenced-by-count":88,"title":["Where to put the image in an image caption generator"],"prefix":"10.1017","volume":"24","author":[{"given":"MARC","family":"TANTI","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"ALBERT","family":"GATT","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0436-6408","authenticated-orcid":false,"given":"KENNETH P.","family":"CAMILLERI","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2018,4,23]]},"reference":[{"key":"S1351324918000098_ref018","unstructured":"Kiros R. , Salakhutdinov R. , and Zemel R. S. 2014b. Unifying visual-semantic embeddings with multimodal neural language models. CoRR, 1411.2539."},{"key":"S1351324918000098_ref017","unstructured":"Kiros R. , Salakhutdinov R. , and Zemel R. S. 2014a. Multimodal neural language models. In Proceedings of the ICML\u201914, pp. 595\u2013603."},{"key":"S1351324918000098_ref004","doi-asserted-by":"crossref","unstructured":"Chen X. , and Zitnick C. L. 2015. Mind\u2019s eye: a recurrent visual representation for image caption generation. In Proceedings of the CVPR\u201915.","DOI":"10.1109\/CVPR.2015.7298856"},{"key":"S1351324918000098_ref010","first-page":"249","article-title":"Understanding the difficulty of training deep feedforward neural networks","volume":"9","author":"Glorot","year":"2010","journal-title":"Aistats"},{"key":"S1351324918000098_ref009","doi-asserted-by":"crossref","unstructured":"Donahue J. , Hendricks L. A. , Guadarrama S. , Rohrbach M. , Venugopalan S. , Saenko K. , and Darrell T. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the CVPR\u201915.","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"S1351324918000098_ref006","doi-asserted-by":"crossref","unstructured":"Deng J. , Dong W. , Socher R. , Li L.-J. , Li K. , and Fei-Fei L. 2009. ImageNet: a large-scale hierarchical image database. In Proceedings of the CVPR\u201909.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"S1351324918000098_ref002","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1613\/jair.4900","article-title":"Automatic description generation from images: a survey of models, datasets, and evaluation measures","volume":"55","author":"Bernardi","year":"2016","journal-title":"JAIR"},{"key":"S1351324918000098_ref016","doi-asserted-by":"crossref","unstructured":"Karpathy A. , and Fei-Fei L. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the CVPR\u201915.","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"S1351324918000098_ref028","doi-asserted-by":"crossref","unstructured":"Mnih A. , and Hinton G. 2007. Three new graphical models for statistical language modelling. In Proceedings of the ICML\u201907.","DOI":"10.1145\/1273496.1273577"},{"key":"S1351324918000098_ref007","first-page":"100","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Volume 2: Short Papers","author":"Devlin","year":"2015"},{"key":"S1351324918000098_ref008","unstructured":"Diederik P. K. , and Ba J. 2014. Adam: a method for stochastic optimization. CoRR, 1412.6980."},{"key":"S1351324918000098_ref003","unstructured":"Chen X. and Zitnick C. L. (2014). Learning a recurrent visual representation for image caption generation. CoRR, 1411.5654."},{"key":"S1351324918000098_ref042","first-page":"4904","volume-title":"IEEE International Conference on Computer Vision, ICCV 2017, October 22-29, 2017","author":"Yao","year":"2017"},{"key":"S1351324918000098_ref038","doi-asserted-by":"crossref","unstructured":"Vinyals O. , Toshev A. , Bengio S. , and Erhan D. 2015. Show and tell: A neural image caption generator. In Proceedings of the CVPR\u201915.","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"S1351324918000098_ref013","unstructured":"Hessel J. , Savva N. , and Wilber M. J. 2015. Image representations and new domains in neural image captioning. CoRR, 1508.02091."},{"key":"S1351324918000098_ref026","unstructured":"Mao J. , Xu W. , Yang Y. , Wang J. , and Yuille A. L. 2014. Explain images with multimodal recurrent neural networks. In Proceedings of the NIPS\u201914."},{"key":"S1351324918000098_ref001","unstructured":"Banerjee S. , and Lavie A. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, vol.u 29, pp. 65\u201372."},{"key":"S1351324918000098_ref045","unstructured":"Zhou L. , Xu C. , Koch P. , and Corso J. J. 2016. Image caption generation with text-conditional semantic attention. CoRR, 1606.04621."},{"key":"S1351324918000098_ref032","first-page":"1179","volume-title":"2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, July 21-26, 2017","author":"Rennie","year":"2017"},{"key":"S1351324918000098_ref005","unstructured":"Chung J. , G\u00fcl\u00e7ehre \u00c7. , Cho K. , and Bengio Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, 1412.3555."},{"key":"S1351324918000098_ref044","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1162\/tacl_a_00166","article-title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","volume":"2","author":"Young","year":"2014","journal-title":"TACL"},{"key":"S1351324918000098_ref011","doi-asserted-by":"publisher","DOI":"10.1016\/0167-2789(90)90087-6"},{"key":"S1351324918000098_ref023","doi-asserted-by":"crossref","unstructured":"Ma S. , and Han Y. 2016. Describing images by feeding LSTM with structural words. In Proceedings of the ICME\u201916.","DOI":"10.1109\/ICME.2016.7552883"},{"key":"S1351324918000098_ref019","doi-asserted-by":"crossref","unstructured":"Lin C.-Y. and Och F. J. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the ACL\u201904.","DOI":"10.3115\/1218955.1219032"},{"key":"S1351324918000098_ref021","unstructured":"Liu S. , Zhu Z. , Ye N. , Guadarrama S. , and Murphy K. 2016. Optimization of image description metrics using policy gradient methods. CoRR, 1612.00370."},{"key":"S1351324918000098_ref022","doi-asserted-by":"crossref","unstructured":"Lu J. , Xiong C. , Parikh D. , and Socher R. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 3242\u20133250.","DOI":"10.1109\/CVPR.2017.345"},{"key":"S1351324918000098_ref024","unstructured":"Mao J. , Xu W. , Yang Y. , Wang J. , Huang Z. , and Yuille A. 2015a. Deep captioning with multimodal recurrent neural networks (m-RNN). In Proceedings of the ICLR\u201915."},{"key":"S1351324918000098_ref025","doi-asserted-by":"crossref","unstructured":"Mao J. , Xu W. , Yang Y. , Wang J. , Huang Z. , and Yuille A. 2015b. Learning like a child: fast novel visual concept learning from sentence descriptions of images. In Proceedings of the ICCV\u201915.","DOI":"10.1109\/ICCV.2015.291"},{"key":"S1351324918000098_ref027","unstructured":"Mikolov T. , Chen K. , Corrado G. , and Dean J. 2013. Efficient estimation of word representations in vector space. CoRR, 1301.3781."},{"key":"S1351324918000098_ref033","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2005.04.007"},{"key":"S1351324918000098_ref034","unstructured":"Simonyan K. and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition. CoRR, 1409.1556."},{"key":"S1351324918000098_ref035","doi-asserted-by":"crossref","unstructured":"Song M. , and Yoo C. D. 2016. Multimodal representation: Kneser-Ney smoothing\/skip-gram based neural language model. In Proceedings of the ICIP\u201916.","DOI":"10.1109\/ICIP.2016.7532765"},{"key":"S1351324918000098_ref036","first-page":"3104","volume-title":"Advances in Neural Information Processing Systems 27","author":"Sutskever","year":"2014"},{"key":"S1351324918000098_ref037","doi-asserted-by":"crossref","unstructured":"Vedantam R. , Zitnick C. L. , and Parikh D. 2015. CIDEr: consensus-based image description evaluation. In Proceedings of the CVPR\u201915.","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"S1351324918000098_ref039","doi-asserted-by":"crossref","unstructured":"Wang M. , Song L. , Yang X. , and Luo C. 2016. A parallel-fusion RNN-LSTM architecture for image caption generation. In Proceedings of the ICIP\u201916.","DOI":"10.1109\/ICIP.2016.7533201"},{"key":"S1351324918000098_ref040","unstructured":"Wu Q. , Shen C. , van den Hengel A. , Liu L. , and Dick A. R. 2015. Image captioning with an intermediate attributes layer. CoRR, 1506.01144."},{"key":"S1351324918000098_ref041","unstructured":"Xu K. , Ba J. , Kiros R. , Cho K. , Courville A. C. , Salakhutdinov R. , Zemel R. S. , and Bengio Y. 2015. Show, attend and tell: neural image caption generation with visual attention. In Proceedings of the ICML\u201915."},{"key":"S1351324918000098_ref012","doi-asserted-by":"crossref","unstructured":"Hendricks L. A. , Venugopalan S. , Rohrbach M. , Mooney R. , Saenko K. , and Darrell T. 2016. Deep compositional captioning: describing novel object categories without paired training data. In Proceedings of the CVPR\u201916.","DOI":"10.1109\/CVPR.2016.8"},{"key":"S1351324918000098_ref043","doi-asserted-by":"crossref","unstructured":"You Q. , Jin H. , Wang Z. , Fang C. , and Luo J. 2016. Image captioning with semantic attention. In Proceedings of the CVPR\u201916.","DOI":"10.1109\/CVPR.2016.503"},{"key":"S1351324918000098_ref029","doi-asserted-by":"crossref","unstructured":"Nina O. , and Rodriguez A. 2015. Simplified LSTM unit and search space probability exploration for image description. In Proceedings of the ICICS\u201915.","DOI":"10.1109\/ICICS.2015.7459976"},{"key":"S1351324918000098_ref015","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1613\/jair.3994","article-title":"Framing image description as a ranking task: data, models and evaluation metrics","volume":"47","author":"Hodosh","year":"2013","journal-title":"JAIR"},{"key":"S1351324918000098_ref031","unstructured":"Papineni K. , Roukos S. , Ward T. , and Zhu W.-J. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the ACL\u201902, pp. 311\u20138."},{"key":"S1351324918000098_ref020","doi-asserted-by":"crossref","unstructured":"Lin T.-Y. , Maire M. , Belongie S. , Hays J. , Perona P. , Ramanan D. , Doll\u00e1r P. , and Zitnick C. L. 2014. Microsoft COCO: common objects in context. In Proceedings of the ECCV\u201914, pp. 740\u201355.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"S1351324918000098_ref030","doi-asserted-by":"crossref","unstructured":"Oruganti R. M. , Sah S. , Pillai S. , and Ptucha R. 2016. Image description through fusion based recurrent multi-modal learning. In Proceedings of the ICIP\u201916.","DOI":"10.1109\/ICIP.2016.7533033"},{"key":"S1351324918000098_ref014","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324918000098","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,1]],"date-time":"2020-11-01T00:58:50Z","timestamp":1604192330000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324918000098\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,4,23]]},"references-count":45,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,5]]}},"alternative-id":["S1351324918000098"],"URL":"https:\/\/doi.org\/10.1017\/s1351324918000098","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,4,23]]}}}