{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T02:52:48Z","timestamp":1790823168138,"version":"4.1.0"},"reference-count":61,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2018,4,25]],"date-time":"2018-04-25T00:00:00Z","timestamp":1524614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ISCRA"},{"name":"\u201cJUMP -Una piattaforma sensoristica avanzata per rinnovare la pratica e la fruizione dello sport, del benessere, della riabilitazione e del gioco educativo\u201d"},{"name":"NVIDIA Corporation with the donation of the GPUs used for this research"},{"name":"\u201cCitt\u00e0 educante\u201d of the National Technological Cluster on Smart Communities","award":["CTN01_00034_393801"],"award-info":[{"award-number":["CTN01_00034_393801"]}]},{"name":"Italian Ministry of Education, University and Research - MIUR"},{"DOI":"10.13039\/501100009879","name":"Emilia-Romagna region","doi-asserted-by":"crossref","award":["POR-FESR 2014-2020"],"award-info":[{"award-number":["POR-FESR 2014-2020"]}],"id":[{"id":"10.13039\/501100009879","id-type":"DOI","asserted-by":"crossref"}]},{"name":"CINECA"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2018,5,31]]},"abstract":"<jats:p>Image captioning has been recently gaining a lot of attention thanks to the impressive achievements shown by deep captioning architectures, which combine Convolutional Neural Networks to extract image representations and Recurrent Neural Networks to generate the corresponding captions. At the same time, a significant research effort has been dedicated to the development of saliency prediction models, which can predict human eye fixations. Even though saliency information could be useful to condition an image captioning architecture, by providing an indication of what is salient and what is not, research is still struggling to incorporate these two techniques. In this work, we propose an image captioning approach in which a generative recurrent neural network can focus on different parts of the input image during the generation of the caption, by exploiting the conditioning given by a saliency prediction model on which parts of the image are salient and which are contextual. We show, through extensive quantitative and qualitative experiments on large-scale datasets, that our model achieves superior performance with respect to captioning baselines with and without saliency and to different state-of-the-art approaches combining saliency and captioning.<\/jats:p>","DOI":"10.1145\/3177745","type":"journal-article","created":{"date-parts":[[2018,4,25]],"date-time":"2018-04-25T12:22:17Z","timestamp":1524658937000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":82,"title":["Paying More Attention to Saliency"],"prefix":"10.1145","volume":"14","author":[{"given":"Marcella","family":"Cornia","sequence":"first","affiliation":[{"name":"University of Modena and Reggio Emilia, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lorenzo","family":"Baraldi","sequence":"additional","affiliation":[{"name":"University of Modena and Reggio Emilia, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giuseppe","family":"Serra","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rita","family":"Cucchiara","sequence":"additional","affiliation":[{"name":"University of Modena and Reggio Emilia, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,4,25]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"International Conference on Learning Representations.","author":"Bahdanau Dzmitry","year":"2015"},{"key":"e_1_2_1_2_1","volume-title":"ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization.","author":"Banerjee Satanjeev","year":"2005"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.339"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/2354409.2354875"},{"key":"e_1_2_1_5_1","unstructured":"Zoya Bylinskii Tilke Judd Ali Borji Laurent Itti Fr\u00e9do Durand Aude Oliva and Antonio Torralba. 2017. MIT Saliency Benchmark. Retrieved from http:\/\/saliency.mit.edu\/. Zoya Bylinskii Tilke Judd Ali Borji Laurent Itti Fr\u00e9do Durand Aude Oliva and Antonio Torralba. 2017. MIT Saliency Benchmark. Retrieved from http:\/\/saliency.mit.edu\/."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46454-1_49"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.350"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2016.7900174"},{"key":"e_1_2_1_9_1","unstructured":"Marcella Cornia Lorenzo Baraldi Giuseppe Serra and Rita Cucchiara. 2017. Predicting human eye fixations via an LSTM-based saliency attentive model. arXiv preprint arXiv:1611.09571 (2017). Marcella Cornia Lorenzo Baraldi Giuseppe Serra and Rita Cucchiara. 2017. Predicting human eye fixations via an LSTM-based saliency attentive model. arXiv preprint arXiv:1611.09571 (2017)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-009-0275-4"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/1888089.1888092"},{"key":"e_1_2_1_12_1","volume-title":"International Conference on Artificial Intelligence and Statistics.","author":"Glorot Xavier","year":"2010"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.715"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9280.00255"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Jonathan Harel Christof Koch and Pietro Perona. 2006. Graph-based visual saliency. In Advances in Neural Information Processing Systems. Jonathan Harel Christof Koch and Pietro Perona. 2006. Graph-based visual saliency. In Advances in Neural Information Processing Systems.","DOI":"10.7551\/mitpress\/7503.003.0073"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/2566972.2566993"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/2919332.2919806"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.730558"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.620"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298710"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.494"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2009.5459462"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"e_1_2_1_25_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik","year":"2014"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.356"},{"key":"e_1_2_1_27_1","unstructured":"Alex Krizhevsky Ilya Sutskever and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. Alex Krizhevsky Ilya Sutskever and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems."},{"key":"e_1_2_1_28_1","volume-title":"IEEE International Conference on Computer Vision and Pattern Recognition.","author":"Kruthiventi Srinivas S. S."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.162"},{"key":"e_1_2_1_30_1","volume-title":"International Conference on Learning Representations Workshops.","author":"K\u00fcmmerer Matthias","year":"2015"},{"key":"e_1_2_1_31_1","unstructured":"Matthias K\u00fcmmerer Thomas S. A. Wallis and Matthias Bethge. 2016. DeepGaze II: Reading fixations from deep features trained on object recognition. arXiv preprint arXiv:1610.01563 (2016). Matthias K\u00fcmmerer Thomas S. A. Wallis and Matthias Bethge. 2016. DeepGaze II: Reading fixations from deep features trained on object recognition. arXiv preprint arXiv:1610.01563 (2016)."},{"key":"e_1_2_1_32_1","volume-title":"ACL Workshop on Text Summarization Branches Out.","author":"Lin Chin-Yew","year":"2004"},{"key":"e_1_2_1_33_1","volume-title":"European Conference on Computer Vision.","author":"Lin Tsung-Yi"},{"key":"e_1_2_1_34_1","unstructured":"Nian Liu and Junwei Han. 2016. A deep spatial contextual long-term recurrent convolutional network for saliency detection. arXiv preprint arXiv:1610.01708 (2016). Nian Liu and Junwei Han. 2016. A deep spatial contextual long-term recurrent convolutional network for saliency detection. arXiv preprint arXiv:1610.01708 (2016)."},{"key":"e_1_2_1_35_1","volume-title":"IEEE International Conference on Computer Vision and Pattern Recognition.","author":"Liu Nian","year":"2015"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.119"},{"key":"e_1_2_1_37_1","unstructured":"Vicente Ordonez Girish Kulkarni and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems. 1143\u20131151. Vicente Ordonez Girish Kulkarni and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems. 1143\u20131151."},{"key":"e_1_2_1_38_1","volume-title":"IEEE International Conference on Computer Vision and Pattern Recognition Workshops.","author":"Pan Junting","year":"2017"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.71"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.334"},{"key":"e_1_2_1_42_1","volume-title":"NAACL HLT Workshops.","author":"Rashtchian Cyrus","year":"2010"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1080\/135062800394667"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"Rakshith Shetty Marcus Rohrbach Lisa Anne Hendricks Mario Fritz and Bernt Schiele. 2017. Speaking the same language: Matching machine to human captions by adversarial training. arXiv preprint arXiv:1703.10476 (2017). Rakshith Shetty Marcus Rohrbach Lisa Anne Hendricks Mario Fritz and Bernt Schiele. 2017. Speaking the same language: Matching machine to human captions by adversarial training. arXiv preprint arXiv:1703.10476 (2017).","DOI":"10.1109\/ICCV.2017.445"},{"key":"e_1_2_1_46_1","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014). Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00177"},{"key":"e_1_2_1_48_1","unstructured":"Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203 (2016). Yusuke Sugano and Andreas Bulling. 2016. Seeing with humans: Gaze-assisted neural image captioning. arXiv preprint arXiv:1608.05203 (2016)."},{"key":"e_1_2_1_49_1","volume-title":"International Conference on Machine Learning.","author":"Sutskever Ilya","year":"2013"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.03.018"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.272"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.358"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"e_1_2_1_56_1","volume-title":"International Conference on Machine Learning.","author":"Xu Kelvin","year":"2015"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.5555\/2145432.2145484"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2010.2050411"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.503"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00166"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.26"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3177745","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3177745","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:05:51Z","timestamp":1750273551000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3177745"}},"subtitle":["Image Captioning with Saliency and Context Attention"],"short-title":[],"issued":{"date-parts":[[2018,4,25]]},"references-count":61,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,5,31]]}},"alternative-id":["10.1145\/3177745"],"URL":"https:\/\/doi.org\/10.1145\/3177745","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,4,25]]},"assertion":[{"value":"2017-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-04-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}