{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T18:23:52Z","timestamp":1790360632244,"version":"4.1.0"},"reference-count":50,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2020,7,8]],"date-time":"2020-07-08T00:00:00Z","timestamp":1594166400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,8]],"date-time":"2020-07-08T00:00:00Z","timestamp":1594166400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Image captioning is a challenging multimodal task. Significant improvements could be obtained by deep learning. Yet, captions generated by humans are still considered better, which makes it an interesting application for interactive machine learning and explainable artificial intelligence methods. In this work, we aim at improving the performance and explainability of the state-of-the-art method Show, Attend and Tell by augmenting their attention mechanism using additional bottom-up features. We compute visual attention on the joint embedding space formed by the union of high-level features and the low-level features obtained from the object specific salient regions of the input image. We embed the content of bounding boxes from a pre-trained Mask R-CNN model. This delivers state-of-the-art performance, while it provides explanatory features. Further, we discuss how interactive model improvement can be realized through re-ranking caption candidates using beam search decoders and explanatory features. We show that interactive re-ranking of beam search candidates has the potential to outperform the state-of-the-art in image captioning.<\/jats:p>","DOI":"10.1007\/s13218-020-00679-2","type":"journal-article","created":{"date-parts":[[2020,7,8]],"date-time":"2020-07-08T00:02:46Z","timestamp":1594166566000},"page":"571-584","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6532-4282","authenticated-orcid":false,"given":"Rajarshi","family":"Biswas","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Barz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Sonntag","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,7,8]]},"reference":[{"key":"679_CR1","doi-asserted-by":"crossref","unstructured":"Anderson P, Fernando B, Johnson M, Gould S (2016) Spice: semantic propositional image caption evaluation. In: ECCV","DOI":"10.1007\/978-3-319-46454-1_24"},{"key":"679_CR2","unstructured":"Banerjee S, Lavie A (2005) Meteor: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and\/or summarization, pp 65\u201372"},{"key":"679_CR3","unstructured":"Barz M, Sonntag D (2019) Incremental improvement of a question answering system by re-ranking answer candidates using machine learning. CoRR. arXiv:1908.10149"},{"key":"679_CR4","unstructured":"Biswas R (2019) Diverse image caption generation and automated human judgement through active learning. Master\u2019s thesis, Saarland University"},{"key":"679_CR5","doi-asserted-by":"crossref","unstructured":"Biswas R, Mogadala A, Barz M, Sonntag D, Klakow D (2019) Automatic judgement of neural network-generated image captions. In: Statistical language and speech processing\u20147th international conference, SLSP 2019, Ljubljana, Slovenia, October 14\u201316, 2019, Proceedings, pp 261\u2013272","DOI":"10.1007\/978-3-030-31372-2_22"},{"key":"679_CR6","doi-asserted-by":"crossref","unstructured":"Cho K, Merrienboer B, Gulcehre C, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. EMNLP","DOI":"10.3115\/v1\/D14-1179"},{"key":"679_CR7","doi-asserted-by":"crossref","unstructured":"Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. CVPR","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"679_CR8","doi-asserted-by":"crossref","unstructured":"Denil M, Bazzani L, Larochelle H, Freitas N (2012) Learning where to attend with deep architectures for image tracking. Neural Comput","DOI":"10.1162\/NECO_a_00312"},{"key":"679_CR9","doi-asserted-by":"crossref","unstructured":"Donahue J, Hendrikcs L, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, Darrell T (2015) Long-term recurrent convolutional networks for visual recognition and description. CVPR","DOI":"10.21236\/ADA623249"},{"key":"679_CR10","doi-asserted-by":"publisher","unstructured":"Dudley JJ, Kristensson PO (2018) A review of user interface design for interactive machine learning. ACM Trans Interact Intell Syst 8(2):1\u201337. https:\/\/doi.org\/10.1145\/3185517. http:\/\/dl.acm.org\/citation.cfm?doid=3232718.3185517","DOI":"10.1145\/3185517"},{"issue":"2","key":"679_CR11","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1609\/aimag.v40i2.2850","volume":"40","author":"D Gunning","year":"2019","unstructured":"Gunning D, Aha D (2019) DARPA\u2019s explainable artificial intelligence (XAI) program. AI Mag 40(2):44\u201358. https:\/\/doi.org\/10.1609\/aimag.v40i2.2850","journal-title":"AI Mag"},{"key":"679_CR12","doi-asserted-by":"crossref","unstructured":"Harnad S (1990) The symbol grounding problem. Physica 335\u2013346","DOI":"10.1016\/0167-2789(90)90087-6"},{"key":"679_CR13","doi-asserted-by":"crossref","unstructured":"He K, Gkioxari G, Doll\u00e1r P, Girshick R (2017) Mask R-CNN. In: IEEE international conference on computer vision (ICCV), pp 2980\u20132988","DOI":"10.1109\/ICCV.2017.322"},{"key":"679_CR14","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016, Las Vegas, NV, USA, June 27\u201330, 2016, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"679_CR15","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. CVPR","DOI":"10.1109\/CVPR.2016.90"},{"key":"679_CR16","doi-asserted-by":"crossref","unstructured":"Hochreiter S, Schmidhuber J (1997) Long short term memory. Neural Comput 1735\u20131780","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"679_CR17","unstructured":"Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167"},{"key":"679_CR18","first-page":"2017","volume":"28","author":"M Jaderberg","year":"2015","unstructured":"Jaderberg M, Simonyan K, Zisserman A, Kavukcuoglu K (2015) Spatial transformer networks. Adv Neural Inf Process Syst 28:2017\u20132025","journal-title":"Adv Neural Inf Process Syst"},{"key":"679_CR19","unstructured":"Jin J, Fu K, Cui R, Sha F, Zhang C (2015) Aligning where to see and what to tell: image caption with region-based attention and scene factorization. arXiv:1506.06272"},{"key":"679_CR20","doi-asserted-by":"publisher","unstructured":"Kalimuthu M, Barz M, Sonntag D (2019) Incremental domain adaptation for neural machine translation in low-resource settings. In: Proceedings of the fourth arabic natural language processing workshop, pp 1\u201310. Association for Computational Linguistics (ACL). https:\/\/doi.org\/10.18653\/v1\/w19-4601. https:\/\/www.aclweb.org\/anthology\/W19-4601","DOI":"10.18653\/v1\/w19-4601"},{"key":"679_CR21","doi-asserted-by":"crossref","unstructured":"Karpathy A, Fei-Fei L (2015) Deep visual-semantic alignments for generating image descriptions. CVPR","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"679_CR22","unstructured":"Karpathy A, Joulin A, Fei-Fei L (2014) Deep fragment embeddings for bidirectional image sentence mapping. NIPS"},{"key":"679_CR23","unstructured":"Kingma D, Ba J (2015) Adam: a method for stochastic optimization. ICLR"},{"key":"679_CR24","unstructured":"Kiros R, Salahutdinov R, Zemel R (2014) Multimodal neural language models. ICLR, pp 595\u2013603"},{"key":"679_CR25","unstructured":"Kiros R, Salahutdinov R, Zemel R (2014) Unifying visual-semantic embeddings with multimodal neural language models. arXiv:1411.2539"},{"key":"679_CR26","unstructured":"Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks. NIPS"},{"key":"679_CR27","unstructured":"Larochelle H, Hinton G (2010) Learning to combine foveal glimpses with a third-order Boltzmann machine. NIPS"},{"key":"679_CR28","doi-asserted-by":"crossref","unstructured":"Li J, Galley M, Brockett C, Gao J, Dolan B (2015) A diversity-promoting objective function for neural conversation models. arXiv:1506.01057","DOI":"10.18653\/v1\/N16-1014"},{"key":"679_CR29","unstructured":"Li J, Monroe W, Jurafsky D (2016) A simple, fast diverse decoding algorithm for neural generation. arXiv:1611.08562"},{"key":"679_CR30","unstructured":"Lin C (2004) Rouge: a package for automatic evaluation of summaries. In: Text summarization branches out"},{"key":"679_CR31","doi-asserted-by":"crossref","unstructured":"Lin T, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollar P, Zitnick C (2014) Microsoft coco: common objects in context. ECCV, pp 740\u2013755","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"679_CR32","doi-asserted-by":"crossref","unstructured":"Lowerre B, Reddy R (1990) The harpy speech understanding system. In: Readings in speech recognition, pp 576\u2013586","DOI":"10.1016\/B978-0-08-051584-7.50053-X"},{"key":"679_CR33","doi-asserted-by":"crossref","unstructured":"Oviatt S, Schuller B, Cohen P, Sonntag D, Potamianos G (2017) The handbook of multimodal-multisensor interfaces In: Foundations, user modeling, and common modality combinations, vol 1. ACM, New York","DOI":"10.1145\/3015783"},{"key":"679_CR34","doi-asserted-by":"crossref","unstructured":"Papineni K, Roukos S, Ward T, Zhu W (2002) Bleu: a method for automatic evaluation of machine translation. In: Association for computational linguistics, pp 311\u2013318","DOI":"10.3115\/1073083.1073135"},{"key":"679_CR35","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: an imperative style, high-performance deep learning library. In: Wallach H, Larochelle H, Beygelzimer A, d\u2019Alch\u00e9 Buc F, Fox E, Garnett R (eds) Advances in neural information processing systems, vol 32, pp 8026\u20138037. Curran Associates, Inc., New York. http:\/\/papers.nips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"},{"key":"679_CR36","doi-asserted-by":"crossref","unstructured":"Pedersoli M, Lucas T, Schmid C, Verbeek J (2017) Areas of attention for image captioning. In: 2017 IEEE international conference on computer vision (ICCV), pp 1251\u20131259","DOI":"10.1109\/ICCV.2017.140"},{"key":"679_CR37","first-page":"91","volume":"28","author":"S Ren","year":"2015","unstructured":"Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28:91\u201399","journal-title":"Adv Neural Inf Process Syst"},{"key":"679_CR38","doi-asserted-by":"crossref","unstructured":"Roy D, Reiter E (2005) Connecting language to the world. Artif Intell 1\u201312","DOI":"10.1016\/j.artint.2005.06.002"},{"key":"679_CR39","doi-asserted-by":"crossref","unstructured":"Savenkov D, Agichtein E (2016) CRQA: crowd-powered real-time automatic question answering system. In: Fourth AAAI conference on human computation and crowdsourcing","DOI":"10.1609\/hcomp.v4i1.13291"},{"key":"679_CR40","doi-asserted-by":"crossref","unstructured":"Selvaraju R, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: International conference on computer vision, ICCV 2017, Venice, Italy, October 22\u201329, 2017, pp 618\u2013626","DOI":"10.1109\/ICCV.2017.74"},{"key":"679_CR41","unstructured":"Settles B (2010) Active learning literature survey. University of Wisconsin, Madison, vol 52, no 55\u201366, p 11"},{"key":"679_CR42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1613\/jair.4539","volume":"53","author":"P Shivaswamy","year":"2015","unstructured":"Shivaswamy P, Joachims T (2015) Coactive learning. J Artif Intell Res 53:1\u201340","journal-title":"J Artif Intell Res"},{"key":"679_CR43","unstructured":"Simard P, Amershi S, Chickering M, Edelman Pelton A, Ghorashi S, Meek C, Ramos G, Suh J, Verwey J, Wang M, Wernsing J (2017) Machine teaching: a new paradigm for building machine learning systems. Technical report. https:\/\/www.microsoft.com\/en-us\/research\/publication\/machine-teaching-new-paradigm-building-machine-learning-systems\/"},{"key":"679_CR44","doi-asserted-by":"crossref","unstructured":"Teso S, Kersting K (2019) Explanatory interactive machine learning. In: Proceedings of the 2019 AAAI\/ACM conference on AI, ethics, and society, pp 239\u2013245","DOI":"10.1145\/3306618.3314293"},{"key":"679_CR45","doi-asserted-by":"crossref","unstructured":"Uijlings J, van de Sande K, Gevers T, Smeulders A (2013) Selective search for object recognition. Int J Comput Vis","DOI":"10.1007\/s11263-013-0620-5"},{"key":"679_CR46","doi-asserted-by":"crossref","unstructured":"Vedantam R, Zitnick C, Parikh D (2015) Cider: consensus-based image description evaluation. Comput Vis Pattern Recognit 4566\u20134575","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"679_CR47","doi-asserted-by":"crossref","unstructured":"Vinyals O, Toshev A, Bengio S, Erhan D (2015) Show and tell: a neural image caption generator. CVPR","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"679_CR48","doi-asserted-by":"publisher","unstructured":"Williams JD, Niraula NB, Dasigi P, Lakshmiratan A, Suarez CGJ, Reddy M, Zweig G (2015) Rapidly scaling dialog systems with interactive learning. In: Natural language dialog systems and intelligent assistants, pp 1\u201313. Springer International Publishing, Cham. https:\/\/doi.org\/10.1007\/978-3-319-19291-8_1","DOI":"10.1007\/978-3-319-19291-8_1"},{"key":"679_CR49","unstructured":"Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhutdinov R, Zemel R, Bengio Y (2015) Show, attend and tell: neural image caption generation with visual attention. ICML"},{"key":"679_CR50","first-page":"391","volume":"2014","author":"L Zitnick","year":"2014","unstructured":"Zitnick L, Doll\u00e1r P (2014) Edge boxes: Locating object proposals from edges. Comput Vis ECCV 2014:391\u2013405","journal-title":"Comput Vis ECCV"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00679-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13218-020-00679-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00679-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,1]],"date-time":"2022-11-01T17:32:18Z","timestamp":1667323938000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13218-020-00679-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,8]]},"references-count":50,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["679"],"URL":"https:\/\/doi.org\/10.1007\/s13218-020-00679-2","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,8]]},"assertion":[{"value":"30 March 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 June 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 July 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}