{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T16:10:41Z","timestamp":1783527041418,"version":"3.55.0"},"reference-count":175,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2019,10,16]],"date-time":"2019-10-16T00:00:00Z","timestamp":1571184000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ARC Discovery","award":["DP160101458 and DP190102443"],"award-info":[{"award-number":["DP160101458 and DP190102443"]}]},{"DOI":"10.13039\/100000183","name":"Army Research Office","doi-asserted-by":"crossref","award":["W911NF-19-1-0356"],"award-info":[{"award-number":["W911NF-19-1-0356"]}],"id":[{"id":"10.13039\/100000183","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2020,11,30]]},"abstract":"<jats:p>Video description is the automatic generation of natural language sentences that describe the contents of a given video. It has applications in human-robot interaction, helping the visually impaired and video subtitling. The past few years have seen a surge of research in this area due to the unprecedented success of deep learning in computer vision and natural language processing. Numerous methods, datasets, and evaluation metrics have been proposed in the literature, calling the need for a comprehensive survey to focus research efforts in this flourishing new direction. This article fills the gap by surveying the state-of-the-art approaches with a focus on deep learning models; comparing benchmark datasets in terms of their domains, number of classes, and repository size; and identifying the pros and cons of various evaluation metrics, such as SPICE, CIDEr, ROUGE, BLEU, METEOR, and WMD. Classical video description approaches combined subject, object, and verb detection with template-based language models to generate sentences. However, the release of large datasets revealed that these methods cannot cope with the diversity in unconstrained open domain videos. Classical approaches were followed by a very short era of statistical methods that were soon replaced with deep learning, the current state-of-the-art in video description. Our survey shows that despite the fast-paced developments, video description research is still in its infancy due to the following reasons: Analysis of video description models is challenging, because it is difficult to ascertain the contributions towards accuracy or errors of the visual features and the adopted language model in the final description. Existing datasets neither contain adequate visual diversity nor complexity of linguistic structures. Finally, current evaluation metrics fall short of measuring the agreement between machine-generated descriptions with that of humans. We conclude our survey by listing promising future research directions.<\/jats:p>","DOI":"10.1145\/3355390","type":"journal-article","created":{"date-parts":[[2019,10,16]],"date-time":"2019-10-16T18:55:35Z","timestamp":1571252135000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":149,"title":["Video Description"],"prefix":"10.1145","volume":"52","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2763-2094","authenticated-orcid":false,"given":"Nayyer","family":"Aafaq","sequence":"first","affiliation":[{"name":"The University of Western Australia, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ajmal","family":"Mian","sequence":"additional","affiliation":[{"name":"The University of Western Australia, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Liu","sequence":"additional","affiliation":[{"name":"The University of Western Australia, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Syed Zulqarnain","family":"Gilani","sequence":"additional","affiliation":[{"name":"The University of Western Australia and Edith Cowan University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mubarak","family":"Shah","sequence":"additional","affiliation":[{"name":"University of Central Florida, Orlando, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,10,16]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Casting Words transcription service 2014. Retrieved from: http:\/\/castingwords.com\/.  Casting Words transcription service 2014. Retrieved from: http:\/\/castingwords.com\/."},{"key":"e_1_2_1_2_1","unstructured":"Language in Vision 2017. Retrieved from: https:\/\/www.sciencedirect.com\/journal\/computer-vision-and-image-understanding\/vol\/163.  Language in Vision 2017. Retrieved from: https:\/\/www.sciencedirect.com\/journal\/computer-vision-and-image-understanding\/vol\/163."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the CVPR.","author":"Aafaq N."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the CVPR.","author":"Alayrac J."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the ECCV.","author":"Anderson P."},{"key":"e_1_2_1_6_1","unstructured":"B. Andrei E. Georgios H. Daniel M. Krystian N. Siddharth X. Caiming and Z. Yibiao. 2015. A workshop on language and vision at CVPR 2015.  B. Andrei E. Georgios H. Daniel M. Krystian N. Siddharth X. Caiming and Z. Yibiao. 2015. A workshop on language and vision at CVPR 2015."},{"key":"e_1_2_1_7_1","unstructured":"B. Andrei M. Tao N. Siddharth Z. Quanshi S. Nishant L. Jiebo and S. Rahul. 2018. A workshop on language and vision at CVPR 2018. http:\/\/languageandvision.com\/.  B. Andrei M. Tao N. Siddharth Z. Quanshi S. Nishant L. Jiebo and S. Rahul. 2018. A workshop on language and vision at CVPR 2018. http:\/\/languageandvision.com\/."},{"key":"e_1_2_1_8_1","unstructured":"R. Anna T. Atousa R. Marcus P. Christopher L. Hugo C. Aaron and S. Bernt. 2015. The joint video and language understanding workshop at ICCV 2015.  R. Anna T. Atousa R. Marcus P. Christopher L. Hugo C. Aaron and S. Bernt. 2015. The joint video and language understanding workshop at ICCV 2015."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the ICCV.","author":"Antol S."},{"key":"e_1_2_1_10_1","unstructured":"D. Bahdanau K. Cho and Y. Bengio. 2014. Neural machine translation by jointly learning to align and translate. Retrieved from: arXiv preprint arXiv:1409.0473 (2014).  D. Bahdanau K. Cho and Y. Bengio. 2014. Neural machine translation by jointly learning to align and translate. Retrieved from: arXiv preprint arXiv:1409.0473 (2014)."},{"key":"e_1_2_1_11_1","unstructured":"N. Ballas L. Yao C. Pal and A. Courville. 2015. Delving deeper into convolutional networks for learning video representations. Retrieved from: arXiv preprint arXiv:1511.06432 (2015).  N. Ballas L. Yao C. Pal and A. Courville. 2015. Delving deeper into convolutional networks for learning video representations. Retrieved from: arXiv preprint arXiv:1511.06432 (2015)."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization. 65--72","author":"Banerjee S."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the CVPR.","author":"Baraldi L."},{"key":"e_1_2_1_14_1","volume-title":"Salvi et\u00a0al","author":"Barbu A.","year":"2012"},{"key":"e_1_2_1_15_1","unstructured":"K. Barnard P. Duygulu D. Forsyth N. D. Freitas D. M. Blei and M. I. Jordan. 2003. Matching words and pictures. J. Mach. Learn. Res. 3 (Feb. 2003) 1107--1135.  K. Barnard P. Duygulu D. Forsyth N. D. Freitas D. M. Blei and M. I. Jordan. 2003. Matching words and pictures. J. Mach. Learn. Res. 3 (Feb. 2003) 1107--1135."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the CVPR.","author":"Berg T."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.643892"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the OPTRO.","author":"Burghouts G."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the CVPR.","author":"Heilbron F. C."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the AAAI\/IAAI. Citeseer, 132--137","author":"Brand M.","year":"1997"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the CVPR.","author":"Chaudhry R."},{"key":"e_1_2_1_22_1","unstructured":"D. Chen and W. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In ACL: Human Language Technologies-Volume 1. ACL 190--200.  D. Chen and W. Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In ACL: Human Language Technologies-Volume 1. ACL 190--200."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/1861751.1861761"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Y. Chen S. Wang W. Zhang and Q. Huang. 2018. Less is more: Picking informative frames for video captioning. Retrieved from: arXiv preprint arXiv:1803.01457 (2018).  Y. Chen S. Wang W. Zhang and Q. Huang. 2018. Less is more: Picking informative frames for video captioning. Retrieved from: arXiv preprint arXiv:1803.01457 (2018).","DOI":"10.1007\/978-3-030-01261-8_22"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"K. Cho B. V. Merri\u00ebnboer D. Bahdanau and Y. Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. Retrieved from: arXiv preprint arXiv:1409.1259 (2014).  K. Cho B. V. Merri\u00ebnboer D. Bahdanau and Y. Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. Retrieved from: arXiv preprint arXiv:1409.1259 (2014).","DOI":"10.3115\/v1\/W14-4012"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the CVPR.","author":"Dalal N."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the ECCV.","author":"Dalal N."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the CVPR.","author":"Das P."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the CVPR.","author":"Das A."},{"key":"e_1_2_1_31_1","unstructured":"J. Deng K. Li M. Do H. Su and L. Fei-Fei. 2009. Construction and analysis of a large scale image ontology. Vis. Sci. Soc. 186 2 (2009).  J. Deng K. Li M. Do H. Su and L. Fei-Fei. 2009. Construction and analysis of a large scale image ontology. Vis. Sci. Soc. 186 2 (2009)."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the ICMR.","author":"Ding D."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the CVPR.","author":"Donahue J."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2964284.2984064"},{"key":"e_1_2_1_35_1","first-page":"457","article-title":"Comparing automatic evaluation measures for image description","volume":"452","author":"Elliott D.","year":"2014","journal-title":"Proceedings of the ACL: Short Papers"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-009-0275-4"},{"key":"e_1_2_1_37_1","volume-title":"Platt et\u00a0al","author":"Fang H.","year":"2015"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the ECCV.","author":"Farhadi A."},{"key":"e_1_2_1_39_1","volume-title":"Wiley Online Library","author":"Fellbaum C."},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the CVPR.","author":"Felzenszwalb P."},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the CVPR.","author":"Felzenszwalb P. F."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2009.167"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the CVPR.","author":"Gan Z."},{"key":"e_1_2_1_44_1","volume-title":"TRECVID 2017: Evaluating ad hoc and instance video search, events detection, video captioning, and hyperlinking. In Proceedings of the TRECVID.","author":"George A."},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the EMNLP. 968--974","author":"Gella S."},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the ICCV.","author":"Gong S."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the ICML. 1764--1772","author":"Graves A."},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the ICASSP. 6645--6649","author":"Graves A."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the ICCV.","author":"Guadarrama S."},{"key":"e_1_2_1_50_1","volume-title":"Darrell et\u00a0al","author":"Guadarrama S.","year":"2013"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the AAAI. 263--268","author":"Hakeem A."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the ECCV.","author":"Hanckmann P."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the ECCV.","author":"Harwath D."},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the CVPR.","author":"He K."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the ICPR","volume":"1","author":"Hongeng S."},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the ICLR.","author":"Hudson Drew A.","year":"2018"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the MM. ACM, 675--678","author":"Jia Y."},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the MM. ACM, 1087--1091","author":"Jin Q."},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the CVPR.","author":"Johnson J."},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the EACL Workshop on Innovative Hybrid Approaches to the Processing of Textual Data. ACL, 27--35","author":"Khan M. U. G."},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the ICCV Workshops.","author":"Khan M. U. G."},{"key":"e_1_2_1_63_1","doi-asserted-by":"crossref","unstructured":"M. Kilickaya A. Erdem N. Ikizler-Cinbis and E. Erdem. 2016. Re-evaluating automatic metrics for image captioning. Retrieved from: arXiv preprint arXiv:1612.07600 (2016).  M. Kilickaya A. Erdem N. Ikizler-Cinbis and E. Erdem. 2016. Re-evaluating automatic metrics for image captioning. Retrieved from: arXiv preprint arXiv:1612.07600 (2016).","DOI":"10.18653\/v1\/E17-1019"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the ECCV.","author":"Kim J."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11265-009-0446-0"},{"key":"e_1_2_1_66_1","unstructured":"R. Kiros R. Salakhutdinov and R. S. Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. Retrieved from: arXiv preprint arXiv:1411.2539 (2014).  R. Kiros R. Salakhutdinov and R. S. Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. Retrieved from: arXiv preprint arXiv:1411.2539 (2014)."},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the 45th Meeting of the ACL on Interactive Poster and Demonstration Sessions. ACL, 177--180","author":"Koehn P."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1020346032608"},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the CVPR. 90--95","author":"Koller D."},{"key":"e_1_2_1_70_1","doi-asserted-by":"crossref","unstructured":"R. Krishna K. Hata F. Ren L. Fei-Fei and J. C. Niebles. 2017. Dense-captioning events in videos. Retrieved from: arXiv:1705.00754.  R. Krishna K. Hata F. Ren L. Fei-Fei and J. C. Niebles. 2017. Dense-captioning events in videos. Retrieved from: arXiv:1705.00754.","DOI":"10.1109\/ICCV.2017.83"},{"key":"e_1_2_1_71_1","first-page":"2","article-title":"Generating natural-language video descriptions using text-mined knowledge","volume":"1","author":"Krishnamoorthy N.","year":"2013","journal-title":"Proceedings of the AAAI"},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the NIPS. 1097--1105","author":"Krizhevsky A."},{"key":"e_1_2_1_73_1","first-page":"3","article-title":"Human face detection and tracking using skin color modeling and connected component operators","volume":"48","author":"Kuchi P.","year":"2002","journal-title":"IEEE J. Res."},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the CVPR.","author":"Kulkarni G."},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the NIPS. 3675--3683","author":"Kulkarni T. D."},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the ICML.","author":"Kusner M."},{"key":"e_1_2_1_77_1","unstructured":"I. Langkilde-Geary and K. Knight. Halogen Input Representation. [Online]. http:\/\/www.isi.edu\/publications\/licensed-sw\/halogen\/interlingua.html.  I. Langkilde-Geary and K. Knight. Halogen Input Representation. [Online]. http:\/\/www.isi.edu\/publications\/licensed-sw\/halogen\/interlingua.html."},{"key":"e_1_2_1_78_1","volume-title":"Proceedings of the CVPR Workshops. 1--8.","author":"Lee M. W."},{"key":"e_1_2_1_79_1","unstructured":"L. Li and B. Gong. 2018. End-to-end video captioning with multitask reinforcement learning. Retrieved from: arXiv preprint arXiv:1803.07950 (2018).  L. Li and B. Gong. 2018. End-to-end video captioning with multitask reinforcement learning. Retrieved from: arXiv preprint arXiv:1803.07950 (2018)."},{"key":"e_1_2_1_80_1","volume-title":"Proceedings of the CNLL.","author":"Li S."},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the CVPR.","author":"Li Y."},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the ACL-04 Workshop on Text Summarization Branches Out. 74--81","author":"Lin C.","year":"2004"},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the ECCV.","author":"Lin T."},{"key":"e_1_2_1_84_1","volume-title":"Proceedings of the MM. ACM, 631--634","author":"Liu Y."},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.5555\/850924.851523"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-007-0165-0"},{"key":"e_1_2_1_87_1","volume-title":"Proceedings of the NIPS. 1682--1690","author":"Malinowski M."},{"key":"e_1_2_1_88_1","volume-title":"Proceedings of the ICCV.","author":"Mao J."},{"key":"e_1_2_1_89_1","volume-title":"Proceedings of the First Workshop on Storytelling. ACL. https:\/\/www.aclweb.org\/anthology\/W18-1500","author":"Margaret M."},{"key":"e_1_2_1_90_1","volume-title":"Proceedings of the HRI.","author":"Matuszek C."},{"key":"e_1_2_1_91_1","volume-title":"Proceedings of the NIPS. 3111--3119","author":"Mikolov T."},{"key":"e_1_2_1_92_1","unstructured":"V. Mnih K. Kavukcuoglu D. Silver A. Graves I. Antonoglou D. Wierstra and M. Riedmiller. 2013. Playing Atari with deep reinforcement learning. Retrieved from: arXiv preprint arXiv:1312.5602. (2013).  V. Mnih K. Kavukcuoglu D. Silver A. Graves I. Antonoglou D. Wierstra and M. Riedmiller. 2013. Playing Atari with deep reinforcement learning. Retrieved from: arXiv preprint arXiv:1312.5602. (2013)."},{"key":"e_1_2_1_93_1","doi-asserted-by":"crossref","unstructured":"V. Mnih K. Kavukcuoglu D. Silver A. A. Rusu J. Veness M. G. Bellemare A. Graves M. Riedmiller A. K. Fidjeland G. Ostrovski S. Petersen C. Beattie A. Sadik I. Antonoglou H. King D. Kumaran D. Wierstra S. Legg and D. Hassbis. 2015. Human-level control through deep reinforcement learning. Nature 518 7540 (2015) 9529.  V. Mnih K. Kavukcuoglu D. Silver A. A. Rusu J. Veness M. G. Bellemare A. Graves M. Riedmiller A. K. Fidjeland G. Ostrovski S. Petersen C. Beattie A. Sadik I. Antonoglou H. King D. Kumaran D. Wierstra S. Legg and D. Hassbis. 2015. Human-level control through deep reinforcement learning. Nature 518 7540 (2015) 9529.","DOI":"10.1038\/nature14236"},{"key":"e_1_2_1_94_1","volume-title":"Proceedings of the AAAI\/IAAI. 770--776","author":"Moore D."},{"key":"e_1_2_1_95_1","volume-title":"Proceedings of the CVPR Workshop. 119--119","author":"Nevatia R."},{"key":"e_1_2_1_96_1","volume-title":"Proceedings of the COLING","volume":"1","author":"Nishida F."},{"key":"e_1_2_1_97_1","volume-title":"Proceedings of the COLING","volume":"2","author":"Nishida F."},{"key":"e_1_2_1_98_1","volume-title":"Proceedings of the ECCV.","author":"Owens A."},{"key":"e_1_2_1_99_1","volume-title":"Proceedings of the CVPR.","author":"Pan P."},{"key":"e_1_2_1_100_1","volume-title":"Proceedings of the CVPR.","author":"Pan Y."},{"key":"e_1_2_1_101_1","volume-title":"Proceedings of the CVPR.","author":"Pan Y."},{"key":"e_1_2_1_102_1","volume-title":"Proceedings of the ACL. 311--318","author":"Papineni K."},{"key":"e_1_2_1_103_1","doi-asserted-by":"crossref","unstructured":"R. Pasunuru and M. Bansal. 2017. Reinforced video captioning with entailment rewards. Retrieved from: arXiv preprint arXiv:1708.02300 (2017).  R. Pasunuru and M. Bansal. 2017. Reinforced video captioning with entailment rewards. Retrieved from: arXiv preprint arXiv:1708.02300 (2017).","DOI":"10.18653\/v1\/D17-1103"},{"key":"e_1_2_1_104_1","unstructured":"S. Phan G. E. Henter Y. Miyao and S. Satoh. 2017. Consensus-based sequence training for video captioning. Retrieved from: arXiv preprint arXiv:1712.09532 (2017).  S. Phan G. E. Henter Y. Miyao and S. Satoh. 2017. Consensus-based sequence training for video captioning. Retrieved from: arXiv preprint arXiv:1712.09532 (2017)."},{"key":"e_1_2_1_105_1","volume-title":"Proceedings of the CVPR.","author":"Pinhanez C. S."},{"key":"e_1_2_1_106_1","unstructured":"C. Pollard and I. A. Sag. 1994. Head-driven Phrase Structure Grammar. University of Chicago Press.  C. Pollard and I. A. Sag. 1994. Head-driven Phrase Structure Grammar. University of Chicago Press."},{"key":"e_1_2_1_107_1","unstructured":"S. Chen Y. Song Y. Zhao J. Qiu Q. Jin and A. Hauptmann. 2017. RUC-CMU: System descriptions for the dense video captioning task. Retrieved from: arXiv preprint arXiv:1710.08011 (2017).  S. Chen Y. Song Y. Zhao J. Qiu Q. Jin and A. Hauptmann. 2017. RUC-CMU: System descriptions for the dense video captioning task. Retrieved from: arXiv preprint arXiv:1710.08011 (2017)."},{"key":"e_1_2_1_108_1","volume-title":"Proceedings of the MM. ACM, 1092--1096","author":"Ramanishka V."},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00207"},{"key":"e_1_2_1_110_1","doi-asserted-by":"crossref","unstructured":"E. Reiter and R. Dale. 2000. Building Natural Language Generation Systems. Cambridge University Press.  E. Reiter and R. Dale. 2000. Building Natural Language Generation Systems. Cambridge University Press.","DOI":"10.1017\/CBO9780511519857"},{"key":"e_1_2_1_111_1","volume-title":"Proceedings of the NIPS. 2953--2961","author":"Ren M."},{"key":"e_1_2_1_112_1","doi-asserted-by":"crossref","unstructured":"Z. Ren X. Wang N. Zhang X. Lv and L. Li. 2017. Deep reinforcement learning-based image captioning with embedding reward. Retrieved from: arXiv preprint arXiv:1704.03899 (2017).  Z. Ren X. Wang N. Zhang X. Lv and L. Li. 2017. Deep reinforcement learning-based image captioning with embedding reward. Retrieved from: arXiv preprint arXiv:1704.03899 (2017).","DOI":"10.1109\/CVPR.2017.128"},{"key":"e_1_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.1108\/00220410410560582"},{"key":"e_1_2_1_114_1","volume-title":"Proceedings of the GCPR.","author":"Rohrbach A."},{"key":"e_1_2_1_115_1","volume-title":"Proceedings of the GCPR. 209--221","author":"Rohrbach A."},{"key":"e_1_2_1_116_1","volume-title":"Proceedings of the CVPR.","author":"Rohrbach A."},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0987-1"},{"key":"e_1_2_1_118_1","volume-title":"Proceedings of the CVPR.","author":"Rohrbach M."},{"key":"e_1_2_1_119_1","volume-title":"Proceedings of the ICCV.","author":"Rohrbach M."},{"key":"e_1_2_1_120_1","volume-title":"Proceedings of the ECCV.","author":"Rohrbach M."},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.5555\/1113166.1644543"},{"key":"e_1_2_1_122_1","doi-asserted-by":"publisher","DOI":"10.5555\/1113166.1644543"},{"key":"e_1_2_1_123_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026543900054"},{"key":"e_1_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_2_1_125_1","doi-asserted-by":"publisher","DOI":"10.1109\/78.650093"},{"key":"e_1_2_1_126_1","volume-title":"Proceedings of the CVPR.","author":"Shen Z."},{"key":"e_1_2_1_127_1","volume-title":"Proceedings of the MM. ACM, 1073--1076","author":"Shetty R."},{"key":"e_1_2_1_128_1","volume-title":"Proceedings of the CVPR.","author":"Shi J."},{"key":"e_1_2_1_129_1","volume-title":"Proceedings of the ICIP.","author":"Shin A."},{"key":"e_1_2_1_130_1","volume-title":"Proceedings of the ECCV.","author":"Sigurdsson G. A."},{"key":"e_1_2_1_131_1","unstructured":"K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from: arXiv preprint arXiv:1409.1556 (2014).  K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from: arXiv preprint arXiv:1409.1556 (2014)."},{"key":"e_1_2_1_132_1","volume-title":"Proceedings of the ICML. 843--852","author":"Srivastava N."},{"key":"e_1_2_1_133_1","volume-title":"Proceedings of the ECCV.","author":"Sun C."},{"key":"e_1_2_1_134_1","volume-title":"Proceedings of the NIPS. 3104--3112","author":"Sutskever I."},{"key":"e_1_2_1_135_1","volume-title":"Proceedings of the CVPR.","author":"Szegedy C."},{"key":"e_1_2_1_136_1","volume-title":"Proceedings of the AAAI.","author":"Tellex S.","year":"2011"},{"key":"e_1_2_1_137_1","first-page":"9","article-title":"Integrating language and vision to generate natural language descriptions of videos in the wild","volume":"2","author":"Thomason J.","year":"2014","journal-title":"Proceedings of the COLING"},{"key":"e_1_2_1_138_1","volume-title":"Technical Report CMU-CS-91-132. Carnegie Mellon University.","author":"Tomasi C.","year":"1991"},{"key":"e_1_2_1_139_1","unstructured":"A. Torabi C. Pal H. Larochelle and A. Courville. 2015. Using descriptive video services to create a large data source for video annotation research. Retrieved from: arXiv preprint arXiv:1503.01070 (2015).  A. Torabi C. Pal H. Larochelle and A. Courville. 2015. Using descriptive video services to create a large data source for video annotation research. Retrieved from: arXiv preprint arXiv:1503.01070 (2015)."},{"key":"e_1_2_1_140_1","volume-title":"Proceedings of the ICCV.","author":"Torralba A."},{"key":"e_1_2_1_141_1","unstructured":"D. Tran L. D. Bourdev R. Fergus L. Torresani and M. Paluri. 2014. C3D: Generic features for video analysis. Retrieved from: CoRR abs\/1412.0767 (2014).  D. Tran L. D. Bourdev R. Fergus L. Torresani and M. Paluri. 2014. C3D: Generic features for video analysis. Retrieved from: CoRR abs\/1412.0767 (2014)."},{"key":"e_1_2_1_142_1","volume-title":"Proceedings of the CVPR.","author":"Vedantam R."},{"key":"e_1_2_1_143_1","doi-asserted-by":"crossref","unstructured":"S. Venugopalan L. A. Hendricks R. Mooney and K. Saenko. 2016. Improving LSTM-based video description with linguistic knowledge mined from text. Retrieved from: arXiv preprint arXiv:1604.01729 (2016).  S. Venugopalan L. A. Hendricks R. Mooney and K. Saenko. 2016. Improving LSTM-based video description with linguistic knowledge mined from text. Retrieved from: arXiv preprint arXiv:1604.01729 (2016).","DOI":"10.18653\/v1\/D16-1204"},{"key":"e_1_2_1_144_1","volume-title":"Proceedings of the ICCV.","author":"Venugopalan S."},{"key":"e_1_2_1_145_1","doi-asserted-by":"crossref","unstructured":"S. Venugopalan H. Xu J. Donahue M. Rohrbach R. Mooney and K. Saenko. 2014. Translating videos to natural language using deep recurrent neural networks. Retrieved from: arXiv preprint arXiv:1412.4729 (2014).  S. Venugopalan H. Xu J. Donahue M. Rohrbach R. Mooney and K. Saenko. 2014. Translating videos to natural language using deep recurrent neural networks. Retrieved from: arXiv preprint arXiv:1412.4729 (2014).","DOI":"10.3115\/v1\/N15-1173"},{"key":"e_1_2_1_146_1","unstructured":"A. S. Vezhnevets S. Osindero T. Schaul N. Heess M. Jaderberg D. Silver and K. Kavukcuoglu. 2017. Feudal networks for hierarchical reinforcement learning. Retrieved from: arXiv preprint arXiv:1703.01161 (2017).  A. S. Vezhnevets S. Osindero T. Schaul N. Heess M. Jaderberg D. Silver and K. Kavukcuoglu. 2017. Feudal networks for hierarchical reinforcement learning. Retrieved from: arXiv preprint arXiv:1703.01161 (2017)."},{"key":"e_1_2_1_147_1","volume-title":"Proceedings of the CVPR.","author":"Vinyals O."},{"key":"e_1_2_1_148_1","volume-title":"Proceedings of the CVPR.","author":"Viola P."},{"key":"e_1_2_1_149_1","volume-title":"Talk the walk: Navigating New York City through grounded dialogue. Retrieved from: CoRR abs\/1807.03367","author":"de Vries Harm","year":"2018"},{"key":"e_1_2_1_150_1","volume-title":"Proceedings of the CVPR.","author":"Wang B."},{"key":"e_1_2_1_151_1","volume-title":"Proceedings of the BMVC. BMVA Press, 124--1.","author":"Wang H."},{"key":"e_1_2_1_152_1","volume-title":"Proceedings of the CVPR.","author":"Wang J."},{"key":"e_1_2_1_153_1","volume-title":"Proceedings of the CVPR.","author":"Wang J."},{"key":"e_1_2_1_154_1","volume-title":"Proceedings of the LREC. European Language Resources Association, 3059--3066","author":"Wang J. K."},{"key":"e_1_2_1_155_1","unstructured":"X. Wang W. Chen J. Wu Y. Wang and W. Y. Wang. 2017. Video captioning via hierarchical reinforcement learning. Retrieved from: arXiv preprint arXiv:1711.11135 (2017).  X. Wang W. Chen J. Wu Y. Wang and W. Y. Wang. 2017. Video captioning via hierarchical reinforcement learning. Retrieved from: arXiv preprint arXiv:1711.11135 (2017)."},{"key":"e_1_2_1_156_1","volume-title":"Proceedings of the CVPR.","author":"Wu X."},{"key":"e_1_2_1_157_1","volume-title":"Proceedings of the CVPR.","author":"Wu Q."},{"key":"e_1_2_1_158_1","doi-asserted-by":"crossref","unstructured":"H. Xu B. Li V. Ramanishka L. Sigal and K. Saenko. 2018. Joint event detection and description in continuous video streams. Retrieved from: arXiv preprint arXiv:1802.10250 (2018).  H. Xu B. Li V. Ramanishka L. Sigal and K. Saenko. 2018. Joint event detection and description in continuous video streams. Retrieved from: arXiv preprint arXiv:1802.10250 (2018).","DOI":"10.1109\/WACVW.2019.00011"},{"key":"e_1_2_1_159_1","unstructured":"H. Xu S. Venugopalan V. Ramanishka M. Rohrbach and K. Saenko. 2015. A multi-scale multiple instance video description network. Retrieved from: arXiv preprint arXiv:1505.05914 (2015).  H. Xu S. Venugopalan V. Ramanishka M. Rohrbach and K. Saenko. 2015. A multi-scale multiple instance video description network. Retrieved from: arXiv preprint arXiv:1505.05914 (2015)."},{"key":"e_1_2_1_160_1","volume-title":"Proceedings of the CVPR.","author":"Xu J."},{"key":"e_1_2_1_161_1","first-page":"6","article-title":"Jointly modeling deep video and compositional text to bridge vision and language in a unified framework","volume":"5","author":"Xu R.","year":"2015","journal-title":"Proceedings of the AAAI"},{"key":"e_1_2_1_162_1","volume-title":"Proceedings of the ICCV.","author":"Yao L."},{"key":"e_1_2_1_163_1","volume-title":"Proceedings of the MSR Asia MSM at ActivityNet Challenge","author":"Yao T.","year":"2017"},{"key":"e_1_2_1_164_1","first-page":"67","article-title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","volume":"2","author":"Young P.","year":"2014","journal-title":"ACL"},{"key":"e_1_2_1_165_1","volume-title":"Proceedings of the ACL 1. 53--63","author":"Yu H."},{"key":"e_1_2_1_166_1","volume-title":"Proceedings of the AAAI. 3855--3863","author":"Yu H."},{"key":"e_1_2_1_167_1","volume-title":"Proceedings of the CVPR.","author":"Yu H."},{"key":"e_1_2_1_168_1","volume-title":"Proceedings of the ICCV.","author":"Yu L."},{"key":"e_1_2_1_169_1","volume-title":"Proceedings of the CVPR.","author":"Yu Y."},{"key":"e_1_2_1_170_1","doi-asserted-by":"crossref","unstructured":"Y. Yu H. Ko J. Choi and G. Kim. 2016. End-to-end concept word detection for video captioning retrieval and question answering. Retrieved from: arXiv preprint arXiv:1610.02947 (2016).  Y. Yu H. Ko J. Choi and G. Kim. 2016. End-to-end concept word detection for video captioning retrieval and question answering. Retrieved from: arXiv preprint arXiv:1610.02947 (2016).","DOI":"10.1109\/CVPR.2017.347"},{"key":"e_1_2_1_171_1","volume-title":"Proceedings of the ECCV.","author":"Zeng K."},{"key":"e_1_2_1_172_1","volume-title":"Proceedings of the CVPR.","author":"Zhang X."},{"key":"e_1_2_1_173_1","unstructured":"L. Zhou Y. Kalantidis X. Chen J. J. Corso and M. Rohrbach. 2018. Grounded video description. Retrieved from: arXiv preprint arXiv:1812.06587 (2018).  L. Zhou Y. Kalantidis X. Chen J. J. Corso and M. Rohrbach. 2018. Grounded video description. Retrieved from: arXiv preprint arXiv:1812.06587 (2018)."},{"key":"e_1_2_1_174_1","volume-title":"Proceedings of the AAAI.","author":"Zhou L."},{"key":"e_1_2_1_175_1","volume-title":"Proceedings of the CVPR.","author":"Zhou L."},{"key":"e_1_2_1_176_1","doi-asserted-by":"publisher","DOI":"10.1561\/0600000018"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3355390","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3355390","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3355390","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:34Z","timestamp":1750202614000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3355390"}},"subtitle":["A Survey of Methods, Datasets, and Evaluation Metrics"],"short-title":[],"issued":{"date-parts":[[2019,10,16]]},"references-count":175,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2020,11,30]]}},"alternative-id":["10.1145\/3355390"],"URL":"https:\/\/doi.org\/10.1145\/3355390","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,16]]},"assertion":[{"value":"2019-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}