{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T05:13:05Z","timestamp":1780463585146,"version":"3.54.1"},"reference-count":62,"publisher":"Springer Science and Business Media LLC","issue":"25","license":[{"start":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T00:00:00Z","timestamp":1679616000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T00:00:00Z","timestamp":1679616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008252","name":"Universit\u00e0 degli Studi di Udine","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008252","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Video Question Answering (VideoQA) concerns the realization of models able to analyze a video, and produce a meaningful answer to visual content-related questions. To encode the given question, word embedding techniques are used to compute a representation of the tokens suitable for neural networks. Yet almost all the works in the literature use the same technique, although recent advancements in NLP brought better solutions. This lack of analysis is a major shortcoming. To address it, in this paper we present a twofold contribution about this inquiry and its relation with question encoding. First of all, we integrate four of the most popular word embedding techniques in three recent VideoQA architectures, and investigate how they influence the performance on two public datasets: EgoVQA and PororoQA. Thanks to the learning process, we show that embeddings carry question type-dependent characteristics. Secondly, to leverage this result, we propose a simple yet effective multi-task learning protocol which uses an auxiliary task defined on the question types. By using the proposed learning strategy, significant improvements are observed in most of the combinations of network architecture and embedding under analysis.<\/jats:p>","DOI":"10.1007\/s11042-023-14333-0","type":"journal-article","created":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T09:02:57Z","timestamp":1679648577000},"page":"38799-38826","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Video question answering supported by a multi-task learning objective"],"prefix":"10.1007","volume":"82","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6325-9066","authenticated-orcid":false,"given":"Alex","family":"Falcon","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giuseppe","family":"Serra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oswald","family":"Lanz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,3,24]]},"reference":[{"key":"14333_CR1","unstructured":"Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: ICLR"},{"key":"14333_CR2","doi-asserted-by":"crossref","unstructured":"Del Barrio E, Cuesta-Albertos JA, Matr\u00e1n C (2018) An optimal transportation approach for assessing almost stochastic order. In: The mathematics of the uncertain. Springer, p 33\u201344","DOI":"10.1007\/978-3-319-73848-2_3"},{"key":"14333_CR3","unstructured":"Devlin J, Chang MW, Lee K, et al. (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT"},{"key":"14333_CR4","doi-asserted-by":"publisher","unstructured":"Dror R, Shlomov S, Reichart R (2019) Deep dominance - how to properly compare deep neural models. In: Korhonen A, Traum D R, M\u00e0rquez L (eds) Proceedings of the 57th conference of the association for computational linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1 : Long Papers. Association for Computational Linguistics, pp 2773\u20132785. https:\/\/doi.org\/10.18653\/v1\/p19-1266","DOI":"10.18653\/v1\/p19-1266"},{"key":"14333_CR5","doi-asserted-by":"crossref","unstructured":"Fan C (2019) Egovqa - an egocentric video question answering benchmark dataset. In: ICCV Workshop","DOI":"10.1109\/ICCVW.2019.00536"},{"key":"14333_CR6","doi-asserted-by":"crossref","unstructured":"Fan C, Zhang X, Zhang S et al (2019) Heterogeneous memory enhanced multimodal attention model for video question answering. In: CVPR","DOI":"10.1109\/CVPR.2019.00210"},{"key":"14333_CR7","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1016\/j.patcog.2019.01.038","volume":"90","author":"Z Fang","year":"2019","unstructured":"Fang Z, Liu J, Li Y et al (2019) Improving visual question answering using dropout and enhanced question encoder. Pattern Recogn 90:404\u2013414","journal-title":"Pattern Recogn"},{"key":"14333_CR8","doi-asserted-by":"crossref","unstructured":"Gao J, Ge R, Chen K et al (2018) Motion-appearance co-memory networks for video question answering. In: CVPR","DOI":"10.1109\/CVPR.2018.00688"},{"key":"14333_CR9","doi-asserted-by":"crossref","unstructured":"Garcia N, Otani M, Chu C et al (2020) Knowit vqa: answering knowledge-based questions about videos. In: Proceedings of the AAAI conference on artificial intelligence, pp 10,826\u201310,834","DOI":"10.1609\/aaai.v34i07.6713"},{"key":"14333_CR10","doi-asserted-by":"crossref","unstructured":"Gardner M, Grus J, Neumann M et al (2017) Allennlp: a deep semantic natural language processing platform. arXiv:180307640","DOI":"10.18653\/v1\/W18-2501"},{"key":"14333_CR11","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: CVPR","DOI":"10.1109\/CVPR.2016.90"},{"key":"14333_CR12","doi-asserted-by":"crossref","unstructured":"Hori C, Hori T, Lee TY et al (2017) Attention-based multimodal fusion for video description. In: ICCV","DOI":"10.1109\/ICCV.2017.450"},{"key":"14333_CR13","doi-asserted-by":"crossref","unstructured":"Huang D, Chen P, Zeng R et al (2020) Location-aware graph convolutional networks for video question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 11,021\u201311,028","DOI":"10.1609\/aaai.v34i07.6737"},{"key":"14333_CR14","unstructured":"Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, PMLR, pp 448\u2013456"},{"key":"14333_CR15","doi-asserted-by":"crossref","unstructured":"Jang Y, Song Y, Yu Y, et al. (2017) Tgif-qa: toward spatio-temporal reasoning in visual question answering. In: CVPR","DOI":"10.1109\/CVPR.2017.149"},{"key":"14333_CR16","doi-asserted-by":"crossref","unstructured":"Jiang P, Han Y (2020) Reasoning with heterogeneous graph alignment for video question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 11,109\u201311,116","DOI":"10.1609\/aaai.v34i07.6767"},{"key":"14333_CR17","doi-asserted-by":"crossref","unstructured":"Karpathy A, Toderici G, Shetty S et al (2014) Large-scale video classification with convolutional neural networks. In: CVPR","DOI":"10.1109\/CVPR.2014.223"},{"key":"14333_CR18","doi-asserted-by":"crossref","unstructured":"Kim J, Ma M, Kim K, et al. (2019) Gaining extra supervision via multi-task learning for multi-modal video question answering. In: International Joint Conference on Neural Networks (IJCNN). IEEE, pp 1-8","DOI":"10.1109\/IJCNN.2019.8852087"},{"key":"14333_CR19","doi-asserted-by":"crossref","unstructured":"Kim KM, Heo MO, Choi SH, et al. (2017) Deepstory: video story qa by deep embedded memory networks. In: IJCAI","DOI":"10.24963\/ijcai.2017\/280"},{"key":"14333_CR20","doi-asserted-by":"crossref","unstructured":"Kim KM, Choi SH, Kim JH, et al. (2018) Multimodal dual attention memory for video story question answering. In: ECCV","DOI":"10.1109\/CVPR.2019.00853"},{"key":"14333_CR21","unstructured":"Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: ICLR"},{"key":"14333_CR22","unstructured":"Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. ICLR"},{"issue":"1","key":"14333_CR23","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","volume":"123","author":"R Krishna","year":"2017","unstructured":"Krishna R, Zhu Y, Groth O, et al. (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. Int J Comput Vis 123(1):32\u201373","journal-title":"Int J Comput Vis"},{"key":"14333_CR24","unstructured":"Lample G, Conneau A (2019) Cross-lingual language model pretraining. In: NeurIPS"},{"key":"14333_CR25","doi-asserted-by":"crossref","unstructured":"Lei J, Yu L, Bansal M et al (2018) Tvqa: localized, compositional video question answering. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 1369\u20131379","DOI":"10.18653\/v1\/D18-1167"},{"key":"14333_CR26","doi-asserted-by":"crossref","unstructured":"Lei J, Li L, Zhou L et al (2021) Less is more: clipbert for video-and-language learning via sparse sampling. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 7331\u20137341","DOI":"10.1109\/CVPR46437.2021.00725"},{"key":"14333_CR27","doi-asserted-by":"crossref","unstructured":"Li L, Chen YC, Cheng Y et al (2020) Hero: hierarchical encoder for video+ language omni-representation pre-training. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 2046\u20132065","DOI":"10.18653\/v1\/2020.emnlp-main.161"},{"key":"14333_CR28","doi-asserted-by":"crossref","unstructured":"Li X, Song J, Gao L et al (2019) Beyond rnns: positional self-attention with co-attention for video question answering. In: Proceedings of the AAAI conference on artificial intelligence, pp 8658\u20138665","DOI":"10.1609\/aaai.v33i01.33018658"},{"key":"14333_CR29","unstructured":"Liu Y, Ott M, Goyal N et al (2019) Roberta: a robustly optimized bert pretraining approach. arXiv:190711692"},{"key":"14333_CR30","doi-asserted-by":"crossref","unstructured":"Lu J, Goswami V, Rohrbach M et al (2020) 12-in-1: multi-task vision and language representation learning. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 10,437\u201310,446","DOI":"10.1109\/CVPR42600.2020.01045"},{"key":"14333_CR31","doi-asserted-by":"crossref","unstructured":"Luo Y, Zhao H, Zhan J (2020) Named entity recognition only from word embeddings. EMNLP","DOI":"10.18653\/v1\/2020.emnlp-main.723"},{"key":"14333_CR32","doi-asserted-by":"crossref","unstructured":"Miech A, Zhukov D, Alayrac JB et al (2019) Howto100m: learning a text-video embedding by watching hundred million narrated video clips. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 2630\u20132640","DOI":"10.1109\/ICCV.2019.00272"},{"key":"14333_CR33","unstructured":"Mikolov T, Sutskever I, Chen K, et al. (2013) Distributed representations of words and phrases and their compositionality. In: NeurIPS"},{"key":"14333_CR34","doi-asserted-by":"crossref","unstructured":"Park J, Lee J, Sohn K (2021) Bridge to answer: structure-aware graph interaction network for video question answering. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 15,526\u201315,535","DOI":"10.1109\/CVPR46437.2021.01527"},{"key":"14333_CR35","doi-asserted-by":"crossref","unstructured":"Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: EMNLP","DOI":"10.3115\/v1\/D14-1162"},{"key":"14333_CR36","doi-asserted-by":"crossref","unstructured":"Peters ME, Neumann M, Iyyer M, et al. (2018) Deep contextualized word representations. In: NAACL","DOI":"10.18653\/v1\/N18-1202"},{"key":"14333_CR37","doi-asserted-by":"crossref","unstructured":"Pfeiffer J, Vuli\u0107 I, Gurevych I et al (2020) Mad-x: an adapter-based framework for multi-task cross-lingual transfer. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 7654\u20137673","DOI":"10.18653\/v1\/2020.emnlp-main.617"},{"key":"14333_CR38","unstructured":"Sanh V, Debut L, Chaumond J, et al. (2019) Distilbert, a distilled version of bert: smaller, faster cheaper and lighter. In: NeurIPS Workshop"},{"key":"14333_CR39","doi-asserted-by":"crossref","unstructured":"Sennrich R, Haddow B, Birch A (2016) Neural machine translation of rare words with subword units. In: Proceedings of the 54th annual meeting of the association for computational linguistics (Volume 1 : Long Papers), pp 1715\u20131725","DOI":"10.18653\/v1\/P16-1162"},{"issue":"2","key":"14333_CR40","first-page":"93","volume":"10","author":"M Seok","year":"2016","unstructured":"Seok M, Song HJ, Park CY, et al. (2016) Named entity recognition using word embedding as a feature. Int J Softw Eng Appl 10(2):93\u2013104","journal-title":"Int J Softw Eng Appl"},{"key":"14333_CR41","unstructured":"Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR"},{"key":"14333_CR42","unstructured":"Soomro K, Zamir AR, Shah M (2012). In: Ucf101: a dataset of 101 human actions classes from videos in the wild. arXiv:12120402"},{"key":"14333_CR43","unstructured":"Standley T, Zamir A, Chen D et al (2020) Which tasks should be learned together in multi-task learning?. In: International conference on machine learning, PMLR, pp 9120\u20139132"},{"key":"14333_CR44","doi-asserted-by":"crossref","unstructured":"Sun G, Liang L, Li T, et al. (2021) Video question answering: a survey of models and datasets. Mob Netw Appl, 1\u201334","DOI":"10.1007\/s11036-020-01730-0"},{"key":"14333_CR45","doi-asserted-by":"crossref","unstructured":"Tapaswi M, Zhu Y, Stiefelhagen R, et al. (2016) Movieqa: understanding stories in movies through question-answering. In: CVPR","DOI":"10.1109\/CVPR.2016.501"},{"key":"14333_CR46","doi-asserted-by":"crossref","unstructured":"Tran D, Bourdev L, Fergus R, et al. (2015) Learning spatiotemporal features with 3d convolutional networks. In: ICCV","DOI":"10.1109\/ICCV.2015.510"},{"key":"14333_CR47","unstructured":"Ulmer D, Hardmeier C, Frellsen J (2022) Deep-significance-easy and meaningful statistical significance testing in the age of neural networks. arXiv:220406815"},{"key":"14333_CR48","unstructured":"Vaswani A, Shazeer N, Parmar N, et al. (2017) Attention is all you need. In: NeurIPS"},{"key":"14333_CR49","doi-asserted-by":"crossref","unstructured":"Wang Y, Zhai C, Awadalla HH (2020) Multi-task learning for multilingual neural machine translation. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 1022\u20131034","DOI":"10.18653\/v1\/2020.emnlp-main.75"},{"key":"14333_CR50","doi-asserted-by":"crossref","unstructured":"Wang J, Bao B, Xu C (2021) Dualvgr: a dual-visual graph reasoning unit for video question answering. IEEE Trans Multimedia","DOI":"10.1109\/TMM.2021.3097171"},{"key":"14333_CR51","unstructured":"Winterbottom T, Xiao S, McLean A et al (2020 ) On modality bias in the tvqa dataset. BMVC"},{"key":"14333_CR52","doi-asserted-by":"crossref","unstructured":"Wolf T, Debut L, Sanh V, other (2019) Transformers: state-of-the-art natural language processing. arXiv:191003771","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"14333_CR53","unstructured":"Wu Y, Schuster M, Chen Z, other (2016) Google\u2019s neural machine translation system: bridging the gap between human and machine translation. arXiv:160908144"},{"key":"14333_CR54","doi-asserted-by":"crossref","unstructured":"Xiao J, Shang X, Yao A et al (2021) Next-qa: next phase of question-answering to explaining temporal actions. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 9777\u20139786","DOI":"10.1109\/CVPR46437.2021.00965"},{"key":"14333_CR55","doi-asserted-by":"crossref","unstructured":"Xu D, Zhao Z, Xiao J, et al. (2017) Video question answering via gradually refined attention over appearance and motion. In: ACM Multimedia","DOI":"10.1145\/3123266.3123427"},{"key":"14333_CR56","doi-asserted-by":"crossref","unstructured":"Xu H, Ghosh G, Huang P Y, et al. (2021) Vlm: task-agnostic video-language model pre-training for video understanding. In: Findings of the association for computational linguistics : ACL-IJCNLP, pp 4227-4239","DOI":"10.18653\/v1\/2021.findings-acl.370"},{"key":"14333_CR57","doi-asserted-by":"crossref","unstructured":"Yang Z, Garcia N, Chu C, et al. (2020) Bert representations for video question answering. In: WACV","DOI":"10.1109\/WACV45572.2020.9093596"},{"key":"14333_CR58","doi-asserted-by":"crossref","unstructured":"Yang A, Miech A, Sivic J et al (2021) Just ask: learning to answer questions from millions of narrated videos. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 1686\u20131697","DOI":"10.1109\/ICCV48922.2021.00171"},{"key":"14333_CR59","doi-asserted-by":"crossref","unstructured":"Zamir AR, Sax A, Cheerla N et al (2020) Robust learning through cross-task consistency. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 11,197\u201311,206","DOI":"10.1109\/CVPR42600.2020.01121"},{"key":"14333_CR60","doi-asserted-by":"crossref","unstructured":"Zhang Y, Yang Q (2021) A survey on multi-task learning. IEEE Trans Knowl Data Eng","DOI":"10.1109\/TKDE.2021.3070203"},{"key":"14333_CR61","doi-asserted-by":"crossref","unstructured":"Zhao Z, Zhang Z, Xiao S et al (2018) Open-ended long-form video question answering via adaptive hierarchical reinforced networks. In: IJCAI, p 4","DOI":"10.24963\/ijcai.2018\/512"},{"key":"14333_CR62","doi-asserted-by":"crossref","unstructured":"Zhu L, Yang Y (2020) Actbert: learning global-local video-text representations. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 8746\u20138755","DOI":"10.1109\/CVPR42600.2020.00877"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-14333-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-023-14333-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-14333-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,10]],"date-time":"2023-10-10T10:16:14Z","timestamp":1696932974000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-023-14333-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,24]]},"references-count":62,"journal-issue":{"issue":"25","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["14333"],"URL":"https:\/\/doi.org\/10.1007\/s11042-023-14333-0","relation":{},"ISSN":["1380-7501","1573-7721"],"issn-type":[{"value":"1380-7501","type":"print"},{"value":"1573-7721","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,24]]},"assertion":[{"value":"26 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 June 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 January 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 March 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Competing interests"}}]}}