{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T13:47:31Z","timestamp":1773841651583,"version":"3.50.1"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T00:00:00Z","timestamp":1679616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Young Faculty Research Fellowship program of Visvesvaraya Ph.D. Scheme of Ministry of Electronics and Information Technology, Government of India"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>In sequence-to-sequence modeling tasks, such as image captioning, machine translation, and visual question answering, encoder-decoder architectures are state of the art. An encoder, convolutional neural network (CNN) encodes input images into fixed dimensional vector representation in the image captioning task, whereas a decoder, a recurrent neural network, performs language modeling and generates the target descriptions. Recent CNNs use the same operation over every pixel; however, all the image pixels are not equally important. To address this, the proposed method uses a dynamic convolution-based encoder for image encoding or feature extraction, Long-Short-Term-Memory as a decoder for language modeling, and X-Linear attention to make the system robust. Encoders, attentions, and decoders are important aspects of the image captioning task; therefore, we experiment with various encoders, decoders, and attention mechanisms. Most of the works for image captioning have been carried out for the English language in the existing literature. We propose a novel approach for caption generation from images in Hindi. Hindi, widely spoken in South Asia and India, is the fourth most-spoken language globally; it is India\u2019s official language. The proposed method utilizes dynamic convolution operation on the encoder side to obtain a better image encoding quality. The Hindi image captioning dataset is manually created by translating the popular MSCOCO dataset from English to Hindi. In terms of BLEU scores, the performance of the proposed method is compared with other baselines, and the results obtained show that the proposed method outperforms different baselines. Manual human assessment in terms of adequacy and fluency of the captions generated further determines the efficacy of the proposed method in generating good-quality captions.<\/jats:p>","DOI":"10.1145\/3573891","type":"journal-article","created":{"date-parts":[[2022,12,26]],"date-time":"2022-12-26T12:58:40Z","timestamp":1672059520000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Dynamic Convolution-based Encoder-Decoder Framework for Image Captioning in Hindi"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4639-5506","authenticated-orcid":false,"given":"Santosh Kumar","family":"Mishra","sequence":"first","affiliation":[{"name":"Indian Institute of Technology Patna"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8555-1810","authenticated-orcid":false,"given":"Sushant","family":"Sinha","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology Patna"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5458-9381","authenticated-orcid":false,"given":"Sriparna","family":"Saha","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology Patna"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5319-5508","authenticated-orcid":false,"given":"Pushpak","family":"Bhattacharyya","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology Bombay"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,3,24]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00636"},{"key":"e_1_3_1_3_2","unstructured":"Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs\/1409.0473 (2014). Retrieved from https:\/\/arxiv.org\/abs\/s1409.0473."},{"key":"e_1_3_1_4_2","unstructured":"Kyunghyun Cho Bart Van Merri\u00ebnboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Conference on Empirical Methods in Natural Language Processing . Retrieved from https:\/\/arxiv.org\/abs\/1406.1078."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01059"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2005.177"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01095"},{"issue":"3","key":"e_1_3_1_8_2","article-title":"A deep attention based framework for image caption generation in Hindi language","volume":"23","author":"Dhir Rijul","year":"2019","unstructured":"Rijul Dhir, Santosh Kumar Mishra, Sriparna Saha, and Pushpak Bhattacharyya. 2019. A deep attention based framework for image caption generation in Hindi language. Comput. Sist. 23, 3 (2019).","journal-title":"Comput. Sist."},{"key":"e_1_3_1_9_2","first-page":"1292","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Elliott Desmond","year":"2013","unstructured":"Desmond Elliott and Frank Keller. 2013. Image description using visual dependency representations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1292\u20131302."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/1888089.1888092"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00425"},{"key":"e_1_3_1_12_2","doi-asserted-by":"crossref","unstructured":"Akira Fukui Dong Huk Park Daylen Yang Anna Rohrbach Trevor Darrell and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. Conference on Empirical Methods in Natural Language Processing . Retrieved from https:\/\/arxiv.org\/abs\/1606.01847.","DOI":"10.18653\/v1\/D16-1044"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.41"},{"key":"e_1_3_1_14_2","unstructured":"Jane Gary and Carl Rubino. 2001. Facts About the World\u2019s Languages: An Encyclopedia of the World\u2019s Major Languages ."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC52423.2021.9658586"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10593-2_35"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.5555\/2566972.2566993"},{"key":"e_1_3_1_20_2","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201918).","author":"Hu Jie","year":"2018","unstructured":"Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks, 7132\u20137141. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201918)."},{"key":"e_1_3_1_21_2","unstructured":"Eric Jang Shixiang Gu and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. International Conference on Learning Representations . Retrieved from https:\/\/arxiv.org\/abs\/1611.01144."},{"key":"e_1_3_1_22_2","volume-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","author":"Jiang Wenhao","year":"2018","unstructured":"Wenhao Jiang, Lin Ma, Xinpeng Chen, Hanwang Zhang, and Wei Liu. 2018. Learning to guide decoding for image captioning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecoinf.2021.101549"},{"key":"e_1_3_1_25_2","first-page":"1564","volume-title":"Advances in Neural Information Processing Systems","author":"Kim Jin-Hwa","year":"2018","unstructured":"Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear attention networks. In Advances in Neural Information Processing Systems. 1564\u20131574."},{"key":"e_1_3_1_26_2","volume-title":"Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201911)","author":"Kulkarni Girish","year":"2011","unstructured":"Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C. Berg, and Tamara L. Berg. 2011. Baby talk: Understanding and generating image descriptions. In Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201911). Citeseer."},{"key":"e_1_3_1_27_2","first-page":"220","volume-title":"Proceedings of the 15th Conference on Computational Natural Language Learning","author":"Li Siming","year":"2011","unstructured":"Siming Li, Girish Kulkarni, Tamara L. Berg, Alexander C. Berg, and Yejin Choi. 2011. Composing simple image descriptions using web-scale n-grams. In Proceedings of the 15th Conference on Computational Natural Language Learning. Association for Computational Linguistics, 220\u2013228."},{"key":"e_1_3_1_28_2","first-page":"11588","volume-title":"Proceedings of the AAAI Annual Conference on Artificial Intelligence (AAAI\u201920)","author":"Liu Junhao","year":"2020","unstructured":"Junhao Liu, Kai Wang, Chunpu Xu, Zhou Zhao, Ruifeng Xu, Ying Shen, and Min Yang. 2020. Interactive dual generative adversarial networks for image captioning. In Proceedings of the AAAI Annual Conference on Artificial Intelligence (AAAI\u201920). 11588\u201311595."},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:VISI.0000029664.99615.94"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.345"},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","unstructured":"Minh-Thang Luong Hieu Pham and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing . 1412\u20131421.","DOI":"10.18653\/v1\/D15-1166"},{"key":"e_1_3_1_32_2","first-page":"5988","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"MacLeod Haley","year":"2017","unstructured":"Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017. Understanding blind people\u2019s experiences with computer-generated captions of social media images. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 5988\u20135999."},{"key":"e_1_3_1_33_2","unstructured":"Junhua Mao Wei Xu Yi Yang Jiang Wang and Alan L. Yuille. 2015. Deep captioning with multimodal recurrent neural networks (m-RNN). 3rd International Conference on Learning Representations (ICLR\u201915 San Diego CA USA May 7-9 2015 Conference Track Proceedings) Yoshua Bengio and Yann LeCun. http:\/\/arxiv.org\/abs\/1412.6632"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3432246"},{"key":"e_1_3_1_35_2","unstructured":"Santosh Kumar Mishra Rijul Dhir Sriparna Saha Pushpak Bhattacharyya and Amit Kumar Singh. 2020. Image captioning in Hindi language using transformer networks. (unpublished)."},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.compeleceng.2021.107114"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC52423.2021.9658859"},{"issue":"3","key":"e_1_3_1_38_2","first-page":"1","article-title":"Efficient channel attention based encoder\u2013decoder approach for image captioning in Hindi","volume":"21","author":"Mishra Santosh Kumar","year":"2021","unstructured":"Santosh Kumar Mishra, Gaurav Rai, Sriparna Saha, and Pushpak Bhattacharyya. 2021. Efficient channel attention based encoder\u2013decoder approach for image captioning in Hindi. Trans. Asian Low-Resour. Lang. Inf. Process. 21, 3 (2021), 1\u201317.","journal-title":"Trans. Asian Low-Resour. Lang. Inf. Process."},{"key":"e_1_3_1_39_2","first-page":"251","volume-title":"Proceedings of the 18th International Conference on Natural Language Processing (ICON\u201921)","author":"Mishra Santosh Kumar","year":"2021","unstructured":"Santosh Kumar Mishra, Sriparna Saha, and Pushpak Bhattacharyya. 2021. A scaled encoder decoder network for image captioning in Hindi. In Proceedings of the 18th International Conference on Natural Language Processing (ICON\u201921). 251\u2013260."},{"key":"e_1_3_1_40_2","article-title":"An object localization based dense image captioning framework in Hindi","author":"Mishra Santosh Kumar","year":"2022","unstructured":"Santosh Kumar Mishra, Sriparna Saha, and Pushpak Bhattacharyya. 2022. An object localization based dense image captioning framework in Hindi. Trans. Asian Low-Resour. Lang. Inf. Process. (2022).","journal-title":"Trans. Asian Low-Resour. Lang. Inf. Process."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-021-06144-x"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-020-05018-z"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-020-09827-4"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-021-10775-6"},{"key":"e_1_3_1_45_2","first-page":"404","volume-title":"European Conference on Computer Vision","author":"Ojala Timo","year":"2000","unstructured":"Timo Ojala, Matti Pietik\u00e4inen, and Topi M\u00e4enp\u00e4\u00e4. 2000. Gray scale and rotation invariant texture classification with local binary patterns. In European Conference on Computer Vision. Springer, 404\u2013420."},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01098"},{"key":"e_1_3_1_47_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_1_48_2","first-page":"91","volume-title":"Advances in Neural Information Processing Systems","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems. 91\u201399."},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"e_1_3_1_50_2","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998\u20136008."},{"key":"e_1_3_1_51_2","first-page":"2320","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Verelst Thomas","year":"2020","unstructured":"Thomas Verelst and Tinne Tuytelaars. 2020. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 2320\u20132329."},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"issue":"4","key":"e_1_3_1_53_2","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1109\/TPAMI.2016.2587640","article-title":"Show and tell: Lessons learned from the 2015 mscoco image captioning challenge","volume":"39","author":"Vinyals Oriol","year":"2016","unstructured":"Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE Trans. Pattern Anal. Mach. Intell. 39, 4 (2016), 652\u2013663.","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"e_1_3_1_54_2","first-page":"12637","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Xu Guanghui","year":"2021","unstructured":"Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo, Qing Du, and Qi Wu. 2021. Towards accurate text-based image captioning with content diversity exploration. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 12637\u201312646."},{"key":"e_1_3_1_55_2","first-page":"2048","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Xu Kelvin","year":"2015","unstructured":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning. 2048\u20132057."},{"key":"e_1_3_1_56_2","first-page":"2361","volume-title":"Advances in Neural Information Processing Systems","author":"Yang Zhilin","year":"2016","unstructured":"Zhilin Yang, Ye Yuan, Yuexin Wu, William W. Cohen, and Ruslan R. Salakhutdinov. 2016. Review networks for caption generation. In Advances in Neural Information Processing Systems. 2361\u20132369."},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.503"},{"key":"e_1_3_1_58_2","first-page":"574","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV\u201918)","author":"Yu Chaojian","year":"2018","unstructured":"Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You. 2018. Hierarchical bilinear pooling for fine-grained visual recognition. In Proceedings of the European Conference on Computer Vision (ECCV\u201918). 574\u2013589."},{"key":"e_1_3_1_59_2","first-page":"13041","volume-title":"Proceedings of the AAAI Annual Conference on Artificial Intelligence (AAAI\u201920)","author":"Zhou Luowei","year":"2020","unstructured":"Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and VQA. In Proceedings of the AAAI Annual Conference on Artificial Intelligence (AAAI\u201920). 13041\u201313049."}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3573891","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3573891","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:21Z","timestamp":1750182561000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3573891"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,24]]},"references-count":58,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3573891"],"URL":"https:\/\/doi.org\/10.1145\/3573891","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"value":"2375-4699","type":"print"},{"value":"2375-4702","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,24]]},"assertion":[{"value":"2021-12-10","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-11-12","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}