{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T12:00:16Z","timestamp":1769169616847,"version":"3.49.0"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2023,4,10]],"date-time":"2023-04-10T00:00:00Z","timestamp":1681084800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,10]],"date-time":"2023-04-10T00:00:00Z","timestamp":1681084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Ambient Intell Human Comput"],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Urdu is morphologically rich language and lacks the resources available in English. While several studies on the image captioning task in English have been published, this is among the pioneer studies on Urdu generative image captioning. The study makes several key contributions: (i) it presents a new dataset for Urdu image captioning, and (ii) it presents different attention-based architectures for image captioning in the Urdu language. These attention mechanisms are new to the Urdu language, as those have never been used for the Urdu image captioning task (iii) Finally, it performs quantitative and qualitative analysis of the results by studying the impact of different model architectures on Urdu\u2019s image caption generation task. The extensive experiments on the Urdu image caption generation task show encouraging results such as a BLEU-1 score of 72.5, BLEU-2 of 56.9, BLEU-3 of 42.8, and BLEU-4 of 31.6. Finally, we present data and code used in the study for future research via GitHub (<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/saeedhas\/Urdu_cap_gen\">https:\/\/github.com\/saeedhas\/Urdu_cap_gen<\/jats:ext-link>).<\/jats:p>","DOI":"10.1007\/s12652-023-04584-y","type":"journal-article","created":{"date-parts":[[2023,4,10]],"date-time":"2023-04-10T11:02:31Z","timestamp":1681124551000},"page":"7719-7731","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Generative image captioning in Urdu using deep learning"],"prefix":"10.1007","volume":"14","author":[{"given":"Muhammad Kashif","family":"Afzal","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Shardlow","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Suppawong","family":"Tuarob","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Farooq","family":"Zaman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0640-807X","authenticated-orcid":false,"given":"Raheem","family":"Sarwar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohsen","family":"Ali","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Naif Radi","family":"Aljohani","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miltiades D.","family":"Lytras","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Raheel","family":"Nawaz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saeed-Ul","family":"Hassan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,4,10]]},"reference":[{"key":"4584_CR1","doi-asserted-by":"crossref","unstructured":"Agrawal H, Desai K, Wang Y, Chen X, Jain R, Johnson M, Batra D, Parikh D, Lee S, Anderson P (2019) Nocaps: novel object captioning at scale. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp. 8948\u20138957","DOI":"10.1109\/ICCV.2019.00904"},{"key":"4584_CR2","unstructured":"Amjad M, Sidorov G, Zhila A (2020) Data augmentation using machine translation for fake news detection in the urdu language. In: Proceedings of the 12th language resources and evaluation conference, LREC 2020, Marseille, France, May 11-16, 2020. European Language Resources Association, pp. 2537\u20132542"},{"key":"4584_CR3","doi-asserted-by":"crossref","unstructured":"Aneja J, Deshpande A, Schwing AG (2018) Convolutional image captioning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5561\u20135570","DOI":"10.1109\/CVPR.2018.00583"},{"key":"4584_CR4","doi-asserted-by":"publisher","first-page":"597","DOI":"10.1162\/tacl_a_00288","volume":"7","author":"M Artetxe","year":"2019","unstructured":"Artetxe M, Schwenk H (2019) Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transac Assoc Comput Linguist 7:597\u2013610","journal-title":"Transac Assoc Comput Linguist"},{"key":"4584_CR5","unstructured":"Bahdanau D, Cho KH, Bengio Y (2015) Neural machine translation by jointly learning to align and translate.\u201d In: 3rd International Conference on Learning Representations"},{"key":"4584_CR6","doi-asserted-by":"crossref","unstructured":"Chen X, Lawrence Zitnick C (2015) Mind\u2019s eye: A recurrent visual representation for image caption generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2422\u20132431","DOI":"10.1109\/CVPR.2015.7298856"},{"key":"4584_CR7","doi-asserted-by":"crossref","unstructured":"Farhadi A, Hejrati M, Sadeghi MA, Young P, Rashtchian C, Hockenmaier J, Forsyth D (2010) Every picture tells a story: generating sentences from images.\u201d In: European conference on computer vision. Springer, pp. 15\u201329","DOI":"10.1007\/978-3-642-15561-1_2"},{"key":"4584_CR8","doi-asserted-by":"crossref","unstructured":"Fisch A, Lee K, Chang M, Clark JH, Barzilay R (2020) Capwap: image captioning with a purpose. In: Webber B, Cohn T, He Y, Liu Y (Eds.) Proceedings of the 2020 Conference on empirical methods in natural language processing. Association for computational linguistics, pp 8755\u20138768","DOI":"10.18653\/v1\/2020.emnlp-main.705"},{"key":"4584_CR9","doi-asserted-by":"crossref","unstructured":"Gong Y, Wang L, Hodosh M, Hockenmaier J, Lazebnik S (2014) Improving image-sentence embeddings using large weakly annotated photo collections. In: European conference on computer vision. Springer, pp 529\u2013545","DOI":"10.1007\/978-3-319-10593-2_35"},{"key":"4584_CR10","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770\u2013778","DOI":"10.1109\/CVPR.2016.90"},{"key":"4584_CR11","doi-asserted-by":"publisher","first-page":"853","DOI":"10.1613\/jair.3994","volume":"47","author":"M Hodosh","year":"2013","unstructured":"Hodosh M, Young P, Hockenmaier J (2013) Framing image description as a ranking task: data, models and evaluation metrics. J Artif Intell Res 47:853\u2013899","journal-title":"J Artif Intell Res"},{"key":"4584_CR12","doi-asserted-by":"crossref","unstructured":"Huang L, Wang W, Chen J, Wei X-Y (2019) Attention on attention for image captioning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4634\u20134643","DOI":"10.1109\/ICCV.2019.00473"},{"issue":"1","key":"4584_CR13","doi-asserted-by":"publisher","first-page":"8:1","DOI":"10.1145\/3329710","volume":"19","author":"S Kanwal","year":"2020","unstructured":"Kanwal S, Malik K, Shahzad K, Aslam F, Nawaz Z (2020) Urdu named entity recognition corpus generation and deep learning applications. ACM Trans Asian Low Resour Lang Inf Process 19(1):8:1-8:13. https:\/\/doi.org\/10.1145\/3329710","journal-title":"ACM Trans Asian Low Resour Lang Inf Process"},{"key":"4584_CR14","unstructured":"Karpathy A, Joulin A, Fei-Fei L (2014) Deep fragment embeddings for bidirectional image sentence mapping. arXiv preprint arXiv:1406.5679"},{"key":"4584_CR15","doi-asserted-by":"crossref","unstructured":"Kim J, Rohrbach A, Darrell T, Canny J, Akata Z (2018) Textual explanations for self-driving vehicles. In: Proceedings of the European conference on computer vision, pp 563\u2013578","DOI":"10.1007\/978-3-030-01216-8_35"},{"key":"4584_CR16","unstructured":"Kiros R, Salakhutdinov R, Zemel RS (2014) Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539"},{"issue":"12","key":"4584_CR17","doi-asserted-by":"publisher","first-page":"2891","DOI":"10.1109\/TPAMI.2012.162","volume":"35","author":"G Kulkarni","year":"2013","unstructured":"Kulkarni G, Premraj V, Ordonez V, Dhar S, Li S, Choi Y, Berg AC, Berg TL (2013) Babytalk: understanding and generating simple image descriptions. IEEE Transac Pattern Anal Mach Intell 35(12):2891\u20132903","journal-title":"IEEE Transac Pattern Anal Mach Intell"},{"key":"4584_CR18","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1162\/tacl_a_00188","volume":"2","author":"P Kuznetsova","year":"2014","unstructured":"Kuznetsova P, Ordonez V, Berg TL, Choi Y (2014) Treetalk: composition and compression of trees for image descriptions. Transac Assoc Comput Linguist 2:351\u2013362","journal-title":"Transac Assoc Comput Linguist"},{"key":"4584_CR20","unstructured":"Li S, Kulkarni G, Berg T, Berg A, Choi Y (2011) Composing simple image descriptions using web-scale n-grams. In: Proceedings of the fifteenth conference on computational natural language learning, pp. 220\u2013228"},{"issue":"3","key":"4584_CR19","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.102185","volume":"57","author":"C Li","year":"2020","unstructured":"Li C, Bao Z, Li L, Zhao Z (2020) Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNS for multi-modal emotion recognition. Inf Process Manag 57(3):102185","journal-title":"Inf Process Manag"},{"key":"4584_CR21","doi-asserted-by":"crossref","unstructured":"Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Doll\u00e1r P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision. Springer, pp. 740\u2013755","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"4584_CR22","doi-asserted-by":"crossref","unstructured":"Lindh A, Ross RJ, Kelleher JD (2020) Language-driven region pointer advancement for controllable image captioning. In: Scott D, Bel N, Zong C (Eds.) Proceedings of the 28th international conference on computational linguistics. International Committee on Computational Linguistics, pp 1922\u20131935","DOI":"10.18653\/v1\/2020.coling-main.174"},{"key":"4584_CR23","doi-asserted-by":"crossref","unstructured":"Liu J, Wang K, Xu C, Zhao Z, Xu R, Shen Y, Yang M (2020) Interactive dual generative adversarial networks for image captioning. In: The thirty-fourth AAAI conference on artificial intelligence, AAAI. AAAI Press, pp 11 588\u201311 595","DOI":"10.1609\/aaai.v34i07.6826"},{"issue":"4","key":"4584_CR24","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2020.102233","volume":"57","author":"Z Mahmood","year":"2020","unstructured":"Mahmood Z, Safder I, Nawab RMA, Bukhari F, Nawaz R, Alfakeeh AS, Aljohani NR, Hassan S-U (2020) Deep sentiments in roman Urdu text using recurrent convolutional neural network model. Inf Process Manag 57(4):102233","journal-title":"Inf Process Manag"},{"key":"4584_CR25","doi-asserted-by":"crossref","unstructured":"Makav B, K\u0131l\u0131\u00e7 V (2019) A new image captioning approach for visually impaired people. In: 2019 11th International conference on electrical and electronics engineering (ELECO). IEEE, pp 945\u2013949","DOI":"10.23919\/ELECO47770.2019.8990630"},{"issue":"1","key":"4584_CR26","volume":"38","author":"MI Malik","year":"2021","unstructured":"Malik MI, Sindhu MA, Khattak AS, Abbasi RA, Saleem K (2021) Automating test oracles from restricted natural language agile requirements. Expert Syst J Knowl Eng 38(1):e12608","journal-title":"Expert Syst J Knowl Eng"},{"key":"4584_CR27","unstructured":"Mao J, Xu W, Yang Y, Wang J, Huang Z, Yuille A (2014) Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632"},{"issue":"2","key":"4584_CR28","doi-asserted-by":"publisher","first-page":"32:1","DOI":"10.1145\/3432246","volume":"20","author":"SK Mishra","year":"2021","unstructured":"Mishra SK, Dhir R, Saha S, Bhattacharyya P (2021) A Hindi image caption generation framework using deep learning. ACM Trans Asian Low Resour Lang Inf Process 20(2):32:1-32:19","journal-title":"ACM Trans Asian Low Resour Lang Inf Process"},{"key":"4584_CR29","doi-asserted-by":"crossref","unstructured":"Pan Y, Yao T, Li Y, Mei T (2020) X-linear attention networks for image captioning. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp. 10971\u201310980","DOI":"10.1109\/CVPR42600.2020.01098"},{"key":"4584_CR30","doi-asserted-by":"crossref","unstructured":"Papineni K, Roukos S, Ward T, Zhu W-J (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the association for computational linguistics, pp. 311\u2013318","DOI":"10.3115\/1073083.1073135"},{"key":"4584_CR31","first-page":"1","volume":"25","author":"R Shaik","year":"2021","unstructured":"Shaik R, Venkatramaphanikumar S (2021) Sentiment analysis with word-based Urdu speech recognition. J Ambient Intell Humanized Comput 25:1\u201321","journal-title":"J Ambient Intell Humanized Comput"},{"key":"4584_CR32","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1162\/tacl_a_00177","volume":"2","author":"R Socher","year":"2014","unstructured":"Socher R, Karpathy A, Le QV, Manning CD, Ng AY (2014) Grounded compositional semantics for finding and describing images with sentences. Transac Assoc Comput Linguist 2:207\u2013218","journal-title":"Transac Assoc Comput Linguist"},{"key":"4584_CR34","doi-asserted-by":"crossref","unstructured":"Tanti M, Gatt A, Camilleri K (2017) What is the role of recurrent neural networks (rnns) in an image caption generator? In: Proceedings of the 10th international conference on natural language generation, pp. 51\u201360","DOI":"10.18653\/v1\/W17-3506"},{"issue":"3","key":"4584_CR33","doi-asserted-by":"publisher","first-page":"467","DOI":"10.1017\/S1351324918000098","volume":"24","author":"M Tanti","year":"2018","unstructured":"Tanti M, Gatt A, Camilleri KP (2018) Where to put the image in an image caption generator. Nat Lang Eng 24(3):467","journal-title":"Nat Lang Eng"},{"key":"4584_CR35","doi-asserted-by":"crossref","unstructured":"Vinyals O, Toshev A, Bengio S, Erhan D (2015) Show and tell: A neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156\u20133164","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"4584_CR36","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2019.107075","volume":"98","author":"J Wang","year":"2020","unstructured":"Wang J, Wang W, Wang L, Wang Z, Feng DD, Tan T (2020) Learning visual relationship and context-aware attention for image captioning. Pattern Recognit 98:107075","journal-title":"Pattern Recognit"},{"key":"4584_CR37","first-page":"117174","volume-title":"Expert systems with applications","author":"C Wang","year":"2022","unstructured":"Wang C, Shen Y, Ji L (2022) Geometry attention transformer with position-aware ISTMS for image captioning. Expert systems with applications. Elsevier, p 117174"},{"key":"4584_CR38","unstructured":"Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning, pp 2048\u20132057"},{"issue":"8","key":"4584_CR39","doi-asserted-by":"publisher","first-page":"1485","DOI":"10.1109\/JPROC.2010.2050411","volume":"98","author":"BZ Yao","year":"2010","unstructured":"Yao BZ, Yang X, Lin L, Lee MW, Zhu S-C (2010) I2t: image parsing to text description. Proc IEEE 98(8):1485\u20131508","journal-title":"Proc IEEE"},{"key":"4584_CR40","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1162\/tacl_a_00166","volume":"2","author":"P Young","year":"2014","unstructured":"Young P, Lai A, Hodosh M, Hockenmaier J (2014) From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transac Assoc Comput Linguist 2:67\u201378","journal-title":"Transac Assoc Comput Linguist"},{"key":"4584_CR41","unstructured":"Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y (2019) \u201cBertscore: Evaluating text generation with bert.\u201d In: International conference on learning representations"},{"key":"4584_CR42","doi-asserted-by":"crossref","unstructured":"Zhang P, Li X, Hu X, Yang J, Zhang L, Wang L, Choi Y, Gao J (2021) Vinvl: Revisiting visual representations in vision-language models. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp. 5579\u20135588","DOI":"10.1109\/CVPR46437.2021.00553"},{"issue":"07","key":"4584_CR43","first-page":"13041","volume":"34","author":"L Zhou","year":"2020","unstructured":"Zhou L, Palangi H, Zhang L, Hu H, Corso J, Gao J (2020a) Unified vision-language pre-training for image captioning and VGA. Proc AAAI Conf Artif Intell 34(07):13041\u201313049","journal-title":"Proc AAAI Conf Artif Intell"},{"key":"4584_CR44","doi-asserted-by":"crossref","unstructured":"Zhou Y, Wang M, Liu D, Hu Z, Zhang H (2020b) More grounded image captioning by distilling image-text matching model. In: 2020 IEEE\/CVF conference on computer vision and pattern recognition. IEEE, pp. 4776\u20134785","DOI":"10.1109\/CVPR42600.2020.00483"}],"container-title":["Journal of Ambient Intelligence and Humanized Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12652-023-04584-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12652-023-04584-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12652-023-04584-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,23]],"date-time":"2023-05-23T18:21:59Z","timestamp":1684866119000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12652-023-04584-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,10]]},"references-count":44,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,6]]}},"alternative-id":["4584"],"URL":"https:\/\/doi.org\/10.1007\/s12652-023-04584-y","relation":{},"ISSN":["1868-5137","1868-5145"],"issn-type":[{"value":"1868-5137","type":"print"},{"value":"1868-5145","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,10]]},"assertion":[{"value":"22 April 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 February 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 April 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}