{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T06:24:43Z","timestamp":1777703083281,"version":"3.51.4"},"reference-count":35,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2019,5,14]],"date-time":"2019-05-14T00:00:00Z","timestamp":1557792000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"published-print":{"date-parts":[[2019,5,14]]},"abstract":"<jats:p>\u00a0Caption generation requires best of both Computer Vision and Natural Language Processing. Due to recent improvements in both of them many efficient models have been developed. Automatic Image Captioning can be utilized to provide descriptions of website content or to engender frame-by-frame descriptions of video for the vision-impaired and in many such applications. In this work, a model is described which is utilized to generate novel image captions for a previously unseen image by utilizing a multimodal architecture by amalgamation of a Recurrent Neural Network (RNN) and a Convolutional Neural Network (CNN). The model is trained on Microsoft Common Objects in Context (MSCOCO), an image captioning dataset that aligns captions and images in the same representation space, so that an image is close to its relevant captions in that space and far away from dissimilar captions and dissimilar images. ResNet-50 architecture is used for extracting features from the images and GloVe embeddings are used along with Gated Recurrent Unit (GRU) in Recurrent Neural Network (RNN) for text representation. MSCOCO evaluation server is used for evaluation of the machine generated caption for a given image.<\/jats:p>","DOI":"10.3233\/jifs-179027","type":"journal-article","created":{"date-parts":[[2019,5,14]],"date-time":"2019-05-14T12:04:11Z","timestamp":1557835451000},"page":"4787-4796","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":2,"title":["Generating image captions through multimodal embedding"],"prefix":"10.1177","volume":"36","author":[{"given":"Sandeep Kumar","family":"Dash","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, National Institute of Technology Mizoram, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saurav","family":"Saha","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, National Institute of Technology Mizoram, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Partha","family":"Pakray","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, National Institute of Technology Silchar, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander","family":"Gelbukh","sequence":"additional","affiliation":[{"name":"Natural Language Lab, Center for Computing Research, National Polytechnic Institute, Mexico"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2019,5,14]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"282","volume-title":"18th International Conference on Machine Learning Conditional Random Fields","author":"Lafferty J.","year":"2001","unstructured":"LaffertyJ., McCallumA. and PereiraF.C., Probabilistic models for segmenting and labeling sequence data, in: 18th International Conference on Machine Learning Conditional Random Fields, 2001, pp. 282\u2013289."},{"key":"e_1_3_2_3_2","first-page":"15","volume-title":"European Conference on Computer Vision","author":"Farhadi A.","year":"2010","unstructured":"FarhadiA., HejratiM., SadeghiM.A., YoungP., RashtchianC., HockenmaierJ. and ForsythD., Every picture tells a story: Generating sentences from images, in: European Conference on Computer Vision, Springer, Berlin, Heidelberg, 2010, pp. 15\u201329."},{"key":"e_1_3_2_4_2","first-page":"444","volume-title":"Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics","author":"Yang Y.","year":"2011","unstructured":"YangY., TeoC.L., Daum\u00c3l\u2019 IIIH. and AloimonosY., Corpusguided sentence generation of natural images, in: Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2011, pp. 444\u2013454."},{"key":"e_1_3_2_5_2","first-page":"1601","author":"Kulkarni G.","year":"2011","unstructured":"KulkarniG., PremrajV., DharS., LiS., ChoiY., BergA.C. and BergT.L., Baby talk: Understanding and generating image descriptions, in: 24th CVPR, 2011, pp. 1601\u20131608.","journal-title":"24th CVPR"},{"key":"e_1_3_2_6_2","first-page":"747","volume-title":"13th Conference of the European Chapter of the Association for Computational Linguistics","author":"Mitchell M.","year":"2012","unstructured":"MitchellM., HanX., DodgeJ., MenschA., GoyalA., BergA., YamaguchiK., BergT., StratosK. and Daum\u00c3l\u2019 IIIH., , Midge: Generating image descriptions from computer vision detections, in: 13th Conference of the European Chapter of the Association for Computational Linguistics, 2012, pp. 747\u2013756."},{"key":"e_1_3_2_7_2","first-page":"110","volume-title":"Third Joint Conference on Lexical and Computational Semantics","author":"Yatskar M.","year":"2014","unstructured":"YatskarM., GalleyM., VanderwendeL. and ZettlemoyerL., See No Evil, Say No Evil: Description Generation from Densely Labeled Images, in: Third Joint Conference on Lexical and Computational Semantics, 2014, pp. 110\u2013120."},{"key":"e_1_3_2_8_2","first-page":"1473","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Fang H.","year":"2015","unstructured":"FangH., GuptaS., IandolaF., SrivastavaR.K., DengL., Doll\u00c3\u0105rP. and GaoJ., From captions to visual concepts and back, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1473\u20131482."},{"key":"e_1_3_2_9_2","first-page":"1143","author":"Ordonez V.","year":"2011","unstructured":"OrdonezV., KulkarniG. and BergT.L., Im2text: Describing images using 1 million captioned photographs, in: Advances in Neural Information Processing Systems, 2011, pp. 1143\u20131151.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1011139631724"},{"issue":"11","key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"1958","DOI":"10.1109\/TPAMI.2008.128","volume":"30","author":"Torralba A.","year":"2008","unstructured":"TorralbaA., FergusR. and FreemanW.T., 80 million tiny images: A large data set for nonparametric object and scene recognition, in: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 11, 2008, 1958\u20131970.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_2_12_2","first-page":"294","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition Workshops","author":"Hodosh M.","year":"2013","unstructured":"HodoshM. and HockenmaierJ., Sentence-based image description with scalable, explicit models, in: IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2013, pp. 294\u2013300."},{"issue":"1","key":"e_1_3_2_13_2","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1162\/tacl_a_00177","article-title":"Grounded compositional semantics for finding and describing images with sentences","volume":"2","author":"Socher R.","year":"2014","unstructured":"SocherR., KarpathyA., LeQ.V., ManningC.D. and NgA.Y., Grounded compositional semantics for finding and describing images with sentences, in: Transactions of the Association of Computational Linguistics2(1) (2014), 207\u2013218.","journal-title":"Transactions of the Association of Computational Linguistics"},{"key":"e_1_3_2_14_2","first-page":"1889","author":"Karpathy A.","year":"2014","unstructured":"KarpathyA., JoulinA. and Fei-FeiL.F., Deep fragment embeddings for bidirectional image sentence mapping, in: Advances in Neural Information Processing Systems, 2014, pp. 1889\u20131897.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_15_2","unstructured":"BahdanauD. ChoK. and BengioY. Neural machine translation by jointly learning to align and translate arXiv preprint arXiv:1409.0473 2014."},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","unstructured":"ChoK. MerrienboerB.V. GulcehreC. BahdanauD. BougaresF. SchwenkH. and BengioY. Learning phrase representations using rnn encoder decoder for statistical machine translation arXiv preprint arXiv:1406.1078 2014.","DOI":"10.3115\/v1\/D14-1179"},{"key":"e_1_3_2_17_2","first-page":"3104","author":"Sutskever I.","year":"2014","unstructured":"SutskeverI., VinyalsO. and LeQ.V., Sequence to sequence learning with neural networks, in: Advances in Neural Information Processing Systems, 2014, pp. 3104\u20133112.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_18_2","first-page":"595","volume-title":"31st International Conference on Machine Learning (ICML-14)","author":"Kiros R.","year":"2014","unstructured":"KirosR., SalakhutdinovR. and ZemelR., Multimodal Neural Language Models, in: 31st International Conference on Machine Learning (ICML-14), 2014, pp. 595\u2013603."},{"key":"e_1_3_2_19_2","unstructured":"MaoJ. XuW. YangY. WangJ. HuangZ. and YuilleA. Deep captioning with multimodal recurrent neural networks (m-rnn) arXiv preprint arXiv:1412.6632 2014."},{"key":"e_1_3_2_20_2","first-page":"3156","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Oriol V.","year":"2015","unstructured":"OriolV., ToshevA., BengioS. and ErhanD., Show and tell: A neural image caption generator, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3156\u20133164."},{"key":"e_1_3_2_21_2","first-page":"2625","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Donahue J.","year":"2015","unstructured":"DonahueJ., HendricksL.A., GuadarramaS., RohrbachM., VenugopalanS., SaenkoK. and DarrellT., Long-term recurrent convolutional networks for visual recognition and description, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2625\u20132634."},{"key":"e_1_3_2_22_2","unstructured":"VenugopalanS. XuH. DonahueJ. RohrbachM. MooneyR. and SaenkoK. Translating videos to natural language using deep recurrent neural networks arXiv preprint arXiv:1412.4729 2014."},{"key":"e_1_3_2_23_2","first-page":"3128","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Karpathy A.","year":"2015","unstructured":"KarpathyA. and Fei-FeiL., Deep visual-semantic alignments for generating image descriptions, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3128\u20133137."},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","unstructured":"LuJ. XiongC. ParikhD. and SocherR. Knowing when to look: Adaptive attention via A visual sentinel for image captioning arXiv preprint arXiv:1612.01887 2016.","DOI":"10.1109\/CVPR.2017.345"},{"key":"e_1_3_2_25_2","unstructured":"MikolovT. ChenK. CorradoG. and DeanJ. Efficient estimation of word representations in vector space arXiv preprint arXiv:1301.3781 2013."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","unstructured":"RenZ. WangX. ZhangN. LvX. and LiL. Deep Reinforcement Learning-based Image Captioning with Embedding Reward arXiv preprint arXiv:1704.03899 2017.","DOI":"10.1109\/CVPR.2017.128"},{"key":"e_1_3_2_28_2","unstructured":"AndersonP. HeX. BuehlerC. TeneyD. JohnsonM. GouldS. and ZhangL. Bottom-up and top-down attention for image captioning and VQA arXiv preprint arXiv:1707.07998 2017."},{"key":"e_1_3_2_29_2","first-page":"740","author":"Lin T.Y.","year":"2014","unstructured":"LinT.Y., MaireM., BelongieS., HaysJ., PeronaP., RamananD., Doll\u00c3\u0105rP. and ZitnickC.L., Microsoft coco: Common objects in context, in: European Conference on Computer Vision, 2014, pp. 740\u2013755.","journal-title":"European Conference on Computer Vision"},{"key":"e_1_3_2_30_2","first-page":"770","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"He K.","year":"2016","unstructured":"HeK., ZhangX., RenS. and SunJ., Deep residual learning for image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770\u2013778."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_32_2","first-page":"63","author":"Loper E.","year":"2002","unstructured":"LoperE. and BirdS., NLTK: The natural language toolkit, in: ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics-Volume 1, 2002, pp. 63\u201370.","journal-title":"ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics-Volume 1"},{"key":"e_1_3_2_33_2","first-page":"311","volume-title":"40th Annual Meeting on Association for Computational Linguistics","author":"Papineni K.","year":"2002","unstructured":"PapineniK., RoukosS., WardT. and ZhuW.J., BLEU: A method for automatic evaluation of machine translation, in: 40th Annual Meeting on Association for Computational Linguistics, 2002, pp. 311\u2013318."},{"key":"e_1_3_2_34_2","first-page":"4566","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition","author":"Vedantam R.","year":"2015","unstructured":"VedantamR., ZitnickC.L. and ParikhD., Cider: Consensus-based image description evaluation, in: IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4566\u20134575."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"376","DOI":"10.3115\/v1\/W14-3348","author":"Denkowski M.","year":"2014","unstructured":"DenkowskiM. and LavieA., Meteor universal: Language specific translation evaluation for any target language, in: Ninth Workshop on Statistical Machine Translation, 2014, pp. 376\u2013380.","journal-title":"Ninth Workshop on Statistical Machine Translation"},{"key":"e_1_3_2_36_2","volume":"8","author":"Lin C.Y.","year":"2004","unstructured":"LinC.Y., Rouge: A package for automatic evaluation of summaries, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, vol. 8, 2004.","journal-title":"Text Summarization Branches Out: Proceedings of the ACL-04 Workshop"}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-179027","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/JIFS-179027","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/JIFS-179027","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:37:39Z","timestamp":1777455459000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/JIFS-179027"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,14]]},"references-count":35,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,5,14]]}},"alternative-id":["10.3233\/JIFS-179027"],"URL":"https:\/\/doi.org\/10.3233\/jifs-179027","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,5,14]]}}}