{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T17:50:56Z","timestamp":1790272256081,"version":"4.1.0"},"reference-count":30,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T00:00:00Z","timestamp":1764892800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Open Access Publication Fund of Magdeburg University"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bioengineering"],"abstract":"<jats:p>This paper presents a new image captioning system which contains facial expression recognition as a way to provide better emotional and contextual comprehension of the captions generated. A combination of affective cues and visual features is made, which enables semantically full and emotionally conscious descriptions. Experiments were carried out on two created datasets, FlickrFace11k and COCOFace15k, with standard benchmarks such as BLEU, METEOR, ROUGE-L, CIDEr, and SPICE to analyze their effectiveness. The suggested model produced better results in all metrics as compared to baselines, like Show-Attend-Tell and Up-Down, remaining consistently better on all the scores. Remarkably, it has reached gains of 2.5 points on CIDEr and 1.0 on SPICE, which means a closer correlation to the prompt captions made by people. A 5-fold cross-validation confirmed the model\u2019s robustness, with minimal standard deviation across folds (&lt;\u00b10.2). Qualitative results further demonstrated its ability to capture fine-grained emotional expressions often missed by conventional models. These findings underscore the model\u2019s potential in affective computing, assistive technologies, and human-centric AI applications. The pipeline is designed for on-prem\/edge deployment with lightweight interfaces to IoT middleware (MQTT\/OPC UA), enabling smart-factory integration. These characteristics align the method with Industry 4.0 sensor networks and human-centric analytics.<\/jats:p>","DOI":"10.3390\/bioengineering12121325","type":"journal-article","created":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T10:50:38Z","timestamp":1764931838000},"page":"1325","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Image Captioning with Object Detection and Facial Expression Recognition for Smart Industry"],"prefix":"10.3390","volume":"12","author":[{"given":"Abdul Saboor","family":"Khan","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering and Information Technology, Otto-von-Guericke University, 39106 Magdeburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abdul Haseeb","family":"Khan","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Rheinland-Pf\u00e4lzische Technische Universit\u00e4t Kaiserslautern\u2013Landau (RPTU), 67663 Kaiserslautern, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2831-2316","authenticated-orcid":false,"given":"Muhammad Jamshed","family":"Abbass","sequence":"additional","affiliation":[{"name":"Faculty of Electrical Engineering, Wroc\u0142aw University of Science and Technology, 27 Wybrzeze Stanis\u0142awa Wyspianskiego St., 50-370 Wroclaw, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Imran","family":"Shafi","sequence":"additional","affiliation":[{"name":"College of Electrical and Mechanical Engineering, National University of Sciences and Technology (NUST), Islamabad 44000, Pakistan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,12,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"5719","DOI":"10.1007\/s00034-022-02050-2","article-title":"Image captioning encoder\u2013decoder models using CNN-RNN architectures: A comparative study","volume":"41","author":"Suresh","year":"2022","journal-title":"Circuits Syst. Signal Process."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., and Wei, F. (2020, January 23\u201328). Oscar: Object-semantics aligned pre-training for vision-language tasks. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.","DOI":"10.1007\/978-3-030-58577-8_8"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18\u201322). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00636"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1186\/s40537-022-00571-w","article-title":"Image captioning model using attention and object features to mimic human image understanding","volume":"9","author":"Jafar","year":"2022","journal-title":"J. Big Data"},{"key":"ref_5","unstructured":"Li, J., Li, D., Xiong, C., and Hoi, S. (2022, January 17\u201323). BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Proceedings of the 39th International Conference on Machine Learning (ICML), Baltimore, MD, USA."},{"key":"ref_6","unstructured":"Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. (2022, January 17\u201323). OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. Proceedings of the 39th International Conference on Machine Learning (ICML), Baltimore, MD, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mohamad Nezami, O., Dras, M., Anderson, P., and Hamey, L. (2019, January 16\u201320). Face-Cap: Image captioning using facial expression analysis. Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), W\u00fcrzburg, Germany.","DOI":"10.1007\/978-3-030-10925-7_14"},{"key":"ref_8","first-page":"1","article-title":"Image Captioning with Affective Guiding and Selective Attention","volume":"14","author":"Wang","year":"2018","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1499","DOI":"10.1109\/LSP.2016.2603342","article-title":"Joint face detection and alignment using multi-task cascaded convolutional networks","volume":"23","author":"Zhang","year":"2016","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1109\/TPAMI.2016.2598339","article-title":"Deep visual-semantic alignments for generating image descriptions","volume":"39","author":"Karpathy","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7\u201312). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"ref_12","unstructured":"Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015, January 6\u201311). Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Tan, M., Pang, R., and Le, Q.V. (2020, January 14\u201319). EfficientDet: Scalable and efficient object detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01079"},{"key":"ref_14","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7\u201312). Faster R-CNN: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, QC, Canada."},{"key":"ref_15","unstructured":"Jocher, G., Chaurasia, A., and Qiu, J. YOLOv5 by Ultralytics (Version 7.0) [Software]. Zenodo, 2020."},{"key":"ref_16","unstructured":"Parkhi, O., Vedaldi, A., and Zisserman, A. (2015, January 7\u201310). Deep face recognition. Proceedings of the British Machine Vision Conference (BMVC), Swansea, UK."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.H. (2013, January 3\u20137). Challenges in representation learning: A report on three machine learning contests. Proceedings of the International Conference on Neural Information Processing, Daegu, Republic of Korea.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhou, B., Zhai, Y., Li, J., Wang, Z., Zhang, Y., and Li, X. (2025). Improved IEC performance via emotional stimuli-aware captioning. Sci. Rep., 15.","DOI":"10.1038\/s41598-025-06094-7"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wang, L., Zhang, M., Jiao, M., Chen, E., Ma, Y., and Wang, J. (2025). Image Captioning Method Based on CLIP-Combined Local Feature Enhancement and Multi-Scale Semantic Guidance. Electronics, 14.","DOI":"10.3390\/electronics14142809"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_22","unstructured":"Banerjee, S., and Lavie, A. (2005, January 29). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, Ann Arbor, MI, USA."},{"key":"ref_23","unstructured":"Lin, C.-Y. (2004, January 25\u201326). ROUGE: A package for automatic evaluation of summaries. Proceedings of the ACL-04 Workshop on Text Summarization Branches Out, Barcelona, Spain."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Vedantam, R., Zitnick, C.L., and Parikh, D. (2015, January 7\u201312). CIDEr: Consensus-based image description evaluation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Anderson, P., Fernando, B., Johnson, M., and Gould, S. (2016, January 11\u201314). SPICE: Semantic propositional image caption evaluation. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46454-1_24"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014, January 6\u201312). Microsoft COCO: Common objects in context. Proceedings of the European Conference on Computer Vision (ECCV), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_27","unstructured":"Goodfellow, I.J., Bulatov, Y., Ibarz, J., Arnoud, S., and Shet, V. (2013). Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1016\/j.jmsy.2025.02.017","article-title":"MaViLa: Unlocking new potentials in smart manufacturing through vision language models","volume":"80","author":"Fan","year":"2025","journal-title":"J. Manuf. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/s42524-025-4136-9","article-title":"Vision-language model-based human-robot collaboration for smart manufacturing: A state-of-the-art survey","volume":"12","author":"Shi","year":"2025","journal-title":"Front. Eng. Manag."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Khan, A.S., Abbass, M.J., and Khan, A.H. (2025). Towards Fault-Aware Image Captioning: A Review on Integrating Facial Expression Recognition (FER) and Object Detection. Sensors, 25.","DOI":"10.3390\/s25195992"}],"container-title":["Bioengineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/12\/1325\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T12:57:24Z","timestamp":1764939444000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/12\/1325"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,5]]},"references-count":30,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["bioengineering12121325"],"URL":"https:\/\/doi.org\/10.3390\/bioengineering12121325","relation":{},"ISSN":["2306-5354"],"issn-type":[{"value":"2306-5354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,5]]}}}