{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T20:08:33Z","timestamp":1783973313115,"version":"3.55.0"},"reference-count":44,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T00:00:00Z","timestamp":1687219200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>In the field of human\u2013computer interaction, accurate identification of talking objects can help robots to accomplish subsequent tasks such as decision-making or recommendation; therefore, object determination is of great interest as a pre-requisite task. Whether it is named entity recognition (NER) in natural language processing (NLP) work or object detection (OD) task in the computer vision (CV) field, the essence is to achieve object recognition. Currently, multimodal approaches are widely used in basic image recognition and natural language processing tasks. This multimodal architecture can perform entity recognition tasks more accurately, but when faced with short texts and images containing more noise, we find that there is still room for optimization in the image-text-based multimodal named entity recognition (MNER) architecture. In this study, we propose a new multi-level multimodal named entity recognition architecture, which is a network capable of extracting useful visual information for boosting semantic understanding and subsequently improving entity identification efficacy. Specifically, we first performed image and text encoding separately and then built a symmetric neural network architecture based on Transformer for multimodal feature fusion. We utilized a gating mechanism to filter visual information that is significantly related to the textual content, in order to enhance text understanding and achieve semantic disambiguation. Furthermore, we incorporated character-level vector encoding to reduce text noise. Finally, we employed Conditional Random Fields for label classification task. Experiments on the Twitter dataset show that our model works to increase the accuracy of the MNER task.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1181143","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T10:17:32Z","timestamp":1687256252000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["MLNet: a multi-level multimodal named entity recognition architecture"],"prefix":"10.3389","volume":"17","author":[{"given":"Hanming","family":"Zhai","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaojun","family":"Lv","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiwen","family":"Hou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xin","family":"Tong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fanliang","family":"Bu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,6,20]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"3308","DOI":"10.1109\/LRA.2018.2852786","article-title":"Interactive text2pickup networks for natural language-based human\u2013robot collaboration","volume":"3","author":"Ahn","year":"2018","journal-title":"IEEE Robot. Automat. Lett."},{"key":"B2","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1109\/ICDAR.2019.00061","article-title":"\u201cAiding intra-text1 representations with visual context for multimodal named entity recognition,\u201d","volume-title":"2019 International Conference on Document Analysis and Recognition (ICDAR)","author":"Arshad","year":"2019"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-021-06488-4","article-title":"A multimodal deep learning approach for named entity recognition from social media","author":"Asgari-Chenaghlu","year":"2020","journal-title":"arXiv [Preprint]. arXiv:2001.06888"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1506.08700","article-title":"Dropout as data augmentation","author":"Bouthillier","year":"2015","journal-title":"arXiv [Preprint]. arXiv:1506.08700"},{"key":"B5","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1007\/978-3-030-73197-7_12","article-title":"\u201cMultimodal named entity recognition with image attributes and image knowledge,\u201d","volume-title":"Database Systems for Advanced Applications: 26th International Conference, DASFAA 2021","author":"Chen","year":"2021"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.wnut-1.11","article-title":"Can images help recognize entities? a study of the role of images for multimodal NER","author":"Chen","year":"2020","journal-title":"arXiv [Preprint]. arXiv:2010.12712"},{"key":"B7","article-title":"Lightner: a lightweight tuning paradigm for low-resource ner via pluggable prompting","author":"Chen","year":"2021","journal-title":"arXiv [Preprint]. arXiv:2109.00720"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-naacl.121","article-title":"Good visual guidance makes a better extractor: hierarchical visual prefix for multimodal entity and relation extraction","author":"Chen","year":"2022","journal-title":"arXiv [Preprint]. arXiv:2205.03521"},{"key":"B9","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1007\/978-3-030-58577-8_7","article-title":"\u201cUNITER: universal image-text representation learning,\u201d","volume-title":"Computer Vision\u2013ECCV 2020: 16th European Conference","author":"Chen","year":"2020"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04805","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018","journal-title":"arXiv [Preprint]. arXiv:1810.04805"},{"key":"B11","first-page":"6616","article-title":"\u201cLarge-scale adversarial training for vision-and-language representation learning,\u201d","author":"Gan","year":"2020","journal-title":"Advances in Neural Information Processing Systems 33"},{"key":"B12","first-page":"2251","article-title":"\u201cFashionBERT: text and image matching with adaptive loss for cross-modal retrieval,\u201d","author":"Gao","year":"2020","journal-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval"},{"key":"B13","doi-asserted-by":"crossref","first-page":"172","DOI":"10.3115\/1119176.1119202","article-title":"\u201cNamed entity recognition with long short-term memory,\u201d","volume-title":"Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003","author":"Hammerton","year":"2003"},{"key":"B14","first-page":"770","article-title":"\u201cDeep residual learning for image recognition,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"B15","first-page":"1643","article-title":"\u201cVLN BERT: a recurrent vision-and-language bert for navigation,\u201d","volume-title":"Proceedings of the IEEE\/CVF conference on Computer Vision and Pattern Recognition","author":"Hong","year":"2021"},{"key":"B16","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P19-1356","article-title":"\u201cWhat does bert learn about the structure of language?\u201d","volume-title":"ACL 2019-57th Annual Meeting of the Association for Computational Linguistics","author":"Jawahar","year":"2019"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.140","article-title":"Named entity recognition with small strongly labeled and large weakly labeled data","author":"Jiang","year":"2021","journal-title":"arXiv [Preprint]. arXiv:2106.08977"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1603.01360","article-title":"Neural architectures for named entity recognition","author":"Lample","year":"2016","journal-title":"arXiv [Preprint]. arXiv:1603.01360"},{"key":"B19","first-page":"11336","article-title":"\u201cUnicoder-VL: a universal encoder for vision and language by cross-modal pre-training,\u201d","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Li","year":"2020"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.352","article-title":"Weakly supervised named entity tagging with learnable logical rules","author":"Li","year":"2021","journal-title":"arXiv [Preprint]. arXiv:2107.02282"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1908.03557","article-title":"VisualBERT: a simple and performant baseline for vision and language","author":"Li","year":"2019","journal-title":"arXiv [Preprint]. arXiv:1908.03557"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.00794","article-title":"Scaling language-image pre-training via masking","author":"Li","year":"2022","journal-title":"arXiv [Preprint]. arXiv:2212.00794"},{"key":"B23","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1007\/978-3-319-10602-1_48","article-title":"\u201cMicrosoft COCO: common objects in context,\u201d","volume-title":"Computer Vision\u2013ECCV 2014: 13th European Conference","author":"Lin","year":"2014"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1233","article-title":"GCDT: a global context enhanced deep transition architecture for sequence labeling","author":"Liu","year":"2019","journal-title":"arXiv [Preprint]. arXiv:1906.02437"},{"key":"B25","first-page":"1990","article-title":"\u201cVisual attention model for name tagging in multimodal social media,\u201d","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Lu","year":"2018"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1381","DOI":"10.1093\/bioinformatics\/btx761","article-title":"An attention-based biLSTM-CRF approach to document-level chemical named entity recognition","volume":"34","author":"Luo","year":"2018","journal-title":"Bioinformatics"},{"key":"B27","first-page":"8441","article-title":"\u201cHierarchical contextualized representation for named entity recognition,\u201d","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Luo","year":"2020"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1078","article-title":"Multimodal named entity recognition for short social media posts","author":"Moon","year":"2018","journal-title":"arXiv [Preprint]. arXiv:1802.07862"},{"key":"B29","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1007\/978-3-030-58523-5_20","article-title":"\u201cLarge-scale pretraining for visual dialog: A simple state-of-the-art baseline,\u201d","volume-title":"Computer Vision\u2013ECCV 2020: 16th European Conference","author":"Murahari","year":"2020"},{"key":"B30","doi-asserted-by":"crossref","first-page":"6964","DOI":"10.1109\/ICRA.2019.8794394","article-title":"\u201cEfficient generation of motion plans from attribute-based natural language instructions using dynamic constraint mapping,\u201d","volume-title":"2019 International Conference on Robotics and Automation (ICRA)","author":"Park","year":"2019"},{"key":"B31","first-page":"82","article-title":"\u201cRecurrent convolutional neural networks for scene labeling,\u201d","volume-title":"International Conference on Machine Learning","author":"Pinheiro","year":"2014"},{"key":"B32","first-page":"4310","article-title":"\u201cSubsequence based deep active learning for named entity recognition,\u201d","author":"Radmard","year":"2021","journal-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1409.1556","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2014","journal-title":"arXiv [Preprint]. arXiv:1409.1556"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.10649","article-title":"Portuguese named entity recognition using bert-crf","author":"Souza","year":"2019","journal-title":"arXiv [Preprint]. arXiv:1909.10649"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1908.08530","article-title":"VL-BERT: pre-training of generic visual-linguistic representations","author":"Su","year":"2019","journal-title":"arXiv [Preprint]. arXiv:1908.08530"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1514","article-title":"LXMERT: learning cross-modality encoder representations from transformers","author":"Tan","year":"2019","journal-title":"arXiv [Preprint]. arXiv:1908.07490"},{"key":"B37","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1016\/j.neucom.2021.01.060","article-title":"Hierarchical self-adaptation network for multimodal named entity recognition in social media","volume":"439","author":"Tian","year":"2021","journal-title":"Neurocomputing"},{"key":"B38","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1007\/978-3-030-35699-6_26","article-title":"\u201cNeural semantic parsing with anonymization for command understanding in general-purpose service robots,\u201d","volume-title":"RoboCup 2019: Robot World Cup XXIII","author":"Walker","year":"2019"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.17103","article-title":"k nn-ner: named entity recognition with nearest neighbor search. arXiv [Preprint]. arXiv:2203.17103","author":"Wang","year":"2022"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2105.03654","article-title":"Improving named entity recognition by external context retrieving and cooperative learning","author":"Wang","year":"2021","journal-title":"arXiv [Preprint]. arXiv:2105.03654"},{"key":"B41","doi-asserted-by":"crossref","first-page":"1038","DOI":"10.1145\/3394171.3413650","article-title":"\u201cMultimodal representation with embedded visual guiding objects for named entity recognition in social media posts,\u201d","volume-title":"Proceedings of the 28th ACM International Conference on Multimedia","author":"Wu","year":"2020"},{"key":"B42","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.acl-main.306","volume-title":"Improving Multimodal Named Entity Recognition via Entity Span Detection With Unified Multimodal Transformer","author":"Yu","year":"2020"},{"key":"B43","doi-asserted-by":"crossref","first-page":"14347","DOI":"10.1609\/aaai.v35i16.17687","article-title":"\u201cMulti-modal graph fusion for named entity recognition with targeted visual guidance,\u201d","author":"Zhang","year":"2021","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"B44","doi-asserted-by":"crossref","DOI":"10.1609\/aaai.v32i1.11962","article-title":"\u201cAdaptive co-attention network for named entity recognition in tweets,\u201d","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Zhang","year":"2018"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1181143\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T10:18:02Z","timestamp":1687256282000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1181143\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,20]]},"references-count":44,"alternative-id":["10.3389\/fnbot.2023.1181143"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1181143","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,20]]},"article-number":"1181143"}}