{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T18:18:42Z","timestamp":1781806722014,"version":"3.54.5"},"reference-count":34,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T00:00:00Z","timestamp":1699920000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["No. 2020AAA0108904"],"award-info":[{"award-number":["No. 2020AAA0108904"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100013093","name":"Science and Technology Planning Project of Shenzhen Municipality","doi-asserted-by":"publisher","award":["No.JCYJ20200109140410340"],"award-info":[{"award-number":["No.JCYJ20200109140410340"]}],"id":[{"id":"10.13039\/501100013093","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100017607","name":"Shenzhen Fundamental Research Program","doi-asserted-by":"publisher","award":["No.GXWD20201231165807007-20200807164903001"],"award-info":[{"award-number":["No.GXWD20201231165807007-20200807164903001"]}],"id":[{"id":"10.13039\/501100017607","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>The field of human-computer interaction is expanding, especially within the domain of intelligent technologies. Scene understanding, which entails the generation of advanced semantic descriptions from scene content, is crucial for effective interaction. Despite its importance, it remains a significant challenge. This study introduces RGBD2Cap, an innovative method that uses RGBD images for scene semantic description. We utilize a multimodal fusion module to integrate RGB and Depth information for extracting multi-level features. And the method also incorporates target detection and region proposal network and a top-down attention LSTM network to generate semantic descriptions. The experimental data are derived from the ScanRefer indoor scene dataset, with RGB and depth images rendered from ScanNet's 3D scene serving as the model's input. The method outperforms the DenseCap network in several metrics, including BLEU, CIDEr, and METEOR. Ablation studies have confirmed the essential role of the RGBD fusion module in the method's success. Furthermore, the practical applicability of our method was verified within the AI2-THOR embodied intelligence experimental environment, showcasing its reliability.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1280501","type":"journal-article","created":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T11:56:54Z","timestamp":1699963014000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Dense captioning and multidimensional evaluations for indoor robotic scenes"],"prefix":"10.3389","volume":"17","author":[{"given":"Hua","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenshuai","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenhao","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hong","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,11,14]]},"reference":[{"key":"B1","first-page":"6077","article-title":"\u201cBottom-up and top-down attention for image captioning and visual question answering,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Anderson","year":"2018"},{"key":"B2","first-page":"65","article-title":"\u201cMETEOR: an automatic metric for MT evaluation with improved correlation with human judgments,\u201d","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee","year":"2005"},{"key":"B3","first-page":"16464","article-title":"\u201c3DJCG: a unified framework for joint dense captioning and visual grounding on 3D point clouds,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Cai","year":"2022"},{"key":"B4","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1007\/978-3-030-58565-5_13","article-title":"\u201cScanRefer: 3D object localization in RGB-D scans using natural language,\u201d","volume-title":"Computer Vision\u2013ECCV 2020: 16th European Conference","author":"Chen","year":"2020"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2212.00836","article-title":"Unit3D: a unified transformer for 3d dense captioning and visual grounding","author":"Chen","year":"2022","journal-title":"arXiv preprint arXiv:2212.00836"},{"key":"B6","first-page":"3193","article-title":"\u201cScan2CAP: context-aware dense captioning in RGB-D scans,\u201d","author":"Chen","year":"2021","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.3115\/V1\/D14-1179","article-title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","author":"Cho","year":"2014","journal-title":"arXiv preprint arXiv:1406.1078"},{"key":"B8","first-page":"5828","article-title":"\u201cScanNet: richly-annotated 3D reconstructions of indoor scenes,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Dai","year":"2017"},{"key":"B9","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.00323","article-title":"\u201cRoboTHOR: an open simulation-to-real embodied AI platform,\u201d","volume-title":"CVPR","author":"Deitke","year":"2020"},{"key":"B10","first-page":"770","article-title":"\u201cDeep residual learning for image recognition,\u201d","volume-title":"2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"He","year":"2015"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1109\/TRO.2019.2959161","article-title":"An online initialization and self-calibration method for stereo visual-inertial odometry","volume":"36","author":"Huang","year":"2020","journal-title":"IEEE Trans. Robot."},{"key":"B12","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1007\/978-3-031-19833-5_31","article-title":"\u201cMORE: multi-order relation mining for dense captioning in 3D scenes,\u201d","volume-title":"Computer Vision\u2013ECCV 2022: 17th European Conference","author":"Jiao","year":"2022"},{"key":"B13","first-page":"4565","article-title":"\u201cDenseCap: fully convolutional localization networks for dense captioning,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Johnson","year":"2016"},{"key":"B14","doi-asserted-by":"publisher","first-page":"1335","DOI":"10.1109\/TPAMI.2006.153","article-title":"A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses","volume":"28","author":"Kannala","year":"2006","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"B15","first-page":"6271","article-title":"\u201cDense relational captioning: triple-stream networks for relationship-based captioning,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Kim","year":"2019"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1712.05474","article-title":"AI2-THOR: an interactive 3d environment for visual AI","author":"Kolve","year":"2017","journal-title":"arXiv preprint arXiv:1712.05474"},{"key":"B17","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"B18","first-page":"13147","article-title":"\u201cMHFormer: multi-hypothesis transformer for 3D human pose estimation,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Li","year":"2022"},{"key":"B19","first-page":"74","article-title":"\u201cROUGE: a package for automatic evaluation of summaries,\u201d","author":"Lin","year":"2004","journal-title":"Proceedings of the Workshop on Text Summarization Branches Out."},{"key":"B20","doi-asserted-by":"crossref","first-page":"9470","DOI":"10.1109\/ICRA46639.2022.9811641","article-title":"\u201cIntegrating point and line features for visual-inertial initialization,\u201d","volume-title":"2022 International Conference on Robotics and Automation (ICRA)","author":"Liu","year":"2022"},{"key":"B21","article-title":"Deep captioning with multimodal recurrent neural networks (m-RNN)","author":"Mao","year":"2014","journal-title":"arXiv preprint arXiv:1412.6632"},{"key":"B22","first-page":"311","article-title":"\u201cBleu: a method for automatic evaluation of machine translation,\u201d","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"B23","article-title":"\u201cPointNet++: deep hierarchical feature learning on point sets in a metric space,\u201d","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017","author":"Qi","year":"2017"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2007.08501","article-title":"Accelerating 3d deep learning with pytorch3d","author":"Ravi","year":"2020","journal-title":"arXiv preprint arXiv:2007.08501"},{"key":"B25","doi-asserted-by":"publisher","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster r-CNN: towards real-time object detection with region proposal networks","volume":"39","author":"Ren","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1177\/1729881419839596","article-title":"A review of mobile robots: concepts, methods, theoretical framework, and applications","author":"Rubio","year":"2019","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2022.3152990","article-title":"Region-object relation-aware dense captioning via transformer","author":"Shao","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst"},{"key":"B28","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1177\/0018720816644364","article-title":"Human\u2013robot interaction: status and challenges","volume":"58","author":"Sheridan","year":"2016","journal-title":"Hum. Fact."},{"key":"B29","first-page":"4566","article-title":"\u201cCIDER: consensus-based image description evaluation,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Vedantam","year":"2015"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.24963\/IJCAI.2022\/194","article-title":"Spatiality-guided transformer for 3d dense captioning on point clouds","author":"Wang","year":"2022","journal-title":"arXiv preprint arXiv:2204.10688"},{"key":"B31","first-page":"2048","article-title":"\u201cShow, attend and tell: neural image caption generation with visual attention,\u201d","volume-title":"International Conference on Machine Learning","author":"Xu","year":"2015"},{"key":"B32","first-page":"2193","article-title":"\u201cDense captioning with joint inference and visual context,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Yang","year":"2017"},{"key":"B33","first-page":"8563","article-title":"\u201cX-Trans2Cap: cross-modal knowledge transfer using transformer for 3d dense captioning,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Yuan","year":"2022"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2210.03925","article-title":"Contextual modeling for 3d dense captioning on point clouds","author":"Zhong","year":"2022","journal-title":"arXiv preprint arXiv:2210.03925"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1280501\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T11:57:12Z","timestamp":1699963032000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1280501\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,14]]},"references-count":34,"alternative-id":["10.3389\/fnbot.2023.1280501"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1280501","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,14]]},"article-number":"1280501"}}