{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T05:23:01Z","timestamp":1782278581198,"version":"3.54.5"},"reference-count":34,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,3,19]],"date-time":"2022-03-19T00:00:00Z","timestamp":1647648000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61671190"],"award-info":[{"award-number":["61671190"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Traditional endoscopic treatment methods restrict the surgeon\u2019s field of view. New approaches to laparoscopic visualization have emerged due to the advent of robot-assisted surgical techniques. Lumen simultaneous localization and mapping (SLAM) technology can use the image sequence taken by the endoscope to estimate the pose of the endoscope and reconstruct the lumen scene in minimally invasive surgery. This technology gives the surgeon better visual perception and is the basis for the development of surgical navigation systems as well as medical augmented reality. However, the movement of surgical instruments in the internal cavity can interfere with the SLAM algorithm, and the feature points extracted from the surgical instruments may cause errors. Therefore, we propose a modified endocavity SLAM method combined with deep learning semantic segmentation that introduces a convolution neural network based on U-Net architecture with a symmetric encoder\u2013decoder structure in the visual odometry with the goals of solving the binary segmentation problem between surgical instruments and the lumen background and distinguishing dynamic feature points. Its segmentation performance is improved by using pretrained encoders on the network model to obtain more accurate pixel-level instrument segmentation. In this setting, the semantic segmentation is used to reject the feature points on the surgical instruments and reduce the impact caused by dynamic surgical instruments. This can provide more stable and accurate mapping results compared to ordinary SLAM systems.<\/jats:p>","DOI":"10.3390\/sym14030614","type":"journal-article","created":{"date-parts":[[2022,3,20]],"date-time":"2022-03-20T21:37:17Z","timestamp":1647812237000},"page":"614","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Semantic SLAM Based on Deep Learning in Endocavity Environment"],"prefix":"10.3390","volume":"14","author":[{"given":"Haibin","family":"Wu","sequence":"first","affiliation":[{"name":"Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianbo","family":"Zhao","sequence":"additional","affiliation":[{"name":"Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaiyang","family":"Xu","sequence":"additional","affiliation":[{"name":"Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yan","family":"Zhang","sequence":"additional","affiliation":[{"name":"Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruotong","family":"Xu","sequence":"additional","affiliation":[{"name":"Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9118-230X","authenticated-orcid":false,"given":"Aili","family":"Wang","sequence":"additional","affiliation":[{"name":"Heilongjiang Province Key Laboratory of Laser Spectroscopy Technology and Application, Harbin University of Science and Technology, Harbin 150080, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1016-1636","authenticated-orcid":false,"given":"Yuji","family":"Iwahori","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Chubu University, Aichi 487-8501, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1016\/j.suronc.2019.05.007","article-title":"Retrospective comparison of outcomes of laparoscopic and open surgery for t2 gallbladder cancer\u2014Thirteen-year experience","volume":"29","author":"Jang","year":"2019","journal-title":"Surg. Oncol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1007\/s11548-011-0631-z","article-title":"Enhanced visualisation for minimally invasive surgery","volume":"7","author":"Totz","year":"2012","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_3","unstructured":"Vemuri, A.S., Liu, K.C., Ho, Y., Wu, H.S., and Ku, M.C. (2012, January 1). Endoscopic Video Mosaicing: Application to Surgery and Diagnostics. Proceedings of the Living Imaging Workshop, Strasbourg, France."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Afifi, A., Takada, C., Yoshimura, Y., and Nakaguchi, T. (2021). Real-time expanded field-of-view for minimally invasive surgery using multi-camera visual simultaneous localization and mapping. Sensors, 21.","DOI":"10.3390\/s21062106"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Brandt, O., and Munwes, Y. (2018, January 1). Commissioning and First Image Reconstruction with a New Time-of-Flight PET Prototype. Proceedings of the 2018 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS\/MIC), Sydney, Australia.","DOI":"10.1109\/NSSMIC.2018.8824716"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Furukawa, R., Oka, S., Kotachi, T., Okamoto, Y., Tanaka, S., Sagawa, R., and Kawasaki, H. (2020, January 20\u201324). Fully Auto-calibrated Active-stereo-based 3D Endoscopic System using Correspondence Estimation with Graph Convolutional Network. Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Montreal, QC, Canada.","DOI":"10.1109\/EMBC44109.2020.9176417"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1109\/TNS.2016.2514600","article-title":"Asymmetric data acquisition system for an endoscopic pet-us detector","volume":"63","author":"Zorraquino","year":"2016","journal-title":"IEEE Trans. Nucl. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sun, D., Liu, J., Linte, C.A., Duan, H., and Robb, R.A. (2013). Surface Reconstruction from Tracked Endoscopic Video Using the Structure from Motion Approach. Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions, Springer.","DOI":"10.1007\/978-3-642-40843-4_14"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2536","DOI":"10.3788\/OPE.20182610.2536","article-title":"Hand-eye calibration method without a calibration reference based on second-order cone programming","volume":"26","author":"Wei","year":"2018","journal-title":"Opt. Precis. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Collins, T., and Bartoli, A. (2012, January 27). Towards Live Monocular 3D Laparoscopy Using Shading and Specularity Information. Proceedings of the International Conference on Information Processing in Computer-Assisted Interventions, Pisa, Italy.","DOI":"10.1007\/978-3-642-30618-1_2"},{"key":"ref_11","first-page":"6","article-title":"Survey on Monocular Visual Inertial SLAM Algorithms","volume":"19","author":"Qi","year":"2020","journal-title":"Softw. Guide"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Mountney, P., and Yang, G.Z. (2009, January 3\u20136). Dynamic view expansion for minimally invasive surgery using simultaneous localization and mapping. Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA.","DOI":"10.1109\/IEMBS.2009.5333939"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lin, B., Johnson, A., Qian, X., Sanchez, J., and Sun, Y. (2013). Simultaneous tracking, 3D reconstruction and deforming point detection for stereoscope guided surgery. Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions, Springer.","DOI":"10.1007\/978-3-642-40843-4_5"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1314","DOI":"10.1016\/j.jiph.2019.06.028","article-title":"Endoscope localization and gastrointestinal feature map construction based on monocular SLAM technology","volume":"13","author":"Xie","year":"2020","journal-title":"J. Infect. Public Health"},{"key":"ref_15","unstructured":"Peng, X. (2017). Research on Endoscopic Visual SLAM for Minimally Invasive Surgery. [Master\u2019s Thesis, University of Electronic Science and Technology of China]."},{"key":"ref_16","unstructured":"Wang, C., and Thorpe, C. (2002, January 11\u201315). Simultaneous localization and mapping with detection and tracking of moving objects. Proceedings of the 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), Washington, DC, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kundu, A., Krishna, K.M., and Sivaswamy, J. (2009, January 10\u201315). Moving object detection by multi-view geometric techniques from a single camera mounted robot. Proceedings of the 2009 IEEE\/RSJ International Conference on Intelligent Robots and Systems, St. Louis, MO, USA.","DOI":"10.1109\/IROS.2009.5354227"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Wang, Y., and Huang, S. (2014, January 10\u201312). Towards dense moving object segmentation based robust dense RGB-D SLAM in dynamic scenarios. Proceedings of the 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV), Singapore.","DOI":"10.1109\/ICARCV.2014.7064596"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Yu, C., Liu, Z., Liu, X.-J., Xie, F., Yang, Y., Wei, Q., and Fei, Q. (2018, January 1\u20135). DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments. Proceedings of the 2018 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.","DOI":"10.1109\/IROS.2018.8593691"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4076","DOI":"10.1109\/LRA.2018.2860039","article-title":"DynaSLAM: Tracking, Mapping, and Inpainting in Dynamic Scenes","volume":"3","author":"Bescos","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Laina, I., Rieke, N., Rupprecht, C., Vizca\u00edno, J.P., Eslami, A., Tombari, F., and Navab, N. (2017). Concurrent Segmentation and Localization for Tracking of Surgical Instruments. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-319-66185-8_75"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Pakhomov, D., Premachandran, V., Allan, M., Azizian, M., and Navab, N. (2019). Deep residual learning for instrument segmentation in robotic surgery. International Workshop on Machine Learning in Medical Imaging, Springer.","DOI":"10.1007\/978-3-030-32692-0_65"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1007\/s11517-019-02071-1","article-title":"Nucleus and cytoplasm-based segmentation and actor-critic neural network for acute lymphocytic leukaemia detection in single cell blood smear images","volume":"58","author":"Jha","year":"2020","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"38493","DOI":"10.1109\/ACCESS.2020.2975745","article-title":"Chaining a U-net with a residual U-net for retinal blood vessels segmentation","volume":"8","author":"Francia","year":"2020","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ding, H., Cen, Q., Si, X., Pan, Z., and Chen, X. (2022). Automatic glottis segmentation for laryngeal endoscopic images based on U-Net. Biomed. Signal Process. Control, accepted.","DOI":"10.1016\/j.bspc.2021.103116"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Siddique, N., Sidike, P., Elkin, C., and Devabhaktuni, V. (2020). U-Net and its variants for medical image segmentation: Theory and applications. arXiv.","DOI":"10.1109\/ACCESS.2021.3086020"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.1109\/TRO.2017.2705103","article-title":"Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras","volume":"33","year":"2017","journal-title":"IEEE Trans. Robot."},{"key":"ref_29","unstructured":"Iglovikov, V., and Shvets, A. (2018). Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Shvets, A., Rakhlin, A., Kalinin, A.A., and Iglovikov, V. (2018, January 17). Automatic instrument segmentation in robot-assisted surgery using deep learning. Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA.","DOI":"10.1109\/ICMLA.2018.00100"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Mahmoud, N., Cirauqui, I., Hostettler, A., Doignon, C., Soler, L., Marescaux, J., and Montiel, J.M. (2016, January 17). ORBSLAM-based endoscope tracking and 3D reconstruction. Proceedings of the International Workshop on Computer-Assisted and Robotic Endoscopy, Athens, Greece.","DOI":"10.1007\/978-3-319-54057-3_7"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/MSP.2010.936728","article-title":"Three-dimensional tissue deformation recovery and tracking","volume":"27","author":"Mountney","year":"2010","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Piccinelli, N., Roberti, A., Tagliabue, E., Setti, F., Kronreif, G., Muradore, R., and Fiorini, P. (2020, January 18\u201320). Rigid 3D registration of pre-operative information for semi-autonomous surgery. Proceedings of the 2020 International Symposium on Medical Robotics (ISMR), Atlanta, GA, USA.","DOI":"10.1109\/ISMR48331.2020.9312949"},{"key":"ref_34","unstructured":"Allan, M., Shvets, A., Kurmann, T., Zhang, Z., Duggal, R., Su, Y.H., Rieke, N., Laina, I., Kalavakonda, N., and Bodenstedt, S. (2019). 2017 robotic instrument segmentation challenge. arXiv."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/3\/614\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:39:26Z","timestamp":1760135966000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/14\/3\/614"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,19]]},"references-count":34,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["sym14030614"],"URL":"https:\/\/doi.org\/10.3390\/sym14030614","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,19]]}}}