{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T10:46:02Z","timestamp":1772275562846,"version":"3.50.1"},"reference-count":27,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,2,12]],"date-time":"2023-02-12T00:00:00Z","timestamp":1676160000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Major Program of National Natural Science Foundation of China","award":["T2293751"],"award-info":[{"award-number":["T2293751"]}]},{"name":"Major Program of National Natural Science Foundation of China","award":["2021YFC2400103"],"award-info":[{"award-number":["2021YFC2400103"]}]},{"name":"Major Program of National Natural Science Foundation of China","award":["LGF20F050006"],"award-info":[{"award-number":["LGF20F050006"]}]},{"name":"Major Program of National Natural Science Foundation of China","award":["2019MC0AD02"],"award-info":[{"award-number":["2019MC0AD02"]}]},{"name":"Major Program of National Natural Science Foundation of China","award":["2022MG0AL01"],"award-info":[{"award-number":["2022MG0AL01"]}]},{"name":"National Key Research and Development Program of China","award":["T2293751"],"award-info":[{"award-number":["T2293751"]}]},{"name":"National Key Research and Development Program of China","award":["2021YFC2400103"],"award-info":[{"award-number":["2021YFC2400103"]}]},{"name":"National Key Research and Development Program of China","award":["LGF20F050006"],"award-info":[{"award-number":["LGF20F050006"]}]},{"name":"National Key Research and Development Program of China","award":["2019MC0AD02"],"award-info":[{"award-number":["2019MC0AD02"]}]},{"name":"National Key Research and Development Program of China","award":["2022MG0AL01"],"award-info":[{"award-number":["2022MG0AL01"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["T2293751"],"award-info":[{"award-number":["T2293751"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["2021YFC2400103"],"award-info":[{"award-number":["2021YFC2400103"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LGF20F050006"],"award-info":[{"award-number":["LGF20F050006"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["2019MC0AD02"],"award-info":[{"award-number":["2019MC0AD02"]}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["2022MG0AL01"],"award-info":[{"award-number":["2022MG0AL01"]}]},{"name":"Key Research Project of Zhejiang Lab","award":["T2293751"],"award-info":[{"award-number":["T2293751"]}]},{"name":"Key Research Project of Zhejiang Lab","award":["2021YFC2400103"],"award-info":[{"award-number":["2021YFC2400103"]}]},{"name":"Key Research Project of Zhejiang Lab","award":["LGF20F050006"],"award-info":[{"award-number":["LGF20F050006"]}]},{"name":"Key Research Project of Zhejiang Lab","award":["2019MC0AD02"],"award-info":[{"award-number":["2019MC0AD02"]}]},{"name":"Key Research Project of Zhejiang Lab","award":["2022MG0AL01"],"award-info":[{"award-number":["2022MG0AL01"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Binocular endoscopy is gradually becoming the future of minimally invasive surgery (MIS) thanks to the development of stereo vision. However, some problems still exist, such as the low reconstruction accuracy, small surgical field, and low computational efficiency. To solve these problems, we designed a framework for real-time dense reconstruction in binocular endoscopy scenes. First, we obtained the initial disparity map using an SGBM algorithm and proposed the disparity confidence map as a dataset to provide StereoNet training. Then, based on the depth map predicted by StereoNet, the corresponding left image of each depth map was input into the Oriented Fast and Brief-Simultaneous Localization and Mapping (ORB-SLAM) framework using an RGB-D camera to realize the real-time dense reconstruction of the binocular endoscopy scene. The proposed algorithm was verified in the stomach phantom and a real pig stomach. Compared with the ground truth, the proposed algorithm\u2019s RMSE is 1.620 mm, and the number of effective points in the point cloud is 834,650, which is a significant improvement in the mapping ability compared with binocular SLAM and ensures the real-time performance of the algorithm while performing dense reconstruction. The effectiveness of the proposed algorithm is verified.<\/jats:p>","DOI":"10.3390\/s23042074","type":"journal-article","created":{"date-parts":[[2023,2,13]],"date-time":"2023-02-13T02:14:11Z","timestamp":1676254451000},"page":"2074","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Real-Time Dense Reconstruction with Binocular Endoscopy Based on StereoNet and ORB-SLAM"],"prefix":"10.3390","volume":"23","author":[{"given":"Jiayi","family":"Huo","sequence":"first","affiliation":[{"name":"State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Changjiang","family":"Zhou","sequence":"additional","affiliation":[{"name":"Research Center for Humanoid Sensing, Zhejiang Lab, Hangzhou 311100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bo","family":"Yuan","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Engineering, Zhejiang University, Hangzhou 310027, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qing","family":"Yang","sequence":"additional","affiliation":[{"name":"Research Center for Humanoid Sensing, Zhejiang Lab, Hangzhou 311100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liqiang","family":"Wang","sequence":"additional","affiliation":[{"name":"Research Center for Humanoid Sensing, Zhejiang Lab, Hangzhou 311100, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Mountney, P., and Yang, G.-Z. (2009, January 3\u20136). Dynamic view expansion for minimally invasive surgery using simultaneous localization and mapping. Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA.","DOI":"10.1109\/IEMBS.2009.5333939"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/JTEHM.2019.2946802","article-title":"Whole stomach 3D reconstruction and frame localization from monocular endoscope video","volume":"7","author":"Widya","year":"2019","journal-title":"IEEE J. Transl. Eng. Health Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.neucom.2020.12.089","article-title":"Deep learning for monocular depth estimation: A review","volume":"438","author":"Ming","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.neucom.2019.04.051","article-title":"Single image super-resolution incorporating example-based gradient profile estimation and weighted adaptive p-norm","volume":"355","author":"Li","year":"2019","journal-title":"Neurocomputing"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/j.neucom.2019.09.035","article-title":"Effective image super resolution via hierarchical convolutional neural network","volume":"374","author":"Liu","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Atapour-Abarghouei, A., and Breckon, T.P. (2019, January 15\u201320). Veritatem dies aperit-temporally consistent depth prediction enabled by a multi-task geometric and semantic scene understanding approach. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00349"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Spencer, J., Bowden, R., and Hadfield, S. (2020, January 13\u201319). Defeat-net: General monocular depth via simultaneous unsupervised representation learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01441"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Spencer, J., Bowden, R., and Hadfield, S. (2019, January 15\u201320). Scale-adaptive neural dense features: Learning via hierarchical context aggregation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00636"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"093901","DOI":"10.1103\/PhysRevLett.123.093901","article-title":"Revealing the Transition Dynamics from Q Switching to Mode Locking in a Soliton Laser","volume":"123","author":"Liu","year":"2019","journal-title":"Phys. Rev. Lett."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"023905","DOI":"10.1103\/PhysRevLett.121.023905","article-title":"Real-Time Observation of the Buildup of Soliton Molecules","volume":"121","author":"Liu","year":"2018","journal-title":"Phys. Rev. Lett."},{"key":"ref_11","unstructured":"Hirschmuller, H. (2005, January 20\u201325). Accurate and efficient stereo processing by semi-global matching and mutual information. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1651","DOI":"10.1109\/TMI.2022.3147414","article-title":"A Robust Edge-Preserving Stereo Matching Method for Laparoscopic Images","volume":"41","author":"Xia","year":"2022","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1080\/21681163.2020.1835561","article-title":"HAPNet: Hierarchically aggregated pyramid network for real-time stereo matching","volume":"9","author":"Brandao","year":"2021","journal-title":"Comput. Methods Biomech. Biomed. Eng. Imaging Vis."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ye, M., Johns, E., Handa, A., Zhang, L., Pratt, P., and Yang, G.-Z. (2017). Self-supervised siamese learning on stereo image pairs for depth estimation in robotic surgery. arXiv.","DOI":"10.31256\/HSMR2017.14"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"73241","DOI":"10.1109\/ACCESS.2020.2987767","article-title":"Deep convolutional network for stereo depth mapping in binocular endoscopy","volume":"8","author":"Wang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"994343","DOI":"10.3389\/fphys.2022.994343","article-title":"A 3D reconstruction based on an unsupervised domain adaptive for binocular endoscopy","volume":"13","author":"Zhang","year":"2022","journal-title":"Front. Physiol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1109\/TRO.2015.2463671","article-title":"ORB-SLAM: A versatile and accurate monocular SLAM system","volume":"31","author":"Montiel","year":"2015","journal-title":"IEEE Trans. Robot."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.1109\/TRO.2017.2705103","article-title":"ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras","volume":"33","author":"Tardos","year":"2017","journal-title":"IEEE Trans. Robot."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1619","DOI":"10.1007\/s11548-020-02241-9","article-title":"A visual SLAM-based bronchoscope tracking scheme for bronchoscopic navigation","volume":"15","author":"Wang","year":"2020","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Mahmoud, N., Cirauqui, I., Hostettler, A., Doignon, C., Soler, L., Marescaux, J., and Montiel, J.M.M. (2017, January 17). ORBSLAM-based endoscope tracking and 3D reconstruction. Proceedings of the International Workshop on Computer-Assisted and Robotic Endoscopy, Athens, Greece.","DOI":"10.1007\/978-3-319-54057-3_7"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1007\/s11633-017-1090-y","article-title":"A brain-inspired SLAM system based on ORB features","volume":"14","author":"Zhou","year":"2017","journal-title":"Int. J. Autom. Comput."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Whelan, T., Leutenegger, S., Salas-Moreno, R., Glocker, B., and Davison, A. (2015, January 13\u201317). ElasticFusion: Dense SLAM without a Pose Graph. Proceedings of the Robotics: Science and Systems 2015, Rome, Italy.","DOI":"10.15607\/RSS.2015.XI.001"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Docea, R., Pfeiffer, M., Bodenstedt, S., Kolbinger, F.R., H\u00f6ller, L., Wittig, I., Hoffmann, R.-T., Troost, E.G., Riediger, C., and Weitz, J. (2021, January 15\u201320). Simultaneous localisation and mapping for laparoscopic liver navigation: A comparative evaluation study. Proceedings of the Medical Imaging 2021: Image-Guided Procedures, Robotic Interventions, and Modeling, Online.","DOI":"10.1117\/12.2582121"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Khamis, S., Fanello, S., Rhemann, C., Kowdle, A., Valentin, J., and Izadi, S. (2018, January 8\u201314). StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01267-0_35"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1109\/LRA.2017.2735487","article-title":"Dynamic reconstruction of deformable soft-tissue with stereo scope in minimal invasive surgery","volume":"3","author":"Song","year":"2017","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"4068","DOI":"10.1109\/LRA.2018.2856519","article-title":"Mis-slam: Real-time large-scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing","volume":"3","author":"Song","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"107546","DOI":"10.1016\/j.optlastec.2021.107546","article-title":"Porous nickel oxide micron polyhedral particles for high-performance ultrafast photonics","volume":"146","author":"Zhang","year":"2022","journal-title":"Opt. Laser Technol."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/2074\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:32:26Z","timestamp":1760121146000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/4\/2074"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,12]]},"references-count":27,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["s23042074"],"URL":"https:\/\/doi.org\/10.3390\/s23042074","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,12]]}}}