{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T07:42:43Z","timestamp":1761896563229,"version":"build-2065373602"},"reference-count":27,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2017,5,24]],"date-time":"2017-05-24T00:00:00Z","timestamp":1495584000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Image-guided surgical procedures are challenged by mono image modality, two-dimensional anatomical guidance and non-intuitive human-machine interaction. The introduction of Tablet-based augmented reality (AR) into surgical robots may assist surgeons with overcoming these problems. In this paper, we proposed and developed a robot-assisted surgical system with interactive surgical guidance using tablet-based AR with a Kinect sensor for three-dimensional (3D) localization of patient anatomical structures and intraoperative 3D surgical tool navigation. Depth data acquired from the Kinect sensor was visualized in cone-shaped layers for 3D AR-assisted navigation. Virtual visual cues generated by the tablet were overlaid on the images of the surgical field for spatial reference. We evaluated the proposed system and the experimental results showed that the tablet-based visual guidance system could assist surgeons in locating internal organs, with errors between 1.74 and 2.96 mm. We also demonstrated that the system was able to provide mobile augmented guidance and interaction for surgical tool navigation.<\/jats:p>","DOI":"10.3390\/robotics6020013","type":"journal-article","created":{"date-parts":[[2017,5,24]],"date-time":"2017-05-24T12:13:11Z","timestamp":1495627991000},"page":"13","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery"],"prefix":"10.3390","volume":"6","author":[{"given":"Rong","family":"Wen","sequence":"first","affiliation":[{"name":"Singapore Institute of Manufacturing Technology, 138634, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chin-Boon","family":"Chng","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, National University of Singapore, 117575, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chee-Kong","family":"Chui","sequence":"additional","affiliation":[{"name":"Singapore Institute of Manufacturing Technology, 138634, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2017,5,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"730","DOI":"10.1067\/j.cpsurg.2009.04.001","article-title":"Image guided surgical interventions","volume":"46","author":"Perrin","year":"2009","journal-title":"Curr. Probl Surg."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1109\/THMS.2014.2339324","article-title":"The Validity and Reliability of a Hybrid Reality Simulator for Wire Navigation in Orthopedic Surgery","volume":"45","author":"Thomas","year":"2015","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.fjs.2011.12.003","article-title":"Non-invasive, fluoroscopy-based, image-guided surgery reduces radiation exposure for vertebral compression fractures: A preliminary survey","volume":"45","author":"Yang","year":"2012","journal-title":"Formos. J. Surg."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1016\/j.eururo.2009.11.025","article-title":"Treatment of Renal Tumors by Percutaneous Ultrasound-Guided Radiofrequency Ablation Using a Multitined Electrode: Effectiveness and Complications","volume":"57","author":"Cura","year":"2010","journal-title":"Eur. Urol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1007\/s11548-013-0897-4","article-title":"Projection-based visual guidance for robot-aided RF needle insertion","volume":"8","author":"Wen","year":"2013","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wen, R., Yang, L., Chui, C., Lim, K., and Chang, S. (2010, January 9\u201311). Intraoperative visual guidance and control interface for augmented reality robotic surgery. Proceedings of the 8th IEEE International Conference on Control and Automation 2010, Xiamen, China.","DOI":"10.1109\/ICCA.2010.5524421"},{"key":"ref_7","unstructured":"Birkfellner, W., Huber, K., Watzinger, F., Figl, M., Wanschitz, F., Hanel, R., Rafolt, D., Ewers, R., and Bergmann, H. (2000, January 6). Development of the Varioscope AR. A see-through HMD for computer-aided surgery. Proceedings of the IEEE and ACM International Symposium on Augmented Reality, Munich, Germany."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1415","DOI":"10.1109\/TBME.2005.851493","article-title":"Image overlay guidance for needle insertion in CT scanner","volume":"52","author":"Fichtinger","year":"2005","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_9","unstructured":"Mercier-Ganady, J., Lotte, F., Loup-Escande, E., Marchal, M., and Lecuyer, A. (April, January 29). The Mind-Mirror: See your brain in action in your head using EEG and augmented reality. Proceedings of the IEEE Virtual Reality Conference, Minneapolis, MN, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1016\/j.eururo.2011.12.024","article-title":"iPad-assisted percutaneous access to the kidney using marker-based navigation: Initial clinical experience","volume":"61","author":"Rassweiler","year":"2012","journal-title":"Eur. Urol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1007\/s11548-013-0828-4","article-title":"Mobile augmented reality for computer-assisted percutaneous nephrolithotomy","volume":"8","author":"Rassweiler","year":"2013","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_12","first-page":"1","article-title":"Image-guided procedures: A review","volume":"3","author":"Yaniv","year":"2006","journal-title":"Comput. Aided Interv. Med. Robot."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"071","DOI":"10.1055\/s-0032-1333415","article-title":"Preliminary study on the clinical application of augmented reality neuronavigation","volume":"74","author":"Inoue","year":"2013","journal-title":"J. Neurol. Surg. Part A Cent. Eur. Neurosurg."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Figl, M., Rueckert, D., and Edwards, P. (2009, January 7\u201312). Registration of a Cardiac Motion Model to Video for Augmented Reality Image Guidance of Coronary Artery Bypass. Proceedings of the World Congress on Medical Physics and Biomedical Engineering, Munich, Germany.","DOI":"10.1007\/978-3-642-03882-2_449"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1186\/1749-799X-5-50","article-title":"An interactive surgical planning tool for acetabular fractures: Initial results","volume":"5","author":"Fornaro","year":"2010","journal-title":"J. Orthop. Surg. Res."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1109\/TPAMI.2009.189","article-title":"An adaptive and stable method for fitting implicit polynomial curves and surfaces","volume":"32","author":"Zheng","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Rouhani, M., and Sappa, A.D. (2010, January 26\u201329). A fast accurate implicit polynomial fitting approach. Proceedings of the IEEE International Conference on Image Processing, Hong Kong, China.","DOI":"10.1109\/ICIP.2010.5654043"},{"key":"ref_18","unstructured":"Jolliffe, I.T. (2002). Principal component analysis and factor analysis. Princ. Compon. Anal., 150\u2013166."},{"key":"ref_19","unstructured":"Besl, P.J., and McKay, N.D. (1992, January 30). Method for registration of 3-D shapes in Robotics-DL tentative. Proceedings of the International Society for Optics and Photonics, Boston, MA, USA."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Agrawal, M., Konolige, K., and Blas, M.R. (2008, January 12\u201318). Censure: Center surround extremas for realtime feature detection and matching. Proceedings of the European Conference on Computer Vision, Marseille, France.","DOI":"10.1007\/978-3-540-88693-8_8"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.cviu.2007.09.014","article-title":"Speeded-up robust features (SURF)","volume":"110","author":"Bay","year":"2008","journal-title":"Comput. Vis. Imag. Underst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.cmpb.2013.12.018","article-title":"Hand gesture guided robot-assisted surgery based on a direct augmented reality interface","volume":"116","author":"Wen","year":"2014","journal-title":"Comput. Method. Progr. Biomed."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1089\/end.2010.0494","article-title":"The Locator: Novel percutaneous nephrolithotomy apparatus to aid collecting system puncture\u2014A preliminary report","volume":"25","author":"Lazarus","year":"2011","journal-title":"J. Endourol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1007\/s00345-012-0923-0","article-title":"Influence of surgeon\u2019s experience on fluoroscopy time during endourological interventions","volume":"31","author":"Ritter","year":"2013","journal-title":"World J. Urol."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Xiao, C., and Zhang, L. (2014, January 27\u201329). Implementation of mobile augmented reality based on Vuforia and Rawajali. Proceedings of the IEEE 5th International Conference on Software Engineering and Service Science, Beijing, China.","DOI":"10.1109\/ICSESS.2014.6933713"},{"key":"ref_26","first-page":"5575","article-title":"Future practicability of Android application development with new Android libraries and frameworks","volume":"5","author":"Mukherjee","year":"2014","journal-title":"Int. J. Comput. Sci. Inf. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"594","DOI":"10.1016\/j.media.2005.04.005","article-title":"The medical imaging interaction toolkit","volume":"9","author":"Wolf","year":"2005","journal-title":"Med. Imag. Anal."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/6\/2\/13\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:36:53Z","timestamp":1760207813000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/6\/2\/13"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,5,24]]},"references-count":27,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2017,6]]}},"alternative-id":["robotics6020013"],"URL":"https:\/\/doi.org\/10.3390\/robotics6020013","relation":{},"ISSN":["2218-6581"],"issn-type":[{"type":"electronic","value":"2218-6581"}],"subject":[],"published":{"date-parts":[[2017,5,24]]}}}