{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,23]],"date-time":"2026-02-23T16:31:22Z","timestamp":1771864282365,"version":"3.50.1"},"reference-count":20,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,10,6]],"date-time":"2022-10-06T00:00:00Z","timestamp":1665014400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100012331","name":"Agentschap Innoveren en Ondernemen, Belgium","doi-asserted-by":"publisher","award":["HBC.2020.2246"],"award-info":[{"award-number":["HBC.2020.2246"]}],"id":[{"id":"10.13039\/100012331","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012331","name":"Agentschap Innoveren en Ondernemen, Belgium","doi-asserted-by":"publisher","award":["101016985"],"award-info":[{"award-number":["101016985"]}],"id":[{"id":"10.13039\/100012331","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100012331","name":"Agentschap Innoveren en Ondernemen, Belgium","doi-asserted-by":"publisher","award":["1S36322N"],"award-info":[{"award-number":["1S36322N"]}],"id":[{"id":"10.13039\/100012331","id-type":"DOI","asserted-by":"publisher"}]},{"name":"European Union\u2019s Horizon 2020 research and innovation programme","award":["HBC.2020.2246"],"award-info":[{"award-number":["HBC.2020.2246"]}]},{"name":"European Union\u2019s Horizon 2020 research and innovation programme","award":["101016985"],"award-info":[{"award-number":["101016985"]}]},{"name":"European Union\u2019s Horizon 2020 research and innovation programme","award":["1S36322N"],"award-info":[{"award-number":["1S36322N"]}]},{"name":"Flemish Research Foundation","award":["HBC.2020.2246"],"award-info":[{"award-number":["HBC.2020.2246"]}]},{"name":"Flemish Research Foundation","award":["101016985"],"award-info":[{"award-number":["101016985"]}]},{"name":"Flemish Research Foundation","award":["1S36322N"],"award-info":[{"award-number":["1S36322N"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Robot-assisted surgery is becoming popular in the operation room (OR) for, e.g., orthopedic surgery (among other surgeries). However, robotic executions related to surgical steps cannot simply rely on preoperative plans. Using pedicle screw placement as an example, extra adjustments are needed to adapt to the intraoperative changes when the preoperative planning is outdated. During surgery, adjusting a surgical plan is non-trivial and typically rather complex since the available interfaces used in current robotic systems are not always intuitive to use. Recently, thanks to technical advancements in head-mounted displays (HMD), augmented reality (AR)-based medical applications are emerging in the OR. The rendered virtual objects can be overlapped with real-world physical objects to offer intuitive displays of the surgical sites and anatomy. Moreover, the potential of combining AR with robotics is even more promising; however, it has not been fully exploited. In this paper, an innovative AR-based robotic approach is proposed and its technical feasibility in simulated pedicle screw placement is demonstrated. An approach for spatial calibration between the robot and HoloLens 2 without using an external 3D tracking system is proposed. The developed system offers an intuitive AR\u2013robot interaction approach between the surgeon and the surgical robot by projecting the current surgical plan to the surgeon for fine-tuning and transferring the updated surgical plan immediately back to the robot side for execution. A series of bench-top experiments were conducted to evaluate system accuracy and human-related errors. A mean calibration error of 3.61 mm was found. The overall target pose error was 3.05 mm in translation and 1.12\u2218 in orientation. The average execution time for defining a target entry point intraoperatively was 26.56 s. This work offers an intuitive AR-based robotic approach, which could facilitate robotic technology in the OR and boost synergy between AR and robots for other medical applications.<\/jats:p>","DOI":"10.3390\/jimaging8100273","type":"journal-article","created":{"date-parts":[[2022,10,8]],"date-time":"2022-10-08T23:39:31Z","timestamp":1665272371000},"page":"273","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["An Augmented Reality-Based Interaction Scheme for Robotic Pedicle Screw Placement"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1426-4309","authenticated-orcid":false,"given":"Viktor","family":"V\u00f6r\u00f6s","sequence":"first","affiliation":[{"name":"Robot-Assisted Surgery Group, Department of Mechanical Engineering, KU Leuven, Celestijnenlaan 300, 3000 Leuven, Belgium"},{"name":"Healthcare Division, Barco NV, Beneluxpark 21, 8500 Kortrijk, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2882-8398","authenticated-orcid":false,"given":"Ruixuan","family":"Li","sequence":"additional","affiliation":[{"name":"Robot-Assisted Surgery Group, Department of Mechanical Engineering, KU Leuven, Celestijnenlaan 300, 3000 Leuven, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7407-5000","authenticated-orcid":false,"given":"Ayoob","family":"Davoodi","sequence":"additional","affiliation":[{"name":"Robot-Assisted Surgery Group, Department of Mechanical Engineering, KU Leuven, Celestijnenlaan 300, 3000 Leuven, Belgium"}]},{"given":"Gauthier","family":"Wybaillie","sequence":"additional","affiliation":[{"name":"Robot-Assisted Surgery Group, Department of Mechanical Engineering, KU Leuven, Celestijnenlaan 300, 3000 Leuven, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3764-9551","authenticated-orcid":false,"given":"Emmanuel","family":"Vander Poorten","sequence":"additional","affiliation":[{"name":"Robot-Assisted Surgery Group, Department of Mechanical Engineering, KU Leuven, Celestijnenlaan 300, 3000 Leuven, Belgium"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5498-4311","authenticated-orcid":false,"given":"Kenan","family":"Niu","sequence":"additional","affiliation":[{"name":"Robot-Assisted Surgery Group, Department of Mechanical Engineering, KU Leuven, Celestijnenlaan 300, 3000 Leuven, Belgium"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"784","DOI":"10.1177\/2192568218770769","article-title":"Degenerative lumbar spine disease: Estimating global incidence and worldwide volume","volume":"8","author":"Ravindra","year":"2018","journal-title":"Glob. Spine J."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"E465","DOI":"10.1097\/BRS.0b013e3181d1021a","article-title":"Complications of pedicle screw fixation in scoliosis surgery: A systematic review","volume":"35","author":"Hicks","year":"2010","journal-title":"Spine"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"50","DOI":"10.3109\/17453674.2010.548032","article-title":"The rate of screw misplacement in segmental pedicle screw fixation in adolescent idiopathic scoliosis: The effect of learning and cumulative experience","volume":"82","author":"Ohlin","year":"2011","journal-title":"Acta Orthop."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Waschke, A., Walter, J., Duenisch, P., Reichart, R., Kalff, R., and Ewald, C. (2013). CT-navigation versus fluoroscopy-guided placement of pedicle screws at the thoracolumbar spine: Single center experience of 4500 screws. Eur. Spine J., 654\u2013660.","DOI":"10.1007\/s00586-012-2509-3"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1007\/s12178-021-09699-3","article-title":"Augmented Reality in Orthopedic Surgery Is Emerging from Proof of Concept towards Clinical Studies: A Literature Review Explaining the Technology and Current State of the Art","volume":"14","author":"Casari","year":"2021","journal-title":"Curr. Rev. Musculoskelet. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1007\/s11548-018-1814-7","article-title":"Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography","volume":"14","author":"Gibby","year":"2019","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"542","DOI":"10.3171\/2019.10.SPINE19969","article-title":"Percutaneous placement of lumbar pedicle screws via intraoperative CT image\u2013based augmented reality\u2013guided technology","volume":"32","author":"Liu","year":"2019","journal-title":"J. Neurosurg. Spine"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13018-020-01690-x","article-title":"Augmented reality-based navigation increases precision of pedicle screw insertion","volume":"15","author":"Dennler","year":"2020","journal-title":"J. Orthop. Surg. Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1016\/j.spinee.2019.10.012","article-title":"Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging","volume":"20","author":"Muller","year":"2020","journal-title":"Spine J."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2554","DOI":"10.1109\/LRA.2020.2972892","article-title":"A Surgeon-Robot Shared Control for Ergonomic Pedicle Screw Fixation","volume":"5","author":"Lauretti","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1109\/TMRB.2022.3155288","article-title":"An Automatic Path Planning Method of Pedicle Screw Placement Based on Preoperative CT Images","volume":"4","author":"Qi","year":"2022","journal-title":"IEEE Trans. Med. Robot. Bionics"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1109\/TMRB.2020.3037339","article-title":"Automated Polyaxial Screw Placement Using a Commercial-Robot-Based, Image-Guided Spine Surgery System","volume":"3","author":"Smith","year":"2021","journal-title":"IEEE Trans. Med. Robot. Bionics"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1097\/BSD.0b013e318218a5ef","article-title":"Assessment of pedicle screw placement accuracy, procedure time, and radiation exposure using a miniature robotic guidance system","volume":"25","author":"Lieberman","year":"2012","journal-title":"J. Spinal Disord. Tech."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"4556","DOI":"10.1109\/LRA.2020.3002216","article-title":"Augmented Reality and Robotic-Assistance for Percutaneous Nephrolithotomy","volume":"5","author":"Ferraguti","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2593","DOI":"10.1109\/TBME.2022.3150952","article-title":"Ultrasound Image Guided and Mixed Reality-Based Surgical System with Real-Time Soft Tissue Deformation Computing for Robotic Cervical Pedicle Screw Placement","volume":"69","author":"Tu","year":"2022","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_16","unstructured":"Bruyninckx, H. (2001, January 21\u201326). Open robot control software: The OROCOS project. Proceedings of the Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), Seoul, Korea."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1421","DOI":"10.1177\/0278364918768950","article-title":"Admittance control for physical human\u2013robot interaction","volume":"37","author":"Keemink","year":"2018","journal-title":"Int. J. Robot. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02289451","article-title":"A generalized solution of the orthogonal procrustes problem","volume":"31","year":"1966","journal-title":"Psychometrika"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"101795","DOI":"10.1109\/ACCESS.2019.2930701","article-title":"An Online Calibration Method for Microsoft HoloLens","volume":"7","author":"Guo","year":"2019","journal-title":"IEEE Access"},{"key":"ref_20","first-page":"261","article-title":"Feasibility of image-based augmented reality guidance of total shoulder arthroplasty using microsoft HoloLens 1","volume":"9","author":"Gu","year":"2021","journal-title":"Spine J."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/8\/10\/273\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:47:15Z","timestamp":1760143635000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/8\/10\/273"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,6]]},"references-count":20,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["jimaging8100273"],"URL":"https:\/\/doi.org\/10.3390\/jimaging8100273","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,6]]}}}