{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T22:59:15Z","timestamp":1768085955578,"version":"3.49.0"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,5,27]],"date-time":"2022-05-27T00:00:00Z","timestamp":1653609600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,27]],"date-time":"2022-05-27T00:00:00Z","timestamp":1653609600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Virtual Reality"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Augmented reality (AR) permits the visualization of pre-operative data in the surgical field of view of the surgeon. This requires the alignment of the AR device\u2019s coordinate system with the used navigation\/tracking system. We propose a multimodal marker approach to align an AR device with a tracking system: in our implementation, an electromagnetic tracking system (EMTS). The solution makes use of a calibration method which determines the relationship between a 2D pattern detected by an RGB camera and an electromagnetic sensor of the EMTS. This allowed the projection of a 3D skull model on its physical counterpart. This projection was evaluated using a monocular camera and an optical see-through device (HoloLens 2) (<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/www.microsoft.com\/en-us\/hololens\/\">https:\/\/www.microsoft.com\/en-us\/hololens\/<\/jats:ext-link>) achieving an accuracy of less than 2.5 mm in the image plane of the HoloLens 2 (HL2). Additionally, 10 volunteers participated in a user study consisting of an alignment task of a pointer with 25 projections viewed through the HL2. The participants achieved a mean error of 2.7 1.3 mm and 2.9 2.9<jats:inline-formula><jats:alternatives><jats:tex-math>$$^\\circ $$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:msup>\n                    <mml:mrow\/>\n                    <mml:mo>\u2218<\/mml:mo>\n                  <\/mml:msup>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> in positional and orientation error. This study showcases the feasibility of the approach, provides an evaluation of the alignment, and finally, discusses its advantages and limitations.<\/jats:p>","DOI":"10.1007\/s10055-022-00653-3","type":"journal-article","created":{"date-parts":[[2022,5,27]],"date-time":"2022-05-27T12:03:01Z","timestamp":1653652981000},"page":"1637-1650","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Multimodal markers for technology-independent integration of augmented reality devices and surgical navigation systems"],"prefix":"10.1007","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1830-2480","authenticated-orcid":false,"given":"Mohamed","family":"Benmahdjoub","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wiro J.","family":"Niessen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eppo B.","family":"Wolvius","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Theo van","family":"Walsum","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,5,27]]},"reference":[{"key":"653_CR1","doi-asserted-by":"publisher","unstructured":"Azimi E, et al. (2017) Robust optical see- through head-mounted display calibration: taking anisotropic nature of user interaction errors into account. In: Proceedings\u2014IEEE virtual reality, pp 219\u2013220. https:\/\/doi.org\/10.1109\/VR.2017.7892255","DOI":"10.1109\/VR.2017.7892255"},{"key":"653_CR2","doi-asserted-by":"crossref","unstructured":"Azimi E, et al. (2020) An interactive mixed re- ality platform for bedside surgical procedures. In: Medical image computing and computer assisted intervention - MICCAI 2020. Ed. by Anne L. Mar- tel et al. Springer, Cham, pp 65\u201375. isbn: 978-3-030-59716-0","DOI":"10.1007\/978-3-030-59716-0_7"},{"issue":"8","key":"653_CR3","doi-asserted-by":"publisher","first-page":"1970","DOI":"10.1016\/j.jcms.2014.09.001","volume":"42","author":"G Badiali","year":"2014","unstructured":"Badiali G et al (2014) Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning. J Cranio-Maxillof Surg 42(8):1970\u20131976. https:\/\/doi.org\/10.1016\/j.jcms.2014.09.001","journal-title":"J Cranio-Maxillof Surg"},{"issue":"4","key":"653_CR4","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1007\/s00423-016-1417-0","volume":"401","author":"VM Banz","year":"2016","unstructured":"Banz VM et al (2016) Intraoperative image-guided navigation system: development and applicability in 65 patients undergoing liver surgery. Langenbeck\u2019s Arch Surg 401(4):495\u2013502. https:\/\/doi.org\/10.1007\/s00423-016-1417-0","journal-title":"Langenbeck\u2019s Arch Surg"},{"key":"653_CR5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijom.2020.11.015","author":"M Benmahdjoub","year":"2020","unstructured":"Benmahdjoub M et al (2020) Augmented reality in craniomaxillofacial surgery: added value and pro- posed recommendations through a systematic review of the literature. Int J Oral Maxillof Surg. https:\/\/doi.org\/10.1016\/j.ijom.2020.11.015","journal-title":"Int J Oral Maxillof Surg"},{"issue":"11","key":"653_CR6","doi-asserted-by":"publisher","first-page":"4332","DOI":"10.1109\/TVCG.2021.3106506","volume":"27","author":"M Benmahdjoub","year":"2021","unstructured":"Benmahdjoub M et al (2021) Virtual extensions improve perception-based instrument alignment using optical see-through devices. IEEE Trans Visual Comput Graph 27(11):4332\u20134341. https:\/\/doi.org\/10.1109\/TVCG.2021.3106506","journal-title":"IEEE Trans Visual Comput Graph"},{"issue":"9","key":"653_CR7","doi-asserted-by":"publisher","first-page":"1731","DOI":"10.1016\/J.JCMS.2015.08.022","volume":"43","author":"M Berger","year":"2015","unstructured":"Berger M et al (2015) Approach to intraoperative electromagnetic navigation in orthognathic surgery: a phantom skull based trial. J Cranio-Maxillof Surg 43(9):1731\u20131736. https:\/\/doi.org\/10.1016\/J.JCMS.2015.08.022","journal-title":"J Cranio-Maxillof Surg"},{"key":"653_CR8","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1016\/J.JBI.2015.04.003","volume":"55","author":"X Chen","year":"2015","unstructured":"Chen X et al (2015) Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. J Biomed Informatics 55:124\u2013131. https:\/\/doi.org\/10.1016\/J.JBI.2015.04.003","journal-title":"J Biomed Informatics"},{"key":"653_CR9","first-page":"1382","volume":"91","author":"T Chotanaphuti","year":"2008","unstructured":"Chotanaphuti T et al (2008) Comparative study between computer assisted-navigation and conventional technique in minimally invasive surgery total knee arthroplasty, prospective control study. J Med Assoc Thailand Chotmaihet Thangphaet 91:1382\u20138","journal-title":"J Med Assoc Thailand Chotmaihet Thangphaet"},{"issue":"1","key":"653_CR10","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1146\/annurev-bioeng-070909-105249","volume":"12","author":"K Cleary","year":"2010","unstructured":"Cleary K, Terry MP (2010) Image- guided interventions: technology review and clinical applications. Ann Rev Biomed Eng 12(1):119\u2013142. https:\/\/doi.org\/10.1146\/annurev-bioeng-070909-105249","journal-title":"Ann Rev Biomed Eng"},{"key":"653_CR11","doi-asserted-by":"crossref","unstructured":"Do TD, Joseph JL, Ryan PM (2020) The effects of object shape, fidelity, color, and luminance on depth percep- tion in handheld mobile augmented reality. In: Proceedings\u20142020 IEEE international symposium on mixed and augmented reality, ISMAR 2020, pp 64\u201372. arXiv: 2008.05505","DOI":"10.1109\/ISMAR50242.2020.00026"},{"key":"653_CR12","doi-asserted-by":"publisher","unstructured":"Eggers G, et al. (2009) Intraoperative computed to- mography and automated registration for image- guided cranial surgery. In: Dentomaxillofacial Radiology 38.1. PMID: 19114421, pp. 28\u201333. https:\/\/doi.org\/10.1259\/dmfr\/26098099","DOI":"10.1259\/dmfr\/26098099"},{"issue":"4","key":"653_CR13","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1007\/s11548-011-0660-7","volume":"7","author":"K Gavaghan","year":"2012","unstructured":"Gavaghan K et al (2012) Evaluation of a portable image overlay projector for the visualisation of surgical navigation data: phantom studies. Int J Comput Assist Radiol Surg 7(4):547\u2013556. https:\/\/doi.org\/10.1007\/s11548-011-0660-7","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"653_CR14","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1109\/ISMAR.2002.1115086","volume":"2002","author":"Y Genc","year":"2002","unstructured":"Genc Y, Tuceryan M, Navab N (2002) Practical solutions for calibration of optical see-through devices. Proc Int Symp Mixed Augment Real ISMAR 2002:169\u2013175. https:\/\/doi.org\/10.1109\/ISMAR.2002.1115086","journal-title":"Proc Int Symp Mixed Augment Real ISMAR"},{"key":"653_CR15","doi-asserted-by":"publisher","unstructured":"Gsaxner C, et al. (2019) Markerless image-to- face registration for untethered augmented reality in head and neck surgery. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 11768 LNCS. Springer, pp 236\u2013244. ISBN: 9783030322533. https:\/\/doi.org\/10.1007\/978-3-030-32254-0_27","DOI":"10.1007\/978-3-030-32254-0_27"},{"key":"653_CR16","doi-asserted-by":"publisher","unstructured":"Hara T, et al. (2020) Efficacy of atlantoax- ial transarticular screw fixation using navigation- guided drill: technical note. In: World Neurosurgery 134, pp 378\u2013382. ISSN: 1878-8750. https:\/\/doi.org\/10.1016\/j.wneu.2019.10.176","DOI":"10.1016\/j.wneu.2019.10.176"},{"issue":"4","key":"653_CR17","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1364\/JOSAA.4.000629","volume":"4","author":"BKP Horn","year":"1987","unstructured":"Horn BKP (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A 4(4):629\u2013642. https:\/\/doi.org\/10.1364\/JOSAA.4.000629","journal-title":"J Opt Soc Am A"},{"key":"653_CR18","doi-asserted-by":"publisher","first-page":"e422","DOI":"10.1016\/j.wneu.2018.06.208","volume":"118","author":"F Incekara","year":"2018","unstructured":"Incekara F et al (2018) Clinical feasibility of a wearable mixed-reality device in neuro- surgery. World Neurosurg 118:e422\u2013e427. https:\/\/doi.org\/10.1016\/j.wneu.2018.06.208","journal-title":"World Neurosurg"},{"key":"653_CR19","doi-asserted-by":"publisher","unstructured":"Itoh Y, Gudrun K (2014) Performance and sensitivity analysis of INDICA: INteraction-free display calibration for optical see-through head-mounted displays. In: ISMAR 2014\u2014IEEE international symposium on mixed and augmented reality\u2014science and technology 2014, proceedings, pp 171\u2013176. https:\/\/doi.org\/10.1109\/ISMAR.2014.6948424","DOI":"10.1109\/ISMAR.2014.6948424"},{"key":"653_CR20","doi-asserted-by":"publisher","unstructured":"Jiang T, et al. (2019) Precision of a novel craniofacial surgical navigation system based on augmented reality using an occlusal splint as a registration strategy. In: Scientific reports 9.1, p 501. ISSN: 20452322. https:\/\/doi.org\/10.1038\/s41598-018-36457-2","DOI":"10.1038\/s41598-018-36457-2"},{"key":"653_CR21","doi-asserted-by":"publisher","unstructured":"Khenak N, Jeanne V, Patrick B (2020) Effectiveness of augmented reality guides for blind insertion tasks. In: Frontiers in virtual reality 1, 588217. ISSN: 2673-4192. https:\/\/doi.org\/10.3389\/frvir.2020.588217","DOI":"10.3389\/frvir.2020.588217"},{"key":"653_CR22","doi-asserted-by":"publisher","unstructured":"Kuzhagaliyev T, et al. (2018) Augmented reality needle ablation guidance tool for irreversible electroporation in the pancreas. In: Medical imaging 2018: image-guided procedures, robotic interventions, and modeling. Ed. by Robert J. Webster and Baowei Fei. Vol. 10576. SPIE, p. 30. isbn: 9781510616417. https:\/\/doi.org\/10.1117\/12.2293671","DOI":"10.1117\/12.2293671"},{"key":"653_CR23","unstructured":"Lewis JR, et al. (2011) Aligning interpupillary distance in a near-eye display system. 20130050642A1. url: https:\/\/patents.google. com\/patent\/US20130050642A1\/en"},{"key":"653_CR24","doi-asserted-by":"publisher","unstructured":"Li Y, et al. (2018) A wearable mixed-reality holographic computer for guiding external ventricular drain insertion at the bedside. en. In: J Neurosurg, p. 1. ISSN: 0022-3085. https:\/\/doi.org\/10.3171\/2018.4.JNS18124","DOI":"10.3171\/2018.4.JNS18124"},{"key":"653_CR25","doi-asserted-by":"publisher","unstructured":"Makibuchi N, Haruhisa K, Akio Y (2013) Vision-based robust calibration for optical see-through head-mounted displays. In: 2013 IEEE international conference on image processing, ICIP 2013\u2014proceedings, pp 2177\u20132181. https:\/\/doi.org\/10.1109\/ICIP.2013.6738449","DOI":"10.1109\/ICIP.2013.6738449"},{"key":"653_CR26","doi-asserted-by":"publisher","unstructured":"Martin-Gomez A, et al. (2020) Augmented mirrors. In: Proceedings\u20142020 IEEE international symposium on mixed and augmented reality, ISMAR 2020. Institute of Electrical and Electronics Engineers Inc., pp. 217\u2013226. https:\/\/doi.org\/10.1109\/ISMAR50242.2020.00045","DOI":"10.1109\/ISMAR50242.2020.00045"},{"key":"653_CR27","doi-asserted-by":"publisher","unstructured":"Martin-Gomez A, et al. (2020) Gain a new perspective: towards exploring multi-view alignment in mixed reality. In: Proceedings \u20142020 IEEE international symposium on mixed and augmented reality, ISMAR 2020. Institute of Electrical 15 and Electronics Engineers Inc., pp. 207\u2013216. https:\/\/doi.org\/10.1109\/ISMAR50242.2020.00044","DOI":"10.1109\/ISMAR50242.2020.00044"},{"issue":"1","key":"653_CR28","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1177\/1553350618799552","volume":"26","author":"JW Meulstee","year":"2019","unstructured":"Meulstee JW et al (2019) Toward holographic-guided surgery. Surg Innov 26(1):86\u201394. https:\/\/doi.org\/10.1177\/1553350618799552","journal-title":"Surg Innov"},{"key":"653_CR29","unstructured":"MevisLab (2020). https:\/\/www.mevislab.de\/. (Visited on 05\/11\/2020)"},{"key":"653_CR30","doi-asserted-by":"publisher","unstructured":"Mezger U, Claudia J, Michael B (2013) Navigation in surgery. In: Langen beck\u2019s archives of surgery 398.4, pp 501\u2013514. ISSN: 1435-2451. https:\/\/doi.org\/10.1007\/s00423-013-1059-4","DOI":"10.1007\/s00423-013-1059-4"},{"issue":"11","key":"653_CR31","doi-asserted-by":"publisher","first-page":"892","DOI":"10.1002\/jsid.947","volume":"28","author":"J Ping","year":"2020","unstructured":"Ping J et al (2020) Effects of shading model and opacity on depth perception in optical see-through augmented reality. J Soc Inf Disp 28(11):892\u2013904. https:\/\/doi.org\/10.1002\/jsid.947","journal-title":"J Soc Inf Disp"},{"key":"653_CR32","doi-asserted-by":"publisher","unstructured":"Singh G, et al. (2010) Depth judgment measures and occluding surfaces in near-field augmented reality. In: Proceedings\u2014APGV 2010: symposium on applied perception in graphics and visualization, pp 149\u2013156. isbn: 9781450302487. https:\/\/doi.org\/10.1145\/1836248.1836277","DOI":"10.1145\/1836248.1836277"},{"issue":"11","key":"653_CR33","doi-asserted-by":"publisher","first-page":"1289","DOI":"10.1109\/TVCG.2015","volume":"21","author":"JE Swan","year":"2015","unstructured":"Swan JE, Gurjot S, Stephen RE (2015) Matching and reaching depth judgments with real and augmented reality targets. IEEE Trans Visual Comput Graph 21(11):1289\u20131298. https:\/\/doi.org\/10.1109\/TVCG.2015","journal-title":"IEEE Trans Visual Comput Graph"},{"key":"653_CR34","doi-asserted-by":"publisher","unstructured":"Tuceryan M, Navab N(2000) Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR. In: pp 149\u2013158. https:\/\/doi.org\/10.1109\/ISAR.2000.880938","DOI":"10.1109\/ISAR.2000.880938"},{"key":"653_CR35","unstructured":"Vuforia (2020). https:\/\/developer.vuforia.com\/downloads\/sdk. (Visited on 05\/11\/2020)"},{"issue":"4","key":"653_CR36","doi-asserted-by":"publisher","first-page":"1295","DOI":"10.1109\/TBME.2014.2301191","volume":"61","author":"J Wang","year":"2014","unstructured":"Wang J et al (2014) Augmented reality navigation with automatic marker-free image registration using 3-d image overlay for dental surgery. IEEE Trans Biomed Eng 61(4):1295\u20131304. https:\/\/doi.org\/10.1109\/TBME.2014.2301191","journal-title":"IEEE Trans Biomed Eng"},{"issue":"1","key":"653_CR37","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1007\/s40593-014-0032-x","volume":"25","author":"G Westerfield","year":"2015","unstructured":"Westerfield G, Antonija M, Mark B (2015) Intelligent augmented reality training for motherboard assembly. Int J Artif Intell Educ 25(1):157\u2013172. https:\/\/doi.org\/10.1007\/s40593-014-0032-x","journal-title":"Int J Artif Intell Educ"},{"key":"653_CR38","doi-asserted-by":"publisher","first-page":"42365","DOI":"10.1038\/srep42365","volume":"7","author":"M Zhu","year":"2017","unstructured":"Zhu M et al (2017) A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery. Sci Rep 7:42365. https:\/\/doi.org\/10.1038\/srep42365","journal-title":"Sci Rep"},{"issue":"8","key":"653_CR39","doi-asserted-by":"publisher","first-page":"1188","DOI":"10.1016\/j.bjps.2018.03.018","volume":"71","author":"M Zhu","year":"2018","unstructured":"Zhu M et al (2018) Does intraoperative navigation improve the accuracy of mandibular angle osteotomy: comparison between augmented reality navigation, individualised templates and free-hand techniques. J Plast Reconstruct Aesthet Surg 71(8):1188\u20131195. https:\/\/doi.org\/10.1016\/j.bjps.2018.03.018","journal-title":"J Plast Reconstruct Aesthet Surg"},{"issue":"8","key":"653_CR40","doi-asserted-by":"publisher","first-page":"827","DOI":"10.1016\/J.BJOMS.2013.06.014","volume":"51","author":"MJ Zinser","year":"2013","unstructured":"Zinser MJ et al (2013) Computer-assisted orthognathic surgery: waferless maxillary positioning, versatility, and accuracy of an image-guided visualisation display. Br J Oral Maxillof Surg 51(8):827\u2013833. https:\/\/doi.org\/10.1016\/J.BJOMS.2013.06.014","journal-title":"Br J Oral Maxillof Surg"}],"container-title":["Virtual Reality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-022-00653-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10055-022-00653-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-022-00653-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,22]],"date-time":"2022-10-22T17:16:29Z","timestamp":1666458989000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10055-022-00653-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,27]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["653"],"URL":"https:\/\/doi.org\/10.1007\/s10055-022-00653-3","relation":{},"ISSN":["1359-4338","1434-9957"],"issn-type":[{"value":"1359-4338","type":"print"},{"value":"1434-9957","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,27]]},"assertion":[{"value":"13 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 April 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflicts of interest to declare that are relevant to the content of this article. The authors declare that no patient consent was required.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}