{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T17:16:07Z","timestamp":1786122967783,"version":"build-2736575974"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T00:00:00Z","timestamp":1779321600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T00:00:00Z","timestamp":1779321600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005713","name":"Technische Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005713","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>The integration of multimodal imaging into operating rooms paves the way for comprehensive surgical scene understanding. In ophthalmic surgery, by now, two complementary imaging modalities are available: operating microscope (OPMI) imaging and real-time intraoperative optical coherence tomography (iOCT). This first work toward temporal OPMI and iOCT feature fusion demonstrates the potential of multimodal image processing for multi-head prediction through the example of precise instrument tracking in vitreoretinal surgery.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We propose a multimodal, temporal, real-time capable network architecture to perform joint instrument detection, keypoint localization, and tool-tissue distance estimation. Our network design integrates a cross-attention fusion module to merge OPMI and iOCT image features, which are efficiently extracted via a Yolo-NAS and a CNN encoder, respectively. Furthermore, a region-based recurrent module leverages temporal coherence.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Our experiments demonstrate reliable instrument localization and keypoint detection (95.79% mAP50) and show that the incorporation of iOCT significantly improves tool-tissue distance estimation, while achieving real-time processing rates of 22.5 ms per frame. Especially for close distances to the retina (below 1\u00a0mm), the distance estimation accuracy improved from\n                      <jats:inline-formula>\n                        <jats:alternatives>\n                          <jats:tex-math>$$284\\,\\upmu \\hbox {m}$$<\/jats:tex-math>\n                          <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                            <mml:mrow>\n                              <mml:mn>284<\/mml:mn>\n                              <mml:mspace\/>\n                              <mml:mi>\u03bc<\/mml:mi>\n                              <mml:mtext>m<\/mml:mtext>\n                            <\/mml:mrow>\n                          <\/mml:math>\n                        <\/jats:alternatives>\n                      <\/jats:inline-formula>\n                      (OPMI only) to\n                      <jats:inline-formula>\n                        <jats:alternatives>\n                          <jats:tex-math>$$33\\,\\upmu \\hbox {m}$$<\/jats:tex-math>\n                          <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                            <mml:mrow>\n                              <mml:mn>33<\/mml:mn>\n                              <mml:mspace\/>\n                              <mml:mi>\u03bc<\/mml:mi>\n                              <mml:mtext>m<\/mml:mtext>\n                            <\/mml:mrow>\n                          <\/mml:math>\n                        <\/jats:alternatives>\n                      <\/jats:inline-formula>\n                      (multimodal).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Feature fusion of multimodal imaging can enhance multitask prediction accuracy compared to single-modality processing, and real-time processing performance can be achieved through tailored network design. While our results demonstrate the potential of multimodal processing for image-guided vitreoretinal surgery, they also underline key challenges that motivate future research toward more reliable, consistent, and comprehensive surgical scene understanding.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s11548-026-03711-2","type":"journal-article","created":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T07:21:14Z","timestamp":1779348074000},"page":"1359-1367","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Toward comprehensive real-time scene understanding in ophthalmic surgery through multimodal image fusion"],"prefix":"10.1007","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-4242-2604","authenticated-orcid":false,"given":"Nikolo","family":"Rohrmoser","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghazal","family":"Ghazaei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Sommersperger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nassir","family":"Navab","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,21]]},"reference":[{"issue":"4","key":"3711_CR1","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.oret.2017.05.006","volume":"2","author":"JP Ehlers","year":"2018","unstructured":"Ehlers JP, Khan M, Petkovsek D, Stiegel L, Kaiser PK, Singh RP, Reese JL, Srivastava SK (2018) Outcomes of intraoperative oct-assisted epiretinal membrane surgery from the pioneer study. Ophthalmol Retina 2(4):263\u2013267","journal-title":"Ophthalmol Retina"},{"key":"3711_CR2","doi-asserted-by":"crossref","unstructured":"Posarelli C, Sartini F, Casini G, Passani A, Toro MD, Vella G, Figus M (2020) What is the impact of intraoperative microscope-integrated oct in ophthalmic surgery? relevant applications and outcomes. a systematic review. JCM 9(6):1682","DOI":"10.3390\/jcm9061682"},{"issue":"1","key":"3711_CR3","doi-asserted-by":"publisher","first-page":"24848","DOI":"10.1038\/s41598-025-07670-7","volume":"15","author":"L Sch\u00fctz","year":"2025","unstructured":"Sch\u00fctz L, Dehghani S, Sommersperger M, Faridpooya K, Navab N (2025) The impact of intraoperative optical coherence tomography on cognitive load in virtual reality vitreoretinal surgery training. Sci Rep 15(1):24848","journal-title":"Sci Rep"},{"key":"3711_CR4","doi-asserted-by":"crossref","unstructured":"Roodaki H, Filippatos K, Eslami A, Navab N (2015) Introducing augmented reality to optical coherence tomography in ophthalmic microsurgery. IEEE ISMAR 1\u20136IEEE","DOI":"10.1109\/ISMAR.2015.15"},{"key":"3711_CR5","doi-asserted-by":"crossref","unstructured":"Matinfar S, Dehghani S, Sommersperger M, Faridpooya K, Fairhurst M, Navab N (2024) Ocular stethoscope: Auditory support for retinal membrane peeling. In: MICCAI, Springer 433\u2013443","DOI":"10.1007\/978-3-031-72089-5_41"},{"key":"3711_CR6","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.media.2016.05.003","volume":"34","author":"N Rieke","year":"2016","unstructured":"Rieke N, Tan DJ, San Filippo CA, Tombari F, Alsheakhali M, Belagiannis V, Eslami A, Navab N (2016) Real-time localization of articulated surgical instruments in retinal microsurgery. Med Image Anal 34:82\u2013100","journal-title":"Med Image Anal"},{"key":"3711_CR7","doi-asserted-by":"crossref","unstructured":"K\u00f6ksal \u00c7, Ghazaei G, Holm F, Farshad A, Navab N (2024) Sangria: surgical video scene graph optimization for surgical workflow prediction. In: GRAIL, pp. 106\u2013117. Springer","DOI":"10.1007\/978-3-031-83243-7_10"},{"issue":"1","key":"3711_CR8","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1007\/s11548-025-03472-4","volume":"21","author":"D Gastager","year":"2026","unstructured":"Gastager D, Ghazaei G, Patsch C (2026) Watch and learn: leveraging expert knowledge and language for surgical video understanding. Int J Comput Assist Radiol Surg 21(1):185\u2013194. https:\/\/doi.org\/10.1007\/s11548-025-03472-4","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"3711_CR9","doi-asserted-by":"crossref","unstructured":"Kim JW, He C, Urias M, Gehlbach P, Hager GD, Iordachita I, Kobilarov M (2020) Autonomously navigating a surgical tool inside the eye by learning from demonstration. In: 2020 IEEE (ICRA), pp. 7351\u20137357 . IEEE","DOI":"10.1109\/ICRA40945.2020.9196537"},{"key":"3711_CR10","doi-asserted-by":"crossref","unstructured":"Yang J, Zhao Z, Maier M, Huang K, Navab N, Ali Nasseri M (2024) Shadow-based 3d pose estimation of intraocular instrument using only 2d images. In: 2024 IEEE (ICRA), pp. 1323\u20131329","DOI":"10.1109\/ICRA57147.2024.10611011"},{"key":"3711_CR11","doi-asserted-by":"crossref","unstructured":"Arikan D, Zhang P, Sommersperger M, Dehghani S, Esfandiari M, Taylor RH, Nasseri MA, Gehlbach P, Navab N, Iordachita I (2025) Real-time deformation-aware control for autonomous robotic subretinal injection under ioct guidance. IEEE ICRA 10531\u201310537 IEEE","DOI":"10.1109\/ICRA55743.2025.11128595"},{"key":"3711_CR12","unstructured":"Park S-J, Hong K-S, Lee S (2017) Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation. IEEE ICCV. 4980\u20134989"},{"key":"3711_CR13","doi-asserted-by":"crossref","unstructured":"Wang C, Xu D, Zhu Y, Mart\u00edn-Mart\u00edn R, Lu C, Fei-Fei L, Savarese S (2019) Densefusion: 6d object pose estimation by iterative dense fusion. IEEE\/CVF CVPR. 3343\u20133352","DOI":"10.1109\/CVPR.2019.00346"},{"key":"3711_CR14","unstructured":"Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Neur IPS 27"},{"key":"3711_CR15","doi-asserted-by":"crossref","unstructured":"Carreira J, Zisserman A (2017) Quo vadis, action recognition? A new model and the kinetics dataset. IEEE CVPR. 6299\u20136308","DOI":"10.1109\/CVPR.2017.502"},{"issue":"12","key":"3711_CR16","first-page":"14679","volume":"24","author":"J Zhang","year":"2023","unstructured":"Zhang J, Liu H, Yang K, Hu X, Liu R, Stiefelhagen R (2023) Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. IEEE T-ITS 24(12):14679\u201314694","journal-title":"IEEE T-ITS"},{"key":"3711_CR17","doi-asserted-by":"crossref","unstructured":"Laina I, Rieke N, Rupprecht C, Vizca\u00edno JP, Eslami A, Tombari F, Navab N (2017)Concurrent segmentation and localization for tracking of surgical instruments. In: MICCAI, pp. 664\u2013672. Springer","DOI":"10.1007\/978-3-319-66185-8_75"},{"key":"3711_CR18","doi-asserted-by":"crossref","unstructured":"Weerasinghe K, Roodabeh SHR, Hutchinson K, Alemzadeh H (2024) Multimodal transformers for real-time surgical activity prediction. In: 2024 IEEE (ICRA), pp. 13323\u201313330. IEEE","DOI":"10.1109\/ICRA57147.2024.10611048"},{"key":"3711_CR19","unstructured":"Mostafa ML, Alperovich A, Fedotov D, Ghazaei G, Saur S, Farshad A, Navab N (2025) Surgical flow masked autoencoder for event recognition. p MIDL"},{"issue":"1","key":"3711_CR20","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1186\/s40662-024-00405-1","volume":"11","author":"S Wang","year":"2024","unstructured":"Wang S, He X, Jian Z, Li J, Xu C, Chen Y, Liu Y, Chen H, Huang C, Hu J et al (2024) Advances and prospects of multi-modal ophthalmic artificial intelligence based on deep learning: a review. Eye Vision 11(1):38","journal-title":"Eye Vision"},{"key":"3711_CR21","unstructured":"Aharon S, Louis-Dupont Ofri Masad Yurkova K, Lotem Fridman Lkdci Khvedchenya E, Rubin R, Bagrov N, Tymchenko B, Keren T, Zhilko A (2021) Eran-Deci: Super-Gradients. GitHub . https:\/\/zenodo.org\/records\/7789328"},{"key":"3711_CR22","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need. NeurIPS 30"},{"key":"3711_CR23","doi-asserted-by":"crossref","unstructured":"Li X, Wang W, Hu X, Li J, Tang J, Yang J (2021) Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (CVPR), pp. 11632\u201311641","DOI":"10.1109\/CVPR46437.2021.01146"},{"issue":"3","key":"3711_CR24","doi-asserted-by":"publisher","first-page":"271","DOI":"10.3390\/bioengineering12030271","volume":"12","author":"F Turgut","year":"2025","unstructured":"Turgut F, Ueda K, Saad A, Spitznagel T, Felten L, Matsumoto T, Santos R, Smet MD, Nagy ZZ, Becker MD et al (2025) Real-time integration of optical coherence tomography thickness map overlays for enhanced visualization in epiretinal membrane surgery: A pilot study. Bioengineering 12(3):271","journal-title":"Bioengineering"},{"key":"3711_CR25","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1007\/978-3-319-10602-1_48","volume-title":"Computer Vision - ECCV 2014","author":"T-Y Lin","year":"2014","unstructured":"Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Doll\u00e1r P, Zitnick CL (2014) Microsoft coco: common objects in context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer Vision - ECCV 2014. Springer, Cham, pp 740\u2013755"},{"key":"3711_CR26","unstructured":"Chaudhuri A, Dutta A, Bui T, Georgescu S (2025) A closer look at multimodal representation collapse. International conference on machine learning. pp 7555\u20137577 (PMLR)"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-026-03711-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-026-03711-2","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-026-03711-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T13:19:57Z","timestamp":1786108797000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-026-03711-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,21]]},"references-count":26,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,6]]}},"alternative-id":["3711"],"URL":"https:\/\/doi.org\/10.1007\/s11548-026-03711-2","relation":{},"ISSN":["1861-6429"],"issn-type":[{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,21]]},"assertion":[{"value":"15 January 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"M.S. and N.N. are shareholders of SynthesEyes GmbH, which provided the synthetic datasets.","order":1,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}}]}}