{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T08:04:38Z","timestamp":1791533078107,"version":"4.3.4"},"reference-count":62,"publisher":"Optica Publishing Group","issue":"3","license":[{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/doi.org\/10.1364\/OA_License_v2#VOR-OA"},{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/opg.optica.org\/policies\/opg-tdm-policy.json"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01-EY030490"],"award-info":[{"award-number":["R01-EY030490"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01-EY031769"],"award-info":[{"award-number":["R01-EY031769"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["T32-EB021937"],"award-info":[{"award-number":["T32-EB021937"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["opg.optica.org"],"crossmark-restriction":false},"short-container-title":["Biomed. Opt. Express"],"published-print":{"date-parts":[[2022,3,1]]},"abstract":"<jats:p>Intraoperative image-guidance provides enhanced feedback that facilitates surgical decision-making in a wide variety of medical fields and is especially useful when haptic feedback is limited. In these cases, automated instrument-tracking and localization are essential to guide surgical maneuvers and prevent damage to underlying tissue. However, instrument-tracking is challenging and often confounded by variations in the surgical environment, resulting in a trade-off between accuracy and speed. Ophthalmic microsurgery presents additional challenges due to the nonrigid relationship between instrument motion and instrument deformation inside the eye, image field distortion, image artifacts, and bulk motion due to patient movement and physiological tremor. We present an automated instrument-tracking method by leveraging multimodal imaging and deep-learning to dynamically detect surgical instrument positions and re-center imaging fields for 4D video-rate visualization of ophthalmic surgical maneuvers. We are able to achieve resolution-limited tracking accuracy at varying instrument orientations as well as at extreme instrument speeds and image defocus beyond typical use cases. As proof-of-concept, we perform automated instrument-tracking and 4D imaging of a mock surgical task. Here, we apply our methods for specific applications in ophthalmic microsurgery, but the proposed technologies are broadly applicable for intraoperative image-guidance with high speed and accuracy.<\/jats:p>","DOI":"10.1364\/boe.450814","type":"journal-article","created":{"date-parts":[[2022,2,10]],"date-time":"2022-02-10T07:30:26Z","timestamp":1644478226000},"page":"1471","update-policy":"https:\/\/doi.org\/10.1364\/crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Automated instrument-tracking for 4D video-rate imaging of ophthalmic surgical maneuvers"],"prefix":"10.1364","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1021-8914","authenticated-orcid":true,"given":"Eric M.","family":"Tang","sequence":"first","affiliation":[{"name":"Vanderbilt University, Department of Biomedical Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1344-3631","authenticated-orcid":true,"given":"Mohamed T.","family":"El-Haddad","sequence":"additional","affiliation":[{"name":"Vanderbilt University, Department of Biomedical Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shriji N.","family":"Patel","sequence":"additional","affiliation":[{"name":"Vanderbilt University Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuankai K.","family":"Tao","sequence":"additional","affiliation":[{"name":"Vanderbilt University, Department of Biomedical Engineering"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"285","published-online":{"date-parts":[[2022,2,15]]},"reference":[{"key":"boe-13-3-1471-R1","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1109\/42.491414","volume":"15","author":"Peters","year":"1996","journal-title":"IEEE Trans. Med. Imaging"},{"key":"boe-13-3-1471-R2","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1148\/radiology.211.2.r99ma26477","volume":"211","author":"Schwartz","year":"1999","journal-title":"Radiology"},{"key":"boe-13-3-1471-R3","doi-asserted-by":"publisher","first-page":"787","DOI":"10.1118\/1.598942","volume":"27","author":"Comeau","year":"2000","journal-title":"Med. Phys."},{"key":"boe-13-3-1471-R4","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1002\/ana.21801","volume":"66","author":"Martin","year":"2009","journal-title":"Ann. Neurol."},{"key":"boe-13-3-1471-R5","doi-asserted-by":"publisher","first-page":"1457","DOI":"10.1097\/IAE.0b013e3181b266bc","volume":"29","author":"Dayani","year":"2009","journal-title":"Retina"},{"key":"boe-13-3-1471-R6","doi-asserted-by":"publisher","first-page":"3315","DOI":"10.1364\/OL.35.003315","volume":"35","author":"Tao","year":"2010","journal-title":"Opt. Lett."},{"key":"boe-13-3-1471-R7","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1097\/ICU.0000000000000044","volume":"25","author":"Ehlers","year":"2014","journal-title":"Curr. Opin. Ophthalmol."},{"key":"boe-13-3-1471-R8","doi-asserted-by":"publisher","first-page":"844","DOI":"10.1007\/s11605-007-0090-6","volume":"11","author":"Cash","year":"2007","journal-title":"J. Gastrointest. Surg."},{"key":"boe-13-3-1471-R9","doi-asserted-by":"publisher","first-page":"1763","DOI":"10.1007\/s00268-008-9620-y","volume":"32","author":"Aoki","year":"2008","journal-title":"World J. Surg."},{"key":"boe-13-3-1471-R10","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1002\/rcs.360","volume":"7","author":"Peterhans","year":"2011","journal-title":"Int. J. Med. Robot. Comput. Assist. Surg."},{"key":"boe-13-3-1471-R11","doi-asserted-by":"publisher","first-page":"938","DOI":"10.1200\/JCO.2006.09.9515","volume":"25","author":"Dawson","year":"2007","journal-title":"J. Clin. Oncol."},{"key":"boe-13-3-1471-R12","doi-asserted-by":"publisher","first-page":"1180","DOI":"10.1007\/s00464-008-0298-x","volume":"23","author":"Van Der Meijden","year":"2009","journal-title":"Surg. Endosc."},{"key":"boe-13-3-1471-R13","doi-asserted-by":"publisher","first-page":"633","DOI":"10.1016\/j.media.2016.09.003","volume":"35","author":"Bouget","year":"2017","journal-title":"Med. Image Anal."},{"key":"boe-13-3-1471-R14","first-page":"664","article-title":"Concurrent segmentation and localization for tracking of surgical instruments","volume-title":"","author":"Laina","year":"2017"},{"key":"boe-13-3-1471-R15","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1109\/TMI.2004.825614","volume":"23","author":"West","year":"2004","journal-title":"IEEE Trans. Med. Imaging"},{"key":"boe-13-3-1471-R16","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.cobme.2017.09.007","volume":"3","author":"El-Haddad","year":"2017","journal-title":"Curr. Opin. Biomed. Eng."},{"key":"boe-13-3-1471-R17","doi-asserted-by":"publisher","first-page":"2212","DOI":"10.1016\/j.ophtha.2011.04.012","volume":"118","author":"Ray","year":"2011","journal-title":"Ophthalmology"},{"key":"boe-13-3-1471-R18","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.1364\/BOE.5.001877","volume":"5","author":"Tao","year":"2014","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R20","doi-asserted-by":"publisher","first-page":"1014","DOI":"10.1016\/j.ophtha.2017.12.037","volume":"125","author":"Ehlers","year":"2018","journal-title":"Ophthalmology"},{"key":"boe-13-3-1471-R22","doi-asserted-by":"publisher","first-page":"OCT146","DOI":"10.1167\/iovs.15-18818","volume":"57","author":"Todorich","year":"2016","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"boe-13-3-1471-R23","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1136\/bjophthalmol-2020-316045","volume":"105","author":"Yee","year":"2021","journal-title":"Br. J. Ophthalmol."},{"key":"boe-13-3-1471-R24","doi-asserted-by":"publisher","first-page":"1607","DOI":"10.1364\/BOE.8.001607","volume":"8","author":"Carrasco-Zevallos","year":"2017","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R25","doi-asserted-by":"publisher","first-page":"1329","DOI":"10.1136\/bjophthalmol-2014-305294","volume":"98","author":"Ehlers","year":"2014","journal-title":"Br. J. Ophthalmol."},{"key":"boe-13-3-1471-R26","doi-asserted-by":"publisher","first-page":"216","DOI":"10.3928\/23258160-20170301-04","volume":"48","author":"Runkle","year":"2017","journal-title":"Ophthalmic Surg. Lasers Imaging Retin."},{"key":"boe-13-3-1471-R27","doi-asserted-by":"publisher","first-page":"OCT37","DOI":"10.1167\/iovs.16-19277","volume":"57","author":"Carrasco-Zevallos","year":"2016","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"boe-13-3-1471-R28","doi-asserted-by":"publisher","first-page":"31689","DOI":"10.1038\/srep31689","volume":"6","author":"Carrasco-Zevallos","year":"2016","journal-title":"Sci. Rep."},{"key":"boe-13-3-1471-R29","doi-asserted-by":"publisher","first-page":"1815","DOI":"10.1364\/BOE.7.001815","volume":"7","author":"Viehland","year":"2016","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R30","doi-asserted-by":"publisher","first-page":"5052","DOI":"10.1364\/BOE.9.005052","volume":"9","author":"Carrasco-Zevallos","year":"2018","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R31","doi-asserted-by":"publisher","first-page":"e0213144","DOI":"10.1371\/journals.phone.0213144","volume":"14","author":"Kolb","year":"2019","journal-title":"PLoS One"},{"key":"boe-13-3-1471-R32","doi-asserted-by":"publisher","first-page":"1890","DOI":"10.1364\/BOE.4.001890","volume":"4","author":"Klein","year":"2013","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1167\/tvst.4.4.1","volume":"4","author":"Hahn","year":"2015","journal-title":"Transl. Vis. Sci. Technol."},{"key":"boe-13-3-1471-R34","doi-asserted-by":"publisher","first-page":"999","DOI":"10.1016\/j.ajo.2014.07.034","volume":"158","author":"Ehlers","year":"2014","journal-title":"Am. J. Ophthalmol."},{"key":"boe-13-3-1471-R35","doi-asserted-by":"publisher","first-page":"2716","DOI":"10.1364\/BOE.9.002716","volume":"9","author":"Keller","year":"2018","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R36","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1016\/j.media.2018.03.002","volume":"46","author":"Gessert","year":"2018","journal-title":"Med. Image Anal."},{"key":"boe-13-3-1471-R37","doi-asserted-by":"publisher","first-page":"787","DOI":"10.1007\/s11548-018-1751-5","volume":"13","author":"Weiss","year":"2018","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"boe-13-3-1471-R38","doi-asserted-by":"publisher","first-page":"63113","DOI":"10.1109\/ACCESS.2019.2912327","volume":"7","author":"Zhou","year":"2019","journal-title":"IEEE Access"},{"key":"boe-13-3-1471-R39","doi-asserted-by":"publisher","first-page":"3014","DOI":"10.1364\/BOE.6.003014","volume":"6","author":"El-Haddad","year":"2015","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R40","doi-asserted-by":"publisher","first-page":"332","DOI":"10.1007\/s10439-011-0441-z","volume":"40","author":"Stoyanov","year":"2012","journal-title":"Ann. Biomed. Eng."},{"key":"boe-13-3-1471-R42","doi-asserted-by":"publisher","first-page":"065005","DOI":"10.1117\/1.JBO.22.6.065005","volume":"22","author":"Zhou","year":"2017","journal-title":"J. Biomed. Opt."},{"key":"boe-13-3-1471-R43","first-page":"1173","article-title":"Automatic detection of instruments in laparoscopic images: A first step towards high-level command of robotic endoscopic holders","volume-title":"","volume":"26","author":"Voros","year":"2007"},{"key":"boe-13-3-1471-R44","first-page":"72610A","article-title":"Automatic classification of minimally invasive instruments based on endoscopic image sequences","volume-title":"","author":"Speidel","year":"2009"},{"key":"boe-13-3-1471-R45","first-page":"2919","article-title":"Full state visual forceps tracking under a microscope using projective contour models","volume-title":"","author":"Baek","year":"2012"},{"key":"boe-13-3-1471-R46","first-page":"568","article-title":"Data-driven visual tracking in retinal microsurgery","volume-title":"","author":"Sznitman","year":"2012"},{"key":"boe-13-3-1471-R47","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1111\/j.1525-1594.2012.01543.x","volume":"37","author":"Ryu","year":"2013","journal-title":"Artif. Organs"},{"key":"boe-13-3-1471-R48","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1080\/24699322.2017.1378777","volume":"22","author":"Zhao","year":"2017","journal-title":"Comput. Assist. Surg."},{"key":"boe-13-3-1471-R49","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1049\/htl.2019.0068","volume":"6","author":"Qiu","year":"2019","journal-title":"Healthcare Technology Letters"},{"key":"boe-13-3-1471-R50","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1080\/24699322.2018.1560097","volume":"24","author":"Zhao","year":"2019","journal-title":"Comput. Assist. Surg."},{"key":"boe-13-3-1471-R51","doi-asserted-by":"publisher","first-page":"e201700268","DOI":"10.1002\/jbio.201700268","volume":"11","author":"El-Haddad","year":"2018","journal-title":"J. Biophotonics"},{"key":"boe-13-3-1471-R52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1117\/1.NPh.6.4.041102","volume":"6","author":"Malone","year":"2019","journal-title":"Neurophotonics"},{"key":"boe-13-3-1471-R53","first-page":"57","article-title":"Deep-learning based automated instrument tracking and adaptive-sampling of intraoperative OCT for video-rate volumetric imaging of ophthalmic surgical maneuvers","volume-title":", p.","volume":"10867","author":"El-Haddad","year":"2019"},{"key":"boe-13-3-1471-R55","first-page":"12993","article-title":"Distance-IoU loss: Faster and better learning for bounding box regression","volume-title":"","author":"Zheng","year":"2020"},{"key":"boe-13-3-1471-R56","doi-asserted-by":"publisher","first-page":"6701","DOI":"10.1364\/BOE.430586","volume":"12","author":"Tang","year":"2021","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R57","first-page":"244","article-title":"Motorized force-sensing micro-forceps with tremor cancelling and controlled micro-vibrations for easier membrane peeling","volume-title":"","author":"Gonenc","year":"2014"},{"key":"boe-13-3-1471-R58","first-page":"698","article-title":"Force-sensing microneedle for assisted retinal vein cannulation","volume-title":"","author":"Gonenc","year":"2014"},{"key":"boe-13-3-1471-R59","doi-asserted-by":"publisher","first-page":"1086","DOI":"10.1097\/ICO.0000000000002338","volume":"39","author":"Borroni","year":"2020","journal-title":"Cornea"},{"key":"boe-13-3-1471-R60","doi-asserted-by":"publisher","first-page":"1323","DOI":"10.1016\/j.mri.2012.05.001","volume":"30","author":"Fedorov","year":"2012","journal-title":"Magn. Reson. Imaging"},{"key":"boe-13-3-1471-R61","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"boe-13-3-1471-R62","doi-asserted-by":"publisher","first-page":"1285","DOI":"10.1109\/TMI.2016.2528162","volume":"35","author":"Shin","year":"2016","journal-title":"IEEE Trans. Med. Imaging"},{"key":"boe-13-3-1471-R63","first-page":"100530I","article-title":"Image-guided feedback for ophthalmic microsurgery using multimodal intraoperative swept-source spectrally encoded scanning laser ophthalmoscopy and optical coherence tomography","volume-title":"","author":"Li","year":"2017"},{"key":"boe-13-3-1471-R64","doi-asserted-by":"crossref","DOI":"10.1117\/12.2583849","article-title":"Intraoperative spectrally encoded coherence tomography and reflectometry (iSECTR) for ophthalmic surgical guidance","volume-title":"","author":"Ringel","year":"2021"},{"key":"boe-13-3-1471-R65","doi-asserted-by":"publisher","first-page":"764","DOI":"10.1364\/BOE.2.000764","volume":"2","author":"Zhang","year":"2011","journal-title":"Biomed. Opt. Express"},{"key":"boe-13-3-1471-R66","doi-asserted-by":"publisher","first-page":"14685","DOI":"10.1364\/OE.18.014685","volume":"18","author":"Wieser","year":"2010","journal-title":"Opt. Express"}],"container-title":["Biomedical Optics Express"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/opg.optica.org\/viewmedia.cfm?URI=boe-13-3-1471&seq=0","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T19:30:47Z","timestamp":1722972647000},"score":1,"resource":{"primary":{"URL":"https:\/\/opg.optica.org\/abstract.cfm?URI=boe-13-3-1471"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,15]]},"references-count":62,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022]]},"published-print":{"date-parts":[[2022]]}},"URL":"https:\/\/doi.org\/10.1364\/boe.450814","relation":{},"ISSN":["2156-7085","2156-7085"],"issn-type":[{"value":"2156-7085","type":"print"},{"value":"2156-7085","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,15]]},"assertion":[{"value":"Optica Publishing Group","name":"publisher","label":"This article is maintained by"},{"value":"https:\/\/doi.org\/10.1364\/BOE.450814","name":"articlelink","label":"Crossref DOI link to publisher maintained version"},{"value":"research-article","name":"content_type","label":"Article type"},{"value":"Screened by Similarity Check","name":"cross_check","label":"Similarity check"},{"value":"Yes","order":0,"name":"peer_reviewed","label":"Peer reviewed","group":{"name":"peer_review","label":"Peer review"}},{"value":"Single blind","order":1,"name":"review_process","label":"Review process","group":{"name":"peer_review","label":"Peer review"}},{"value":"13 December 2021","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"9 February 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"15 February 2022","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}},{"value":"\u00a9 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement","name":"copyright","label":"Copyright"}]}}