{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T02:53:12Z","timestamp":1772247192017,"version":"3.50.1"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2020,10,10]],"date-time":"2020-10-10T00:00:00Z","timestamp":1602288000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,10,10]],"date-time":"2020-10-10T00:00:00Z","timestamp":1602288000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008678","name":"Universit\u00e4t Leipzig","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008678","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Purpose<\/jats:title>\n<jats:p>In the context of aviation and automotive navigation technology, assistance functions are associated with predictive planning and wayfinding tasks. In endoscopic minimally invasive surgery, however, assistance so far relies primarily on image-based localization and classification. We show that navigation workflows can be described and used for the prediction of navigation steps.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Methods<\/jats:title>\n<jats:p>A natural description vocabulary for observable anatomical landmarks in endoscopic images was defined to create 3850 navigation workflow sentences from 22 annotated functional endoscopic sinus surgery (FESS) recordings. Resulting FESS navigation workflows showed an imbalanced data distribution with over-represented landmarks in the ethmoidal sinus. A transformer model was trained to predict navigation sentences in sequence-to-sequence tasks. The training was performed with the Adam optimizer and label smoothing in a leave-one-out cross-validation study. The sentences were generated using an adapted beam search algorithm with exponential decay beam rescoring. The transformer model was compared to a standard encoder-decoder-model, as well as HMM and LSTM baseline models.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>The transformer model reached the highest prediction accuracy for navigation steps at 0.53, followed by 0.35 of the LSTM and 0.32 for the standard encoder-decoder-network. With an accuracy of sentence generation of 0.83, the prediction of navigation steps at sentence-level benefits from the additional semantic information. While standard class representation predictions suffer from an imbalanced data distribution, the attention mechanism also considered underrepresented classes reasonably well.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusion<\/jats:title>\n<jats:p>We implemented a natural language-based prediction method for sentence-level navigation steps in endoscopic surgery. The sentence-level prediction method showed a potential that word relations to navigation tasks can be learned and used for predicting future steps. Further studies are needed to investigate the functionality of path prediction. The prediction approach is a first step in the field of visuo-linguistic navigation assistance for endoscopic minimally invasive surgery.<\/jats:p>\n<\/jats:sec>","DOI":"10.1007\/s11548-020-02264-2","type":"journal-article","created":{"date-parts":[[2020,10,10]],"date-time":"2020-10-10T03:33:07Z","timestamp":1602300787000},"page":"2089-2100","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Language-based translation and prediction of surgical navigation steps for endoscopic wayfinding assistance in minimally invasive surgery"],"prefix":"10.1007","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0950-0637","authenticated-orcid":false,"given":"Richard","family":"Bieck","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katharina","family":"Heuermann","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Markus","family":"Pirlich","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juliane","family":"Neumann","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thomas","family":"Neumuth","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,10,10]]},"reference":[{"issue":"1","key":"2264_CR1","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1146\/annurev-bioeng-062117-120917","volume":"20","author":"X Luo","year":"2018","unstructured":"Luo X, Mori K, Peters TM (2018) Advanced endoscopic navigation: surgical big data, methodology, and applications. Annu Rev Biomed Eng 20(1):221\u2013251. https:\/\/doi.org\/10.1146\/annurev-bioeng-062117-120917","journal-title":"Annu Rev Biomed Eng"},{"key":"2264_CR2","doi-asserted-by":"crossref","unstructured":"Speidel S, Bodenstedt S, Vasconcelos F, Stoyanov D (2020) Interventional imaging: Vision. In: Handbook of Medical Image Computing and Computer Assisted Intervention, Elsevier, pp 721\u2013745","DOI":"10.1016\/B978-0-12-816176-0.00034-X"},{"key":"2264_CR3","doi-asserted-by":"publisher","unstructured":"Leonard S, Reiter A, Sinha A, Ishii M, Taylor RH, Hager GD (2016) Image-based navigation for functional endoscopic sinus surgery using structure from motion, presented at the SPIE Medical Imaging, San Diego, California, United States, p 97840\u00a0V, https:\/\/doi.org\/10.1117\/12.2217279","DOI":"10.1117\/12.2217279"},{"key":"2264_CR4","doi-asserted-by":"publisher","unstructured":"Sinha A, Liu X, Reiter A, Ishii M, Hager GD, Taylor RH (2018) Endoscopic navigation in the absence of CT imaging, ArXiv180603997 Cs Eess, vol. 11073, pp 64\u201371, https:\/\/doi.org\/10.1007\/978-3-030-00937-3_8","DOI":"10.1007\/978-3-030-00937-3_8"},{"key":"2264_CR5","doi-asserted-by":"publisher","first-page":"316","DOI":"10.1007\/978-3-319-10470-6_40","volume-title":"Medical image computing and computer-assisted intervention\u2014MICCAI 2014","author":"M Ye","year":"2014","unstructured":"Ye M, Johns E, Giannarou S, Yang G-Z (2014) Online Scene Association for Endoscopic Navigation. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention\u2014MICCAI 2014, vol 8674. Springer, Cham, pp 316\u2013323"},{"key":"2264_CR6","doi-asserted-by":"publisher","DOI":"10.1007\/s11548-020-02148-5","author":"Q He","year":"2020","unstructured":"He Q, Bano S, Ahmad OF, Yang B, Chen X, Valdastri P, Lovat LB, Stoyanov D, Zuo S (2020) Deep learning-based anatomical site classification for upper gastrointestinal endoscopy. Int J Comput Assist Radiol Surg. https:\/\/doi.org\/10.1007\/s11548-020-02148-5","journal-title":"Int J Comput Assist Radiol Surg"},{"issue":"12","key":"2264_CR7","doi-asserted-by":"publisher","first-page":"196","DOI":"10.1007\/s10916-017-0836-y","volume":"41","author":"J Ahmad","year":"2017","unstructured":"Ahmad J, Muhammad K, Lee MY, Baik SW (2017) Endoscopic image classification and retrieval using clustered convolutional features. J Med Syst 41(12):196. https:\/\/doi.org\/10.1007\/s10916-017-0836-y","journal-title":"J Med Syst"},{"issue":"2","key":"2264_CR8","doi-asserted-by":"publisher","first-page":"370","DOI":"10.2514\/1.26750","volume":"30","author":"JL Yepes","year":"2007","unstructured":"Yepes JL, Hwang I, Rotea M (2007) New algorithms for aircraft intent inference and trajectory prediction. J Guid Control Dyn 30(2):370\u2013382. https:\/\/doi.org\/10.2514\/1.26750","journal-title":"J Guid Control Dyn"},{"key":"2264_CR9","doi-asserted-by":"publisher","unstructured":"Froehlich J, Krumm J (2008) Route Prediction from Trip Observations, presented at the SAE World Congress & Exhibition, pp 2008-01\u20130201, https:\/\/doi.org\/10.4271\/2008-01-0201","DOI":"10.4271\/2008-01-0201"},{"issue":"3","key":"2264_CR10","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1007\/s11548-013-0940-5","volume":"9","author":"F Lalys","year":"2014","unstructured":"Lalys F, Jannin P (2014) Surgical process modelling: a review. Int J Comput Assist Radiol Surg 9(3):495\u2013511. https:\/\/doi.org\/10.1007\/s11548-013-0940-5","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"2264_CR11","doi-asserted-by":"publisher","DOI":"10.1002\/rcs.2136","author":"L Tanzi","year":"2020","unstructured":"Tanzi L, Piazzolla P, Vezzetti E (2020) Intraoperative surgery room management: a deep learning perspective. Int J Med Robot. https:\/\/doi.org\/10.1002\/rcs.2136","journal-title":"Int J Med Robot"},{"issue":"1","key":"2264_CR12","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1016\/j.jbi.2012.10.002","volume":"46","author":"S Franke","year":"2013","unstructured":"Franke S, Meixensberger J, Neumuth T (2013) Intervention time prediction from surgical low-level tasks. J Biomed Inform 46(1):152\u2013159. https:\/\/doi.org\/10.1016\/j.jbi.2012.10.002","journal-title":"J Biomed Inform"},{"issue":"6","key":"2264_CR13","doi-asserted-by":"publisher","first-page":"1089","DOI":"10.1007\/s11548-019-01966-6","volume":"14","author":"S Bodenstedt","year":"2019","unstructured":"Bodenstedt S, Wagner M, M\u00fcndermann L, Kenngott H, M\u00fcller-Stich B, Breucha M, Mees ST, Weitz J, Speidel S (2019) Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data. Int J Comput Assist Radiol Surg 14(6):1089\u20131095. https:\/\/doi.org\/10.1007\/s11548-019-01966-6","journal-title":"Int J Comput Assist Radiol Surg"},{"issue":"7","key":"2264_CR14","doi-asserted-by":"publisher","first-page":"1247","DOI":"10.1007\/s11548-019-02003-2","volume":"14","author":"F Fuentes-Hurtado","year":"2019","unstructured":"Fuentes-Hurtado F, Kadkhodamohammadi A, Flouty E, Barbarisi S, Luengo I, Stoyanov D (2019) EasyLabels: weak labels for scene segmentation in laparoscopic videos. Int J Comput Assist Radiol Surg 14(7):1247\u20131257. https:\/\/doi.org\/10.1007\/s11548-019-02003-2","journal-title":"Int J Comput Assist Radiol Surg"},{"issue":"7","key":"2264_CR15","doi-asserted-by":"publisher","first-page":"1217","DOI":"10.1007\/s11548-019-01995-1","volume":"14","author":"I Funke","year":"2019","unstructured":"Funke I, Mees ST, Weitz J, Speidel S (2019) Video-based surgical skill assessment using 3D convolutional neural networks. Int J Comput Assist Radiol Surg 14(7):1217\u20131225. https:\/\/doi.org\/10.1007\/s11548-019-01995-1","journal-title":"Int J Comput Assist Radiol Surg"},{"issue":"3","key":"2264_CR16","doi-asserted-by":"publisher","first-page":"632","DOI":"10.1016\/j.media.2010.10.001","volume":"16","author":"N Padoy","year":"2012","unstructured":"Padoy N, Blum T, Ahmadi S-A, Feussner H, Berger M-O, Navab N (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632\u2013641. https:\/\/doi.org\/10.1016\/j.media.2010.10.001","journal-title":"Med Image Anal"},{"issue":"9","key":"2264_CR17","doi-asserted-by":"publisher","first-page":"2522","DOI":"10.1109\/TKDE.2015.2416723","volume":"27","author":"A Bagnall","year":"2015","unstructured":"Bagnall A, Lines J, Hills J, Bostrom A (2015) Time-series classification with COTE: the collective of transformation-based ensembles. IEEE Trans Knowl Data Eng 27(9):2522\u20132535. https:\/\/doi.org\/10.1109\/TKDE.2015.2416723","journal-title":"IEEE Trans Knowl Data Eng"},{"issue":"4","key":"2264_CR18","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1007\/s11548-018-1882-8","volume":"14","author":"H Nakawala","year":"2019","unstructured":"Nakawala H, Bianchi R, Pescatori LE, De Cobelli O, Ferrigno G, De Momi E (2019) \u2018Deep-Onto\u2019 network for surgical workflow and context recognition. Int J Comput Assist Radiol Surg 14(4):685\u2013696. https:\/\/doi.org\/10.1007\/s11548-018-1882-8","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"2264_CR19","doi-asserted-by":"publisher","unstructured":"Kati\u0107 D, Wekerle A-L, G\u00e4rtner F, Kenngott H, M\u00fcller-Stich BP, Dillmann R, Speidel S (2013) Ontology-based prediction of surgical events in laparoscopic surgery, presented at the SPIE Medical Imaging, Lake Buena Vista (Orlando Area), Florida, USA, p 86711A, https:\/\/doi.org\/10.1117\/12.2007895","DOI":"10.1117\/12.2007895"},{"key":"2264_CR20","doi-asserted-by":"crossref","unstructured":"Anderson P, Wu Q, Teney D, Bruce J, Johnson M, S\u00fcnderhauf N, Reid I, Gould S, van den Hengel A (2018) Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments, ArXiv171107280 Cs","DOI":"10.1109\/CVPR.2018.00387"},{"key":"2264_CR21","doi-asserted-by":"crossref","unstructured":"Neumuth T, Strau\u00df G, Meixensberger J, Lemke HU, Burgert O (2006) \u201cAcquisition of Process Descriptions from Surgical Interventions,\u201d In: Database and Expert Systems Applications, vol. 4080, Bressan S, K\u00fcng J, Wagner R (Eds.) Berlin, Heidelberg: Springer Berlin Heidelberg, pp 602\u2013611","DOI":"10.1007\/11827405_59"},{"key":"2264_CR22","first-page":"5","volume":"2050","author":"S Siemoleit","year":"2017","unstructured":"Siemoleit S, Uciteli A, Bieck R, Herre H (2017) Ontological modelling of situational awareness in surgical interventions. JOWO 2050:5","journal-title":"JOWO"},{"key":"2264_CR23","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention Is All You Need,\u201d ArXiv170603762 Cs"},{"key":"2264_CR24","doi-asserted-by":"crossref","unstructured":"Klein G, Kim Y, Deng Y, Senellart J, Rush AM (Mar. 2017) \u201cOpenNMT: Open-Source Toolkit for Neural Machine Translation,\u201d ArXiv170102810 Cs","DOI":"10.18653\/v1\/P17-4012"},{"key":"2264_CR25","doi-asserted-by":"crossref","unstructured":"Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2015) \u201cRethinking the Inception Architecture for Computer Vision,\u201d ArXiv151200567 Cs","DOI":"10.1109\/CVPR.2016.308"},{"key":"2264_CR26","doi-asserted-by":"crossref","unstructured":"Wei J, Zou K (2019) \u201cEDA: easy data augmentation techniques for boosting performance on text classification tasks,\u201d ArXiv190111196 Cs","DOI":"10.18653\/v1\/D19-1670"},{"key":"2264_CR27","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S \u201cPyTorch: An Imperative Style, High-Performance Deep Learning Library,\u201d p 12"},{"key":"2264_CR28","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.21437\/Interspeech.2018-2416","volume":"2018","author":"I Williams","year":"2018","unstructured":"Williams I, Kannan A, Aleksic P, Rybach D, Sainath T (2018) Contextual speech recognition in end-to-end neural network systems using beam search. Interspeech 2018:2227\u20132231. https:\/\/doi.org\/10.21437\/Interspeech.2018-2416","journal-title":"Interspeech"},{"key":"2264_CR29","unstructured":"Franke S, Neumuth T (2015) \u201cAdaptive surgical process models for prediction of surgical work steps from surgical low-level activities,\u201d presented at the 6th Workshop on Modeling and Monitoring of Computer Assisted Interventions (M2CAI) at the 18th International Conference on Medical Image Computing and Computer Assisted Interventions (MICCAI), Munich, Germany"},{"key":"2264_CR30","doi-asserted-by":"publisher","unstructured":"Lin C-Y, Och FJ (2004) \u201cAutomatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL\u201904, Barcelona, Spain, pp 605-es, https:\/\/doi.org\/10.3115\/1218955.1219032","DOI":"10.3115\/1218955.1219032"},{"key":"2264_CR31","unstructured":"Bodenstedt S, Wagner M, Kati\u0107 D, Mietkowski P, Mayer B, Kenngott H, M\u00fcller-Stich B, Dillmann R, Speidel S (2017) Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis,\u201d ArXiv170203684 Cs"},{"key":"2264_CR32","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, ArXiv181004805 Cs"},{"key":"2264_CR33","unstructured":"Gowda T, May J (2020) Neural Machine Translation with Imbalanced Classes, ArXiv200402334 Cs Stat"},{"key":"2264_CR34","unstructured":"Chan W, Jaitly N, Le QV, Vinyals O (2015) Listen, Attend and Spell,\u201d ArXiv150801211 Cs Stat"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-020-02264-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-020-02264-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-020-02264-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,11]],"date-time":"2021-10-11T00:47:54Z","timestamp":1633913274000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-020-02264-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,10]]},"references-count":34,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["2264"],"URL":"https:\/\/doi.org\/10.1007\/s11548-020-02264-2","relation":{},"ISSN":["1861-6410","1861-6429"],"issn-type":[{"value":"1861-6410","type":"print"},{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,10]]},"assertion":[{"value":"13 January 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 September 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 October 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Informed consent was obtained from all individual participants from which recorded endoscopic video data were included in the study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}]}}