{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T16:10:03Z","timestamp":1790525403764,"version":"4.1.0"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T00:00:00Z","timestamp":1726531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T00:00:00Z","timestamp":1726531200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2321684"],"award-info":[{"award-number":["2321684"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100028897","name":"Wellcome Leap","doi-asserted-by":"publisher","award":["SAVE Program"],"award-info":[{"award-number":["SAVE Program"]}],"id":[{"id":"10.13039\/100028897","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100028897","name":"Wellcome Leap","doi-asserted-by":"publisher","award":["SAVE Program"],"award-info":[{"award-number":["SAVE Program"]}],"id":[{"id":"10.13039\/100028897","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Purpose<\/jats:title>\n            <jats:p>In order to produce a surgical gesture recognition system that can support a wide variety of procedures, either a very large annotated dataset must be acquired, or fitted models must generalize to new labels (so-called zero-shot capability). In this paper we investigate the feasibility of latter option.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Methods<\/jats:title>\n            <jats:p>Leveraging the bridge-prompt framework, we prompt-tune a pre-trained vision-text model (CLIP) for gesture recognition in surgical videos. This can utilize extensive outside video data such as text, but also make use of label meta-data and weakly supervised contrastive losses.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>Our experiments show that prompt-based video encoder outperforms standard encoders in surgical gesture recognition tasks. Notably, it displays strong performance in zero-shot scenarios, where gestures\/tasks that were not provided during the encoder training phase are included in the prediction phase. Additionally, we measure the benefit of inclusion text descriptions in the feature extractor training schema.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusion<\/jats:title>\n            <jats:p>Bridge-prompt and similar pre-trained\u00a0+\u00a0prompt-tuned video encoder models present significant visual representation for surgical robotics, especially in gesture recognition tasks. Given the diverse range of surgical tasks (gestures), the ability of these models to zero-shot transfer without the need for any task (gesture) specific retraining makes them invaluable.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1007\/s11548-024-03257-1","type":"journal-article","created":{"date-parts":[[2024,9,18]],"date-time":"2024-09-18T13:43:13Z","timestamp":1726666993000},"page":"311-321","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Zero-shot prompt-based video encoder for surgical gesture recognition"],"prefix":"10.1007","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-6582-4598","authenticated-orcid":false,"given":"Mingxing","family":"Rao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4526-4139","authenticated-orcid":false,"given":"Yinhong","family":"Qin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8495-5362","authenticated-orcid":false,"given":"Soheil","family":"Kolouri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7306-8140","authenticated-orcid":false,"given":"Jie Ying","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4428-5012","authenticated-orcid":false,"given":"Daniel","family":"Moyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,9,17]]},"reference":[{"issue":"6","key":"3257_CR1","doi-asserted-by":"publisher","first-page":"2021","DOI":"10.1109\/TBME.2021.3054828","volume":"68","author":"B Amsterdam","year":"2021","unstructured":"Amsterdam B, Clarkson MJ, Stoyanov D (2021) Gesture recognition in robotic surgery: a review. IEEE Trans Biomed Eng 68(6):2021\u20132035","journal-title":"IEEE Trans Biomed Eng"},{"key":"3257_CR2","doi-asserted-by":"crossref","unstructured":"Carreira J, Zisserman A (2017) Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6299\u20136308","DOI":"10.1109\/CVPR.2017.502"},{"key":"3257_CR3","doi-asserted-by":"crossref","unstructured":"Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 4489\u20134497","DOI":"10.1109\/ICCV.2015.510"},{"key":"3257_CR4","doi-asserted-by":"crossref","unstructured":"Hara K, Kataoka H, Satoh Y (2017) Learning spatio-temporal features with 3D residual networks for action recognition. In: Proceedings of the IEEE international conference on computer vision workshops, pp 3154\u20133160","DOI":"10.1109\/ICCVW.2017.373"},{"key":"3257_CR5","doi-asserted-by":"crossref","unstructured":"Zhang J, Nie Y, Lyu Y, Li H, Chang J, Yang X, Zhang JJ (2020) Symmetric dilated convolution for surgical gesture recognition. In: Medical image computing and computer assisted intervention\u2014MICCAI 2020: 23rd international conference, Lima, Peru, October 4\u20138, 2020, proceedings, Part III 23, pp 409\u2013418. Springer","DOI":"10.1007\/978-3-030-59716-0_39"},{"issue":"7","key":"3257_CR6","doi-asserted-by":"publisher","first-page":"1677","DOI":"10.1109\/TMI.2022.3147640","volume":"41","author":"B Van Amsterdam","year":"2022","unstructured":"Van Amsterdam B, Funke I, Edwards E, Speidel S, Collins J, Sridhar A, Kelly J, Clarkson MJ, Stoyanov D (2022) Gesture recognition in robotic surgery with multimodal attention. IEEE Trans Med Imaging 41(7):1677\u20131687","journal-title":"IEEE Trans Med Imaging"},{"key":"3257_CR7","doi-asserted-by":"publisher","first-page":"1675","DOI":"10.1007\/s11548-021-02495-x","volume":"16","author":"J Zhang","year":"2021","unstructured":"Zhang J, Nie Y, Lyu Y, Yang X, Chang J, Zhang JJ (2021) SD-Net: joint surgical gesture recognition and skill assessment. Int J Comput Assist Radiol Surg 16:1675\u20131682","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"3257_CR8","doi-asserted-by":"crossref","unstructured":"Goldbraikh A, Avisdris N, Pugh CM, Laufer S (2022) Bounded future MS-TCN++ for surgical gesture recognition. In: European conference on computer vision, pp 406\u2013421. Springer","DOI":"10.1007\/978-3-031-25066-8_22"},{"issue":"2","key":"3257_CR9","first-page":"1","volume":"10","author":"W Wang","year":"2019","unstructured":"Wang W, Zheng VW, Yu H, Miao C (2019) A survey of zero-shot learning: settings, methods, and applications. ACM Trans Intell Syst Technol TIST 10(2):1\u201337","journal-title":"ACM Trans Intell Syst Technol TIST"},{"key":"3257_CR10","doi-asserted-by":"crossref","unstructured":"Li M, Chen L, Duan Y, Hu Z, Feng J, Zhou J, Lu J (2022) Bridge-prompt: towards ordinal action understanding in instructional videos. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 19880\u201319889","DOI":"10.1109\/CVPR52688.2022.01926"},{"key":"3257_CR11","unstructured":"Gao Y, Vedula SS, Reiley CE, Ahmidi N, Varadarajan B, Lin HC, Tao L, Zappella L, B\u00e9jar B, Yuh DD et\u00a0al (2014) JHU-ISI gesture and skill assessment working set (JIGSAWS): a surgical activity dataset for human motion modeling. In: MICCAI workshop: M2cai, vol 3"},{"key":"3257_CR12","doi-asserted-by":"publisher","first-page":"6647","DOI":"10.1109\/TPAMI.2020.3021756","volume":"45","author":"S Li","year":"2020","unstructured":"Li S, Farha YA, Liu Y, Cheng M-M, Gall J (2020) MS-TCN++: multi-stage temporal convolutional network for action segmentation. IEEE Trans Pattern Anal Mach Intell 45:6647\u20136658","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"3257_CR13","doi-asserted-by":"crossref","unstructured":"DiPietro R, Lea C, Malpani A, Ahmidi N, Vedula SS, Lee GI, Lee MR, Hager GD (2016) Recognizing surgical activities with recurrent neural networks. In: Medical image computing and computer-assisted intervention\u2014MICCAI 2016: 19th international conference, Athens, Greece, October 17\u201321, 2016, proceedings, Part I 19. Springer, pp 551\u2013558","DOI":"10.1007\/978-3-319-46720-7_64"},{"key":"3257_CR14","doi-asserted-by":"crossref","unstructured":"Tao L, Zappella L, Hager GD, Vidal R (2013) Surgical gesture segmentation and recognition. In: Medical image computing and computer-assisted intervention\u2014MICCAI 2013: 16th international conference, Nagoya, Japan, September 22\u201326, 2013, proceedings, Part III 16. Springer, pp 339\u2013346","DOI":"10.1007\/978-3-642-40760-4_43"},{"key":"3257_CR15","unstructured":"Reiley CE, Lin HC, Varadarajan B, Vagvolgyi B, Khudanpur S, Yuh DD, Hager GD (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. In: MMVR, vol 132, pp 396\u2013401"},{"key":"3257_CR16","doi-asserted-by":"crossref","unstructured":"Funke I, Bodenstedt S, Oehme F, Bechtolsheim F, Weitz J, Speidel S (2019) Using 3D convolutional neural networks to learn spatiotemporal features for automatic surgical gesture recognition in video. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 467\u2013475","DOI":"10.1007\/978-3-030-32254-0_52"},{"key":"3257_CR17","doi-asserted-by":"crossref","unstructured":"Lea C, Reiter A, Vidal R, Hager GD (2016) Segmental spatiotemporal CNNs for fine-grained action segmentation. In: Computer vision\u2014ECCV 2016: 14th European conference, Amsterdam, The Netherlands, October 11\u201314, 2016, proceedings, Part III 14. Springer, pp 36\u201352","DOI":"10.1007\/978-3-319-46487-9_3"},{"key":"3257_CR18","doi-asserted-by":"crossref","unstructured":"Zappella L, B\u00e9jar B, Hager G, Vidal R (2013) Surgical gesture classification from video and kinematic data. Med Image Anal 17(7):732\u2013745","DOI":"10.1016\/j.media.2013.04.007"},{"key":"3257_CR19","doi-asserted-by":"crossref","unstructured":"Long Y, Wu JY, Lu B, Jin Y, Unberath M, Liu Y-H, Heng PA, Dou Q (2021) Relational graph learning on visual and kinematics embeddings for accurate gesture recognition in robotic surgery. In: 2021 IEEE international conference on robotics and automation (ICRA). IEEE, pp 13346\u201313353","DOI":"10.1109\/ICRA48506.2021.9561028"},{"key":"3257_CR20","doi-asserted-by":"crossref","unstructured":"Qin Y, Feyzabadi S, Allan M, Burdick JW, Azizian M (2020) davincinet: Joint prediction of motion and surgical state in robot-assisted surgery. In: 2020 IEEE\/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 2921\u20132928","DOI":"10.1109\/IROS45743.2020.9340723"},{"key":"3257_CR21","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1007\/s11548-021-02343-y","volume":"16","author":"JY Wu","year":"2021","unstructured":"Wu JY, Tamhane A, Kazanzides P, Unberath M (2021) Cross-modal self-supervised representation learning for gesture and skill recognition in robotic surgery. Int J Comput Assist Radiol Surg 16:779\u2013787","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"3257_CR22","doi-asserted-by":"crossref","unstructured":"Tao L, Elhamifar E, Khudanpur S, Hager GD, Vidal R (2012) Sparse hidden Markov models for surgical gesture classification and skill evaluation. In: Information processing in computer-assisted interventions: third international conference, IPCAI 2012, Pisa, Italy, June 27, 2012. Proceedings 3. Springer, pp 167\u2013177","DOI":"10.1007\/978-3-642-30618-1_17"},{"issue":"11","key":"3257_CR23","doi-asserted-by":"publisher","first-page":"2005","DOI":"10.1007\/s11548-019-01953-x","volume":"14","author":"R DiPietro","year":"2019","unstructured":"DiPietro R, Ahmidi N, Malpani A, Waldram M, Lee GI, Lee MR, Vedula SS, Hager GD (2019) Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks. Int J Comput Assist Radiol Surg 14(11):2005\u20132020","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"3257_CR24","doi-asserted-by":"crossref","unstructured":"Lea C, Flynn MD, Vidal R, Reiter A, Hager GD (2017) Temporal convolutional networks for action segmentation and detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 156\u2013165","DOI":"10.1109\/CVPR.2017.113"},{"key":"3257_CR25","doi-asserted-by":"crossref","unstructured":"Farha YA, Gall J (2019) MS-TCN: multi-stage temporal convolutional network for action segmentation. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 3575\u20133584","DOI":"10.1109\/CVPR.2019.00369"},{"key":"3257_CR26","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2022.102611","volume":"82","author":"K Yuan","year":"2022","unstructured":"Yuan K, Holden M, Gao S, Lee W (2022) Anticipation for surgical workflow through instrument interaction and recognized signals. Med Image Anal 82:102611","journal-title":"Med Image Anal"},{"key":"3257_CR27","doi-asserted-by":"crossref","unstructured":"Czempiel T, Paschali M, Keicher M, Simson W, Feussner H, Kim ST, Navab N (2020) Tecno: surgical phase recognition with multi-stage temporal convolutional networks. In: Medical image computing and computer assisted intervention\u2014MICCAI 2020: 23rd international conference, Lima, Peru, October 4\u20138, 2020, proceedings, part III 23. Springer, pp 343\u2013352","DOI":"10.1007\/978-3-030-59716-0_33"},{"key":"3257_CR28","unstructured":"Bengio Y, Courville AC, Vincent P (2012) Unsupervised feature learning and deep learning: a review and new perspectives. CoRR arXiv:1206.5538"},{"key":"3257_CR29","unstructured":"Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J et\u00a0al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748\u20138763"},{"key":"3257_CR30","unstructured":"Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. In: International conference on machine learning. PMLR, pp 1597\u20131607"},{"key":"3257_CR31","doi-asserted-by":"crossref","unstructured":"Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Doll\u00e1r P, Zitnick CL (2014) Microsoft coco: common objects in context. In: Computer vision\u2014ECCV 2014: 13th European conference, Zurich, Switzerland, September 6\u201312, 2014, proceedings, Part V 13. Springer, pp 740\u2013755","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"3257_CR32","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1007\/s11263-016-0981-7","volume":"123","author":"R Krishna","year":"2017","unstructured":"Krishna R, Zhu Y, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li L-J, Shamma DA et al (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. Int J Comput Vis 123:32\u201373","journal-title":"Int J Comput Vis"},{"key":"3257_CR33","doi-asserted-by":"crossref","unstructured":"Thomee B, Shamma DA, Friedland G, Elizalde B, Ni K, Poland D, Borth D, Li L-J (2016) Yfcc100m: The new data in multimedia research. Commun ACM 59(2):64\u201373","DOI":"10.1145\/2812802"},{"issue":"8","key":"3257_CR34","first-page":"9","volume":"1","author":"A Radford","year":"2019","unstructured":"Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9","journal-title":"OpenAI blog"},{"key":"3257_CR35","unstructured":"Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et\u00a0al (2021) An image is worth $$16 \\times 16$$ words: transformers for image recognition at scale. In: International conference on learning representations"},{"key":"3257_CR36","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770\u2013778","DOI":"10.1109\/CVPR.2016.90"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-024-03257-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-024-03257-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-024-03257-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,10]],"date-time":"2025-02-10T12:20:31Z","timestamp":1739190031000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-024-03257-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,17]]},"references-count":36,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,2]]}},"alternative-id":["3257"],"URL":"https:\/\/doi.org\/10.1007\/s11548-024-03257-1","relation":{},"ISSN":["1861-6429"],"issn-type":[{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,17]]},"assertion":[{"value":"9 June 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 August 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 September 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}