{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T00:43:56Z","timestamp":1778028236933,"version":"3.51.4"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2024,4,1]],"date-time":"2024-04-01T00:00:00Z","timestamp":1711929600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,1]],"date-time":"2024-04-01T00:00:00Z","timestamp":1711929600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"publisher","award":["JPMJCR20D5"],"award-info":[{"award-number":["JPMJCR20D5"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p><jats:bold>Purpose<\/jats:bold> Surgical workflow recognition is a challenging task that requires understanding multiple aspects of surgery, such as gestures, phases, and steps. However, most existing methods focus on single-task or single-modal models and rely on costly annotations for training. To address these limitations, we propose a novel semi-supervised learning approach that leverages multimodal data and self-supervision to create meaningful representations for various surgical tasks. <jats:bold>Methods<\/jats:bold> Our representation learning approach conducts two processes. In the first stage, time contrastive learning is used to learn spatiotemporal visual features from video data, without any labels. In the second stage, multimodal VAE fuses the visual features with kinematic data to obtain a shared representation, which is fed into recurrent neural networks for online recognition. <jats:bold>Results<\/jats:bold> Our method is evaluated on two datasets: JIGSAWS and MISAW. We confirmed that it achieved comparable or better performance in multi-granularity workflow recognition compared to fully supervised models specialized for each task. On the JIGSAWS Suturing dataset, we achieve a gesture recognition accuracy of 83.3%. In addition, our model is more efficient in annotation usage, as it can maintain high performance with only half of the labels. On the MISAW dataset, we achieve 84.0% AD-Accuracy in phase recognition and 56.8% AD-Accuracy in step recognition. <jats:bold>Conclusion<\/jats:bold> Our multimodal representation exhibits versatility across various surgical tasks and enhances annotation efficiency. This work has significant implications for real-time decision-making systems within the operating room.<\/jats:p>","DOI":"10.1007\/s11548-024-03101-6","type":"journal-article","created":{"date-parts":[[2024,4,1]],"date-time":"2024-04-01T09:13:14Z","timestamp":1711962794000},"page":"1075-1083","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows"],"prefix":"10.1007","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7750-5683","authenticated-orcid":false,"given":"Yutaro","family":"Yamada","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8833-2215","authenticated-orcid":false,"given":"Jacinto","family":"Colan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2076-6842","authenticated-orcid":false,"given":"Ana","family":"Davila","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9917-098X","authenticated-orcid":false,"given":"Yasuhisa","family":"Hasegawa","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,4,1]]},"reference":[{"key":"3101_CR1","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1007\/s11548-016-1474-4","volume":"12","author":"M Maktabi","year":"2017","unstructured":"Maktabi M, Neumuth T (2017) Online time and resource management based on surgical workflow time series analysis. Int J Comput Assist Radiol Surg 12:325\u2013338","journal-title":"Int J Comput Assist Radiol Surg"},{"issue":"9","key":"3101_CR2","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1038\/s41551-017-0132-7","volume":"1","author":"L Maier-Hein","year":"2017","unstructured":"Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feussner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, M\u00e4rz K, Neumuth T, Padoy N, Pugh C, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1(9):691\u2013696. https:\/\/doi.org\/10.1038\/s41551-017-0132-7","journal-title":"Nat Biomed Eng"},{"key":"3101_CR3","doi-asserted-by":"publisher","unstructured":"Yamada Y, Colan J, Davila A, Hasegawa Y(2023) Task segmentation based on transition state clustering for surgical robot assistance. In: 2023 8th international conference on control and robotics engineering (ICCRE), pp 260\u2013264 . https:\/\/doi.org\/10.1109\/ICCRE57112.2023.10155581","DOI":"10.1109\/ICCRE57112.2023.10155581"},{"issue":"6","key":"3101_CR4","doi-asserted-by":"publisher","first-page":"2021","DOI":"10.1109\/TBME.2021.3054828","volume":"68","author":"B Amsterdam","year":"2021","unstructured":"Amsterdam B, Clarkson MJ, Stoyanov D (2021) Gesture recognition in robotic surgery: a review. IEEE Trans Biomed Eng 68(6):2021\u20132035. https:\/\/doi.org\/10.1109\/TBME.2021.3054828","journal-title":"IEEE Trans Biomed Eng"},{"issue":"5\u20136","key":"3101_CR5","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1080\/01691864.2022.2035253","volume":"36","author":"M Suzuki","year":"2022","unstructured":"Suzuki M, Matsuo Y (2022) A survey of multimodal deep generative models. Adv Robot 36(5\u20136):261\u2013278. https:\/\/doi.org\/10.1080\/01691864.2022.2035253","journal-title":"Adv Robot"},{"issue":"9","key":"3101_CR6","doi-asserted-by":"publisher","first-page":"2025","DOI":"10.1109\/TBME.2016.2647680","volume":"64","author":"N Ahmidi","year":"2017","unstructured":"Ahmidi N, Tao L, Sefati S, Gao Y, Lea C, Haro BB, Zappella L, Khudanpur S, Vidal R, Hager GD (2017) A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Trans Biomed Eng 64(9):2025\u20132041. https:\/\/doi.org\/10.1109\/TBME.2016.2647680","journal-title":"IEEE Trans Biomed Eng"},{"key":"3101_CR7","doi-asserted-by":"crossref","unstructured":"DiPietro R, Lea C, Malpani A, Ahmidi N, Vedula S.S, Lee G.I, Lee M.R, Hager G.D (2016) Recognizing surgical activities with recurrent neural networks. In: International conference on medical image computing and computer-assisted intervention, pp 551\u2013558. Springer","DOI":"10.1007\/978-3-319-46720-7_64"},{"key":"3101_CR8","doi-asserted-by":"crossref","unstructured":"Funke I, Bodenstedt S, Oehme F, Bechtolsheim F, Weitz J, Speidel S (2019) Using 3d convolutional neural networks to learn spatiotemporal features for automatic surgical gesture recognition in video. In: International conference on medical image computing and computer-assisted intervention, pp 467\u2013475 . Springer","DOI":"10.1007\/978-3-030-32254-0_52"},{"key":"3101_CR9","doi-asserted-by":"crossref","unstructured":"Qin Y, Pedram S.A, Feyzabadi S, Allan M, McLeod A.J, Burdick J.W, Azizian M (2020) Temporal segmentation of surgical sub-tasks through deep learning with multiple data sources. In: Proceeding of IEEE international conference on robotics and automation (ICRA), pp 371\u2013377. IEEE","DOI":"10.1109\/ICRA40945.2020.9196560"},{"key":"3101_CR10","doi-asserted-by":"crossref","unstructured":"Long Y, Wu J.Y, Lu B, Jin Y, Unberath M, Liu Y.-H, Heng P.A, Dou Q (2021) Relational graph learning on visual and kinematics embeddings for accurate gesture recognition in robotic surgery. In: Proceedings of IEEE international conference on robotics and automation (ICRA), pp 13346\u201313353. IEEE","DOI":"10.1109\/ICRA48506.2021.9561028"},{"issue":"7","key":"3101_CR11","doi-asserted-by":"publisher","first-page":"1677","DOI":"10.1109\/TMI.2022.3147640","volume":"41","author":"B Van Amsterdam","year":"2022","unstructured":"Van Amsterdam B, Funke I, Edwards E, Speidel S, Collins J, Sridhar A, Kelly J, Clarkson MJ, Stoyanov D (2022) Gesture recognition in robotic surgery with multimodal attention. IEEE Trans Med Imaging 41(7):1677\u20131687. https:\/\/doi.org\/10.1109\/TMI.2022.3147640","journal-title":"IEEE Trans Med Imaging"},{"key":"3101_CR12","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102158","volume":"73","author":"X Shi","year":"2021","unstructured":"Shi X, Jin Y, Dou Q, Heng P-A (2021) Semi-supervised learning with progressive unlabeled data excavation for label-efficient surgical workflow recognition. Med Image Anal 73:102158. https:\/\/doi.org\/10.1016\/j.media.2021.102158","journal-title":"Med Image Anal"},{"key":"3101_CR13","doi-asserted-by":"crossref","unstructured":"Tanwani AK, Sermanet P, Yan A, Anand R, Phielipp M, Goldberg K (2020) Motion2vec: semi-supervised representation learning from surgical videos. In: Proceedings of IEEE international conference on robotics and automation (ICRA), pp 1\u20138 . IEEE","DOI":"10.1109\/ICRA40945.2020.9197324"},{"key":"3101_CR14","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1007\/s11548-021-02343-y","volume":"16","author":"JY Wu","year":"2021","unstructured":"Wu JY, Tamhane A, Kazanzides P, Unberath M (2021) Cross-modal self-supervised representation learning for gesture and skill recognition in robotic surgery. Int J Comput Assisted Radiol Surg 16:779\u2013787. https:\/\/doi.org\/10.1007\/s11548-021-02343-y","journal-title":"Int J Comput Assisted Radiol Surg"},{"key":"3101_CR15","doi-asserted-by":"crossref","unstructured":"Yao T, Zhang Y, Qiu Z, Pan Y, Mei T (2021) Seco: exploring sequence supervision for unsupervised representation learning. In: Proceedings of the AAAI conference on artificial intelligence, vol 35, pp 10656\u201310664","DOI":"10.1609\/aaai.v35i12.17274"},{"key":"3101_CR16","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2022.103406","volume":"219","author":"I Dave","year":"2022","unstructured":"Dave I, Gupta R, Rizve MN, Shah M (2022) Tclr: temporal contrastive learning for video representation. Comput Vis Image Understand 219:103406. https:\/\/doi.org\/10.1016\/j.cviu.2022.103406","journal-title":"Comput Vis Image Understand"},{"key":"3101_CR17","doi-asserted-by":"crossref","unstructured":"Sermanet P, Lynch C, Chebotar Y, Hsu J, Jang E, Schaal S, Levine S, Brain G (2018) Time-contrastive networks: self-supervised learning from video. In: Proceedings of IEEE international conference on robotics and automation (ICRA), pp 1134\u20131141. IEEE","DOI":"10.1109\/ICRA.2018.8462891"},{"key":"3101_CR18","unstructured":"Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. In: International conference on machine learning, pp 1597\u20131607. PMLR"},{"key":"3101_CR19","doi-asserted-by":"publisher","unstructured":"Schroff F, Kalenichenko D, Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815\u2013823. https:\/\/doi.org\/10.1109\/CVPR.2015.7298682","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"3101_CR20","unstructured":"Wu M, Goodman N (2018) Multimodal generative models for scalable weakly-supervised learning. Adv Neural Inf Process Syst 31"},{"issue":"8","key":"3101_CR21","doi-asserted-by":"publisher","first-page":"1771","DOI":"10.1162\/089976602760128018","volume":"14","author":"GE Hinton","year":"2002","unstructured":"Hinton GE (2002) Training products of experts by minimizing contrastive divergence. Neural Comput 14(8):1771\u20131800","journal-title":"Neural Comput"},{"key":"3101_CR22","unstructured":"Higgins I, Matthey L, Pal A, Burgess C, Glorot X, Botvinick M, Mohamed S, Lerchner A (2017) beta-VAE: learning basic visual concepts with a constrained variational framework. In: International conference on learning representations"},{"key":"3101_CR23","unstructured":"Gao Y, Vedula S.S, Reiley C.E, Ahmidi N, Varadarajan B, Lin H.C, Tao L, Zappella L, B\u00e9jar B, Yuh D.D, et al (2014) Jhu-isi gesture and skill assessment working set (jigsaws): a surgical activity dataset for human motion modeling. In: MICCAI Workshop: M2cai, vol. 3"},{"key":"3101_CR24","doi-asserted-by":"publisher","unstructured":"Huaulm\u00e9 A, Sarikaya D, Le Mut K, Despinoy F, Long Y, Dou Q, Chng C-B, Lin W, Kondo S, Bravo-S\u00e1nchez L, Arbel\u00e1ez P, Reiter W, Mitsuishi M, Harada K, Jannin P (2021) Micro-surgical anastomose workflow recognition challenge report. Comput Methods Programs Biomed 212:106452. https:\/\/doi.org\/10.1016\/j.cmpb.2021.106452","DOI":"10.1016\/j.cmpb.2021.106452"},{"issue":"29","key":"3101_CR25","doi-asserted-by":"publisher","first-page":"861","DOI":"10.21105\/joss.00861","volume":"3","author":"L McInnes","year":"2018","unstructured":"McInnes L, Healy J, Saul N, Gro\u00dfberger L (2018) Umap: Uniform manifold approximation and projection. J Open Source Softw 3(29):861","journal-title":"J Open Source Softw"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-024-03101-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-024-03101-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-024-03101-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,14]],"date-time":"2024-06-14T11:23:59Z","timestamp":1718364239000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-024-03101-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,1]]},"references-count":25,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["3101"],"URL":"https:\/\/doi.org\/10.1007\/s11548-024-03101-6","relation":{},"ISSN":["1861-6429"],"issn-type":[{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,1]]},"assertion":[{"value":"29 February 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 March 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Conflict of interest The authors declare no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}}]}}