{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T01:41:22Z","timestamp":1792806082375,"version":"4.5.0"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T00:00:00Z","timestamp":1632096000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T00:00:00Z","timestamp":1632096000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003006","name":"ETH Zurich","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003006","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"published-print":{"date-parts":[[2021,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p><jats:italic>Purpose:<\/jats:italic> Virtual reality-based simulators have the potential to become an essential part of surgical education. To make full use of this potential, they must be able to automatically recognize activities performed by users and assess those. Since annotations of trajectories by human experts are expensive, there is a need for methods that can learn to recognize surgical activities in a data-efficient way. <jats:italic>Methods:<\/jats:italic> We use self-supervised training of deep encoder\u2013decoder architectures to learn representations of surgical trajectories from video data. These representations allow for semi-automatic extraction of features that capture information about semantically important events in the trajectories. Such features are processed as inputs of an unsupervised surgical activity recognition pipeline. <jats:italic>Results:<\/jats:italic> Our experiments document that the performance of hidden semi-Markov models used for recognizing activities in a simulated myomectomy scenario benefits from using features extracted from representations learned while training a deep encoder\u2013decoder network on the task of predicting the remaining surgery progress. <jats:italic>Conclusion:<\/jats:italic> Our work is an important first step in the direction of making efficient use of features obtained from deep representation learning for surgical activity recognition in settings where only a small fraction of the existing data is annotated by human domain experts and where those annotations are potentially incomplete.<\/jats:p>","DOI":"10.1007\/s11548-021-02493-z","type":"journal-article","created":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T09:14:54Z","timestamp":1632129294000},"page":"2037-2044","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Self-supervised representation learning for surgical activity recognition"],"prefix":"10.1007","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4995-7527","authenticated-orcid":false,"given":"Daniel","family":"Paysan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luis","family":"Haug","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Bajka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Markus","family":"Oelhafen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joachim M.","family":"Buhmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,9,20]]},"reference":[{"issue":"9","key":"2493_CR1","doi-asserted-by":"publisher","first-page":"2025","DOI":"10.1109\/TBME.2016.2647680","volume":"64","author":"N Ahmidi","year":"2017","unstructured":"Ahmidi N, Tao L, Sefati S, Gao Y, Lea C, Haro BB, Zappella L, Khudanpur S, Vidal R, Hager GD (2017) A dataset and benchmarks for segmentation and recognition of gestures in robotic surgery. IEEE Trans Biomed Eng 64(9):2025\u20132041","journal-title":"IEEE Trans Biomed Eng"},{"key":"2493_CR2","unstructured":"Arlot S, Celisse A, Harchaoui Z (2019). A kernel multiple change-point algorithm via model selection. Journal of Machine Learning Research 20(162), 1\u201356 http:\/\/jmlr.org\/papers\/v20\/16-155.html"},{"issue":"1","key":"2493_CR3","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1007\/s00464-009-0582-4","volume":"24","author":"M Bajka","year":"2010","unstructured":"Bajka M, Tuchschmid S, Fink D, Sz\u00e9kely G, Harders M (2010) Establishing construct validity of a virtual-reality training simulator for hysteroscopy via a multimetric scoring system. Surg Endoscopy 24(1):79","journal-title":"Surg Endoscopy"},{"issue":"7","key":"2493_CR4","doi-asserted-by":"publisher","first-page":"668","DOI":"10.1080\/0142159X.2018.1472754","volume":"40","author":"F Bjerrum","year":"2018","unstructured":"Bjerrum F, Thomsen ASS, Nayahangan LJ, Konge L (2018) Surgical simulation: current practices and future perspectives for technical skills training. Med Teacher 40(7):668\u2013675","journal-title":"Med Teacher"},{"issue":"1","key":"2493_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1687-5281-2014-1","volume":"2018","author":"Y Chen","year":"2018","unstructured":"Chen Y, Sun QL, Zhong K (2018) Semi-supervised spatio-temporal cnn for recognition of surgical workflow. EURASIP J Image Video Process 2018(1):1\u20139","journal-title":"EURASIP J Image Video Process"},{"key":"2493_CR6","unstructured":"Dauphin Y.N., de Vries H, Chung J, Bengio Y (2015) Rmsprop and equilibrated adaptive learning rates for non-convex optimization. CoRR abs\/1502.04390"},{"key":"2493_CR7","doi-asserted-by":"crossref","unstructured":"Deng J, Dong W, Socher R, Li L, Kai Li, Li Fei-Fei (2009) Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248\u2013255","DOI":"10.1109\/CVPR.2009.5206848"},{"issue":"11","key":"2493_CR8","doi-asserted-by":"publisher","first-page":"2005","DOI":"10.1007\/s11548-019-01953-x","volume":"14","author":"R DiPietro","year":"2019","unstructured":"DiPietro R, Ahmidi N, Malpani A, Waldram M, Lee GI, Lee MR, Vedula SS, Hager GD (2019) Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks. Int J Comput Assis Radiol Surg 14(11):2005\u20132020","journal-title":"Int J Comput Assis Radiol Surg"},{"key":"2493_CR9","doi-asserted-by":"crossref","unstructured":"DiPietro R, Hager G.D. (2018) Unsupervised learning for surgical motion by learning to predict the future. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 281\u2013288. Springer","DOI":"10.1007\/978-3-030-00937-3_33"},{"key":"2493_CR10","doi-asserted-by":"crossref","unstructured":"DiPietro R, Hager G.D. (2019) Automated surgical activity recognition with one labeled sequence. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 458\u2013466. Springer","DOI":"10.1007\/978-3-030-32254-0_51"},{"key":"2493_CR11","doi-asserted-by":"crossref","unstructured":"DiPietro R, Lea C, Malpani A, Ahmidi N, Vedula S.S., Lee G.I., Lee M.R., Hager G.D. (2016) Recognizing surgical activities with recurrent neural networks. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 551\u2013558. Springer","DOI":"10.1007\/978-3-319-46720-7_64"},{"key":"2493_CR12","doi-asserted-by":"crossref","unstructured":"Gong G, Wang X, Mu Y, Tian Q (2020) Learning temporal co-attention models for unsupervised video action localization. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 9819\u20139828","DOI":"10.1109\/CVPR42600.2020.00984"},{"issue":"500","key":"2493_CR13","doi-asserted-by":"publisher","first-page":"1590","DOI":"10.1080\/01621459.2012.737745","volume":"107","author":"R Killick","year":"2012","unstructured":"Killick R, Fearnhead P, Eckley IA (2012) Optimal detection of changepoints with a linear computational cost. J Am Stat Assoc 107(500):1590\u20131598","journal-title":"J Am Stat Assoc"},{"key":"2493_CR14","first-page":"8545","volume":"33","author":"D Kim","year":"2019","unstructured":"Kim D, Cho D, Kweon IS (2019) Self-supervised video representation learning with space-time cubic puzzles. Proc AAAI Conf Artific Intell 33:8545\u20138552","journal-title":"Proc AAAI Conf Artific Intell"},{"key":"2493_CR15","unstructured":"Kingma D.P., Ba J (2015) Adam: A method for stochastic optimization. In: Y.\u00a0Bengio, Y.\u00a0LeCun (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings"},{"key":"2493_CR16","doi-asserted-by":"crossref","unstructured":"Lu, A.X., Kraus, O.Z., Cooper, S., Moses, A.M.: Learning unsupervised feature representations for single cell microscopy images with paired cell inpainting. PLoS computational biology 15(9), e1007348 (2019)","DOI":"10.1371\/journal.pcbi.1007348"},{"key":"2493_CR17","doi-asserted-by":"crossref","unstructured":"Noroozi M, Favaro P (2016) Unsupervised learning of visual representations by solving jigsaw puzzles. In: European Conference on Computer Vision, pp. 69\u201384. Springer","DOI":"10.1007\/978-3-319-46466-4_5"},{"key":"2493_CR18","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan, G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019)Pytorch: An imperative style, high-performance deep learning library. In: H.\u00a0Wallach, H.\u00a0Larochelle, A.\u00a0Beygelzimer, F.\u00a0d\u2019Alch\u00e9 Buc, E.\u00a0Fox, R.\u00a0Garnett (eds.) Advances in Neural Information Processing Systems 32, pp. 8024\u20138035"},{"key":"2493_CR19","unstructured":"Srivastava N, Mansimov E, Salakhutdinov R (2015) Unsupervised learning of video representations using lstms. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, p. 843\u2013852"},{"issue":"4","key":"2493_CR20","doi-asserted-by":"publisher","first-page":"1069","DOI":"10.1109\/TMI.2018.2878055","volume":"38","author":"AP Twinanda","year":"2018","unstructured":"Twinanda AP, Yengera G, Mutter D, Marescaux J, Padoy N (2018) Rsdnet: learning to predict remaining surgery duration from laparoscopic videos without manual annotations. IEEE Trans Med Imag 38(4):1069\u20131078","journal-title":"IEEE Trans Med Imag"},{"key":"2493_CR21","unstructured":"Yengera G., Mutter D, Marescaux J, Padoy N (2018) Less is more: Surgical phase recognition with less annotations through self-supervised pre-training of CNN-LSTM networks. arXiv preprint arXiv:1805.08569"},{"key":"2493_CR22","unstructured":"Yu T, Mutter D, Marescaux J, Padoy N (2019) Learning from a tiny dataset of manual annotations: a teacher\/student approach for surgical phase recognition. In: International Conference on Information Processing in Computer-Assisted Interventions (IPCAI)"},{"key":"2493_CR23","doi-asserted-by":"crossref","unstructured":"Zhang R, Isola P, Efros A.A. (2017) Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058\u20131067","DOI":"10.1109\/CVPR.2017.76"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-021-02493-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-021-02493-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-021-02493-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,6]],"date-time":"2021-12-06T12:06:01Z","timestamp":1638792361000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-021-02493-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,20]]},"references-count":23,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2021,11]]}},"alternative-id":["2493"],"URL":"https:\/\/doi.org\/10.1007\/s11548-021-02493-z","relation":{},"ISSN":["1861-6410","1861-6429"],"issn-type":[{"value":"1861-6410","type":"print"},{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,20]]},"assertion":[{"value":"25 January 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 September 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 December 2021","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Electronic Supplemental Information is missing","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest\/Competing interest"}},{"value":"The data to reproduce the experiments can be made available upon request.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Availability of data and material"}},{"value":"The code required to reproduce the experiments can be made available by the authors upon request.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code availability"}}]}}