{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T00:26:54Z","timestamp":1780705614720,"version":"3.54.1"},"reference-count":32,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T00:00:00Z","timestamp":1664236800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Wyss Medical Foundation and Feldberg Chair for Spinal Research"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and time-series kinematic data. All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). This work is significant since it demonstrates that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. The study demonstrates an algorithm with state-of-the-art performance in surgical skill assessment. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.<\/jats:p>","DOI":"10.3390\/s22197328","type":"journal-article","created":{"date-parts":[[2022,9,28]],"date-time":"2022-09-28T03:30:37Z","timestamp":1664335837000},"page":"7328","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["Multi-Modal Deep Learning for Assessing Surgeon Technical Skill"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9934-6324","authenticated-orcid":false,"given":"Kevin","family":"Kasa","sequence":"first","affiliation":[{"name":"Orthopaedic Biomechanics Lab, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Burns","sequence":"additional","affiliation":[{"name":"Orthopaedic Biomechanics Lab, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"},{"name":"Institute of Biomedical Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada"},{"name":"Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, Toronto, ON M5S 1A1, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4601-5721","authenticated-orcid":false,"given":"Mitchell G.","family":"Goldenberg","sequence":"additional","affiliation":[{"name":"Division of Urology, Department of Surgery, University of Toronto, Toronto, ON M5S 1A1, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omar","family":"Selim","sequence":"additional","affiliation":[{"name":"Department of Surgery, Royal Victoria Regional Health Center, Barrie, ON L4M 6M2, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cari","family":"Whyne","sequence":"additional","affiliation":[{"name":"Orthopaedic Biomechanics Lab, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"},{"name":"Institute of Biomedical Engineering, University of Toronto, Toronto, ON M5S 1A1, Canada"},{"name":"Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, Toronto, ON M5S 1A1, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8941-3543","authenticated-orcid":false,"given":"Michael","family":"Hardisty","sequence":"additional","affiliation":[{"name":"Orthopaedic Biomechanics Lab, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"},{"name":"Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, Toronto, ON M5S 1A1, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2664","DOI":"10.1056\/NEJMra054785","article-title":"Teaching surgical skills\u2013changes in the wind","volume":"355","author":"Reznick","year":"2006","journal-title":"N. Engl. J. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.jsurg.2013.06.020","article-title":"Reflections on Competency-Based Education and Training for Surgical Residents","volume":"71","author":"Sonnadara","year":"2014","journal-title":"J. Surg. Educ."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e15443","DOI":"10.2196\/15443","article-title":"Implementation of the Operating Room Black Box Research Program at the Ottawa Hospital Through Patient, Clinical, and Organizational Engagement: Case Study","volume":"23","author":"Boet","year":"2021","journal-title":"J. Med. Internet Res."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Poursartip, B., LeBel, M.E., McCracken, L.C., Escoto, A., Patel, R.V., Naish, M.D., and Trejos, A.L. (2017). Energy-Based Metrics for Arthroscopic Skills Assessment. Sensors, 17.","DOI":"10.3390\/s17081808"},{"key":"ref_5","first-page":"159","article-title":"Deep neural networks for the assessment of surgical skills: A systematic review","volume":"19","author":"Yanik","year":"2021","journal-title":"J. Def. Model. Simul. Appl. Methodol. Technol."},{"key":"ref_6","unstructured":"Gao, Y., Vedula, S.S., Reiley, C.E., Ahmidi, N., Varadarajan, B., Lin, H.C., Tao, L., Zappella, L., B\u00e9jar, B., and Yuh, D.D. (2014, January 14\u201318). JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS): A Surgical Activity Dataset for Human Motion Modeling. Proceedings of the Modeling and Monitoring of Computer Assisted Interventions (M2CAI)\u2014MICCAI Workshop, Boston, MA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e1850","DOI":"10.1002\/rcs.1850","article-title":"Automated robot-assisted surgical skill evaluation: Predictive analytics approach","volume":"14","author":"Fard","year":"2018","journal-title":"Int. J. Med Robot. Comput. Assist. Surg."},{"key":"ref_8","unstructured":"Law, H., Ghani, K., and Deng, J. (2017, January 18\u201319). Surgeon Technical Skill Assessment Using Computer Vision Based Analysis. Proceedings of the 2nd Machine Learning for Healthcare Conference, Boston, MA, USA. Available online: https:\/\/proceedings.mlr.press\/v68\/law17a.html."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1163","DOI":"10.1097\/ACM.0000000000000316","article-title":"Use of a machine learning algorithm to classify expertise: Analysis of hand motion patterns during a simulated surgical task","volume":"89","author":"Watson","year":"2014","journal-title":"Acad. Med."},{"key":"ref_10","first-page":"273","article-title":"Objective structured assessment of technical skill (OSATS) for surgical residents","volume":"84","author":"Martin","year":"1997","journal-title":"Br. J. Surg."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"e201664","DOI":"10.1001\/jamanetworkopen.2020.1664","article-title":"Evaluation of Deep Learning Models for Identifying Surgical Actions and Measuring Performance","volume":"3","author":"Khalid","year":"2020","journal-title":"JAMA Netw. Open"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1007\/s11548-018-1704-z","article-title":"Video and accelerometer-based motion analysis for automated surgical skills assessment","volume":"13","author":"Aneeq","year":"2018","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1611","DOI":"10.1007\/s11548-019-02039-4","article-title":"Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks","volume":"14","author":"Forestier","year":"2019","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_14","first-page":"315","article-title":"Object detection to compute performance metrics for skill assessment in central venous catheterization","volume":"Volume 11598","author":"Linte","year":"2021","journal-title":"SPIE 11598, Proceedings of the Medical Imaging 2021: Image-Guided Procedures, Robotic Interventions, and Modeling, Online, 15\u201319 February 2021"},{"key":"ref_15","first-page":"358","article-title":"Feasibility of object detection for skill assessment in central venous catheterization","volume":"Volume 12034","author":"Linte","year":"2022","journal-title":"SPIE 12034, Proceedings of the Medical Imaging 2022: Image-Guided Procedures, Robotic Interventions, and Modeling, San Diego, CA, USA, 20\u201323 February 2022"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"731","DOI":"10.1007\/s11548-018-1735-5","article-title":"Automated surgical skill assessment in RMIS training","volume":"13","author":"Zia","year":"2018","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-L\u00f3pez, C., and Fichtinger, G. (2018, January 16\u201320). Evaluating Surgical Skills from Kinematic Data Using Convolutional Neural Networks. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2018, Granada, Spain.","DOI":"10.1007\/978-3-030-00931-1"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ordonez, F.J., and Roggen, D. (2016). Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors, 16.","DOI":"10.3390\/s16010115"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"075007","DOI":"10.1088\/1361-6579\/aacfd9","article-title":"Shoulder Physiotherapy Exercise Recognition: Machine Learning the Inertial Signals from a Smartwatch","volume":"39","author":"Burns","year":"2018","journal-title":"Physiol. Meas."},{"key":"ref_20","unstructured":"Hammerla, N.Y., Halloran, S., and Ploetz, T. (2016). Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Rueda, F.M., Grzeszick, R., Fink, G.A., Feldhorst, S., and ten Hompel, M. (2018). Convolutional Neural Networks for Human Activity Recognition Using Body-Worn Sensors. Informatics, 5.","DOI":"10.3390\/informatics5020026"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1109\/JBHI.2019.2909688","article-title":"TSE-CNN: A Two-Stage End-to-End CNN for Human Activity Recognition","volume":"24","author":"Huang","year":"2020","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3403574","article-title":"Identifying Child Users via Touchscreen Interactions","volume":"16","author":"Cheng","year":"2020","journal-title":"ACM Trans. Sens. Netw. (TOSN)"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Seeland, M., and M\u00e4der, P. (2021). Multi-view classification with convolutional neural networks. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0245230"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Rosen, J., Hannaford, B., and Satava, R. (2011). The da Vinci Surgical System. Surgical Robotics, Springer.","DOI":"10.1007\/978-1-4419-1126-1"},{"key":"ref_26","first-page":"3238","article-title":"Seglearn: A Python Package for Learning Sequences and Time Series","volume":"19","author":"Burns","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Itzkovich, D., Sharon, Y., Jarc, A., Refaely, Y., and Nisky, I. (2019, January 20\u201324). Using Augmentation to Improve the Robustness to Rotation of Deep Learning Segmentation in Robotic-Assisted Surgical Data. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8793963"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_29","unstructured":"Varno, F., Soleimani, B.H., Saghayi, M., Di-Jorio, L., and Matwin, S. (2019). Efficient Neural Task Adaptation by Maximum Entropy Initialization. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research","volume":"15","author":"Koo","year":"2016","journal-title":"J. Chiropr. Med."},{"key":"ref_31","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16 \u00d7 16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1078","DOI":"10.1016\/j.surg.2014.04.052","article-title":"Assessment of surgery residents\u2019 operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): A prospective multicenter study","volume":"156","author":"Hopmans","year":"2014","journal-title":"Surgery"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/19\/7328\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:40:21Z","timestamp":1760143221000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/19\/7328"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,27]]},"references-count":32,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["s22197328"],"URL":"https:\/\/doi.org\/10.3390\/s22197328","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,27]]}}}