{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T01:35:39Z","timestamp":1778808939503,"version":"3.51.4"},"reference-count":44,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2023,5,5]],"date-time":"2023-05-05T00:00:00Z","timestamp":1683244800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Sugarcane Special Scientific Research Project of Guangxi University","award":["2022GZB008"],"award-info":[{"award-number":["2022GZB008"]}]},{"name":"Sugarcane Special Scientific Research Project of Guangxi University","award":["2021AC19324"],"award-info":[{"award-number":["2021AC19324"]}]},{"name":"Sugarcane Special Scientific Research Project of Guangxi University","award":["20-065-40S006"],"award-info":[{"award-number":["20-065-40S006"]}]},{"DOI":"10.13039\/501100018571","name":"Guangxi Science and Technology Base and Talent Special Project","doi-asserted-by":"publisher","award":["2022GZB008"],"award-info":[{"award-number":["2022GZB008"]}],"id":[{"id":"10.13039\/501100018571","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100018571","name":"Guangxi Science and Technology Base and Talent Special Project","doi-asserted-by":"publisher","award":["2021AC19324"],"award-info":[{"award-number":["2021AC19324"]}],"id":[{"id":"10.13039\/501100018571","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100018571","name":"Guangxi Science and Technology Base and Talent Special Project","doi-asserted-by":"publisher","award":["20-065-40S006"],"award-info":[{"award-number":["20-065-40S006"]}],"id":[{"id":"10.13039\/501100018571","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Guangxi Key Laboratory of Manufacturing System &amp; Advanced Manufacturing Technology","award":["2022GZB008"],"award-info":[{"award-number":["2022GZB008"]}]},{"name":"Guangxi Key Laboratory of Manufacturing System &amp; Advanced Manufacturing Technology","award":["2021AC19324"],"award-info":[{"award-number":["2021AC19324"]}]},{"name":"Guangxi Key Laboratory of Manufacturing System &amp; Advanced Manufacturing Technology","award":["20-065-40S006"],"award-info":[{"award-number":["20-065-40S006"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Surgical skill assessment can quantify the quality of the surgical operation via the motion state of the surgical instrument tip (SIT), which is considered one of the effective primary means by which to improve the accuracy of surgical operation. Traditional methods have displayed promising results in skill assessment. However, this success is predicated on the SIT sensors, making these approaches impractical when employing the minimally invasive surgical robot with such a tiny end size. To address the assessment issue regarding the operation quality of robot-assisted minimally invasive surgery (RAMIS), this paper proposes a new automatic framework for assessing surgical skills based on visual motion tracking and deep learning. The new method innovatively combines vision and kinematics. The kernel correlation filter (KCF) is introduced in order to obtain the key motion signals of the SIT and classify them by using the residual neural network (ResNet), realizing automated skill assessment in RAMIS. To verify its effectiveness and accuracy, the proposed method is applied to the public minimally invasive surgical robot dataset, the JIGSAWS. The results show that the method based on visual motion tracking technology and a deep neural network model can effectively and accurately assess the skill of robot-assisted surgery in near real-time. In a fairly short computational processing time of 3 to 5 s, the average accuracy of the assessment method is 92.04% and 84.80% in distinguishing two and three skill levels. This study makes an important contribution to the safe and high-quality development of RAMIS.<\/jats:p>","DOI":"10.3390\/s23094496","type":"journal-article","created":{"date-parts":[[2023,5,5]],"date-time":"2023-05-05T03:57:31Z","timestamp":1683259051000},"page":"4496","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["An Automated Skill Assessment Framework Based on Visual Motion Signals and a Deep Neural Network in Robot-Assisted Minimally Invasive Surgery"],"prefix":"10.3390","volume":"23","author":[{"given":"Mingzhang","family":"Pan","sequence":"first","affiliation":[{"name":"College of Mechanical Engineering, Guangxi University, Nanning 530004, China"},{"name":"State Key Laboratory for Conservation and Utilization of Subtropical Agro-Bioresources, Nanning 530004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuo","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Mechanical Engineering, Guangxi University, Nanning 530004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingao","family":"Li","sequence":"additional","affiliation":[{"name":"College of Mechanical Engineering, Guangxi University, Nanning 530004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing","family":"Li","sequence":"additional","affiliation":[{"name":"College of Mechanical Engineering, Guangxi University, Nanning 530004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiuze","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Mechanical Engineering, Guangxi University, Nanning 530004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3785-4806","authenticated-orcid":false,"given":"Ke","family":"Liang","sequence":"additional","affiliation":[{"name":"College of Mechanical Engineering, Guangxi University, Nanning 530004, China"},{"name":"Guangxi Key Laboratory of Manufacturing System & Advanced Manufacturing Technology, School of Mechanical Engineering, Guangxi University, Nanning 530004, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,5,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1308\/rcsann.supp1.5","article-title":"A short history of robotic surgery","volume":"100","author":"Lane","year":"2018","journal-title":"Ann. R. Coll. Surg. Engl."},{"key":"ref_2","first-page":"68","article-title":"A DVRK-based Framework for Surgical","volume":"16","author":"Nagy","year":"2019","journal-title":"Acta Polytech. Hung."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2664","DOI":"10.1056\/NEJMra054785","article-title":"Teaching surgical skills--changes in the wind","volume":"355","author":"Reznick","year":"2006","journal-title":"N. Engl. J. Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"i34","DOI":"10.1136\/qshc.2009.038562","article-title":"Training and simulation for patient safety","volume":"19","author":"Aggarwal","year":"2010","journal-title":"Qual. Saf Health Care"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1434","DOI":"10.1056\/NEJMsa1300625","article-title":"Surgical skill and complication rates after bariatric surgery","volume":"369","author":"Birkmeyer","year":"2013","journal-title":"N. Eng. J. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1016\/j.surge.2014.07.001","article-title":"Assessment of surgical training","volume":"12","author":"Sanfey","year":"2014","journal-title":"Surgeon"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"ii64","DOI":"10.1136\/qhc.0100064","article-title":"Assessment of surgical competence","volume":"10","author":"Darzi","year":"2001","journal-title":"Qual. Health Care"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3130","DOI":"10.1007\/s00464-016-5337-4","article-title":"Analysis of motion in laparoscopy: The deconstruction of an intra-corporeal suturing task","volume":"31","author":"Farcas","year":"2017","journal-title":"Surg. Endosc."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1097\/DCR.0000000000000031","article-title":"Virtual reality simulator training for laparoscopic colectomy: What metrics have construct validity?","volume":"57","author":"Shanmugan","year":"2014","journal-title":"Dis. Colon Rectum"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"4399","DOI":"10.1007\/s00464-020-07940-7","article-title":"Motion analysis for better understanding of psychomotor skills in laparoscopy: Objective assessment-based simulation training using animal organs","volume":"35","author":"Ebina","year":"2021","journal-title":"Surg. Endosc."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.cmpb.2017.09.007","article-title":"Evaluation of robotic surgery skills using dynamic time warping","volume":"152","author":"Jiang","year":"2017","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1840","DOI":"10.1007\/s00464-017-5873-6","article-title":"Automatically rating trainee skill at a pediatric laparoscopic suturing task","volume":"32","author":"Oquendo","year":"2018","journal-title":"Surg. Endosc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1109\/THMS.2017.2776603","article-title":"Sensory-Glove-Based Open Surgery Skill Evaluation","volume":"48","author":"Sbernini","year":"2018","journal-title":"IEEE Trans. Hum. Mach. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"e2090","DOI":"10.1002\/rcs.2090","article-title":"Analysis of the video motion tracking system \u201cKinovea\u201d to assess surgical movements during robot-assisted radical prostatectomy","volume":"16","author":"Beulens","year":"2020","journal-title":"Int. J. Med. Robot."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2994","DOI":"10.1007\/s00464-018-6023-5","article-title":"A software-based tool for video motion tracking in the surgical skills assessment landscape","volume":"32","author":"Ganni","year":"2018","journal-title":"Surg. Endosc."},{"key":"ref_16","unstructured":"Rivas-Blanco, I., P\u2019erez-del-Pulgar, C.J., Mariani, A., Quaglia, C., Tortora, G., Menciassi, A., and Mu\u00f1oz, V.F. (2021). A surgical dataset from the da Vinci Research Kit for task automation and recognition. arXiv."},{"key":"ref_17","unstructured":"Gao, Y., Vedula, S.S., Reiley, C.E., Ahmidi, N., Varadarajan, B., Lin, H.C., Tao, L., Zappella, L., B\u00e9jar, B., and Yuh, D.D. (2022, September 22). JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS): A Surgical Activity Dataset for Human Motion Modeling. Available online: https:\/\/cirl.lcsr.jhu.edu\/wp-content\/uploads\/2015\/11\/JIGSAWS.pdf."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"4924","DOI":"10.1007\/s00464-019-07281-0","article-title":"Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach","volume":"34","author":"Kitaguchi","year":"2019","journal-title":"Surg. Endosc."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"105234","DOI":"10.1016\/j.cmpb.2019.105234","article-title":"Towards near real-time assessment of surgical skills: A comparison of feature extraction techniques","volume":"187","author":"Anh","year":"2020","journal-title":"Comput. Methods Programs Biomed"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1007\/s11548-021-02495-x","article-title":"SD-Net: Joint surgical gesture recognition and skill assessment","volume":"16","author":"Zhang","year":"2021","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cmpb.2019.05.008","article-title":"Surgical skill levels: Classification and analysis using deep neural network model and motion signals","volume":"177","author":"Nguyen","year":"2019","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1959","DOI":"10.1007\/s11548-018-1860-1","article-title":"Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery","volume":"13","author":"Wang","year":"2018","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1177\/15485129211034586","article-title":"Deep neural networks for the assessment of surgical skills: A systematic review","volume":"19","author":"Yanik","year":"2022","journal-title":"J. Def. Model. Simul."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lee, D., Yu, H.W., Kwon, H., Kong, H.-J., Lee, K.E., and Kim, H.C. (2020). Evaluation of Surgical Skills during Robotic Surgery by Deep Learning-Based Multiple Surgical Instrument Tracking in Training and Actual Operations. J. Clin. Med., 9.","DOI":"10.3390\/jcm9061964"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1217","DOI":"10.1007\/s11548-019-01995-1","article-title":"Video-based surgical skill assessment using 3D convolutional neural networks","volume":"14","author":"Funke","year":"2019","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ming, Y., Cheng, Y., Jing, Y., Liangzhe, L., Pengcheng, Y., Guang, Z., and Feng, C. (2021, January 22\u201324). Surgical skills assessment from robot assisted surgery video data. Proceedings of the 2021 IEEE International Conference on Power Electronics, Computer Applications (ICPECA), Shenyang, China.","DOI":"10.1109\/ICPECA51329.2021.9362525"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lajk\u00f3, G., Nagyn\u00e9 Elek, R., and Haidegger, T. (2021). Endoscopic Image-Based Skill Assessment in Robot-Assisted Minimally Invasive Surgery. Sensors, 21.","DOI":"10.3390\/s21165412"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1109\/TPAMI.2014.2345390","article-title":"High-Speed Tracking with Kernelized Correlation Filters","volume":"37","author":"Henriques","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zheng, K., Zhang, Z., and Qiu, C. (2022). A Fast Adaptive Multi-Scale Kernel Correlation Filter Tracker for Rigid Object. Sensors, 22.","DOI":"10.3390\/s22207812"},{"key":"ref_31","unstructured":"Rifkin, R., Yeo, G., and Poggio, T. (2003). Advances in Learning Theory: Methods, Model and Applications, NATO Science Series III: Computer and Systems Sciences, IOS Press."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014). Computer Vision\u2014ECCV 2014, Springer International Publishing.","DOI":"10.1007\/978-3-319-10599-4"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2017","DOI":"10.1007\/s11548-020-02259-z","article-title":"Motion analysis of the JHU-ISI Gesture and Skill Assessment Working Set using Robotics Video and Motion Assessment Software","volume":"15","author":"Lefor","year":"2020","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_34","unstructured":"(2022, September 29). da Vinci Surgical System, Intuitive Surgical, Inc.. Available online: https:\/\/www.davincisurgery.com\/."},{"key":"ref_35","first-page":"273","article-title":"Objective structured assessment of technical skill (OSATS) for surgical residents","volume":"84","author":"Martin","year":"2005","journal-title":"Br. J. Surg."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1097\/SLA.0000000000002478","article-title":"Modeling Surgical Technical Skill Using Expert Assessment for Automated Computer Rating","volume":"269","author":"Azari","year":"2019","journal-title":"Ann. Surg."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1400","DOI":"10.1016\/j.surg.2016.05.004","article-title":"A marker-less technique for measuring kinematics in the operating room","volume":"160","author":"Frasier","year":"2016","journal-title":"Surgery"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"e1845","DOI":"10.1002\/rcs.1845","article-title":"Motion control skill assessment based on kinematic analysis of robotic end-effector movements","volume":"14","author":"Liang","year":"2018","journal-title":"Int. J. Med. Robot."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2025","DOI":"10.1109\/TBME.2016.2647680","article-title":"A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery","volume":"64","author":"Ahmidi","year":"2017","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1002\/rcs.449","article-title":"Assessing system operation skills in robotic surgery trainees","volume":"8","author":"Kumar","year":"2012","journal-title":"Int. J. Med. Robot."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1146\/annurev-bioeng-071516-044435","article-title":"Objective Assessment of Surgical Technical Skill and Competency in the Operating Room","volume":"19","author":"Vedula","year":"2017","journal-title":"Annu. Rev. Biomed. Eng."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"101994","DOI":"10.1016\/j.media.2021.101994","article-title":"Detection, segmentation, and 3D pose estimation of surgical tools using convolutional neural networks and algebraic geometry","volume":"70","author":"Hasan","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1151","DOI":"10.1007\/s11548-017-1610-9","article-title":"The minimally acceptable classification criterion for surgical skill: Intent vectors and separability of raw motion data","volume":"12","author":"Dockter","year":"2017","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017, January 22\u201329). Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.97"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/9\/4496\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:29:35Z","timestamp":1760124575000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/9\/4496"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,5]]},"references-count":44,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["s23094496"],"URL":"https:\/\/doi.org\/10.3390\/s23094496","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,5]]}}}