{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T04:22:48Z","timestamp":1792124568928,"version":"4.4.0"},"reference-count":38,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2022,12,19]],"date-time":"2022-12-19T00:00:00Z","timestamp":1671408000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Austrian Center for Medical Innovation and Technology","award":["2019-1.3.1-KK-2019-00007"],"award-info":[{"award-number":["2019-1.3.1-KK-2019-00007"]}]},{"DOI":"10.13039\/501100011019","name":"National Research, Development and Innovation Fund of Hungary","doi-asserted-by":"publisher","award":["2019-1.3.1-KK-2019-00007"],"award-info":[{"award-number":["2019-1.3.1-KK-2019-00007"]}],"id":[{"id":"10.13039\/501100011019","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: It is well understood that surgical skills largely define patient outcomes both in Minimally Invasive Surgery (MIS) and Robot-Assisted MIS (RAMIS). Non-technical surgical skills, including stress and distraction resilience, decision-making and situation awareness also contribute significantly. Autonomous, technologically supported objective skill assessment can be efficient tools to improve patient outcomes without the need to involve expert surgeon reviewers. However, autonomous non-technical skill assessments are unstandardized and open for more research. Recently, Surgical Data Science (SDS) has become able to improve the quality of interventional healthcare with big data and data processing techniques (capture, organization, analysis and modeling of data). SDS techniques can also help to achieve autonomous non-technical surgical skill assessments. Methods: An MIS training experiment is introduced to autonomously assess non-technical skills and to analyse the workload based on sensory data (video image and force) and a self-rating questionnaire (SURG-TLX). A sensorized surgical skill training phantom and adjacent training workflow were designed to simulate a complicated Laparoscopic Cholecystectomy task; the dissection of the cholecyst\u2019s peritonial layer and the safe clip application on the cystic artery in an uncomfortable environment. A total of 20 training sessions were recorded from 7 subjects (3 non-medicals, 2 residents, 1 expert surgeon and 1 expert MIS surgeon). Workload and learning curves were studied via SURG-TLX. For autonomous non-technical skill assessment, video image data with tracked instruments based on Channel and Spatial Reliability Tracker (CSRT) and force data were utilized. An autonomous time series classification was achieved by a Fully Convolutional Neural Network (FCN), where the class labels were provided by SURG-TLX. Results: With unpaired t-tests, significant differences were found between the two groups (medical professionals and control) in certain workload components (mental demands, physical demands, and situational stress, p&lt;0.0001, 95% confidence interval, p&lt;0.05 for task complexity). With paired t-tests, the learning curves of the trials were also studied; the task complexity resulted in a significant difference between the first and the second trials. Autonomous non-technical skill classification was based on the FCN by applying the tool trajectories and force data as input. This resulted in a high accuracy (85%) on temporal demands classification based on the z component of the used forces and 75% accuracy for classifying mental demands\/situational stress with the x component of the used forces validated with Leave One Out Cross-Validation. Conclusions: Non-technical skills and workload components can be classified autonomously based on measured training data. SDS can be effective via automated non-technical skill assessment.<\/jats:p>","DOI":"10.3390\/jcm11247533","type":"journal-article","created":{"date-parts":[[2022,12,19]],"date-time":"2022-12-19T08:41:41Z","timestamp":1671439301000},"page":"7533","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Next in Surgical Data Science: Autonomous Non-Technical Skill Assessment in Minimally Invasive Surgery Training"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3030-254X","authenticated-orcid":false,"given":"Ren\u00e1ta","family":"Nagyn\u00e9 Elek","sequence":"first","affiliation":[{"name":"Antal Bejczy Center for Intelligent Robotics, University Research and Innovation Center, \u00d3buda University, H-1034 Budapest, Hungary"},{"name":"Doctoral School of Applied Informatics and Applied Mathematics, \u00d3buda University, H-1034 Budapest, Hungary"},{"name":"John von Neumann Faculty of Informatics, \u00d3buda University, H-1034 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1402-1139","authenticated-orcid":false,"given":"Tam\u00e1s","family":"Haidegger","sequence":"additional","affiliation":[{"name":"Antal Bejczy Center for Intelligent Robotics, University Research and Innovation Center, \u00d3buda University, H-1034 Budapest, Hungary"},{"name":"Austrian Center for Medical Innovation and Technology, 2700 Wiener Neustadt, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"681","DOI":"10.1007\/s00464-010-1243-3","article-title":"Surgery in Space: The Future of Robotic Telesurgery","volume":"25","author":"Haidegger","year":"2011","journal-title":"Surg. Endosc."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1016\/j.juro.2018.06.078","article-title":"Objective Assessment of Robotic Surgical Technical Skill: A Systemic Review (accepted manuscript)","volume":"201","author":"Chen","year":"2018","journal-title":"J. Urol."},{"key":"ref_3","first-page":"141","article-title":"Robot-Assisted Minimally Invasive Surgical Skill Assessment\u2014Manual and Automated Platforms","volume":"16","author":"Haidegger","year":"2019","journal-title":"Acta Polytech. Hung."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1629","DOI":"10.1016\/j.jsurg.2019.06.011","article-title":"Automated methods of technical skill assessment in surgery: A systematic review","volume":"76","author":"Levin","year":"2019","journal-title":"J. Surg. Educ."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Nagy, D.A., Rudas, I.J., and Haidegger, T. (2017, January 24\u201325). Surgical data science, an emerging field of medicine. Proceedings of the 2017 IEEE 30th Neumann Colloquium, Budapest, Hungary.","DOI":"10.1109\/NC.2017.8263251"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1177\/155335060501200112","article-title":"Assessing Mental Workload during Laparoscopic Surgery","volume":"12","author":"Carswell","year":"2005","journal-title":"Surg. Innov."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1016\/j.jtcvs.2009.10.048","article-title":"Is the \u201cSterile Cockpit\u201d Concept Applicable to Cardiovascular Surgery Critical Intervals or Critical Events? The Impact of Protocol-Driven Communication during Cardiopulmonary Bypass","volume":"139","author":"Wadhera","year":"2010","journal-title":"J. Thorac. Cardiovasc. Surg."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Shafiei, S.B., Hussein, A.A., and Guru, K.A. (2018). Dynamic Changes of Brain Functional States during Surgical Skill Acquisition. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0204836"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1136\/leader-2018-000128","article-title":"Do Surgeon Non-Technical Skills Correlate with Teamwork-Related Outcomes during Robot-Assisted Surgery?","volume":"3","author":"Ahmed","year":"2019","journal-title":"BMJ Lead."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1016\/j.surg.2014.06.012","article-title":"The Impact of Heat Stress on Operative Performance and Cognitive Function during Simulated Laparoscopic Operative Tasks","volume":"157","author":"Berg","year":"2015","journal-title":"Surgery"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1007\/s00464-014-3668-6","article-title":"The Impact of Intra-Operative Interruptions on Surgeons\u2019 Perceived Workload: An Observational Study in Elective General and Orthopedic Surgery","volume":"29","author":"Weigl","year":"2015","journal-title":"Surg. Endosc."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1007\/s00464-013-3213-z","article-title":"Comparative Assessment of Physical and Cognitive Ergonomics Associated with Robotic and Traditional Laparoscopic Surgeries","volume":"28","author":"Lee","year":"2014","journal-title":"Surg. Endosc."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2553","DOI":"10.1007\/s00464-014-3967-y","article-title":"Surgeons\u2019 Display Reduced Mental Effort and Workload While Performing Robotically Assisted Surgical Tasks, When Compared to Conventional Laparoscopy","volume":"29","author":"Moore","year":"2015","journal-title":"Surg. Endosc."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1621","DOI":"10.1007\/s10439-018-2049-z","article-title":"Robotic Surgery Improves Technical Performance and Enhances Prefrontal Activation During High Temporal Demand","volume":"46","author":"Singh","year":"2018","journal-title":"Ann. Biomed. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.amjsurg.2007.08.070","article-title":"Reliability of a Revised NOTECHS Scale for Use in Surgical Teams","volume":"196","author":"Sevdalis","year":"2008","journal-title":"Am. J. Surg."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1016\/j.jamcollsurg.2010.11.001","article-title":"Observational Teamwork Assessment for Surgery: Content Validation and Tool Refinement","volume":"212","author":"Hull","year":"2011","journal-title":"J. Am. Coll. Surg."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.surg.2005.06.017","article-title":"Non-Technical Skills for Surgeons in the Operating Room: A Review of the Literature","volume":"139","author":"Yule","year":"2006","journal-title":"Surgery"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"5403","DOI":"10.1007\/s00464-017-5622-x","article-title":"Development and Validation of a Tool for Non-Technical Skills Evaluation in Robotic Surgery-the ICARS System","volume":"31","author":"Raison","year":"2017","journal-title":"Surg. Endosc."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"731","DOI":"10.1007\/s11548-018-1735-5","article-title":"Automated Surgical Skill Assessment in RMIS Training","volume":"13","author":"Zia","year":"2018","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Lajk\u00f3, G., Elek, R.N., and Haidegger, T. (2021, January 7\u20139). Surgical Skill Assessment Automation Based on Sparse Optical Flow Data. Proceedings of the IEEE International Conference on Intelligent Engineering Systems (INES), Budapest, Hungary.","DOI":"10.1109\/INES52918.2021.9512917"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ahn, J.W., Ku, Y., and Kim, H.C. (2019). A Novel Wearable EEG and ECG Recording System for Stress Assessment. Sensors, 19.","DOI":"10.3390\/s19091991"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"65","DOI":"10.12700\/APH.18.1.2021.1.5","article-title":"EEG-based Speech Activity Detection","volume":"18","year":"2021","journal-title":"Acta Polytech. Hung."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1016\/j.physbeh.2015.09.032","article-title":"Skin Temperature Reveals the Intensity of Acute Stress","volume":"152","author":"Herborn","year":"2015","journal-title":"Physiol. Behav."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1038\/srep00305","article-title":"Fast by Nature\u2014How Stress Patterns Define Human Experience and Performance in Dexterous Tasks","volume":"2","author":"Pavlidis","year":"2012","journal-title":"Sci. Rep."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1097\/SLA.0000000000000260","article-title":"Saccadic Eye Movement Metrics Reflect Surgical Residents\u2019 Fatigue","volume":"259","author":"McCamy","year":"2014","journal-title":"Ann. Surg."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"79","DOI":"10.12700\/APH.18.1.2021.1.6","article-title":"Clean and dirty code comprehension by eye-tracking based evaluation using GP3 eye tracker","volume":"18","author":"Katona","year":"2021","journal-title":"Acta Polytech. Hung."},{"key":"ref_27","first-page":"1672","article-title":"Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals","volume":"2004","author":"Nasoz","year":"2004","journal-title":"EURASIP J. Appl. Signal Process."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"612","DOI":"10.1016\/j.media.2010.07.007","article-title":"Gaze-Contingent Motor Channelling, Haptic Constraints and Associated Cognitive Demand for Robotic MIS","volume":"16","author":"Mylonas","year":"2012","journal-title":"Med. Image Anal."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1159\/000365580","article-title":"Relax, It\u2019s Just Laparoscopy! A Prospective Randomized Trial on Heart Rate Variability of the Surgeon in Robot-Assisted versus Conventional Laparoscopic Cholecystectomy","volume":"31","author":"Heemskerk","year":"2014","journal-title":"Dig. Surg."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1109\/MRA.2021.3101646","article-title":"Accelerating Surgical Robotics Research: A Review of 10 Years With the Da Vinci Research Kit","volume":"28","author":"Mariani","year":"2021","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1109\/JPROC.2022.3180350","article-title":"Robot-assisted minimally invasive surgery\u2014Surgical robotics in the data age","volume":"110","author":"Haidegger","year":"2022","journal-title":"Proc. IEEE"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1961","DOI":"10.1007\/s00268-011-1141-4","article-title":"Development and Validation of a Surgical Workload Measure: The Surgery Task Load Index (SURG-TLX)","volume":"35","author":"Wilson","year":"2011","journal-title":"World J. Surg."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Nagyn\u00e9 Elek, R., and Haidegger, T. (2021). Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery. Sensors, 21.","DOI":"10.3390\/s21082666"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lukezic, A., Vojir, T., Cehovin Zajc, L., Matas, J., and Kristan, M. (2017, January 21\u201326). Discriminative correlation filter with channel and spatial reliability. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.515"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wang, Z., Yan, W., and Oates, T. (2017, January 14\u201319). Time series classification from scratch with deep neural networks: A strong baseline. Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, USA.","DOI":"10.1109\/IJCNN.2017.7966039"},{"key":"ref_36","unstructured":"Hassler, K.R., Collins, J.T., Philip, K., and Jones, M.W. (2022). Laparoscopic Cholecystectomy. StatPearls, StatPearls Publishing."},{"key":"ref_37","unstructured":"(2022, May 10). Laparoscopyboxx | Laparoscopic Training Boxes & Instruments. Available online: https:\/\/laparoscopyboxx.com\/."},{"key":"ref_38","unstructured":"Gao, Y., Vedula, S.S., Reiley, C.E., Ahmidi, N., Varadarajan, B., Lin, H.C., Tao, L., Zappella, L., B\u00e9jar, B., and Yuh, D.D. (2022, November 26). Jhu-isi Gesture and Skill Assessment Working Set (jigsaws): A Surgical Activity Dataset for Human Motion Modeling. Available online: https:\/\/cirl.lcsr.jhu.edu\/wp-content\/uploads\/2015\/11\/JIGSAWS.pdf."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/11\/24\/7533\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:44:12Z","timestamp":1760147052000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/11\/24\/7533"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,19]]},"references-count":38,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["jcm11247533"],"URL":"https:\/\/doi.org\/10.3390\/jcm11247533","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,19]]}}}