{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T19:35:27Z","timestamp":1778268927054,"version":"3.51.4"},"reference-count":61,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T00:00:00Z","timestamp":1650931200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T00:00:00Z","timestamp":1650931200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In procedural-based medicine, the technical ability can be a critical determinant of patient outcomes. Psychomotor performance occurs in real-time, hence a continuous assessment is necessary to provide action-oriented feedback and error avoidance guidance. We outline a deep learning application, the Intelligent Continuous Expertise Monitoring System (ICEMS), to assess surgical bimanual performance at 0.2-s intervals. A long-short term memory network was built using neurosurgeon and student performance in 156 virtually simulated tumor resection tasks. Algorithm predictive ability was tested separately on 144 procedures by scoring the performance of neurosurgical trainees who are at different training stages. The ICEMS successfully differentiated between neurosurgeons, senior trainees, junior trainees, and students. Trainee average performance score correlated with the year of training in neurosurgery. Furthermore, coaching and risk assessment for critical metrics were demonstrated. This work presents a comprehensive technical skill monitoring system with predictive validation throughout surgical residency training, with the ability to detect errors.<\/jats:p>","DOI":"10.1038\/s41746-022-00596-8","type":"journal-article","created":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T10:10:43Z","timestamp":1650967843000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0521-3227","authenticated-orcid":false,"given":"Recai","family":"Yilmaz","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2110-9879","authenticated-orcid":false,"given":"Alexander","family":"Winkler-Schwartz","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3742-4611","authenticated-orcid":false,"given":"Nykan","family":"Mirchi","sequence":"additional","affiliation":[]},{"given":"Aiden","family":"Reich","sequence":"additional","affiliation":[]},{"given":"Sommer","family":"Christie","sequence":"additional","affiliation":[]},{"given":"Dan Huy","family":"Tran","sequence":"additional","affiliation":[]},{"given":"Nicole","family":"Ledwos","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7400-5558","authenticated-orcid":false,"given":"Ali M.","family":"Fazlollahi","sequence":"additional","affiliation":[]},{"given":"Carlo","family":"Santaguida","sequence":"additional","affiliation":[]},{"given":"Abdulrahman J.","family":"Sabbagh","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7925-5959","authenticated-orcid":false,"given":"Khalid","family":"Bajunaid","sequence":"additional","affiliation":[]},{"given":"Rolando","family":"Del Maestro","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,4,26]]},"reference":[{"key":"596_CR1","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1016\/j.amjsurg.2012.11.009","volume":"206","author":"O Anderson","year":"2013","unstructured":"Anderson, O., Davis, R., Hanna, G. B. & Vincent, C. A. Surgical adverse events: a systematic review. Am. J. Surg. 206, 253\u2013262 (2013).","journal-title":"Am. J. Surg."},{"key":"596_CR2","doi-asserted-by":"publisher","first-page":"960","DOI":"10.1001\/jamasurg.2020.3007","volume":"155","author":"JJ Stulberg","year":"2020","unstructured":"Stulberg, J. J. et al. Association Between Surgeon Technical Skills and Patient Outcomes. JAMA Surg. 155, 960\u2013968 (2020).","journal-title":"JAMA Surg"},{"key":"596_CR3","doi-asserted-by":"publisher","first-page":"705","DOI":"10.1097\/SLA.0b013e31815865f8","volume":"246","author":"SE Regenbogen","year":"2007","unstructured":"Regenbogen, S. E. et al. Patterns of Technical Error Among Surgical Malpractice Claims: An Analysis of Strategies to Prevent Injury to Surgical Patients. Ann. Surg. 246, 705\u2013711 (2007).","journal-title":"Ann. Surg."},{"key":"596_CR4","doi-asserted-by":"publisher","first-page":"S30","DOI":"10.1227\/NEU.0000000000000115","volume":"73","author":"N G\u00e9linas-Phaneuf","year":"2013","unstructured":"G\u00e9linas-Phaneuf, N. & Del Maestro, R. F. Surgical Expertise in Neurosurgery: Integrating Theory Into Practice. Neurosurgery 73, S30\u2013S38 (2013).","journal-title":"Neurosurgery"},{"key":"596_CR5","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1136\/postgradmedj-2012-130881","volume":"89","author":"A Brightwell","year":"2013","unstructured":"Brightwell, A. & Grant, J. Competency-based training: who benefits? Postgrad. Med. J. 89, 107 (2013).","journal-title":"Postgrad. Med. J."},{"key":"596_CR6","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1037\/0003-066X.49.8.725","volume":"49","author":"KA Ericsson","year":"1994","unstructured":"Ericsson, K. A. & Charness, N. Expert performance: Its structure and acquisition. Am. Psychologist 49, 725\u2013747 (1994).","journal-title":"Am. Psychologist"},{"key":"596_CR7","doi-asserted-by":"publisher","first-page":"14","DOI":"10.3171\/2020.4.JNS201012","volume":"133","author":"BT Samuel","year":"2020","unstructured":"Samuel, B. T., Benjamin, K. H. & Aaron, A. C.-G. Editorial. Innovations in neurosurgical education during the COVID-19 pandemic: is it time to reexamine our neurosurgical training models? J. Neurosurg. JNS 133, 14\u201315 (2020).","journal-title":"J. Neurosurg. JNS"},{"key":"596_CR8","doi-asserted-by":"publisher","first-page":"e198363","DOI":"10.1001\/jamanetworkopen.2019.8363","volume":"2","author":"A Winkler-Schwartz","year":"2019","unstructured":"Winkler-Schwartz, A. et al. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw. Open 2, e198363 (2019).","journal-title":"JAMA Netw. Open"},{"key":"596_CR9","doi-asserted-by":"publisher","first-page":"e0229596","DOI":"10.1371\/journal.pone.0229596","volume":"15","author":"N Mirchi","year":"2020","unstructured":"Mirchi, N. et al. The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLOS ONE 15, e0229596 (2020).","journal-title":"PLOS ONE"},{"key":"596_CR10","doi-asserted-by":"publisher","first-page":"180","DOI":"10.4103\/2152-7806.90714","volume":"2","author":"AO Hebb","year":"2011","unstructured":"Hebb, A. O., Yang, T. & Silbergeld, D. L. The sub-pial resection technique for intrinsic tumor surgery. Surgical Neurol. Int. 2, 180\u2013180 (2011).","journal-title":"Surgical Neurol. Int."},{"key":"596_CR11","first-page":"E8","volume":"28","author":"G-R Santiago","year":"2010","unstructured":"Santiago, G.-R. & Hugues, D. Surgical management of World Health Organization Grade II gliomas in eloquent areas: the necessity of preserving a margin around functional structures. Neurosurgical Focus FOC 28, E8 (2010).","journal-title":"Neurosurgical Focus FOC"},{"key":"596_CR12","doi-asserted-by":"publisher","first-page":"e220","DOI":"10.1016\/j.wneu.2020.03.187","volume":"139","author":"AJ Sabbagh","year":"2020","unstructured":"Sabbagh, A. J. et al. Roadmap for Developing Complex Virtual Reality Simulation Scenarios: Subpial Neurosurgical Tumor Resection Model. World Neurosurg. 139, e220\u2013e229 (2020).","journal-title":"World Neurosurg."},{"key":"596_CR13","doi-asserted-by":"publisher","first-page":"ons32","DOI":"10.1227\/NEU.0b013e318249c744","volume":"71","author":"S Delorme","year":"2012","unstructured":"Delorme, S., Laroche, D., DiRaddo, R. & Del Maestro, R. F. NeuroTouch: A Physics-Based Virtual Simulator for Cranial Microneurosurgery Training. Operative Neurosurg. 71, ons32\u2013ons42 (2012).","journal-title":"Operative Neurosurg."},{"key":"596_CR14","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1016\/j.jsurg.2014.12.014","volume":"72","author":"G AlZhrani","year":"2015","unstructured":"AlZhrani, G. et al. Proficiency Performance Benchmarks for Removal of Simulated Brain Tumors Using a Virtual Reality Simulator NeuroTouch. J. Surgical Educ. 72, 685\u2013696 (2015).","journal-title":"J. Surgical Educ."},{"key":"596_CR15","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1016\/j.jsurg.2017.06.018","volume":"75","author":"A Bugdadi","year":"2018","unstructured":"Bugdadi, A. et al. Automaticity of force application during simulated brain tumor resection: testing the Fitts and Posner model. J. surgical Educ. 75, 104\u2013115 (2018).","journal-title":"J. surgical Educ."},{"key":"596_CR16","doi-asserted-by":"publisher","first-page":"E15","DOI":"10.3171\/2021.5.FOCUS21210","volume":"51","author":"J Chan","year":"2021","unstructured":"Chan, J. et al. A systematic review of virtual reality for the assessment of technical skills in neurosurgery. Neurosurgical Focus 51, E15 (2021).","journal-title":"Neurosurgical Focus"},{"key":"596_CR17","unstructured":"Norman, G. R. et al. Expertise in Medicine and Surgery. In The Cambridge Handbook of Expertise and Expert Performance (eds. Williams, A. M., Kozbelt, A., Ericsson, K. A. & Hoffman, R. R.) 331\u2013355 (Cambridge University Press, Cambridge, 2018)."},{"key":"596_CR18","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter, S. & Schmidhuber, J. Long Short-Term Memory. Neural Comput. 9, 1735\u20131780 (1997).","journal-title":"Neural Comput."},{"key":"596_CR19","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"596_CR20","first-page":"00019","volume":"1506","author":"ZC Lipton","year":"2015","unstructured":"Lipton, Z. C., Berkowitz, J. & Elkan, C. A critical review of recurrent neural networks for sequence learning. arXiv Prepr. arXiv 1506, 00019 (2015).","journal-title":"arXiv Prepr. arXiv"},{"key":"596_CR21","doi-asserted-by":"crossref","unstructured":"Cohen, J. Statistical power analysis for the behavioral sciences, (Academic press, 2013).","DOI":"10.4324\/9780203771587"},{"key":"596_CR22","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1093\/ons\/opx189","volume":"14","author":"R Sawaya","year":"2017","unstructured":"Sawaya, R. et al. Virtual Reality Tumor Resection: The Force Pyramid Approach. Operative Neurosurg. 14, 686\u2013696 (2017).","journal-title":"Operative Neurosurg."},{"key":"596_CR23","doi-asserted-by":"publisher","first-page":"942","DOI":"10.1016\/j.jsurg.2016.04.013","volume":"73","author":"A Winkler-Schwartz","year":"2016","unstructured":"Winkler-Schwartz, A. et al. Bimanual psychomotor performance in neurosurgical resident applicants assessed using NeuroTouch, a virtual reality simulator. J. surgical Educ. 73, 942\u2013953 (2016).","journal-title":"J. surgical Educ."},{"key":"596_CR24","doi-asserted-by":"publisher","first-page":"e2031217","DOI":"10.1001\/jamanetworkopen.2020.31217","volume":"3","author":"R Lohre","year":"2020","unstructured":"Lohre, R. et al. Effectiveness of Immersive Virtual Reality on Orthopedic Surgical Skills and Knowledge Acquisition Among Senior Surgical Residents: A Randomized Clinical Trial. JAMA Netw. Open 3, e2031217\u2013e2031217 (2020).","journal-title":"JAMA Netw. Open"},{"key":"596_CR25","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1097\/00000658-200210000-00008","volume":"236","author":"NE Seymour","year":"2002","unstructured":"Seymour, N. E. et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann. Surg. 236, 458\u2013464 (2002).","journal-title":"Ann. Surg."},{"key":"596_CR26","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1002\/bjs.4407","volume":"91","author":"TP Grantcharov","year":"2004","unstructured":"Grantcharov, T. P. et al. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br. J. Surg. 91, 146\u2013150 (2004).","journal-title":"Br. J. Surg."},{"key":"596_CR27","doi-asserted-by":"publisher","first-page":"e62","DOI":"10.1016\/j.wneu.2020.07.209","volume":"144","author":"A Winkler-Schwartz","year":"2020","unstructured":"Winkler-Schwartz, A. et al. Creating a Comprehensive Research Platform for Surgical Technique and Operative Outcome in Primary Brain Tumor Neurosurgery. World Neurosurg. 144, e62\u2013e71 (2020).","journal-title":"World Neurosurg"},{"key":"596_CR28","first-page":"00097","volume":"2101","author":"M Ershad","year":"2020","unstructured":"Ershad, M., Rege, R. & Fey, A. M. Adaptive Surgical Robotic Training Using Real-Time Stylistic Behavior Feedback Through Haptic Cues. arXiv Prepr. arXiv 2101, 00097 (2020).","journal-title":"arXiv Prepr. arXiv"},{"key":"596_CR29","doi-asserted-by":"publisher","first-page":"586707","DOI":"10.3389\/frobt.2020.586707","volume":"7","author":"P Fekri","year":"2021","unstructured":"Fekri, P., Dargahi, J. & Zadeh, M. Deep Learning-Based Haptic Guidance for Surgical Skills Transfer. Front. Robot. AI. 7, 586707 (2021).","journal-title":"Front. Robot. AI"},{"key":"596_CR30","doi-asserted-by":"publisher","first-page":"103760","DOI":"10.1016\/j.engappai.2020.103760","volume":"94","author":"N Vaughan","year":"2020","unstructured":"Vaughan, N. & Gabrys, B. Scoring and assessment in medical VR training simulators with dynamic time series classification. Eng. Appl. Artif. Intell. 94, 103760 (2020).","journal-title":"Eng. Appl. Artif. Intell."},{"key":"596_CR31","doi-asserted-by":"publisher","first-page":"192","DOI":"10.3171\/2018.2.JNS172327","volume":"131","author":"R Sawaya","year":"2018","unstructured":"Sawaya, R. et al. Development of a performance model for virtual reality tumor resections. J. Neurosurg. JNS 131, 192\u2013200 (2018).","journal-title":"J. Neurosurg. JNS"},{"key":"596_CR32","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.artmed.2018.08.002","volume":"91","author":"G Forestier","year":"2018","unstructured":"Forestier, G. et al. Surgical motion analysis using discriminative interpretable patterns. Artif. Intell. Med. 91, 3\u201311 (2018).","journal-title":"Artif. Intell. Med."},{"key":"596_CR33","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-021-02524-y","volume":"21","author":"G Chartrand","year":"2021","unstructured":"Chartrand, G. et al. Self-directed learning by video as a means to improve technical skills in surgery residents: a randomized controlled trial. BMC Med. Educ. 21, 91 (2021).","journal-title":"BMC Med. Educ."},{"key":"596_CR34","doi-asserted-by":"crossref","unstructured":"Sadeghi Esfahlani, S., et al. Development of an Interactive Virtual Reality for Medical Skills Training Supervised by Artificial Neural Network. in Intelligent Systems and Applications (eds. Bi, Y., Bhatia, R. & Kapoor, S.) 473\u2013482 (Springer International Publishing, Cham, 2020).","DOI":"10.1007\/978-3-030-29513-4_34"},{"key":"596_CR35","doi-asserted-by":"publisher","first-page":"102007","DOI":"10.1016\/j.artmed.2020.102007","volume":"112","author":"P Castillo-Segura","year":"2021","unstructured":"Castillo-Segura, P., Fern\u00e1ndez-Panadero, C., Alario-Hoyos, C. & Mu\u00f1oz-Merino, P. J. & Delgado Kloos, C. Objective and automated assessment of surgical technical skills with IoT systems: A systematic literature review. Artif. Intell. Med. 112, 102007 (2021).","journal-title":"Artif. Intell. Med."},{"key":"596_CR36","doi-asserted-by":"publisher","first-page":"1178","DOI":"10.1111\/j.1365-2923.2007.02924.x","volume":"41","author":"B Charlin","year":"2007","unstructured":"Charlin, B., Boshuizen, H. P. A., Custers, E. J. & Feltovich, P. J. Scripts and clinical reasoning. Med. Educ. 41, 1178\u20131184 (2007).","journal-title":"Med. Educ."},{"key":"596_CR37","doi-asserted-by":"publisher","first-page":"449","DOI":"10.2307\/258285","volume":"9","author":"DA Gioia","year":"1984","unstructured":"Gioia, D. A. & Poole, P. P. Scripts in Organizational Behavior. Acad. Manag. Rev. 9, 449\u2013459 (1984).","journal-title":"Acad. Manag. Rev."},{"key":"596_CR38","doi-asserted-by":"publisher","first-page":"2365","DOI":"10.1109\/TMECH.2016.2563384","volume":"21","author":"K Zareinia","year":"2016","unstructured":"Zareinia, K. et al. A Force-Sensing Bipolar Forceps to Quantify Tool\u2013Tissue Interaction Forces in Microsurgery. IEEE\/ASME Trans. Mechatron. 21, 2365\u20132377 (2016).","journal-title":"IEEE\/ASME Trans. Mechatron."},{"key":"596_CR39","doi-asserted-by":"publisher","first-page":"e669","DOI":"10.1016\/j.wneu.2021.01.117","volume":"149","author":"J Davids","year":"2021","unstructured":"Davids, J. et al. Automated Vision-Based Microsurgical Skill Analysis in Neurosurgery Using Deep Learning: Development and Preclinical Validation. World Neurosurg. 149, e669\u2013e686 (2021).","journal-title":"World Neurosurg."},{"key":"596_CR40","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1093\/bjs\/znab016","volume":"108","author":"M Levin","year":"2021","unstructured":"Levin, M. et al. Surgical data recording in the operating room: a systematic review of modalities and metrics. Br. J. Surg. 108, 613\u2013621 (2021).","journal-title":"Br. J. Surg"},{"key":"596_CR41","doi-asserted-by":"publisher","first-page":"1434","DOI":"10.1056\/NEJMsa1300625","volume":"369","author":"JD Birkmeyer","year":"2013","unstructured":"Birkmeyer, J. D. et al. Surgical Skill and Complication Rates after Bariatric Surgery. N. Engl. J. Med. 369, 1434\u20131442 (2013).","journal-title":"N. Engl. J. Med."},{"key":"596_CR42","first-page":"12478","volume":"2002","author":"Q Wen","year":"2020","unstructured":"Wen, Q. et al. Time series data augmentation for deep learning: A survey. arXiv Prepr. arXiv 2002, 12478 (2020).","journal-title":"arXiv Prepr. arXiv"},{"key":"596_CR43","doi-asserted-by":"publisher","first-page":"e2149008","DOI":"10.1001\/jamanetworkopen.2021.49008","volume":"5","author":"AM Fazlollahi","year":"2022","unstructured":"Fazlollahi, A. M. et al. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw. Open 5, e2149008\u2013e2149008 (2022).","journal-title":"JAMA Netw. Open"},{"key":"596_CR44","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1038\/nature23474","volume":"549","author":"J Biamonte","year":"2017","unstructured":"Biamonte, J. et al. Quantum machine learning. Nature 549, 195\u2013202 (2017).","journal-title":"Nature"},{"key":"596_CR45","doi-asserted-by":"publisher","first-page":"012080","DOI":"10.1103\/PhysRevResearch.2.012080","volume":"2","author":"H Fan","year":"2020","unstructured":"Fan, H., Jiang, J., Zhang, C., Wang, X. & Lai, Y.-C. Long-term prediction of chaotic systems with machine learning. Phys. Rev. Res. 2, 012080 (2020).","journal-title":"Phys. Rev. Res."},{"key":"596_CR46","doi-asserted-by":"publisher","first-page":"20180377","DOI":"10.1098\/rstb.2018.0377","volume":"374","author":"LF Seoane","year":"2019","unstructured":"Seoane, L. F. Evolutionary aspects of reservoir computing. Philos. Trans. R. Soc. B: Biol. Sci. 374, 20180377 (2019).","journal-title":"Philos. Trans. R. Soc. B: Biol. Sci."},{"key":"596_CR47","first-page":"1","volume":"10","author":"N Mirchi","year":"2020","unstructured":"Mirchi, N., Ledwos, N. & Del Maestro, R. F. Intelligent Tutoring Systems: Re-Envisioning Surgical Education in Response to COVID-19. Can. J. Neurol. Sci. 10, 1\u20133 (2020).","journal-title":"Can. J. Neurol. Sci"},{"key":"596_CR48","doi-asserted-by":"publisher","first-page":"2191","DOI":"10.1001\/jama.2013.281053","volume":"310","author":"A World Medical","year":"2013","unstructured":"World Medical, A. World Medical Association Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects. JAMA 310, 2191\u20132194 (2013).","journal-title":"JAMA"},{"key":"596_CR49","doi-asserted-by":"publisher","first-page":"1681","DOI":"10.1016\/j.jsurg.2019.05.015","volume":"76","author":"A Winkler-Schwartz","year":"2019","unstructured":"Winkler-Schwartz, A. et al. Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation. J. Surg. Educ. 76, 1681\u20131690 (2019).","journal-title":"J. Surg. Educ."},{"key":"596_CR50","doi-asserted-by":"publisher","DOI":"10.1186\/s41077-016-0025-y","volume":"1","author":"A Cheng","year":"2016","unstructured":"Cheng, A. et al. Reporting guidelines for health care simulation research: extensions to the CONSORT and STROBE statements. Adv. Simul. 1, 25 (2016).","journal-title":"Adv. Simul."},{"key":"596_CR51","doi-asserted-by":"publisher","first-page":"W1","DOI":"10.7326\/M14-0698","volume":"162","author":"KG Moons","year":"2015","unstructured":"Moons, K. G. et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann. Intern Med 162, W1\u201373 (2015).","journal-title":"Ann. Intern Med"},{"key":"596_CR52","doi-asserted-by":"publisher","first-page":"e297","DOI":"10.1371\/journal.pmed.0040297","volume":"4","author":"JP Vandenbroucke","year":"2007","unstructured":"Vandenbroucke, J. P. et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med 4, e297 (2007).","journal-title":"PLoS Med"},{"key":"596_CR53","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1227\/NEU.0000000000000631","volume":"11","author":"FE Alotaibi","year":"2015","unstructured":"Alotaibi, F. E. et al. Assessing Bimanual Performance in Brain Tumor Resection With NeuroTouch, a Virtual Reality Simulator. Operative Neurosurg. 11, 89\u201398 (2015).","journal-title":"Operative Neurosurg."},{"key":"596_CR54","doi-asserted-by":"publisher","first-page":"636","DOI":"10.1177\/1553350615579729","volume":"22","author":"FE Alotaibi","year":"2015","unstructured":"Alotaibi, F. E. et al. Neurosurgical Assessment of Metrics Including Judgment and Dexterity Using the Virtual Reality Simulator NeuroTouch (NAJD Metrics). Surgical Innov. 22, 636\u2013642 (2015).","journal-title":"Surgical Innov."},{"key":"596_CR55","doi-asserted-by":"publisher","first-page":"603","DOI":"10.1007\/s11548-014-1091-z","volume":"10","author":"H Azarnoush","year":"2015","unstructured":"Azarnoush, H. et al. Neurosurgical virtual reality simulation metrics to assess psychomotor skills during brain tumor resection. Int. J. Computer Assist. Radiol. Surg. 10, 603\u2013618 (2015).","journal-title":"Int. J. Computer Assist. Radiol. Surg."},{"key":"596_CR56","doi-asserted-by":"publisher","first-page":"171","DOI":"10.3171\/2016.7.JNS16322","volume":"127","author":"H Azarnoush","year":"2016","unstructured":"Azarnoush, H. et al. The force pyramid: a spatial analysis of force application during virtual reality brain tumor resection. J. Neurosurg. 127, 171 (2016).","journal-title":"J. Neurosurg"},{"key":"596_CR57","doi-asserted-by":"publisher","first-page":"71","DOI":"10.3171\/2015.5.JNS15558","volume":"126","author":"B Khalid","year":"2017","unstructured":"Khalid, B. et al. Impact of acute stress on psychomotor bimanual performance during a simulated tumor resection task. J. Neurosurg. JNS 126, 71\u201380 (2017).","journal-title":"J. Neurosurg. JNS"},{"key":"596_CR58","doi-asserted-by":"publisher","first-page":"e127","DOI":"10.2106\/JBJS.18.01197","volume":"101","author":"V Bissonnette","year":"2019","unstructured":"Bissonnette, V. et al. Artificial Intelligence Distinguishes Surgical Training Levels in a Virtual Reality Spinal Task. J. Bone Jt. Surg. Am. 101, e127 (2019).","journal-title":"J. Bone Jt. Surg. Am."},{"key":"596_CR59","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1093\/ons\/opz359","volume":"19","author":"N Mirchi","year":"2019","unstructured":"Mirchi, N. et al. Artificial Neural Networks to Assess Virtual Reality Anterior Cervical Discectomy Performance. Oper.Neurosurg. (Hagerstown). 19, 65\u201375 (2019).","journal-title":"Oper.Neurosurg. (Hagerstown)"},{"key":"596_CR60","doi-asserted-by":"crossref","unstructured":"Brian, J. T., Marjorie, A. D. & Christina, D. M. Verification and validation of neural networks: a sampling of research in progress. in Proc.SPIE, 5103 (2003).","DOI":"10.1117\/12.487527"},{"key":"596_CR61","doi-asserted-by":"publisher","unstructured":"Yilmaz, R. SubPialResection101-KFMC_scenario.xml:2015-Oct-22_14h06m26s_log.csv. https:\/\/doi.org\/10.6084\/m9.figshare.15132507.v1. (2021).","DOI":"10.6084\/m9.figshare.15132507.v1."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00596-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00596-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00596-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,22]],"date-time":"2024-09-22T23:45:11Z","timestamp":1727048711000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00596-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,26]]},"references-count":61,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["596"],"URL":"https:\/\/doi.org\/10.1038\/s41746-022-00596-8","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,26]]},"assertion":[{"value":"23 August 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"R.Y., N.M., A.W.S., and R.D.M. report having a patent pending to \u2018Methods and systems for continuous monitoring of task performance on virtual simulators\u2019 (2020; patent No. 05001770-883USPR). The remaining authors declare no competing interests. The funding sources had not been involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"54"}}