{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T15:49:03Z","timestamp":1774367343839,"version":"3.50.1"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,1,30]],"date-time":"2020-01-30T00:00:00Z","timestamp":1580342400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100010477","name":"Intuitive Surgical","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100010477","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2020,6,30]]},"abstract":"<jats:p>Monitoring surgeon workload during robot-assisted surgery can guide allocation of task demands, adapt system interfaces, and assess the robotic system's usability. Current practices for measuring cognitive load primarily rely on questionnaires that are subjective and disrupt surgical workflow. To address this limitation, a computational framework is demonstrated to predict user workload during telerobotic surgery. This framework leverages wireless sensors to monitor surgeons\u2019 cognitive load and predict their cognitive states. Continuous data across multiple physiological modalities (e.g., heart rate variability, electrodermal, and electroencephalogram activity) were simultaneously recorded for twelve surgeons performing surgical skills tasks on the validated da Vinci Skills Simulator. These surgical tasks varied in difficulty levels, e.g., requiring varying visual processing demand and degree of fine motor control. Collected multimodal physiological signals were fused using independent component analysis, and the predicted results were compared to the ground-truth workload level. Results compared performance of different classifiers, sensor fusion schemes, and physiological modality (i.e., prediction with single vs. multiple modalities). It was found that our multisensor approach outperformed individual signals and can correctly predict cognitive workload levels 83.2% of the time during basic and complex surgical skills tasks.<\/jats:p>","DOI":"10.1145\/3368589","type":"journal-article","created":{"date-parts":[[2020,4,4]],"date-time":"2020-04-04T00:54:52Z","timestamp":1585961692000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":53,"title":["Multimodal Physiological Signals for Workload Prediction in Robot-assisted Surgery"],"prefix":"10.1145","volume":"9","author":[{"given":"Tian","family":"Zhou","sequence":"first","affiliation":[{"name":"Purdue University, West Lafayette, IN, USA"}]},{"given":"Jackie S.","family":"Cha","sequence":"additional","affiliation":[{"name":"Purdue University, West Lafayette, IN, USA"}]},{"given":"Glebys","family":"Gonzalez","sequence":"additional","affiliation":[{"name":"Purdue University, West Lafayette, IN, USA"}]},{"given":"Juan P.","family":"Wachs","sequence":"additional","affiliation":[{"name":"Purdue University, West Lafayette, IN, USA"}]},{"given":"Chandru P.","family":"Sundaram","sequence":"additional","affiliation":[{"name":"Indiana University School of Medicine, IN, USA"}]},{"given":"Denny","family":"Yu","sequence":"additional","affiliation":[{"name":"Purdue University, West Lafayette, IN, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,1,30]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Hirshfield L. M.","year":"2009"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2008.917246"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/02783649922067807"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (M2CAI\u201914)","author":"Gao Y.","year":"2014"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/IEMBS.2006.259707"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.surg.2015.03.032"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsurg.2017.05.013"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1111\/bju.13445"},{"key":"e_1_2_1_9_1","volume-title":"Adv. Psychol. 52","author":"Hart S. G.","year":"1988"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62387-0"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1464-0597.2004.00161.x"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting","volume":"57","author":"Scerbo M. W."},{"key":"e_1_2_1_13_1","first-page":"7","article-title":"Measuring surgeons\u2019 mental workload with a time-based secondary task","volume":"21","author":"Grant R. C.","year":"2013","journal-title":"Ergon. Des."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"A. Darzi S. Smith and N. Taffinder. 1999. Assessing operative skill: Needs to become more objective. BMJ 318 7188 887 1999.  A. Darzi S. Smith and N. Taffinder. 1999. Assessing operative skill: Needs to become more objective. BMJ 318 7188 887 1999.","DOI":"10.1136\/bmj.318.7188.887"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.45.4.575.27094"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbr.2013.10.042"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"P. G. Jorna. Spectral analysis of heart rate and psychological state: A review of its validity as a workload index. Biol. Psychol. 34 2 (1992) 237--257.  P. G. Jorna. Spectral analysis of heart rate and psychological state: A review of its validity as a workload index. Biol. Psychol. 34 2 (1992) 237--257.","DOI":"10.1016\/0301-0511(92)90017-O"},{"key":"e_1_2_1_18_1","volume-title":"Advances in Psychology","author":"Meshkati N."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327108IJAP1201_2"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijpsycho.2016.10.013"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.3141\/2138-02"},{"key":"e_1_2_1_22_1","first-page":"B231","article-title":"EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks","volume":"78","author":"Berka C.","year":"2007","journal-title":"Aviat. Space Environ. Med."},{"key":"e_1_2_1_23_1","first-page":"360","article-title":"A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight","volume":"69","author":"Hankins T. C.","year":"1998","journal-title":"Aviat. Space Environ. Med."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00464-013-3334-4"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00464-010-1443-x"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00268-011-1141-4"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00464-016-5047-y"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140139608964470"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1002\/bjs.1800840237"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. 4057--4066","author":"Solovey E. T."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1093\/iwc\/iwu021"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.urology.2015.07.028"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"H. Aghajani M. Garbey and A. Omurtag. 2017. Measuring mental workload with EEG+ fNIRS. Front. Hum. Neurosci. 11 359.  H. Aghajani M. Garbey and A. Omurtag. 2017. Measuring mental workload with EEG+ fNIRS. Front. Hum. Neurosci. 11 359.","DOI":"10.3389\/fnhum.2017.00359"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1177\/001872089203400406"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.surg.2007.07.034"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00464-008-0017-7"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00268-008-9624-7"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2005.04.004"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajog.2016.06.033"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/0301-0511(95)05167-8"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2012.10.003"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1111\/bju.12657"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-8986.1994.tb01040.x"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.2114\/jpa2.27.173"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the 1st International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 115--118","author":"Reimer B."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 480--487","author":"Rowe D. W."},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the CHI\u201904: Extended Abstracts on Human Factors in Computing Systems. 1513--1516","author":"Chen D."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1518\/001872095778995517"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/0301-0511(92)90018-P"},{"key":"e_1_2_1_51_1","volume-title":"Mental Workload","author":"De Waard D."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s004640080030"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.3390\/s17102315"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.45.4.635.27088"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2017.00389"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.3390\/s18020458"},{"key":"e_1_2_1_57_1","unstructured":"\u201cMimic Simulation | Leader in Robotic Surgery Simulation 8 Training \u201d Mimic Simulation. Retrieved from http:\/\/mimicsimulation.com\/.  \u201cMimic Simulation | Leader in Robotic Surgery Simulation 8 Training \u201d Mimic Simulation. Retrieved from http:\/\/mimicsimulation.com\/."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11858-015-0754-8"},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the 37th Annual International Conference of the IEEE on Engineering in Medicine and Biology Society (EMBC\u201915)","author":"Fan J.","year":"2015"},{"key":"e_1_2_1_60_1","first-page":"1","article-title":"Fundamentals of EEG measurement","volume":"2","author":"Teplan M.","year":"2002","journal-title":"Meas. Sci. Rev."},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the 27th Annual International Conference of the Engineering in Medicine and Biology Society (IEEE-EMBS\u201905)","author":"Liang S. F."},{"key":"e_1_2_1_62_1","unstructured":"EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis--ScienceDirect. Retrieved from http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0165017398000563.  EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis--ScienceDirect. Retrieved from http:\/\/www.sciencedirect.com\/science\/article\/pii\/S0165017398000563."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jneumeth.2003.10.009"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBME.2003.810706"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2010.04.043"},{"key":"e_1_2_1_66_1","first-page":"647","article-title":"Decomposition of skin conductance data by means of nonnegative deconvolution","volume":"47","author":"Benedek M.","year":"2010","journal-title":"Psychophysiology"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.biopsycho.2014.08.006"},{"key":"e_1_2_1_68_1","doi-asserted-by":"crossref","unstructured":"W. Boucsein. 2012. Electrodermal Activity. Springer Science 8 Business Media.  W. Boucsein. 2012. Electrodermal Activity. Springer Science 8 Business Media.","DOI":"10.1007\/978-1-4614-1126-0"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.icte.2016.11.003"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00420-013-0847-z"},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the 6th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. 590--597","author":"Mehler B."},{"key":"e_1_2_1_72_1","volume-title":"Regression Diagnostics: Identifying Influential Data and Sources of Collinearity","author":"Belsley D. A.","year":"2005"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2004.09.041"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the 16th International Conference on Multimodal Interaction","author":"Mart\u00ednez H. P.","year":"2014"},{"key":"e_1_2_1_75_1","unstructured":"T. Zhou and J. P. Wachs. 2017. Early prediction for physical human robot collaboration in the operating room. Auton. Robots 1--19.  T. Zhou and J. P. Wachs. 2017. Early prediction for physical human robot collaboration in the operating room. Auton. Robots 1--19."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijar.2015.12.009"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2014.2366914"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1177\/154193120605000909"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.urology.2012.02.059"},{"key":"e_1_2_1_80_1","doi-asserted-by":"crossref","unstructured":"F. T. Eggemeier and G. F. Wilson. 1991. Performance-based and subjective assessment of workload in multi-task environments. Mult.-Task Perform. 217--278.  F. T. Eggemeier and G. F. Wilson. 1991. Performance-based and subjective assessment of workload in multi-task environments. Mult.-Task Perform. 217--278.","DOI":"10.1201\/9781003069447-13"},{"key":"e_1_2_1_81_1","unstructured":"da Vinci Surgery\u2014Minimally Invasive Robotic Surgery with the da Vinci surgical system. Retrieved from http:\/\/www.davincisurgery.com\/.  da Vinci Surgery\u2014Minimally Invasive Robotic Surgery with the da Vinci surgical system. Retrieved from http:\/\/www.davincisurgery.com\/."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1177\/001872088402600504"},{"key":"e_1_2_1_83_1","doi-asserted-by":"crossref","unstructured":"J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 159--174 1977.  J. R. Landis and G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 159--174 1977.","DOI":"10.2307\/2529310"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btm344"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13042-011-0012-5"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.5555\/2188385.2188395"},{"key":"e_1_2_1_87_1","volume-title":"Proceedings of the 20th International Conference on Pattern Recognition (ICPR\u201910)","author":"Putze F."},{"key":"e_1_2_1_88_1","volume-title":"Proceedings of the 13th International Conference on Multimodal Interfaces. 205--208","author":"Jarvis J."},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2014.00322"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2014.6944254"},{"key":"e_1_2_1_91_1","first-page":"271","article-title":"Absolute and relative user perception of classification accuracy in an affective video game","volume":"29","author":"McCrea S.","year":"2016","journal-title":"Interact. Comput."},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814526617"},{"key":"e_1_2_1_93_1","first-page":"344","article-title":"EEG and ECG changes during simulator operation reflect mental workload and vigilance","volume":"76","author":"Dussault C.","year":"2005","journal-title":"Aviat. Space Environ. Med."},{"key":"e_1_2_1_94_1","volume-title":"Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201913)","author":"Rossi S."},{"key":"e_1_2_1_95_1","unstructured":"L. J. Trejo N. J. McDonald R. Matthews and B. Z. Allison. 2007. Experimental design and testing of a multimodal cognitive overload classifier. Found. Augment. Cogn. 13--22.  L. J. Trejo N. J. McDonald R. Matthews and B. Z. Allison. 2007. Experimental design and testing of a multimodal cognitive overload classifier. Found. Augment. Cogn. 13--22."},{"key":"e_1_2_1_96_1","doi-asserted-by":"crossref","unstructured":"M. Jacob C. Cange R. Packer and J. P. Wachs. 2012. Intention context and gesture recognition for sterile MRI navigation in the operating room. In Progress in Pattern Recognition Image Analysis Computer Vision and Applications L. Alvarez M. Mejail L. Gomez and J. Jacobo Eds. Springer Berlin 220--227.  M. Jacob C. Cange R. Packer and J. P. Wachs. 2012. Intention context and gesture recognition for sterile MRI navigation in the operating room. In Progress in Pattern Recognition Image Analysis Computer Vision and Applications L. Alvarez M. Mejail L. Gomez and J. Jacobo Eds. Springer Berlin 220--227.","DOI":"10.1007\/978-3-642-33275-3_27"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368589","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368589","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:26Z","timestamp":1750197686000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368589"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,30]]},"references-count":96,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6,30]]}},"alternative-id":["10.1145\/3368589"],"URL":"https:\/\/doi.org\/10.1145\/3368589","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,30]]},"assertion":[{"value":"2018-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}