{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T23:19:26Z","timestamp":1769555966772,"version":"3.49.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,6,3]],"date-time":"2019-06-03T00:00:00Z","timestamp":1559520000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"publisher","award":["FA9550-09-1-0108"],"award-info":[{"award-number":["FA9550-09-1-0108"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000104","name":"NASA","doi-asserted-by":"crossref","award":["NNX16AB24A"],"award-info":[{"award-number":["NNX16AB24A"]}],"id":[{"id":"10.13039\/100000104","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100007297","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N00014-12-1-0987"],"award-info":[{"award-number":["N00014-12-1-0987"]}],"id":[{"id":"10.13039\/100007297","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1659746, IIS-00643100"],"award-info":[{"award-number":["CNS-1659746, IIS-00643100"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2019,6,30]]},"abstract":"<jats:p>High-stress environments, such as first-response or a NASA control room, require optimal task performance, as a single mistake may cause monetary loss or even the loss of human life. Robots can partner with humans in a collaborative or supervisory paradigm to augment the human\u2019s abilities and increase task performance. Such teaming paradigms require the robot to appropriately interact with the human without decreasing either\u2019s task performance. Workload is related to task performance; thus, a robot may use a human\u2019s workload state to modify its interactions with the human. Assessing the human\u2019s workload state may also allow for dynamic task (re-)allocation, as a robot can predict whether a task may overload the human and, if so, allocate it elsewhere. A diagnostic workload assessment algorithm that accurately estimates workload using results from two evaluations, one peer based and one supervisory based, is presented. The algorithm correctly classified workload at least 90% of the time when trained on data from the same human--robot teaming paradigm. This algorithm is an initial step toward robots that can adapt their interactions and intelligently (re-)allocate tasks.<\/jats:p>","DOI":"10.1145\/3314387","type":"journal-article","created":{"date-parts":[[2019,6,4]],"date-time":"2019-06-04T16:01:38Z","timestamp":1559664098000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["A Diagnostic Human Workload Assessment Algorithm for Collaborative and Supervisory Human--Robot Teams"],"prefix":"10.1145","volume":"8","author":[{"given":"Jamison","family":"Heard","sequence":"first","affiliation":[{"name":"Vanderbilt University, Nashville, TN"}]},{"given":"Rachel","family":"Heald","sequence":"additional","affiliation":[{"name":"University of Kansas, Lawrence, KS"}]},{"given":"Caroline E.","family":"Harriott","sequence":"additional","affiliation":[{"name":"Vanderbilt University, Nashville, TN"}]},{"given":"Julie A.","family":"Adams","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR"}]}],"member":"320","published-online":{"date-parts":[[2019,6,3]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1177\/001872088702900204"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916)","volume":"16","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , 2016 . Tensorflow: A system for large-scale machine learning . In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916) , Vol. 16 . 265--283. Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916), Vol. 16. 265--283."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting","volume":"58","author":"Abbass H. A.","unstructured":"H. A. Abbass , J. Tang , R. Amin , M. Ellejmi , and S. Kirby . 2014. Augmented cognition using real-time EEG-based adaptive strategies for air traffic control . In Proceedings of the Human Factors and Ergonomics Society Annual Meeting , Vol. 58 . Sage Publications, 230--234. H. A. Abbass, J. Tang, R. Amin, M. Ellejmi, and S. Kirby. 2014. Augmented cognition using real-time EEG-based adaptive strategies for air traffic control. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 58. Sage Publications, 230--234."},{"key":"e_1_2_1_4_1","first-page":"19","article-title":"New capabilities of the Armys maintenance manpower modeling tool","volume":"26","author":"Archer S.","year":"2005","unstructured":"S. Archer , M. Gosakan , P. Shorter , and J. Lockett . 2005 . New capabilities of the Armys maintenance manpower modeling tool . J. Int. Test Eval. Assoc. 26 , 1 (2005), 19 -- 26 . S. Archer, M. Gosakan, P. Shorter, and J. Lockett. 2005. New capabilities of the Armys maintenance manpower modeling tool. J. Int. Test Eval. Assoc. 26, 1 (2005), 19--26.","journal-title":"J. Int. Test Eval. Assoc."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/0003-6870(92)90152-L"},{"key":"e_1_2_1_6_1","volume-title":"International Conference on User Modeling","author":"Berthold A.","year":"1999","unstructured":"A. Berthold and A. Jameson . 1999. Interpreting symptoms of cognitive load in speech input . In International Conference on User Modeling ( 1999 ), 235--244. A. Berthold and A. Jameson. 1999. Interpreting symptoms of cognitive load in speech input. In International Conference on User Modeling (1999), 235--244."},{"key":"e_1_2_1_7_1","volume-title":"Pattern Recognition","author":"Bishop C. M","unstructured":"C. M Bishop . 2006. Pattern Recognition . Vol. 128 . Springer-Verlag , New York, NY . C. M Bishop. 2006. Pattern Recognition. Vol. 128. Springer-Verlag, New York, NY."},{"key":"e_1_2_1_8_1","first-page":"21","article-title":"Speech measures indicating workload demand","volume":"65","author":"Brenner M.","year":"1994","unstructured":"M. Brenner , E. Doherty , and T. Shipp . 1994 . Speech measures indicating workload demand . Aviat. Space Environ. Med. 65 , 1 (1994), 21 -- 26 . M. Brenner, E. Doherty, and T. Shipp. 1994. Speech measures indicating workload demand. Aviat. Space Environ. Med. 65, 1 (1994), 21--26.","journal-title":"Aviat. Space Environ. Med."},{"key":"e_1_2_1_9_1","unstructured":"B. Cain. 2007. A Review of Mental Workoad Literature. Techreport RTO-TR-HFM-121-Part-II. Defence Research and Development Toronto.  B. Cain. 2007. A Review of Mental Workoad Literature. Techreport RTO-TR-HFM-121-Part-II. Defence Research and Development Toronto."},{"key":"e_1_2_1_10_1","volume-title":"GARTEUR Handbook of Mental Workload Measurement","author":"Castor M.","unstructured":"M. Castor . 2003. GARTEUR Handbook of Mental Workload Measurement . Group for Aeronautical Research and Technology in Europe. M. Castor. 2003. GARTEUR Handbook of Mental Workload Measurement. Group for Aeronautical Research and Technology in Europe."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2011.07.091"},{"key":"e_1_2_1_12_1","unstructured":"J. Raymond Comstock and R. J. Arnegard. 1992. The Multi-Attribute Task Battery for Operator Workload and Strategic Behavior Research. Technical Report NASA Tech. Memorandum 104174. NASA Langley Research Center.  J. Raymond Comstock and R. J. Arnegard. 1992. The Multi-Attribute Task Battery for Operator Workload and Strategic Behavior Research. Technical Report NASA Tech. Memorandum 104174. NASA Langley Research Center."},{"key":"e_1_2_1_13_1","unstructured":"Aymeric Damien et al. 2016. TFLearn. Retrieved from https:\/\/github.com\/tflearn\/tflearn.  Aymeric Damien et al. 2016. TFLearn. Retrieved from https:\/\/github.com\/tflearn\/tflearn."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-39955-3_18"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000005"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2016.2525002"},{"key":"e_1_2_1_17_1","volume-title":"Szalma","author":"Hancock Peter A.","year":"2008","unstructured":"Peter A. Hancock and James L . Szalma . 2008 . Performance under Stress. Ashgate Publishing, Ltd . Peter A. Hancock and James L. Szalma. 2008. Performance under Stress. Ashgate Publishing, Ltd."},{"key":"e_1_2_1_18_1","first-page":"360","article-title":"A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight","volume":"69","author":"Hankins T. C.","year":"1998","unstructured":"T. C. Hankins and G. F. Wilson . 1998 . A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight . Aviat. Space Environ. Med. 69 , 4 (1998), 360 -- 367 . T. C. Hankins and G. F. Wilson. 1998. A comparison of heart rate, eye activity, EEG and subjective measures of pilot mental workload during flight. Aviat. Space Environ. Med. 69, 4 (1998), 360--367.","journal-title":"Aviat. Space Environ. Med."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2013.04.005"},{"key":"e_1_2_1_21_1","volume-title":"Human Performance on the Flight Deck","author":"Harris D.","unstructured":"D. Harris . 2011. Human Performance on the Flight Deck . Ashgate Publishing Limited Surrey , U.K. D. Harris. 2011. Human Performance on the Flight Deck. Ashgate Publishing Limited Surrey, U.K."},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"S. G. Hart and L. E. Staveland. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. 52 1 (1988) 139--183.  S. G. Hart and L. E. Staveland. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. 52 1 (1988) 139--183.","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201917)","author":"Heard J.","unstructured":"J. Heard , C. E. Harriott , and J. A. Adams . 2017. A human workload assessment algorithm for collaborative human-machine teams . In Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201917) . 366--371. J. Heard, C. E. Harriott, and J. A. Adams. 2017. A human workload assessment algorithm for collaborative human-machine teams. In Proceedings of the IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN\u201917). 366--371."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2782483"},{"key":"e_1_2_1_25_1","first-page":"1","article-title":"Mental workload during n-back task-quantified in the prefrontal cortex using fNIRS","volume":"7","author":"Herff C.","year":"2013","unstructured":"C. Herff , D. Heger , O. Fortmann , J. Hennrich , F. Putze , and T. Schultz . 2013 . Mental workload during n-back task-quantified in the prefrontal cortex using fNIRS . Front. Hum. Neurosci. 7 (2013), 1 -- 9 . C. Herff, D. Heger, O. Fortmann, J. Hennrich, F. Putze, and T. Schultz. 2013. Mental workload during n-back task-quantified in the prefrontal cortex using fNIRS. Front. Hum. Neurosci. 7 (2013), 1--9.","journal-title":"Front. Hum. Neurosci."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1519035"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2013.6728565"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1080\/14639221003602473"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"A. Jameson J. Kiefer C. M\u00fcller B. Gro\u00dfmann-Hutter F. Wittig and R. Rummer. 2010. Assessment of a users time pressure and cognitive load on the basis of features of speech. In Resource-Adaptive Processes. Springer-Berlin Heidelberg 171--204.  A. Jameson J. Kiefer C. M\u00fcller B. Gro\u00dfmann-Hutter F. Wittig and R. Rummer. 2010. Assessment of a users time pressure and cognitive load on the basis of features of speech. In Resource-Adaptive Processes. Springer-Berlin Heidelberg 171--204.","DOI":"10.1007\/978-3-540-89408-7_9"},{"key":"e_1_2_1_31_1","first-page":"409","article-title":"BioHarness\u2122 multivariable monitoring device","volume":"11","author":"Johnstone J.","year":"2012","unstructured":"J. Johnstone , P. Ford , G. Hughes , T. Watson , and A. Garrett . 2012 . BioHarness\u2122 multivariable monitoring device : Part. II: Reliability. J. Sports Sci. Med. 11 , 3 (2012), 409 . J. Johnstone, P. Ford, G. Hughes, T. Watson, and A. Garrett. 2012. BioHarness\u2122 multivariable monitoring device: Part. II: Reliability. J. Sports Sci. Med. 11, 3 (2012), 409.","journal-title":"Part. II: Reliability. J. Sports Sci. Med."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1080\/00140139308967976"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2011.02.004"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the International Conference on Learning Representations. 8--13","author":"Kingma D.","unstructured":"D. Kingma and J. Ba . 2015. Adam: A method for stochastic optimization . In Proceedings of the International Conference on Learning Representations. 8--13 . D. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations. 8--13."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/10.83593"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.promfg.2015.07.783"},{"key":"e_1_2_1_37_1","volume-title":"Technical Report ASI-479-024-84B. U.S. Army Research Institution.","author":"McCraken J.","year":"1984","unstructured":"J. McCraken and T. Aldrich . 1984 . Implications of Operator Workload and System Automation Goals . Technical Report ASI-479-024-84B. U.S. Army Research Institution. J. McCraken and T. Aldrich. 1984. Implications of Operator Workload and System Automation Goals. Technical Report ASI-479-024-84B. U.S. Army Research Institution."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.apergo.2009.02.005"},{"key":"e_1_2_1_40_1","first-page":"1581","article-title":"Measuring facial skin temperature changes caused by mental work-load with infrared thermography","volume":"136","author":"Mizuno T.","year":"2016","unstructured":"T. Mizuno , T. Sakai , S. Kawazura , H. Asano , K. Akehi , S. Matsuno , K. Mito , Y. Kume , and N. Itakura . 2016 . Measuring facial skin temperature changes caused by mental work-load with infrared thermography . IEEE Trans. Electron. Inf. Syst. 136 , 11 (2016), 1581 -- 1585 . T. Mizuno, T. Sakai, S. Kawazura, H. Asano, K. Akehi, S. Matsuno, K. Mito, Y. Kume, and N. Itakura. 2016. Measuring facial skin temperature changes caused by mental work-load with infrared thermography. IEEE Trans. Electron. Inf. Syst. 136, 11 (2016), 1581--1585.","journal-title":"IEEE Trans. Electron. Inf. Syst."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2015.2458274"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1080\/14639220210199753"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1080\/001401399185054"},{"key":"e_1_2_1_44_1","doi-asserted-by":"crossref","unstructured":"D. Popovic M. Stikic T. Rosenthal D. Klyde and T. Schnell. 2015. Sensitive diagnostic and multifaceted mental workload classifier (PHYSIOPRINT). In Foundations of Augmented Cognition D.D. Schmorrow and C. M. Fidopiastis (Eds.). Springer-Verlag Switzerland 101--111.  D. Popovic M. Stikic T. Rosenthal D. Klyde and T. Schnell. 2015. Sensitive diagnostic and multifaceted mental workload classifier (PHYSIOPRINT). In Foundations of Augmented Cognition D.D. Schmorrow and C. M. Fidopiastis (Eds.). Springer-Verlag Switzerland 101--111.","DOI":"10.1007\/978-3-319-20816-9_11"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2010.913"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1620509.1620531"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/0301-0511(92)90018-P"},{"key":"e_1_2_1_48_1","volume-title":"Dylan D. Schmorrow and Cali M. Fidopiastis (Eds.)","author":"Rusnock C.","unstructured":"C. Rusnock , B. Borghetti , and I. McQuaid . 2015. Objective-analytical measures of workload -- the third pillar of workload triangulation? In Foundations of Augmented Cognition , Dylan D. Schmorrow and Cali M. Fidopiastis (Eds.) . Springer Science . Switzerland, 124--135. C. Rusnock, B. Borghetti, and I. McQuaid. 2015. Objective-analytical measures of workload -- the third pillar of workload triangulation? In Foundations of Augmented Cognition, Dylan D. Schmorrow and Cali M. Fidopiastis (Eds.). Springer Science. Switzerland, 124--135."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0847-4"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/820752.821584"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting","volume":"37","author":"Sterman M. B.","unstructured":"M. B. Sterman , D. A. Kaiser , C. A. Mann , B. Y. Suyenobu , D. C. Beyma , and J. R. Francis . 1993. Application of quantitative EEG analysis to workload assessment in an advanced aircraft simulator . In Proceedings of the Human Factors and Ergonomics Society Annual Meeting , Vol. 37 . Sage Publications, 118--121. M. B. Sterman, D. A. Kaiser, C. A. Mann, B. Y. Suyenobu, D. C. Beyma, and J. R. Francis. 1993. Application of quantitative EEG analysis to workload assessment in an advanced aircraft simulator. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 37. Sage Publications, 118--121."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/0301-0511(95)05101-5"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2009.2035301"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/001872088702900205"},{"key":"e_1_2_1_55_1","volume-title":"An Introduction to Human Factors Engineering","author":"Wickens C. D.","unstructured":"C. D. Wickens , J. D. Lee , Y. Liu , and S. E. Gordon Becker . 2004. An Introduction to Human Factors Engineering ( 2 nd ed.). Pearson Education, Inc. C. D. Wickens, J. D. Lee, Y. Liu, and S. E. Gordon Becker. 2004. An Introduction to Human Factors Engineering (2nd ed.). Pearson Education, Inc.","edition":"2"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the IEEE Chinese Control Conference. 4700--4705","author":"Yin Z.","unstructured":"Z. Yin and J. Zhang . 2014. Recognition of mental workload levels by combining adaptive exponential feature smoothing and locality preservation projection techniques . In Proceedings of the IEEE Chinese Control Conference. 4700--4705 . Z. Yin and J. Zhang. 2014. Recognition of mental workload levels by combining adaptive exponential feature smoothing and locality preservation projection techniques. In Proceedings of the IEEE Chinese Control Conference. 4700--4705."}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3314387","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3314387","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:29Z","timestamp":1750204409000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3314387"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,3]]},"references-count":53,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,6,30]]}},"alternative-id":["10.1145\/3314387"],"URL":"https:\/\/doi.org\/10.1145\/3314387","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,3]]},"assertion":[{"value":"2018-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-06-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}