{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T18:10:33Z","timestamp":1773943833503,"version":"3.50.1"},"reference-count":86,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T00:00:00Z","timestamp":1602720000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Hum Factors"],"published-print":{"date-parts":[[2021,8]]},"abstract":"<jats:sec><jats:title>Objective<\/jats:title><jats:p> To investigate speech features, human\u2013machine alarms, and operator\u2013system interaction for the estimation of cognitive workload in full-scale realistic simulated scenarios. <\/jats:p><\/jats:sec><jats:sec><jats:title>Background<\/jats:title><jats:p> Theories and models of cognitive workload are critical for the design and evaluation of human\u2013machine systems. Unfortunately, there are very few nonintrusive cognitive workload measures available for realistic dynamic human\u2013machine interaction. <\/jats:p><\/jats:sec><jats:sec><jats:title>Method<\/jats:title><jats:p> The study was conducted in a full-scope control room research simulator of an advanced nuclear reactor. Six crews, each consisting of three operators, participated in 12 scenarios. The operators rated their workload every second minute. Machine learning algorithms were trained to estimate operators\u2019 workload based on crew communication, operator\u2013system interaction, and system alarms. <\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p> Random Forest (RF) utilizing speech and system features achieved an accuracy of 67% on test data. Utilizing speech features only, the accuracy achieved was 63%. The most important speech features were pitch, amplitude, and articulation rate. A 61% accuracy was achieved when alarms and operator\u2013system interaction features were used. The most important features were the number of alarms and amount of operator\u2013system interaction. Accuracy for algorithms trained for each operator ranged from 39% to 98%, with an average of 72%. For a majority of analyses performed, RF and extreme gradient boosting (XGB) outperformed other algorithms. <\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p> The results demonstrate that the features investigated and machine learning models developed provide a potential for the dynamic nonintrusive measurement of cognitive workload. <\/jats:p><\/jats:sec><jats:sec><jats:title>Application<\/jats:title><jats:p> The approach presented can be developed for nonintrusive workload measurement in real-world human\u2013machine applications, simulator-based training, and research. <\/jats:p><\/jats:sec>","DOI":"10.1177\/0018720820961730","type":"journal-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T18:52:32Z","timestamp":1602787952000},"page":"736-756","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":8,"title":["An Investigation of Speech Features, Plant System Alarms, and Operator\u2013System Interaction for the Classification of Operator Cognitive Workload During Dynamic Work"],"prefix":"10.1177","volume":"63","author":[{"given":"Per \u00d8.","family":"Braarud","sequence":"first","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]},{"given":"Terje","family":"Bodal","sequence":"additional","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]},{"given":"John E.","family":"Hulsund","sequence":"additional","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0226-2357","authenticated-orcid":false,"given":"Michael N.","family":"Louka","sequence":"additional","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]},{"given":"Christer","family":"Nihlwing","sequence":"additional","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]},{"given":"Espen","family":"Nystad","sequence":"additional","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]},{"given":"H\u00e5kan","family":"Svengren","sequence":"additional","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]},{"given":"Emil","family":"Wingstedt","sequence":"additional","affiliation":[{"name":"Institute for Energy Technology, Halden, Norway"}]}],"member":"179","published-online":{"date-parts":[[2020,10,15]]},"reference":[{"key":"bibr1-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2006.04.002"},{"key":"bibr2-0018720820961730","first-page":"115","volume-title":"Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, ACM","author":"Arshad S.","year":"2013"},{"key":"bibr3-0018720820961730","first-page":"341","volume":"5","author":"Boersma P.","year":"2001","journal-title":"Glot International"},{"key":"bibr4-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2019.102904"},{"key":"bibr5-0018720820961730","first-page":"233","volume-title":"Simulator-based human factors studies across 25 years: The history of the Halden Man-Machine Laboratory","author":"Braarud P. \u00d8.","year":"2011"},{"key":"bibr6-0018720820961730","volume-title":"Human-system validation experiment 2018: Multistage assessment, operators self-evaluation, and mental workload assessment","author":"Braarud P. \u00d8.","year":"2020"},{"key":"bibr8-0018720820961730","first-page":"21","volume":"65","author":"Brenner M.","year":"1994","journal-title":"Aviation, Space, and Environmental Medicine"},{"key":"bibr7-0018720820961730","first-page":"239","volume-title":"Vocal fold physiology, biomechanics, acoustics, and phonatory control","author":"Brenner M.","year":"1985"},{"key":"bibr9-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1009213286"},{"key":"bibr10-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1044\/1092-4388(2013\/12-0103)"},{"key":"bibr11-0018720820961730","first-page":"481","volume-title":"Advances in industrial ergonomics and safety I","author":"Byers J. C.","year":"1989"},{"key":"bibr12-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1037\/xhp0000638"},{"key":"bibr13-0018720820961730","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.40.4.988"},{"key":"bibr14-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2013.6637699"},{"key":"bibr15-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1145\/2395123.2395127"},{"key":"bibr16-0018720820961730","unstructured":"Chen T., Benesty M., Khotilovich V., Tang Y., Cho H., Chen K., Mitchell R., Cano I., Zhou T., Li M., Xie J., Lin M., Geng Y., Li Y., He T. (2019). xgboost: Extreme gradient boosting. R package version 0.90.0.2. https:\/\/CRAN.R-project.org\/package=xgboost"},{"key":"bibr17-0018720820961730","doi-asserted-by":"publisher","DOI":"10.2345\/0899-8205-46.4.268"},{"key":"bibr18-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017197"},{"key":"bibr19-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-75555-5_52"},{"key":"bibr20-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1109\/CCMB.2014.7020692"},{"key":"bibr21-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2013.790483"},{"key":"bibr22-0018720820961730","volume-title":"Handbook of perception and human performance, vol II, cognitive processes and performance","author":"Gopher D.","year":"1986"},{"key":"bibr23-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/1541931215591373"},{"key":"bibr24-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1080\/1463922X.2013.869371"},{"key":"bibr25-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-61061-0_1"},{"key":"bibr90-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/0018720818809590"},{"key":"bibr87-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/001872088903100503"},{"key":"bibr26-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"bibr88-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/S0301-0511(96)05223-4"},{"key":"bibr28-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/0018720819825803"},{"key":"bibr29-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/001872089403600101"},{"key":"bibr30-0018720820961730","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1881.001.0001"},{"key":"bibr31-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.apergo.2010.08.005"},{"key":"bibr32-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1121\/1.3543948"},{"key":"bibr33-0018720820961730","unstructured":"Institute for Energy Technology (IFE). (2019). HAMMLAB. Institute for Energy Technology. https:\/\/ife.no\/en\/laboratory\/hammlab\/"},{"key":"bibr34-0018720820961730","volume-title":"ISO 11064-7, 2006. Ergonomic design of control centres part 7: Principles for the evaluation of control centres","author":"International Organization for Standardization (ISO)","year":"2006"},{"key":"bibr35-0018720820961730","volume-title":"Road vehicles\u2014Transport information and control systems\u2014Detection-response task (DRT) for assessing attentional effects of cognitive load in driving","author":"International Organization for Standardization (ISO)","year":"2016"},{"key":"bibr36-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.wocn.2018.07.001"},{"key":"bibr37-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-7138-7"},{"key":"bibr39-0018720820961730","volume-title":"DRA Technical Memorandum (CAD5) 92011. DRA Maritime Command Control Division","author":"Jordan C. S","year":"1992"},{"key":"bibr40-0018720820961730","unstructured":"Karatzoglou A., Smola A., Hornik K., Maniscalco M. A., Teo C. H. (2019). Kernlab: Kernel-based machine learning lab. R package version 0.9-29. https:\/\/CRAN.R-project.org\/package=kernlab"},{"key":"bibr41-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/s002139900335"},{"key":"bibr42-0018720820961730","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v028.i05"},{"key":"bibr43-0018720820961730","unstructured":"Kuhn M, Wing J, Weston S, Williams A, Keefer C, Engelhardt A, Cooper T, Mayer Z, Kenkel B, Benesty M, Lescarbeau R, Ziem A, Scrucca L, Tang Y, Candan C, Hunt T, R Core Team. (2019). Caret: classification and regression training. R package version 6.0-84. https:\/\/CRAN.R-project.org\/package=caret"},{"key":"bibr44-0018720820961730","unstructured":"Liaw A., Wiener M. (2018). randomForest: Breiman and Cutler\u2019s random forests for classification and regression. R package version 4.6-14. https:\/\/CRAN.R-project.org\/package=randomForest"},{"key":"bibr45-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1086\/509092"},{"key":"bibr46-0018720820961730","doi-asserted-by":"publisher","DOI":"10.4018\/jthi.2013070106"},{"key":"bibr47-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1121\/1.405815"},{"key":"bibr48-0018720820961730","volume-title":"Proceedings of the 27th International Congress on the Aeronautical Sciences","author":"Luig J.","year":"2010"},{"key":"bibr49-0018720820961730","unstructured":"Majka M. (2019). naivebayes: High performance implementation of the naive Bayes algorithm. R package version 0.9.6. https:\/\/CRAN.R-project.org\/package=naivebayes"},{"key":"bibr50-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814539505"},{"key":"bibr89-0018720820961730","volume":"4","author":"Matthews G.","year":"2020","journal-title":"Theoretical Issues in Ergonomics Science"},{"key":"bibr51-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/0018720819856454"},{"key":"bibr52-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2019.8857501"},{"key":"bibr54-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/0001-6918(67)90048-0"},{"key":"bibr53-0018720820961730","first-page":"123","volume":"2","author":"Moray N","year":"1988","journal-title":"International Review of Ergonomics"},{"key":"bibr56-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/0301-0511(92)90016-N"},{"key":"bibr58-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1177\/001872089103300102"},{"key":"bibr57-0018720820961730","unstructured":"OECD NEA. (2020). OECD Nuclear Energy Agency. https:\/\/www.oecd-nea.org\/jointproj\/halden.html"},{"key":"bibr59-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2015.03.105"},{"key":"bibr60-0018720820961730","volume-title":"R: A language and environment for statistical computing","author":"R Core Team","year":"2019"},{"key":"bibr62-0018720820961730","first-page":"256","volume":"8814","author":"Ragab A.","year":"2014","journal-title":"Proceedings of the International Conference on Image Analysis and Recognition"},{"key":"bibr61-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2019.101634"},{"key":"bibr63-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62387-0"},{"key":"bibr64-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2018.12.001"},{"key":"bibr65-0018720820961730","unstructured":"Ripley B., Venables W. (2016). nnet: Feed-forward neural networks and multinomial log-linear models. R package version 7.3-12. https:\/\/CRAN.R-project.org\/package=nnet"},{"key":"bibr66-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1097\/ACO.0000000000000260"},{"key":"bibr67-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2016.09.003"},{"key":"bibr68-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1093\/bja\/aeq307"},{"key":"bibr69-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1145\/1240866.1241057"},{"key":"bibr70-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557068"},{"key":"bibr71-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1186\/s41235-019-0166-3"},{"key":"bibr72-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.jairtraman.2014.04.002"},{"key":"bibr73-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/s10648-005-3951-0"},{"key":"bibr74-0018720820961730","doi-asserted-by":"crossref","DOI":"10.21437\/Interspeech.2014-114","volume-title":"Classification of cognitive load from speech using an I-Vector framework","author":"van Segbroeck M.","year":"2014"},{"key":"bibr75-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-21706-2"},{"key":"bibr76-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/S0967-0661(97)00018-X"},{"key":"bibr77-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2018.12.003"},{"key":"bibr78-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1037\/14528-018"},{"key":"bibr79-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.45.4.635.27088"},{"key":"bibr85-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/s00421-002-0625-1"},{"key":"bibr86-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39143-9_39"},{"key":"bibr80-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-019-00553-8"},{"key":"bibr81-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2010.5494987"},{"key":"bibr82-0018720820961730","first-page":"2041","volume-title":"Proceedings IEEE ICASSP,","author":"Yin B.","year":"2007"},{"key":"bibr83-0018720820961730","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2014.956151"}],"container-title":["Human Factors: The Journal of the Human Factors and Ergonomics Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0018720820961730","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0018720820961730","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0018720820961730","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T17:06:33Z","timestamp":1740848793000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0018720820961730"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,15]]},"references-count":86,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2021,8]]}},"alternative-id":["10.1177\/0018720820961730"],"URL":"https:\/\/doi.org\/10.1177\/0018720820961730","relation":{},"ISSN":["0018-7208","1547-8181"],"issn-type":[{"value":"0018-7208","type":"print"},{"value":"1547-8181","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,15]]}}}