{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T21:10:04Z","timestamp":1777669804960,"version":"3.51.4"},"reference-count":81,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T00:00:00Z","timestamp":1667347200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/100000104","name":"National Aeronautics and Space Administration","doi-asserted-by":"publisher","award":["80NSSC19K1052"],"award-info":[{"award-number":["80NSSC19K1052"]}],"id":[{"id":"10.13039\/100000104","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Hum Factors"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We use a set of unobtrusive measures to estimate subjectively reported trust, mental workload, and situation awareness (henceforth \u201cTWSA\u201d).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Subjective questionnaires are commonly used to assess human cognitive states. However, they are obtrusive and usually impractical to administer during operations. Measures derived from actions operators take while working (which we call \u201cembedded measures\u201d) have been proposed as an unobtrusive way to obtain TWSA estimates. Embedded measures have not been systematically investigated for each of TWSA, which prevents their operational utility.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Fifteen participants completed twelve trials of spaceflight-relevant tasks while using a simulated autonomous system. Embedded measures of TWSA were obtained during each trial and participants completed TWSA questionnaires after each trial. Statistical models incorporating our embedded measures were fit with various formulations, interaction effects, and levels of personalization to understand their benefits and improve model accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The stepwise algorithm for building statistical models usually included embedded measures, which frequently corresponded to an intuitive increase or decrease in reported TWSA. Embedded measures alone could not accurately capture an operator\u2019s cognitive state, but combining the measures with readily observable task information or information about participants\u2019 backgrounds enabled the models to achieve good descriptive fit and accurate prediction of TWSA.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Statistical models leveraging embedded measures of TWSA can be used to accurately estimate responses on subjective questionnaires that measure TWSA.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Application<\/jats:title>\n                    <jats:p>Our systematic approach to investigating embedded measures and fitting models allows for cognitive state estimation without disrupting tasks when administering questionnaires would be impractical.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/00187208221129371","type":"journal-article","created":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T08:03:49Z","timestamp":1667376229000},"page":"1142-1160","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":15,"title":["Estimation of Subjectively Reported Trust, Mental Workload, and Situation Awareness Using Unobtrusive Measures"],"prefix":"10.1177","volume":"65","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9444-7409","authenticated-orcid":false,"given":"Jacob R.","family":"Kintz","sequence":"first","affiliation":[{"name":"Smead Department of Aerospace Engineering Sciences, University of Colorado\u2013Boulder, Boulder, Colorado, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6825-917X","authenticated-orcid":false,"given":"Neil T.","family":"Banerjee","sequence":"additional","affiliation":[{"name":"Smead Department of Aerospace Engineering Sciences, University of Colorado\u2013Boulder, Boulder, Colorado, USA"}]},{"given":"Johnny Y.","family":"Zhang","sequence":"additional","affiliation":[{"name":"Smead Department of Aerospace Engineering Sciences, University of Colorado\u2013Boulder, Boulder, Colorado, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7808-8557","authenticated-orcid":false,"given":"Allison P.","family":"Anderson","sequence":"additional","affiliation":[{"name":"Smead Department of Aerospace Engineering Sciences, University of Colorado\u2013Boulder, Boulder, Colorado, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9345-9712","authenticated-orcid":false,"given":"Torin K.","family":"Clark","sequence":"additional","affiliation":[{"name":"Smead Department of Aerospace Engineering Sciences, University of Colorado\u2013Boulder, Boulder, Colorado, USA"}]}],"member":"179","published-online":{"date-parts":[[2022,11,2]]},"reference":[{"key":"bibr1-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/ICHMS53169.2021.9582622"},{"key":"bibr2-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1145\/3132743"},{"key":"bibr3-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/MCS.2020.3019151"},{"key":"bibr4-00187208221129371","doi-asserted-by":"publisher","DOI":"10.3357\/AMHP.4343.2015"},{"key":"bibr5-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2012.6232134"},{"key":"bibr6-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/IS.2012.6335127"},{"key":"bibr7-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/001872088302500602"},{"key":"bibr8-00187208221129371","unstructured":"Casner S. M., Gore B. F. (2010). Measuring and evaluating workload: A primer. NASA Technical Memorandum, 216395, 2010."},{"key":"bibr9-00187208221129371","volume-title":"ordinal\u2014Regression models for ordinal data","author":"Christensen R. H. B.","year":"2019"},{"key":"bibr10-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/1071181319631517"},{"key":"bibr11-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-018-0527-6"},{"key":"bibr12-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2020.1759699"},{"key":"bibr13-00187208221129371","doi-asserted-by":"publisher","DOI":"10.2514\/atcq.6.1.1"},{"key":"bibr14-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/0018720024494856"},{"key":"bibr15-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/NAECON.1988.195097"},{"key":"bibr16-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/154193128803200221"},{"key":"bibr82-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/001872095779049499"},{"key":"bibr18-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/0018720819875376"},{"key":"bibr19-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/0018720815592752"},{"key":"bibr20-00187208221129371","unstructured":"Ezekiel M. (1930). Methods of correlation analysis (pp. xiv, 427). Wiley."},{"key":"bibr21-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/0018720812443983"},{"key":"bibr22-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-01818-3_25"},{"key":"bibr23-00187208221129371","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2018.00073"},{"key":"bibr24-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/0278364918778344"},{"key":"bibr25-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-020-00703-3"},{"key":"bibr26-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/1071181319631201"},{"key":"bibr27-00187208221129371","doi-asserted-by":"publisher","DOI":"10.2514\/1.A32267"},{"key":"bibr28-00187208221129371","volume-title":"Workload and performance: Associations, insensitivities, and dissociations: Human Factors","author":"Hancock P. A.","year":"2018"},{"key":"bibr29-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1016\/0001-4575(90)90019-H"},{"key":"bibr30-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"bibr31-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/1555343419847906"},{"key":"bibr32-00187208221129371","first-page":"1","author":"Heard J.","year":"2020","journal-title":"Management (CogSIMA)"},{"key":"bibr33-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2782483"},{"key":"bibr34-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/001872087902100201"},{"key":"bibr35-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2759758"},{"key":"bibr36-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1016\/j.ifacol.2016.12.188"},{"key":"bibr37-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1207\/S15327566IJCE0401_04"},{"key":"bibr38-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/001872097778543831"},{"key":"bibr39-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/AERO.2016.7500734"},{"key":"bibr40-00187208221129371","doi-asserted-by":"publisher","DOI":"10.2514\/6.2017-1314"},{"key":"bibr41-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/001872086300500206"},{"key":"bibr42-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1007\/s43154-020-00029-y"},{"key":"bibr43-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2018.1547842"},{"key":"bibr44-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50_30392"},{"issue":"1","key":"bibr45-00187208221129371","first-page":"31","volume":"49","author":"Lin L.-W.","year":"2017","journal-title":"Journal of Aeronautics, Astronautics and Aviation"},{"key":"bibr46-00187208221129371","doi-asserted-by":"publisher","DOI":"10.3233\/BME-141044"},{"key":"bibr47-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/2327857914031019"},{"key":"bibr48-00187208221129371","unstructured":"Mangiafico S. S. (2016). Summary and analysis of extension program evaluation in R. Rutgers Cooperative Extension: New Brunswick, NJ, USA, 125, 16\u201322."},{"key":"bibr49-00187208221129371","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2019.00225"},{"key":"bibr50-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/78.3.691"},{"key":"bibr51-00187208221129371","volume-title":"Psychometric Theory\/Jum C. Nunnaly","author":"Nunnally J. C.","year":"1978"},{"key":"bibr52-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/001872097778543886"},{"key":"bibr53-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/155534308X284417"},{"key":"bibr54-00187208221129371","doi-asserted-by":"publisher","DOI":"10.4271\/12-02-02-0009"},{"key":"bibr55-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.1988.10509698"},{"key":"bibr56-00187208221129371","volume-title":"R: A language and environment for statistical computing","author":"R Core Team","year":"2019"},{"key":"bibr57-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/01466216970214001"},{"key":"bibr58-00187208221129371","unstructured":"Reaction Time Test. (n.d.). Human Benchmark. https:\/\/www.humanbenchmark.com\/tests\/reactiontime."},{"key":"bibr59-00187208221129371","volume-title":"AGARDograph No. 246; Survey of Methods to Assess Workload","author":"Roscoe A. H.","year":"1979"},{"key":"bibr60-00187208221129371","volume-title":"Proceedings of NATO Advisory Group for Aerospace Research and Development (AGARD)","author":"Roscoe A. H.","year":"1984"},{"key":"bibr61-00187208221129371","volume-title":"A subjective rating scale for assessing pilot workload in flight: A decade of practical use","author":"Roscoe A. H.","year":"1990"},{"key":"bibr62-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/0018720818816838"},{"key":"bibr63-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1109\/SMC.2018.00128"},{"key":"bibr64-00187208221129371","first-page":"1","volume":"478","author":"Selcon S. J.","year":"1990","journal-title":"Situational Awareness in Aerospace Operations (AGARD-CP-478)"},{"key":"bibr65-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/107118191786755706"},{"key":"bibr66-00187208221129371","volume-title":"Towards an empirically developed scale for system trust: Take two: Proceedings of the Human Factors and Ergonomics Society Annual Meeting","author":"Spain R. D.","year":"2008"},{"key":"bibr67-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1016\/S0925-7535(01)00010-8"},{"key":"bibr68-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1080\/14639220902893613"},{"key":"bibr69-00187208221129371","first-page":"1","volume":"478","author":"Taylor R. M.","year":"1990","journal-title":"Situational Awareness in Aerospace Operations (AGARD-CP-478)"},{"key":"bibr70-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-018-0481-3"},{"key":"bibr71-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-21706-2"},{"key":"bibr72-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1080\/14639220500370105"},{"key":"bibr73-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/1071181320641470"},{"key":"bibr74-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1177\/001872089303500205"},{"key":"bibr75-00187208221129371","first-page":"221","author":"Xu A.","year":"2015","journal-title":"Proceedings of the Tenth Annual ACM\/IEEE International Conference on Human-Robot Interaction"},{"key":"bibr76-00187208221129371","doi-asserted-by":"crossref","unstructured":"Yang X. J., Schemanske C., Searle C. (2021). Toward quantifying trust dynamics: How people adjust their trust after moment-to-moment interaction with automation. Human Factors, 00187208211034716. https:\/\/doi.org\/10.1177\/00187208211034716","DOI":"10.1177\/00187208211034716"},{"issue":"1","key":"bibr77-00187208221129371","first-page":"408","volume":"1","author":"Yang X. J.","year":"2017","journal-title":"Proceedings of the 2017 ACM\/IEEE International Conference on Human-Robot Interaction"},{"key":"bibr78-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1002\/cne.920180503"},{"key":"bibr79-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1145\/3171221.3171281"},{"key":"bibr80-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2014.956151"},{"key":"bibr81-00187208221129371","doi-asserted-by":"publisher","DOI":"10.1518\/0018720024497709"}],"container-title":["Human Factors: The Journal of the Human Factors and Ergonomics Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00187208221129371","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/00187208221129371","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00187208221129371","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T07:53:47Z","timestamp":1777449227000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/00187208221129371"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,2]]},"references-count":81,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["10.1177\/00187208221129371"],"URL":"https:\/\/doi.org\/10.1177\/00187208221129371","relation":{},"ISSN":["0018-7208","1547-8181"],"issn-type":[{"value":"0018-7208","type":"print"},{"value":"1547-8181","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,2]]}}}