{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,19]],"date-time":"2025-11-19T11:18:05Z","timestamp":1763551085275,"version":"3.41.2"},"reference-count":52,"publisher":"ASME International","issue":"2","content-domain":{"domain":["asmedigitalcollection.asme.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2014,6,1]]},"abstract":"<jats:p>Usability of in-vehicle systems has become increasingly important for ease of operations and safety of driving. The user interface (UI) of in-vehicle systems is a critical focus of usability study. This paper studies how to use advanced computational, physiology- and behavior-based tools and methodologies to determine affective\/emotional states and behavior of an individual in real time and in turn how to adapt the human-vehicle interaction to meet users' cognitive needs based on the real-time assessment. Specifically, we set up a set of physiological sensors that are capable of collecting EEG, facial EMG, skin conductance response, and respiration data and a set of motion sensing and tracking equipment that is capable of capturing eye ball movement and objects which the user is interacting with. All hardware components and software are integrated into an augmented sensor platform that can perform as \u201cone coherent system\u201d to enable multimodal data processing and information inference for context-aware analysis of emotional states and cognitive behavior based on the rough set inference engine. Meanwhile subjective data are also recorded for comparison. A usability study of in-vehicle system UI is shown to demonstrate the potential of the proposed methodology.<\/jats:p>","DOI":"10.1115\/1.4026222","type":"journal-article","created":{"date-parts":[[2013,12,11]],"date-time":"2013-12-11T17:36:40Z","timestamp":1386783400000},"update-policy":"https:\/\/doi.org\/10.1115\/crossmarkpolicy-asme","source":"Crossref","is-referenced-by-count":10,"title":["Augmented Affective-Cognition for Usability Study of In-Vehicle System User Interface"],"prefix":"10.1115","volume":"14","author":[{"given":"Feng","family":"Zhou","sequence":"first","affiliation":[{"name":"Mem. ASME The G.W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology, 813 Ferst Drive, Atlanta, GA 30332-0405 e-mail:"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yangjian","family":"Ji","sequence":"additional","affiliation":[{"name":"Industrial Engineering Centre, Department of Mechanical Engineering, Zhejiang University, 38 Zheda Rd, Hangzhou, Zhejiang 310027, China e-mail:"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roger J.","family":"Jiao","sequence":"additional","affiliation":[{"name":"The G.W. Woodruff School of Mechanical Engineering, Georgia Institute of Technology, 813 Ferst Drive, Atlanta, GA 30332-0405 e-mail:"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"33","published-online":{"date-parts":[[2014,2,12]]},"reference":[{"key":"2019100601465556300_B1","doi-asserted-by":"crossref","unstructured":"Kim, S., Dey, A. K., Lee, J., and Forlizzi, J., 2011, \u201cUsability of Car Dashboard Displays for Elder Drivers,\u201d Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Vancouver, BC, Canada, pp. 493\u2013502.","DOI":"10.1145\/1978942.1979011"},{"issue":"4","key":"2019100601465556300_B2","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1016\/j.apergo.2010.09.013","article-title":"A Usability Evaluation Toolkit for In-Vehicle Information Systems (IVISs)","volume":"42","year":"2011","journal-title":"Appl. Ergon."},{"key":"2019100601465556300_B3","doi-asserted-by":"crossref","unstructured":"Pauzi\u00e9, A., 2008, \u201cEvaluating Driver Mental Workload Using the Driving Activity Load Index (DALI),\u201d European Conference on Human Centred Design for Intelligent Transport Systems, Lyon, France.","DOI":"10.1049\/iet-its:20080023"},{"key":"2019100601465556300_B4","first-page":"379","article-title":"Driver distraction: A review of the literature","volume-title":"Distracted Driving","year":"2007"},{"key":"2019100601465556300_B5","doi-asserted-by":"crossref","unstructured":"Hix, D., Swan, E. J., Gabbard, J. L., McGee, M., Durbin, J., and King, T., \u201cUser-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment,\u201d Proc. IEEE VR'99 Conference, IEEE Computer Society Press, pp. 96\u2013103.","DOI":"10.1109\/VR.1999.756939"},{"key":"2019100601465556300_B6","doi-asserted-by":"crossref","unstructured":"Gabbard, J. L., Swan, J. E., Hix, D., Lanzagorta, M., Livingston, M., Brown, D., and Julier, S., 2002, \u201cUsability Engineering: Domain Analysis Activities for Augmented Reality Systems,\u201d 9th Annual Engineering Reality of Virtual Reality Conference, San Jose, CA.","DOI":"10.1117\/12.468073"},{"volume-title":"Usability Engineering: Scenario-Based Development of Human-Computer Interaction","year":"2002","key":"2019100601465556300_B7"},{"volume-title":"Developing User Interfaces: Ensuring Usability through Product & Process","year":"1993","key":"2019100601465556300_B8"},{"key":"2019100601465556300_B9","unstructured":"Gabbard, J. L., Hix, D., Swan, J. E., Livingston, M. A., H\u00f6llerer, T. H., Julier, S. J., Brown, D., and Baillot, Y., 2003, \u201cUsability Engineering for Complex Interactive Systems Development,\u201d Human Systems Integration Symposium, Vienna, VA."},{"issue":"1","key":"2019100601465556300_B10","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/s00163-010-0096-z","article-title":"Fundamentals of Product Ecosystem Design for User Experience","volume":"22","year":"2011","journal-title":"Res. Eng. Des."},{"issue":"5","key":"2019100601465556300_B11","first-page":"1047","article-title":"Affective and Cognitive Design for Mass Personalization: Status and Prospect","volume":"24","year":"2012","journal-title":"J. Intell. Manuf."},{"issue":"2","key":"2019100601465556300_B12","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1016\/j.concog.2008.03.019","article-title":"Conscious Emotional Experience Emerges as a Function of Multilevel, Appraisal-Driven Response Synchronization","volume":"17","year":"2008","journal-title":"Conscious Cogn."},{"issue":"3","key":"2019100601465556300_B13","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1093\/abm\/16.3.199","article-title":"Ecological Momentary Assessment (EMA) in Behavioral Medicine","volume":"16","year":"1994","journal-title":"Ann. Behav. Med."},{"issue":"1","key":"2019100601465556300_B14","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1207\/S15326985EP3801_8","article-title":"Cognitive Load Measurement as a Means to Advance Cognitive Load Theory","volume":"38","year":"2003","journal-title":"Educ. Psychol."},{"key":"2019100601465556300_B15","doi-asserted-by":"crossref","unstructured":"Haapalainen, E., Kim, S., Forlizzi, J. F., and Dey, A. K., 2010, \u201cPsycho-Physiological Measures for Assessing Cognitive Load,\u201d Proceedings of the 12th ACM international conference on Ubiquitous computing, ACM, Copenhagen, Denmark, pp. 301\u2013310.","DOI":"10.1145\/1864349.1864395"},{"volume-title":"Protocol Analysis: Verbal Reports as Data","year":"1993","key":"2019100601465556300_B16"},{"key":"2019100601465556300_B17","unstructured":"Detenber, B. H., 2001, \u201cMeasuring Emotional Responses in Human Factors Research: Some Theoretical and Practical Considerations,\u201d Proceedings of the International Conference on Affective Human Factors Design, M.Helander, H.Khalid, and M.Tham, eds., Asean Academic Press, London, pp. 124\u2013130."},{"issue":"14","key":"2019100601465556300_B18","doi-asserted-by":"crossref","first-page":"966","DOI":"10.1080\/00140130210166951","article-title":"Subjective Rating Scales: Science or Art?","volume":"45","year":"2002","journal-title":"Ergonomics"},{"key":"2019100601465556300_B19","unstructured":"Reid, G. B., Shingledecker, C. A., Nygren, T. E., and Eggemeier, F. T., 1981, \u201cDevelopment of Multidimensional Subjective Measures of Workload,\u201d IEEE International Conference on Cybernetics & Society, Atlanta, GA."},{"article-title":"Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research","volume-title":"Human Mental Workload","year":"1988","key":"2019100601465556300_B20"},{"volume-title":"Usability Engineering","year":"1994","key":"2019100601465556300_B21"},{"issue":"3","key":"2019100601465556300_B22","first-page":"190","article-title":"Is It Possible to Have a Single Measure for All Work?","volume":"6","year":"1999","journal-title":"Int. J. Indus. Eng. Theory"},{"key":"2019100601465556300_B23","first-page":"85","article-title":"Measuring the Effects of Driver Distraction: Direct Driving Performance Methods and Measures","volume-title":"Driver Distraction: Theory, Effects and Mitigation","year":"2009"},{"issue":"2","key":"2019100601465556300_B24","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1016\/j.apergo.2005.06.002","article-title":"Effect of Display Position of a Visual In-Vehicle Task on Simulated Driving","volume":"37","year":"2006","journal-title":"Appl. Ergon."},{"issue":"3","key":"2019100601465556300_B25","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/j.apergo.2004.01.005","article-title":"On the Highway Measures of Driver Glance Behavior With An Example Automobile Navigation System","volume":"35","year":"2004","journal-title":"Appl. Ergon."},{"issue":"3","key":"2019100601465556300_B26","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/j.apergo.2008.11.014","article-title":"Evidence for the Selective Attention Mechanism and Dual-Task Interference","volume":"40","year":"2009","journal-title":"Appl. Ergon."},{"issue":"12","key":"2019100601465556300_B27","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1016\/j.ijhcs.2011.07.005","article-title":"Affect Prediction From Physiological Measures Via Visual Stimuli","volume":"69","year":"2011","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"2019100601465556300_B28","doi-asserted-by":"crossref","unstructured":"Schiano, D. J., Ehrlich, S. M., and Sheridan, K., 2004, \u201cCategorical Imperative NOT: Facial Affect is Perceived Continuously,\u201d SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria.","DOI":"10.1145\/985692.985699"},{"issue":"5","key":"2019100601465556300_B29","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/j.ijhcs.2007.10.011","article-title":"Real-Time Classification of Evoked Emotions Using Facial Feature Tracking and Physiological Responses","volume":"66","year":"2008","journal-title":"Int. J. Hum.-Comput. Stud."},{"issue":"1-2","key":"2019100601465556300_B30","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1016\/j.intcom.2008.10.011","article-title":"Fundamentals of Physiological Computing","volume":"21","year":"2009","journal-title":"Interact. Comput."},{"issue":"4","key":"2019100601465556300_B31","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1016\/j.ijhcs.2006.11.011","article-title":"A Fuzzy Physiological Approach for Continuously Modeling Emotion During Interaction With Play Technologies","volume":"65","year":"2007","journal-title":"Int. J. Hum.-Comput. Stud."},{"volume-title":"Electroencephalography: Basic Principles, Clinical Applications, and Related Fields","year":"2004","key":"2019100601465556300_B32"},{"year":"2008","key":"2019100601465556300_B33","article-title":"EEG-Based Emotion Recognition: The Influence of Visual and Auditory Stimuli"},{"issue":"3","key":"2019100601465556300_B34","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1111\/j.1469-8986.1993.tb03352.x","article-title":"Looking at Pictures: Affective, Facial, Visceral, and Behavioral Reactions","volume":"30","year":"1993","journal-title":"Psychophysiology"},{"key":"2019100601465556300_B35","doi-asserted-by":"crossref","unstructured":"Ablassmeier, M., Poitschke, T., Wallhoff, F., Bengler, K., and Rigoll, G., 2007, \u201cEye Gaze Studies Comparing Head-Up and Head-Down Displays in Vehicles,\u201d IEEE International Conference on Multimedia and Expo, Beijing, China.","DOI":"10.1109\/ICME.2007.4285134"},{"volume-title":"Rough Sets: Theoretical Aspects of Reasoning About Data","year":"1991","key":"2019100601465556300_B36"},{"issue":"1","key":"2019100601465556300_B37","first-page":"109","article-title":"Uncertainty Measures of Rough Set Prediction","volume":"106","year":"1998","journal-title":"AI Commun."},{"issue":"10","key":"2019100601465556300_B38","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1109\/34.954607","article-title":"Toward Machine Emotional Intelligence: Analysis of Affective Physiological State","volume":"23","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"2019100601465556300_B39","first-page":"70","article-title":"The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms","volume":"AU-15","year":"1967","journal-title":"IEEE Trans. Audio Electroacoust."},{"year":"2002","key":"2019100601465556300_B40","article-title":"A Survey of Dimension Reduction Techniques"},{"issue":"2","key":"2019100601465556300_B41","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1109\/34.908974","article-title":"PCA Versus LDA","volume":"23","year":"2001","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"issue":"4","key":"2019100601465556300_B42","doi-asserted-by":"crossref","first-page":"376","DOI":"10.1111\/j.1469-1809.1938.tb02189.x","article-title":"The Statistical Utilization of Multiple Measurements","volume":"8","year":"1938","journal-title":"Ann. Eugen."},{"first-page":"592","volume-title":"RSES and RSESlib\u2014A Collection of Tools for Rough Set Computations","year":"2000","key":"2019100601465556300_B43"},{"issue":"4","key":"2019100601465556300_B44","first-page":"369","article-title":"Riona: A New Classification System Combining Rule Induction and Instance-Based Learning","volume":"51","year":"2002","journal-title":"Fund. Inform."},{"issue":"1","key":"2019100601465556300_B45","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1037\/0003-066X.45.1.16","article-title":"Inferring Psychological Significance From Physiological Signals","volume":"45","year":"1990","journal-title":"Am. Psychol."},{"key":"2019100601465556300_B46","doi-asserted-by":"crossref","unstructured":"Gu, Y., Tan, S. L., Wong, K. J., Ho, M.-H. R., and Qu, L., 2010, \u201cA GMM Based 2-Stage Architecture for Multi-Subject Emotion Recognition Using Physiological Responses,\u201d The 1st Augmented Human International Conference, Meg\u00e8ve, France.","DOI":"10.1145\/1785455.1785458"},{"journal-title":"IEEE Trans. Affect. Comput.","article-title":"A Hierarchical Bayesian Model for Affective Parameter Shaping in User Experience Prospect Evaluation","year":"2013","key":"2019100601465556300_B47"},{"issue":"2","key":"2019100601465556300_B48","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.trf.2005.04.014","article-title":"Sensitivity of Eye-Movement Measures to In-Vehicle Task Difficulty","volume":"8","year":"2005","journal-title":"Transp. Res. Part F"},{"year":"2006","key":"2019100601465556300_B49","article-title":"The 100-Car Naturalistic Driving Study, Phase II\u2014Results of the 100-Car Field Experiment"},{"key":"2019100601465556300_B50","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1126\/science.1168085","article-title":"Can Technology Get Your Eyes Back on the Road?","volume":"324","year":"2009","journal-title":"Science"},{"key":"2019100601465556300_B51","first-page":"62","article-title":"Fatigue Indicators of Drowsy Drivers Based on Analysis of Physiological Signals","volume-title":"Medical Data Analysis","year":"2001"},{"key":"2019100601465556300_B52","unstructured":"Grace, R., Byrne, V. E., Bierman, D. M., Legrand, J.-M., Gricourt, D., Davis, B. K., Staszewski, J. J., and Carnahan., B., 1998, \u201cA Drowsy Driver Detection System for Heavy Vehicles,\u201d 17th Digital Avionics Systems Conference, Seattle, WA."}],"container-title":["Journal of Computing and Information Science in Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/doi\/10.1115\/1.4026222\/6099446\/jcise_014_02_021001.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/doi\/10.1115\/1.4026222\/6099446\/jcise_014_02_021001.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,1]],"date-time":"2025-05-01T04:18:58Z","timestamp":1746073138000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article\/doi\/10.1115\/1.4026222\/371429\/Augmented-AffectiveCognition-for-Usability-Study"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,2,12]]},"references-count":52,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2014,6,1]]}},"URL":"https:\/\/doi.org\/10.1115\/1.4026222","relation":{},"ISSN":["1530-9827","1944-7078"],"issn-type":[{"type":"print","value":"1530-9827"},{"type":"electronic","value":"1944-7078"}],"subject":[],"published":{"date-parts":[[2014,2,12]]},"article-number":"021001"}}