{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:59:57Z","timestamp":1782845997348,"version":"3.54.5"},"reference-count":49,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T00:00:00Z","timestamp":1719273600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"BMW Group"},{"name":"European Union Next-Generation EU"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>We investigate personalised feedback mechanisms to help drivers regulate their emotions, aiming to improve road safety. We systematically evaluate driver-preferred feedback modalities and their impact on emotional states. Using unobtrusive vision-based emotion detection and self-labeling, we captured the emotional states and feedback preferences of 21 participants in a simulated driving environment. Results show that in-car feedback systems effectively influence drivers\u2019 emotional states, with participants reporting positive experiences and varying preferences based on their emotions. We also developed a machine learning classification system using facial marker data to demonstrate the feasibility of our approach for classifying emotional states. Our contributions include design guidelines for tailored feedback systems, a systematic analysis of user reactions across three feedback channels with variations, an emotion classification system, and a dataset with labeled face landmark annotations for future research.<\/jats:p>","DOI":"10.3390\/mti8070054","type":"journal-article","created":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T11:09:09Z","timestamp":1719400149000},"page":"54","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Emotion-Aware In-Car Feedback: A Comparative Study"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0652-8503","authenticated-orcid":false,"given":"Kevin Fred","family":"Mwaita","sequence":"first","affiliation":[{"name":"Faculty of Engineering, Free University of Bozen-Bolzano, 39100 Bolzano, BZ, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rahul","family":"Bhaumik","sequence":"additional","affiliation":[{"name":"Faculty of Engineering, Free University of Bozen-Bolzano, 39100 Bolzano, BZ, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aftab","family":"Ahmed","sequence":"additional","affiliation":[{"name":"Faculty of Engineering, Free University of Bozen-Bolzano, 39100 Bolzano, BZ, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5676-3136","authenticated-orcid":false,"given":"Adwait","family":"Sharma","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Bath, Claverton Down, Bath BA2 7AY, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Antonella","family":"De Angeli","sequence":"additional","affiliation":[{"name":"Faculty of Engineering, Free University of Bozen-Bolzano, 39100 Bolzano, BZ, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4522-3155","authenticated-orcid":false,"given":"Michael","family":"Haller","sequence":"additional","affiliation":[{"name":"Faculty of Engineering, Free University of Bozen-Bolzano, 39100 Bolzano, BZ, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Staiano, J., Men\u00e9ndez, M., Battocchi, A., De Angeli, A., and Sebe, N. (2012, January 11\u201315). UX_Mate: From Facial Expressions to UX Evaluation. Proceedings of the Designing Interactive Systems Conference, Newcastle Upon Tyne, UK.","DOI":"10.1145\/2317956.2318068"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Xiao, H., Li, W., Zeng, G., Wu, Y., Xue, J., Zhang, J., Li, C., and Guo, G. (2022). On-Road Driver Emotion Recognition Using Facial Expression. Appl. Sci., 12.","DOI":"10.3390\/app12020807"},{"key":"ref_3","unstructured":"Kosch, T., Hassib, M., Reutter, R., and Alt, F. (October, January 28). Emotions on the Go: Mobile Emotion Assessment in Real-Time Using Facial Expressions. Proceedings of the International Conference on Advanced Visual Interfaces, Salerno, Italy."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Antoniou, N., Katsamanis, A., Giannakopoulos, T., and Narayanan, S. (2023, January 4\u201310). Designing and Evaluating Speech Emotion Recognition Systems: A Reality Check Case Study with IEMOCAP. Proceedings of the ICASSP 2023\u20142023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.","DOI":"10.1109\/ICASSP49357.2023.10096808"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"102974","DOI":"10.1016\/j.specom.2023.102974","article-title":"Speech Emotion Recognition Approaches: A Systematic Review","volume":"154","author":"Hashem","year":"2023","journal-title":"Speech Commun."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tscharn, R., Latoschik, M.E., L\u00f6ffler, D., and Hurtienne, J. (2017, January 13\u201317). \u201cStop over There\u201d: Natural Gesture and Speech Interaction for Non-Critical Spontaneous Intervention in Autonomous Driving. Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK.","DOI":"10.1145\/3136755.3136787"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Dobbins, C., and Fairclough, S. (2018, January 19\u201323). Detecting Negative Emotions during Real-Life Driving via Dynamically Labelled Physiological Data. Proceedings of the 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Athens, Greece.","DOI":"10.1109\/PERCOMW.2018.8480369"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/TITS.2005.848368","article-title":"Detecting Stress during Real-World Driving Tasks Using Physiological Sensors","volume":"6","author":"Healey","year":"2005","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Zepf, S., Dittrich, M., Hernandez, J., and Schmitt, A. (2019, January 4\u20139). Towards Empathetic Car Interfaces: Emotional Triggers While Driving. Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.","DOI":"10.1145\/3290607.3312883"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Damasio, A.R. (2004). Emotions and Feelings, Cambridge University Press.","DOI":"10.1017\/CBO9780511806582.004"},{"key":"ref_11","unstructured":"Wagener, N., Niess, J., Rogers, Y., and Sch\u00f6ning, J. (May, January 29). Mood Worlds: A Virtual Environment for Autonomous Emotional Expression. Proceedings of the CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A Circumplex Model of Affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ekkekakis, P., and Russell, J.A. (2013). The Measurement of Affect, Mood, and Emotion: A Guide for Health-Behavioral Research, Cambridge University Press. [1st ed.].","DOI":"10.1017\/CBO9780511820724"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ayoub, J., Zhou, F., Bao, S., and Yang, X.J. (2019, January 21\u201325). From Manual Driving to Automated Driving: A Review of 10 Years of AutoUI. Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Utrecht, The Netherlands.","DOI":"10.1145\/3342197.3344529"},{"key":"ref_15","unstructured":"Mehrabian, A., and Russell, J.A. (1974). An Approach to Environmental Psychology, MIT Press."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1511\/2001.28.344","article-title":"The Nature of Emotions: Human Emotions Have Deep Evolutionary Roots, a Fact That May Explain Their Complexity and Provide Tools for Clinical Practice","volume":"89","author":"Plutchik","year":"2001","journal-title":"Am. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1037\/0278-7393.18.2.379","article-title":"Remembering Pictures: Pleasure and Arousal in Memory","volume":"18","author":"Bradley","year":"1992","journal-title":"J. Exp. Psychol. Learn. Mem. Cogn."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.1037\/0022-3514.54.6.1063","article-title":"Development and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales","volume":"54","author":"Watson","year":"1988","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring Emotion: The Self-Assessment Manikin and the Semantic Differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Ther. Exp. Psychiatry"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Dillen, N., Ilievski, M., Law, E., Nacke, L.E., Czarnecki, K., and Schneider, O. (2020, January 25\u201330). Keep Calm and Ride Along: Passenger Comfort and Anxiety as Physiological Responses to Autonomous Driving Styles. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3313831.3376247"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hayashi, E.C.S., Posada, J.E.G., Maike, V.R.M.L., and Baranauskas, M.C.C. (2016, January 4\u20137). Exploring New Formats of the Self-Assessment Manikin in the Design with Children. Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, S\u00e3o Paulo, Brazil.","DOI":"10.1145\/3033701.3033728"},{"key":"ref_22","unstructured":"Wampfler, R., Klingler, S., Solenthaler, B., Schinazi, V.R., Gross, M., and Holz, C. (May, January 29). Affective State Prediction from Smartphone Touch and Sensor Data in the Wild. Proceedings of the CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA."},{"key":"ref_23","first-page":"678","article-title":"Automatic Detection of a Driver\u2019s Complex Mental States","volume":"Volume 10406","author":"Gervasi","year":"2017","journal-title":"Computational Science and Its Applications\u2014ICCSA 2017, Proceedings of the 17th International Conference, Trieste, Italy, 3\u20136 July 2017"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Paschero, M., Del Vescovo, G., Benucci, L., Rizzi, A., Santello, M., Fabbri, G., and Mascioli, F.M.F. (2012, January 28\u201331). A Real Time Classifier for Emotion and Stress Recognition in a Vehicle Driver. Proceedings of the 2012 IEEE International Symposium on Industrial Electronics, Hangzhou, China.","DOI":"10.1109\/ISIE.2012.6237345"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/TMM.2010.2058095","article-title":"Speech Emotion Analysis: Exploring the Role of Context","volume":"12","author":"Tawari","year":"2010","journal-title":"IEEE Trans. Multimed."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1007\/978-3-540-74889-2_12","article-title":"On the Necessity and Feasibility of Detecting a Driver\u2019s Emotional State While Driving","volume":"Volume 4738","author":"Paiva","year":"2007","journal-title":"Affective Computing and Intelligent Interaction"},{"key":"ref_27","first-page":"1085","article-title":"Bimodal Fusion of Emotional Data in an Automotive Environment","volume":"Volume 2","author":"Hoch","year":"2005","journal-title":"Proceedings of the Proceedings. (ICASSP \u201905). IEEE International Conference on Acoustics, Speech, and Signal Processing"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/TITS.2011.2168215","article-title":"Real-Time Driver\u2019s Stress Event Detection","volume":"13","author":"Rigas","year":"2012","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Paredes, P.E., Ordonez, F., Ju, W., and Landay, J.A. (2018, January 21\u201326). Fast & Furious: Detecting Stress with a Car Steering Wheel. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3174239"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Taib, R., Tederry, J., and Itzstein, B. (2014, January 26). Quantifying Driver Frustration to Improve Road Safety. Proceedings of the CHI\u201914 Extended Abstracts on Human Factors in Computing Systems, Toronto, ON, Canada.","DOI":"10.1145\/2559206.2581258"},{"key":"ref_31","first-page":"64","article-title":"Driver Emotion Recognition for Intelligent Vehicles: A Survey","volume":"53","author":"Zepf","year":"2020","journal-title":"ACM Comput. Surv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Fakhrhosseini, S.M., Landry, S., Tan, Y.Y., Bhattarai, S., and Jeon, M. (2014, January 17\u201319). If You\u2019re Angry, Turn the Music on: Music Can Mitigate Anger Effects on Driving Performance. Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, Seattle, WA, USA.","DOI":"10.1145\/2667317.2667410"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1145\/3463493","article-title":"AmbientBreath: Unobtrusive Just-in-Time Breathing Intervention Using Multi-Sensory Stimulation and Its Evaluation in a Car Simulator","volume":"5","author":"Lee","year":"2021","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"20646","DOI":"10.1038\/s41598-021-00062-7","article-title":"Individualized Stress Detection Using an Unmodified Car Steering Wheel","volume":"11","author":"Balters","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1145\/3191760","article-title":"Just Breathe: In-Car Interventions for Guided Slow Breathing","volume":"2","author":"Paredes","year":"2018","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Choi, K.Y., Lee, J., ElHaouij, N., Picard, R., and Ishii, H. (2021, January 8\u201313). aSpire: Clippable, Mobile Pneumatic-Haptic Device for Breathing Rate Regulation via Personalizable Tactile Feedback. Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411763.3451602"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Lee, J.-H., and Spence, C. (2008, January 1\u20135). Assessing the Benefits of Multimodal Feedback on Dual-Task Performance under Demanding Conditions. Proceedings of the People and Computers XXII Culture, Creativity, Interaction (HCI), Liverpool, UK.","DOI":"10.14236\/ewic\/HCI2008.18"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Roumen, T., Perrault, S.T., and Zhao, S. (2015, January 18\u201323). NotiRing: A Comparative Study of Notification Channels for Wearable Interactive Rings. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea.","DOI":"10.1145\/2702123.2702350"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1153","DOI":"10.1080\/02699930903274322","article-title":"Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers","volume":"24","author":"Schaefer","year":"2010","journal-title":"Cognit. Emot."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1016\/j.cortex.2010.04.004","article-title":"Is evaluation of humorous stimuli associated with frontal cortex morphology? A pilot study using facial micro-movement analysis and MRI","volume":"47","author":"Juckel","year":"2011","journal-title":"Cortex"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wilson, G., Davidson, G., and Brewster, S.A. (2015, January 18). In the Heat of the Moment: Subjective Interpretations of Thermal Feedback During Interaction. Proceedings of the Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea.","DOI":"10.1145\/2702123.2702219"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ju, Y., Zheng, D., Hynds, D., Chernyshov, G., Kunze, K., and Minamizawa, K. (2021, January 8). Haptic Empathy: Conveying Emotional Meaning through Vibrotactile Feedback. Proceedings of the Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411763.3451640"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Wilson, G., Dobrev, D., and Brewster, S.A. (2016, January 7). Hot Under the Collar: Mapping Thermal Feedback to Dimensional Models of Emotion. Proceedings of the Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.","DOI":"10.1145\/2858036.2858205"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1111\/j.1467-8721.2007.00514.x","article-title":"Color and Psychological Functioning","volume":"16","author":"Elliot","year":"2007","journal-title":"Curr. Dir. Psychol. Sci."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Juslin, P.N. (1993). Handbook of Music and Emotion: Theory, Research, Applications, Oxford University Press.","DOI":"10.1093\/acprof:oso\/9780199230143.001.0001"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Thoma, M.V., La Marca, R., Br\u00f6nnimann, R., Finkel, L., Ehlert, U., and Nater, U.M. (2013). The Effect of Music on the Human Stress Response. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0070156"},{"key":"ref_47","unstructured":"Th\u00fcrmel, V. (2021). I Know How You Feel: An Investigation of Users\u2019 Trust in Emotion-Based Personalization Systems. AMCIS 2021 Proceedings, Association for Information Systems."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1093\/scan\/nsab092","article-title":"Emotion depends on context, culture, and their interaction: Evidence from effective connectivity","volume":"17","author":"Pugh","year":"2022","journal-title":"Soc. Cogn. Affect. Neurosci."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"E7900","DOI":"10.1073\/pnas.1702247114","article-title":"Self-report captures 27 distinct categories of emotion bridged by continuous gradients","volume":"114","author":"Cowen","year":"2017","journal-title":"Proc. Natl. Acad. Sci. USA"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/8\/7\/54\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:03:51Z","timestamp":1760108631000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/8\/7\/54"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,25]]},"references-count":49,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["mti8070054"],"URL":"https:\/\/doi.org\/10.3390\/mti8070054","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,25]]}}}