{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T05:24:48Z","timestamp":1780637088238,"version":"3.54.1"},"reference-count":42,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,6,10]],"date-time":"2022-06-10T00:00:00Z","timestamp":1654819200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Hyundai Motor Group, the Knowledge Service Industry Core Technology Development Program funded by the Ministry of Trade, Industry, and Energy of Korea","award":["20003519"],"award-info":[{"award-number":["20003519"]}]},{"name":"Hyundai Motor Group, the Knowledge Service Industry Core Technology Development Program funded by the Ministry of Trade, Industry, and Energy of Korea","award":["2021R1A2C1005433"],"award-info":[{"award-number":["2021R1A2C1005433"]}]},{"name":"Hyundai Motor Group, the Knowledge Service Industry Core Technology Development Program funded by the Ministry of Trade, Industry, and Energy of Korea","award":["5199990814084"],"award-info":[{"award-number":["5199990814084"]}]},{"name":"Hyundai Motor Group, the Knowledge Service Industry Core Technology Development Program funded by the Ministry of Trade, Industry, and Energy of Korea","award":["092021C26S03000"],"award-info":[{"award-number":["092021C26S03000"]}]},{"name":"Ministry of Science, ICT, and Future Planning","award":["20003519"],"award-info":[{"award-number":["20003519"]}]},{"name":"Ministry of Science, ICT, and Future Planning","award":["2021R1A2C1005433"],"award-info":[{"award-number":["2021R1A2C1005433"]}]},{"name":"Ministry of Science, ICT, and Future Planning","award":["5199990814084"],"award-info":[{"award-number":["5199990814084"]}]},{"name":"Ministry of Science, ICT, and Future Planning","award":["092021C26S03000"],"award-info":[{"award-number":["092021C26S03000"]}]},{"name":"Ministry of Education","award":["20003519"],"award-info":[{"award-number":["20003519"]}]},{"name":"Ministry of Education","award":["2021R1A2C1005433"],"award-info":[{"award-number":["2021R1A2C1005433"]}]},{"name":"Ministry of Education","award":["5199990814084"],"award-info":[{"award-number":["5199990814084"]}]},{"name":"Ministry of Education","award":["092021C26S03000"],"award-info":[{"award-number":["092021C26S03000"]}]},{"name":"Korea government (KNPA)","award":["20003519"],"award-info":[{"award-number":["20003519"]}]},{"name":"Korea government (KNPA)","award":["2021R1A2C1005433"],"award-info":[{"award-number":["2021R1A2C1005433"]}]},{"name":"Korea government (KNPA)","award":["5199990814084"],"award-info":[{"award-number":["5199990814084"]}]},{"name":"Korea government (KNPA)","award":["092021C26S03000"],"award-info":[{"award-number":["092021C26S03000"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>As vehicles provide various services to drivers, research on driver emotion recognition has been expanding. However, current driver emotion datasets are limited by inconsistencies in collected data and inferred emotional state annotations by others. To overcome this limitation, we propose a data collection system that collects multimodal datasets during real-world driving. The proposed system includes a self-reportable HMI application into which a driver directly inputs their current emotion state. Data collection was completed without any accidents for over 122 h of real-world driving using the system, which also considers the minimization of behavioral and cognitive disturbances. To demonstrate the validity of our collected dataset, we also provide case studies for statistical analysis, driver face detection, and personalized driver emotion recognition. The proposed data collection system enables the construction of reliable large-scale datasets on real-world driving and facilitates research on driver emotion recognition. The proposed system is avaliable on GitHub.<\/jats:p>","DOI":"10.3390\/s22124402","type":"journal-article","created":{"date-parts":[[2022,6,13]],"date-time":"2022-06-13T02:01:44Z","timestamp":1655085704000},"page":"4402","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0498-5631","authenticated-orcid":false,"given":"Geesung","family":"Oh","sequence":"first","affiliation":[{"name":"Graduate School of Automotive Engineering, Kookmin University, Seoul 02707, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Euiseok","family":"Jeong","sequence":"additional","affiliation":[{"name":"Graduate School of Automotive Engineering, Kookmin University, Seoul 02707, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rak Chul","family":"Kim","sequence":"additional","affiliation":[{"name":"Graduate School of Automotive Engineering, Kookmin University, Seoul 02707, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8349-6931","authenticated-orcid":false,"given":"Ji Hyun","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Automobile and IT Convergence, Kookmin University, Seoul 02707, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sungwook","family":"Hwang","sequence":"additional","affiliation":[{"name":"Chassis System Control Research Lab, Hyundai Motor Group, Hwaseong 18280, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sangho","family":"Lee","sequence":"additional","affiliation":[{"name":"Chassis System Control Research Lab, Hyundai Motor Group, Hwaseong 18280, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1917-699X","authenticated-orcid":false,"given":"Sejoon","family":"Lim","sequence":"additional","affiliation":[{"name":"Department of Automobile and IT Convergence, Kookmin University, Seoul 02707, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1109\/TAFFC.2018.2890471","article-title":"Deep learning for human affect recognition: Insights and new developments","volume":"12","author":"Rouast","year":"2019","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/S1369-8478(99)00006-6","article-title":"Anger while driving","volume":"2","author":"Underwood","year":"1999","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"777","DOI":"10.1080\/10447318.2016.1198524","article-title":"Don\u2019t cry while you\u2019re driving: Sad driving is as bad as angry driving","volume":"32","author":"Jeon","year":"2016","journal-title":"Int. J. Hum.-Comput. Interact."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Hassib, M., Braun, M., Pfleging, B., and Alt, F. (2019, January 2\u20136). Detecting and influencing driver emotions using psycho-physiological sensors and ambient light. Proceedings of the IFIP Conference on Human-Computer Interactionr, Paphos, Cyprus.","DOI":"10.1007\/978-3-030-29381-9_43"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Gao, H., Y\u00fcce, A., and Thiran, J.P. (2014, January 27\u201330). Detecting emotional stress from facial expressions for driving safety. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.","DOI":"10.1109\/ICIP.2014.7026203"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Oh, G., Ryu, J., Jeong, E., Yang, J.H., Hwang, S., Lee, S., and Lim, S. (2021). Drer: Deep learning-based driver\u2019s real emotion recognizer. Sensors, 21.","DOI":"10.3390\/s21062166"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/TITS.2005.848368","article-title":"Detecting stress during real-world driving tasks using physiological sensors","volume":"6","author":"Healey","year":"2005","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Angkititrakul, P., Petracca, M., Sathyanarayana, A., and Hansen, J.H. (2007, January 13\u201315). UTDrive: Driver behavior and speech interactive systems for in-vehicle environments. Proceedings of the 2007 IEEE Intelligent Vehicles Symposium, Istanbul, Turkey.","DOI":"10.1109\/IVS.2007.4290175"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Singh, R.R., Conjeti, S., and Banerjee, R. (2012, January 3\u20135). Biosignal based on-road stress monitoring for automotive drivers. Proceedings of the 2012 National Conference on Communications (NCC), Kharagpur, India.","DOI":"10.1109\/NCC.2012.6176845"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Jones, C., and Jonsson, I.M. (2008). Using paralinguistic cues in speech to recognise emotions in older car drivers. Affect and Emotion in Human-Computer Interaction, Springer.","DOI":"10.1007\/978-3-540-85099-1_20"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ma, Z., Mahmoud, M., Robinson, P., Dias, E., and Skrypchuk, L. (2017, January 3\u20136). Automatic detection of a driver\u2019s complex mental states. Proceedings of the International Conference on Computational Science and Its Applications, Trieste, Italy.","DOI":"10.1007\/978-3-319-62398-6_48"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kato, T., Kawanaka, H., Bhuiyan, M.S., and Oguri, K. (2011, January 5\u20137). Classification of positive and negative emotion evoked by traffic jam based on electrocardiogram (ECG) and pulse wave. Proceedings of the 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), Washington, DC, USA.","DOI":"10.1109\/ITSC.2011.6083113"},{"key":"ref_13","unstructured":"Taib, R., Tederry, J., and Itzstein, B. (May, January 26). Quantifying driver frustration to improve road safety. Proceedings of the CHI\u201914 Extended Abstracts on Human Factors in Computing Systems, Toronto, ON, Canada."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1075\/is.17005.ihm","article-title":"Frustration in the face of the driver: A simulator study on facial muscle activity during frustrated driving","volume":"19","author":"Ihme","year":"2018","journal-title":"Interact. Stud."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3388790","article-title":"Driver emotion recognition for intelligent vehicles: A survey","volume":"53","author":"Zepf","year":"2020","journal-title":"ACM Comput. Surv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Ortega, J.D., Kose, N., Ca\u00f1as, P., Chao, M.A., Unnervik, A., Nieto, M., Otaegui, O., and Salgado, L. (2020, January 23\u201328). Dmd: A large-scale multi-modal driver monitoring dataset for attention and alertness analysis. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-66823-5_23"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"115960","DOI":"10.1016\/j.image.2020.115960","article-title":"A novel public dataset for multimodal multiview and multispectral driver distraction analysis: 3MDAD","volume":"88","author":"Jegham","year":"2020","journal-title":"Signal Process. Image Commun."},{"key":"ref_18","unstructured":"Martin, M., Roitberg, A., Haurilet, M., Horne, M., Rei\u00df, S., Voit, M., and Stiefelhagen, R. (November, January 27). Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1109\/TIV.2019.2955364","article-title":"Looking at the driver\/rider in autonomous vehicles to predict take-over readiness","volume":"5","author":"Deo","year":"2019","journal-title":"IEEE Trans. Intell. Veh."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1109\/TAFFC.2018.2817622","article-title":"EEG emotion recognition using dynamical graph convolutional neural networks","volume":"11","author":"Song","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_21","unstructured":"Tao, W., Li, C., Song, R., Cheng, J., Liu, Y., Wan, F., and Chen, X. (2020). EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans. Affect. Comput."},{"key":"ref_22","unstructured":"Li, S., and Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kawaguchi, N., Matsubara, S., Takeda, K., and Itakura, F. (2001, January 3\u20137). Multimedia data collection of in-car speech communication. Proceedings of the 7th European Conference on Speech Communication and Technology, Aalborg, Denmark.","DOI":"10.21437\/Eurospeech.2001-478"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring emotion: The self-assessment manikin and the semantic differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Ther. Exp. Psychiatry"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"McCarthy, C., Pradhan, N., Redpath, C., and Adler, A. (2016, January 29\u201331). Validation of the Empatica E4 wristband. Proceedings of the 2016 IEEE EMBS International Student Conference (ISC), Ottawa, ON, Canada.","DOI":"10.1109\/EMBSISC.2016.7508621"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ragot, M., Martin, N., Em, S., Pallamin, N., and Diverrez, J.M. (2017, January 17\u201321). Emotion recognition using physiological signals: Laboratory vs. wearable sensors. Proceedings of the International Conference on Applied Human Factors and Ergonomics, Los Angeles, CA, USA.","DOI":"10.1007\/978-3-319-60639-2_2"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1146\/annurev.clinpsy.3.022806.091415","article-title":"Ecological momentary assessment","volume":"4","author":"Shiffman","year":"2008","journal-title":"Annu. Rev. Clin. Psychol."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Jeon, M., and Walker, B.N. (2011, January 19\u201323). What to detect? Analyzing factor structures of affect in driving contexts for an emotion detection and regulation system. Proceedings of the 55th Annual Meeting of the Human Factors and Ergonomics Society, Human Factors and Ergonomics Society, Las Vegas, NV, USA.","DOI":"10.1177\/1071181311551393"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_30","first-page":"218","article-title":"Tranquilizing effect of color reduces aggressive behavior and potential violence","volume":"8","author":"Schauss","year":"1979","journal-title":"J. Orthomol. Psychiatry"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","article-title":"Use of ranks in one-criterion variance analysis","volume":"47","author":"Kruskal","year":"1952","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"115","DOI":"10.4028\/www.scientific.net\/AMM.611.115","article-title":"Methodology and application of the Kruskal-Wallis test","volume":"Volume 611","author":"Ostertagova","year":"2014","journal-title":"Applied Mechanics and Materials"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1080\/01621459.1957.10501395","article-title":"Historical notes on the Wilcoxon unpaired two-sample test","volume":"52","author":"Kruskal","year":"1957","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1136\/bmj.323.7309.391","article-title":"Mann-Whitney test is not just a test of medians: Differences in spread can be important","volume":"323","author":"Hart","year":"2001","journal-title":"Bmj"},{"key":"ref_35","unstructured":"Jain, V., and Learned-Miller, E. (2010). Fddb: A Benchmark for Face Detection in Unconstrained Settings, University of Massachusetts. Technical Report UMCS-2010-009."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Yang, S., Luo, P., Loy, C.C., and Tang, X. (2016, January 27\u201330). Wider face: A face detection benchmark. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.596"},{"key":"ref_37","unstructured":"Viola, P., and Jones, M. (2001, January 8\u201314). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, USA."},{"key":"ref_38","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201325). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","article-title":"Robust real-time face detection","volume":"57","author":"Viola","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_40","unstructured":"King, D.E. (2015). Max-margin object detection. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Oh, G., Jeong, E., and Lim, S. (2021, January 11\u201317). Causal affect prediction model using a past facial image sequence. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCVW54120.2021.00395"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Fang, H.S., Xie, S., Tai, Y.W., and Lu, C. (2017, January 22\u201329). RMPE: Regional Multi-person Pose Estimation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.256"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/12\/4402\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:27:43Z","timestamp":1760138863000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/12\/4402"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,10]]},"references-count":42,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["s22124402"],"URL":"https:\/\/doi.org\/10.3390\/s22124402","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,10]]}}}