{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,23]],"date-time":"2025-10-23T11:22:28Z","timestamp":1761218548075,"version":"build-2065373602"},"reference-count":36,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"EU Mobilitas Pluss","award":["MOBTT90"],"award-info":[{"award-number":["MOBTT90"]}]},{"name":"Pia Tikka, Enactive Virtuality Lab, Tallinn University","award":["MOBTT90"],"award-info":[{"award-number":["MOBTT90"]}]},{"name":"Estonian Centre of Excellence in IT (EXCITE)","award":["MOBTT90"],"award-info":[{"award-number":["MOBTT90"]}]},{"name":"European Regional Development Fund","award":["MOBTT90"],"award-info":[{"award-number":["MOBTT90"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Emotion recognition is a significant issue in many sectors that use human emotion reactions as communication for marketing, technological equipment, or human\u2013robot interaction. The realistic facial behavior of social robots and artificial agents is still a challenge, limiting their emotional credibility in dyadic face-to-face situations with humans. One obstacle is the lack of appropriate training data on how humans typically interact in such settings. This article focused on collecting the facial behavior of 60 participants to create a new type of dyadic emotion reaction database. For this purpose, we propose a methodology that automatically captures the facial expressions of participants via webcam while they are engaged with other people (facial videos) in emotionally primed contexts. The data were then analyzed using three different Facial Expression Analysis (FEA) tools: iMotions, the Mini-Xception model, and the Py-Feat FEA toolkit. Although the emotion reactions were reported as genuine, the comparative analysis between the aforementioned models could not agree with a single emotion reaction prediction. Based on this result, a more-robust and -effective model for emotion reaction prediction is needed. The relevance of this work for human\u2013computer interaction studies lies in its novel approach to developing adaptive behaviors for synthetic human-like beings (virtual or robotic), allowing them to simulate human facial interaction behavior in contextually varying dyadic situations with humans. This article should be useful for researchers using human emotion analysis while deciding on a suitable methodology to collect facial expression reactions in a dyadic setting.<\/jats:p>","DOI":"10.3390\/s23010458","type":"journal-article","created":{"date-parts":[[2023,1,2]],"date-time":"2023-01-02T03:08:59Z","timestamp":1672628939000},"page":"458","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Towards Context-Aware Facial Emotion Reaction Database for Dyadic Interaction Settings"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1773-904X","authenticated-orcid":false,"given":"Abdallah Hussein","family":"Sham","sequence":"first","affiliation":[{"name":"Digital Technology Institute, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia"},{"name":"Enactive Virtuality Lab, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amna","family":"Khan","sequence":"additional","affiliation":[{"name":"Digital Technology Institute, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Lamas","sequence":"additional","affiliation":[{"name":"Digital Technology Institute, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3658-7993","authenticated-orcid":false,"given":"Pia","family":"Tikka","sequence":"additional","affiliation":[{"name":"Enactive Virtuality Lab, Tallinn University, Narva mnt 25, 10120 Tallinn, Estonia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8460-5717","authenticated-orcid":false,"given":"Gholamreza","family":"Anbarjafari","sequence":"additional","affiliation":[{"name":"iCV Lab, University of Tartu, Narva mnt 18, 51009 Tartu, Estonia"},{"name":"iVCV O\u00dc, 51009 Tartu, Estonia"},{"name":"PwC Advisory, 00180 Helsinki, Finland"},{"name":"Institute of Higher Education, Yildiz Technical University, Be\u015fikta\u015f, Istanbul 34349, Turkey"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Ko, B.C. (2018). A brief review of facial emotion recognition based on visual information. Sensors, 18.","DOI":"10.3390\/s18020401"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1018","DOI":"10.1177\/0146167204264762","article-title":"Emotional intelligence and social interaction","volume":"30","author":"Lopes","year":"2004","journal-title":"Personal. Soc. Psychol. Bull."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1111\/j.1467-8721.2009.01633.x","article-title":"How emotions regulate social life: The emotions as social information (EASI) model","volume":"18","year":"2009","journal-title":"Curr. Dir. Psychol. Sci."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1042","DOI":"10.1016\/j.jesp.2010.05.015","article-title":"Can expressions of anger enhance creativity? A test of the emotions as social information (EASI) model","volume":"46","author":"Anastasopoulou","year":"2010","journal-title":"J. Exp. Soc. Psychol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1607","DOI":"10.1177\/1362361320925334","article-title":"Measuring change in facial emotion recognition in individuals with autism spectrum disorder: A systematic review","volume":"24","author":"Wieckowski","year":"2020","journal-title":"Autism"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Sapi\u0144ski, T., Kami\u0144ska, D., Pelikant, A., Ozcinar, C., Avots, E., and Anbarjafari, G. (2018, January 20\u201324). Multimodal database of emotional speech, video and gestures. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.","DOI":"10.1007\/978-3-030-05792-3_15"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1109\/TAFFC.2018.2874986","article-title":"Survey on emotional body gesture recognition","volume":"12","author":"Noroozi","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Dzedzickis, A., Kaklauskas, A., and Bucinskas, V. (2020). Human emotion recognition: Review of sensors and methods. Sensors, 20.","DOI":"10.3390\/s20030592"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Avots, E., Jermakovs, K., Bachmann, M., P\u00e4eske, L., Ozcinar, C., and Anbarjafari, G. (2022). Ensemble approach for detection of depression using EEG features. Entropy, 24.","DOI":"10.3390\/e24020211"},{"key":"ref_10","first-page":"243","article-title":"Hawthorne Effect","volume":"1","year":"2021","journal-title":"Encycl. Res. Methods Criminol. Crim. Justice"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Burns, A., and Tulip, J. (2017, January 22\u201325). Detecting Flow in Games Using Facial Expressions. Proceedings of the 2017 IEEE Conference on Computational Intelligence and Games (CIG), New York, NY, USA.","DOI":"10.1109\/CIG.2017.8080414"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"24","DOI":"10.5565\/rev\/elcvia.795","article-title":"A survey on human emotion recognition approaches, databases and applications","volume":"14","author":"Vinola","year":"2015","journal-title":"ELCVIA Electron. Lett. Comput. Vis. Image Anal."},{"key":"ref_13","first-page":"39","article-title":"Review on emotion recognition databases","volume":"3","author":"Haamer","year":"2017","journal-title":"Hum. Robot Interact. Theor. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"19301","DOI":"10.1007\/s11042-017-5317-2","article-title":"A survey: Facial micro-expression recognition","volume":"77","author":"Takalkar","year":"2018","journal-title":"Multimed. Tools Appl."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"eaaw4358","DOI":"10.1126\/sciadv.aaw4358","article-title":"Emotion schemas are embedded in the human visual system","volume":"5","author":"Kragel","year":"2019","journal-title":"Sci. Adv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1109\/TAFFC.2020.2981446","article-title":"Deep facial expression recognition: A survey","volume":"13","author":"Li","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Chowdhury, A., and Chakraborty, P. (2021, January 7\u20139). Memes that Evoke Emotions: A Neurodesign Strategy for Brand Communication and Experience. Proceedings of the International Conference on Research into Design, Mumbai, India.","DOI":"10.1007\/978-981-16-0041-8_13"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Kami\u0144ska, D., Aktas, K., Rizhinashvili, D., Kuklyanov, D., Sham, A.H., Escalera, S., Nasrollahi, K., Moeslund, T.B., and Anbarjafari, G. (2021). Two-stage recognition and beyond for compound facial emotion recognition. Electronics, 10.","DOI":"10.3390\/electronics10222847"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ringeval, F., Sonderegger, A., Sauer, J., and Lalanne, D. (2013, January 22\u201326). Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.","DOI":"10.1109\/FG.2013.6553805"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Douglas-Cowie, E., Cox, C., Martin, J.C., Devillers, L., Cowie, R., Sneddon, I., McRorie, M., Pelachaud, C., Peters, C., and Lowry, O. (2011). The HUMAINE database. Emotion-Oriented Systems, Springer.","DOI":"10.1007\/978-3-642-15184-2_14"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1022","DOI":"10.1109\/TPAMI.2019.2944808","article-title":"Sewa db: A rich database for audio-visual emotion and sentiment research in the wild","volume":"43","author":"Kossaifi","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"907","DOI":"10.1007\/s11263-019-01158-4","article-title":"Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond","volume":"127","author":"Kollias","year":"2019","journal-title":"Int. J. Comput. Vis."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3488242","article-title":"Non-dyadic interaction: A literature review of 15 years of human-robot interaction conference publications","volume":"11","author":"Schneiders","year":"2022","journal-title":"ACM Trans. Hum. Robot. Interact. THRI"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Huang, W., Liu, G., and Wen, W. (2014, January 13\u201314). MAPD: A Multi-Subject Affective Physiological Database. Proceedings of the 2014 Seventh International Symposium on Computational Intelligence and Design, Hangzhou, China.","DOI":"10.1109\/ISCID.2014.247"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"L\u00fcsi, I., Escarela, S., and Anbarjafari, G. (2016, January 11\u201314). Sase: Rgb-depth database for human head pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-49409-8_26"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"L\u00fcsi, I., Junior, J.C.J., Gorbova, J., Bar\u00f3, X., Escalera, S., Demirel, H., Allik, J., Ozcinar, C., and Anbarjafari, G. (June, January 30). Joint Challenge on Dominant and Complementary Emotion Recognition Using Micro Emotion Features and Head-Pose Estimation: Databases. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.","DOI":"10.1109\/FG.2017.102"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Haamer, R.E., Kulkarni, K., Imanpour, N., Haque, M.A., Avots, E., Breisch, M., Nasrollahi, K., Escalera, S., Ozcinar, C., and Baro, X. (2018, January 15\u201319). Changes in Facial Expression as Biometric: A Database and Benchmarks of Identification. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00098"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Schmidt, P., Reiss, A., D\u00fcrichen, R., and Laerhoven, K.V. (2019). Wearable-based affect recognition\u2014A review. Sensors, 19.","DOI":"10.3390\/s19194079"},{"key":"ref_29","unstructured":"Zhang, Z., Girard, J.M., Wu, Y., Zhang, X., Liu, P., Ciftci, U., Canavan, S., Reale, M., Horowitz, A., and Yang, H. (July, January 26). Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1007\/s10919-019-00320-3","article-title":"Are you on my wavelength? Interpersonal coordination in dyadic conversations","volume":"44","author":"Hale","year":"2020","journal-title":"J. Nonverbal Behav."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hadley, L.V., and Ward, J.A. (2021). Synchrony as a measure of conversation difficulty: Movement coherence increases with background noise level and complexity in dyads and triads. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0258247"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"102819","DOI":"10.1016\/j.ijhcs.2022.102819","article-title":"Nonverbal communication in virtual reality: Nodding as a social signal in virtual interactions","volume":"164","author":"Aburumman","year":"2022","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Sham, A.H., Tikka, P., Lamas, D., and Anbarjafari, G. (2022). Automatic reaction emotion estimation in a human\u2013human dyadic setting using Deep Neural Networks. Signal Image Video Process., 1\u20138.","DOI":"10.1007\/s11760-022-02257-5"},{"key":"ref_34","unstructured":"(2022, December 27). iMotions A. Affectiva iMotions Biometric Research Platform. Available online: https:\/\/imotions.com\/."},{"key":"ref_35","unstructured":"Cheong, J.H., Xie, T., Byrne, S., and Chang, L.J. (2021). Py-feat: Python facial expression analysis toolbox. arXiv."},{"key":"ref_36","unstructured":"Tkalcic, M., Kosir, A., and Tasic, J. (2011). Affective Recommender Systems: The Role of Emotions in Recommender Systems. Proceedings of the RecSys 2011 Workshop on Human Decision Making in Recommender Systems, Citeseer."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/1\/458\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T17:55:14Z","timestamp":1760118914000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/1\/458"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,1]]},"references-count":36,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,1]]}},"alternative-id":["s23010458"],"URL":"https:\/\/doi.org\/10.3390\/s23010458","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,1,1]]}}}