{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T02:26:57Z","timestamp":1772591217167,"version":"3.50.1"},"reference-count":44,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T00:00:00Z","timestamp":1730246400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JST Center of Innovation (COI) STREAM"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Facial expression recognition (FER) is significantly influenced by the cultural background (CB) of observers and the masking conditions of the target face. This study aimed to clarify these factors\u2019 impact on FER, particularly in machine-learning datasets, increasingly used in human-computer interaction and automated systems. We conducted an FER experiment with East Asian participants and compared the results with the FERPlus dataset, evaluated by Western raters. Our novel analysis approach focused on variability between images and participants within a \"majority\" category and the eye-opening rate of target faces, providing a deeper understanding of FER processes. Notable findings were differences in \"fear\" perception between East Asians and Westerners, with East Asians more likely to interpret \"fear\" as \"surprise.\" Masking conditions significantly affected emotion categorization, with \"fear\" perceived by East Asians for non-masked faces interpreted as \"surprise\" for masked faces. Then, the emotion labels were perceived as different emotions across categories in the masking condition, rather than simply lower recognition rates or confusion as in existing studies. Additionally, \"sadness\" perceived by Westerners was often interpreted as \"disgust\" by East Asians. These results suggest that one-to-one network learning models, commonly trained using majority labels, might overlook important minority response information, potentially leading to biases in automated FER systems. In conclusion, FER dataset characteristics differ depending on the target face\u2019s masking condition and the diversity among evaluation groups. This study highlights the need to consider these factors in machine-learning-based FER that relies on human-judged labels, to contribute to the development of more nuanced and fair automated FER systems. Our findings emphasize the novelty of our approach compared to existing studies and the importance of incorporating a broader range of human variability in FER research, setting the stage for future evaluations of machine learning classifiers on similar data.<\/jats:p>","DOI":"10.1371\/journal.pone.0313029","type":"journal-article","created":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T17:37:15Z","timestamp":1730309835000},"page":"e0313029","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":4,"title":["Effect of observer\u2019s cultural background and masking condition of target face on facial expression recognition for machine-learning dataset"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-5407-5566","authenticated-orcid":true,"given":"Masahito","family":"Nagata","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katsunori","family":"Okajima","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2024,10,30]]},"reference":[{"issue":"6","key":"pone.0313029.ref001","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1016\/j.jksuci.2018.09.002","article-title":"A survey on human face expression recognition techniques","volume":"33","author":"IM Revina","year":"2021","journal-title":"Journal of King Saud University Computer and Information Sciences"},{"key":"pone.0313029.ref002","article-title":"FER-2013 face database.","author":"PL Carrier","year":"2013","journal-title":"Universit de Montreal"},{"key":"pone.0313029.ref003","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1145\/2993148.2993165","article-title":"Training deep networks for facial expression recognition with crowd-sourced label distribution.","author":"E Barsoum","year":"2016","journal-title":"Proceedings of the 18th ACM International Conference on Multimodal Interaction"},{"key":"pone.0313029.ref004","doi-asserted-by":"crossref","first-page":"416","DOI":"10.3390\/s18020416","article-title":"Facial emotion recognition: A survey and real-world user experiences in mixed reality.","volume":"18","author":"D Mehta","year":"2018","journal-title":"Sensors (Basel, Switzerland)."},{"key":"pone.0313029.ref005","first-page":"1","article-title":"Deep learning-based facial emotion recognition for human\u2013computer interaction applications","author":"MK Chowdary","year":"2021","journal-title":"Neural Computing and Applications"},{"key":"pone.0313029.ref006","first-page":"1","article-title":"Deep leaning-based facial expression recognition in FER2013 database.","author":"GK Sahoo","year":"2022","journal-title":"An in-vehicle application. 2022 IEEE 19th India Council International Conference (INDICON)."},{"key":"pone.0313029.ref007","doi-asserted-by":"crossref","first-page":"807","DOI":"10.3390\/app12020807","article-title":"On-road driver emotion recognition using facial expression.","volume":"12","author":"H Xiao","year":"2022","journal-title":"Applied Sciences"},{"key":"pone.0313029.ref008","article-title":"Pervasive label errors in test sets destabilize machine learning benchmarks.","author":"CG Northcutt","year":"2021","journal-title":"35th Conference on Neural Information Processing Systems (NeurIPS 2021)."},{"key":"pone.0313029.ref009","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"Imagenet: A large-scale hierarchical image database.","author":"J Deng","year":"2009","journal-title":"2009 IEEE conference on computer vision and pattern recognition."},{"issue":"2","key":"pone.0313029.ref010","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1037\/h0030377","article-title":"Constants across cultures in the face and emotion","volume":"17","author":"P Ekman","year":"1971","journal-title":"Journal of Personality and Social Psychology"},{"key":"pone.0313029.ref011","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1023\/A:1024902500935","article-title":"Matsumoto and Ekman\u2019s Japanese and Caucasian facial expressions of emotion (JACFEE): Reliability data and cross-national differences.","volume":"21","author":"M Biehl","year":"1997","journal-title":"Journal of Nonverbal Behavior"},{"key":"pone.0313029.ref012","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1007\/s10919-019-00293-3","article-title":"Emotional expression: Advances in basic emotion theory","volume":"43","author":"D Keltner","year":"2019","journal-title":"Journal of Nonverbal Behavior"},{"key":"pone.0313029.ref013","doi-asserted-by":"crossref","first-page":"4282","DOI":"10.1109\/ICPR48806.2021.9413141","article-title":"Continuous learning of face attribute synthesis.","author":"X Ning","year":"2021","journal-title":"2020 25th International Conference on Pattern Recognition (ICPR)."},{"key":"pone.0313029.ref014","doi-asserted-by":"crossref","first-page":"64827","DOI":"10.1109\/ACCESS.2019.2917266","article-title":"Local learning with deep and handcrafted features for facial expression recognition.","volume":"7","author":"MI Georgescu","year":"2019","journal-title":"IEEE Access"},{"key":"pone.0313029.ref015","first-page":"221","article-title":"Real-time convolutional neural networks for emotion and gender classification.","author":"O Arriaga","year":"2019","journal-title":"27th European Symposium on Artificial Neural Networks (ESANN 2019)."},{"key":"pone.0313029.ref016","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1177\/0022022192231005","article-title":"American-Japanese cultural differences in the recognition of universal facial expressions","volume":"23","author":"D. Matsumoto","year":"1992","journal-title":"Journal of Cross-Cultural Psychology"},{"issue":"2","key":"pone.0313029.ref017","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1037\/0033-2909.115.2.268","article-title":"Strong evidence for universals in facial expressions: A reply to Russell\u2019s mistaken critique.","volume":"115","author":"P. Ekman","year":"1994","journal-title":"Psychological Bulletin"},{"key":"pone.0313029.ref018","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1007\/978-0-387-89676-2_8","article-title":"The expression of anger across cultures.","author":"D Matsumoto","year":"2010","journal-title":"International handbook of anger: Constituent and concomitant biological, psychological, and social processes."},{"issue":"18","key":"pone.0313029.ref019","doi-asserted-by":"crossref","first-page":"1543","DOI":"10.1016\/j.cub.2009.07.051","article-title":"Cultural confusions show that facial expressions are not universal","volume":"19","author":"RE Jack","year":"2009","journal-title":"Current Biology"},{"issue":"2","key":"pone.0313029.ref020","first-page":"13","article-title":"A racial \/ ethnic performance disparity on the facial recognition test","volume":"2","author":"HE Fouty","year":"2016","journal-title":"GSTF Journal of Psychology (JPsych)."},{"issue":"4","key":"pone.0313029.ref021","doi-asserted-by":"crossref","first-page":"e0249792","DOI":"10.1371\/journal.pone.0249792","article-title":"Face masks reduce emotion-recognition accuracy and perceived closeness.","volume":"16","author":"F Grundmann","year":"2021","journal-title":"PLoS One"},{"key":"pone.0313029.ref022","doi-asserted-by":"crossref","DOI":"10.1016\/j.nbd.2021.105319","article-title":"Facemasks and face recognition: Potential impact on synaptic plasticity","volume":"153","author":"C Ferrari","year":"2021","journal-title":"Neurobiology of Disease"},{"key":"pone.0313029.ref023","doi-asserted-by":"crossref","first-page":"2601","DOI":"10.1038\/s41598-021-81780-w","article-title":"The consequences of COVID-19 on social interactions: an online study on face covering.","volume":"11","author":"M Calbi","year":"2021","journal-title":"Scientific Reports"},{"issue":"59","key":"pone.0313029.ref024","article-title":"Surgical face masks impair human face matching performance for familiar and unfamiliar faces.","volume":"5","author":"DJ Carragher","year":"2010","journal-title":"Cognitive Research: Principles and Implications."},{"key":"pone.0313029.ref025","doi-asserted-by":"crossref","first-page":"22344","DOI":"10.1038\/s41598-020-78986-9","article-title":"The COVID-19 pandemic masks the way people perceive faces.","volume":"10","author":"E Freud","year":"2020","journal-title":"Scientific Reports."},{"key":"pone.0313029.ref026","doi-asserted-by":"crossref","first-page":"669432","DOI":"10.3389\/fpsyg.2021.669432","article-title":"Masking emotions: Face masks impair how we read emotions.","volume":"12","author":"M Gori","year":"2021","journal-title":"Frontiers in Psychology"},{"key":"pone.0313029.ref027","doi-asserted-by":"crossref","first-page":"5577","DOI":"10.1038\/s41598-021-84806-5","article-title":"The impact of facemasks on emotion recognition, trust attribution and re-identification.","volume":"11","author":"M Marini","year":"2021","journal-title":"Scientific Reports"},{"key":"pone.0313029.ref028","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1080\/02642069.2021.1873296","article-title":"Masked smiles matter employee verbal expertise and emotion display during COVID-19","volume":"41","author":"H Verena","year":"2021","journal-title":"The Service Industries Journal"},{"key":"pone.0313029.ref029","doi-asserted-by":"crossref","first-page":"201169","DOI":"10.1098\/rsos.201169","article-title":"The effect of face masks and sunglasses on identity and expression recognition with super-recognizers and typical observers","volume":"8","author":"E Noyes","year":"2021","journal-title":"Royal Society Open Science"},{"key":"pone.0313029.ref030","doi-asserted-by":"crossref","first-page":"566886","DOI":"10.3389\/fpsyg.2020.566886","article-title":"Wearing face masks strongly confuses counterparts in reading emotions.","volume":"11","author":"CC Carbon","year":"2020","journal-title":"Frontiers in Psychology"},{"issue":"12","key":"pone.0313029.ref031","doi-asserted-by":"crossref","first-page":"e0243708","DOI":"10.1371\/journal.pone.0243708","article-title":"Children\u2019s emotion inferences from masked faces: implications for social interactions during COVID-19.","volume":"15","author":"AL Ruba","year":"2020","journal-title":"PLoS One"},{"issue":"1","key":"pone.0313029.ref032","doi-asserted-by":"crossref","first-page":"351","DOI":"10.3758\/BRM.42.1.351","article-title":"FACES\u2014A database of facial expressions in young, middle-aged, and older women and men: development and validation","volume":"42","author":"NC Ebner","year":"2010","journal-title":"Behavior Research Methods"},{"key":"pone.0313029.ref033","first-page":"1755","article-title":"Dlib-ml: A machine learning toolkit","volume":"10","author":"DE King","year":"2009","journal-title":"Journal of Machine Learning Research"},{"key":"pone.0313029.ref034","article-title":"Ongoing Face Recognition Vendor Test (FRVT) Part 6A: Face recognition accuracy with masks using pre-COVID-19 algorithms.","author":"M Ngan","year":"2020","journal-title":"National Institute of Standards and Technology (NIST)."},{"issue":"1","key":"pone.0313029.ref035","first-page":"33","article-title":"Masked emotions: Do face mask patterns and colors affect the recognition of emotions?","volume":"7","author":"O Blazhenkova","year":"2022","journal-title":"Cognitive Research: Principles and Implications."},{"issue":"1","key":"pone.0313029.ref036","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1038\/s41598-022-27049-2","article-title":"Emotional face recognition when a colored mask is worn: a cross-sectional study.","volume":"13","author":"S Gil","year":"2023","journal-title":"Scientific Reports."},{"key":"pone.0313029.ref037","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"J. Cohen","year":"1960","journal-title":"Educational and Psychological Measurement"},{"key":"pone.0313029.ref038","article-title":"A simple method for between-group agreement assessment.","author":"P R\u00fcegg","year":"2018","journal-title":"arXiv preprint arXiv:1806.05821"},{"issue":"86","key":"pone.0313029.ref039","first-page":"2579","article-title":"Visualizing data using t-SNE.","volume":"9","author":"L Maaten","year":"2008","journal-title":"Journal of Machine Learning Research"},{"key":"pone.0313029.ref040","first-page":"3","article-title":"Real-time eye blink detection using facial landmarks.","author":"T Soukupov\u00e1","year":"2016","journal-title":"Computer Vision Winter Workshop, Rimske Toplice, Slovenia."},{"issue":"5","key":"pone.0313029.ref041","doi-asserted-by":"crossref","first-page":"e0177239","DOI":"10.1371\/journal.pone.0177239","article-title":"Mapping the emotional face. How individual face parts contribute to successful emotion recognition.","volume":"12","author":"M Wegrzyn","year":"2017","journal-title":"PLoS One."},{"issue":"4","key":"pone.0313029.ref042","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1037\/h0028434","article-title":"Recognition for faces of own and other race","volume":"13","author":"RS Malpass","year":"1969","journal-title":"Journal of Personality and Social Psychology"},{"issue":"6","key":"pone.0313029.ref043","doi-asserted-by":"crossref","first-page":"964","DOI":"10.1037\/a0020400","article-title":"Mere social categorization modulates identification of facial expressions of emotion","volume":"99","author":"SG Young","year":"2010","journal-title":"Journal of Personality and Social Psychology"},{"issue":"6","key":"pone.0313029.ref044","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1080\/00207594.2011.626049","article-title":"In the eye of the beholder? Universality and cultural specificity in the expression and perception of emotion","volume":"46","author":"KR Scherer","year":"2011","journal-title":"International Journal of Psychology"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0313029","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T17:37:44Z","timestamp":1730309864000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0313029"}},"subtitle":[],"editor":[{"given":"Antoine","family":"Coutrot","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2024,10,30]]},"references-count":44,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,10,30]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0313029","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,30]]}}}