{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,26]],"date-time":"2026-07-26T04:00:46Z","timestamp":1785038446496,"version":"3.55.0"},"reference-count":88,"publisher":"Springer Science and Business Media LLC","issue":"38","license":[{"start":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T00:00:00Z","timestamp":1724803200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T00:00:00Z","timestamp":1724803200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100011033","name":"Agencia Estatal de Investigaci\u00f3n","doi-asserted-by":"publisher","award":["Grant PID2019-104829RA-I00 funded by MCIN\/ AEI \/10.13039\/501100011033"],"award-info":[{"award-number":["Grant PID2019-104829RA-I00 funded by MCIN\/ AEI \/10.13039\/501100011033"]}],"id":[{"id":"10.13039\/501100011033","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100011033","name":"Agencia Estatal de Investigaci\u00f3n","doi-asserted-by":"publisher","award":["Grant PID2022-136779OB-C32 funded by MCIN\/AEI\/ 10.13039\/501100011033"],"award-info":[{"award-number":["Grant PID2022-136779OB-C32 funded by MCIN\/AEI\/ 10.13039\/501100011033"]}],"id":[{"id":"10.13039\/501100011033","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100022397","name":"Govern de les Illes Balears","doi-asserted-by":"publisher","award":["FPU scholarship"],"award-info":[{"award-number":["FPU scholarship"]}],"id":[{"id":"10.13039\/501100022397","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Facial expression recognition is vital for human behavior analysis, and deep learning has enabled models that can outperform humans. However, it is unclear how closely they mimic human processing. This study aims to explore the similarity between deep neural networks and human perception by comparing twelve different networks, including both general object classifiers and FER-specific models. We employ an innovative global explainable AI method to generate heatmaps, revealing crucial facial regions for the twelve networks trained on six facial expressions. We assess these results both quantitatively and qualitatively, comparing them to ground truth masks based on Friesen and Ekman\u2019s description and among them. We use Intersection over Union (IoU) and normalized correlation coefficients for comparisons. We generate 72 heatmaps to highlight critical regions for each expression and architecture. Qualitatively, models with pre-trained weights show more similarity in heatmaps compared to those without pre-training. Specifically, eye and nose areas influence certain facial expressions, while the mouth is consistently important across all models and expressions. Quantitatively, we find low average IoU values (avg. 0.2702) across all expressions and architectures. The best-performing architecture averages 0.3269, while the worst-performing one averages 0.2066. Dendrograms, built with the normalized correlation coefficient, reveal two main clusters for most expressions: models with pre-training and models without pre-training. Findings suggest limited alignment between human and AI facial expression recognition, with network architectures influencing the similarity, as similar architectures prioritize similar facial regions.<\/jats:p>","DOI":"10.1007\/s11042-024-20090-5","type":"journal-article","created":{"date-parts":[[2024,8,28]],"date-time":"2024-08-28T07:02:29Z","timestamp":1724828549000},"page":"85725-85753","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Unveiling the human-like similarities of automatic facial expression recognition: An empirical exploration through explainable ai"],"prefix":"10.1007","volume":"83","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1231-7235","authenticated-orcid":false,"given":"F. Xavier","family":"Gaya-Morey","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1039-4387","authenticated-orcid":false,"given":"Silvia","family":"Ramis-Guarinos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8482-7552","authenticated-orcid":false,"given":"Cristina","family":"Manresa-Yee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6137-9558","authenticated-orcid":false,"given":"Jos\u00e9 M.","family":"Buades-Rubio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,8,28]]},"reference":[{"issue":"1","key":"20090_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/1529100619832930","volume":"20","author":"LF Barrett","year":"2019","unstructured":"Barrett LF, Adolphs R, Marsella S, Martinez AM, Pollak SD (2019) Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements. Psychologic Sci Public Interest 20(1):1\u201368. https:\/\/doi.org\/10.1177\/1529100619832930","journal-title":"Psychologic Sci Public Interest"},{"issue":"3\u20134","key":"20090_CR2","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1080\/02699939208411068","volume":"6","author":"P Ekman","year":"1992","unstructured":"Ekman P (1992) An argument for basic emotions. Cogn Emot 6(3\u20134):169\u2013200. https:\/\/doi.org\/10.1080\/02699939208411068","journal-title":"Cogn Emot"},{"key":"20090_CR3","unstructured":"Group I (2023) Affective Computing Market Report (2024-2032). Report ID: SR112024A3711. Technical report, IMARC Group . https:\/\/www.imarcgroup.com\/affective-computing-market"},{"key":"20090_CR4","doi-asserted-by":"publisher","unstructured":"Grabowski K, Rynkiewicz A, Lassalle A, Baron-Cohen S, Schuller B, Cummins N, Baird A, Podg\u00f3rska-Bednarz J, Pieni\u017cek A, \u0141ucka I (2019) Emotional expression in psychiatric conditions: New technology for clinicians. Psych Clinical Neurosci. 73(2):50\u201362 https:\/\/doi.org\/10.1111\/pcn.12799","DOI":"10.1111\/pcn.12799"},{"issue":"June","key":"20090_CR5","first-page":"163","volume":"9","author":"AM Barreto","year":"2017","unstructured":"Barreto AM (2017) Application of facial expression studies on the field of marketing. Emotional expression: the brain and the face. 9(June):163\u2013189","journal-title":"Emotional expression: the brain and the face."},{"issue":"2","key":"20090_CR6","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1007\/s00530-021-00854-x","volume":"28","author":"J Shen","year":"2022","unstructured":"Shen J, Yang H, Li J, Cheng Z (2022) Assessing learning engagement based on facial expression recognition in MOOC\u2019s scenario. Multimedia Syst 28(2):469\u2013478. https:\/\/doi.org\/10.1007\/s00530-021-00854-x","journal-title":"Multimedia Syst"},{"issue":"7","key":"20090_CR7","doi-asserted-by":"publisher","first-page":"0235908","DOI":"10.1371\/journal.pone.0235908","volume":"15","author":"S Medjden","year":"2020","unstructured":"Medjden S, Ahmed N, Lataifeh M (2020) Adaptive user interface design and analysis using emotion recognition through facial expressions and body posture from an RGB-D sensor. PLoS ONE 15(7):0235908. https:\/\/doi.org\/10.1371\/journal.pone.0235908","journal-title":"PLoS ONE"},{"key":"20090_CR8","doi-asserted-by":"publisher","unstructured":"Ramis S, Buades JM, Perales FJ (2020) Using a Social Robot to Evaluate Facial Expressions in the Wild. Sensors. 20:(23) https:\/\/doi.org\/10.3390\/s20236716","DOI":"10.3390\/s20236716"},{"issue":"12","key":"20090_CR9","doi-asserted-by":"publisher","first-page":"1424","DOI":"10.1109\/34.895976","volume":"22","author":"M Pantic","year":"2000","unstructured":"Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424\u20131445. https:\/\/doi.org\/10.1109\/34.895976","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"1","key":"20090_CR10","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/S0031-3203(02)00052-3","volume":"36","author":"B Fasel","year":"2003","unstructured":"Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259\u2013275. https:\/\/doi.org\/10.1016\/S0031-3203(02)00052-3","journal-title":"Pattern Recogn"},{"key":"20090_CR11","doi-asserted-by":"publisher","unstructured":"Li S, Deng W (2020) Deep Facial Expression Recognition: A Survey. IEEE Trans Affect Comput, 1 https:\/\/doi.org\/10.1109\/TAFFC.2020.2981446","DOI":"10.1109\/TAFFC.2020.2981446"},{"key":"20090_CR12","doi-asserted-by":"publisher","unstructured":"Mellouk W, Handouzi W (2020) Facial emotion recognition using deep learning: review and insights. The 17th International Conference on Mobile Systems and Pervasive Computing (MobiSPC),The 15th International Conference on Future Networks and Communications (FNC),The 10th International Conference on Sustainable Energy Information Technology. Procedia Computer Science. 175:689\u2013694 https:\/\/doi.org\/10.1016\/j.procs.2020.07.101","DOI":"10.1016\/j.procs.2020.07.101"},{"issue":"4","key":"20090_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pcbi.1004896","volume":"12","author":"J Kubilius","year":"2016","unstructured":"Kubilius J, Bracci S, Beeck HP (2016) Deep Neural Networks as a Computational Model for Human Shape Sensitivity. PLoS Comput Biol 12(4):1\u201326. https:\/\/doi.org\/10.1371\/journal.pcbi.1004896","journal-title":"PLoS Comput Biol"},{"key":"20090_CR14","doi-asserted-by":"crossref","unstructured":"Mehrer J, Spoerer CJ, Jones EC, Kriegeskorte N, Kietzmann TC (2021) An ecologically motivated image dataset for deep learning yields better models of human vision. Proc Natl Acad Sci 118(8):2011417118","DOI":"10.1073\/pnas.2011417118"},{"issue":"9","key":"20090_CR15","doi-asserted-by":"publisher","first-page":"5044","DOI":"10.1167\/jov.23.9.5044","volume":"23","author":"Y Chen","year":"2023","unstructured":"Chen Y, Cui L, Ding M (2023) Neural Processing of Affective Scenes: A Comparison between Convolutional Neural Networks and Human Visual Pathways. J Vis 23(9):5044. https:\/\/doi.org\/10.1167\/jov.23.9.5044","journal-title":"J Vis"},{"issue":"3","key":"20090_CR16","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1177\/1064804620920870","volume":"28","author":"ST Mueller","year":"2020","unstructured":"Mueller ST (2020) Cognitive anthropomorphism of ai: How humans and computers classify images. Ergonomics in Design. 28(3):12\u201319. https:\/\/doi.org\/10.1177\/1064804620920870","journal-title":"Ergonomics in Design."},{"key":"20090_CR17","doi-asserted-by":"publisher","unstructured":"Li M, Suh A (2021) Machinelike or humanlike? a literature review of anthropomorphism in ai-enabled technology. In: Proceedings of the 54th Hawaii International Conference on System Sciences. Proceedings of the Annual Hawaii International Conference on System Sciences, Research Unit(s) information for this publications provided by the author(s) concerned.; 54th Hawaii International Conference on System Sciences (HICSS 2021), HICSS-54 ; Conference date: 04-01-2021 Through 08-01-2021. pp 4053\u20134062 . https:\/\/doi.org\/10.24251\/HICSS.2021.493https:\/\/scholarspace.manoa.hawaii.edu\/handle\/10125\/72112","DOI":"10.24251\/HICSS.2021.493"},{"key":"20090_CR18","doi-asserted-by":"publisher","unstructured":"Borowski J, Funke CM, Stosio K, Brendel W, Wallis TSA, Bethge M (2019) The Notorious Difficulty of Comparing Human and Machine Perception, 642\u2013646 https:\/\/doi.org\/10.32470\/ccn.2019.1295-0","DOI":"10.32470\/ccn.2019.1295-0"},{"key":"20090_CR19","doi-asserted-by":"publisher","unstructured":"Fu K, Du C, Wang S, He H (2023) Improved video emotion recognition with alignment of cnn and human brain representations. IEEE Trans Affect Comput, 1\u201315 https:\/\/doi.org\/10.1109\/TAFFC.2023.3316173","DOI":"10.1109\/TAFFC.2023.3316173"},{"key":"20090_CR20","doi-asserted-by":"publisher","unstructured":"M\u00fcller R, D\u00fcrschmidt M, Ullrich J, Knoll C, Weber S, Seitz S (2024) Do humans and convolutional neural networks attend to similar areas during scene classification: Effects of task and image type. Appl Sci. 14(6) https:\/\/doi.org\/10.3390\/app14062648","DOI":"10.3390\/app14062648"},{"key":"20090_CR21","unstructured":"Lee J, Kim S, Won S, Lee J, Ghassemi M, Thorne J, Choi J, Kwon O-K, Choi E (2023) Visalign: Dataset for measuring the alignment between ai and humans in visual perception. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems, vol 36, pp 77119\u201377148. Curran Associates, Inc., ??? . https:\/\/proceedings.neurips.cc\/paper\/_files\/paper\/2023\/file\/f37aba0f53fdb59f53254fe9098b2177-Paper-Datasets\/_and\/_Benchmarks.pdf"},{"key":"20090_CR22","unstructured":"Geirhos R, Janssen DHJ, Sch\u00fctt HH, Rauber J, Bethge M, Wichmann FA (2017) Comparing deep neural networks against humans: object recognition when the signal gets weaker"},{"issue":"1","key":"20090_CR23","doi-asserted-by":"publisher","first-page":"32672","DOI":"10.1038\/srep32672","volume":"6","author":"SR Kheradpisheh","year":"2016","unstructured":"Kheradpisheh SR, Ghodrati M, Ganjtabesh M, Masquelier T (2016) Deep Networks Can Resemble Human Feed-forward Vision in Invariant Object Recognition. Sci Rep 6(1):32672. https:\/\/doi.org\/10.1038\/srep32672","journal-title":"Sci Rep"},{"key":"20090_CR24","doi-asserted-by":"publisher","unstructured":"Bowers JS, Malhotra G, Dujmovi\u0107 M, Llera\u00a0Montero M, Tsvetkov C, Biscione V, Puebla G, Adolfi F, Hummel JE, Heaton RF, al (2023) Deep problems with neural network models of human vision. Behavioral and Brain Sciences. 46:385 https:\/\/doi.org\/10.1017\/S0140525X22002813","DOI":"10.1017\/S0140525X22002813"},{"issue":"2","key":"20090_CR25","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1016\/j.neuron.2017.06.011","volume":"95","author":"D Hassabis","year":"2017","unstructured":"Hassabis D, Kumaran D, Summerfield C, Botvinick M (2017) Neuroscience-inspired artificial intelligence. Neuron 95(2):245\u2013258. https:\/\/doi.org\/10.1016\/j.neuron.2017.06.011","journal-title":"Neuron"},{"issue":"1","key":"20090_CR26","doi-asserted-by":"publisher","first-page":"1872","DOI":"10.1038\/s41467-021-22078-3","volume":"12","author":"G Jacob","year":"2021","unstructured":"Jacob G, Pramod RT, Katti H, Arun SP (2021) Qualitative similarities and differences in visual object representations between brains and deep networks. Nat Commun 12(1):1872. https:\/\/doi.org\/10.1038\/s41467-021-22078-3","journal-title":"Nat Commun"},{"issue":"10","key":"20090_CR27","doi-asserted-by":"publisher","first-page":"2744","DOI":"10.1073\/pnas.1513198113","volume":"113","author":"S Ullman","year":"2016","unstructured":"Ullman S, Assif L, Fetaya E, Harari D (2016) Atoms of recognition in human and computer vision. Proc Natl Acad Sci 113(10):2744\u20132749. https:\/\/doi.org\/10.1073\/pnas.1513198113","journal-title":"Proc Natl Acad Sci"},{"key":"20090_CR28","doi-asserted-by":"crossref","unstructured":"Ekman P, Friesen WV (1978) Manual for the Facial Action Coding System. Consulting Psychologists Press, ???","DOI":"10.1037\/t27734-000"},{"key":"20090_CR29","doi-asserted-by":"crossref","unstructured":"Khan RA, Meyer A, Konik H, Bouakaz S, Khan RA, Meyer A, Konik H, Bouakaz S (2013) Human vision inspired framework for facial expressions recognition. In: Image Processing (ICIP), 2012 19th IEEE International Conference On, Sep 2012, Orlando, FL, United States., pp 2593\u20132596","DOI":"10.1109\/ICIP.2012.6467429"},{"key":"20090_CR30","doi-asserted-by":"publisher","unstructured":"Benitez-Quiroz CF, Wang Y, Martinez AM (2017) Recognition of Action Units in the Wild with Deep Nets and a New Global-Local Loss. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 3990\u20133999 . https:\/\/doi.org\/10.1109\/ICCV.2017.428","DOI":"10.1109\/ICCV.2017.428"},{"key":"20090_CR31","doi-asserted-by":"publisher","unstructured":"Pham TTD, Won CS (2019) Facial action units for training convolutional neural networks. IEEE Access. 7:77816\u201377824 https:\/\/doi.org\/10.1109\/ACCESS.2019.2921241","DOI":"10.1109\/ACCESS.2019.2921241"},{"key":"20090_CR32","unstructured":"Benitez-Quiroz CF, Srinivasan R, Feng Q, Wang Y, Martinez AM (2017) EmotioNet Challenge: Recognition of facial expressions of emotion in the wild"},{"issue":"10","key":"20090_CR33","doi-asserted-by":"publisher","first-page":"10490","DOI":"10.1109\/TCYB.2021.3062830","volume":"52","author":"C Xu","year":"2022","unstructured":"Xu C, Liu H, Guan Z, Wu X, Tan J, Ling B (2022) Adversarial incomplete multiview subspace clustering networks. IEEE Trans Cyber 52(10):10490\u201310503. https:\/\/doi.org\/10.1109\/TCYB.2021.3062830","journal-title":"IEEE Trans Cyber"},{"issue":"2","key":"20090_CR34","doi-asserted-by":"publisher","first-page":"1456","DOI":"10.1109\/TII.2022.3206343","volume":"19","author":"C Xu","year":"2023","unstructured":"Xu C, Zhao W, Zhao J, Guan Z, Song X, Li J (2023) Uncertainty-aware multiview deep learning for internet of things applications. IEEE Trans Industr Inf 19(2):1456\u20131466. https:\/\/doi.org\/10.1109\/TII.2022.3206343","journal-title":"IEEE Trans Industr Inf"},{"key":"20090_CR35","doi-asserted-by":"publisher","unstructured":"Barredo Arrieta A, D\u00edaz-Rodr\u00edguez N, Del Ser J, Bennetot A, Tabik S, Barbado A, Garcia S, Gil-Lopez S, Molina D, Benjamins R, Chatila R, Herrera F (2020) Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion. 58:82\u2013115 https:\/\/doi.org\/10.1016\/j.inffus.2019.12.012arXiv:1910.10045","DOI":"10.1016\/j.inffus.2019.12.012"},{"key":"20090_CR36","doi-asserted-by":"publisher","unstructured":"Adadi A, Berrada M (2018) Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access. 6:52138\u201352160 https:\/\/doi.org\/10.1109\/ACCESS.2018.2870052","DOI":"10.1109\/ACCESS.2018.2870052"},{"issue":"2","key":"20090_CR37","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1609\/aimag.v40i2.2850","volume":"40","author":"D Gunning","year":"2019","unstructured":"Gunning D, Aha DW (2019) DARPA\u2019s Explainable Artificial Intelligence (XAI) Program. AI Mag 40(2):44\u201358. https:\/\/doi.org\/10.1609\/aimag.v40i2.2850","journal-title":"AI Mag"},{"key":"20090_CR38","doi-asserted-by":"publisher","unstructured":"Speith T (2022) A review of taxonomies of explainable artificial intelligence (xai) methods. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. FAccT \u201922, pp. 2239\u20132250. Association for Computing Machinery, New York, NY, USA . https:\/\/doi.org\/10.1145\/3531146.3534639. https:\/\/doi.org\/10.1145\/3531146.3534639","DOI":"10.1145\/3531146.3534639"},{"key":"20090_CR39","unstructured":"Friesen WV, Ekman P (1983) EMFACS-7: Emotional Facial Action Coding System"},{"key":"20090_CR40","doi-asserted-by":"crossref","unstructured":"Kandeel AA, Abbas HM, Hassanein HS (2021) Explainable Model Selection of a Convolutional Neural Network for Driver\u2019s Facial Emotion Identification. In: Del Bimbo, A., Cucchiara, R., Sclaroff, S., Farinella, G.M., Mei, T., Bertini, M., Escalante, H.J., Vezzani, R. (eds.) Pattern Recognition. ICPR International Workshops and Challenges, pp 699\u2013713. Springer, Cham","DOI":"10.1007\/978-3-030-68780-9_53"},{"key":"20090_CR41","doi-asserted-by":"crossref","unstructured":"Weitz K, Hassan T, Schmid U, Garbas J (2019) Deep-learned faces of pain and emotions: Elucidating the differences of facial expressions with the help of explainable AI methods. tm - Technisches Messen. 86:404\u2013412","DOI":"10.1515\/teme-2019-0024"},{"key":"20090_CR42","doi-asserted-by":"publisher","unstructured":"Manresa-Yee C, Ramis S, Buades JM (2023) Analysis of Gender Differences in Facial Expression Recognition Based on Deep Learning Using Explainable Artificial Intelligence. International Journal of Interactive Multimedia and Artificial Intelligence (In press). https:\/\/doi.org\/10.9781\/ijimai.2023.04.003","DOI":"10.9781\/ijimai.2023.04.003"},{"key":"20090_CR43","doi-asserted-by":"publisher","unstructured":"Manresa-Yee C, Ramis\u00a0Guarinos S, Buades\u00a0Rubio JM (2022) Facial expression recognition: Impact of gender on fairness and expressions. In: Proceedings of the XXII International Conference on Human Computer Interaction. Interacci\u00f3n \u201922. Association for Computing Machinery, New York, NY, USA . https:\/\/doi.org\/10.1145\/3549865.3549904","DOI":"10.1145\/3549865.3549904"},{"key":"20090_CR44","doi-asserted-by":"publisher","unstructured":"Sabater-G\u00e1rriz A, Gaya-Morey FX, Buades JM, Manresa-Yee C, Montoya P, Riquelme I (2024) Automated facial recognition system using deep learning for pain assessment in adults with cerebral palsy. Digital Health. (In press) https:\/\/doi.org\/10.1177\/20552076241259664","DOI":"10.1177\/20552076241259664"},{"key":"20090_CR45","doi-asserted-by":"publisher","unstructured":"Schiller D, Huber T, Dietz M, Andr\u00e9 E (2020) Relevance-Based Data Masking: A Model-Agnostic Transfer Learning Approach for Facial Expression Recognition. Frontier Compu Sci. 2:6 https:\/\/doi.org\/10.3389\/fcomp.2020.00006","DOI":"10.3389\/fcomp.2020.00006"},{"issue":"1","key":"20090_CR46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TAFFC.2020.3043603","volume":"1","author":"A Heimerl","year":"2020","unstructured":"Heimerl A, Weitz K, Baur T, Andre E (2020) Unraveling ML Models of Emotion with NOVA: Multi-Level Explainable AI for Non-Experts. IEEE Trans Affect Comput 1(1):1\u201313. https:\/\/doi.org\/10.1109\/TAFFC.2020.3043603","journal-title":"IEEE Trans Affect Comput"},{"key":"20090_CR47","doi-asserted-by":"publisher","unstructured":"Khorrami P, Paine TL, Huang TS (2015) Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition? In: 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), pp 19\u201327 . https:\/\/doi.org\/10.1109\/ICCVW.2015.12","DOI":"10.1109\/ICCVW.2015.12"},{"key":"20090_CR48","doi-asserted-by":"crossref","unstructured":"Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 Ieee Computer Society Conference on Computer Vision and Pattern Recognition-workshops, pp 94\u2013101 . IEEE","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"20090_CR49","unstructured":"Susskind JM, Anderson AK, Hinton GE (2010) The Toronto Face Database. Technical report"},{"key":"20090_CR50","doi-asserted-by":"publisher","unstructured":"Prajod P, Schiller D, Huber T, Andr\u00e9 E (2022) In: Shaban-Nejad, A., Michalowski, M., Bianco, S. (eds.) Do Deep Neural Networks Forget Facial Action Units?\u2014Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition, pp 217\u2013233. Springer, Cham . https:\/\/doi.org\/10.1007\/978-3-030-93080-6_16","DOI":"10.1007\/978-3-030-93080-6_16"},{"key":"20090_CR51","doi-asserted-by":"publisher","unstructured":"Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011) Painful data: The unbc-mcmaster shoulder pain expression archive database. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 57\u201364 . https:\/\/doi.org\/10.1109\/FG.2011.5771462","DOI":"10.1109\/FG.2011.5771462"},{"key":"20090_CR52","doi-asserted-by":"publisher","DOI":"10.1109\/IST50367.2021.9651357","author":"M Deramgozin","year":"2021","unstructured":"Deramgozin M, Jovanovic S, Rabah H, Ramzan N (2021). A Hybrid Explainable AI Framework Applied to Global and Local Facial Expression Recognition. https:\/\/doi.org\/10.1109\/IST50367.2021.9651357","journal-title":"A Hybrid Explainable AI Framework Applied to Global and Local Facial Expression Recognition"},{"key":"20090_CR53","doi-asserted-by":"publisher","unstructured":"Gund M, Bharadwaj AR, Nwogu I (2021) Interpretable emotion classification using temporal convolutional models. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp 6367\u20136374 . https:\/\/doi.org\/10.1109\/ICPR48806.2021.9412134","DOI":"10.1109\/ICPR48806.2021.9412134"},{"issue":"1","key":"20090_CR54","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1109\/TAFFC.2016.2573832","volume":"9","author":"AK Davison","year":"2018","unstructured":"Davison AK, Lansley C, Costen N, Tan K, Yap MH (2018) Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116\u2013129. https:\/\/doi.org\/10.1109\/TAFFC.2016.2573832","journal-title":"IEEE Trans Affect Comput"},{"issue":"12","key":"20090_CR55","doi-asserted-by":"publisher","first-page":"4383","DOI":"10.1126\/sciadv.abj4383","volume":"8","author":"L Zhou","year":"2022","unstructured":"Zhou L, Yang A, Meng M, Zhou K (2022) Emerged human-like facial expression representation in a deep convolutional neural network. Sci Adv 8(12):4383. https:\/\/doi.org\/10.1126\/sciadv.abj4383","journal-title":"Sci Adv"},{"key":"20090_CR56","unstructured":"Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pp 211\u2013216 . IEEE"},{"key":"20090_CR57","unstructured":"Lyons MJ, Akamatsu S, Kamachi M, Gyoba J, Budynek J (1998) The japanese female facial expression (jaffe) database. In: Third International Conference on Automatic Face and Gesture Recognition, pp 14\u201316"},{"key":"20090_CR58","doi-asserted-by":"publisher","first-page":"1516","DOI":"10.3389\/fpsyg.2014.01516","volume":"5","author":"M Olszanowski","year":"2015","unstructured":"Olszanowski M, Pochwatko G, Kuklinski K, Scibor-Rylski M, Lewinski P, Ohme RK (2015) Warsaw set of emotional facial expression pictures: a validation study of facial display photographs. Front Psychol 5:1516","journal-title":"Front Psychol"},{"issue":"27","key":"20090_CR59","doi-asserted-by":"publisher","first-page":"39507","DOI":"10.1007\/s11042-022-13117-2","volume":"81","author":"S Ramis","year":"2022","unstructured":"Ramis S, Buades JM, Perales FJ, Manresa-Yee C (2022) A novel approach to cross dataset studies in facial expression recognition. Multimedia Tools Appl. 81(27):39507\u201339544. https:\/\/doi.org\/10.1007\/s11042-022-13117-2","journal-title":"Multimedia Tools Appl."},{"issue":"4","key":"20090_CR60","doi-asserted-by":"publisher","first-page":"2091","DOI":"10.1137\/17M1118774","volume":"10","author":"J-L Lisani","year":"2017","unstructured":"Lisani J-L, Ramis S, Perales FJ (2017) A contrario detection of faces: A case example. SIAM J Imag Sci 10(4):2091\u20132118. https:\/\/doi.org\/10.1137\/17M1118774","journal-title":"SIAM J Imag Sci"},{"key":"20090_CR61","doi-asserted-by":"publisher","unstructured":"Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp 1867\u20131874 . https:\/\/doi.org\/10.1109\/CVPR.2014.241","DOI":"10.1109\/CVPR.2014.241"},{"issue":"2","key":"20090_CR62","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1007\/s11263-018-1097-z","volume":"127","author":"Y Wu","year":"2019","unstructured":"Wu Y, Ji Q (2019) Facial Landmark Detection: A Literature Survey. Int J Comput Vision 127(2):115\u2013142. https:\/\/doi.org\/10.1007\/s11263-018-1097-z","journal-title":"Int J Comput Vision"},{"key":"20090_CR63","doi-asserted-by":"publisher","unstructured":"McReynolds T, Blythe D (2005) Chapter 3 - color, shading, and lighting. In: McReynolds, T., Blythe, D. (eds.) Advanced Graphics Programming Using OpenGL. The Morgan Kaufmann Series in Computer Graphics, pp 35\u201356. Morgan Kaufmann, San Francisco . https:\/\/doi.org\/10.1016\/B978-155860659-3.50005-6","DOI":"10.1016\/B978-155860659-3.50005-6"},{"key":"20090_CR64","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol 25. Curran Associates, Inc., ???"},{"key":"20090_CR65","doi-asserted-by":"crossref","unstructured":"Simonyan K, Zisserman A (2015) Very Deep Convolutional Networks for Large-Scale Image Recognition","DOI":"10.1109\/ICCV.2015.314"},{"key":"20090_CR66","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J (2015) Deep Residual Learning for Image Recognition","DOI":"10.1109\/CVPR.2016.90"},{"key":"20090_CR67","doi-asserted-by":"crossref","unstructured":"Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2015) Rethinking the Inception Architecture for Computer Vision","DOI":"10.1109\/CVPR.2016.308"},{"key":"20090_CR68","doi-asserted-by":"crossref","unstructured":"Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceed IEEE Conf Comp Vis Pattern Recog (CVPR)","DOI":"10.1109\/CVPR.2017.195"},{"key":"20090_CR69","doi-asserted-by":"crossref","unstructured":"Howard A, Sandler M, Chu G, Chen L-C, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, Le QV, Adam H (2019) Searching for MobileNetV3","DOI":"10.1109\/ICCV.2019.00140"},{"key":"20090_CR70","unstructured":"Tan M, Le QV (2021) EfficientNetV2: Smaller Models and Faster Training"},{"key":"20090_CR71","doi-asserted-by":"publisher","unstructured":"Song I, Kim H-J, Jeon PB (2014) Deep learning for real-time robust facial expression recognition on a smartphone. In: 2014 IEEE International Conference on Consumer Electronics (ICCE), pp 564\u2013567 . https:\/\/doi.org\/10.1109\/ICCE.2014.6776135","DOI":"10.1109\/ICCE.2014.6776135"},{"key":"20090_CR72","doi-asserted-by":"crossref","unstructured":"Li W, Li M, Su Z, Zhu Z (2015) A deep-learning approach to facial expression recognition with candid images. In: 2015 14th IAPR International Conference on Machine Vision Applications (MVA), pp 279\u2013282 . IEEE","DOI":"10.1109\/MVA.2015.7153185"},{"key":"20090_CR73","doi-asserted-by":"publisher","unstructured":"He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conf Compu Vis Pattern Recog (CVPR), pp 770\u2013778 . https:\/\/doi.org\/10.1109\/CVPR.2016.90","DOI":"10.1109\/CVPR.2016.90"},{"key":"20090_CR74","doi-asserted-by":"crossref","unstructured":"Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke, V, Rabinovich A (2014) Going Deeper with Convolutions","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"20090_CR75","unstructured":"Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: Efficient Conv Neural Netw Mobile Vis Appl"},{"key":"20090_CR76","unstructured":"Tan M, Le QV (2020) EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks"},{"key":"20090_CR77","doi-asserted-by":"publisher","unstructured":"van der Velden BHM, Kuijf HJ, Gilhuijs KGA, Viergever MA (2022) Explainable artificial intelligence (xai) in deep learning-based medical image analysis. Medical Image Analysis. 79:102470 https:\/\/doi.org\/10.1016\/j.media.2022.102470","DOI":"10.1016\/j.media.2022.102470"},{"key":"20090_CR78","doi-asserted-by":"publisher","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \"Why should i trust you?\" Explaining the predictions of any classifier. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 13-17-Augu, 1135\u20131144 https:\/\/doi.org\/10.1145\/2939672.2939778","DOI":"10.1145\/2939672.2939778"},{"key":"20090_CR79","doi-asserted-by":"publisher","unstructured":"Alicioglu G, Sun B (2022) A survey of visual analytics for explainable artificial intelligence methods. Computers & Graphics. 102:502\u2013520 https:\/\/doi.org\/10.1016\/j.cag.2021.09.002","DOI":"10.1016\/j.cag.2021.09.002"},{"issue":"11","key":"20090_CR80","doi-asserted-by":"publisher","first-page":"2274","DOI":"10.1109\/TPAMI.2012.120","volume":"34","author":"R Achanta","year":"2012","unstructured":"Achanta R, Shaji A, Smith K, Lucchi A, Fua P, S\u00fcsstrunk S (2012) Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274\u20132282. https:\/\/doi.org\/10.1109\/TPAMI.2012.120","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"20090_CR81","doi-asserted-by":"publisher","unstructured":"Perveen N, Mohan C (2020) Configural Representation of Facial Action Units for Spontaneous Facial Expression Recognition in the Wild. In: 15th International Conference on Computer Vision Theory and Applications . https:\/\/doi.org\/10.5220\/0009099700930102","DOI":"10.5220\/0009099700930102"},{"issue":"1","key":"20090_CR82","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1109\/TSMC.1979.4310076","volume":"9","author":"N Otsu","year":"1979","unstructured":"Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62\u201366. https:\/\/doi.org\/10.1109\/TSMC.1979.4310076","journal-title":"IEEE Trans Syst Man Cybern"},{"key":"20090_CR83","doi-asserted-by":"publisher","unstructured":"Manresa-Yee C, Ramis S, Gaya-Morey FX, Buades JM (2024) Impact of explanations for trustworthy and transparent artificial intelligence. In: Proceedings of the XXIII International Conference on Human Computer Interaction. Interacci\u00f3n \u201923. Association for Computing Machinery, New York, NY, USA . https:\/\/doi.org\/10.1145\/3612783.3612798","DOI":"10.1145\/3612783.3612798"},{"key":"20090_CR84","first-page":"207","volume":"19","author":"P Ekman","year":"1971","unstructured":"Ekman P (1971) Universals and cultural differences in facial expressions of emotion. Nebr Symp Motiv 19:207\u2013283","journal-title":"Nebr Symp Motiv"},{"key":"20090_CR85","doi-asserted-by":"crossref","unstructured":"Peterson JC, Abbott JT, Griffiths TL (2018) Evaluating (and improving) the correspondence between deep neural networks and human representations. Cogn Sci 42(8):2648\u20132669","DOI":"10.1111\/cogs.12670"},{"key":"20090_CR86","unstructured":"Muttenthaler L, Linhardt L, Dippel J, Vandermeulen RA, Hermann K, Lampinen A, Kornblith S (2023) Improving neural network representations using human similarity judgments. In: Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S (eds.) Advances in Neural Information Processing Systems, vol 36, pp 50978\u201351007. Curran Associates, Inc., ??? . https:\/\/proceedings.neurips.cc\/paper\/_files\/paper\/2023\/file\/9febda1c8344cc5f2d51713964864e93-Paper-Conference.pdf"},{"key":"20090_CR87","unstructured":"Geirhos R, Meding K, Wichmann FA (2020) Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS \u201920. Curran Associates Inc., Red Hook, NY, USA"},{"key":"20090_CR88","doi-asserted-by":"crossref","unstructured":"Guidotti R, Monreale A, Ruggieri S, Turini F, Pedreschi D, Giannotti F (2018) A Survey Of Methods For Explaining Black Box Models","DOI":"10.1145\/3236009"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-024-20090-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-024-20090-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-024-20090-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T14:08:36Z","timestamp":1732025316000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-024-20090-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,28]]},"references-count":88,"journal-issue":{"issue":"38","published-online":{"date-parts":[[2024,11]]}},"alternative-id":["20090"],"URL":"https:\/\/doi.org\/10.1007\/s11042-024-20090-5","relation":{},"ISSN":["1573-7721"],"issn-type":[{"value":"1573-7721","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,28]]},"assertion":[{"value":"22 February 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 July 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 August 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 August 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}