{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T12:29:49Z","timestamp":1783427389242,"version":"3.54.6"},"reference-count":92,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2022,7,1]],"date-time":"2022-07-01T00:00:00Z","timestamp":1656633600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Big Data &amp; Society"],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>Emotion recognition, and in particular acial emotion recognition (FER), is among the most controversial applications of machine learning, not least because of its ethical implications for human subjects. In this article, we address the controversial conjecture that machines can read emotions from our facial expressions by asking whether this task can be performed reliably. This means, rather than considering the potential harms or scientific soundness of facial emotion recognition systems, focusing on the reliability of the ground truths used to develop emotion recognition systems, assessing how well different human observers agree on the emotions they detect in subjects\u2019 faces. Additionally, we discuss the extent to which sharing context can help observers agree on the emotions they perceive on subjects\u2019 faces. Briefly, we demonstrate that when large and heterogeneous samples of observers are involved, the task of emotion detection from static images crumbles into inconsistency. We thus reveal that any endeavour to understand human behaviour from large sets of labelled patterns is over-ambitious, even if it were technically feasible. We conclude that we cannot speak of actual accuracy for facial emotion recognition systems for any practical purposes.<\/jats:p>","DOI":"10.1177\/20539517221129549","type":"journal-article","created":{"date-parts":[[2022,10,3]],"date-time":"2022-10-03T02:09:22Z","timestamp":1664762962000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":25,"title":["The unbearable (technical) unreliability of automated facial emotion recognition"],"prefix":"10.1177","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4065-3415","authenticated-orcid":false,"given":"Federico","family":"Cabitza","sequence":"first","affiliation":[{"name":"University of Milan-Bicocca, Milan, Italy"},{"name":"IRCCS Istituto Ortopedico Galeazzi, Milan, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0027-5157","authenticated-orcid":false,"given":"Andrea","family":"Campagner","sequence":"additional","affiliation":[{"name":"University of Milan-Bicocca, Milan, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martina","family":"Mattioli","sequence":"additional","affiliation":[{"name":"University of Milan-Bicocca, Milan, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2022,10,2]]},"reference":[{"key":"bibr1-20539517221129549","first-page":"1","volume":"1","author":"Abu-Lughod L","year":"1990","journal-title":"Language and the Politics of Emotion"},{"key":"bibr2-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1186\/1745-6215-14-161"},{"key":"bibr3-20539517221129549","unstructured":"Authors V (2020) Emotional entanglement: China\u2019s emotion recognition market and its implications for human rights. Technical report, ARTICLE 19."},{"key":"bibr4-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1177\/0301006616638595"},{"key":"bibr5-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1207\/s15327957pspr1001_2"},{"key":"bibr6-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1177\/1529100619832930"},{"key":"bibr7-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2018.8489099"},{"key":"bibr8-20539517221129549","unstructured":"Basile V, Cabitza F, Campagner A, et al. (2021) Toward a perspectivist turn in ground truthing for predictive computing.\n                      arXiv preprint arXiv:2109.04270\n                      ."},{"key":"bibr9-20539517221129549","unstructured":"Bender EM (2022) Look behind the curtain: Don\u2019t be dazzled by claims of \u2018artificial intelligence\u2019.\n                      The Seattle Times\n                      ."},{"key":"bibr10-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00041"},{"key":"bibr11-20539517221129549","unstructured":"Benitez-Quiroz CF, Srinivasan R, Feng Q, et al. (2017) Emotionet challenge: Recognition of facial expressions of emotion in the wild.\n                      arXiv preprint arXiv:1703.01210\n                      ."},{"key":"bibr12-20539517221129549","doi-asserted-by":"crossref","unstructured":"Birhane A, Prabhu VU (2021) Large image datasets: A pyrrhic win for computer vision? In:\n                      2021 IEEE Winter Conference on Applications of Computer Vision (WACV)\n                      . IEEE, pp. 1536\u20131546.","DOI":"10.1109\/WACV48630.2021.00158"},{"key":"bibr13-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/ICoCS.2019.8930761"},{"key":"bibr14-20539517221129549","doi-asserted-by":"publisher","DOI":"10.17265\/2159-5313\/2016.09.003"},{"key":"bibr15-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1177\/01634437211036975"},{"key":"bibr16-20539517221129549","doi-asserted-by":"crossref","unstructured":"Bullington J (2005) \u2019affective\u2019 computing and emotion recognition systems: the future of biometric surveillance? In:\n                      Proceedings of the 2nd annual conference on Information security curriculum development\n                      . pp. 95\u201399.","DOI":"10.1145\/1107622.1107644"},{"key":"bibr17-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-90503-7_10"},{"key":"bibr18-20539517221129549","doi-asserted-by":"publisher","DOI":"10.3390\/app10114014"},{"issue":"1","key":"bibr19-20539517221129549","first-page":"1","volume":"20","author":"Cabitza F","journal-title":"BMC Medical Informatics and Decision Making"},{"key":"bibr20-20539517221129549","doi-asserted-by":"crossref","unstructured":"Cabitza F, Campagner A, Datteri E (2021) To err is (only) human. reflections on how to move from accuracy to trust for medical ai. In:\n                      Exploring Innovation in a Digital World\n                      . Springer, pp. 36\u201349.","DOI":"10.1007\/978-3-030-87842-9_4"},{"issue":"2","key":"bibr21-20539517221129549","first-page":"249","volume":"22","author":"Carletta J","year":"1996","journal-title":"Computational Linguistics"},{"key":"bibr22-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287587"},{"key":"bibr23-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"bibr24-20539517221129549","unstructured":"Crawford K (2021) Artificial intelligence is misreading human emotion.\n                      The Atlantic\n                      https:\/\/www.theatlantic.com\/technology\/archive\/2021\/04\/artifi."},{"key":"bibr25-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0079131"},{"key":"bibr26-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3131733"},{"key":"bibr27-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2011.6130508"},{"issue":"45","key":"bibr28-20539517221129549","first-page":"16","volume":"98","author":"Ekman P","year":"1999","journal-title":"Handbook of Cognition and Emotion"},{"key":"bibr29-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992253"},{"key":"bibr30-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.128.2.203"},{"key":"bibr31-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-020-03509-5"},{"key":"bibr32-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"bibr33-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1145\/3358695.3361749"},{"key":"bibr34-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"bibr35-20539517221129549","doi-asserted-by":"crossref","unstructured":"Ghotbi N, Ho MT, Mantello P (2021) Attitude of college students towards ethical issues of artificial intelligence in an international university in japan.\n                      AI & SOCIETY\n                      : 1\u20138.","DOI":"10.1007\/s00146-021-01168-2"},{"key":"bibr36-20539517221129549","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9302.001.0001"},{"key":"bibr37-20539517221129549","volume-title":"The Emotions: A Philosophical Exploration","author":"Goldie P","year":"2000"},{"key":"bibr38-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"bibr39-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1145\/3123266.3123383"},{"key":"bibr40-20539517221129549","unstructured":"Harvey A, LaPlace J (2021) Exposing.ai. https:\/\/exposing.ai."},{"key":"bibr41-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1080\/19312450709336664"},{"key":"bibr42-20539517221129549","unstructured":"Holland S, Hosny A, Newman S, et al. (2018) The dataset nutrition label: A framework to drive higher data quality standards.\n                      arXiv preprint arXiv:1805.03677\n                      ."},{"key":"bibr43-20539517221129549","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.3840403"},{"key":"bibr44-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2018.04.060"},{"key":"bibr45-20539517221129549","unstructured":"Jay S (2017) What\u2019s wrong with airport face recognition?\n                      ACLU\n                      https:\/\/www.aclu.org\/blog\/privacy-technology\/surveillance-tec."},{"key":"bibr46-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2019.8857005"},{"key":"bibr47-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/SEmotion.2019.00008"},{"key":"bibr48-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2019.02.004"},{"key":"bibr49-20539517221129549","doi-asserted-by":"publisher","DOI":"10.3390\/s18020401"},{"key":"bibr50-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-08491-6_5"},{"key":"bibr51-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.212"},{"key":"bibr52-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-2958.2004.tb00738.x"},{"key":"bibr53-20539517221129549","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff K","year":"2018"},{"key":"bibr54-20539517221129549","doi-asserted-by":"publisher","DOI":"10.2307\/2529786"},{"key":"bibr55-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1080\/02699930903485076"},{"key":"bibr56-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-25352-6"},{"key":"bibr57-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1016\/j.cobeha.2017.09.011"},{"key":"bibr58-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2018.2868382"},{"key":"bibr59-20539517221129549","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2014.01532"},{"key":"bibr60-20539517221129549","first-page":"1","author":"Mantello P","year":"2021","journal-title":"AI & Society"},{"key":"bibr61-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1016\/S0263-2241(03)00016-2"},{"key":"bibr62-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720904386"},{"issue":"10","key":"bibr63-20539517221129549","volume":"24","author":"McStay A","year":"2019","journal-title":"First Monday"},{"key":"bibr64-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/s42452-020-2234-1"},{"key":"bibr65-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/BF02686918"},{"key":"bibr66-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2020.07.101"},{"key":"bibr67-20539517221129549","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-016-0756-7"},{"key":"bibr68-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2017.2740923"},{"key":"bibr69-20539517221129549","unstructured":"Mozur P (2020) One month, 500, 000 face scans: How china is using a.i. to profile a minority.\n                      New York Times\n                      www.nytimes.com\/2019\/04\/14\/technology\/china-surveillance-art."},{"key":"bibr70-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/Confluence47617.2020.9057985"},{"key":"bibr71-20539517221129549","volume-title":"Logic of Feeling: Technology\u2019s Quest to Capitalize Emotion","author":"Munn L","year":"2020"},{"key":"bibr72-20539517221129549","doi-asserted-by":"publisher","DOI":"10.4135\/9781071802878"},{"key":"bibr73-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.9"},{"key":"bibr74-20539517221129549","doi-asserted-by":"publisher","DOI":"10.17265\/2159-5313\/2016.09.003"},{"key":"bibr75-20539517221129549","unstructured":"Northcutt CG, Athalye A, Mueller J (2021) Pervasive label errors in test sets destabilize machine learning benchmarks.\n                      arXiv preprint arXiv:2103.14749\n                      ."},{"key":"bibr76-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1037\/pne0000047"},{"key":"bibr77-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2021.100336"},{"key":"bibr78-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517578"},{"key":"bibr79-20539517221129549","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v23i9.9406"},{"key":"bibr80-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445939"},{"key":"bibr81-20539517221129549","unstructured":"Stark L, Hutson J (Forthcoming) Physiognomic artificial intelligence.\n                      Fordham Intellectual Property, Media & Entertainment Law Journal\n                      Available at SSRN: https:\/\/ssrn.com\/abstract=3927300."},{"key":"bibr82-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29739-8_17"},{"key":"bibr83-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00583"},{"key":"bibr84-20539517221129549","unstructured":"Wakefield J (2021) Ai emotion-detection software tested on uyghurs.\n                      BBC News\n                      https:\/\/www.bbc.com\/news\/technology-57101248."},{"key":"bibr85-20539517221129549","doi-asserted-by":"publisher","DOI":"10.17265\/2159-5313\/2016.09.003"},{"key":"bibr86-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2010.2060716"},{"key":"bibr87-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1007\/s12559-021-09936-4"},{"key":"bibr88-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1080\/02699052.2018.1531300"},{"key":"bibr89-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1016\/j.beth.2017.12.010"},{"key":"bibr90-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0086041"},{"key":"bibr91-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0231304"},{"key":"bibr92-20539517221129549","doi-asserted-by":"publisher","DOI":"10.1145\/3388790"}],"container-title":["Big Data &amp; Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20539517221129549","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/20539517221129549","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20539517221129549","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T12:58:40Z","timestamp":1777381120000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/20539517221129549"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":92,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,7]]}},"alternative-id":["10.1177\/20539517221129549"],"URL":"https:\/\/doi.org\/10.1177\/20539517221129549","relation":{},"ISSN":["2053-9517","2053-9517"],"issn-type":[{"value":"2053-9517","type":"print"},{"value":"2053-9517","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7]]},"article-number":"20539517221129549"}}