{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T02:30:07Z","timestamp":1775097007335,"version":"3.50.1"},"reference-count":61,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2021,7,25]],"date-time":"2021-07-25T00:00:00Z","timestamp":1627171200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Applied Sciences"],"abstract":"<jats:p>Understanding how acting bridges the emotional bond between spectators and films is essential to depict how humans interact with this rapidly growing digital medium. In recent decades, the research community made promising progress in developing facial expression recognition (FER) methods. However, no emphasis has been put in cinematographic content, which is complex by nature due to the visual techniques used to convey the desired emotions. Our work represents a step towards emotion identification in cinema through facial expressions\u2019 analysis. We presented a comprehensive overview of the most relevant datasets used for FER, highlighting problems caused by their heterogeneity and to the inexistence of a universal model of emotions. Built upon this understanding, we evaluated these datasets with a standard image classification models to analyze the feasibility of using facial expressions to determine the emotional charge of a film. To cope with the problem of lack of datasets for the scope under analysis, we demonstrated the feasibility of using a generic dataset for the training process and propose a new way to look at emotions by creating clusters of emotions based on the evidence obtained in the experiments.<\/jats:p>","DOI":"10.3390\/app11156827","type":"journal-article","created":{"date-parts":[[2021,7,25]],"date-time":"2021-07-25T22:06:21Z","timestamp":1627250781000},"page":"6827","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Emotion Identification in Movies through Facial Expression Recognition"],"prefix":"10.3390","volume":"11","author":[{"given":"Jo\u00e3o","family":"Almeida","sequence":"first","affiliation":[{"name":"INESC TEC, 4200-465 Porto, Portugal"},{"name":"Faculty of Engineering, University of Porto, 4200-465 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3640-7019","authenticated-orcid":false,"given":"Lu\u00eds","family":"Vila\u00e7a","sequence":"additional","affiliation":[{"name":"INESC TEC, 4200-465 Porto, Portugal"},{"name":"School of Engineering, Polytechnic of Porto, 4200-072 Porto, Portugal"}]},{"given":"In\u00eas N.","family":"Teixeira","sequence":"additional","affiliation":[{"name":"INESC TEC, 4200-465 Porto, Portugal"},{"name":"Faculty of Engineering, University of Porto, 4200-465 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8447-2360","authenticated-orcid":false,"given":"Paula","family":"Viana","sequence":"additional","affiliation":[{"name":"INESC TEC, 4200-465 Porto, Portugal"},{"name":"School of Engineering, Polytechnic of Porto, 4200-072 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,25]]},"reference":[{"key":"ref_1","unstructured":"Segerstrale, U., and Molnar, P. (1997). Universal facial expressions of emotion. Nonverbal Communication: Where Nature Meets Culture, Routledge."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1080\/02699939208411068","article-title":"An argument for basic emotions","volume":"6","author":"Ekman","year":"1992","journal-title":"Cogn. Emot."},{"key":"ref_3","unstructured":"Ortony, A., Clore, G.L., and Collins, A. (1990). The Cognitive Structure of Emotions, Cambridge University Press."},{"key":"ref_4","unstructured":"Prinz, J.J. (2004). Gut Reactions: A Perceptual Theory of Emotion, Oxford University Press."},{"key":"ref_5","unstructured":"Parrott, W.G. (2001). Emotions in Social Psychology: Essential Readings, Psychology Press."},{"key":"ref_6","first-page":"5","article-title":"Facial action coding system: A technique for the measurement of facial movement","volume":"3","author":"Friesen","year":"1978","journal-title":"Palo Alto"},{"key":"ref_7","unstructured":"Ekman, P., and Rosenberg, E.L. (2020). What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS), Oxford University Press."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Fabian Benitez-Quiroz, C., Srinivasan, R., and Martinez, A.M. (2016, January 27\u201330). Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.600"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1017\/S0954579405050340","article-title":"The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology","volume":"17","author":"Posner","year":"2005","journal-title":"Dev. Psychopathol."},{"key":"ref_10","first-page":"173","article-title":"The psychophysiology of emotion","volume":"Volume 2","author":"Cacioppo","year":"2000","journal-title":"Handbook of Emotions"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"7241","DOI":"10.1073\/pnas.1200155109","article-title":"Facial expressions of emotion are not culturally universal","volume":"109","author":"Jack","year":"2012","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_12","unstructured":"Saarni, C. (1999). The Development of Emotional Competence, Guilford Press."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Whissell, C.M. (1989). The dictionary of affect in language. The Measurement of Emotions, Elsevier.","DOI":"10.1016\/B978-0-12-558704-4.50011-6"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/BF02686918","article-title":"Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament","volume":"14","author":"Mehrabian","year":"1996","journal-title":"Curr. Psychol."},{"key":"ref_16","unstructured":"Greenwald, M.K., Cook, E.W., and Lang, P.J. (1989). Affective judgment and psychophysiological response: Dimensional covariation in the evaluation of pictorial stimuli. J. Psychophysiol., Available online: https:\/\/psycnet.apa.org\/record\/1990-03841-001."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1111\/j.1467-9280.2007.02024.x","article-title":"The world of emotions is not two-dimensional","volume":"18","author":"Fontaine","year":"2007","journal-title":"Psychol. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Gebhard, P. (2005, January 25\u201329). ALMA: A layered model of affect. Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems, Utrecht, The Netherlands.","DOI":"10.1145\/1082473.1082478"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s12193-011-0071-7","article-title":"Affective transfer computing model based on attenuation emotion mechanism","volume":"5","author":"Shi","year":"2012","journal-title":"J. MultiModal User Interfaces"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Landowska, A. (2018). Towards new mappings between emotion representation models. Appl. Sci., 8.","DOI":"10.3390\/app8020274"},{"key":"ref_21","unstructured":"Bradley, M.M., and Lang, P.J. (1999). Affective Norms for English Words (ANEW): Instruction Manual and Affective Ratings, The Center for Research in Psychophysiology, University of Florida. Technical report, Technical report C-1."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1109\/T-AFFC.2013.18","article-title":"Synesketch: An open source library for sentence-based emotion recognition","volume":"4","author":"Krcadinac","year":"2013","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1222","DOI":"10.3758\/s13428-014-0552-1","article-title":"Nencki affective word list (NAWL): The cultural adaptation of the Berlin affective word list\u2013reloaded (BAWL-R) for Polish","volume":"47","author":"Riegel","year":"2015","journal-title":"Behav. Res. Methods"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wierzba, M., Riegel, M., Wypych, M., Jednor\u00f3g, K., Turnau, P., Grabowska, A., and Marchewka, A. (2015). Basic emotions in the Nencki Affective Word List (NAWL BE): New method of classifying emotional stimuli. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0132305"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1177\/0305735610362821","article-title":"A comparison of the discrete and dimensional models of emotion in music","volume":"39","author":"Eerola","year":"2011","journal-title":"Psychol. Music."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.H. (2013). Challenges in representation learning: A report on three machine learning contests. International Conference on Neural Information Processing, Springer.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Dhall, A., Goecke, R., Joshi, J., Hoey, J., and Gedeon, T. (2016, January 12\u201316). Emotiw 2016: Video and group-level emotion recognition challenges. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan.","DOI":"10.1145\/2993148.2997638"},{"key":"ref_28","first-page":"34","article-title":"Collecting large, richly annotated facial-expression databases from movies","volume":"19","author":"Dhall","year":"2012","journal-title":"IEEE Ann. Hist. Comput."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.imavis.2017.02.001","article-title":"AFEW-VA database for valence and arousal estimation in-the-wild","volume":"65","author":"Kossaifi","year":"2017","journal-title":"Image Vis. Comput."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/TAFFC.2017.2740923","article-title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"Mollahosseini","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_31","unstructured":"Kollias, D., and Zafeiriou, S. (2018). Aff-wild2: Extending the aff-wild database for affect recognition. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1109\/TAFFC.2018.2801311","article-title":"Am-fed+: An extended dataset of naturalistic facial expressions collected in everyday settings","volume":"10","author":"McDuff","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13\u201318). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"ref_34","unstructured":"Lyons, M., Akamatsu, S., Kamachi, M., and Gyoba, J. (1998, January 14\u201316). Coding facial expressions with gabor wavelets. Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"109","DOI":"10.3758\/BRM.40.1.109","article-title":"Facial expressions of emotion (KDEF): Identification under different display-duration conditions","volume":"40","author":"Calvo","year":"2008","journal-title":"Behav. Res. Methods"},{"key":"ref_36","unstructured":"Pantic, M., Valstar, M., Rademaker, R., and Maat, L. (2005, January 6). Web-based database for facial expression analysis. Proceedings of the 2005 IEEE International Conference on Multimedia and Expo, Amsterdam, The Netherlands."},{"key":"ref_37","unstructured":"Valstar, M., and Pantic, M. (2010, January 17\u201323). Induced disgust, happiness and surprise: An addition to the mmi facial expression database. Proceedings of the 3rd International Workshop on EMOTION (Satellite of LREC): Corpora for Research on Emotion and Affect, Valletta, Malta."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1016\/j.imavis.2011.07.002","article-title":"Facial expression recognition from near-infrared videos","volume":"29","author":"Zhao","year":"2011","journal-title":"Image Vis. Comput."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Li, S., Deng, W., and Du, J. (2017, January 21\u201326). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.277"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1109\/TIP.2018.2868382","article-title":"Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_41","unstructured":"Dhall, A., Goecke, R., Lucey, S., and Gedeon, T. (2011). Acted Facial Expressions in the Wild Database, Australian National University. Technical Report TR-CS-11."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Cohn, J.F., Ertugrul, I.O., Chu, W.S., Girard, J.M., Jeni, L.A., and Hammal, Z. (2019). Affective facial computing: Generalizability across domains. Multimodal Behavior Analysis in the Wild, Elsevier.","DOI":"10.1016\/B978-0-12-814601-9.00026-2"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Meng, D., Peng, X., Wang, K., and Qiao, Y. (2019, January 22\u201325). Frame attention networks for facial expression recognition in videos. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803603"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1007\/s11263-017-1055-1","article-title":"From facial expression recognition to interpersonal relation prediction","volume":"126","author":"Zhang","year":"2018","journal-title":"Int. J. Comput. Vis."},{"key":"ref_45","unstructured":"Breuer, R., and Kimmel, R. (2017). A deep learning perspective on the origin of facial expressions. arXiv."},{"key":"ref_46","unstructured":"Pramerdorfer, C., and Kampel, M. (2016). Facial expression recognition using convolutional neural networks: State of the art. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1109\/TAFFC.2017.2695999","article-title":"Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition","volume":"10","author":"Kim","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Hamester, D., Barros, P., and Wermter, S. (2015, January 12\u201317). Face expression recognition with a 2-channel convolutional neural network. Proceedings of the 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland.","DOI":"10.1109\/IJCNN.2015.7280539"},{"key":"ref_49","unstructured":"Minaee, S., and Abdolrashidi, A. (2019). Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Yu, Z., and Zhang, C. (2015, January 9\u201313). Image based static facial expression recognition with multiple deep network learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830595"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Kim, B.K., Lee, H., Roh, J., and Lee, S.Y. (2015, January 9\u201313). Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830590"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"2439","DOI":"10.1109\/TIP.2018.2886767","article-title":"Occlusion aware facial expression recognition using CNN with attention mechanism","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Yang, H., Zhang, Z., and Yin, L. (2018, January 15\u201319). Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00050"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Ng, H.W., Nguyen, V.D., Vonikakis, V., and Winkler, S. (2015, January 9\u201313). Deep learning for emotion recognition on small datasets using transfer learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830593"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Ding, H., Zhou, S.K., and Chellappa, R. (June, January 30). Facenet2expnet: Regularizing a deep face recognition net for expression recognition. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.","DOI":"10.1109\/FG.2017.23"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Yao, A., Cai, D., Hu, P., Wang, S., Sha, L., and Chen, Y. (2016, January 12\u201316). HoloNet: Towards robust emotion recognition in the wild. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan.","DOI":"10.1145\/2993148.2997639"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Hu, P., Cai, D., Wang, S., Yao, A., and Chen, Y. (2017, January 13\u201317). Learning supervised scoring ensemble for emotion recognition in the wild. Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK.","DOI":"10.1145\/3136755.3143009"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Cai, J., Meng, Z., Khan, A.S., Li, Z., O\u2019Reilly, J., and Tong, Y. (2018, January 15\u201319). Island loss for learning discriminative features in facial expression recognition. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00051"},{"key":"ref_59","unstructured":"Guo, Y., Tao, D., Yu, J., Xiong, H., Li, Y., and Tao, D. (2016, January 11\u201315). Deep neural networks with relativity learning for facial expression recognition. Proceedings of the 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Seattle, WA, USA."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Liu, X., Vijaya Kumar, B., You, J., and Jia, P. (2017, January 21\u201326). Adaptive deep metric learning for identity-aware facial expression recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.79"},{"key":"ref_61","unstructured":"Li, S., and Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput."}],"container-title":["Applied Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3417\/11\/15\/6827\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:34:39Z","timestamp":1760164479000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3417\/11\/15\/6827"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,25]]},"references-count":61,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["app11156827"],"URL":"https:\/\/doi.org\/10.3390\/app11156827","relation":{},"ISSN":["2076-3417"],"issn-type":[{"value":"2076-3417","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,25]]}}}