{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T19:40:43Z","timestamp":1776886843032,"version":"3.51.2"},"reference-count":46,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2024,4,19]],"date-time":"2024-04-19T00:00:00Z","timestamp":1713484800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the JOY Neuro-wellness Grant"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants\u2019 emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.<\/jats:p>","DOI":"10.3390\/s24082620","type":"journal-article","created":{"date-parts":[[2024,4,19]],"date-time":"2024-04-19T08:44:31Z","timestamp":1713516271000},"page":"2620","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-5936-6119","authenticated-orcid":false,"given":"Shlomi","family":"Talala","sequence":"first","affiliation":[{"name":"Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva 84105, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaul","family":"Shvimmer","sequence":"additional","affiliation":[{"name":"Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva 84105, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rotem","family":"Simhon","sequence":"additional","affiliation":[{"name":"School of Psychology, Tel Aviv University, Tel Aviv 39040, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Gilead","sequence":"additional","affiliation":[{"name":"School of Psychology, Tel Aviv University, Tel Aviv 39040, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4974-9683","authenticated-orcid":false,"given":"Yitzhak","family":"Yitzhaky","sequence":"additional","affiliation":[{"name":"Department of Electro-Optics and Photonics Engineering, School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Beer Sheva 84105, Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,4,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"\u0160imi\u0107, G., Tkal\u010di\u0107, M., Vuki\u0107, V., Mulc, D., \u0160pani\u0107, E., \u0160agud, M., Olucha-Bordonau, F.E., Vuk\u0161i\u0107, M., and Hof, P.R. (2021). Understanding Emotions: Origins and Roles of the Amygdala. Biomolecules, 11.","DOI":"10.3390\/biom11060823"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"23311","DOI":"10.1007\/s00521-021-06012-8","article-title":"Deep learning-based facial emotion recognition for human\u2013computer interaction applications","volume":"35","author":"Chowdary","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1016\/j.ins.2021.10.005","article-title":"A survey on facial emotion recognition techniques: A state-of-the-art literature review","volume":"582","author":"Canal","year":"2022","journal-title":"Inf. Sci."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1109\/TAFFC.2020.2981446","article-title":"Deep Facial Expression Recognition: A Survey","volume":"13","author":"Li","year":"2022","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2525","DOI":"10.1007\/s11277-023-10244-3","article-title":"Speech Emotion Recognition: A Comprehensive Survey","volume":"129","author":"Jawad","year":"2023","journal-title":"Wirel. Pers. Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1016\/j.biopsycho.2010.03.010","article-title":"Autonomic nervous system activity in emotion: A review","volume":"84","author":"Kreibig","year":"2010","journal-title":"Biol. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1177\/1754073913512003","article-title":"The Autonomic Nervous System and Emotion","volume":"6","author":"Levenson","year":"2014","journal-title":"Emot. Rev."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lin, W., and Li, C. (2023). Review of Studies on Emotion Recognition and Judgment Based on Physiological Signals. Appl. Sci., 13.","DOI":"10.3390\/app13042573"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1006\/nimg.2002.1087","article-title":"Functional Neuroanatomy of Emotion: A Meta-Analysis of Emotion Activation Studies in PET and fMRI","volume":"16","author":"Phan","year":"2002","journal-title":"NeuroImage"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Lee, M.S., Lee, Y.K., Pae, D.S., Lim, M.T., Kim, D.W., and Kang, T.K. (2019). Fast Emotion Recognition Based on Single Pulse PPG Signal with Convolutional Neural Network. Appl. Sci., 9.","DOI":"10.3390\/app9163355"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"801709","DOI":"10.3389\/fphys.2022.801709","article-title":"Contactless Vital Signs Monitoring From Videos Recorded With Digital Cameras: An Overview","volume":"13","author":"Molinaro","year":"2022","journal-title":"Front. Physiol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, E.J., Li, W., Hawkins, D., Gernsheimer, T., Norby-Slycord, C., and Patel, S.N. (2016, January 12\u201316). HemaApp: Noninvasive blood screening of hemoglobin using smartphone cameras. Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Heidelberg, Germany.","DOI":"10.1145\/2971648.2971653"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Baranoski, G.V.G., and Krishnaswamy, A. (2010). Light & Skin Interactions, Morgan Kaufmann.","DOI":"10.1016\/B978-0-12-375093-8.00001-0"},{"key":"ref_14","first-page":"081","article-title":"Skin, Light and their Interactions, an In-Depth Review for Modern Light-Based Skin Therapies","volume":"7","author":"Alhallak","year":"2021","journal-title":"J. Clin. Derm. Ther."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wang, E.J., Li, W., Zhu, J., Rana, R., and Patel, S.N. (2017, January 11\u201315). Noninvasive hemoglobin measurement using unmodified smartphone camera and white flash. Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju, Republic of Korea.","DOI":"10.1109\/EMBC.2017.8037323"},{"key":"ref_16","unstructured":"Abdallah, O., Natsheh, M., Alam, K.A., Qananwah, Q., Nabulsi, A.A., and Bolz, A. (2010). Biophotonics: Photonic Solutions for Better Health Care II, SPIE."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"11188","DOI":"10.1038\/s41598-022-14808-4","article-title":"Classification of emotional states via transdermal cardiovascular spatiotemporal facial patterns using multispectral face videos","volume":"12","author":"Shvimmer","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Benezeth, Y., Li, P., Macwan, R., Nakamura, K., Gomez, R., and Yang, F. (2018, January 4\u20137). Remote heart rate variability for emotional state monitoring. Proceedings of the 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), Las Vegas, NV, USA.","DOI":"10.1109\/BHI.2018.8333392"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mellouk, W., and Handouzi, W. (2023). CNN-LSTM for automatic emotion recognition using contactless photoplythesmographic signals. Biomed. Signal Process. Control, 85.","DOI":"10.1016\/j.bspc.2023.104907"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"120948","DOI":"10.1016\/j.eswa.2023.120948","article-title":"Trusted emotion recognition based on multiple signals captured from video","volume":"233","author":"Zhang","year":"2023","journal-title":"Expert Syst. Appl."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1016\/j.ymeth.2023.08.014","article-title":"Dimensional emotion recognition from camera-based PRV features","volume":"218","author":"Zhou","year":"2023","journal-title":"Methods"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"10588","DOI":"10.1038\/s41598-018-28804-0","article-title":"Transdermal optical imaging revealed different spatiotemporal patterns of facial cardiovascular activities","volume":"8","author":"Liu","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"808451","DOI":"10.3389\/fphys.2021.808451","article-title":"Photoplethysmogram Analysis and Applications: An Integrative Review","volume":"12","author":"Park","year":"2021","journal-title":"Front. Physiol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"E7900","DOI":"10.1073\/pnas.1702247114","article-title":"Self-report captures 27 distinct categories of emotion bridged by continuous gradients","volume":"114","author":"Cowen","year":"2017","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1037\/0033-295X.99.3.550","article-title":"Are there basic emotions?","volume":"99","author":"Ekman","year":"1992","journal-title":"Psychol. Rev."},{"key":"ref_26","unstructured":"Viola, P., and Jones, M. (2001, January 8\u201314). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR 2001, Kauai, HI, USA."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/TSMC.1979.4310076","article-title":"A Threshold Selection Method from Gray-Level Histograms","volume":"9","author":"Otsu","year":"1979","journal-title":"IEEE Trans. Syst. Man Cybern."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., and Pang, R. (November, January 27). Searching for MobileNetV3. Proceedings of the 2019 IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.","DOI":"10.1109\/ICCV.2019.00140"},{"key":"ref_29","unstructured":"Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., and Lee, J. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1479","DOI":"10.1109\/TBME.2016.2609282","article-title":"Algorithmic Principles of Remote-PPG","volume":"64","author":"Wang","year":"2016","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2878","DOI":"10.1109\/TBME.2013.2266196","article-title":"Robust Pulse Rate From Chrominance-Based rPPG","volume":"60","author":"Jeanne","year":"2013","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1109\/RBME.2021.3121476","article-title":"A Review of Wearable Multi-Wavelength Photoplethysmography","volume":"16","author":"Ray","year":"2023","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_33","first-page":"4414","article-title":"An Exploration of Blue PPG Signal Using a Novel Color Sensorbased PPG System","volume":"2020","author":"Shchelkanova","year":"2020","journal-title":"Annu. Int. Conf. IEEE Eng. Med. Biol. Soc."},{"key":"ref_34","unstructured":"Simchon, R., Meiran, N., Shvimmer, S., Yitzhaky, Y., Rosenblatt, J., and Gilad, M. (2024). Beyond Valence and Arousal: Distributed facial patterns linked to specific emotions cannot be reduced to core affect. Affect. Sci., submitted."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"3717717","DOI":"10.1109\/JPHOT.2022.3153506","article-title":"Wearable Wrist Photoplethysmography for Optimal Monitoring of Vital Signs: A Unified Perspective on Pulse Waveforms","volume":"14","author":"Long","year":"2022","journal-title":"IEEE Photonics J."},{"key":"ref_36","unstructured":"Tan, M., and Le, Q. (2019, January 9\u201315). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, PMLR, Long Beach, CA, USA. Available online: https:\/\/proceedings.mlr.press\/v97\/tan19a.html."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10\u201317). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the 2021 IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"133982","DOI":"10.1109\/ACCESS.2020.3010715","article-title":"Deep Neural Networks for Human Activity Recognition With Wearable Sensors: Leave-One-Subject-Out Cross-Validation for Model Selection","volume":"8","author":"Gholamiangonabadi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_39","unstructured":"Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., and Antiga, L. (2019). Advances in Neural Information Processing Systems, Curran Associates, Inc.. Available online: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2019\/hash\/bdbca288fee7f92f2bfa9f7012727740-Abstract.html."},{"key":"ref_40","unstructured":"Grandini, M., Bagli, E., and Visani, G. (2020). Metrics for Multi-Class Classification: An Overview. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kosonogov, V., De Zorzi, L., Honor\u00e9, J., Mart\u00ednez-Vel\u00e1zquez, E.S., Nandrino, J.-L., Martinez-Selva, J.M., and Sequeira, H. (2017). Facial thermal variations: A new marker of emotional arousal. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0183592"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1037\/1528-3542.1.3.276","article-title":"Emotion and motivation I: Defensive and appetitive reactions in picture processing","volume":"1","author":"Bradley","year":"2001","journal-title":"Emotion"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1016\/j.biopsycho.2009.10.007","article-title":"Emotion and the motivational brain","volume":"84","author":"Lang","year":"2010","journal-title":"Biol. Psychol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1037\/emo0001112","article-title":"Angry and happy expressions affect forward gait initiation only when task relevant","volume":"23","author":"Mirabella","year":"2023","journal-title":"Emotion"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"20183","DOI":"10.1038\/s41598-023-47385-1","article-title":"Unveiling the influence of task-relevance of emotional faces on behavioral reactions in a multi-face context using a novel Flanker-Go\/No-go task","volume":"13","author":"Montalti","year":"2023","journal-title":"Sci. Rep."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Dupr\u00e9, D., Krumhuber, E.G., K\u00fcster, D., and McKeown, G.J. (2020). A performance comparison of eight commercially available automatic classifiers for facial affect recognition. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0231968"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/8\/2620\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:31:05Z","timestamp":1760106665000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/8\/2620"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,19]]},"references-count":46,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["s24082620"],"URL":"https:\/\/doi.org\/10.3390\/s24082620","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,19]]}}}