{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T15:56:29Z","timestamp":1778082989245,"version":"3.51.4"},"reference-count":30,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2023,11,25]],"date-time":"2023-11-25T00:00:00Z","timestamp":1700870400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Machine learning is used for a fast pre-diagnosis approach to prevent the effects of Major Depressive Disorder (MDD). The objective of this research is to detect depression using a set of important facial features extracted from interview video, e.g., radians, gaze at angles, action unit intensity, etc. The model is based on LSTM with an attention mechanism. It aims to combine those features using the intermediate fusion approach. The label smoothing was presented to further improve the model\u2019s performance. Unlike other black-box models, the integrated gradient was presented as the model explanation to show important features of each patient. The experiment was conducted on 474 video samples collected at Chulalongkorn University. The data set was divided into 134 depressed and 340 non-depressed categories. The results showed that our model is the winner, with a 88.89% F1-score, 87.03% recall, 91.67% accuracy, and 91.40% precision. Moreover, the model can capture important features of depression, including head turning, no specific gaze, slow eye movement, no smiles, frowning, grumbling, and scowling, which express a lack of concentration, social disinterest, and negative feelings that are consistent with the assumptions in the depressive theories.<\/jats:p>","DOI":"10.3390\/s23239402","type":"journal-article","created":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T03:48:17Z","timestamp":1701056897000},"page":"9402","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Explainable Depression Detection Based on Facial Expression Using LSTM on Attentional Intermediate Feature Fusion with Label Smoothing"],"prefix":"10.3390","volume":"23","author":[{"given":"Yanisa","family":"Mahayossanunt","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Natawut","family":"Nupairoj","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"},{"name":"Center of Excellence in Digital and AI Innovation for Mental Health (AIMET), Chulalongkorn Unversity, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Solaphat","family":"Hemrungrojn","sequence":"additional","affiliation":[{"name":"Center of Excellence in Digital and AI Innovation for Mental Health (AIMET), Chulalongkorn Unversity, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"},{"name":"Department of Psychiatry, Faculty of Medicine, Chulalongkorn University, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"},{"name":"Cognitive Fitness and Biopsychiatry Technology Research Unit, Faculty of Medicine, Chulalongkorn University, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peerapon","family":"Vateekul","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Faculty of Engineering, Chulalongkorn University, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"},{"name":"Center of Excellence in Digital and AI Innovation for Mental Health (AIMET), Chulalongkorn Unversity, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"},{"name":"Cognitive Fitness and Biopsychiatry Technology Research Unit, Faculty of Medicine, Chulalongkorn University, Phayathai Rd, Pathumwan, Bangkok 10330, Thailand"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,11,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1700","DOI":"10.1016\/S0140-6736(21)02143-7","article-title":"Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic","volume":"398","author":"Santomauro","year":"2021","journal-title":"Lancet"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"3222","DOI":"10.1017\/S0033291720005358","article-title":"Increases in depression and anxiety symptoms in adolescents and young adults during the COVID-19 pandemic","volume":"52","author":"Hawes","year":"2022","journal-title":"Psychol. Med."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tabur, A., Elkefi, S., Emhan, A., Mengenci, C., Bez, Y., and Asan, O. (2022). Anxiety, burnout and depression, psychological well-being as predictor of healthcare professionals\u2019 turnover during the COVID-19 pandemic: Study in a pandemic hospital. Healthcare, 10.","DOI":"10.3390\/healthcare10030525"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1016\/j.jad.2022.05.052","article-title":"Closed doors: Predictors of stress, anxiety, depression, and PTSD during the onset of COVID-19 pandemic in Brazil","volume":"310","author":"Calegaro","year":"2022","journal-title":"J. Affect. Disord."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.jad.2022.05.084","article-title":"Prevalence of depressive disorders and associated demographic characteristics in Shandong: An epidemiological investigation","volume":"311","author":"Cheng","year":"2022","journal-title":"J. Affect. Disord."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.comppsych.2012.06.006","article-title":"The diagnosis of depression: Current and emerging methods","volume":"54","author":"Smith","year":"2013","journal-title":"Compr. Psychiatry"},{"key":"ref_7","unstructured":"Ekman, P., and Friesen, W.V. (1978). Environmental Psychology & Nonverbal Behavior, American Psychological Association."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Baltru\u0161aitis, T., Robinson, P., and Morency, L.P. (2016, January 7\u201310). Openface: An open source facial behavior analysis toolkit. Proceedings of the 2016 IEEE winter conference on applications of computer vision (WACV), Lake Placid, NY, USA.","DOI":"10.1109\/WACV.2016.7477553"},{"key":"ref_9","unstructured":"Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6\u201311). Axiomatic attribution for deep networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"109158","DOI":"10.1016\/j.jneumeth.2021.109158","article-title":"Representation of the brain network by electroencephalograms during facial expressions","volume":"357","author":"Watanabe","year":"2021","journal-title":"J. Neurosci. Methods"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1176\/ajp.2007.164.4.599","article-title":"Neural responses to happy facial expressions in major depression following antidepressant treatment","volume":"164","author":"Fu","year":"2007","journal-title":"Am. J. Psychiatry"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"100295","DOI":"10.1016\/j.jadr.2021.100295","article-title":"The cerebellum as a moderator of negative bias of facial expression processing in depressive patients","volume":"7","author":"Nakamura","year":"2022","journal-title":"J. Affect. Disord. Rep."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1016\/j.tics.2017.01.001","article-title":"Emotion perception from face, voice, and touch: Comparisons and convergence","volume":"21","author":"Schirmer","year":"2017","journal-title":"Trends Cogn. Sci."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1016\/j.psychres.2013.02.027","article-title":"Facial cues to depressive symptoms and their associated personality attributions","volume":"208","author":"Scott","year":"2013","journal-title":"Psychiatry Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1784","DOI":"10.1017\/S0033291720003608","article-title":"Prediction of depression symptoms in individual subjects with face and eye movement tracking","volume":"52","author":"Stolicyn","year":"2022","journal-title":"Psychol. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1016\/j.eng.2019.06.008","article-title":"A brief review of artificial intelligence applications and algorithms for psychiatric disorders","volume":"6","author":"Liu","year":"2020","journal-title":"Engineering"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ringeval, F., Schuller, B., Valstar, M., Cummins, N., Cowie, R., Tavabi, L., Schmitt, M., Alisamir, S., Amiriparian, S., and Messner, E.M. (2019, January 21\u201325). AVEC 2019 workshop and challenge: State-of-mind, detecting depression with AI, and cross-cultural affect recognition. Proceedings of the 9th International on Audio\/Visual Emotion Challenge and Workshop, Nice, France.","DOI":"10.1145\/3347320.3357688"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ray, A., Kumar, S., Reddy, R., Mukherjee, P., and Garg, R. (2019, January 21\u201325). Multi-level attention network using text, audio and video for depression prediction. Proceedings of the 9th International on Audio\/Visual Emotion Challenge and Workshop, Nice, France.","DOI":"10.1145\/3347320.3357697"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sun, H., Liu, J., Chai, S., Qiu, Z., Lin, L., Huang, X., and Chen, Y. (2021). Multi-modal adaptive fusion transformer network for the estimation of depression level. Sensors, 21.","DOI":"10.3390\/s21144764"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Akbar, H., Dewi, S., Rozali, Y.A., Lunanta, L.P., Anwar, N., and Anwar, D. (2021, January 28). Exploiting facial action unit in video for recognizing depression using metaheuristic and neural networks. Proceedings of the 2021 1st International Conference on Computer Science and Artificial Intelligence (ICCSAI), Jakarta, Indonesia.","DOI":"10.1109\/ICCSAI53272.2021.9609747"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"17703","DOI":"10.1007\/s11042-022-12420-2","article-title":"Selection of relevant visual feature sets for enhanced depression detection using incremental linear discriminant analysis","volume":"81","author":"Rathi","year":"2022","journal-title":"Multimed. Tools Appl."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1017\/S0033291720002718","article-title":"Deep learning-based classification of posttraumatic stress disorder and depression following trauma utilizing visual and auditory markers of arousal and mood","volume":"52","author":"Schultebraucks","year":"2022","journal-title":"Psychol. Med."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Serengil, S.I., and Ozpinar, A. (2021, January 27\u201328). HyperExtended LightFace: A Facial Attribute Analysis Framework. Proceedings of the 2021 International Conference on Engineering and Emerging Technologies (ICEET), Istanbul, Turkey.","DOI":"10.1109\/ICEET53442.2021.9659697"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1136\/jnnp.23.1.56","article-title":"A rating scale for depression","volume":"23","author":"Hamilton","year":"1960","journal-title":"J. Neurol. Neurosurg. Psychiatry"},{"key":"ref_25","unstructured":"M\u00fcller, R., Kornblith, S., and Hinton, G.E. (2019). When does label smoothing help?. Adv. Neural Inf. Process. Syst., 32."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1097\/00005053-198406000-00004","article-title":"The ethological approach to the assessment of depressive disorders","volume":"172","author":"Fossi","year":"1984","journal-title":"J. Nerv. Ment. Dis."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1097\/00005053-199803000-00001","article-title":"Major depression: Behavioral markers of depression and recovery","volume":"186","author":"Schelde","year":"1998","journal-title":"J. Nerv. Ment. Dis."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1114","DOI":"10.1016\/j.jad.2013.05.002","article-title":"Talking bodies: Nonverbal behavior in the assessment of depression severity","volume":"150","author":"Fiquer","year":"2013","journal-title":"J. Affect. Disord."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Gahalawat, M., Fernandez Rojas, R., Guha, T., Subramanian, R., and Goecke, R. (2023, January 9\u201313). Explainable Depression Detection via Head Motion Patterns. Proceedings of the International Conference on Multimodal Interaction, Paris, France.","DOI":"10.1145\/3577190.3614130"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"101306","DOI":"10.1016\/j.rineng.2023.101306","article-title":"A comprehensive review of wearable assistive robotic devices used for head and neck rehabilitation","volume":"19","author":"Doss","year":"2023","journal-title":"Results Eng."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/23\/9402\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:30:12Z","timestamp":1760131812000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/23\/9402"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,25]]},"references-count":30,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["s23239402"],"URL":"https:\/\/doi.org\/10.3390\/s23239402","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,25]]}}}