{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,8]],"date-time":"2026-11-08T14:14:45Z","timestamp":1794147285802,"version":"4.8.0"},"reference-count":52,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T00:00:00Z","timestamp":1728950400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Brain Sciences"],"abstract":"<jats:p>Background\/Objectives: This study investigates the classification of Major Depressive Disorder (MDD) using electroencephalography (EEG) Short-Time Fourier-Transform (STFT) spectrograms and audio Mel-spectrogram data of 52 subjects. The objective is to develop a multimodal classification model that integrates audio and EEG data to accurately identify depressive tendencies. Methods: We utilized the Multimodal open dataset for Mental Disorder Analysis (MODMA) and trained a pre-trained Densenet121 model using transfer learning. Features from both the EEG and audio modalities were extracted and concatenated before being passed through the final classification layer. Additionally, an ablation study was conducted on both datasets separately. Results: The proposed multimodal classification model demonstrated superior performance compared to existing methods, achieving an Accuracy of 97.53%, Precision of 98.20%, F1 Score of 97.76%, and Recall of 97.32%. A confusion matrix was also used to evaluate the model\u2019s effectiveness. Conclusions: The paper presents a robust multimodal classification approach that outperforms state-of-the-art methods with potential application in clinical diagnostics for depression assessment.<\/jats:p>","DOI":"10.3390\/brainsci14101018","type":"journal-article","created":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T08:09:08Z","timestamp":1728979748000},"page":"1018","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Multimodal Fusion of EEG and Audio Spectrogram for Major Depressive Disorder Recognition Using Modified DenseNet121"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-5264-488X","authenticated-orcid":false,"given":"Musyyab","family":"Yousufi","sequence":"first","affiliation":[{"name":"Centre of Real Time Computer Systems, Kaunas University of Technology, 51368 Kaunas, Lithuania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9990-1084","authenticated-orcid":false,"given":"Robertas","family":"Dama\u0161evi\u010dius","sequence":"additional","affiliation":[{"name":"Centre of Real Time Computer Systems, Kaunas University of Technology, 51368 Kaunas, Lithuania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2809-2213","authenticated-orcid":false,"given":"Rytis","family":"Maskeli\u016bnas","sequence":"additional","affiliation":[{"name":"Centre of Real Time Computer Systems, Kaunas University of Technology, 51368 Kaunas, Lithuania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,10,15]]},"reference":[{"key":"ref_1","first-page":"85","article-title":"When does depression become a mental disorder?","volume":"2","author":"Mario","year":"2011","journal-title":"Br. J. Psychiatry"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"73804","DOI":"10.1109\/ACCESS.2022.3190502","article-title":"Automated detection of major depressive disorder with EEG signals: A time series classification using deep learning","volume":"10","author":"Alireza","year":"2022","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"112481","DOI":"10.1016\/j.psychres.2019.112481","article-title":"Suicide attempts in chronically depressed individuals: What are the risk factors?","volume":"287","author":"Ernst","year":"2020","journal-title":"Psychiatry Res."},{"key":"ref_4","first-page":"244","article-title":"The comorbidity of major depression and anxiety disorders: Recognition and management in primary care","volume":"3","author":"Hirschfeld","year":"2001","journal-title":"Prim. Care Companion J. Clin. Psychiatry"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental Disorders, The American Psychiatric Association.","DOI":"10.1176\/appi.books.9780890425596"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Marcus, M., Yasamy, M.T., Ommeren, M.V., Chisholm, D., and Saxena, S. (2012). Depression: A Global Public Health Concern, American Psychological Association.","DOI":"10.1037\/e517532013-004"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1038\/515180a","article-title":"A world of depression","volume":"515","author":"Smith","year":"2014","journal-title":"Nature"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2391","DOI":"10.1001\/jama.286.19.2391","article-title":"Mental health: New understanding, new hope","volume":"286","author":"Brundtland","year":"2001","journal-title":"JAMA"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"28196","DOI":"10.1109\/ACCESS.2019.2901950","article-title":"Multimodal mild depression recognition based on EEG-EM synchronization acquisition network","volume":"7","author":"Zhu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"99847","DOI":"10.1109\/ACCESS.2023.3315243","article-title":"Taking all the factors we need: A multimodal depression classification with uncertainty approximation","volume":"11","author":"Ahmad","year":"2023","journal-title":"IEEE Access"},{"key":"ref_11","unstructured":"World Health Organization (2024, April 25). Depressive Disorder (Depression). Available online: https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/depression."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.psychres.2019.07.005","article-title":"Heterogeneity in psychiatric diagnostic classification","volume":"279","author":"Allsopp","year":"2019","journal-title":"Psychiatry Res."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2587","DOI":"10.1109\/TCBB.2023.3257175","article-title":"High-density electroencephalography and speech signal based deep framework for clinical depression diagnosis","volume":"20","author":"Qayyum","year":"2023","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1001\/jama.2015.18406","article-title":"Recommendations for screening for depression in adults","volume":"4","author":"Thase","year":"2016","journal-title":"JAMA"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1111\/pcn.12360","article-title":"Multidimensional anatomy of \u2018modern type depression\u2019in J apan: A proposal for a different diagnostic approach to depression beyond the DSM-5","volume":"1","author":"Kato","year":"2016","journal-title":"Psychiatry Clin. Neurosci."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1002\/hbm.20275","article-title":"Impaired functional connectivity at EEG alpha and theta frequency bands in major depression","volume":"28","author":"Fingelkurts","year":"2007","journal-title":"Hum. Brain Mapp."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1109\/JBHI.2017.2676878","article-title":"Dynamic multimodal measurement of depression severity using deep autoencoding","volume":"22","author":"Dibeklioglu","year":"2017","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1159\/000197898","article-title":"Electrophysiological markers of rapid cognitive decline in mild cognitive impairment","volume":"24","author":"Giannakopoulos","year":"2009","journal-title":"Dement. Clin. Pract."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2525","DOI":"10.1016\/j.proeng.2012.06.298","article-title":"Analysis of electroencephalography (EEG) signals and its categorization\u2014A study","volume":"38","author":"Kumar","year":"2012","journal-title":"Procedia Eng."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1097\/01.wnp.0000220079.61973.6c","article-title":"Neurophysiologic basis of EEG","volume":"23","author":"Olejniczak","year":"2006","journal-title":"J. Clin. Neurophysiol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1250002","DOI":"10.1142\/S0129065712500025","article-title":"Application of non-linear and wavelet based features for the automated identification of epileptic EEG signals","volume":"22","author":"Acharya","year":"2012","journal-title":"Int. J. Neural Syst."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Cavanagh, J.F., Napolitano, A., Wu, C., and Mueen, A. (2017). The patient repository for EEG data+ computational tools (PRED + CT). Front. Neurosci., 11.","DOI":"10.3389\/fninf.2017.00067"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Othmani, A., and Muzammel, M. (2023). An Ambient Intelligence-Based Approach for Longitudinal Monitoring of Verbal and Vocal Depression Symptoms. International Workshop on PRedictive Intelligence in Medicine, Springer Nature.","DOI":"10.1007\/978-3-031-46005-0_18"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1007\/s42600-020-00100-9","article-title":"Detection of major depressive disorder using vocal acoustic analysis and machine learning an exploratory study","volume":"37","author":"Espinola","year":"2021","journal-title":"Res. Biomed. Eng."},{"key":"ref_25","unstructured":"Gao, H., Liu, Z., Maaten, L.V.D., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3466","DOI":"10.1109\/JBHI.2022.3165640","article-title":"Computer-aided recognition based on decision-level multimodal fusion for depression","volume":"7","author":"Zhang","year":"2022","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1038\/s41597-022-01211-x","article-title":"A multi-modal open dataset for mental-disorder analysis","volume":"9","author":"Cai","year":"2022","journal-title":"Sci. Data"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wu, C.-T., Huang, H.-C., Huang, S., Chen, I.-M., Liao, S.-C., Chen, C.-K., Lin, C., Lee, S.-H., Chen, M.-H., and Tsai, C.-F. (2021). Resting-state EEG signal for major depressive disorder detection: A systematic validation on a large and diverse dataset. Biosensors, 11.","DOI":"10.3390\/bios11120499"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Mumtaz, W., Xia, L., Mohd Yasin, M.A., Azhar Ali, S.S., and Malik, A.S. (2017). A wavelet-based technique to predict treatment out-come for majordepressive disorder. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0171409"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1007\/s00542-018-4075-z","article-title":"Detection of major depressive disorder using linear and non-linear features from EEG signals","volume":"25","author":"Mahato","year":"2019","journal-title":"Microsyst. Technol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1016\/j.cmpb.2012.10.008","article-title":"Classifying depression patients and normal subjects using machine learning techniques and nonlinear features from EEG signal","volume":"1109","author":"Hosseinifard","year":"2013","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"6449","DOI":"10.1007\/s10489-021-02426-y","article-title":"Automated major depressive disorder detection using melamine pattern with EEG signals","volume":"51","author":"Aydemir","year":"2021","journal-title":"Appl. Intell."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1177\/1550059414523764","article-title":"Feature selection and classification of electroencephalographic signals: An artificial neural network and genetic algorithm based approach","volume":"46","author":"TErguzel","year":"2015","journal-title":"Clin. EEG Neurosci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1007\/s10916-019-1486-z","article-title":"Classification of depression patients and normal subjects based on electroencephalogram (EEG) signal using alpha power and theta asymmetry","volume":"44","author":"Mahato","year":"2019","journal-title":"J. Med. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"5238028","DOI":"10.1155\/2018\/5238028","article-title":"A pervasive approach to eeg-based depression detection","volume":"2018","author":"Cai","year":"2018","journal-title":"Complexity"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1016\/j.bspc.2015.10.006","article-title":"Geriatric depression symptoms coexisting with cognitive decline: A comparison of classification methodologies","volume":"25","author":"Spyrou","year":"2016","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.cmpb.2018.04.012","article-title":"Automated EEG-based screening of depression using deep convolutional neural network","volume":"161","author":"Acharya","year":"2018","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Sandheep, P., Vineeth, S., Poulose, M., and Subha, D.P. (2019, January 17\u201320). Performance analysis of deep learning CNN in classification of depression EEG signals. Proceedings of the TENCON 2019\u20132019 IEEE Region 10 Conference (TENCON), Kochi, India.","DOI":"10.1109\/TENCON.2019.8929254"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1341","DOI":"10.1007\/s11517-019-01959-2","article-title":"EEG-based mild depression recognition using convolutional neural network","volume":"57","author":"Li","year":"2019","journal-title":"Med. Biol. Eng. Comput."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1007\/s10916-019-1345-y","article-title":"Automated depression detection using deep representation and sequence learning with EEG signals","volume":"43","author":"Ay","year":"2019","journal-title":"J. Med. Syst."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"667","DOI":"10.1007\/s11071-020-05665-9","article-title":"Multilayer brain network combined with deep convolutional neural network for detecting major depressive disorder","volume":"102","author":"Dang","year":"2020","journal-title":"Nonlinear Dyn."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1007\/s11571-020-09619-0","article-title":"Major depressive disorder diagnosis based on effective connectivity in EEG signals: A convolutional neural network and long short-term memory approach","volume":"15","author":"Saeedi","year":"2020","journal-title":"Cogn. Neurodynamics"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Gupta, R., Malandrakis, N., Xiao, B., Guha, T., Segbroeck, M.V., Black, M., Potamianos, A., and Narayanan, S. (2014, January 7). Multimodal prediction of affective dimensions and depression in human-computer interactions. Proceedings of the 4th International Workshop on Audio\/Visual Emotion Challenge, Orlando, FL, USA.","DOI":"10.1145\/2661806.2661810"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1007\/s00702-017-1699-6","article-title":"Differential diagnosis between patients with probable Alzheimer\u2019s disease, Parkinson\u2019s disease dementia, or dementia with Lewy bodies and frontotemporal dementia, behavioral variant, using quantitative electroencephalographic features","volume":"124","author":"Garn","year":"2017","journal-title":"J. Neural Transm."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"101943","DOI":"10.1016\/j.jocs.2023.101943","article-title":"Multiple classification of EEG signals and epileptic seizure diagnosis with combined deep learning","volume":"67","author":"Varli","year":"2023","journal-title":"J. Comput. Sci."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"47003","DOI":"10.1007\/s11042-023-15653-x","article-title":"Electroencephalogram based brain-computer interface: Applications, challenges, and opportunities","volume":"82","author":"Yadav","year":"2023","journal-title":"Multimed. Tools Appl."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Kim, H., Luo, J., Chu, S., Cannard, C., Hoffmann, S., and Miyakoshi, M. (2023). ICA\u2019s bug: How ghost ICs emerge from effective rank deficiency caused by EEG electrode interpolation and incorrect re-referencing. Front. Signal Process., 3.","DOI":"10.3389\/frsip.2023.1064138"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Husain, S.F., Tang, T.-B., Yu, R., Tam, W.W., Tran, B., Quek, T.T., Hwang, S.-H., Chang, C.W., Ho, C.S., and Ho, R.C. (2020). Cortical haemodynamic response measured by functional near infrared spectroscopy during a verbal fluency task in patients with major depression and borderline personality disorder. EBioMedicine, 51.","DOI":"10.1016\/j.ebiom.2019.11.047"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Kumfor, F., Irish, M., Hodges, J.R., and Piguet, O. (2014). Frontal and temporal lobe contributions to emotional enhancement of memory in behavioral-variant frontotemporal dementia and Alzheimer\u2019s disease. Front. Behav. Neurosci., 8.","DOI":"10.3389\/fnbeh.2014.00225"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Li, Y., Zhao, T., and Kawahara, T. (2019, January 15\u201319). Improved End-to-End Speech Emotion Recognition Using Self Attention Mechanism and Multitask Learning. Proceedings of the Interspeech 2019, Graz, Austria.","DOI":"10.21437\/Interspeech.2019-2594"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Xin, C., and Pan, Z. (2021). A convenient and low-cost model of depression screening and early warning based on voice data using for public mental health. Int. J. Environ. Res. Public Health, 18.","DOI":"10.3390\/ijerph18126441"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Wang, Y., Liu, F., and Yang, L. (2021, January 20\u201322). EEG-based depression recognition using intrinsic time-scale decomposition and temporal convolution network. Proceedings of the 5th International Conference on Biological Information and Biomedical Engineering, Hangzhou, China.","DOI":"10.1145\/3469678.3469683"}],"container-title":["Brain Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3425\/14\/10\/1018\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:13:24Z","timestamp":1760112804000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3425\/14\/10\/1018"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,15]]},"references-count":52,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,10]]}},"alternative-id":["brainsci14101018"],"URL":"https:\/\/doi.org\/10.3390\/brainsci14101018","relation":{},"ISSN":["2076-3425"],"issn-type":[{"value":"2076-3425","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,15]]}}}