{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T04:23:50Z","timestamp":1792038230355,"version":"4.4.0"},"reference-count":25,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T00:00:00Z","timestamp":1785974400000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/publishingsupport.iopscience.iop.org\/iop-standard\/v1"},{"start":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T00:00:00Z","timestamp":1785974400000},"content-version":"tdm","delay-in-days":5,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Biomed. Phys. Eng. Express"],"published-print":{"date-parts":[[2026,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The dynamic assessment of mental health has emerged as a hotspot for study and application due to the rise in social pressure. However, onventional methods rely mostly on static scales or single-modal data, failing to fully capture multifaceted emotional and behavioral features. This study suggests a deep learning model based on multi-modal data fusion to address this problem. By combining information from multiple sources, including text and visuals, the model effectively identifies and dynamically monitors mental states. In the study, a multi-modal dataset containing text and image information is constructed. Mental health levels and emotional tendency labels are generated by combining self-assessment questionnaires and expert annotations. The model uses Bi-LSTM to process the text modality, convolutional neural network to process the image modality, and realizes deep feature interaction and integration through an improved multi-modal fusion strategy. Experimental results show that in the emotion recognition task, the precision, recall, and F-score of the proposed model reach 91.3%, 88.9%, and 90.1% respectively. In the classification of \u2018possible depression\u2019 and \u2018existing depression\u2019, the accuracy rates are 75.89% and 85.32%. All these performance indicators are superior to those of single-modal models and traditional average fusion methods, indicating that multi-modal fusion can significantly improve discriminative ability, robustness, and the accuracy of dynamic assessment. This study\u2019s primary contributions are the creation of a jointly annotated dataset, the proposal of a multi-modal deep learning framework appropriate for dynamic mental health assessment, and the confirmation of the fusion strategy\u2019s contribution to enhanced assessment performance. The study\u2019s findings broaden the range of applications for multi-modal deep learning in the field of mental health and offer a feasible technological solution for public health management, psychological services, and smart healthcare. They also offer theoretical and practical support for early screening, risk early warning, and personalized intervention of mental health.<\/jats:p>","DOI":"10.1088\/2057-1976\/ae89bd","type":"journal-article","created":{"date-parts":[[2026,7,14]],"date-time":"2026-07-14T02:47:41Z","timestamp":1783997261000},"page":"045058","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Design and optimization of deep learning model based on multimodal data fusion for dynamic mental health assessment"],"prefix":"10.1088","volume":"12","author":[{"given":"Xin","family":"Mao","sequence":"first","affiliation":[{"name":"Zhuhai College of Science and Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Xie","sequence":"additional","affiliation":[{"name":"Zhuhai College of Science and Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0804-474X","authenticated-orcid":true,"given":"Lu","family":"Liu","sequence":"additional","affiliation":[{"name":"ChengDe College of Applied Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2026,8,6]]},"reference":[{"key":"bpexae89bdbib1","doi-asserted-by":"publisher","first-page":"13","DOI":"10.64252\/sfqnv852","type":"journal-article","article-title":"Optimizing mental health diagnostics with hybrid deep learning and multimodal data fusion","volume":"11","author":"Nayak","year":"2025","journal-title":"Int. J. Environ. Sci."},{"key":"bpexae89bdbib2","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1504\/IJICT.2025.146669","type":"journal-article","article-title":"Dynamic evaluation of college students\u2019 psychological state based on multimodal physiological signal fusion and deep generation model","volume":"26","author":"Li","year":"2025","journal-title":"Int. J. Inf. Commun. Technol."},{"key":"bpexae89bdbib3","doi-asserted-by":"publisher","first-page":"1205","DOI":"10.1109\/TVCG.2023.3326943","type":"journal-article","article-title":"Healthprism: a visual analytics system for exploring children\u2019s physical and mental health profiles with multimodal data","volume":"30","author":"Jiang","year":"2023","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"bpexae89bdbib4","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1007\/s10462-024-10712-7","type":"journal-article","article-title":"Artificial intelligence and multimodal data fusion for smart healthcare: topic modeling and bibliometrics","volume":"57","author":"Chen","year":"2024","journal-title":"Artif. Intell. Rev."},{"key":"bpexae89bdbib5","doi-asserted-by":"publisher","first-page":"24435","DOI":"10.1007\/s00521-023-09036-4","type":"journal-article","article-title":"Multi-modal fusion learning through biosignal, audio, and visual content for detection of mental stress","volume":"35","author":"Dogan","year":"2023","journal-title":"Neural Comput. Appl."},{"key":"bpexae89bdbib6","doi-asserted-by":"publisher","DOI":"10.1016\/j.apacoust.2022.108849","type":"journal-article","article-title":"An auto-encoder based LSTM model for prediction of ambient noise levels","volume":"195","author":"Tiwari","year":"2022","journal-title":"Appl. Acoust."},{"key":"bpexae89bdbib7","doi-asserted-by":"publisher","first-page":"49666","DOI":"10.1007\/s11356-023-25509-4","type":"journal-article","article-title":"A hybrid deep leaning model for prediction and parametric sensitivity analysis of noise annoyance","volume":"30","author":"Tiwari","year":"2023","journal-title":"Environ. Sci. Pollut. Res."},{"key":"bpexae89bdbib8","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-20377-z","type":"journal-article","article-title":"An automated framework for traffic noise level analysis using explainable artificial intelligence techniques","volume":"15","author":"Patel","year":"2025","journal-title":"Sci. Rep."},{"key":"bpexae89bdbib9","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2022.947168","type":"journal-article","article-title":"Real-time mental stress detection using multimodality expressions with a deep learning framework","volume":"16","author":"Zhang","year":"2022","journal-title":"Front. Neurosci."},{"key":"bpexae89bdbib10","doi-asserted-by":"publisher","first-page":"22935","DOI":"10.1007\/s00521-022-06913-2","type":"journal-article","article-title":"Real-time emotional health detection using fine-tuned transfer networks with multimodal fusion","volume":"35","author":"Sharma","year":"2023","journal-title":"Neural Comput. Appl."},{"key":"bpexae89bdbib11","doi-asserted-by":"publisher","first-page":"1082","DOI":"10.3390\/diagnostics13061082","type":"journal-article","article-title":"Common mental disorders in smart city settings and use of multimodal medical sensor fusion to detect them","volume":"13","author":"Alwakeel","year":"2023","journal-title":"Diagnostics"},{"key":"bpexae89bdbib12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3556980","type":"journal-article","article-title":"A critical review of multimodal-multisensor analytics for anxiety assessment","volume":"3","author":"Senaratne","year":"2022","journal-title":"ACM Trans. Comput. Healthcare"},{"key":"bpexae89bdbib13","doi-asserted-by":"publisher","first-page":"1821","DOI":"10.1080\/02648725.2023.2196846","type":"journal-article","article-title":"College students\u2019 mental health evaluation model based on tensor fusion network with multimodal data during the COVID-19 pandemic","volume":"40","author":"Zhu","year":"2024","journal-title":"Biotechnol. Genet. Eng. Rev."},{"key":"bpexae89bdbib14","doi-asserted-by":"publisher","first-page":"70370","DOI":"10.1109\/ACCESS.2022.3187502","type":"journal-article","article-title":"Multimodal educational data fusion for students\u2019 mental health detection","volume":"10","author":"Guo","year":"2022","journal-title":"IEEE Access"},{"key":"bpexae89bdbib15","doi-asserted-by":"publisher","first-page":"569","DOI":"10.1093\/bib\/bbab569","type":"journal-article","article-title":"Multimodal deep learning for biomedical data fusion: a review","volume":"23","author":"Stahlschmidt","year":"2022","journal-title":"Brief. Bioinform."},{"key":"bpexae89bdbib16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3545572","type":"journal-article","article-title":"A review on methods and applications in multimodal deep learning","volume":"19","author":"Jabeen","year":"2023","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"bpexae89bdbib17","doi-asserted-by":"publisher","DOI":"10.1016\/j.compag.2024.109245","type":"journal-article","article-title":"RustQNet: Multimodal deep learning for quantitative inversion of wheat stripe rust disease index","volume":"225","author":"Deng","year":"2024","journal-title":"Comput. Electron. Agric."},{"key":"bpexae89bdbib18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104482","type":"journal-article","article-title":"A scoping review on multimodal deep learning in biomedical images and texts","volume":"146","author":"Sun","year":"2023","journal-title":"J. Biomed. Inform."},{"key":"bpexae89bdbib19","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1038\/s42256-023-00633-5","type":"journal-article","article-title":"Multimodal data fusion for cancer biomarker discovery with deep learning","volume":"5","author":"Steyaert","year":"2023","journal-title":"Nat. Mach. Intell."},{"key":"bpexae89bdbib20","first-page":"74","type":"book","article-title":"Deep learning algorithms and architectures for multimodal data analysis","author":"Sathio","year":"2024"},{"key":"bpexae89bdbib21","doi-asserted-by":"publisher","first-page":"7627","DOI":"10.1109\/TNNLS.2022.3219551","type":"journal-article","article-title":"Attention-like multimodality fusion with data augmentation for diagnosis of mental disorders using MRI","volume":"35","author":"Liu","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"bpexae89bdbib22","doi-asserted-by":"publisher","first-page":"79","DOI":"10.7840\/kics.2022.47.1.79","type":"journal-article","article-title":"Deep learning based data fusion methods for multimodal emotion recognition","volume":"47","author":"Njoku","year":"2022","journal-title":"J. Korean Inst. Commun. Inf. Sci."},{"key":"bpexae89bdbib23","doi-asserted-by":"publisher","first-page":"2263","DOI":"10.1109\/TMI.2022.3161828","type":"journal-article","article-title":"Multi-modal imaging genetics data fusion via a hypergraph-based manifold regularization: Application to schizophrenia study","volume":"41","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Med. Imaging"},{"key":"bpexae89bdbib24","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1109\/TNSRE.2023.3237375","type":"journal-article","article-title":"Schizo-Net: A novel Schizophrenia Diagnosis framework using late fusion multimodal deep learning on Electroencephalogram-based Brain connectivity indices","volume":"31","author":"Grover","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"bpexae89bdbib25","doi-asserted-by":"publisher","first-page":"630","DOI":"10.1007\/s11682-024-00857-y","type":"journal-article","article-title":"A deep learning approach for mental health quality prediction using functional network connectivity and assessment data","volume":"18","author":"Ajith","year":"2024","journal-title":"Brain Imaging Behav."}],"container-title":["Biomedical Physics &amp; Engineering Express"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T06:36:20Z","timestamp":1785998180000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2057-1976\/ae89bd"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,1]]},"references-count":25,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,8,6]]},"published-print":{"date-parts":[[2026,8,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2057-1976\/ae89bd","relation":{},"ISSN":["2057-1976"],"issn-type":[{"value":"2057-1976","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,1]]},"assertion":[{"value":"Design and optimization of deep learning model based on multimodal data fusion for dynamic mental health assessment","name":"article_title","label":"Article Title"},{"value":"Biomedical Physics & Engineering Express","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2026 IOP Publishing Ltd. All rights, including for text and data mining, AI training, and similar technologies, are reserved.","name":"copyright_information","label":"Copyright Information"},{"value":"2025-11-18","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2026-07-13","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2026-08-06","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}