{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T05:54:09Z","timestamp":1781589249999,"version":"3.54.5"},"reference-count":45,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T00:00:00Z","timestamp":1781568000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychiatry"],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Depression assessment remains largely dependent on symptom reports and clinician judgment, while objective tools for depressive-spectrum stratification and severity estimation remain limited. Existing digital and multimodal depression-detection studies often focus on binary case-control classification, handle missing modalities incompletely, provide limited calibration assessment, and rarely combine depressive-spectrum classification with continuous symptom-severity estimation. We therefore developed a quality-aware multimodal framework integrating eye tracking, facial behavior, and transcript-derived language for classification across normal control (NC), subthreshold depression (SD), and major depressive disorder (MDD), together with prediction of 17-item Hamilton Depression Rating Scale (HAMD-17) severity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A total of 186 participants completed a controlled task battery including interview, emotional reading, free viewing with verbal description, fixation, gaze orienting, smooth pursuit, prosaccade, and antisaccade tasks. Eye-tracking, facial-video, and transcript-derived language data were converted into modality-specific features. Baseline-3 combined modality-specific encoders, quality-aware gated fusion, and joint classification-regression learning under a nested repeated-resampling framework with explicit missing-modality handling. Baseline-3+ further incorporated Transformer-based cross-modal interaction and uncertainty-based dynamic task weighting. Performance was evaluated on held-out outer-loop test sets after temperature scaling. Interpretability analyses included gate profiling, selective prediction, SHAP, Integrated Gradients, and counterfactual analysis.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Baseline-3+ showed the most favorable classification and calibration profile, with accuracy, balanced accuracy, and F1-macro approaching 0.90 across both classification routes and lower expected calibration error than Baseline-3. For severity estimation, the improvement was route-dependent and mainly reduced the regression disadvantage observed under the hierarchical route. Misclassification was concentrated near the SD boundary. Interpretability analyses showed stable quality-aware modality reweighting, with facial features providing the dominant signal, complemented by eye tracking and smaller but meaningful language contributions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>This framework addresses key limitations of prior binary and incompletely calibrated depression-detection models by jointly supporting depressive-spectrum classification, severity estimation, missing-modality handling, calibrated prediction, and individual-level interpretation. Its most plausible role is to augment clinical assessment, particularly for boundary states such as SD.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fpsyt.2026.1842005","type":"journal-article","created":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T05:38:57Z","timestamp":1781588337000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Multimodal behavioral phenotyping for depressive-spectrum classification and severity estimation using eye tracking, facial behavior, and transcript-derived language"],"prefix":"10.3389","volume":"17","author":[{"given":"Xiang-Ting","family":"Chen","sequence":"first","affiliation":[{"name":"Department of General Medicine, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Nanjing Medical University","place":["Suzhou, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Min","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of General Medicine, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Nanjing Medical University","place":["Suzhou, Jiangsu, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,6,16]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1038\/s41392-024-01738-y","article-title":"Major depressive disorder: hypothesis, mechanism, prevention and treatment","volume":"9","author":"Cui","year":"2024","journal-title":"Signal Transduct Target Ther"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2024.03.163","article-title":"Digital biomarkers in depression: a systematic review and call for standardization and harmonization of feature engineering","volume":"356","author":"Zierer","year":"2024","journal-title":"J Affect Disord"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.2196\/59826","article-title":"Beyond step count: are we ready to use digital phenotyping to make actionable individual predictions in psychiatry","volume":"26","author":"Ortiz","year":"2024","journal-title":"J Med Internet Res"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.2196\/40689","article-title":"Digital phenotyping for stress, anxiety, and mild depression: systematic literature review","volume":"12","author":"Choi","year":"2024","journal-title":"JMIR Mhealth Uhealth"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2024.105541","article-title":"From smartphone data to clinically relevant predictions: a systematic review of digital phenotyping methods in depression","volume":"158","author":"Leaning","year":"2024","journal-title":"Neurosci Biobehav Rev"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1001\/jamapsychiatry.2023.5083","article-title":"A systematic evaluation of machine learning-based biomarkers for major depressive disorder","volume":"81","author":"Winter","year":"2024","journal-title":"JAMA Psychiatry"},{"key":"B7","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/s44184-023-00040-z","article-title":"A systematic review on automated clinical depression diagnosis","volume":"2","author":"Mao","year":"2023","journal-title":"NPJ Ment Health Res"},{"key":"B8","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1038\/s41746-023-00828-5","article-title":"Systematic review and meta-analysis of performance of wearable artificial intelligence in detecting and predicting depression","volume":"6","author":"Abd-Alrazaq","year":"2023","journal-title":"NPJ Digit Med"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2023.999047","article-title":"Predictive potential of somatic symptoms for the identification of subthreshold depression and major depressive disorder in primary care settings","volume":"14","author":"Li","year":"2023","journal-title":"Front Psychiatry"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.2147\/ndt.S425509","article-title":"Subthreshold depression: a systematic review and network meta-analysis of non-pharmacological interventions","volume":"19","author":"Hao","year":"2023","journal-title":"Neuropsychiatr Dis Treat"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1017\/s0033291722000241","article-title":"The prevalence and risk of developing major depression among individuals with subthreshold depression in the general population","volume":"53","author":"Zhang","year":"2023","journal-title":"Psychol Med"},{"key":"B12","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1080\/13651501.2022.2087530","article-title":"Subthreshold depression - concept, operationalisation and epidemiological data. A scoping review","volume":"27","author":"Volz","year":"2023","journal-title":"Int J Psychiatry Clin Pract"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1017\/s0033291720003608","article-title":"Prediction of depression symptoms in individual subjects with face and eye movement tracking","volume":"52","author":"Stolicyn","year":"2022","journal-title":"Psychol Med"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2024.1459831","article-title":"Depressive self-focus bias following failure: an eye-tracking study among individuals with clinical depression","volume":"15","author":"Mon\u00e9ger","year":"2024","journal-title":"Front Psychiatry"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1016\/j.pscychresns.2024.111914","article-title":"The association between eye movement characteristics and cognitive function in adolescents with major depressive disorder","volume":"345","author":"Lu","year":"2024","journal-title":"Psychiatry Res Neuroimaging"},{"key":"B16","doi-asserted-by":"publisher","first-page":"12000","DOI":"10.1038\/s41598-024-62251-4","article-title":"Eye-tracking evidence of a relationship between attentional bias for emotional faces and depression severity in patients with treatment-resistant depression","volume":"14","author":"Imbert","year":"2024","journal-title":"Sci Rep"},{"key":"B17","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1186\/s12888-024-05633-2","article-title":"Depressive symptoms and visual attention to others' eyes in healthy individuals","volume":"24","author":"Suslow","year":"2024","journal-title":"BMC Psychiatry"},{"key":"B18","doi-asserted-by":"publisher","first-page":"899","DOI":"10.1016\/j.jad.2023.08.087","article-title":"Emotion regulation in adolescents with major depression - evidence from a combined EEG and eye-tracking study","volume":"340","author":"Feldmann","year":"2023","journal-title":"J Affect Disord"},{"key":"B19","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1186\/s12888-021-03093-6","article-title":"Efficient visual search for facial emotions in patients with major depression","volume":"21","author":"Bodenschatz","year":"2021","journal-title":"BMC Psychiatry"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1007\/s10488-023-01310-w","article-title":"Facing change: using automated facial expression analysis to examine emotional flexibility in the treatment of depression","volume":"51","author":"Slonim","year":"2024","journal-title":"Adm Policy Ment Health"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1266","DOI":"10.1038\/s41598-024-51813-1","article-title":"High depressive symptomatology reduces emotional reactions to pictures of social interaction","volume":"14","author":"Lacerda","year":"2024","journal-title":"Sci Rep"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.3390\/tomography9020043","article-title":"Recognition of facial emotion expressions in patients with depressive disorders: a functional MRI study","volume":"9","author":"Ternovoy","year":"2023","journal-title":"Tomography"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2021.703888","article-title":"Confusion effects of facial expression recognition in patients with major depressive disorder and healthy controls","volume":"12","author":"Mo","year":"2021","journal-title":"Front Psychol"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1007\/s40120-021-00231-w","article-title":"Recognition of facial emotion expressions in patients with depressive disorders: a prospective, observational study","volume":"10","author":"Akhapkin","year":"2021","journal-title":"Neurol Ther"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2021.06.053","article-title":"Facial emotion recognition in major depressive disorder: a meta-analytic review","volume":"293","author":"Krause","year":"2021","journal-title":"J Affect Disord"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1109\/jbhi.2025.3570900","article-title":"Automated depression detection from text and audio: a systematic review","volume":"29","author":"Li","year":"2025","journal-title":"IEEE J BioMed Health Inform"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.2196\/67802","article-title":"Performance of automatic speech analysis in detecting depression: systematic review and meta-analysis","volume":"12","author":"Maran","year":"2025","journal-title":"JMIR Ment Health"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocae189","article-title":"Diagnostic accuracy of deep learning using speech samples in depression: a systematic review and meta-analysis","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"B29","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1038\/s41746-024-01100-0","article-title":"Robust language-based mental health assessments in time and space through social media","volume":"7","author":"Mangalik","year":"2024","journal-title":"NPJ Digit Med"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2024.1355734","article-title":"Linguistic markers for major depressive disorder: a cross-sectional study using an automated procedure","volume":"15","author":"Trifu","year":"2024","journal-title":"Front Psychol"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.2196\/55067","article-title":"Screening for depression using natural language processing: literature review","volume":"13","author":"Teferra","year":"2024","journal-title":"Interact J Med Res"},{"key":"B32","doi-asserted-by":"publisher","first-page":"794","DOI":"10.1186\/s12888-024-06253-6","article-title":"The voice of depression: speech features as biomarkers for major depressive disorder","volume":"24","author":"Menne","year":"2024","journal-title":"BMC Psychiatry"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2025.1694762","article-title":"Depression diagnosis from patient interviews using multimodal machine learning","volume":"16","author":"Weber","year":"2025","journal-title":"Front Psychiatry"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2025.1508772","article-title":"Diagnosis of depression based on facial multimodal data","volume":"16","author":"Jin","year":"2025","journal-title":"Front Psychiatry"},{"key":"B35","doi-asserted-by":"publisher","first-page":"21907","DOI":"10.1038\/s41598-025-03524-4","article-title":"Depression detection methods based on multimodal fusion of voice and text","volume":"15","author":"Xu","year":"2025","journal-title":"Sci Rep"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.3390\/bs15111476","article-title":"Systematic review and meta-analysis of explainable machine learning models for clinical depression detection","volume":"15","author":"Trelles","year":"2025","journal-title":"Behav Sci (Basel)"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.2196\/72038","article-title":"Explainable AI for depression detection and severity classification from activity data: development and evaluation study of an interpretable framework","volume":"12","author":"Ahmed","year":"2025","journal-title":"JMIR Ment Health"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1111\/jocn.17694","article-title":"Artificial intelligence in diagnosing depression through behavioural cues: a diagnostic accuracy systematic review and meta-analysis","author":"Goh","year":"2025","journal-title":"J Clin Nurs"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2024.115817","article-title":"Explainable multimodal prediction of treatment-resistance in patients with depression leveraging brain morphometry and natural language processing","volume":"334","author":"Lee","year":"2024","journal-title":"Psychiatry Res"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1007\/s43657-023-00152-8","article-title":"A multimodal approach for detection and assessment of depression using text, audio and video","volume":"4","author":"Zhang","year":"2024","journal-title":"Phenomics"},{"key":"B41","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1038\/s44184-024-00112-8","article-title":"Harnessing multimodal approaches for depression detection using large language models and facial expressions","volume":"3","author":"Sadeghi","year":"2024","journal-title":"NPJ Ment Health Res"},{"key":"B42","doi-asserted-by":"publisher","first-page":"523","DOI":"10.1038\/s41746-025-01933-3","article-title":"AI-assisted multi-modal information for the screening of depression: a systematic review and meta-analysis","volume":"8","author":"Wang","year":"2025","journal-title":"NPJ Digit Med"},{"key":"B43","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1038\/s41746-024-01208-3","article-title":"Evaluating multimodal AI in medical diagnostics","volume":"7","author":"Kaczmarczyk","year":"2024","journal-title":"NPJ Digit Med"},{"key":"B44","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1038\/s41746-022-00712-8","article-title":"Multimodal machine learning in precision health: a scoping review","volume":"5","author":"Kline","year":"2022","journal-title":"NPJ Digit Med"},{"key":"B45","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.inffus.2021.10.012","article-title":"Deep learning for depression recognition with audiovisual cues: a review","volume":"80","author":"He","year":"2022","journal-title":"Inf Fusion"}],"container-title":["Frontiers in Psychiatry"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyt.2026.1842005\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T05:38:58Z","timestamp":1781588338000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyt.2026.1842005\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,16]]},"references-count":45,"alternative-id":["10.3389\/fpsyt.2026.1842005"],"URL":"https:\/\/doi.org\/10.3389\/fpsyt.2026.1842005","relation":{},"ISSN":["1664-0640"],"issn-type":[{"value":"1664-0640","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,16]]},"article-number":"1842005"}}