{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T23:04:43Z","timestamp":1793315083432,"version":"4.5.0"},"reference-count":79,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,9,11]],"date-time":"2023-09-11T00:00:00Z","timestamp":1694390400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychol."],"abstract":"<jats:sec><jats:title>Background<\/jats:title><jats:p>The rise of depression, anxiety, and suicide rates has led to increased demand for telemedicine-based mental health screening and remote patient monitoring (RPM) solutions to alleviate the burden on, and enhance the efficiency of, mental health practitioners. Multimodal dialog systems (MDS) that conduct on-demand, structured interviews offer a scalable and cost-effective solution to address this need.<\/jats:p><\/jats:sec><jats:sec><jats:title>Objective<\/jats:title><jats:p>This study evaluates the feasibility of a cloud based MDS agent, Tina, for mental state characterization in participants with depression, anxiety, and suicide risk.<\/jats:p><\/jats:sec><jats:sec><jats:title>Method<\/jats:title><jats:p>Sixty-eight participants were recruited through an online health registry and completed 73 sessions, with 15 (20.6%), 21 (28.8%), and 26 (35.6%) sessions screening positive for depression, anxiety, and suicide risk, respectively using conventional screening instruments. Participants then interacted with Tina as they completed a structured interview designed to elicit calibrated, open-ended responses regarding the participants' feelings and emotional state. Simultaneously, the platform streamed their speech and video recordings in real-time to a HIPAA-compliant cloud server, to compute speech, language, and facial movement-based biomarkers. After their sessions, participants completed user experience surveys. Machine learning models were developed using extracted features and evaluated with the area under the receiver operating characteristic curve (AUC).<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>For both depression and suicide risk, affected individuals tended to have a higher percent pause time, while those positive for anxiety showed reduced lip movement relative to healthy controls. In terms of single-modality classification models, speech features performed best for depression (AUC = 0.64; 95% CI = 0.51\u20130.78), facial features for anxiety (AUC = 0.57; 95% CI = 0.43\u20130.71), and text features for suicide risk (AUC = 0.65; 95% CI = 0.52\u20130.78). Best overall performance was achieved by decision fusion of all models in identifying suicide risk (AUC = 0.76; 95% CI = 0.65\u20130.87). Participants reported the experience comfortable and shared their feelings.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>MDS is a feasible, useful, effective, and interpretable solution for RPM in real-world clinical depression, anxiety, and suicidal populations. Facial information is more informative for anxiety classification, while speech and language are more discriminative of depression and suicidality markers. In general, combining speech, language, and facial information improved model performance on all classification tasks.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fpsyg.2023.1135469","type":"journal-article","created":{"date-parts":[[2023,9,11]],"date-time":"2023-09-11T15:27:49Z","timestamp":1694446069000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["A multimodal dialog approach to mental state characterization in clinically depressed, anxious, and suicidal populations"],"prefix":"10.3389","volume":"14","author":[{"given":"Joshua","family":"Cohen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vanessa","family":"Richter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Neumann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Black","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Allie","family":"Haq","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Wright-Berryman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vikram","family":"Ramanarayanan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,9,11]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"e0248842","DOI":"10.1371\/journal.pone.0248842","article-title":"Association between acoustic speech features and non-severe levels of anxiety and depression symptoms across lifespan","volume":"16","author":"Albuquerque","year":"2021","journal-title":"PLoS ONE"},{"key":"B2","doi-asserted-by":"publisher","first-page":"95","DOI":"10.36959\/584\/455","article-title":"Classification of anxiety disorders using machine learning methods: a literature review","volume":"4","author":"Arif","year":"2020","journal-title":"Insights Biomed. Res."},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1111\/j.1600-0447.1987.tb02891.x","article-title":"Acoustic analysis of speech variables during depression and after improvement","year":"1987","journal-title":"Acta Psychiatr. Scand."},{"key":"B4","article-title":"Blazeface: sub-millisecond neural face detection on mobile GPUs","author":"Bazarevsky","year":"2019","journal-title":"CoRR"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1155\/2013\/158746","article-title":"Psychomotor retardation in depression: a systematic review of diagnostic, pathophysiologic, and therapeutic implications","author":"Bennabi","year":"2013","journal-title":"BioMed Res. Int"},{"key":"B6","first-page":"2453","article-title":"\u201cAre reported accuracies in the clinical speech machine learning literature overoptimistic?,\u201d","volume-title":"Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, Vol. 2022","author":"Berisha","year":"2022"},{"key":"B7","doi-asserted-by":"publisher","first-page":"5929","DOI":"10.3390\/ijerph17165929","article-title":"Artificial intelligence and suicide prevention: a systematic review of machine learning investigations","volume":"17","author":"Bernert","year":"2020","journal-title":"Int. J. Environ. Res. Publ. Health"},{"key":"B8","first-page":"341","article-title":"Speak and unspeak with praat","volume":"5","author":"Boersma","year":"2001","journal-title":"Glot. Int."},{"key":"B9","first-page":"1921","author":"Botelho","year":"2022"},{"key":"B10","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1016\/j.bandc.2004.05.003","article-title":"Voice acoustical measurement of the severity of major depression","volume":"56","author":"Cannizzaro","year":"2004","journal-title":"Brain Cogn."},{"key":"B11","first-page":"343","article-title":"The psychological functions of function words","volume":"1","author":"Chung","year":"2007","journal-title":"Soc. Commun."},{"key":"B12","volume-title":"Statistical Power Analysis for the Behavioral Sciences, 2nd Edn)","author":"Cohen","year":"1988"},{"key":"B13","doi-asserted-by":"publisher","first-page":"818705","DOI":"10.3389\/fdgth.2022.818705","article-title":"Integration and validation of a natural language processing machine learning suicide risk prediction model based on open-ended interview language in the emergency department","volume":"4","author":"Cohen","year":"2022","journal-title":"Front. Digit. Health"},{"key":"B14","doi-asserted-by":"publisher","first-page":"21","DOI":"10.3390\/ijerph17218187","article-title":"A feasibility study using a machine learning suicide risk prediction model based on open-ended interview language in adolescent therapy sessions","volume":"17","author":"Cohen","year":"2020","journal-title":"Int. J. Environ. Res. Public Health"},{"key":"B15","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1016\/j.ymeth.2018.07.007","article-title":"Speech analysis for health: current state-of-the-art and the increasing impact of deep learning","volume":"151","author":"Cummins","year":"2018","journal-title":"Methods"},{"key":"B16","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1016\/j.specom.2015.03.004","article-title":"A review of depression and suicide risk assessment using speech analysis","volume":"71","author":"Cummins","year":"2015","journal-title":"Speech Commun."},{"key":"B17","first-page":"1061","author":"DeVault","year":"2014"},{"key":"B18","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/0301-0511(90)90024-Q","article-title":"Gender differences in facial reactions to facial expressions","volume":"30","author":"Dimberg","year":"1990","journal-title":"Biol. Psychol."},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0227-5","article-title":"Towards the automatic detection of social biomarkers in autism spectrum disorder: introducing the simulated interaction task (SIT)","author":"Drimalla","year":"2020","journal-title":"NPJ Digit. Med."},{"key":"B20","first-page":"363","article-title":"Hoarseness in adults","volume":"80","author":"Feierabend","year":"2009","journal-title":"Am. Fam. Phys."},{"key":"B21","doi-asserted-by":"publisher","first-page":"829","DOI":"10.1109\/10.846676","article-title":"Acoustical properties of speech as indicators of depression and suicidal risk","volume":"47","author":"France","year":"2000","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"B22","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1037\/bul0000084","article-title":"Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research","volume":"143","author":"Franklin","year":"2017","journal-title":"Psychol. Bullet."},{"key":"B23","doi-asserted-by":"publisher","first-page":"25199","DOI":"10.2196\/preprints.25199","article-title":"Validation of visual and auditory digital markers of suicidality in acutely suicidal psychiatric inpatients: proof-of-concept study","volume":"23","author":"Galatzer-Levy","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"B24","first-page":"85","article-title":"\u201cIt's only a computer: the impact of human-agent interaction in clinical interviews,\u201d","volume-title":"Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems","author":"Gratch","year":"2014"},{"key":"B25","first-page":"286","article-title":"\u201cCan virtual humans be more engaging than real ones?,\u201d","volume-title":"International Conference on Human-Computer Interaction","author":"Gratch","year":"2007"},{"key":"B26","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MCSE.2007.55","article-title":"Matplotlib: a 2D graphics environment","volume":"9","author":"Hunter","year":"2007","journal-title":"Comput. Sci. Eng."},{"key":"B27","article-title":"Real-time facial surface geometry from monocular video on mobile GPUs","author":"Kartynnik","year":"2019","journal-title":"CoRR"},{"key":"B28","first-page":"3658","author":"Kothare","year":"2022"},{"key":"B29","author":"Kothare","year":"2021"},{"key":"B30","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1046\/j.1525-1497.2001.016009606.x","article-title":"The PHQ-9: Validity of a brief depression severity measure","volume":"16","author":"Kroenke","year":"2001","journal-title":"J. Gen. Internal Med."},{"key":"B31","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1016\/j.jad.2008.06.026","article-title":"The PHQ-8 as a measure of current depression in the general population","volume":"114","author":"Kroenke","year":"2009","journal-title":"J. Affect. Disord."},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpsychires.2022.09.050","article-title":"The performance of machine learning models in predicting suicidal ideation, attempts, and deaths: a meta-analysis and systematic review","author":"Kusuma","year":"2022","journal-title":"J. Psychiatr. Res"},{"key":"B33","first-page":"770","author":"Laksana","year":"2017"},{"key":"B34","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1007\/s13218-015-0357-0","article-title":"Now all together: overview of virtual health assistants emulating face-to-face health interview experience","volume":"29","author":"Lisetti","year":"2015","journal-title":"KI-K\u00fcnstliche Intelligenz"},{"key":"B35","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1002\/lio2.354","article-title":"Automated assessment of psychiatric disorders using speech: a systematic review","volume":"5","author":"Low","year":"2019","journal-title":"Laryngos. Investig. Otolaryngol."},{"key":"B36","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1016\/j.chb.2014.04.043","article-title":"It's only a computer: virtual humans increase willingness to disclose","volume":"37","author":"Lucas","year":"2014","journal-title":"Comput. Hum. Behav."},{"key":"B37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/9780470479216.corpsy0491","article-title":"Kruskal-wallis test","volume":"2010","author":"McKight","year":"2010","journal-title":"Corsini Encycl. Psychol."},{"key":"B38","first-page":"51","author":"McKinney","year":"2010"},{"key":"B39","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/S0892-1997(96)80019-1","article-title":"Differences in voice quality between men and women: use of the long-term average spectrum (LTAS)","volume":"10","author":"Mendoza","year":"1996","journal-title":"J. Voice"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1301.3781","article-title":"Efficient estimation of word representations in vector space","author":"Mikolov","year":"2013","journal-title":"arXiv preprint arXiv:1301.3781"},{"key":"B41","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1016\/j.jneuroling.2006.04.001","article-title":"Voice acoustic measures of depression severity and treatment response collected via interactive voice response (IVR) technology","volume":"20","author":"Mundt","year":"2007","journal-title":"J. Neurolinguist."},{"key":"B42","doi-asserted-by":"publisher","first-page":"580","DOI":"10.1016\/j.biopsych.2012.03.015","article-title":"Vocal acoustic biomarkers of depression severity and treatment response","volume":"72","author":"Mundt","year":"2012","journal-title":"Biol. Psychiatry"},{"key":"B43","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1121\/1.405558","article-title":"Toward the simulation of emotion in synthetic speech: a review of the literature on human vocal emotion","volume":"93","author":"Murray","year":"1993","journal-title":"J. Acoust. Soc. Am."},{"key":"B44","first-page":"4783","article-title":"\u201cInvestigating the utility of multimodal conversational technology and audiovisual analytic measures for the assessment and monitoring of amyotrophic lateral sclerosis at scale,\u201d","volume-title":"Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association","author":"Neumann","year":"2021"},{"key":"B45","doi-asserted-by":"crossref","first-page":"47","DOI":"10.18653\/v1\/2020.nlpmc-1.7","article-title":"\u201cOn the utility of audiovisual dialog technologies and signal analytics for real-time remote monitoring of depression biomarkers,\u201d","author":"Neumann","year":"2020","journal-title":"Proceedings of the First Workshop on Natural Language Processing for Medical Conversations"},{"key":"B46","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1109\/MCSE.2007.58","article-title":"Python for scientific computing","volume":"9","author":"Oliphant","year":"2007","journal-title":"Comput. Sci. Eng."},{"key":"B47","first-page":"2825","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Machine Learn. Res."},{"key":"B48","first-page":"1532","author":"Pennington","year":"2014"},{"key":"B49","doi-asserted-by":"publisher","first-page":"516","DOI":"10.1521\/suli.2010.40.5.516","article-title":"A conversation with edwin shneidman","volume":"40","author":"Pestian","year":"2010","journal-title":"Suicide Life-Threat. Behav."},{"key":"B50","doi-asserted-by":"publisher","DOI":"10.4137\/BII.S4706","article-title":"Suicide note classification using natural language processing: a content analysis","author":"Pestian","year":"2010","journal-title":"Biomed. Informat. Insights"},{"key":"B51","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1111\/sltb.12180","article-title":"A controlled trial using natural language processing to examine the language of suicidal adolescents in the emergency department","volume":"46","author":"Pestian","year":"2016","journal-title":"Suicide Life-Threat. Behav."},{"key":"B52","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1111\/sltb.12312","article-title":"A machine learning approach to identifying the thought markers of suicidal subjects: a prospective multicenter trial","volume":"47","author":"Pestian","year":"2017","journal-title":"Suicide Life-Threat. Behav."},{"key":"B53","doi-asserted-by":"publisher","first-page":"1266","DOI":"10.1176\/appi.ajp.2011.10111704","article-title":"The columbia-suicide severity rating scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults","volume":"168","author":"Posner","year":"2011","journal-title":"Am. J. Psychiatry"},{"key":"B54","first-page":"1","volume-title":"Data Mining","author":"Rajaraman","year":"2011"},{"key":"B55","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1044\/2021_PERSP-21-00174","article-title":"Speech as a biomarker: opportunities, interpretability, and challenges","volume":"7","author":"Ramanarayanan","year":"2022","journal-title":"Perspect. ASHA Spec. Interest Groups"},{"key":"B56","first-page":"492","author":"Ramanarayanan","year":"2020"},{"key":"B57","first-page":"1135","author":"Ribeiro","year":"2016"},{"key":"B58","doi-asserted-by":"publisher","first-page":"615","DOI":"10.1007\/s10979-005-6832-7","article-title":"Comparing effect sizes in follow-up studies: ROC area, Cohen's D, and R","volume":"29","author":"Rice","year":"2005","journal-title":"Law Hum. Behav."},{"key":"B59","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1145\/3536220.3558075","article-title":"\u201cTowards multimodal dialog-based speech and facial biomarkers of schizophrenia,\u201d","volume-title":"Companion Publication of the 2022 International Conference on Multimodal Interaction","author":"Richter","year":"2022"},{"key":"B60","first-page":"57","author":"Rizzo","year":"2016"},{"key":"B61","first-page":"160","author":"Roesler","year":"2022"},{"key":"B62","doi-asserted-by":"publisher","first-page":"710","DOI":"10.1176\/appi.ps.201700344","article-title":"Projected workforce of psychiatrists in the united states: a population analysis","volume":"69","author":"Satiani","year":"2018","journal-title":"Psychiatr. Serv."},{"key":"B63","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1037\/0021-843X.102.3.430","article-title":"Voices of fear and anxiety and sadness and depression: the effects of speech rate and loudness on fear and anxiety and sadness and depression","volume":"102","author":"Siegman","year":"1993","journal-title":"J. Abnorm. Psychol."},{"key":"B64","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1111\/j.1749-818X.2009.00125.x","article-title":"Phonetic differences between male and female speech","volume":"3","author":"Simpson","year":"2009","journal-title":"Lang. Linguist. Compass"},{"key":"B65","doi-asserted-by":"publisher","first-page":"1092","DOI":"10.1001\/archinte.166.10.1092","article-title":"A brief measure for assessing generalized anxiety disorder: the GAD-7","volume":"166","author":"Spitzer","year":"2006","journal-title":"Archiv. Internal Med."},{"key":"B66","author":"Suendermann-Oeft","year":"2019"},{"key":"B67","year":"2020","journal-title":"Pandas-dev\/Pandas: Pandas"},{"key":"B68","doi-asserted-by":"publisher","first-page":"1699","DOI":"10.1121\/1.397959","article-title":"Physiologic and acoustic differences between male and female voices","volume":"85","author":"Titze","year":"1989","journal-title":"J. Acousti. Soc. Am."},{"key":"B69","author":"Torchiano","year":"2020","journal-title":"effsize: Efficient Effect Size Computation. R Package Version 0.7.6"},{"key":"B70","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0224365","article-title":"Machine learning algorithm validation with a limited sample size","volume":"14","author":"Vabalas","year":"2019","journal-title":"PLoS ONE"},{"key":"B71","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MCSE.2011.37","article-title":"The numpy array: a structure for efficient numerical computation","volume":"13","author":"Van Der Walt","year":"2011","journal-title":"Comput. Sci. Eng."},{"key":"B72","volume-title":"Python Tutorial, Vol. 620","author":"Van Rossum","year":"1995"},{"key":"B73","first-page":"277","author":"Venek","year":"2015"},{"key":"B74","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-020-0772-5","article-title":"SciPy 1.0: fundamental algorithms for scientific computing in python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat. Methods"},{"key":"B75","volume-title":"Suicide Worldwide in 2019: Global Health Estimates","year":"2021"},{"key":"B76","volume-title":"Mental Health and COVID-19: Early Evidence of the Pandemic's Impact. Technical Report","year":"2022"},{"key":"B77","doi-asserted-by":"publisher","first-page":"1143175","DOI":"10.3389\/fpsyt.2023.1143175","article-title":"Virtually screening adults for depression, anxiety, and suicide risk using machine learning and language from an open-ended interview","volume":"14","author":"Wright-Berryman","year":"2023","journal-title":"Front. Psychiatry"},{"key":"B78","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1093\/jpepsy\/jst062","article-title":"A primer on receiver operating characteristic analysis and diagnostic efficiency statistics for pediatric psychology: we are ready to ROC","volume":"39","author":"Youngstrom","year":"2013","journal-title":"J. Pediatr. Psychol."},{"key":"B79","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1121\/1.387808","article-title":"Harmonics-to-noise ratio as an index of the degree of hoarseness","volume":"71","author":"Yumoto","year":"1982","journal-title":"J. Acoust. Soc. Am."}],"container-title":["Frontiers in Psychology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2023.1135469\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,11]],"date-time":"2023-09-11T15:28:02Z","timestamp":1694446082000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2023.1135469\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,11]]},"references-count":79,"alternative-id":["10.3389\/fpsyg.2023.1135469"],"URL":"https:\/\/doi.org\/10.3389\/fpsyg.2023.1135469","relation":{},"ISSN":["1664-1078"],"issn-type":[{"value":"1664-1078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,11]]},"article-number":"1135469"}}