{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T15:14:30Z","timestamp":1790608470274,"version":"4.1.0"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n There is considerable need to improve and increase the detection and measurement of depression. The use of speech as a digital biomarker of depression represents a considerable opportunity for transforming and accelerating depression identification and treatment; however, research to date has primarily consisted of small-sample feasibility or pilot studies incorporating highly controlled applications and settings. There has been limited examination of the technology in real-world use contexts.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study evaluated the performance of a machine learning (ML) model examining both semantic and acoustic properties of speech in predicting depression across more than 2000 real-world interactions between health plan members and case managers.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A total of 2086 recordings of case management calls with verbally administered Patient Health Questionnaire\u20149 questions (PHQ-9) surveys were analyzed using the ML model after the portions of the recordings with the PHQ-9 survey were manually redacted. The recordings were divided into a Development Set (Dev Set) (n=1336) and a Blind Set (n=671and Patient Health Questionnaire\u20148 questions (PHQ-8) scores were provided for the Dev Set for ML model refinement while PHQ-8 scores from the Blind Set were withheld until after ML model depression severity output was reported.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The Dev Set and the Blind Set were well matched for age (Dev Set: mean 53.7, SD 16.3 years; Blind Set: mean 51.7, SD 16.9 years), gender (Dev Set: 910\/1336, 68.1% of female participants; Blind Set: 462\/671, 68.9% of female participants), and depression severity (Dev Set: mean 10.5, SD 6.1 of PHQ-8 scores; Blind Set: mean 10.9, SD 6.0 of PHQ-8 scores). The concordance correlation coefficient was \u03c1\n c<\/jats:sub>\n =0.57 for the test of the ML model on the Dev Set\n \n and\n \n on the Blind Set\n \n , while the mean absolute error for the Dev Set and for the Blind Set, demonstrating strong model performance. This performance was maintained when dividing each set into subgroups of age brackets (\u226439, 40\u201064, and \u226565 years), biological sex, and the 4 categories of Social Vulnerability Index (an index based on 16 social factors), with concordance correlation coefficients ranging as \u03c1\n c<\/jats:sub>\n =0.44\u20100.61. Performance at PHQ-8 threshold score cutoffs of 5, 10, 15, and 20, representing the depression severity categories of none, mild, moderate, moderately severe, and severe (\u226520), respectively, expressed as area under the receiver operating characteristic curve values, varied between 0.79 and 0.83 in both the Dev and Blind Sets.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Overall, the findings suggest that speech may have significant potential for detection and measurement of depression severity over a variety of ages, gender, and socioeconomic categories that may enhance treatment, improve clinical decision-making, and enable truly personalized treatment recommendations.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/69149","type":"journal-article","created":{"date-parts":[[2025,3,31]],"date-time":"2025-03-31T00:36:27Z","timestamp":1743381387000},"page":"e69149-e69149","source":"Crossref","is-referenced-by-count":7,"title":["Digital Phenotyping for Detecting Depression Severity in a Large Payor-Provider System: Retrospective Study of Speech and Language Model Performance"],"prefix":"10.2196","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8184-3565","authenticated-orcid":false,"given":"Bradley","family":"Karlin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2051-5469","authenticated-orcid":false,"given":"Doug","family":"Henry","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8125-1655","authenticated-orcid":false,"given":"Ryan","family":"Anderson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4358-5903","authenticated-orcid":false,"given":"Salvatore","family":"Cieri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4749-6413","authenticated-orcid":false,"given":"Michael","family":"Aratow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3779-4956","authenticated-orcid":false,"given":"Elizabeth","family":"Shriberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5328-0649","authenticated-orcid":false,"given":"Michelle","family":"Hoy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"R1","unstructured":"Substance Abuse and Mental Health Services Administration. 2021 NSDUH Annual National Report. CBHSQ Data; 2021. URL: https:\/\/www.samhsa.gov\/data\/report\/2021-nsduh-annual-national-report [Accessed 24-06-2024]"},{"issue":"13","key":"R2","doi-asserted-by":"publisher","first-page":"490","DOI":"10.15585\/mmwr.mm7013e2","article-title":"Symptoms of anxiety or depressive disorder and use of mental health care among adults during the COVID-19 pandemic\u2014United States, August 2020\u2013February 2021","volume":"70","author":"Vahratian","journal-title":"MMWR Morb Mortal Wkly Rep"},{"issue":"9","key":"R3","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.19686","article-title":"Prevalence of depression symptoms in US adults before and during the COVID-19 pandemic","volume":"3","author":"Ettman","journal-title":"JAMA Netw Open"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"92","DOI":"10.2105\/ajph.92.1.92","article-title":"Adequacy of treatment for serious mental illness in the United States","volume":"92","author":"Wang","journal-title":"Am J Public Health"},{"key":"R5","unstructured":"Lovett L . BH Business. Highmark health\u2019s behavioral health director: personalized care, upstream prevention will define the industry. Apr 10, 2023. URL: https:\/\/bhbusiness.com\/2023\/04\/10\/highmark-healths-behavioral-health-director-personalized-care-upstream-prevention-will-define-the-industry\/ [Accessed 24-06-2024]"},{"issue":"6","key":"R6","doi-asserted-by":"publisher","DOI":"10.1200\/CCI.22.00095","article-title":"Digital phenotyping of behavioral symptoms as the next frontier for personalized and proactive cancer care","volume":"6","author":"Fahed","journal-title":"JCO Clin Cancer Inform"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0166-1","article-title":"Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety","volume":"2","author":"Huckvale","journal-title":"NPJ Digit Med"},{"issue":"13","key":"R8","doi-asserted-by":"publisher","first-page":"1215","DOI":"10.1001\/jama.2017.11295","article-title":"Digital phenotyping: technology for a new science of behavior","volume":"318","author":"Insel","journal-title":"JAMA"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0251-5","article-title":"Digital phenotyping, behavioral sensing, or personal sensing: names and transparency in the digital age","volume":"3","author":"Mohr","journal-title":"NPJ Digit Med"},{"issue":"6","key":"R10","doi-asserted-by":"publisher","first-page":"1331","DOI":"10.3233\/THC-213648","article-title":"Digital phenotyping in psychiatry: a scoping review","volume":"30","author":"Chia","journal-title":"Technol Health Care"},{"issue":"10328","key":"R11","doi-asserted-by":"publisher","first-page":"957","DOI":"10.1016\/S0140-6736(21)02141-3","article-title":"Time for united action on depression: a Lancet\u2013World Psychiatric Association Commission","volume":"399","author":"Herrman","journal-title":"Lancet"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-clinpsy-080921-073212","article-title":"Machine learning and the digital measurement of psychological health","volume":"19","author":"Galatzer-Levy","journal-title":"Annu Rev Clin Psychol"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1002\/lio2.354","article-title":"Automated assessment of psychiatric disorders using speech: a systematic review","volume":"5","author":"Low","journal-title":"Laryngoscope Investig Otolaryngol"},{"issue":"11","key":"R14","doi-asserted-by":"publisher","first-page":"e829","DOI":"10.1016\/S2589-7500(22)00153-4","article-title":"From promise to practice: towards the realisation of AI-informed mental health care","volume":"4","author":"Koutsouleris","journal-title":"Lancet Digit Health"},{"issue":"9","key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/24352","article-title":"Using acoustic speech patterns from smartphones to investigate mood disorders: scoping review","volume":"9","author":"Flanagan","journal-title":"JMIR Mhealth Uhealth"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Shen Y Yang H Lin L . Automatic depression detection: an emotional audio-textual corpus and a GRU\/bilstm-based model. Presented at: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 6247-6251; Singapore, Singapore. [doi: 10.1109\/ICASSP43922.2022.9746569]","DOI":"10.1109\/ICASSP43922.2022.9746569"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1038\/s41537-020-00115-2","article-title":"Using machine learning of computerized vocal expression to measure blunted vocal affect and alogia","volume":"6","author":"Cohen","journal-title":"NPJ Schizophr"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jad.2017.08.060","article-title":"The psychometric properties of depression screening tools in primary healthcare settings: a systematic review","volume":"225","author":"El-Den","journal-title":"J Affect Disord"},{"issue":"9","key":"R19","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1046\/j.1525-1497.2001.016009606.x","article-title":"The PHQ-9: validity of a brief depression severity measure","volume":"16","author":"Kroenke","journal-title":"J Gen Intern Med"},{"issue":"9","key":"R20","doi-asserted-by":"publisher","first-page":"509","DOI":"10.3928\/0048-5713-20020901-06","article-title":"The PHQ-9: a new depression diagnostic and severity measure","volume":"32","author":"Kroenke","journal-title":"Psychiatr Ann"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Rutowski T Shriberg E Harati A Lu Y Oliveira R Chlebek P . Cross-demographic portability of deep NLP-based depression models. Presented at: 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE. 1052-1057; Shenzhen, China. [doi: 10.1109\/SLT48900.2021.9383609]","DOI":"10.1109\/SLT48900.2021.9383609"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Harati A Shriberg E Rutowski T Chlebek P Lu Y Oliveira R . Speech-based depression prediction using encoder-weight-only transfer learning and a large corpus. Presented at: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 7273-7277; Toronto, ON, Canada. 2021.[doi: 10.1109\/ICASSP39728.2021.9414208]","DOI":"10.1109\/ICASSP39728.2021.9414208"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2022.811517","article-title":"Feasibility of a machine learning-based smartphone application in detecting depression and anxiety in a generally senior population","volume":"13","author":"Lin","journal-title":"Front Psychol"},{"key":"R24","unstructured":"Baevski A Zhou Y Mohamed A Auli M . Wav2vec 2.0: a framework for self-supervised learning of speech representations. Adv Neural Inf Process Syst. Preprint posted online on 2020. [doi: 10.48550\/arXiv.2006.11477]"},{"key":"R25","unstructured":"Beltagy I Peters ME Cohan A . Longformer: the long-document transformer. arXiv. Preprint posted online on 2020. URL: https:\/\/arxiv.org\/abs\/2004.05150"},{"key":"R26","unstructured":"Amazon Web Services, Inc. Speech to text\u2014Amazon Transcribe\u2014AWS. Jun 24, 2024. URL: https:\/\/aws.amazon.com\/transcribe\/ [Accessed 24-06-2024]"},{"key":"R27","unstructured":"Amazon Web Services, Inc. Introducing medical language processing with Amazon Comprehend Medical | AWS ML blog. Nov 27, 2018. URL: https:\/\/aws.amazon.com\/blogs\/machine-learning\/introducing-medical-language-processing-with-amazon-comprehend-medical\/ [Accessed 24-06-2024]"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Fara S Goria S Molimpakis E Cummins N . Speech and the n-Back task as a lens into depression. How combining both may allow us to isolate different core symptoms of depression. In: Interspeech 2022. ISCA; 2022:1911-1915. [doi: 10.48550\/arXiv.2204.00088]","DOI":"10.21437\/Interspeech.2022-10393"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Seneviratne N Espy-Wilson C . Multimodal depression severity score prediction using articulatory coordination features and hierarchical attention based text embeddings. Presented at: Interspeech 2022. 3353-3357; 2022.[doi: 10.21437\/Interspeech.2022-11099]","DOI":"10.21437\/Interspeech.2022-11099"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Wang J Ravi V Flint J Alwan A . Unsupervised instance discriminative learning for depression detection from speech signals. Presented at: Interspeech 2022. 2018-2022; 2022.[doi: 10.21437\/Interspeech.2022-10814]","DOI":"10.21437\/Interspeech.2022-10814"},{"key":"R31","unstructured":"CDC\/ATSDR. Social Vulnerability Index (CDC\/ATSDR SVI). Jun 14, 2024. URL: https:\/\/www.atsdr.cdc.gov\/place-health\/php\/svi\/?CDC_AAref_Val=https:\/\/www.atsdr.cdc.gov\/placeandhealth\/svi\/index.html [Accessed 24-06-2024]"},{"key":"R32","doi-asserted-by":"crossref","unstructured":"Bn S Abdullah S . Privacy sensitive speech analysis using federated learning to assess depression. Presented at: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 6272-6276; 2022.[doi: 10.1109\/ICASSP43922.2022.9746827]","DOI":"10.1109\/ICASSP43922.2022.9746827"},{"key":"R33","doi-asserted-by":"crossref","unstructured":"Rutowski T Harati A Shriberg E Lu Y Chlebek P Oliveira R . Toward corpus size requirements for training and evaluating depression risk models using spoken language. Presented at: Interspeech 2022. 3343-3347; 2022.[doi: 10.21437\/Interspeech.2022-10888]","DOI":"10.21437\/Interspeech.2022-10888"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","DOI":"10.1186\/s12991-021-00375-x","article-title":"Prevalence and determinants of depression among old age: a systematic review and meta-analysis","volume":"20","author":"Zenebe","journal-title":"Ann Gen Psychiatry"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.2147\/NDT.S347356","article-title":"Recognizing depression in the elderly: practical guidance and challenges for clinical management","volume":"18","author":"Devita","journal-title":"Neuropsychiatr Dis Treat"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1097\/00004850-199809005-00004","article-title":"Is there a difference between elderly and younger patients with regard to the symptomatology and aetiology of depression?","volume":"13 Suppl 5","author":"Gottfries","journal-title":"Int Clin Psychopharmacol"},{"issue":"4","key":"R37","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1192\/bjp.bp.111.095950","article-title":"Phenomenology of depression in older compared with younger adults: meta-analysis","volume":"200","author":"Hegeman","journal-title":"Br J Psychiatry"},{"issue":"7","key":"R38","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1001\/jama.2014.8040","article-title":"The mental health comorbidities of diabetes","volume":"312","author":"Ducat","journal-title":"JAMA"},{"key":"R39","unstructured":"Davenport S Gray T Melek S Milliman Research Report. Milliman high-cost patient study 2020. Vol URL. Aug 13, 2020. URL: https:\/\/www.milliman.com\/-\/media\/milliman\/pdfs\/articles\/milliman-high-cost-patient-study-2020.pdf [Accessed 16-05-2025]"},{"issue":"12","key":"R40","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2022.44644","article-title":"Association of outpatient behavioral health treatment with medical and pharmacy costs in the first 27 months following a new behavioral health diagnosis in the US","volume":"5","author":"Bellon","journal-title":"JAMA Netw Open"},{"issue":"643","key":"R41","doi-asserted-by":"publisher","first-page":"e78","DOI":"10.3399\/bjgp16X683473","article-title":"Usefulness of PHQ-9 in primary care to determine meaningful symptoms of low mood: a qualitative study","volume":"66","author":"Malpass","journal-title":"Br J Gen Pract"},{"issue":"2","key":"R42","doi-asserted-by":"publisher","first-page":"E44","DOI":"10.1503\/cmaj.170691","article-title":"Addressing overestimation of the prevalence of depression based on self-report screening questionnaires","volume":"190","author":"Thombs","journal-title":"CMAJ"},{"issue":"6","key":"R43","doi-asserted-by":"publisher","first-page":"737","DOI":"10.1509\/jmr.12.0336","article-title":"Quantifying under- and overreporting in surveys through a dual-questioning-technique design","volume":"52","author":"De Jong","journal-title":"J Mark Res"},{"issue":"1","key":"R44","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.jadohealth.2009.08.008","article-title":"Mental health problems and help-seeking behavior among college students","volume":"46","author":"Hunt","journal-title":"J Adolesc Health"},{"key":"R45","doi-asserted-by":"publisher","DOI":"10.1016\/j.genhosppsych.2020.11.016","article-title":"Discrepancies between self-rated depression and observed depression severity: the effects of personality and dysfunctional attitudes","volume":"70","author":"Ma","journal-title":"Gen Hosp Psychiatry"},{"key":"R46","doi-asserted-by":"crossref","unstructured":"Harati A Rutowski T Lu Y et al. Generalization of deep acoustic and NLP models for large-scale depression screening. In: Obeid I Picone J Selesnick I , editors. Biomedical Sensing and Analysis. Springer International Publishing; 2022:99-132. [doi: 10.1007\/978-3-030-99383-2_318]","DOI":"10.1007\/978-3-030-99383-2_3"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.l1781","article-title":"Accuracy of Patient Health Questionnaire-9 (PHQ-9) for screening to detect major depression: individual participant data meta-analysis","volume":"PMCID","author":"Levis","journal-title":"BMJ"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e69149","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:56:37Z","timestamp":1782842197000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e69149"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/69149","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]},"article-number":"v4i13e69149"}}