{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T18:08:16Z","timestamp":1777486096032,"version":"3.51.4"},"reference-count":43,"publisher":"Oxford University Press (OUP)","issue":"1","funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","award":["ZIC-MH002968"],"award-info":[{"award-number":["ZIC-MH002968"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We aim to use large language models (LLMs) to detect mentions of nuanced psychotherapeutic outcomes and impacts than previously considered in transcripts of interviews with adolescent depression. Our clinical authors previously created a novel coding framework containing fine-grained therapy outcomes beyond the binary classification (eg, depression vs control) based on qualitative analysis embedded within a clinical study of depression. Moreover, we seek to demonstrate that embeddings from LLMs are informative enough to accurately label these experiences.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Data were drawn from interviews, where text segments were annotated with different outcome labels. Five different open-source LLMs were evaluated to classify outcomes from the coding framework. Classification experiments were carried out in the original interview transcripts. Furthermore, we repeated those experiments for versions of the data produced by breaking those segments into conversation turns, or keeping non-interviewer utterances (monologues).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We used classification models to predict 31 outcomes and 8 derived labels, for 3 different text segmentations. Area under the ROC curve scores ranged between 0.6 and 0.9 for the original segmentation and 0.7 and 1.0 for the monologues and turns.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>LLM-based classification models could identify outcomes important to adolescents, such as friendships or academic and vocational functioning, in text transcripts of patient interviews. By using clinical data, we also aim to better generalize to clinical settings compared to studies based on public social media data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Our results demonstrate that fine-grained therapy outcome coding in psychotherapeutic text is feasible, and can be used to support the quantification of important outcomes for downstream uses.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocae298","type":"journal-article","created":{"date-parts":[[2024,11,25]],"date-time":"2024-11-25T15:24:05Z","timestamp":1732548245000},"page":"79-89","source":"Crossref","is-referenced-by-count":10,"title":["Using large language models to detect outcomes in qualitative studies of adolescent depression"],"prefix":"10.1093","volume":"33","author":[{"given":"Alison W","family":"Xin","sequence":"first","affiliation":[{"name":"Machine Learning Core, National Institute of Mental Health, National Institutes of Health , Bethesda, MD 20892,","place":["United States"]}]},{"given":"Dylan M","family":"Nielson","sequence":"additional","affiliation":[{"name":"Machine Learning Core, National Institute of Mental Health, National Institutes of Health , Bethesda, MD 20892,","place":["United States"]}]},{"given":"Karolin Rose","family":"Krause","sequence":"additional","affiliation":[{"name":"Centre of Research in Epidemiology and Statistics (CRESS UMR 1153), Universit\u00e9 Paris Cit\u00e9 , Paris 75004,","place":["France"]}]},{"given":"Guilherme","family":"Fiorini","sequence":"additional","affiliation":[{"name":"Department of Clinical, Educational and Health Psychology, University College , London WC1E 6BT,","place":["United Kingdom"]}]},{"given":"Nick","family":"Midgley","sequence":"additional","affiliation":[{"name":"Department of Clinical, Educational and Health Psychology, University College , London WC1E 6BT,","place":["United Kingdom"]}]},{"given":"Francisco","family":"Pereira","sequence":"additional","affiliation":[{"name":"Machine Learning Core, National Institute of Mental Health, National Institutes of Health , Bethesda, MD 20892,","place":["United States"]}]},{"given":"Juan Antonio","family":"Lossio-Ventura","sequence":"additional","affiliation":[{"name":"Machine Learning Core, National Institute of Mental Health, National Institutes of Health , Bethesda, MD 20892,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2024,12,11]]},"reference":[{"key":"2026010211513393900_ocae298-B1","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1111\/bjc.12333","article-title":"Global prevalence of depression and elevated depressive symptoms among adolescents: a systematic review and meta-analysis","volume":"61","author":"Shorey","year":"2022","journal-title":"Br J Clin Psychol"},{"key":"2026010211513393900_ocae298-B2","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.jaac.2018.07.893","article-title":"Review: what outcomes count? A review of outcomes measured for adolescent depression between 2007 and 2017","volume":"58","author":"Krause","year":"2019","journal-title":"J Am Acad Child Adolesc Psychiatry"},{"key":"2026010211513393900_ocae298-B3","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1037\/a0034179","article-title":"The meaningful assessment of therapy outcomes: Incorporating a qualitative study into a randomized controlled trial evaluating the treatment of adolescent depression","volume":"51","author":"Midgley","year":"2014","journal-title":"Psychotherapy (Chic)"},{"key":"2026010211513393900_ocae298-B4","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1186\/1745-6215-12-175","article-title":"Improving mood with psychoanalytic and cognitive therapies (IMPACT): a pragmatic effectiveness superiority trial to investigate whether specialised psychological treatment reduces the risk for relapse in adolescents with moderate to severe unipolar depression: study protocol for a randomised controlled trial","volume":"12","author":"Goodyer","year":"2011","journal-title":"Trials"},{"key":"2026010211513393900_ocae298-B5","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1016\/S2215-0366(16)30378-9","article-title":"Cognitive behavioural therapy and short-term psychoanalytical psychotherapy versus a brief psychosocial intervention in adolescents with unipolar major depressive disorder (IMPACT): a multicentre, pragmatic, observer-blind, randomised controlled superiority trial","volume":"4","author":"Goodyer","year":"2017","journal-title":"Lancet Psychiatry"},{"key":"2026010211513393900_ocae298-B6","doi-asserted-by":"publisher","first-page":"1779","DOI":"10.1007\/s00787-020-01648-8","article-title":"A comprehensive mapping of outcomes following psychotherapy for adolescent depression: the perspectives of young people, their parents and therapists","volume":"30","author":"Krause","year":"2021","journal-title":"Eur Child Adolesc Psychiatry"},{"key":"2026010211513393900_ocae298-B7","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1038\/s41746-022-00589-7","article-title":"Natural language processing applied to mental illness detection: a narrative review","volume":"5","author":"Zhang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2026010211513393900_ocae298-B8","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1038\/s41746-020-0233-7","article-title":"Methods in predictive techniques for mental health status on social media: a critical review","volume":"3","author":"Chancellor","year":"2020","journal-title":"NPJ Digit Med"},{"key":"2026010211513393900_ocae298-B9","author":"Pennington","year":"2014"},{"key":"2026010211513393900_ocae298-B10","author":"Mikolov","year":"2013"},{"key":"2026010211513393900_ocae298-B11","first-page":"4171","author":"Devlin","year":"2019"},{"key":"2026010211513393900_ocae298-B12","first-page":"1218","author":"Zhuang","year":"2021"},{"key":"2026010211513393900_ocae298-B13","first-page":"98","author":"Guntuku","year":"2018"},{"key":"2026010211513393900_ocae298-B14","first-page":"331","author":"Bandyopadhyay","year":"2019"},{"key":"2026010211513393900_ocae298-B15","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/s40708-023-00188-6","article-title":"Deep learning and machine learning in psychiatry: a survey of current progress in depression detection, diagnosis and treatment","volume":"10","author":"Squires","year":"2023","journal-title":"Brain Inform"},{"key":"2026010211513393900_ocae298-B16","doi-asserted-by":"publisher","author":"Touvron","year":"2023","DOI":"10.48550\/arXiv.2302.13971"},{"key":"2026010211513393900_ocae298-B17","doi-asserted-by":"publisher","author":"Touvron","year":"2023","DOI":"10.48550\/arXiv.2307.09288"},{"key":"2026010211513393900_ocae298-B18","author":"Meta","year":"2024"},{"key":"2026010211513393900_ocae298-B19","doi-asserted-by":"publisher","author":"Jiang","year":"2023","DOI":"10.48550\/arXiv.2310.06825"},{"key":"2026010211513393900_ocae298-B20","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026010211513393900_ocae298-B21","first-page":"147","author":"Jiang","year":"2020"},{"key":"2026010211513393900_ocae298-B22","first-page":"718","author":"Malviya","year":"2021"},{"key":"2026010211513393900_ocae298-B23","first-page":"e50729","article-title":"Safety of large language models in addressing depression","volume":"15","author":"Heston","year":"2023","journal-title":"Cureus"},{"key":"2026010211513393900_ocae298-B24","first-page":"12","author":"Aragon","year":"2024"},{"key":"2026010211513393900_ocae298-B25","first-page":"108","author":"Wang","year":"2024"},{"key":"2026010211513393900_ocae298-B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3643540","article-title":"Mental-LLM: leveraging large language models for mental health prediction via online text data","volume":"8","author":"Xu","year":"2024","journal-title":"Proc ACM Interact Mob Wearable Ubiquitous Technol"},{"key":"2026010211513393900_ocae298-B27","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1007\/978-3-031-42448-9_22","author":"Parapar","year":"2023"},{"key":"2026010211513393900_ocae298-B28","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1037\/1040-3590.10.2.83","article-title":"A psychometric evaluation of the Beck Depression Inventory-II","volume":"10","author":"Dozois","year":"1998","journal-title":"Psychol Assess"},{"key":"2026010211513393900_ocae298-B29","doi-asserted-by":"publisher","author":"Zhang","year":"2022","DOI":"10.24963\/ijcai.2022\/725"},{"key":"2026010211513393900_ocae298-B30","author":"P\u00e9rez","year":"2023"},{"key":"2026010211513393900_ocae298-B31","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1046\/j.1525-1497.2001.016009606.x","article-title":"The PHQ-9: validity of a brief depression severity measure","volume":"16","author":"Kroenke","year":"2001","journal-title":"J Gen Intern Med"},{"key":"2026010211513393900_ocae298-B32","author":"Nguyen","year":"2022"},{"key":"2026010211513393900_ocae298-B33","first-page":"8026","author":"Paszke","year":"2019"},{"key":"2026010211513393900_ocae298-B34","author":"Wolf","year":"2020"},{"key":"2026010211513393900_ocae298-B35","doi-asserted-by":"publisher","author":"Devlin","DOI":"10.18653\/v1\/N19-1423"},{"key":"2026010211513393900_ocae298-B36","first-page":"7184","author":"Ji","year":"2022"},{"key":"2026010211513393900_ocae298-B37","doi-asserted-by":"publisher","author":"Ji","year":"2023","DOI":"10.48550\/arXiv.2304.10447"},{"key":"2026010211513393900_ocae298-B38","doi-asserted-by":"publisher","author":"Beltagy","year":"2020","DOI":"10.48550\/arXiv.2004.05150"},{"key":"2026010211513393900_ocae298-B39","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"2026010211513393900_ocae298-B40","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","article-title":"An introduction to ROC analysis","volume":"27","author":"Fawcett","year":"2006","journal-title":"Pattern Recognit Lett"},{"key":"2026010211513393900_ocae298-B41","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1080\/01621459.1937.10503522","article-title":"The use of ranks to avoid the assumption of normality implicit in the analysis of variance","volume":"32","author":"Friedman","year":"1937","journal-title":"J Am Stat Assoc"},{"key":"2026010211513393900_ocae298-B42","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","author":"Dem\u0161ar","year":"2006","journal-title":"J Mach Learn Res"},{"key":"2026010211513393900_ocae298-B43","first-page":"1","article-title":"Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis","volume":"18","author":"Benavoli","year":"2017","journal-title":"J Mach Learn Res"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocae298\/61052551\/ocae298.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocae298\/61052551\/ocae298.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T16:51:42Z","timestamp":1767372702000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/1\/79\/7921513"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,11]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12,11]]},"published-print":{"date-parts":[[2026,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae298","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,1]]},"published":{"date-parts":[[2024,12,11]]}}}