{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,20]],"date-time":"2026-09-20T01:10:24Z","timestamp":1789866624263,"version":"4.0.1"},"reference-count":49,"publisher":"Public Library of Science (PLoS)","issue":"11","license":[{"start":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T00:00:00Z","timestamp":1701043200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>In treating depression and anxiety, just over half of all clients respond. Monitoring and obtaining early client feedback can allow for rapidly adapted treatment delivery and improve outcomes. This study seeks to develop a state-of-the-art deep-learning framework for predicting clinical outcomes in internet-delivered Cognitive Behavioural Therapy (iCBT) by leveraging large-scale, high-dimensional time-series data of client-reported mental health symptoms and platform interaction data. We use de-identified data from 45,876 clients on SilverCloud Health, a digital platform for the psychological treatment of depression and anxiety. We train deep recurrent neural network (RNN) models to predict whether a client will show reliable improvement by the end of treatment using clinical measures, interaction data with the iCBT program, or both. Outcomes are based on total improvement in symptoms of depression (Patient Health Questionnaire-9, PHQ-9) and anxiety (Generalized Anxiety Disorder-7, GAD-7), as reported within the iCBT program. Using internal and external datasets, we compare the proposed models against several benchmarks and rigorously evaluate them according to their predictive accuracy, sensitivity, specificity and AUROC over treatment. Our proposed RNN models consistently predict reliable improvement in PHQ-9 and GAD-7, using past clinical measures alone, with above 87% accuracy and 0.89 AUROC after three or more review periods, outperforming all benchmark models. Additional evaluations demonstrate the robustness of the achieved models across (i) different health services; (ii) geographic locations; (iii) iCBT programs, and (iv) client severity subgroups. Results demonstrate the robust performance of dynamic prediction models that can yield clinically helpful prognostic information ready for implementation within iCBT systems to support timely decision-making and treatment adjustments by iCBT clinical supporters towards improved client outcomes.<\/jats:p>","DOI":"10.1371\/journal.pone.0272685","type":"journal-article","created":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T19:08:50Z","timestamp":1701112130000},"page":"e0272685","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":21,"title":["Deep learning for the prediction of clinical outcomes in internet-delivered CBT for depression and anxiety"],"prefix":"10.1371","volume":"18","author":[{"given":"Niranjani","family":"Prasad","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Isabel","family":"Chien","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tim","family":"Regan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angel","family":"Enrique","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jorge","family":"Palacios","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dessie","family":"Keegan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Usman","family":"Munir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryutaro","family":"Tanno","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hannah","family":"Richardson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aditya","family":"Nori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0871-4078","authenticated-orcid":true,"given":"Derek","family":"Richards","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9617-7008","authenticated-orcid":true,"given":"Gavin","family":"Doherty","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Danielle","family":"Belgrave","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anja","family":"Thieme","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2023,11,27]]},"reference":[{"key":"pone.0272685.ref001","volume-title":"Depression and other common mental disorders: global health estimates","author":"World Health Organization","year":"2017"},{"key":"pone.0272685.ref002","doi-asserted-by":"crossref","DOI":"10.1002\/9781118470886","volume-title":"Cognitive behavior therapy: Core principles for practice","author":"WT O\u2019Donohue","year":"2012"},{"issue":"2","key":"pone.0272685.ref003","doi-asserted-by":"crossref","first-page":"0","DOI":"10.4088\/JCP.18r12188","article-title":"Computer-assisted cognitive-behavior therapy for depression: a systematic review and meta-analysis","volume":"80","author":"JH Wright","year":"2019","journal-title":"The Journal of clinical psychiatry"},{"key":"pone.0272685.ref004","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/j.janxdis.2018.01.001","article-title":"Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis","volume":"55","author":"G Andrews","year":"2018","journal-title":"Journal of anxiety disorders"},{"issue":"2","key":"pone.0272685.ref005","doi-asserted-by":"crossref","first-page":"196","DOI":"10.1002\/da.23115","article-title":"The effectiveness of Internet\u2010delivered treatment for generalized anxiety disorder: An updated systematic review and meta\u2010analysis","volume":"38","author":"N Eilert","year":"2021","journal-title":"Depression and Anxiety"},{"issue":"4","key":"pone.0272685.ref006","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1145\/2775388","article-title":"Sustaining user engagement with behavior-change tools","volume":"22","author":"E. Karapanos","year":"2015","journal-title":"Interactions"},{"issue":"4","key":"pone.0272685.ref007","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1001\/jamapsychiatry.2020.4364","article-title":"Internet-based cognitive behavioral therapy for depression: a systematic review and individual patient data network meta-analysis","volume":"78","author":"E Karyotaki","year":"2021","journal-title":"JAMA psychiatry"},{"issue":"1","key":"pone.0272685.ref008","first-page":"27","article-title":"Integrating human support into behavioral intervention technologies: The efficiency model of support","volume":"24","author":"SM Schueller","year":"2017","journal-title":"Clinical Psychology: Science and Practice"},{"key":"pone.0272685.ref009","volume-title":"Child & Family Behavior Therapy","author":"SC Hayes","year":"2018"},{"issue":"1","key":"pone.0272685.ref010","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.beth.2012.10.001","article-title":"The relationship between use of CBT skills and depression treatment outcome: A theoretical and methodological review of the literature","volume":"44","author":"NE Hundt","year":"2013","journal-title":"Behavior Therapy"},{"key":"pone.0272685.ref011","doi-asserted-by":"crossref","DOI":"10.1037\/0000039-000","volume-title":"Feedback-informed treatment in clinical practice: Reaching for excellence","author":"DS Prescott","year":"2017"},{"issue":"2","key":"pone.0272685.ref012","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1037\/0022-006X.69.2.159","article-title":"Patient-focused research: Using patient outcome data to enhance treatment effects","volume":"69","author":"M. J. Lambert","year":"2001","journal-title":"Journal of consulting and clinical psychology"},{"issue":"6","key":"pone.0272685.ref013","doi-asserted-by":"crossref","first-page":"669","DOI":"10.1080\/10503307.2014.928756","article-title":"The effect of implementing the Outcome Questionnaire-45.2 feedback system in Norway: A multisite randomized clinical trial in a naturalistic setting","volume":"25","author":"I Amble","year":"2015","journal-title":"Psychotherapy Research"},{"key":"pone.0272685.ref014","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.brat.2017.09.011","article-title":"Improving the efficiency of psychological treatment using outcome feedback technology","volume":"99","author":"J Delgadillo","year":"2017","journal-title":"Behaviour Research and Therapy"},{"issue":"7","key":"pone.0272685.ref015","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1016\/S2215-0366(18)30162-7","article-title":"Feedback-informed treatment versus usual psychological treatment for depression and anxiety: a multisite, open-label, cluster randomised controlled trial","volume":"5","author":"J Delgadillo","year":"2018","journal-title":"The Lancet Psychiatry"},{"key":"pone.0272685.ref016","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1037\/a0019247","article-title":"Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system","volume":"78","author":"K. Shimokawa","year":"2010","journal-title":"Journal of Consulting and Clinical Psychology"},{"issue":"1","key":"pone.0272685.ref017","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1111\/bjc.12031","article-title":"Early changes, attrition, and dose-response in low intensity psychological interventions","volume":"53","author":"J Delgadillo","year":"2014","journal-title":"Br J Clin Psychol"},{"issue":"4","key":"pone.0272685.ref018","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1037\/pst0000246","article-title":"Ethical implications of routine outcomes monitoring for patients, psychotherapists, and mental health care systems","volume":"56","author":"HJ Muir","year":"2019","journal-title":"Psychotherapy"},{"issue":"8","key":"pone.0272685.ref019","doi-asserted-by":"crossref","first-page":"1349","DOI":"10.1016\/j.cpr.2011.09.003","article-title":"Relapse and recurrence prevention in depression: current research and future prospects","volume":"31","author":"S Beshai","year":"2011","journal-title":"Clinical Psychology Review"},{"key":"pone.0272685.ref020","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.brat.2017.04.006","article-title":"How durable is the effect of low intensity CBT for depression and anxiety? Remission and relapse in a longitudinal cohort study","volume":"94","author":"S Ali","year":"2017","journal-title":"Behaviour research and therapy"},{"issue":"4","key":"pone.0272685.ref021","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1037\/ccp0000462","article-title":"Predicting treatment failure in regular care Internet-Delivered Cognitive Behavior Therapy for depression and anxiety using only weekly symptom measures","volume":"88","author":"E Forsell","year":"2020","journal-title":"Journal of consulting and clinical psychology"},{"issue":"4","key":"pone.0272685.ref022","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1176\/appi.ajp.2018.18060699","article-title":"Proof of concept for an adaptive treatment strategy to prevent failures in internet-delivered CBT: a single-blind randomized clinical trial with insomnia patients","volume":"176","author":"E Forsell","year":"2019","journal-title":"American Journal of Psychiatry"},{"key":"pone.0272685.ref023","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.jad.2017.02.010","article-title":"A prognostic index (PI) as a moderator of outcomes in the treatment of depression: A proof of concept combining multiple variables to inform risk-stratified stepped care models","volume":"213","author":"L Lorenzo-Luaces","year":"2017","journal-title":"Journal of Affective Disorders"},{"issue":"4","key":"pone.0272685.ref024","doi-asserted-by":"crossref","first-page":"e231","DOI":"10.1016\/S2589-7500(21)00018-2","article-title":"Dynamic prediction of psychological treatment outcomes: development and validation of a prediction model using routinely collected symptom data","volume":"3","author":"C Bone","year":"2021","journal-title":"The Lancet Digital Health"},{"issue":"2","key":"pone.0272685.ref025","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1002\/wps.20882","article-title":"The promise of machine learning in predicting treatment outcomes in psychiatry","volume":"20","author":"AM Chekroud","year":"2021","journal-title":"World Psychiatry"},{"key":"pone.0272685.ref026","doi-asserted-by":"crossref","first-page":"103530","DOI":"10.1016\/j.brat.2019.103530","article-title":"Predicting cognitive behavioral therapy outcome in the outpatient sector based on clinical routine data: a machine learning approach","volume":"124","author":"K Hilbert","year":"2020","journal-title":"Behaviour research and therapy"},{"key":"pone.0272685.ref027","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12888-020-02655-4","article-title":"Predictors of remission from body dysmorphic disorder after internet-delivered cognitive behavior therapy: a machine learning approach","volume":"20","author":"O Flygare","year":"2020","journal-title":"BMC psychiatry"},{"issue":"1","key":"pone.0272685.ref028","doi-asserted-by":"crossref","first-page":"e1576","DOI":"10.1002\/mpr.1576","article-title":"Prediction of outcome in internet\u2010delivered cognitive behaviour therapy for paediatric obsessive\u2010compulsive disorder: A machine learning approach","volume":"27","author":"F Lenhard","year":"2018","journal-title":"International Journal of Methods in Psychiatric Research"},{"key":"pone.0272685.ref029","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1016\/j.invent.2017.08.003","article-title":"Predicting therapy success for treatment as usual and blended treatment in the domain of depression","volume":"12","author":"W van Breda","year":"2018","journal-title":"Internet interventions"},{"key":"pone.0272685.ref030","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","article-title":"Scalable and accurate deep learning with electronic health records","volume":"1","author":"A Rajkomar","year":"2018","journal-title":"npj Digital Med"},{"key":"pone.0272685.ref031","unstructured":"UAE. 2020. About IAPT and the History of the Programme: Stepped Care Model Information. Last retrieved 22nd June 2020 https:\/\/www.uea.ac.uk\/medicine\/departments\/psychological-sciences\/cognitive-behavioural-therapy-training\/-about-iapt-and-the-history-of-the-programme\/stepped-care-model-information"},{"key":"pone.0272685.ref032","unstructured":"Mental health support for nurses: free access to online support (2021) Last accessed 23 December 2021 https:\/\/rcni.com\/nursing-standard\/newsroom\/news\/mental-health-support-nurses-free-access-to-online-support-and-no-referral-needed-173521"},{"key":"pone.0272685.ref033","unstructured":"Online Behavioral Health Solutions for NHS | SilverCloud Health (2021) Last accessed 22 December 2021 https:\/\/www.silvercloudhealth.com\/uk\/online-behavioral-health-solutions-for-nhs"},{"issue":"12","key":"pone.0272685.ref034","doi-asserted-by":"crossref","first-page":"2011","DOI":"10.1093\/jamia\/ocaa088","article-title":"MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care","volume":"27","author":"T Hernandez-Boussard","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"10","key":"pone.0272685.ref035","doi-asserted-by":"crossref","first-page":"e006556","DOI":"10.1161\/CIRCOUTCOMES.120.006556","article-title":"Recommendations for Reporting Machine Learning Analyses in Clinical Research","volume":"13","author":"LM Stevens","year":"2020","journal-title":"Circ Cardiovasc Qual Outcomes"},{"issue":"9","key":"pone.0272685.ref036","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1046\/j.1525-1497.2001.016009606.x","article-title":"The PHQ\u20109: validity of a brief depression severity measure","volume":"16","author":"Kurt Kroenke","year":"2001","journal-title":"Journal of general internal medicine"},{"key":"pone.0272685.ref037","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1097\/MLR.0b013e318160d093","article-title":"Validation and standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the general population","author":"Bernd L\u00f6we","year":"2008","journal-title":"Medical care"},{"key":"pone.0272685.ref038","author":"Neil S. Jacobson","year":"1992","journal-title":"Clinical significance: a statistical approach to defining meaningful change in psychotherapy research"},{"key":"pone.0272685.ref039","volume-title":"Improving Access to Psychological Therapies Manual (updated)","author":"DM Clark","year":"2021"},{"key":"pone.0272685.ref040","unstructured":"Lipton, ZC, Berkowitz, J, Elkan C.2015. A Critical Review of Recurrent Neural Networks for Sequence Learning. arXiv preprint arXiv:1506.00019."},{"key":"pone.0272685.ref041","unstructured":"Lipton ZC, Kale DC, Elkan C, Wetzel R. Learning to diagnose with LSTM recurrent neural networks. arXiv 2015. arXiv preprint arXiv:1511.03677."},{"key":"pone.0272685.ref042","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1038\/s41398-022-02133-3","article-title":"Predicting remission after internet-delivered psychotherapy in patients with depression using machine learning and multi-modal data","volume":"12","author":"J Wallert","year":"2022","journal-title":"Translational Psychiatry"},{"issue":"4","key":"pone.0272685.ref043","doi-asserted-by":"crossref","first-page":"475","DOI":"10.1007\/s13748-019-00192-0","article-title":"Learning machines in Internet-delivered psychological treatment","volume":"8","author":"M Boman","year":"2019","journal-title":"Progress in Artificial Intelligence"},{"issue":"7","key":"pone.0272685.ref044","doi-asserted-by":"crossref","first-page":"e2010791","DOI":"10.1001\/jamanetworkopen.2020.10791","article-title":"A machine learning approach to understanding patterns of engagement with internet-delivered mental health interventions","volume":"3","author":"I Chien","year":"2020","journal-title":"JAMA network open"},{"issue":"8","key":"pone.0272685.ref045","doi-asserted-by":"crossref","first-page":"e12775","DOI":"10.2196\/12775","article-title":"Exploring the relationship between usage and outcomes of an internet-based intervention for individuals with depressive symptoms: secondary analysis of data from a randomized controlled trial","volume":"21","author":"A Enrique","year":"2019","journal-title":"Journal of medical Internet research"},{"issue":"2","key":"pone.0272685.ref046","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1037\/a0027762","article-title":"The Partners for Change Outcome Management System (PCOMS): The Heart and Soul of Change Project","volume":"53","author":"Barry L. Duncan","year":"2012","journal-title":"Canadian Psychology\/Psychologie canadienne"},{"key":"pone.0272685.ref047","doi-asserted-by":"crossref","first-page":"894","DOI":"10.1037\/0022-006X.67.6.894","article-title":"Sudden gains and critical sessions in cognitive-behavioral therapy for depression","volume":"67","author":"T. Z. Tang","year":"1999","journal-title":"Journal of Consulting and Clinical Psychology"},{"issue":"5","key":"pone.0272685.ref048","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3398069","article-title":"Machine learning in mental health: A systematic review of the HCI literature to support the development of effective and implementable ML systems","volume":"27","author":"A Thieme","year":"2020","journal-title":"ACM Transactions on Computer-Human Interaction (TOCHI)"},{"issue":"27","key":"pone.0272685.ref049","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3564752","article-title":"2023. Designing Human-Centered AI for Mental Health: Developing Clinically Relevant Applications for Online CBT Treatment","volume":"2","author":"A Thieme","year":"2023","journal-title":"ACM Transactions on Computer-Human Interaction (TOCHI)"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0272685","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T19:09:27Z","timestamp":1701112167000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0272685"}},"subtitle":[],"editor":[{"given":"Hugh","family":"Cowley","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2023,11,27]]},"references-count":49,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,11,27]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0272685","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,27]]}}}