{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T04:33:02Z","timestamp":1784608382228,"version":"3.55.0"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2018,4,6]],"date-time":"2018-04-06T00:00:00Z","timestamp":1522972800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/academic.oup.com\/journals\/pages\/about_us\/legal\/notices"}],"funder":[{"DOI":"10.13039\/100000002","name":"US National Institutes of Health","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","award":["R01MH109496"],"award-info":[{"award-number":["R01MH109496"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","award":["R01MH100482"],"award-info":[{"award-number":["R01MH100482"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>While depression and anxiety are common mental health issues, only a small segment of the population has access to standard one-on-one treatment. The use of smartphone apps can fill this gap. An app recommender system may help improve user engagement of these apps and eventually symptoms.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>IntelliCare was a suite of apps for depression and anxiety, with a Hub app that provided app recommendations aiming to increase user engagement. This study captured the records of 8057 users of 12 apps. We measured overall engagement and app-specific usage longitudinally by the number of weekly app sessions (\u201cloyalty\u201d) and the number of days with app usage (\u201cregularity\u201d) over 16 weeks. Hub and non-Hub users were compared using zero-inflated Poisson regression for loyalty, linear regression for regularity, and Cox regression for engagement duration. Adjusted analyses were performed in 4561 users for whom we had baseline characteristics. Impact of Hub recommendations was assessed using the same approach.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>When compared to non-Hub users in adjusted analyses, Hub users had a lower risk of discontinuing IntelliCare (hazard ratio\u2009=\u20090.67, 95% CI, 0.62-0.71), higher loyalty (2- to 5-fold), and higher regularity (0.1\u20130.4 day\/week greater). Among Hub users, Hub recommendations increased app-specific loyalty and regularity in all 12 apps.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion\/Conclusion<\/jats:title><jats:p>Centralized app recommendations increase overall user engagement of the apps, as well as app-specific usage. Further studies relating app usage to symptoms can validate that such a recommender improves clinical benefits and does so at scale.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocy023","type":"journal-article","created":{"date-parts":[[2018,2,26]],"date-time":"2018-02-26T20:17:15Z","timestamp":1519676235000},"page":"955-962","source":"Crossref","is-referenced-by-count":66,"title":["Evaluation of a recommender app for apps for the treatment of depression and anxiety: an analysis of longitudinal user engagement"],"prefix":"10.1093","volume":"25","author":[{"given":"Ken","family":"Cheung","sequence":"first","affiliation":[{"name":"Department of Biostatistics, Columbia University Mailman School of Public Health, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wodan","family":"Ling","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Columbia University Mailman School of Public Health, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris J","family":"Karr","sequence":"additional","affiliation":[{"name":"Audacious software, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kenneth","family":"Weingardt","sequence":"additional","affiliation":[{"name":"Center for Behavioral Intervention Technologies (CBITs), Department of Preventive Medicine, Northwestern University, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen M","family":"Schueller","sequence":"additional","affiliation":[{"name":"Center for Behavioral Intervention Technologies (CBITs), Department of Preventive Medicine, Northwestern University, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David C","family":"Mohr","sequence":"additional","affiliation":[{"name":"Center for Behavioral Intervention Technologies (CBITs), Department of Preventive Medicine, Northwestern University, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2018,4,6]]},"reference":[{"issue":"3","key":"2020110612223653600_ocy023-B1","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1207\/s15324796abm3203_12","article-title":"Barriers to psychotherapy among depressed and nondepressed primary care patients","volume":"32","author":"Mohr","year":"2006","journal-title":"Ann Behav Med"},{"issue":"4","key":"2020110612223653600_ocy023-B2","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1002\/jclp.20659","article-title":"Perceived barriers to psychological treatments and their relationship to depression","volume":"66","author":"Mohr","year":"2010","journal-title":"J Clin Psychol"},{"key":"2020110612223653600_ocy023-B3","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1177\/1745691610393527","article-title":"Rebooting psychotherapy research and practice to reduce the burden of mental illness","volume":"6","author":"Kazdin","year":"2011","journal-title":"Perspect Psychol Sci"},{"issue":"4","key":"2020110612223653600_ocy023-B4","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1001\/jamapsychiatry.2017.0044","article-title":"Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data","volume":"74","author":"Karyotaki","year":"2017","journal-title":"JAMA Psychiatry"},{"issue":"4","key":"2020110612223653600_ocy023-B5","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1016\/j.cpr.2012.02.004","article-title":"Computer-based psychological treatments for depression: a systematic review and meta-analysis","volume":"32","author":"Richards","year":"2012","journal-title":"Clin Psychol Rev"},{"key":"2020110612223653600_ocy023-B6","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1016\/j.invent.2015.07.003","article-title":"The efficacy of internet-delivered treatment for generalized anxiety disorder: a systematic review and meta-analysis","volume":"2","author":"Richards","year":"2015","journal-title":"Internet Interv"},{"issue":"3","key":"2020110612223653600_ocy023-B7","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1002\/wps.20472","article-title":"The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials","volume":"16","author":"Firth","year":"2017","journal-title":"World Psychiatry"},{"key":"2020110612223653600_ocy023-B8","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.jad.2017.04.046","article-title":"Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials","volume":"218","author":"Firth","year":"2017","journal-title":"J Affect Disord"},{"key":"2020110612223653600_ocy023-B9","doi-asserted-by":"crossref","DOI":"10.1145\/1054972.1055101","article-title":"Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI","volume-title":"SIGCHI Conference on Human Factors in Computing Systems","author":"Oulasvirta","year":"2005"},{"key":"2020110612223653600_ocy023-B10","doi-asserted-by":"crossref","DOI":"10.1145\/2556288.2556996","article-title":"Twitch crowdsourcing: crowd contributions in short bursts of time","volume-title":"SIGCHI Conference on Human Factors in Computing Systems","author":"Vaish","year":"2014"},{"issue":"10","key":"2020110612223653600_ocy023-B11","doi-asserted-by":"crossref","first-page":"e0139004","DOI":"10.1371\/journal.pone.0139004","article-title":"Beyond self-report: tools to compare estimated and real-world smartphone use","volume":"10","author":"Andrews","year":"2015","journal-title":"PLoS One"},{"key":"2020110612223653600_ocy023-B12","volume-title":"Psychosocial Interventions for Mental and Substance Use Disorders: A Framework for Establishing Evidence-Based Standards","author":"Institute of Medicine (IOM)","year":"2015"},{"issue":"6","key":"2020110612223653600_ocy023-B13","doi-asserted-by":"crossref","first-page":"e146","DOI":"10.2196\/jmir.3077","article-title":"The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions","volume":"16","author":"Mohr","year":"2014","journal-title":"J Med Internet Res"},{"key":"2020110612223653600_ocy023-B14","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.invent.2016.06.003","article-title":"Uptake and usage of IntelliCare: a publicly available suite of mental health and well-being apps","volume":"4","author":"Lattie","year":"2016","journal-title":"Internet Interv"},{"issue":"3","key":"2020110612223653600_ocy023-B15","doi-asserted-by":"crossref","first-page":"e18","DOI":"10.2196\/humanfactors.7951","article-title":"Exploring user learnability and learning performance in an app for depression: usability study","volume":"4","author":"Stiles-Shields","year":"2017","journal-title":"JMIR Human Factors"},{"issue":"1","key":"2020110612223653600_ocy023-B16","doi-asserted-by":"crossref","first-page":"e10","DOI":"10.2196\/jmir.6645","article-title":"IntelliCare: an eclectic, skills-based app suite for the treatment of depression and anxiety","volume":"19","author":"Mohr","year":"2017","journal-title":"J Med Internet Res"},{"key":"2020110612223653600_ocy023-B17","volume-title":"Measuring the Immeasurable: Visitor Engagement","author":"Peterson"},{"key":"2020110612223653600_ocy023-B18","doi-asserted-by":"crossref","first-page":"e89","DOI":"10.2196\/mhealth.7236","article-title":"Assessing user engagement of an mHealth intervention: development and implementation of the growing healthy app engagement index","volume":"5","author":"Taki","year":"2017","journal-title":"JMIR mHealth and uHealth"},{"key":"2020110612223653600_ocy023-B19","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511814365","volume-title":"Regression Analysis of Count Data","author":"Cameron","year":"1998"},{"key":"2020110612223653600_ocy023-B20","volume-title":"The Inevitable","author":"Kelly","year":"2016"},{"issue":"1","key":"2020110612223653600_ocy023-B21","doi-asserted-by":"crossref","first-page":"e7","DOI":"10.2196\/mental.3935","article-title":"mHealth in the wild: using novel data to examine the reach, use, and impact of PTSD coach","volume":"2","author":"Owen","year":"2015","journal-title":"JMIR Ment Health"},{"issue":"4","key":"2020110612223653600_ocy023-B22","doi-asserted-by":"crossref","first-page":"e109","DOI":"10.2196\/jmir.3084","article-title":"Factors related to sustained use of a free mobile app for dietary self-monitoring with photography and peer feedback: retrospective cohort study","volume":"16","author":"Helander","year":"2014","journal-title":"J Med Internet Res"},{"key":"2020110612223653600_ocy023-B23","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.brat.2015.06.014","article-title":"Comparing human and automated support for depression: fractional factorial randomized controlled trial","volume":"72","author":"Kelders","year":"2015","journal-title":"Behav Res Ther"},{"key":"2020110612223653600_ocy023-B24","first-page":"3063","article-title":"How to evaluate technologies for health behavior change in HCI research","author":"Klasnja"},{"key":"2020110612223653600_ocy023-B25","doi-asserted-by":"crossref","first-page":"e52","DOI":"10.2196\/jmir.1772","article-title":"A systematic review of the impact of adherence on the effectiveness of e-therapies","volume":"13","author":"Donkin","year":"2011","journal-title":"J Med Internet Res"},{"issue":"3","key":"2020110612223653600_ocy023-B26","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1177\/1090198113486805","article-title":"What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation","volume":"40","author":"Glasgow","year":"2013","journal-title":"Health Educ Behav"},{"key":"2020110612223653600_ocy023-B27","first-page":"S902","article-title":"Effects of contextually-tailored suggestions for physical activity: the HeartSteps Micro-Randomized Trial","volume":"51","author":"Klasnja","year":"2017","journal-title":"Ann Behav Med"},{"key":"2020110612223653600_ocy023-B28","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"1998"},{"key":"2020110612223653600_ocy023-B29","article-title":"A first step towards behavioral coaching for managing stress: a case study on optimal policy estimation with multi-stage threshold Q-learning","author":"Hu"},{"issue":"4","key":"2020110612223653600_ocy023-B30","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1016\/j.amepre.2013.06.006","article-title":"Continuous evaluation of evolving behavioral intervention technologies","volume":"45","author":"Mohr","year":"2013","journal-title":"Am J Prev Med"},{"issue":"2","key":"2020110612223653600_ocy023-B31","doi-asserted-by":"crossref","first-page":"450","DOI":"10.1111\/biom.12258","article-title":"Sequential multiple assignment randomized trial (SMART) with adaptive randomization for quality improvement in depression treatment programs","volume":"71","author":"Cheung","year":"2015","journal-title":"Biometrics"},{"issue":"5","key":"2020110612223653600_ocy023-B32","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1001\/jamapsychiatry.2017.0262","article-title":"Needed innovation in digital health and smartphone applications for mental health: transparency and trust","volume":"74","author":"Torous","year":"2017","journal-title":"JAMA Psychiatry"},{"key":"2020110612223653600_ocy023-B33","doi-asserted-by":"crossref","first-page":"526","DOI":"10.1002\/da.22657","article-title":"There is an app for that! The current state of mobile applications (apps) for DSM-5 obsessive-compulsive disorder, posttraumatic stress disorder, anxiety and mood disorders","volume":"34","author":"Ameringen","year":"2017","journal-title":"Depress Anxiety"},{"issue":"11","key":"2020110612223653600_ocy023-B34","first-page":"S40","article-title":"Beacon: a web portal to high-quality mental health websites for use by health professionals and the public","volume":"192","author":"Christensen","year":"2010","journal-title":"Med J Aust"},{"issue":"5","key":"2020110612223653600_ocy023-B35","doi-asserted-by":"crossref","first-page":"e141","DOI":"10.2196\/jmir.6775","article-title":"Efficacy of a web-based guided recommendation service for a curated list of readily available mental health and well-being mobile apps for young people: randomized controlled trial","volume":"19","author":"Bidargaddi","year":"2017","journal-title":"J Med Internet Res"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/955\/34150491\/ocy023.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/955\/34150491\/ocy023.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T15:30:57Z","timestamp":1719847857000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/8\/955\/4963735"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,4,6]]},"references-count":35,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2018,4,6]]},"published-print":{"date-parts":[[2018,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy023","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,8]]},"published":{"date-parts":[[2018,4,6]]}}}