{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T14:13:44Z","timestamp":1777299224645,"version":"3.51.4"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2021,11,22]],"date-time":"2021-11-22T00:00:00Z","timestamp":1637539200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000062","name":"National Institute of Diabetes and Digestive and Kidney Diseases","doi-asserted-by":"publisher","award":["R01-DK100694"],"award-info":[{"award-number":["R01-DK100694"]}],"id":[{"id":"10.13039\/100000062","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Center for Diabetes Translation Research Pilot & Feasibility Award","award":["P30-DK092986"],"award-info":[{"award-number":["P30-DK092986"]}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["K12-HL137943"],"award-info":[{"award-number":["K12-HL137943"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,12,28]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Guidance is needed on studying engagement and treatment effects in digital health interventions, including levels required for benefit. We evaluated multiple analytic approaches for understanding the association between engagement and clinical outcomes.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We defined engagement as intervention participants\u2019 response rate to interactive text messages, and considered moderation, standard regression, mediation, and a modified instrumental variable (IV) analysis to investigate the relationship between engagement and clinical outcomes. We applied each approach to two randomized controlled trials featuring text message content in the intervention: REACH (Rapid Encouragement\/Education and Communications for Health), which targeted diabetes, and VERB (Vanderbilt Emergency Room Bundle), which targeted hypertension.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>In REACH, the treatment effect on hemoglobin A1c was estimated to be \u22120.73% (95% CI: [\u22121.29, \u22120.21]; P\u2009=\u20090.008), and in VERB, the treatment effect on systolic blood pressure was estimated to be \u221210.1 mmHg (95% CI: [\u221217.7, \u22122.8]; P\u2009=\u20090.007). Only the IV analyses suggested an effect of engagement on outcomes; the difference in treatment effects between engagers and non-engagers was \u22120.29% to \u22120.51% in the REACH study and \u22121.08 to \u22123.25 mmHg in the VERB study.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Standard regression and mediation have less power than a modified IV analysis, but the IV approach requires specification of assumptions. This is the first review of the strengths and limitations of various approaches to evaluating the impact of engagement on outcomes.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Understanding the role of engagement in digital health interventions can help reveal when and how these interventions achieve desired outcomes.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab254","type":"journal-article","created":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T20:12:37Z","timestamp":1635797557000},"page":"128-136","source":"Crossref","is-referenced-by-count":19,"title":["Estimating the impact of engagement with digital health interventions on patient outcomes in randomized trials"],"prefix":"10.1093","volume":"29","author":[{"given":"Lyndsay A","family":"Nelson","sequence":"first","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Center for Health Behavior and Health Education, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}]},{"given":"Andrew J","family":"Spieker","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University, Nashville, Tennessee, USA"}]},{"given":"Lindsay S","family":"Mayberry","sequence":"additional","affiliation":[{"name":"Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Center for Health Behavior and Health Education, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Center for Diabetes Translation Research, Vanderbilt University Medical Center, Nashville, Tennessee, USA"}]},{"given":"Candace","family":"McNaughton","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA"},{"name":"Geriatric Research Education Clinical Center, Tennessee Valley Healthcare System VA Medical Center, Nashville, Tennessee, USA"}]},{"given":"Robert A","family":"Greevy","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University, Nashville, Tennessee, USA"}]}],"member":"286","published-online":{"date-parts":[[2021,11,22]]},"reference":[{"issue":"01","key":"2021122823084053600_ocab254-B1","doi-asserted-by":"crossref","first-page":"93","DOI":"10.15265\/IY-2016-020","article-title":"Recent worldwide developments in eHealth and mHealth to more effectively manage cancer and other chronic diseases - a systematic review","volume":"25","author":"Lewis","year":"2016","journal-title":"Yearb Med Inform"},{"issue":"6","key":"2021122823084053600_ocab254-B2","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1093\/jamia\/ocaa029","article-title":"A systematic review of mobile health technologies to support self-management of concurrent diabetes and hypertension","volume":"27","author":"Choi","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2021122823084053600_ocab254-B3","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1177\/1932296817713506","article-title":"A systematic review of reviews evaluating technology-enabled diabetes self-management education and support","volume":"11","author":"Greenwood","year":"2017","journal-title":"J Diabetes Sci Technol"},{"issue":"1","key":"2021122823084053600_ocab254-B4","doi-asserted-by":"crossref","first-page":"e23","DOI":"10.2196\/mhealth.8873","article-title":"The impact of mHealth interventions: systematic review of systematic reviews","volume":"6","author":"Marcolino","year":"2018","journal-title":"JMIR Mhealth Uhealth"},{"issue":"2","key":"2021122823084053600_ocab254-B5","doi-asserted-by":"crossref","first-page":"e52","DOI":"10.2196\/jmir.3951","article-title":"Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review","volume":"17","author":"Hamine","year":"2015","journal-title":"J Med Internet Res"},{"issue":"5","key":"2021122823084053600_ocab254-B6","doi-asserted-by":"crossref","first-page":"843","DOI":"10.1016\/j.amepre.2016.06.008","article-title":"Evaluating digital health interventions: key questions and approaches","volume":"51","author":"Murray","year":"2016","journal-title":"Am J Prev Med"},{"issue":"4","key":"2021122823084053600_ocab254-B7","doi-asserted-by":"crossref","first-page":"e14052","DOI":"10.2196\/14052","article-title":"Understanding health behavior technology engagement: pathway to measuring digital behavior change interventions","volume":"3","author":"Cole-Lewis","year":"2019","journal-title":"JMIR Form Res"},{"issue":"11","key":"2021122823084053600_ocab254-B8","doi-asserted-by":"crossref","first-page":"e292","DOI":"10.2196\/jmir.9397","article-title":"Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies","volume":"20","author":"Short","year":"2018","journal-title":"J Med Internet Res"},{"issue":"2","key":"2021122823084053600_ocab254-B9","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1007\/s13142-016-0453-1","article-title":"Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis","volume":"7","author":"Perski","year":"2017","journal-title":"Transl Behav Med"},{"issue":"2","key":"2021122823084053600_ocab254-B10","doi-asserted-by":"crossref","first-page":"e28","DOI":"10.2196\/jmir.6382","article-title":"Barriers to remote health interventions for type 2 diabetes: a systematic review and proposed classification scheme","volume":"19","author":"Alvarado","year":"2017","journal-title":"J Med Internet Res"},{"issue":"7","key":"2021122823084053600_ocab254-B11","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1007\/s11892-016-0755-1","article-title":"Patterns of user engagement with mobile- and web-delivered self-care interventions for adults with t2dm: a review of the literature","volume":"16","author":"Nelson","year":"2016","journal-title":"Curr Diab Rep"},{"issue":"5","key":"2021122823084053600_ocab254-B12","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1093\/jamia\/ocaa024","article-title":"Patient characteristics associated with objective measures of digital health tool use in the united states: a literature review","volume":"27","author":"Nouri","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"7","key":"2021122823084053600_ocab254-B13","doi-asserted-by":"crossref","first-page":"e10480","DOI":"10.2196\/10480","article-title":"Barriers to and facilitators of engagement with remote measurement technology for managing health: Systematic review and content analysis of findings","volume":"20","author":"Simblett","year":"2018","journal-title":"J Med Internet Res"},{"issue":"5","key":"2021122823084053600_ocab254-B14","doi-asserted-by":"crossref","first-page":"833","DOI":"10.1016\/j.amepre.2016.06.015","article-title":"Understanding and promoting effective engagement with digital behavior change interventions","volume":"51","author":"Yardley","year":"2016","journal-title":"Am J Prev Med"},{"issue":"1","key":"2021122823084053600_ocab254-B15","doi-asserted-by":"crossref","first-page":"e6","DOI":"10.2196\/jmir.4790","article-title":"The effectiveness of prompts to promote engagement with digital interventions: a systematic review","volume":"18","author":"Alkhaldi","year":"2016","journal-title":"J Med Internet Res"},{"issue":"5","key":"2021122823084053600_ocab254-B16","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1089\/dia.2018.0015","article-title":"Engagement with a text-messaging intervention improves adherence in adolescents with type 1 diabetes: brief report","volume":"20","author":"Zhang","year":"2018","journal-title":"Diabetes Technol Ther"},{"key":"2021122823084053600_ocab254-B17","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.jsat.2018.11.001","article-title":"Treatment engagement mediates the links between symptoms of anxiety, depression, and alcohol use disorder with abstinence among smokers registered on an internet cessation program","volume":"98","author":"Cohn","year":"2019","journal-title":"J Subst Abuse Treat"},{"issue":"11","key":"2021122823084053600_ocab254-B18","doi-asserted-by":"crossref","first-page":"e14645","DOI":"10.2196\/14645","article-title":"User engagement and attrition in an app-based physical activity intervention: secondary analysis of a randomized controlled trial","volume":"21","author":"Edney","year":"2019","journal-title":"J Med Internet Res"},{"issue":"2","key":"2021122823084053600_ocab254-B19","doi-asserted-by":"crossref","first-page":"e21","DOI":"10.2196\/diabetes.8039","article-title":"One drop | mobile: an evaluation of hemoglobin a1c improvement linked to app engagement","volume":"2","author":"Osborn","year":"2017","journal-title":"JMIR Diabetes"},{"issue":"1","key":"2021122823084053600_ocab254-B20","doi-asserted-by":"crossref","first-page":"e14","DOI":"10.2196\/jmir.2277","article-title":"Engagement promotes abstinence in a web-based cessation intervention: cohort study","volume":"15","author":"Richardson","year":"2013","journal-title":"J Med Internet Res"},{"issue":"2","key":"2021122823084053600_ocab254-B21","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1177\/193229681200600214","article-title":"Mobile phone-based video messages for diabetes self-care support","volume":"6","author":"Bell","year":"2012","journal-title":"J Diabetes Sci Technol"},{"issue":"1","key":"2021122823084053600_ocab254-B22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/1559325816637515","article-title":"Intervention engagement moderates the dose-response relationships in a dietary intervention","volume":"14","author":"Lippke","year":"2016","journal-title":"Dose Response"},{"issue":"4","key":"2021122823084053600_ocab254-B23","doi-asserted-by":"crossref","first-page":"666","DOI":"10.1111\/j.1083-6101.2009.01514.x","article-title":"The assessment of user engagement with eHealth content: the eHealth engagement scale","volume":"15","author":"Craig Lefebvre","year":"2010","journal-title":"J Comput Mediat Commun"},{"key":"2021122823084053600_ocab254-B24","doi-asserted-by":"crossref","first-page":"37","DOI":"10.21037\/mhealth.2018.08.04","article-title":"If we build it, will they come? Issues of engagement with digital health interventions for trauma recovery","volume":"4","author":"Yeager","year":"2018","journal-title":"mHealth"},{"issue":"4","key":"2021122823084053600_ocab254-B25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3287031","article-title":"Is more always better? Discovering incentivized mHealth intervention engagement related to health behavior trends","volume":"2","author":"Alshurafa","year":"2018","journal-title":"Proc ACM Interact Mob Wearable Ubiquitous Technol"},{"issue":"5","key":"2021122823084053600_ocab254-B26","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1007\/s10865-007-9110-y","article-title":"Psychometric properties of the perceived diabetes self-management scale (PDSMS)","volume":"30","author":"Wallston","year":"2007","journal-title":"J Behav Med"},{"issue":"2","key":"2021122823084053600_ocab254-B27","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.diabres.2013.09.010","article-title":"The ARMS-D out performs the SDSCA, but both are reliable, valid, and predict glycemic control","volume":"102","author":"Mayberry","year":"2013","journal-title":"Diabetes Res Clin Pract"},{"key":"2021122823084053600_ocab254-B28","first-page":"26","article-title":"Effects of a tailored text messaging intervention among diverse adults with type 2 diabetes: evidence from the 15-month REACH randomized controlled trial","author":"Nelson","year":"2020","journal-title":"Diabetes Care"},{"key":"2021122823084053600_ocab254-B29","article-title":"User preferences for and engagement with text messages to support antihypertensive medication adherence: findings from a pilot study evaluating an emergency department-based behavioral intervention","author":"Nelson","journal-title":"Patient Educ Couns"},{"issue":"12","key":"2021122823084053600_ocab254-B30","doi-asserted-by":"crossref","first-page":"2071","DOI":"10.1016\/j.pec.2016.07.020","article-title":"Perceived health competence predicts health behavior and health-related quality of life in patients with cardiovascular disease","volume":"99","author":"Bachmann","year":"2016","journal-title":"Patient Educ Couns"},{"issue":"1","key":"2021122823084053600_ocab254-B31","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1111\/j.1524-4733.2008.00400.x","article-title":"Development and evaluation of the adherence to refills and medications scale (ARMS) among low-literacy patients with chronic disease","volume":"12","author":"Kripalani","year":"2009","journal-title":"Value Health"},{"issue":"5","key":"2021122823084053600_ocab254-B32","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1177\/0193841X8100500502","article-title":"Process analysis: estimating mediation in treatment evaluations","volume":"5","author":"Judd","year":"1981","journal-title":"Eval Rev"},{"key":"2021122823084053600_ocab254-B33","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139025751","volume-title":"Causal Inference in Statistics, Social, and Biomedical Sciences","author":"Imbens","year":"2015"},{"issue":"434","key":"2021122823084053600_ocab254-B34","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1080\/01621459.1996.10476902","article-title":"Identification of causal effects using instrumental variables","volume":"91","author":"Angrist","year":"1996","journal-title":"J Am Stat Assoc"},{"issue":"13","key":"2021122823084053600_ocab254-B35","doi-asserted-by":"crossref","first-page":"2297","DOI":"10.1002\/sim.6128","article-title":"Instrumental variable methods for causal inference","volume":"33","author":"Baiocchi","year":"2014","journal-title":"Stat Med"},{"key":"2021122823084053600_ocab254-B36","article-title":"Bounding the local average treatment effect in an instrumental variable analysis of engagement with a mobile intervention","author":"Spieker","journal-title":"Annals of Applied Statistics"},{"issue":"14","key":"2021122823084053600_ocab254-B37","doi-asserted-by":"crossref","first-page":"2252","DOI":"10.1002\/sim.7654","article-title":"Bootstrap inference when using multiple imputation","volume":"37","author":"Schomaker","year":"2018","journal-title":"Stat Med"},{"issue":"12","key":"2021122823084053600_ocab254-B38","doi-asserted-by":"crossref","first-page":"e16630","DOI":"10.2196\/16630","article-title":"Old-fashioned technology in the era of \u201cbling\u201d: is there a future for text messaging in health care?","volume":"21","author":"Willcox","year":"2019","journal-title":"J Med Internet Res"},{"issue":"2","key":"2021122823084053600_ocab254-B39","doi-asserted-by":"crossref","first-page":"e40","DOI":"10.2196\/jmir.8439","article-title":"Digital health interventions for adults with type 2 diabetes: qualitative study of patient perspectives on diabetes self-management education and support","volume":"20","author":"Pal","year":"2018","journal-title":"J Med Internet Res"},{"issue":"3","key":"2021122823084053600_ocab254-B40","doi-asserted-by":"crossref","first-page":"e52","DOI":"10.2196\/jmir.1772","article-title":"A systematic review of the impact of adherence on the effectiveness of e-therapies","volume":"13","author":"Donkin","year":"2011","journal-title":"J Med Internet Res"},{"issue":"12","key":"2021122823084053600_ocab254-B41","doi-asserted-by":"crossref","first-page":"e21687","DOI":"10.2196\/21687","article-title":"Design features for improving mobile health intervention user engagement: systematic review and thematic analysis","volume":"22","author":"Wei","year":"2020","journal-title":"J Med Internet Res"},{"issue":"2","key":"2021122823084053600_ocab254-B42","doi-asserted-by":"crossref","first-page":"e31","DOI":"10.2196\/mhealth.9265","article-title":"Mobile diabetes intervention study of patient engagement and impact on blood glucose: mixed methods analysis","volume":"6","author":"Quinn","year":"2018","journal-title":"JMIR Mhealth Uhealth"},{"issue":"2","key":"2021122823084053600_ocab254-B43","doi-asserted-by":"crossref","first-page":"e10966","DOI":"10.2196\/10966","article-title":"A framework for analyzing and measuring usage and engagement data (amused) in digital interventions: viewpoint","volume":"21","author":"Miller","year":"2019","journal-title":"J Med Internet Res"},{"issue":"11","key":"2021122823084053600_ocab254-B44","doi-asserted-by":"crossref","first-page":"e16197","DOI":"10.2196\/16197","article-title":"Assessing the psychometric properties of the digital behavior change intervention engagement scale in users of an app for reducing alcohol consumption: Evaluation study","volume":"21","author":"Perski","year":"2019","journal-title":"J Med Internet Res"},{"issue":"3","key":"2021122823084053600_ocab254-B45","doi-asserted-by":"crossref","first-page":"e17776","DOI":"10.2196\/17776","article-title":"The effectiveness of self-management of hypertension in adults using mobile health: systematic review and meta-analysis","volume":"8","author":"Li","year":"2020","journal-title":"JMIR Mhealth Uhealth"},{"issue":"8","key":"2021122823084053600_ocab254-B46","doi-asserted-by":"crossref","first-page":"e15779","DOI":"10.2196\/15779","article-title":"Effectiveness of mobile app-assisted self-care interventions for improving patient outcomes in type 2 diabetes and\/or hypertension: systematic review and meta-analysis of randomized controlled trials","volume":"8","author":"Liu","year":"2020","journal-title":"JMIR Mhealth Uhealth"},{"issue":"11","key":"2021122823084053600_ocab254-B47","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1089\/dia.2018.0216","article-title":"Effectiveness of digital interventions for improving glycemic control in persons with poorly controlled type 2 diabetes: a systematic review, meta-analysis, and meta-regression analysis","volume":"20","author":"Kebede","year":"2018","journal-title":"Diabetes Technol Ther"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/1\/128\/41955618\/ocab254.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/29\/1\/128\/41955618\/ocab254.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,28]],"date-time":"2021-12-28T23:14:51Z","timestamp":1640733291000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/29\/1\/128\/6433096"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,22]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,11,22]]},"published-print":{"date-parts":[[2021,12,28]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab254","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,1,1]]},"published":{"date-parts":[[2021,11,22]]}}}