{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:03:29Z","timestamp":1782846209818,"version":"3.54.5"},"reference-count":42,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Evaluating implementation of digital health interventions (DHIs) in practice settings is complex, involving diverse users and multistep processes. Proactive planning can ensure implementation determinants and outcomes are captured for hybrid studies, but operational guidance for designing or planning hybrid DHI studies is limited.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to proactively define, prioritize, and operationalize measurement of implementation outcomes and determinants for a DHI hybrid effectiveness-implementation trial. We describe unique advantages and limitations of planning the trial implementation evaluation among a large-scale cohort study population and share results of a pretrial organizational readiness assessment.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We planned a cluster-randomized, type II hybrid effectiveness-implementation trial testing\n PositiveLinks<\/jats:italic>\n , a smartphone app for HIV care, compared to usual care (n=6 sites per arm), among HIV outpatient sites in the DC Cohort Longitudinal HIV Study in Washington, DC. We (1) defined components of the DHI and associated implementation strategy; (2) selected implementation science frameworks to accomplish evaluation aims; (3) mapped framework dimensions, domains, and constructs to implementation strategy steps; (4) modified or created instruments to collect data for implementation outcome measures and determinants; and (5) developed a compatible implementation science data collection and management plan. Provider baseline surveys administered at intervention sites probed usage of digital tools and assessed provider readiness for implementation with the Organizational Readiness to Implement Change tool.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n We specified DHI and implementation strategy toward planning measurement of DHI and broader program reach and adoption. Mapping of implementation strategy steps to the Reach Effectiveness Adoption Implementation Maintenance framework prompted considerations for how to capture understudied aspects of each dimension: denominators and demographic representativeness within reach or adoption, and provider or organization-level adaptations, dose, and fidelity within the implementation dimension. Our process also prompted the creation of tools to obtain detailed determinants across domains and constructs of the Consolidated Framework for Implementation Research within a large sample at multiple time points. Some aspects of real-world\n PositiveLinks<\/jats:italic>\n implementation were not reflected within the planned hybrid trial (eg, research assistants selected as de facto site implementation leads) or were modified to preserve internal validity of effectiveness measurement (eg, \u201cCommunity of Practice\u201d). Providers and research assistants (n=17) at intervention sites self-reported high baseline use of digital tools to communicate with patients. Readiness assessment revealed high median (48, IQR 45\u201054) total Organizational Readiness to Implement Change scores, with research assistants scoring higher than physicians (52.5, IQR 44-55 vs 48.0, IQR 46-49).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Key takeaways, challenges, and opportunities arose in planning the implementation evaluation within a hybrid DHI trial among a cohort population. Prospective trial planning must balance generalizability of implementation processes to \u201creal world\u201d conditions with rigorous procedures to measure intervention effectiveness. Rapid, scalable tools require further study to enable evaluations within large multisite hybrid studies.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/76327","type":"journal-article","created":{"date-parts":[[2025,11,30]],"date-time":"2025-11-30T19:55:06Z","timestamp":1764532506000},"page":"e76327-e76327","source":"Crossref","is-referenced-by-count":0,"title":["A Complex Digital Health Intervention to Support People With HIV: Organizational Readiness Survey Study and Preimplementation Planning for a Hybrid Effectiveness-Implementation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9879-7502","authenticated-orcid":false,"given":"Jacqueline","family":"Hodges","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4351-8497","authenticated-orcid":false,"given":"Wendy","family":"Cohn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9095-2413","authenticated-orcid":false,"given":"Amanda","family":"Castel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8820-2536","authenticated-orcid":false,"given":"Tabor","family":"Flickinger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7784-3159","authenticated-orcid":false,"given":"Ava Lena","family":"Waldman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9069-8788","authenticated-orcid":false,"given":"Michelle","family":"Hilgart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6663-065X","authenticated-orcid":false,"given":"Olivia","family":"Kirby","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3222-9355","authenticated-orcid":false,"given":"Sylvia","family":"Caldwell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8115-2548","authenticated-orcid":false,"given":"Karen","family":"Ingersoll","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,1,21]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1111\/cpsp.12174","article-title":"The efficiency model of support and the creation of digital apothecaries","volume":"24","author":"Mu\u00f1oz","journal-title":"Clin Psychol Sci Pract"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.2196\/11752","article-title":"Measuring the implementation of behavioral intervention technologies: recharacterization of established outcomes","volume":"21","author":"Hermes","journal-title":"J Med Internet Res"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1186\/s43058-020-00001-z","article-title":"Implementation science made too simple: a teaching tool","volume":"1","author":"Curran","journal-title":"Implement Sci Commun"},{"issue":"3","key":"R4","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1097\/MLR.0b013e3182408812","article-title":"Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact","volume":"50","author":"Curran","journal-title":"Med Care"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.2196\/17662","article-title":"Workshop on implementation science and digital therapeutics for behavioral health","volume":"8","author":"Lord","journal-title":"JMIR Ment Health"},{"issue":"9","key":"R6","doi-asserted-by":"publisher","DOI":"10.2196\/19898","article-title":"Assessing digital health implementation for a pediatric chronic pain intervention: comparing the RE-AIM and BIT frameworks against real-world trial data and recommendations for future studies","volume":"22","author":"de la Vega","journal-title":"J Med Internet Res"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2019.00064","article-title":"RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review","volume":"7","author":"Glasgow","journal-title":"Front Public Health"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1186\/s13012-022-01258-9","article-title":"Study protocol for a factorial-randomized controlled trial evaluating the implementation, costs, effectiveness, and sustainment of digital therapeutics for substance use disorder in primary care (DIGITS Trial)","volume":"18","author":"Glass","journal-title":"Implement Sci"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1186\/1748-5908-4-50","article-title":"Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science","volume":"4","author":"Damschroder","journal-title":"Implementation Sci"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/s13012-016-0510-7","article-title":"Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update)","volume":"11","author":"Ross","journal-title":"Implement Sci"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-022-08871-w","article-title":"A scoping review of applications of the Consolidated Framework for Implementation Research (CFIR) to telehealth service implementation initiatives","volume":"22","author":"Rangachari","journal-title":"BMC Health Serv Res"},{"issue":"5","key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/18500","article-title":"Theories informing eHealth implementation: systematic review and typology classification","volume":"23","author":"Heinsch","journal-title":"J Med Internet Res"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","DOI":"10.1080\/16549716.2018.1550736","article-title":"Developing a theory-driven contextually relevant mHealth intervention","volume":"12","author":"Jennings","journal-title":"Glob Health Action"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1007\/s41347-020-00165-4","article-title":"Leveraging implementation science to understand factors influencing sustained use of mental health apps: a narrative review","volume":"6","author":"Connolly","journal-title":"J Technol Behav Sci"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/11578","article-title":"Development of PositiveLinks: a mobile phone app to promote linkage and retention in care for people with HIV","volume":"3","author":"Laurence","journal-title":"JMIR Form Res"},{"issue":"6","key":"R16","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1089\/apc.2017.0303","article-title":"PositiveLinks: a mobile health intervention for retention in HIV care and clinical outcomes with 12-month follow-up","volume":"32","author":"Dillingham","journal-title":"AIDS Patient Care STDS"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0226870","article-title":"Long term impact of PositiveLinks: clinic-deployed mobile technology to improve engagement with HIV care","volume":"15","author":"Canan","journal-title":"PLoS ONE"},{"issue":"8","key":"R18","doi-asserted-by":"publisher","first-page":"2708","DOI":"10.1007\/s10461-024-04405-z","article-title":"Three-year follow-up of PositiveLinks: higher use of mHealth platform associated with sustained HIV suppression","volume":"28","author":"Bielick","journal-title":"AIDS Behav"},{"issue":"3","key":"R19","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1089\/apc.2020.0233","article-title":"Implementation of a mobile health strategy to improve linkage to and engagement with HIV care for people living with HIV, Tuberculosis, and substance use in Irkutsk, Siberia","volume":"35","author":"Hodges","journal-title":"AIDS Patient Care STDS"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1186\/s13722-022-00296-4","article-title":"Six-month outcomes of the HOPE smartphone application designed to support treatment with medications for opioid use disorder and piloted during an early statewide COVID-19 lockdown","volume":"17","author":"Hodges","journal-title":"Addict Sci Clin Pract"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/43019","article-title":"Development of an mHealth app to support the prevention of sexually transmitted infections among Black men who have sex with men engaged in pre-exposure prophylaxis care in New Orleans, Louisiana: qualitative user-centered design study","volume":"7","author":"Clement","journal-title":"JMIR Form Res"},{"issue":"3","key":"R22","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooaa068","article-title":"Acceptability and feasibility of a mobile health application for blood pressure monitoring in rural Uganda","volume":"4","author":"Mugabirwe","journal-title":"JAMIA Open"},{"issue":"3","key":"R23","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2021-054867","article-title":"Process evaluation for the adaptation, testing and dissemination of a mobile health platform to support people with HIV and tuberculosis in Irkutsk, Siberia","volume":"12","author":"Hodges","journal-title":"BMJ Open"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1089\/tmr.2020.0018","article-title":"Adapting an m-Health intervention for Spanish-speaking Latinx people living with HIV in the Nonurban Southern United States","volume":"2","author":"Flickinger","journal-title":"Telemed Rep"},{"key":"R25","unstructured":"Interventions: positivelinks. Center for innovation and engagement \/ National Alliance of State and Territorial AIDS Directors (NASTAD); 2021. URL: https:\/\/ciehealth.org\/wp-content\/uploads\/2022\/01\/PositiveLinks_12-16-21_compressed.pdf [Accessed 16-12-2025]"},{"key":"R26","unstructured":"Compendium of evidence-based interventions and best practices for HIV prevention: background, methods, and criteria. Centers for Disease Control and Prevention; 2024. URL: https:\/\/stacks.cdc.gov\/view\/cdc\/149681 [Accessed 16-12-2025]"},{"issue":"11","key":"R27","doi-asserted-by":"publisher","first-page":"e3572","DOI":"10.1093\/cid\/ciaa1391","article-title":"Primary care guidance for persons with human immunodeficiency virus: 2020 update by the HIV medicine association of the infectious diseases society of America","volume":"73","author":"Thompson","journal-title":"Clin Infect Dis"},{"key":"R28","unstructured":"Ryan white HIV\/AIDS program (RWHAP) best practices compilation. Health Resources and Services Administration (HRSA). 2022. URL: https:\/\/targethiv.org\/library\/cie-positivelinks [Accessed 16-12-2025]"},{"issue":"4","key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/37748","article-title":"Evaluation of the implementation and effectiveness of a mobile health intervention to improve outcomes for people with HIV in the Washington, DC Cohort: study protocol for a cluster randomized controlled trial","volume":"11","author":"Hodges","journal-title":"JMIR Res Protoc"},{"issue":"3","key":"R30","doi-asserted-by":"publisher","first-page":"635","DOI":"10.1093\/jamia\/ocv176","article-title":"Development of a large urban longitudinal HIV clinical cohort using a web-based platform to merge electronically and manually abstracted data from disparate medical record systems: technical challenges and innovative solutions","volume":"23","author":"Greenberg","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/1748-5908-9-7","article-title":"Organizational readiness for implementing change: a psychometric assessment of a new measure","volume":"9","author":"Shea","journal-title":"Implement Sci"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1016\/j.annepidem.2025.02.007","article-title":"A framework for designing hybrid effectiveness-implementation trials for digital health interventions","volume":"104","author":"Matson","journal-title":"Ann Epidemiol"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.2196\/48739","article-title":"An mHealth platform for people with HIV receiving care in Washington, district of Columbia: qualitative analysis of stakeholder feedback","volume":"7","author":"Caldwell","journal-title":"JMIR Form Res"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1186\/1748-5908-8-139","article-title":"Implementation strategies: recommendations for specifying and reporting","volume":"8","author":"Proctor","journal-title":"Implement Sci"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2019.00158","article-title":"Implementation mapping: using intervention mapping to develop implementation strategies","volume":"7","author":"Fernandez","journal-title":"Front Public Health"},{"issue":"2","key":"R36","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1177\/1077558711430690","article-title":"A compilation of strategies for implementing clinical innovations in health and mental health","volume":"69","author":"Powell","journal-title":"Med Care Res Rev"},{"issue":"1","key":"R37","doi-asserted-by":"publisher","DOI":"10.1186\/s13012-015-0242-0","article-title":"Making sense of implementation theories, models and frameworks","volume":"10","author":"Nilsen","journal-title":"Implement Sci"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1186\/s13012-017-0635-3","article-title":"Psychometric assessment of three newly developed implementation outcome measures","volume":"12","author":"Weiner","journal-title":"Implement Sci"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1186\/1748-5908-4-67","article-title":"A theory of organizational readiness for change","volume":"4","author":"Weiner","journal-title":"Implement Sci"},{"issue":"4","key":"R40","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002288","article-title":"Implementation science: relevance in the real world without sacrificing rigor","volume":"14","author":"Geng","journal-title":"PLoS Med"},{"issue":"2","key":"R41","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/s10488-010-0319-7","article-title":"Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda","volume":"38","author":"Proctor","journal-title":"Adm Policy Ment Health"},{"issue":"5","key":"R42","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1016\/j.amepre.2016.06.008","article-title":"Evaluating digital health interventions: key questions and approaches","volume":"51","author":"Murray","journal-title":"Am J Prev Med"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e76327","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:07:49Z","timestamp":1782842869000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e76327"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,21]]},"references-count":42,"URL":"https:\/\/doi.org\/10.2196\/76327","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,21]]},"article-number":"v28i1e76327"}}