{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T02:17:58Z","timestamp":1791425878938,"version":"4.3.3"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T00:00:00Z","timestamp":1755561600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T00:00:00Z","timestamp":1755561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"National Institute of Mental Health, National Institute of Drug Abuse, Office of the Director of NIH","award":["R01 MH118213"],"award-info":[{"award-number":["R01 MH118213"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["AIDS Behav"],"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    This study sought to compare implementation and effectiveness outcomes for two delivery strategies of a digital HIV prevention intervention: community-based organization (CBO) versus direct-to-consumer (DTC). A type III hybrid effectiveness\u2013implementation trial compared two implementation strategies of\n                    <jats:italic>Keep It Up<\/jats:italic>\n                    ! (KIU! ) for young men who have sex with men (YMSM) (\n                    <jats:italic>n<\/jats:italic>\n                    \u2009=\u20092124). Data were collected at baseline and 12-weeks in 2019\u20132023. DTC enrolled more participants, but CBO recruited more Black (11.7% vs. 21.7%) and Latino YMSM (27.3% vs. 32.9%). CBO enrollees were at higher HIV risk based on condomless anal sex (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u2009.05) and rectal Gonorrhea rates (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u2009.01). Although pre-exposure prophylaxis (PrEP) use with adherence was 1.94 times higher at follow-up vs. baseline (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u2009.0001) across arms, PrEP increase was 1.60 times higher in CBO vs. DTC (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u2009.0001). CBO delivery averted more HIV infections per 100 people-years than DTC (1.65 vs. 0.62), but the cost of averting an HIV infection was higher in CBO than DTC ($983,029 vs. $173,313). Although CBOs were able to reach participants at higher HIV risk, the DTC strategy was estimated to cost less per infection averted and below established thresholds for cost savings. Findings must be interpreted within the context of data collection during the COVID-19 epidemic. More information about KIU! can be found at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/kiu.northwestern.edu\/\" ext-link-type=\"uri\">https:\/\/kiu.northwestern.edu\/<\/jats:ext-link>\n                    .\n                  <\/jats:p>\n                  <jats:p>Trial registration: NCT03896776 (date of registration: 04\/01\/2019).<\/jats:p>","DOI":"10.1007\/s10461-025-04838-0","type":"journal-article","created":{"date-parts":[[2025,8,19]],"date-time":"2025-08-19T02:13:48Z","timestamp":1755569628000},"page":"4030-4042","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Comparing Implementation and Effectiveness Outcomes for Two Implementation Strategies of the Keep It Up! Digital HIV Prevention Program: A Type 3 Hybrid Effectiveness-Implementation Trial"],"prefix":"10.1007","volume":"29","clinical-trial-number":[{"clinical-trial-number":"nct03896776","registry":"10.18810\/clinical-trials-gov"}],"author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9222-5116","authenticated-orcid":false,"given":"Brian","family":"Mustanski","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nanette","family":"Benbow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathryn","family":"Macapagal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dennis","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Krystal","family":"Madkins","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rana","family":"Saber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Linas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"JD","family":"Smith","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"C. Hendricks","family":"Brown","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Munroe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Susheel","family":"Reddy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bruce R.","family":"Schackman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory","family":"Swann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick","family":"Janulis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alithia","family":"zamantakis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Juan Pablo","family":"Zapata","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,8,19]]},"reference":[{"key":"4838_CR1","unstructured":"Centers for Disease Control and Prevention. HV Surveillance Supplemental Report: estimated HIV Incidence and Prevalence in the United States, 2018\u20132022. 2024. https:\/\/stacks.cdc.gov\/view\/cdc\/156513"},{"key":"4838_CR2","unstructured":"Centers for Disease Control and Prevention. Compendium of evidence-based interventions and best practices for HIV prevention. 2024. https:\/\/www.cdc.gov\/hiv\/research\/interventionresearch\/compendium\/index.html. Accessed 10 Feb 2024"},{"issue":"2","key":"4838_CR3","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1016\/j.amepre.2016.02.026","volume":"51","author":"B Mustanski","year":"2016","unstructured":"Mustanski B, Fisher CB. HIV rates are increasing in gay\/bisexual teens: IRB barriers to research must be resolved to Bend the curve. Am J Prev Med Aug. 2016;51(2):249\u201352.","journal-title":"Am J Prev Med Aug"},{"issue":"1","key":"4838_CR4","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1186\/s13012-024-01345-z","volume":"8","author":"JD Smith","year":"2024","unstructured":"Smith JD, Li DH, Merle JL, Keiser B, Mustanski B, Benbow ND. Adjunctive interventions: change methods directed at recipients that support uptake and use of health innovations. Implement Sci Feb. 2024;8(1):10.","journal-title":"Implement Sci Feb"},{"key":"4838_CR5","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1146\/annurev-psych-032620-035725","volume":"18","author":"B Mustanski","year":"2024","unstructured":"Mustanski B, Queiroz A, Merle JL, et al. A systematic review of implementation research on determinants and strategies of effective HIV interventions for men who have sex with men in the United States. Annu Rev Psychol. 2024;18:55\u201385.","journal-title":"Annu Rev Psychol"},{"issue":"1","key":"4838_CR6","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1007\/s10461-017-1923-2","volume":"22","author":"KD Henny","year":"2018","unstructured":"Henny KD, Wilkes AL, McDonald CM, Denson DJ, Neumann MS. A rapid review of eHealth interventions addressing the continuum of HIV care (2007\u20132017). AIDS Behav. 2018;22(1):43\u201363.","journal-title":"AIDS Behav"},{"issue":"9","key":"4838_CR7","doi-asserted-by":"publisher","first-page":"2253","DOI":"10.1007\/s10461-019-02626-1","volume":"23","author":"LH Nguyen","year":"2019","unstructured":"Nguyen LH, Tran BX, Rocha LEC, et al. A systematic review of eHealth interventions addressing HIV\/STI prevention among men who have sex with men. AIDS Behav. 2019;23(9):2253\u201372.","journal-title":"AIDS Behav"},{"issue":"3","key":"4838_CR8","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1007\/s11904-024-00693-1","volume":"21","author":"DH Li","year":"2024","unstructured":"Li DH, Macapagal K, Mongrella M, Saber R, Mustanski B. Your package could not be delivered: the state of digital HIV intervention implementation in the US. Curr HIV\/AIDS Rep. 2024;21(3):152\u201367.","journal-title":"Curr HIV\/AIDS Rep"},{"issue":"1","key":"4838_CR9","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1007\/s11904-014-0239-3","volume":"12","author":"KE Muessig","year":"2015","unstructured":"Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, internet and web 2.0 interventions to address the HIV continuum of care. Curr HIV\/AIDS Rep. 2015;12(1):173\u201390.","journal-title":"Curr HIV\/AIDS Rep"},{"issue":"5","key":"4838_CR10","doi-asserted-by":"publisher","first-page":"e15977","DOI":"10.2196\/15977","volume":"25","author":"M Xin","year":"2020","unstructured":"Xin M, Viswanath K, Li AY, et al. The effectiveness of electronic health interventions for promoting HIV-preventive behaviors among men who have sex with men: meta-analysis based on an integrative framework of design and implementation features. J Med Internet Res. 2020;25(5):e15977.","journal-title":"J Med Internet Res"},{"issue":"1","key":"4838_CR11","doi-asserted-by":"publisher","first-page":"e25439","DOI":"10.1002\/jia2.25439","volume":"23","author":"KM Maloney","year":"2020","unstructured":"Maloney KM, Bratcher A, Wilkerson R, Sullivan PS. Electronic and other new media technology interventions for HIV care and prevention: a systematic review. J Int AIDS Soc. 2020;23(1):e25439.","journal-title":"J Int AIDS Soc"},{"issue":"2","key":"4838_CR12","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1080\/00224499.2013.867924","volume":"3","author":"B Mustanski","year":"2015","unstructured":"Mustanski B, Greene GJ, Ryan D, Whitton SW. Feasibility, acceptability, and initial efficacy of an online sexual health promotion program for LGBT youth: the Queer sex ed intervention. J Sex Res. 2015;3(2):220\u201330.","journal-title":"J Sex Res"},{"issue":"3","key":"4838_CR13","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1521\/aeap.2016.28.3.231","volume":"28","author":"GJ Greene","year":"2016","unstructured":"Greene GJ, Madkins K, Andrews K, Dispenza J, Mustanski B. Implementation and evaluation of the keep it up! online HIV prevention intervention in a community-based setting. AIDS Educ Prev. 2016;28(3):231\u201345.","journal-title":"AIDS Educ Prev"},{"issue":"2","key":"4838_CR14","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.amepre.2018.04.026","volume":"55","author":"B Mustanski","year":"2018","unstructured":"Mustanski B, Parsons JT, Sullivan PS, Madkins K, Rosenberg E, Swann G. Biomedical and behavioral outcomes of keep it up! an eHealth HIV prevention program RCT. Am J Prev Med. 2018;55(2):151\u20138.","journal-title":"Am J Prev Med"},{"key":"4838_CR15","doi-asserted-by":"publisher","first-page":"1053496","DOI":"10.3389\/frhs.2022.1053496","volume":"2","author":"GM Curran","year":"2022","unstructured":"Curran GM, Landes SJ, McBain SA, et al. Reflections on 10 years of effectiveness-implementation hybrid studies. Front Health Serv. 2022;2:1053496.","journal-title":"Front Health Serv"},{"issue":"1","key":"4838_CR16","first-page":"85","volume":"1","author":"SJ Becker","year":"2015","unstructured":"Becker SJ. Direct-to-consumer marketing: a complementary approach to traditional dissemination and implementation efforts for mental health and substance abuse interventions. Clin Psychol (New York). 2015;1(1):85\u2013100.","journal-title":"Clin Psychol (New York)"},{"key":"4838_CR17","doi-asserted-by":"publisher","first-page":"e50293","DOI":"10.2196\/50293","volume":"5","author":"HC Moungui","year":"2024","unstructured":"Moungui HC, Nana-Djeunga HC, Anyiang CF, Cano M, Ruiz Postigo JA, Carrion C. Dissemination strategies for mHealth apps: systematic review. JMIR Mhealth Uhealth. 2024;5:e50293.","journal-title":"JMIR Mhealth Uhealth"},{"key":"4838_CR18","doi-asserted-by":"publisher","first-page":"107134","DOI":"10.1016\/j.cct.2023.107134","volume":"127","author":"B Mustanski","year":"2023","unstructured":"Mustanski B, Saber R, Jones JP, et al. Keep it up! 3.0: study protocol for a type III hybrid implementation-effectiveness cluster-randomized trial. Contemp Clin Trials. 2023;127:107134.","journal-title":"Contemp Clin Trials"},{"issue":"1","key":"4838_CR19","doi-asserted-by":"publisher","first-page":"e14","DOI":"10.2196\/publichealth.5365","volume":"2","author":"JA Grey","year":"2016","unstructured":"Grey JA, Bernstein KT, Sullivan PS, et al. Estimating the population sizes of men who have sex with men in US States and counties using data from the American community survey. JMIR Public Health Surveill. 2016;2(1):e14.","journal-title":"JMIR Public Health Surveill"},{"issue":"5","key":"4838_CR20","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1093\/her\/cyl081","volume":"21","author":"RE Glasgow","year":"2006","unstructured":"Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res 2006;21(5):688\u201394.","journal-title":"Health Educ Res\u00a0"},{"issue":"2","key":"4838_CR21","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1093\/infdis\/jiz037","volume":"2","author":"C Mullick","year":"2020","unstructured":"Mullick C, Murray J. Correlations between human immunodeficiency virus (HIV) infection and rectal gonorrhea incidence in men who have sex with men: implications for future HIV preexposure prophylaxis trials. J Infect Dis. 2020;2(2):214\u20137.","journal-title":"J Infect Dis"},{"issue":"6","key":"4838_CR22","doi-asserted-by":"publisher","first-page":"1966","DOI":"10.1007\/s10461-019-02766-4","volume":"24","author":"B Mustanski","year":"2020","unstructured":"Mustanski B, Ryan DT, Newcomb ME, D\u2019Aquila RT, Matson M. Very high HIV incidence and associated risk factors in a longitudinal cohort study of diverse adolescent and young adult men who have sex with men and transgender women. AIDS Behav. 2020;24(6):1966\u201375.","journal-title":"AIDS Behav"},{"key":"4838_CR23","unstructured":"Centers for Disease Control and Prevention. Estimated HIV incidence and prevalence in the United States, 2017\u20132021. HIV Surveillance Supplemental Report. 2023. https:\/\/www.cdc.gov\/hiv\/library\/reports\/hiv-surveillance\/vol-28-no-3\/index.html. Accessed 16 Sept 2023."},{"issue":"7","key":"4838_CR24","doi-asserted-by":"publisher","first-page":"S76","DOI":"10.1097\/MLR.0b013e31819bc064","volume":"47","author":"KD Frick","year":"2009","unstructured":"Frick KD. Microcosting quantity data collection methods. Med Care. 2009;47(7):S76-81.","journal-title":"Med Care"},{"issue":"1","key":"4838_CR25","doi-asserted-by":"publisher","first-page":"e26040","DOI":"10.1002\/jia2.26040","volume":"26","author":"RK Shrestha","year":"2023","unstructured":"Shrestha RK, Fanfair RN, Randall LM, et al. Costs and cost-effectiveness of a collaborative data-to-care intervention for HIV treatment and care in the United States. Journal Int AIDS Soc. 2023;26(1):e26040.","journal-title":"Journal Int AIDS Soc"},{"issue":"3","key":"4838_CR26","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1097\/QAI.0000000000003572","volume":"1","author":"S Munroe","year":"2025","unstructured":"Munroe S, Schackman BR, Madkins K, et al. Economic evaluation of delivering an evidence-based online HIV prevention program to MSM via direct-to-consumer marketing versus community-based organization recruitment. J Acquir Immune Defic Syndr. 2025;1(3):257\u201364.","journal-title":"J Acquir Immune Defic Syndr"},{"issue":"6","key":"4838_CR27","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1007\/s11904-018-0415-y","volume":"15","author":"CG Kemp","year":"2018","unstructured":"Kemp CG, Velloza J. Implementation of eHealth interventions across the HIV care cascade: a review of recent research. Curr HIV\/AIDS Rep. 2018;15(6):403\u201313.","journal-title":"Curr HIV\/AIDS Rep"},{"issue":"2","key":"4838_CR28","doi-asserted-by":"publisher","first-page":"e1001382","DOI":"10.1371\/journal.pmed.1001382","volume":"10","author":"M Tomlinson","year":"2013","unstructured":"Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC. Scaling up mHealth: where is the evidence? PLoS Med. 2013;10(2):e1001382.","journal-title":"PLoS Med"},{"issue":"2","key":"4838_CR29","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1007\/s10508-009-9596-1","volume":"20110708","author":"B Mustanski","year":"2011","unstructured":"Mustanski B, Lyons T, Garcia SC. Internet use and sexual health of young men who have sex with men: a mixed-methods study. Arch Sex Behav. 2011;20110708(2):289\u2013300.","journal-title":"Arch Sex Behav"},{"key":"4838_CR30","doi-asserted-by":"publisher","first-page":"3944","DOI":"10.1007\/s10461-024-04525-6","volume":"28","author":"A Zamantakis","year":"2024","unstructured":"Zamantakis A, Zapata JP, Greenawalt I, Knapp AA, Benbow N, Mustanski B. Barriers and facilitators to implementing keep it up! A digital health intervention in community-based organizations. AIDS and Behav. 2024;28:3944\u201355.","journal-title":"AIDS and Behav"},{"key":"4838_CR31","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1186\/s43058-024-00628-2","volume":"5","author":"DH Li","year":"2024","unstructured":"Li DH, Zamantakis A, Zapata JP, et al. A mixed-methods approach to assessing implementers\u2019 readiness to adopt digital health interventions (RADHI). Implement Sci Commun. 2024;5:91.","journal-title":"Implement Sci Commun"},{"key":"4838_CR32","unstructured":"Mustansk B, Benbow N, Saber R et al. Outcomes of a hybrid type 3 trial comparing Direct-to-Consumer (DTC) versus Community-Based Organization (CBO) implementation of the Keep It Up! HIV prevention program. Paper presented at: The 16th Annual Conference on the Science of Dissemination and Implementation2023, Washington"},{"key":"4838_CR33","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-49508-8","volume-title":"Investing in e-health: what it takes to sustain consumer health informatics","author":"DH Gustafson","year":"2007","unstructured":"Gustafson DH, Brennan PF, Hawkins R. Investing in e-health: what it takes to sustain consumer health informatics. New York: Springer; 2007."},{"issue":"6","key":"4838_CR34","doi-asserted-by":"publisher","first-page":"699","DOI":"10.1016\/j.amepre.2016.01.011","volume":"50","author":"F Lin","year":"2016","unstructured":"Lin F, Farnham PG, Shrestha RK, Mermin J, Sansom SL. Cost effectiveness of HIV prevention interventions in the U.S. Am J Prev Med. 2016;50(6):699\u2013708.","journal-title":"Am J Prev Med"}],"container-title":["AIDS and Behavior"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10461-025-04838-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10461-025-04838-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10461-025-04838-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T02:49:23Z","timestamp":1762138163000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10461-025-04838-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,19]]},"references-count":34,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2025,12]]}},"alternative-id":["4838"],"URL":"https:\/\/doi.org\/10.1007\/s10461-025-04838-0","relation":{},"ISSN":["1090-7165","1573-3254"],"issn-type":[{"value":"1090-7165","type":"print"},{"value":"1573-3254","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,19]]},"assertion":[{"value":"20 July 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 August 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"The trial was approved by the IRB at Northwestern University (STU00207476). All participants were informed of the aims of the study and of data protection; all participants consented to participate in our trial.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}},{"value":"All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and\/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Research Involving Human and \/or Animal Participants"}}]}}