{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T09:02:50Z","timestamp":1789549370033,"version":"build-2803163510"},"reference-count":47,"publisher":"BMJ","issue":"3","license":[{"start":{"date-parts":[[2025,9,23]],"date-time":"2025-09-23T00:00:00Z","timestamp":1758585600000},"content-version":"unspecified","delay-in-days":84,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100007181","name":"Quality Enhancement Research Initiative","doi-asserted-by":"publisher","award":["1I50HX003201-01"],"award-info":[{"award-number":["1I50HX003201-01"]}],"id":[{"id":"10.13039\/100007181","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open Qual"],"accepted":{"date-parts":[[2025,8,31]]},"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:p>Quality improvement and implementation science evaluations are often complex mixed methods approaches conducted by interdisciplinary teams. These efforts are challenging in normal circumstances; the COVID-19 pandemic and subsequent shift to remote work have posed additional challenges to this type of work. Novel approaches, tools, and processes may be needed to improve the rigor and cohesion of mixed methods evaluations, especially with a team working remotely.<\/jats:p>\n               <jats:p>Quality improvement and implementation science evaluations are often complex mixed methods approaches conducted by interdisciplinary teams. These efforts are challenging in normal circumstances; the COVID-19 pandemic and subsequent shift to remote work have posed additional challenges to this type of work. Novel approaches, tools, and processes may be needed to improve the rigor and cohesion of mixed methods evaluations, especially with a team working remotely.<\/jats:p>\n               <jats:p>Our aim was to create a rigorous evaluation plan for a large hybrid type III implementation-evaluation trial implementing new evidence-based processes at nine medical centers. Given the trial\u2019s complexity and a geographically-distributed remotely-working interdisciplinary team, we found that existing tools did not meet our needs. We thus created a novel process for developing a rigorous evaluation plan that others could replicate.<\/jats:p>\n               <jats:p>This process has seven steps: 1) select a template and identify point person; 2) complete initial development; 3) obtain targeted asynchronous feedback; 4) identify and analyze gaps; 5) conduct targeted virtual synchronous discussion; 6) finalize working document; and 7) apply the plan and solicit ongoing feedback.<\/jats:p>\n               <jats:p>Interdisciplinary quality improvement and implementation science project teams need tools and processes to ensure clear communication, well-ordered workflow, and rigorous operationalization of evaluation aims. The seven-step evaluation plan tool not only helped to enhance the rigor and execution of a large program evaluation, but the process also served an important convening function to enhance coordination between remote team members. Our work builds on existing processes for evaluation plan development while incorporating team science approaches.<\/jats:p>","DOI":"10.1136\/bmjoq-2023-002620","type":"journal-article","created":{"date-parts":[[2025,9,23]],"date-time":"2025-09-23T04:50:25Z","timestamp":1758603025000},"page":"e002620","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":1,"title":["On the same (evaluation) page: a novel approach to enhance mixed-methods implementation evaluation"],"prefix":"10.1136","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9957-0617","authenticated-orcid":false,"given":"Laura Ellen","family":"Ashcraft","sequence":"first","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"},{"name":"Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4471-7957","authenticated-orcid":false,"given":"Amanda D","family":"Peeples","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6382-0522","authenticated-orcid":false,"given":"Daniel","family":"Hall","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Research, and Promotion (CHERP) and Geriatric Research Education and Clinical Center, VA Pittsburgh Healthcare System, Pittsburgh, Pennsylvania, USA"},{"name":"Department of Surgery, University of Pittsburgh, Pittsburgh, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mary Elizabeth","family":"Bowen","sequence":"additional","affiliation":[{"name":"Geriatrics and Extended Care Promotion, VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7483-4137","authenticated-orcid":false,"given":"Rebecca T","family":"Brown","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"},{"name":"Geriatrics and Extended Care Promotion, VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Judith A","family":"Long","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"},{"name":"Division of General Internal Medicine, Department of Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kirstin Manges","family":"Piazza","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Research, and Promotion (CHERP), VA Pittsburgh Healthcare System, Pittsburgh, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Octavia K","family":"Goodman","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Connor M","family":"Warren","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6007-4511","authenticated-orcid":false,"given":"Lindsay R","family":"Pelcher","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Research, and Promotion (CHERP), VA Pittsburgh Healthcare System, Pittsburgh, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3435-4221","authenticated-orcid":false,"given":"Rachel M","family":"Werner","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"},{"name":"Division of General Internal Medicine, Department of Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9521-8699","authenticated-orcid":false,"given":"Robert E","family":"Burke","sequence":"additional","affiliation":[{"name":"Center for Healthcare Evaluation, Resaerch, and Promotion (CHERP), VA Medical Center Corporal Michael J Crescenz, Philadelphia, Pennsylvania, USA"},{"name":"Division of General Internal Medicine, Department of Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"},{"name":"Section of Hospital Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2025,9,22]]},"reference":[{"key":"2025092221501843000_14.3.e002620.1","doi-asserted-by":"crossref","first-page":"508","DOI":"10.1007\/s10488-014-0552-6","article-title":"Approaches to Mixed Methods Dissemination and Implementation Research: Methods, Strengths, Caveats, and Opportunities","volume":"42","author":"Green","year":"2015","journal-title":"Adm Policy Ment Health"},{"key":"2025092221501843000_14.3.e002620.2","doi-asserted-by":"crossref","unstructured":"Palinkas LA , Cooper BR . Mixed methods evaluation in dissemination and implementation science. In: Dissemination and implementation research in health: translating science to practice. 2017: 335\u201353.","DOI":"10.1093\/oso\/9780190683214.003.0020"},{"key":"2025092221501843000_14.3.e002620.3","doi-asserted-by":"publisher","DOI":"10.1136\/bmjqs-2017-007554"},{"key":"2025092221501843000_14.3.e002620.4","doi-asserted-by":"publisher","DOI":"10.1097\/MLR.0b013e3182408812"},{"key":"2025092221501843000_14.3.e002620.5","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.12335"},{"key":"2025092221501843000_14.3.e002620.6","unstructured":"Office of the Assistant Secretary for Planning and Evaluation . Individuals living in the community with chronic conditions and functional limitations: a closer look. Washington, D.C: United States Department of Health and Human Services, 2010."},{"key":"2025092221501843000_14.3.e002620.7","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.hjdsi.2017.05.005","article-title":"Creating Age-Friendly Health Systems \u2013 A vision for better care of older adults","volume":"6","author":"Mate","year":"2018","journal-title":"Healthcare (Basel)"},{"key":"2025092221501843000_14.3.e002620.8","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1111\/jgs.17489","article-title":"Selecting implementation strategies to drive Age-Friendly Health System Adoption","volume":"70","author":"Burke","year":"2022","journal-title":"J Am Geriatr Soc"},{"key":"2025092221501843000_14.3.e002620.9","unstructured":"Institute for Healthcare Improvement . Age-friendly health systems: guide to using the 4ms in the care of older adults. 2020."},{"key":"2025092221501843000_14.3.e002620.10","doi-asserted-by":"publisher","DOI":"10.1186\/1478-4491-11-1"},{"key":"2025092221501843000_14.3.e002620.11","doi-asserted-by":"publisher","DOI":"10.1080\/00909880903025911"},{"key":"2025092221501843000_14.3.e002620.12","doi-asserted-by":"publisher","DOI":"10.1177\/1049732316634304"},{"key":"2025092221501843000_14.3.e002620.13","doi-asserted-by":"publisher","DOI":"10.1177\/1049732308325535"},{"key":"2025092221501843000_14.3.e002620.14","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1080\/19325037.2013.798216","article-title":"Developing Your Evaluation Plans: A Critical Component of Public Health Program Infrastructure","volume":"44","author":"Lavinghouze","year":"2013","journal-title":"Am J Health Educ"},{"key":"2025092221501843000_14.3.e002620.15","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1002\/nml.11402","article-title":"Improving the Quality of Outcome Evaluation Plans","volume":"11","author":"Poole","year":"2001","journal-title":"Nonprofit Mgmnt & Ldrshp"},{"key":"2025092221501843000_14.3.e002620.16","doi-asserted-by":"publisher","DOI":"10.1177\/1524839904273387"},{"key":"2025092221501843000_14.3.e002620.17","doi-asserted-by":"publisher","DOI":"10.1001\/jamasurg.2016.4202"},{"key":"2025092221501843000_14.3.e002620.18","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1001\/jamasurg.2016.4219","article-title":"Association of a Frailty Screening Initiative With Postoperative Survival at 30, 180, and 365 Days","volume":"152","author":"Hall","year":"2017","journal-title":"JAMA Surg"},{"key":"2025092221501843000_14.3.e002620.19","doi-asserted-by":"crossref","first-page":"e1230","DOI":"10.1097\/SLA.0000000000003808","article-title":"Clinical Utility of the Risk Analysis Index as a Prospective Frailty Screening Tool within a Multi-practice, Multi-hospital Integrated Healthcare System","volume":"274","author":"Varley","year":"2021","journal-title":"Ann Surg"},{"key":"2025092221501843000_14.3.e002620.20","doi-asserted-by":"crossref","first-page":"890","DOI":"10.1001\/jamainternmed.2014.949","article-title":"Reduction of inappropriate benzodiazepine prescriptions among older adults through direct patient education: the EMPOWER cluster randomized trial","volume":"174","author":"Tannenbaum","year":"2014","journal-title":"JAMA Intern Med"},{"key":"2025092221501843000_14.3.e002620.21","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2018.16131"},{"key":"2025092221501843000_14.3.e002620.22","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1111\/jgs.15247","article-title":"Academic Detailing with Provider Audit and Feedback Improve Prescribing Quality for Older Veterans","volume":"66","author":"Vandenberg","year":"2018","journal-title":"J American Geriatrics Society"},{"key":"2025092221501843000_14.3.e002620.23","first-page":"36","article-title":"Reducing benzodiazepine prescribing in older veterans: a direct-to-consumer educational brochure","volume":"35","author":"Mendes","year":"2018","journal-title":"Fed Pract Health Care Prof VA DoD PHS"},{"key":"2025092221501843000_14.3.e002620.24","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1177\/0897190019870949","article-title":"Academic Detailing to Reduce Sedative-Hypnotic Prescribing in Older Veterans","volume":"34","author":"Ragan","year":"2021","journal-title":"J Pharm Pract"},{"key":"2025092221501843000_14.3.e002620.25","doi-asserted-by":"publisher","DOI":"10.1097\/JGP.0b013e318160da72"},{"key":"2025092221501843000_14.3.e002620.26","doi-asserted-by":"publisher","DOI":"10.1093\/geront\/gnp087"},{"key":"2025092221501843000_14.3.e002620.27","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1111\/jgs.15194","article-title":"Targeting Behavioral Symptoms and Functional Decline in Dementia: A Randomized Clinical Trial","volume":"66","author":"Gitlin","year":"2018","journal-title":"J Am Geriatr Soc"},{"key":"2025092221501843000_14.3.e002620.28","doi-asserted-by":"crossref","DOI":"10.1186\/s12877-021-02511-4","article-title":"Effects of the tailored activity program (TAP) on dementia-related symptoms, health events and caregiver wellbeing: a randomized controlled trial","volume":"21","author":"Gitlin","year":"2021","journal-title":"BMC Geriatr"},{"key":"2025092221501843000_14.3.e002620.29","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1001\/jamainternmed.2018.6026","article-title":"Effect of a Biobehavioral Environmental Approach on Disability Among Low-Income Older Adults: A Randomized Clinical Trial","volume":"179","author":"Szanton","year":"2019","journal-title":"JAMA Intern Med"},{"key":"2025092221501843000_14.3.e002620.30","doi-asserted-by":"publisher","DOI":"10.1377\/hlthaff.2016.0140"},{"key":"2025092221501843000_14.3.e002620.31","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1111\/jgs.15713","article-title":"Dissemination of the CAPABLE Model of Care in a Medicaid Waiver Program to Improve Physical Function","volume":"67","author":"Spoelstra","year":"2019","journal-title":"J Am Geriatr Soc"},{"key":"2025092221501843000_14.3.e002620.32","doi-asserted-by":"publisher","DOI":"10.1016\/S1553-7250(08)34030-6"},{"key":"2025092221501843000_14.3.e002620.33","doi-asserted-by":"publisher","DOI":"10.1093\/TBM\/IBZ085"},{"key":"2025092221501843000_14.3.e002620.34","first-page":"663","article-title":"Evaluating the Implementation of Project Re-Engineered Discharge (RED) in Five Veterans Health Administration (VHA) Hospitals","volume":"44","author":"Sullivan","year":"2018","journal-title":"Jt Comm J Qual Patient Saf"},{"key":"2025092221501843000_14.3.e002620.35","doi-asserted-by":"crossref","DOI":"10.1186\/s13012-017-0653-1","article-title":"Implementation and dissemination of a transition of care program for rural veterans: a controlled before and after study","volume":"12","author":"Leonard","year":"2017","journal-title":"Implement Sci"},{"key":"2025092221501843000_14.3.e002620.36","doi-asserted-by":"crossref","first-page":"10","DOI":"10.7812\/TPP\/08-088","article-title":"Using Implementation and Dissemination Concepts to Spread 21st-century Well-Child Care at a Health Maintenance Organization","volume":"13","author":"Beck","year":"2009","journal-title":"Perm J"},{"key":"2025092221501843000_14.3.e002620.37","doi-asserted-by":"crossref","DOI":"10.1186\/s13012-015-0227-z","article-title":"Implementation challenges and successes of a population-based colorectal cancer screening program: a qualitative study of stakeholder perspectives","volume":"10","author":"Liles","year":"2015","journal-title":"Implement Sci"},{"key":"2025092221501843000_14.3.e002620.38","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-018-4617-1"},{"key":"2025092221501843000_14.3.e002620.39","doi-asserted-by":"crossref","first-page":"133","DOI":"10.12788\/jhm.3320","article-title":"Perspectives of Clinicians, Staff, and Veterans in Transitioning Veterans from non-VA Hospitals to Primary Care in a Single VA Healthcare System","volume":"15","author":"Ayele","year":"2020","journal-title":"J Hosp Med"},{"key":"2025092221501843000_14.3.e002620.40","doi-asserted-by":"crossref","DOI":"10.1186\/s43058-023-00431-5","article-title":"Study protocol: Type III hybrid effectiveness-implementation study implementing Age-Friendly evidence-based practices in the VA to improve outcomes in older adults","volume":"4","author":"Piazza","year":"2023","journal-title":"Implement Sci Commun"},{"key":"2025092221501843000_14.3.e002620.41","doi-asserted-by":"crossref","DOI":"10.1186\/s13012-015-0242-0","article-title":"Making sense of implementation theories, models and frameworks","volume":"10","author":"Nilsen","year":"2015","journal-title":"Implement Sci"},{"key":"2025092221501843000_14.3.e002620.42","unstructured":"Bowersox N , Jedele J , Kawentel L . Quality enhancement research initiative (queri) evaluation guide: developing and implementing a high-quality evaluation. United States Department of Veterans Affairs Health Administration Office of Research and Development; 2021."},{"key":"2025092221501843000_14.3.e002620.43","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1017\/cts.2019.413","article-title":"Identifying strategies to promote team science in dissemination and implementation research","volume":"4","author":"Aarons","year":"2020","journal-title":"J Clin Trans Sci"},{"key":"2025092221501843000_14.3.e002620.44","unstructured":"Gittell JH . Relational coordination: guidelines for theory, measurement and analysis. Walth MA Brand Univ; 2011."},{"key":"2025092221501843000_14.3.e002620.45","doi-asserted-by":"publisher","DOI":"10.1891\/1061-3749.23.3.379"},{"key":"2025092221501843000_14.3.e002620.46","doi-asserted-by":"crossref","first-page":"302","DOI":"10.1097\/HMR.0000000000000241","article-title":"The influence of leadership facilitation on relational coordination among primary care team members of accountable care organizations","volume":"45","author":"Huber","year":"2020","journal-title":"Health Care Manage Rev"},{"key":"2025092221501843000_14.3.e002620.47","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1097\/HMR.0000000000000300","article-title":"Practices to support relational coordination in care transitions: Observations from the VA rural Transitions Nurse Program","volume":"47","author":"Gilmartin","year":"2022","journal-title":"Health Care Manage Rev"}],"container-title":["BMJ Open Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjoq-2023-002620","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,23]],"date-time":"2025-09-23T04:50:31Z","timestamp":1758603031000},"score":1,"resource":{"primary":{"URL":"https:\/\/qir.bmj.com\/lookup\/doi\/10.1136\/bmjoq-2023-002620"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":47,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9,22]]},"published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.1136\/bmjoq-2023-002620"],"URL":"https:\/\/doi.org\/10.1136\/bmjoq-2023-002620","relation":{},"ISSN":["2399-6641"],"issn-type":[{"value":"2399-6641","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7]]}}}