{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:12:15Z","timestamp":1787026335858,"version":"build-2736575974"},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T00:00:00Z","timestamp":1780012800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["857223"],"award-info":[{"award-number":["857223"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100008530","name":"European Regional Development Fund","doi-asserted-by":"publisher","award":["857223"],"award-info":[{"award-number":["857223"]}],"id":[{"id":"10.13039\/501100008530","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004587","name":"Instituto de Salud Carlos III","doi-asserted-by":"publisher","award":["ISCIII; Miguel Servet 2019: CP19\/00108"],"award-info":[{"award-number":["ISCIII; Miguel Servet 2019: CP19\/00108"]}],"id":[{"id":"10.13039\/501100004587","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100014013","name":"UK Research and Innovation","doi-asserted-by":"publisher","award":["10031483"],"award-info":[{"award-number":["10031483"]}],"id":[{"id":"10.13039\/100014013","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:p>The large-scale deployment of digital health solutions requires robust operational frameworks capable of coordinating heterogeneous settings, diverse stakeholders, and complex technical infrastructures. However, actionable guidance for executing federated, multinational eHealth pilots remains limited in the implementation literature. Methods: Using a mixed-methods approach, including iterative focus groups, co-creation sessions, and a Delphi study, we developed and refined a practice-derived management framework over four years within the GATEKEEPER project (EU Horizon 2020, Grant Agreement No. 857223). The study involved a federation of four European large-scale pilots. A panel of 23 experts ranked 45 best practices across six operational domains: Engagement, Intervention, Monitoring and Control, Planning, Recruitment, and Other. Results: The resulting framework integrates a structured definition of operative key performance indicators, standardised reporting and analysis tools, and a Business Intelligence dashboard to support real-time monitoring and decision-making across the preparation, deployment, and running phases of large-scale pilots. Among the ranked best practices, usability testing, user-centred digital tool design, and active recruitment strategies emerged as top priorities across pilot sites. Discussion: This management framework addresses a critical gap in implementation science by offering actionable, consensus-validated guidance for coordinating large, distributed, multi-site digital health deployments. The GATEKEEPER experience demonstrates how structured operational governance and shared performance monitoring can support the execution of complex eHealth pilots, with insights that may inform future large-scale initiatives seeking sustainable and patient-centred digital health integration.<\/jats:p>","DOI":"10.3389\/fdgth.2026.1730232","type":"journal-article","created":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T06:08:51Z","timestamp":1780034931000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Best practices and challenges in executing large scale pilots for eHealth deployment and managing innovation: the GATEKEEPER experience"],"prefix":"10.3389","volume":"8","author":[{"given":"Davide","family":"Piaggio","sequence":"first","affiliation":[{"name":"Applied Biomedical Signal Processing Intelligent eHealth Lab, University of Warwick","place":["Coventry, United Kingdom"]},{"name":"European Alliance of Medical and Biological Engineering and Science (EAMBES)","place":["Leuven, Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jordi","family":"de Batlle","sequence":"additional","affiliation":[{"name":"Group of Translational Research in Respiratory Medicine, Institut de Recerca Biom\u00e8dica de Lleida \u2013 Fundaci\u00f3 Dr. Pifarr\u00e9, IRBLleida","place":["Lleida, Spain"]},{"name":"Centro de Investigaci\u00f3n Biom\u00e9dica en Red de Enfermedades Respiratorias (CIBERES)","place":["Madrid, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alba M.","family":"Gallego Montejo","sequence":"additional","affiliation":[{"name":"Life Supporting Technologies-LifeSTech, Universidad Polit\u00e9cnica de Madrid","place":["Madrid, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gloria","family":"Cea S\u00e1nchez","sequence":"additional","affiliation":[{"name":"Unidad de Innovaci\u00f3n, Hospital de Barbastro, Servicio Aragon\u00e9s de Salud","place":["Barbastro, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pedro","family":"Checa Rif\u00e1","sequence":"additional","affiliation":[{"name":"Applied Biomedical Signal Processing Intelligent eHealth Lab, University of Warwick","place":["Coventry, United Kingdom"]},{"name":"European Alliance of Medical and Biological Engineering and Science (EAMBES)","place":["Leuven, Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessia","family":"Maccaro","sequence":"additional","affiliation":[{"name":"Applied Biomedical Signal Processing Intelligent eHealth Lab, University of Warwick","place":["Coventry, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rosana","family":"Angles Barbastro","sequence":"additional","affiliation":[{"name":"Unidad de Innovaci\u00f3n, Hospital de Barbastro, Servicio Aragon\u00e9s de Salud","place":["Barbastro, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Teresa Hurtado","sequence":"additional","affiliation":[{"name":"Unidad de Innovaci\u00f3n, Hospital de Barbastro, Servicio Aragon\u00e9s de Salud","place":["Barbastro, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jon Eneko","family":"Idoyaga Uribarrena","sequence":"additional","affiliation":[{"name":"Internationalization and Project Coordination Unit, Biobizkaia Health Research Institute","place":["Barakaldo (Bizkaia), Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Irati","family":"Erreguerena Redondo","sequence":"additional","affiliation":[{"name":"Biosistemak Institute for Health Systems Research","place":["Bilbao, Spain"]},{"name":"Network for Research on Chronicity, Primary Care, and Health Promotion (RICAPPS)","place":["Bilbao, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Krini","sequence":"additional","affiliation":[{"name":"Cyprus Association of Cancer Patients and Friends (PASYKAF)","place":["Nicosia, Cyprus"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Matsangidou","sequence":"additional","affiliation":[{"name":"CYENS Centre of Excellence","place":["Nicosia, Cyprus"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ioanna","family":"Drympeta","sequence":"additional","affiliation":[{"name":"Information Technologies Institute, Centre for Research and Technology","place":["Thessaloniki, Greece"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Konstantinos","family":"Votis","sequence":"additional","affiliation":[{"name":"Information Technologies Institute, Centre for Research and Technology","place":["Thessaloniki, Greece"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessio","family":"Antonini","sequence":"additional","affiliation":[{"name":"Knowledge Media Institute, The Open University","place":["Milton Keynes, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Przemyslaw","family":"Kardas","sequence":"additional","affiliation":[{"name":"Department of Family Medicine, Medical University of Lodz","place":["\u0141\u00f3dz, Poland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pawel","family":"Lewek","sequence":"additional","affiliation":[{"name":"Faculty of Medicine Carl Gustav Carus, Technical University Dresden","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julia","family":"Schellong","sequence":"additional","affiliation":[{"name":"Faculty of Medicine Carl Gustav Carus, Technical University Dresden","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francesco","family":"Giuliani","sequence":"additional","affiliation":[{"name":"Innovation and Research Unit, Casa Sollievo della Sofferenza Research Hospital","place":["San Giovanni Rotondo, Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Franco","family":"Mercalli","sequence":"additional","affiliation":[{"name":"MultiMed Engineers Srls","place":["Parma, Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giuseppe","family":"Fico","sequence":"additional","affiliation":[{"name":"European Alliance of Medical and Biological Engineering and Science (EAMBES)","place":["Leuven, Belgium"]},{"name":"Life Supporting Technologies-LifeSTech, Universidad Polit\u00e9cnica de Madrid","place":["Madrid, Spain"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leandro","family":"Pecchia","sequence":"additional","affiliation":[{"name":"Applied Biomedical Signal Processing Intelligent eHealth Lab, University of Warwick","place":["Coventry, United Kingdom"]},{"name":"European Alliance of Medical and Biological Engineering and Science (EAMBES)","place":["Leuven, Belgium"]},{"name":"Department of Engineering , University Campus Bio-Medico","place":["Rome, Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"the GATEKEEPER Consortium","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,5,29]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1080\/24725579.2021.1884627","article-title":"Health care 4.0: a vision for smart and connected health care","volume":"11","author":"Li","year":"2021","journal-title":"IISE Trans Healthcare Syst Eng"},{"key":"B2","doi-asserted-by":"publisher","first-page":"e37338","DOI":"10.2196\/37338","article-title":"Using an integrated framework to investigate the facilitators and barriers of health information technology implementation in noncommunicable disease management: systematic review","volume":"24","author":"Sung","year":"2022","journal-title":"J Med Internet Res"},{"key":"B3","doi-asserted-by":"publisher","first-page":"890","DOI":"10.1108\/INTR-10-2021-0774","article-title":"Distance is no longer a barrier to healthcare services: current state and future trends of telehealth research","volume":"33","author":"Sahoo","year":"2023","journal-title":"Internet Res"},{"key":"B4","doi-asserted-by":"publisher","first-page":"1649","DOI":"10.1177\/0733464820983630","article-title":"The effect of telehealth interventions on function and quality of life for older adults with pre-frailty or frailty: a systematic review and meta-analysis","volume":"40","author":"Esfandiari","year":"2021","journal-title":"J Appl Gerontol"},{"key":"B5","doi-asserted-by":"publisher","first-page":"94","DOI":"10.3934\/publichealth.2023008","article-title":"\u201cMind the gap\u201d\u2014an overview of the role of the extensions community healthcare outcomes (ECHO) model in enhancing value in health care delivery","volume":"10","author":"Kenny","year":"2023","journal-title":"AIMS Public Health"},{"key":"B6","doi-asserted-by":"publisher","first-page":"e15935","DOI":"10.2196\/15935","article-title":"Social, organizational, and technological factors impacting clinicians\u2019 adoption of mobile health tools: systematic literature review","volume":"8","author":"Jacob","year":"2020","journal-title":"JMIR Mhealth Uhealth"},{"key":"B7","first-page":"204","article-title":"Industry 5.0 for healthcare 5.0: opportunities, challenges and future research possibilities","author":"Gomathi","year":"2023"},{"key":"B8","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/978-3-319-01583-5_63","article-title":"Integration of health care and social care by technology","volume-title":"Handbook of Smart Homes, Health Care and Well-Being","author":"Bierhoff","year":"2017"},{"key":"B9","doi-asserted-by":"publisher","first-page":"2039","DOI":"10.1093\/jamia\/ocab096","article-title":"Benefits, challenges, and contributors to success for national eHealth systems implementation: a scoping review","volume":"28","author":"Scheibner","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"B10","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/s13012-023-01261-8","article-title":"Implementation costs of hospital-based computerised decision support systems: a systematic review","volume":"18","author":"Donovan","year":"2023","journal-title":"Implement Sci"},{"key":"B11","doi-asserted-by":"publisher","first-page":"100330","DOI":"10.1016\/j.smhl.2022.100330","article-title":"Evaluation of challenges for adoption of smart healthcare strategies","volume":"26","author":"Renukappa","year":"2022","journal-title":"Smart Health"},{"key":"B12","doi-asserted-by":"publisher","first-page":"443","DOI":"10.3390\/jpm14050443","article-title":"Clearing the fog: a scoping literature review on the ethical issues surrounding artificial intelligence-based medical devices","volume":"14","author":"Maccaro","year":"2024","journal-title":"J Pers Med"},{"key":"B13","doi-asserted-by":"publisher","first-page":"e42187","DOI":"10.2196\/42187","article-title":"GATEKEEPER\u2019s strategy for the multinational large-scale piloting of an eHealth platform: tutorial on how to identify relevant settings and use cases","volume":"25","author":"de Batlle","year":"2023","journal-title":"J Med Internet Res"},{"key":"B14","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1038\/s41746-024-01092-x","article-title":"Towards a common European ethical and legal framework for conducting clinical research: the GATEKEEPER experience","volume":"7","author":"Maccaro","year":"2024","journal-title":"npj Digit Med"},{"key":"B15","doi-asserted-by":"publisher","first-page":"e367","DOI":"10.2196\/jmir.8775","article-title":"Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies","volume":"19","author":"Greenhalgh","year":"2017","journal-title":"J Med Internet Res"},{"key":"B16","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1186\/1748-5908-4-50","article-title":"Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science","volume":"4","author":"Damschroder","year":"2009","journal-title":"Implement Sci"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1322","DOI":"10.2105\/AJPH.89.9.1322","article-title":"Evaluating the public health impact of health promotion interventions: the RE-AIM framework","volume":"89","author":"Glasgow","year":"1999","journal-title":"Am J Public Health"},{"key":"B18","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1186\/s12992-021-00718-z","article-title":"A framework for designing medical devices resilient to low-resource settings","volume":"17","author":"Piaggio","year":"2021","journal-title":"Global Health"},{"key":"B19","volume-title":"Research Methodology: An introduction","author":"Goddard","year":"2004"},{"key":"B20","doi-asserted-by":"crossref","DOI":"10.1596\/0-8213-5823-5","volume-title":"Ten Steps to a Results-based Monitoring and Evaluation System: A Handbook for Development Practitioners","author":"Kusek","year":"2004"},{"key":"B21","author":"Sutner","year":"2024"},{"key":"B22","article-title":"A business intelligence dashboard design approach to improve data analytics and decision making","author":"Orlovskyi","year":"2020"},{"key":"B23","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.im.2003.11.002","article-title":"The delphi method as a research tool: an example, design considerations and applications","volume":"42","author":"Okoli","year":"2004","journal-title":"Inf Manag"},{"key":"B24","first-page":"27","article-title":"Applying the borda ranking method","volume":"20","author":"Lansdowne","year":"1996","journal-title":"Air Fornce J Logist"},{"key":"B25","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1198\/108571105X46642","article-title":"Species associations: the Kendall coefficient of concordance revisited","volume":"10","author":"Legendre","year":"2005","journal-title":"J Agric Biol Environ Stat"},{"key":"B26","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"B27","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1016\/j.jclinepi.2013.12.002","article-title":"Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies","volume":"67","author":"Diamond","year":"2014","journal-title":"J Clin Epidemiol"},{"key":"B28","doi-asserted-by":"publisher","first-page":"763","DOI":"10.1111\/j.1540-5915.1997.tb01330.x","article-title":"Managing Delphi surveys using nonparametric statistical techniques*","volume":"28","author":"Schmidt","year":"1997","journal-title":"Decis Sci"},{"key":"B29","first-page":"10","article-title":"The Delphi technique: making sense of consensus","volume-title":"Pract Assess Res Eval","author":"Hsu","year":"2007"},{"key":"B30","first-page":"200","article-title":"A systematic literature review of key performance indicators (KPIs) implementation","volume":"1","author":"Setiawan","year":"2020","journal-title":"J Ind Eng Manage Res"},{"key":"B31","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.ijinfomgt.2013.09.003","article-title":"Towards an implementation framework for business intelligence in healthcare","volume":"34","author":"Foshay","year":"2014","journal-title":"Int J Inf Manage"},{"key":"B32","doi-asserted-by":"publisher","first-page":"247","DOI":"10.3390\/info13050247","article-title":"Digital transformation in healthcare 4.0: critical factors for business intelligence systems","volume":"13","author":"Kitsios","year":"2022","journal-title":"Information"},{"key":"B33","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1504\/IJBIS.2011.041791","article-title":"Business intelligence systems implementation: testing a critical success factors framework in multiple cases","volume":"8","author":"Yeoh","year":"2011","journal-title":"Int J Bus Inf Syst"},{"key":"B34","doi-asserted-by":"publisher","first-page":"e23499","DOI":"10.2196\/23499","article-title":"Maximizing participant engagement, participation, and retention in cohort studies using digital methods: rapid review to inform the next generation of very large birth cohorts","volume":"23","author":"Nkyekyer","year":"2021","journal-title":"J Med Internet Res"},{"key":"B35","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1186\/s12913-025-12663-3","article-title":"Enhancing digital readiness and capability in healthcare: a systematic review of interventions, barriers, and facilitators","volume":"25","author":"Alotaibi","year":"2025","journal-title":"BMC Health Serv Res"},{"key":"B36","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1108\/10878570410699249","article-title":"Co-creating unique value with customers","volume":"32","author":"Prahalad","year":"2004","journal-title":"Strategy Leadership"},{"key":"B37","doi-asserted-by":"publisher","first-page":"3433","DOI":"10.1080\/00207540500536921","article-title":"From critical success factors to critical success processes","volume":"44","author":"Zwikael","year":"2006","journal-title":"Int J Prod Res"},{"key":"B38","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1108\/02683940710733115","article-title":"The job demands-resources model: state of the art","volume":"22","author":"Bakker","year":"2007","journal-title":"J Manage Psychol"},{"key":"B39","volume-title":"Usability Engineering","author":"Nielsen","year":"1994"},{"key":"B40","volume-title":"Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests","author":"Rubin","year":"2008"},{"key":"B41","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1207\/s15324796abm2902s_9","article-title":"Strategies for retaining study participants in behavioral intervention trials: retention experiences of the NIH behavior change consortium","volume":"29","author":"Coday","year":"2005","journal-title":"Ann Behav Med"},{"key":"B42","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1186\/s12875-023-02176-5","article-title":"The impact of eHealth on relationships and trust in primary care: a review of reviews","volume":"24","author":"Ramachandran","year":"2023","journal-title":"BMC Primary Care"},{"key":"B43","doi-asserted-by":"publisher","first-page":"e16","DOI":"10.2105\/AJPH.2013.301706","article-title":"A systematic review of barriers and facilitators to minority research participation among African Americans, latinos, Asian Americans, and Pacific islanders","volume":"104","author":"George","year":"2014","journal-title":"Am J Public Health"},{"key":"B44","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1111\/j.1540-5885.2009.00341.x","article-title":"Product development and learning in project teams: the challenges are the benefits","volume":"26","author":"Edmondson","year":"2009","journal-title":"J Prod Innov Manage"},{"key":"B45","doi-asserted-by":"publisher","first-page":"160940692211488","DOI":"10.1177\/16094069221148871","article-title":"The voting scheme determines the winner. Considering possible options for designing a delphi study","volume":"22","author":"Ehringfeld","year":"2023","journal-title":"Int J Qual Methods"},{"key":"B46","doi-asserted-by":"publisher","first-page":"e324","DOI":"10.2196\/jmir.8106","article-title":"What is eHealth (6)? Development of a Conceptual Model for eHealth: Qualitative Study with Key Informants","volume":"19","author":"Shaw","year":"2017","journal-title":"J Med Internet Res"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2026.1730232\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T06:08:54Z","timestamp":1780034934000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2026.1730232\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,29]]},"references-count":46,"alternative-id":["10.3389\/fdgth.2026.1730232"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2026.1730232","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,29]]},"article-number":"1730232"}}