{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T07:04:32Z","timestamp":1789715072930,"version":"4.0.1"},"update-to":[{"DOI":"10.12688\/f1000research.169999.1","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T00:00:00Z","timestamp":1788998400000}},{"DOI":"10.12688\/f1000research.169999.2","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T00:00:00Z","timestamp":1788998400000}}],"reference-count":83,"publisher":"F1000 Research Ltd","license":[{"start":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T00:00:00Z","timestamp":1788998400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"UNSW Research Infrastructure Scheme"},{"name":"UNSW-UTS Trustworthy Digital Society"},{"name":"MRFF Rapid Applied Research Translation grant","award":["RARUR00125"],"award-info":[{"award-number":["RARUR00125"]}]}],"content-domain":{"domain":["f1000research.com"],"crossmark-restriction":false},"short-container-title":["F1000Res"],"abstract":"<ns3:p>Background The translation of research evidence into routine healthcare practice is often slow and inconsistent, even timely implementation can significantly improve patient outcomes. While implementation science offers strategies to close this gap, current approaches are frequently manual, fragmented, and poorly integrated within healthcare systems. To address these challenges, we propose ImpleMATE \u2013 an AI-assisted implementation science platform designed to streamline implementation efforts. Grounded in the Learning Health System (LHS) model, ImpleMATE aims to establish a continuous, data-driven cycle of learning and improvement in implementation practice. Methods ImpleMATE will be developed through a co-design and co-production approach rooted in human-centred design principles. Development will proceed through four key activities: (1) establishing a data processing pipeline and building an implementation-focused ontology; (2) creating and validating an AI system to extract implementation knowledge, structure the ontology, and support implementation solution delivery; (3) designing an interactive web application to deliver AI-assisted decision support and streamline implementation processes; and (4) developing an evaluation framework to assess platform\u2019s effectiveness and plan for national integration. These activities align with three core components of the LHS model: converting data into knowledge, translating knowledge into practice, and feeding implementation process and outcome data back into the system for continuous learning. The platform will be underpinned by strong ethical and governance frameworks to ensure data privacy, transparency, and responsible AI use. Discussion ImpleMATE aims to transform the adoption of evidence-based innovations in healthcare by embedding trustworthy AI into the core of implementation practice. Through the integration of structured ontologies, real-time AI reasoning, and an interactive user interface, the platform will provide tailored solutions to support implementation efforts. Designed as a dynamic learning system, ImpleMATE will evolve with user input and real-world data, offering a scalable, ethically grounded solution to accelerate and enhance implementation across healthcare settings.<\/ns3:p>","DOI":"10.12688\/f1000research.169999.3","type":"journal-article","created":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T11:49:05Z","timestamp":1789040945000},"page":"1184","update-policy":"https:\/\/doi.org\/10.12688\/f1000research.crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Combining Implementation and Data Sciences to Accelerate Evidence Integration into Healthcare \u2013 ImpleMATE"],"prefix":"10.12688","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7521-9504","authenticated-orcid":false,"given":"Jeffery","family":"Chan","sequence":"first","affiliation":[{"name":"Implementation to Impact (i2i), School of Population Health, Faculty of Medicine and Health Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Janna","family":"Hastings","sequence":"additional","affiliation":[{"name":"Institute for Implementation, Science in Health Care Zurich University of Zurich"},{"name":"School of Medicine St. Gallen University of St. Gallen"},{"name":"Lausanne Swiss Institute of Bioinformatics"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriella","family":"Tiernan","sequence":"additional","affiliation":[{"name":"Implementation to Impact (i2i), School of Population Health, Faculty of Medicine and Health Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeannie","family":"Paterson","sequence":"additional","affiliation":[{"name":"Centre for AI and Digital Ethics Melbourne Victoria The University of Melbourne"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nigel","family":"Lovell","sequence":"additional","affiliation":[{"name":"Graduate School of Biomedical Engineering, Faculty of Engineering Sydney New South Wales University of New South Wales"},{"name":"Tyree Institute of Health Engineering (IHealthE) Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luc","family":"Betbeder-Matibet","sequence":"additional","affiliation":[{"name":"Research Technology Services Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick Kin Man","family":"Tung","sequence":"additional","affiliation":[{"name":"Research Technology Services Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Pink","sequence":"additional","affiliation":[{"name":"Emerging Technologies Lab Melbourne Victoria Monash University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Debora","family":"Lanzeni","sequence":"additional","affiliation":[{"name":"Department of Design Melbourne Victoria Monash University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0127-0091","authenticated-orcid":false,"given":"Thomasina","family":"Donovan","sequence":"additional","affiliation":[{"name":"Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health Brisbane Queensland Queensland University of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Milat","sequence":"additional","affiliation":[{"name":"Sydney New South Wales Agency for Clinical Innovation"},{"name":"School of Public Health Sydney New South Wales University of Sydney"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Louisa","family":"Jorm","sequence":"additional","affiliation":[{"name":"Centre for Big Data Research in Health Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carolyn","family":"Mazariego","sequence":"additional","affiliation":[{"name":"Implementation to Impact (i2i), School of Population Health, Faculty of Medicine and Health Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Implementation to Impact (i2i), School of Population Health, Faculty of Medicine and Health Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7806-814X","authenticated-orcid":false,"given":"Guillaume","family":"Fontaine","sequence":"additional","affiliation":[{"name":"Centre for Implementation Research Ottawa Ottawa Hospital Research Institute"},{"name":"Ingram School of Nursing Montr\u00e9al McGill University"},{"name":"Centre for Clinical Epidemiology Montr\u00e9al Lady Davis Institute for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie","family":"Best","sequence":"additional","affiliation":[{"name":"Melbourne Victoria Australian Genomics"},{"name":"Melbourne School of Health Sciences, Faculty of Medicine, Dentistry and Health Sciences Melbourne Victoria The University of Melbourne"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Georgina","family":"Kennedy","sequence":"additional","affiliation":[{"name":"Faculty of Medicine and Health Sydney New South Wales University of New South Wales"},{"name":"Liverpool, Sydney New South Wales Ingham Institute of Applied Medical Research"},{"name":"Sydney New South Wales Maridulu Budyari Gumal (SPHERE) Cancer Clinical Academic Group"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Susan","family":"Michie","sequence":"additional","affiliation":[{"name":"Centre for Behaviour Change London University College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank","family":"Lin","sequence":"additional","affiliation":[{"name":"Faculty of Medicine and Health Sydney New South Wales University of New South Wales"},{"name":"Department of Medical Oncology Sydney New South Wales Prince of Wales Hospital"},{"name":"Sydney New South Wales Garvan Institute of Medical Research"},{"name":"NHMRC Clinical Trials Centre Sydney New South Wales University of Sydney"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"The ImpleMATE Network","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalie","family":"Taylor","sequence":"additional","affiliation":[{"name":"Implementation to Impact (i2i), School of Population Health, Faculty of Medicine and Health Sydney New South Wales University of New South Wales"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"2560","published-online":{"date-parts":[[2026,9,10]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.3390\/ijms23094645","article-title":"\u201cInnovations in Genomics and Big Data Analytics for Personalized Medicine and Health Care: A Review,\u201d (in eng).","volume":"23","author":"M Hassan","year":"Apr 22 2022","journal-title":"Int. J. Mol. Sci."},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.3390\/biology12070997","article-title":"\u201cNext-Generation Sequencing Technology: Current Trends and Advancements,\u201d (in eng).","volume":"12","author":"H Satam","year":"Jul 13 2023","journal-title":"Biology (Basel)."},{"key":"ref3","doi-asserted-by":"publisher","first-page":"683-692","DOI":"10.1038\/s41586-020-2817-4","article-title":"\u201cStrategic vision for improving human health at The Forefront of Genomics,\u201d (in eng).","volume":"586","author":"E Green","year":"Oct 2020","journal-title":"Nature."},{"key":"ref4","doi-asserted-by":"publisher","first-page":"694-701","DOI":"10.1377\/hlthaff.2017.1624","article-title":"\u201cPrecision Medicine: From Science To Value,\u201d (in eng).","volume":"37","author":"G Ginsburg","year":"2018","journal-title":"Health Aff (Millwood)."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1186\/s13012-014-0200-2","article-title":"\u201cImplementation science in cancer prevention and control: a decade of grant funding by the National Cancer Institute and future directions,\u201d (in eng).","volume":"10","author":"G Neta","year":"Jan 8 2015","journal-title":"Implement. Sci."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"1333-1336","DOI":"10.1001\/jama.2023.4387","article-title":"\u201cIt Takes an Average of 17 Years for Evidence to Change Practice-the Burgeoning Field of Implementation Science Seeks to Speed Things Up,\u201d (in eng).","volume":"329","author":"R Rubin","year":"Apr 25 2023","journal-title":"JAMA."},{"key":"ref7","doi-asserted-by":"crossref","article-title":"Supporting translation of research evidence into practice\u2014the use of Normalisation Process Theory to assess and inform implementation within randomised controlled trials: a systematic review.","author":"A Williams","DOI":"10.1186\/s13012-023-01311-1"},{"key":"ref8","doi-asserted-by":"publisher","first-page":"510-520","DOI":"10.1258\/jrsm.2011.110180","article-title":"\u201cThe answer is 17 years, what is the question: understanding time lags in translational research,\u201d (in eng).","volume":"104","author":"Z Morris","year":"Dec 2011","journal-title":"J. R. Soc. Med."},{"key":"ref9","doi-asserted-by":"publisher","first-page":"12-15","DOI":"10.1159\/000535667","article-title":"\u201cWe Need to Stand Together on the Shoulders of Giants: Consolidating Effective Approaches for Translating Genomics into Practice with Implementation Science,\u201d (in eng).","volume":"27","author":"S Best","year":"2024","journal-title":"Public Health Genomics."},{"key":"ref10","doi-asserted-by":"publisher","first-page":"e18581","DOI":"10.1200\/JCO.2022.40.16_suppl.e18581","article-title":"Two years of the Australian Rare Cancer Portal: a national referral service for rare cancer information and research.","volume":"40","author":"C Georgiou","year":"2022","journal-title":"Proc. Am. Soc. Clin. Oncol."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"e0118253","DOI":"10.1371\/journal.pone.0118253","article-title":"\u201cResearch activity and the association with mortality,\u201d (in eng).","volume":"10","author":"B Ozdemir","year":"2015","journal-title":"PLoS One."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"89-96","DOI":"10.1136\/gutjnl-2015-311308","article-title":"\u201cHigh hospital research participation and improved colorectal cancer survival outcomes: a population-based study,\u201d (in eng).","volume":"66","author":"A Downing","year":"Jan 2017","journal-title":"Gut."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"1941-1942","DOI":"10.1001\/jama.2016.3867","article-title":"\u201cConvergence of Implementation Science, Precision Medicine, and the Learning Health Care System: A New Model for Biomedical Research,\u201d (in eng).","volume":"315","author":"D Chambers","year":"May 10 2016","journal-title":"JAMA."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"e024681","DOI":"10.1136\/bmjopen-2018-024681","article-title":"A transformative translational change programme to introduce genomics into healthcare: a complexity and implementation science study protocol.","volume":"9","author":"N Taylor","year":"2019","journal-title":"BMJ Open."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"866-873","DOI":"10.1038\/s41431-020-0576-1","article-title":"\u201cClinical genomic testing: what matters to key stakeholders?,\" (in eng).","volume":"28","author":"S Best","year":"Jul 2020","journal-title":"Eur. J. Hum. Genet."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"1384-1390","DOI":"10.1038\/s41436-020-0818-1","article-title":"\u201cThe leadership behaviors needed to implement clinical genomics at scale: a qualitative study,\u201d (in eng).","volume":"22","author":"S Best","year":"Aug 2020","journal-title":"Genet. Med."},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.3390\/genes12020317","article-title":"\u201cInvestigating the Adoption of Clinical Genomics in Australia. An Implementation Science Case Study,\u201d (in eng).","volume":"12","author":"S Best","year":"Feb 23 2021","journal-title":"Genes (Basel)."},{"key":"ref19","doi-asserted-by":"publisher","first-page":"1369-1380","DOI":"10.1111\/jep.13573","article-title":"\u201cTeamwork in clinical genomics: A dynamic sociotechnical healthcare setting,\u201d (in eng).","volume":"27","author":"S Best","year":"Dec 2021","journal-title":"J. Eval. Clin. Pract."},{"key":"ref20","doi-asserted-by":"crossref","article-title":"Organizational perspectives on implementing complex health interventions: clinical genomics in Australia.","author":"S Best","DOI":"10.1108\/JHOM-12-2020-0495"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"1415-1424","DOI":"10.1016\/j.gim.2022.03.017","article-title":"\u201cStructured approaches to implementation of clinical genomics: A scoping review,\u201d (in eng).","volume":"24","author":"H Brown","year":"Jul 2022","journal-title":"Genet. Med."},{"key":"ref22","doi-asserted-by":"crossref","article-title":"The demonstration of a theory-based approach to the design of localized patient safety interventions.","author":"N Taylor","DOI":"10.1186\/1748-5908-8-123"},{"key":"ref23","doi-asserted-by":"crossref","article-title":"Collaborating with front-line healthcare professionals: the clinical and cost effectiveness of a theory based approach to the implementation of a national guideline.","author":"N Taylor","DOI":"10.1186\/s12913-014-0648-4"},{"key":"ref24","doi-asserted-by":"publisher","first-page":"72-81","DOI":"10.5694\/mja2.50356","article-title":"The predicted impact and cost-effectiveness of systematic testing of people with incident colorectal cancer for Lynch syndrome.","volume":"212","author":"Y Kang","year":"2020","journal-title":"Med. J. Aust."},{"key":"ref25","doi-asserted-by":"publisher","first-page":"132-147","DOI":"10.1177\/23969873221126027","article-title":"Translation of nurse-initiated protocols to manage fever, hyperglycaemia and swallowing following stroke across Europe (QASC Europe): A pre-test\/post-test implementation study.","volume":"8","author":"S Middleton","year":"2023","journal-title":"Eur. Stroke J."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1186\/s13012-024-01398-0","article-title":"Effects of implementation strategies on nursing practice and patient outcomes: a comprehensive systematic review and meta-analysis.","volume":"19","author":"G Fontaine","year":"2024\/09\/30 2024","journal-title":"Implement. Sci."},{"key":"ref27","doi-asserted-by":"publisher","first-page":"CD005470","DOI":"10.1002\/14651858.CD005470.pub3","article-title":"Tailored interventions to address determinants of practice.","volume":"2015","author":"R Baker","year":"2015","journal-title":"Cochrane Database Syst. Rev."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"2239-2249","DOI":"10.1038\/s41436-021-01272-0","article-title":"\u201cThe design, implementation, and effectiveness of intervention strategies aimed at improving genetic referral practices: a systematic review of the literature,\u201d (in eng).","volume":"23","author":"A Morrow","year":"Dec 2021","journal-title":"Genet. Med."},{"key":"ref29","doi-asserted-by":"publisher","first-page":"136","DOI":"10.3389\/fpubh.2018.00136","article-title":"\u201cFrom Classification to Causality: Advancing Understanding of Mechanisms of Change in Implementation Science,\u201d (in eng).","volume":"6","author":"C Lewis","year":"2018","journal-title":"Front. Public Health."},{"key":"ref30","doi-asserted-by":"crossref","article-title":"Improving measurement-based care implementation in youth mental health through organizational leadership and climate: a mechanistic analysis within a randomized trial.","author":"N Williams","DOI":"10.1186\/s13012-024-01356-w"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"1552","DOI":"10.1186\/s12913-024-11857-5","article-title":"Improving Lynch syndrome detection: a mixed-methods process evaluation of a hybrid type III effectiveness-implementation trial.","volume":"24","author":"A Morrow","year":"2024\/12\/05 2024","journal-title":"BMC Health Serv. Res."},{"key":"ref32","doi-asserted-by":"publisher","first-page":"1-19","DOI":"10.1186\/s12913-024-11857-5","article-title":"Improving Lynch syndrome detection: a mixed-methods process evaluation of a hybrid type III effectiveness-implementation trial.","volume":"24","author":"A Morrow","year":"2024","journal-title":"BMC Health Serv. Res."},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1093\/tbm\/ibab094","article-title":"\u201cBuilding capacity from within: qualitative evaluation of a training program aimed at upskilling healthcare workers in delivering an evidence-based implementation approach,\u201d (in eng).","volume":"12","author":"A Morrow","year":"2022","journal-title":"Transl. Behav. Med."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"g1687","DOI":"10.1136\/bmj.g1687","article-title":"\u201cBetter reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide,\u201d (in eng).","volume":"348","author":"T Hoffmann","year":"Mar 7 2014","journal-title":"BMJ."},{"key":"ref35","doi-asserted-by":"crossref","article-title":"Leveraging academic initiatives to advance implementation practice: a scoping review of capacity building interventions.","author":"L Juckett","DOI":"10.1186\/s13012-022-01216-5"},{"key":"ref36","doi-asserted-by":"publisher","first-page":"588","DOI":"10.1186\/s12913-020-05434-9","article-title":"The barriers and facilitators influencing the sustainability of hospital-based interventions: a systematic review.","volume":"20","author":"J Cowie","year":"2020\/06\/28 2020","journal-title":"BMC Health Serv. Res."},{"key":"ref37","doi-asserted-by":"publisher","first-page":"e087926","DOI":"10.1136\/bmjopen-2024-087926","article-title":"\u201cTrials and tribulations: a qualitative exploration of researcher perspectives on navigating the challenges of health system implementation research,\u201d (in eng).","volume":"15","author":"A Morrow","year":"Jan 15 2025","journal-title":"BMJ Open."},{"key":"ref40","doi-asserted-by":"crossref","article-title":"Closing the gap: advancing implementation science through training and capacity building.","author":"R Brownson","DOI":"10.1186\/s13012-024-01371-x"},{"key":"ref41","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s13012-025-01436-5","article-title":"Advancing the selection of implementation science theories, models, and frameworks: a scoping review and the development of the SELECT-IT meta-framework.","volume":"20","author":"G Fontaine","year":"2025\/05\/28 2025","journal-title":"Implement. Sci."},{"key":"ref42","doi-asserted-by":"publisher","first-page":"113-153","DOI":"10.1016\/j.jiixd.2024.01.002","article-title":"Automated data processing and feature engineering for deep learning and big data applications: A survey.","volume":"3","author":"A Mumuni","year":"2025\/03\/01\/2025","journal-title":"J. Inf. Intell."},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.3390\/computers12020037","article-title":"Artificial Intelligence and Sentiment Analysis: A Review in Competitive Research.","volume":"12","author":"H Taherdoost","journal-title":"Computers."},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.3390\/cancers14051150","article-title":"\u201cMultiomics Topic Modeling for Breast Cancer Classification,\u201d (in eng).","volume":"14","author":"F Valle","year":"Feb 23 2022","journal-title":"Cancers (Basel)."},{"key":"ref45","doi-asserted-by":"publisher","first-page":"lsaa010","DOI":"10.1093\/jlb\/lsaa010","article-title":"\u201cData protection and ethics requirements for multisite research with health data: a comparative examination of legislative governance frameworks and the role of data protection technologies,\u201d (in eng).","volume":"7","author":"J Scheibner","year":"Jan-Jun 2020","journal-title":"J. Law Biosci."},{"key":"ref47","doi-asserted-by":"publisher","first-page":"123","DOI":"10.12688\/wellcomeopenres.15902.2","article-title":"Representation of behaviour change interventions and their evaluation: Development of the Upper Level of the Behaviour Change Intervention Ontology [version 2; peer review: 2 approved].","volume":"5","author":"S Michie","year":"2021","journal-title":"Wellcome Open Res."},{"key":"ref48","doi-asserted-by":"crossref","article-title":"The Human Behaviour-Change Project: harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation.","author":"S Michie","DOI":"10.1186\/s13012-017-0641-5"},{"key":"ref49","doi-asserted-by":"publisher","first-page":"308","DOI":"10.12688\/wellcomeopenres.19363.1","article-title":"\u201cThe Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1,\u201d (in eng).","volume":"8","author":"M Marques","year":"2023","journal-title":"Wellcome Open Res."},{"key":"ref50","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1007\/s42979-021-00592-x","article-title":"\u201cMachine Learning: Algorithms, Real-World Applications and Research Directions,\u201d (in eng).","volume":"2","author":"I Sarker","year":"2021","journal-title":"SN Comput. Sci."},{"key":"ref52","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1186\/s13643-024-02575-4","article-title":"Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain.","volume":"13","author":"F Dennst\u00e4dt","year":"2024\/06\/15 2024","journal-title":"Syst. Rev."},{"key":"ref53","doi-asserted-by":"publisher","DOI":"10.1101\/2024.12.03.24318390","article-title":"Application of a general LLM-based classification system to retrieve information about oncological trials.","author":"F Dennst\u00e4dt","year":"2024","journal-title":"medRxiv."},{"key":"ref55","article-title":"Language Models are Few-Shot Learners.","volume":"2020","author":"T Brown","year":"2020","journal-title":"arXiv [cs.CL]."},{"key":"ref56","article-title":"Mistral 7B.","volume":"2023","author":"A Jiang","year":"2023","journal-title":"arXiv [cs.CL]."},{"key":"ref57","article-title":"LLaMA: Open and Efficient Foundation Language Models.","volume":"2023","author":"H Touvron","year":"2023","journal-title":"arXiv [cs.CL]."},{"key":"ref58","doi-asserted-by":"publisher","first-page":"btae104","DOI":"10.1093\/bioinformatics\/btae104","article-title":"Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES): a method for populating knowledge bases using zero-shot learning.","volume":"40","author":"J Harry Caufield","year":"2024","journal-title":"Bioinformatics."},{"key":"ref59","doi-asserted-by":"publisher","first-page":"e60665","DOI":"10.2196\/60665","article-title":"\u201cAn Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontology-Enhanced Large Language Models: Development Study,\u201d (in eng).","volume":"12","author":"L Cao","year":"Dec 18 2024","journal-title":"JMIR Med. Inform."},{"key":"ref60","article-title":"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.","volume":"2023","author":"J Wei","year":"2023","journal-title":"arXiv [cs.CL]."},{"key":"ref61","doi-asserted-by":"publisher","first-page":"ooab084","DOI":"10.1093\/jamiaopen\/ooab084","article-title":"\u201cA natural language processing pipeline to synthesize patient-generated notes toward improving remote care and chronic disease management: a cystic fibrosis case study,\u201d (in eng).","volume":"4","author":"S Hussain","year":"Jul 2021","journal-title":"JAMIA Open."},{"key":"ref62","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1186\/s13012-021-01120-4","article-title":"5335 days of Implementation Science: using natural language processing to examine publication trends and topics.","volume":"16","author":"J Scaccia","year":"2021\/04\/26 2021","journal-title":"Implement. Sci."},{"key":"ref63","doi-asserted-by":"publisher","first-page":"102096","DOI":"10.1016\/j.artmed.2021.102096","article-title":"\u201cEvaluation of clustering and topic modeling methods over health-related tweets and emails,\u201d (in eng).","volume":"117","author":"J Lossio-Ventura","year":"Jul 2021","journal-title":"Artif. Intell. Med."},{"key":"ref65","doi-asserted-by":"publisher","DOI":"10.17061\/phrp3222211","article-title":"\u201cCo-creation, co-design, co-production for public health - a perspective on definition and distinctions,\u201d (in eng).","volume":"32","author":"C Vargas","year":"Jun 15 2022","journal-title":"Public Health Res Pract."},{"key":"ref66","doi-asserted-by":"crossref","article-title":"An umbrella review on how digital health intervention co-design is conducted and described.","author":"A Kilfoy","DOI":"10.1038\/s41746-024-01385-1"},{"key":"ref67","doi-asserted-by":"publisher","first-page":"e28102","DOI":"10.2196\/28102","article-title":"\u201cThe Application of Human-Centered Design Approaches in Health Research and Innovation: A Narrative Review of Current Practices,\u201d (in eng).","volume":"9","author":"I G\u00f6ttgens","year":"Dec 6 2021","journal-title":"JMIR Mhealth Uhealth."},{"key":"ref68","article-title":"Co-design toolkit."},{"key":"ref69","doi-asserted-by":"publisher","first-page":"26-33","DOI":"10.1136\/qshc.2004.011155","article-title":"\u201cMaking psychological theory useful for implementing evidence based practice: a consensus approach,\u201d (in eng).","volume":"14","author":"S Michie","year":"Feb 2005","journal-title":"Qual. Saf. Health Care."},{"key":"ref70","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1186\/1748-5908-7-37","article-title":"Validation of the theoretical domains framework for use in behaviour change and implementation research.","volume":"7","author":"J Cane","year":"2012\/04\/24 2012","journal-title":"Implement. Sci."},{"key":"ref71","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1186\/1748-5908-4-50","article-title":"Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.","volume":"4","author":"L Damschroder","year":"2009\/08\/07 2009","journal-title":"Implement. Sci."},{"key":"ref72","doi-asserted-by":"crossref","article-title":"Combined use of the Consolidated Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF): a systematic review.","author":"S Birken","DOI":"10.1186\/s13012-016-0534-z"},{"key":"ref73","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1186\/s13012-022-01245-0","article-title":"The updated Consolidated Framework for Implementation Research based on user feedback.","volume":"17","author":"L Damschroder","year":"2022\/10\/29 2022","journal-title":"Implement. Sci."},{"key":"ref74","article-title":"Latent Dirichlet Allocation.","volume":"3","author":"D Blei","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref75","article-title":"BERTopic: Neural topic modeling with a class-based TF-IDF procedure."},{"key":"ref76","doi-asserted-by":"publisher","DOI":"10.1093\/neuonc\/noae064.334","article-title":"HGG-50. LOW GRADE GLIOMA CAUGHT IN TRANSFORMATION: UTILITY OF COMPREHENSIVE GENOMIC PROFILING IN PEDIATRIC LI FRAUMENI SYNDROME ASSOCIATED HIGH-GRADE GLIOMA.","volume":"26","author":"S Nagabushan","year":"2024","journal-title":"Neuro-Oncology."},{"key":"ref77","doi-asserted-by":"publisher","first-page":"65-76","DOI":"10.1007\/s10488-010-0319-7","article-title":"\u201cOutcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda,\u201d (in eng).","volume":"38","author":"E Proctor","year":"Mar 2011","journal-title":"Admin. Pol. Ment. Health."},{"key":"ref78","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1186\/s13012-025-01423-w","article-title":"Development of an instrument (Cost-IS) to estimate costs of implementation strategies for digital health solutions: a modified e-Delphi study.","volume":"20","author":"T Donovan","year":"2025\/03\/07 2025","journal-title":"Implement. Sci."},{"key":"ref79","doi-asserted-by":"publisher","first-page":"114","DOI":"10.1186\/s43058-024-00648-y","article-title":"Challenges and recommendations for collecting and quantifying implementation costs in practice: a qualitative interview study.","volume":"5","author":"T Donovan","year":"2024\/10\/11 2024","journal-title":"Implement. Sci. Commun."},{"key":"ref80","doi-asserted-by":"publisher","first-page":"1251-1255","DOI":"10.1038\/nbt1346","article-title":"\u201cThe OBO Foundry: coordinated evolution of ontologies to support biomedical data integration,\u201d (in eng).","volume":"25","author":"B Smith","year":"Nov 2007","journal-title":"Nat. Biotechnol."},{"key":"ref81","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1038\/s41746-025-01476-7","article-title":"Implementing large language models in healthcare while balancing control, collaboration, costs and security.","volume":"8","author":"F Dennst\u00e4dt","year":"2025\/03\/06 2025","journal-title":"NPJ Digit. Med."},{"key":"ref82","doi-asserted-by":"crossref","article-title":"The sum is the realisation of the parts: interdisciplinary perspectives on care.","author":"L Vaughan","DOI":"10.21606\/drs.2020.118"},{"key":"ref83","article-title":"Smart Homes for Seniors: Intelligent Home Solutions for Independent Living.","author":"Y Strengers"},{"key":"ref84","doi-asserted-by":"publisher","first-page":"07\/12","DOI":"10.1051\/e3sconf\/202339904037","article-title":"Human-Computer Interaction: Enhancing User Experience in Interactive Systems.","volume":"399","author":"R Pushpakumar","year":"2023","journal-title":"E3S Web of Conferences."},{"key":"ref85","doi-asserted-by":"publisher","first-page":"24","DOI":"10.5334\/ijic.5529","article-title":"\u201cEvaluation is Key: Providing Appropriate Evaluation Measures for Participatory and User-Centred Design Processes of Healthcare IT,\u201d (in eng).","volume":"21","author":"L Harst","year":"Jun 21 2021","journal-title":"Int. J. Integr. Care."},{"key":"ref86","doi-asserted-by":"publisher","first-page":"1932-1937","DOI":"10.1016\/j.promfg.2015.07.237","article-title":"Enhancing User Experience Through Customisation of UI Design.","volume":"3","author":"S Hui","year":"2015\/01\/01\/2015","journal-title":"Procedia Manuf."},{"key":"ref87","doi-asserted-by":"publisher","first-page":"561-570","DOI":"10.1093\/tbm\/ibad009","article-title":"\u201cIn person and virtual process mapping experiences to capture and explore variability in clinical practice: application to genetic referral pathways across seven Australian hospital networks,\u201d (in eng).","volume":"13","author":"A Morrow","year":"Aug 11 2023","journal-title":"Transl. Behav. Med."},{"key":"ref88","doi-asserted-by":"publisher","first-page":"100109","DOI":"10.1016\/j.gim.2022.01.004","article-title":"\u201cStandardizing variation: Scaling up clinical genomics in Australia,\u201d (in eng).","volume":"25","author":"S Best","year":"Feb 2023","journal-title":"Genet. Med."},{"key":"ref89","article-title":"Harnessing data science and artificial intelligence to advance implementation research and practice.","author":"J Chan","year":"2026","journal-title":"JBI Evid. Implement."},{"key":"ref90","doi-asserted-by":"publisher","DOI":"10.31222\/osf.io\/3z2ec_v1","article-title":"Prompt-driven large language model for evidence extraction in implementation science: exploring semantic similarity between manual and automated coding across studies.","author":"C Zhang","year":"2026 preprint"}],"container-title":["F1000Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3\/iparadigms","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T06:49:09Z","timestamp":1789714149000},"score":1,"resource":{"primary":{"URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,10]]},"references-count":83,"URL":"https:\/\/doi.org\/10.12688\/f1000research.169999.3","relation":{"has-version":[{"id-type":"doi","id":"10.12688\/f1000research.169999.2","asserted-by":"subject"},{"id-type":"doi","id":"10.12688\/f1000research.169999.1","asserted-by":"subject"}],"has-review":[{"id-type":"doi","id":"10.5256\/f1000research.209793.r520122","asserted-by":"subject"},{"id-type":"doi","id":"10.5256\/f1000research.209793.r520122","asserted-by":"object"}]},"ISSN":["2046-1402"],"issn-type":[{"value":"2046-1402","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,10]]},"assertion":[{"value":"Indexed","URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3#article-reports","order":0,"name":"referee-status","label":"Referee status","group":{"name":"current-referee-status","label":"Current Referee Status"}},{"value":"10.5256\/f1000research.187409.r440859, Yogeesh Nijalingappa, Government First Grade College, Tumkur, 16 Dec 2025, version 1, 2 approved with reservations","URL":"https:\/\/f1000research.com\/articles\/14-1184\/v1#referee-response-440859","order":0,"name":"referee-response-440859","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"<b>Chi Ho Chan<\/b>; \n<i>Posted: 06 Aug 2026<\/i>; We sincerely thank the reviewer for the careful evaluation of our manuscript and for the constructive and insightful comments. We appreciate the time and effort invested in reviewing our work. We have carefully addressed all comments and have revised the manuscript accordingly.  \n<b>Major comments<\/b> \n<b>1. Evaluation plan is not sufficiently defined<\/b> The manuscript mentions KPIs and broad goals, but it does not specify a clear evaluation study design. \n<b>Must add:<\/b>&nbsp;primary outcomes, sample\/setting, comparison condition (if any), study type (pilot\/pre-post\/mixed methods), and a short analysis plan.  \n<b><u>Response:<\/u><\/b> We thank the reviewer for this helpful suggestion. We agree that the original manuscript did not provide sufficient evaluation plan. We have now substantially expanded&nbsp;\n<b>Activity 4 (Design an evaluation framework and roadmap for national integration)<\/b>&nbsp;to specify the evaluation methodology, including the primary and secondary evaluation outcomes for the KPIs, including AI recommendation quality, implementation planning efficiency, usability, user satisfaction, implementation outcomes, and organisational readiness&nbsp; (\n<b>Page 21<\/b>, \u201cPrimary evaluation outcomes will include the accuracy, relevance\u2026\u201d), setting and study design (\n<b>Page 21<\/b>, \u201cWe will conduct a mixed-methods pilot evaluation within implementation projects initially focused on cancer genetics and genomics\u2026\u201d), and quantitative and qualitative analysis plan (\n<b>Page 21<\/b>, \u201cQuantitative outcomes will be analysed using\u2026\u201d). We also describe how the findings will inform iterative refinement prior to broader health system implementation (\n<b>Page 22<\/b>, \u201cThe roadmap for health system integration will follow a phased implementation approach\u2026 Findings from this evaluation will be used to iteratively refine the platform\u2026\u201d).   \n<b>2.&nbsp;Insufficient methodological detail for replication (AI + annotation)<\/b> The protocol states that expert annotation will be used, but does not describe document inclusion criteria, annotation guidelines, inter-rater reliability targets, adjudication, and train\/test splits. \n<b>Must add:<\/b>&nbsp;clear dataset selection rules, annotation process, reliability metrics, and evaluation metrics for extraction\/recommendation performance.  \n<b><u>Response:<\/u><\/b> We thank the reviewer for this helpful suggestion. We agree that additional methodological detail will improve the reproducibility of the proposed AI development and evaluation process. We have expanded&nbsp;\n<b>Activity 2<\/b>&nbsp;to provide a more detailed description of dataset selection (\n<b>Page 14<\/b>, \u201cDocuments will be purposively selected to represent a broad range of implementation studies\u2026\u201d), expert annotation (\n<b>Page 14<\/b>, \u201cAnnotation will be undertaken using a standardised annotation guideline\u2026\u201d), and model evaluation (\n<b>Pages 16 and 17<\/b>, \u201cThe annotated dataset will be randomly divided into training\u2026performance metrics, including precision\u2026\u201d). We now describe the document selection criteria used to construct the gold-standard dataset, including the inclusion of diverse implementation studies, healthcare settings, intervention types, and implementation contexts. We have also clarified that annotation will be conducted using a standardised annotation guideline derived from the implementation ontology, with pilot annotation undertaken to refine the guideline before full annotation. To strengthen annotation quality, we specify that each document will be independently annotated by at least two implementation science experts, with inter-rater agreement assessed using Cohen's kappa and disagreements resolved through consensus or adjudication by a senior implementation scientist. We further clarify that the final consensus annotations will form the gold-standard dataset used for AI development and evaluation (\n<b>Page 15<\/b>, \u201cEach document will be independently annotated by at least two implementation science experts\u2026\u201d).   \n<b>3. LLM safety\/traceability needs stronger operational detail<\/b> Hallucination is acknowledged, but there is no detailed mitigation approach. \n<b>Must add:<\/b>&nbsp;evidence-grounded outputs (citations to source passages\/ontology nodes), human-in-the-loop review points, and explicit limitations that the system supports implementation planning (not clinical decision-making).  \n<b><u>Response:<\/u><\/b> We thank the reviewer for this valuable suggestion. We have strengthened the manuscript to provide additional operational detail regarding LLM safety and traceability. We now clarify that AI-generated outputs will be grounded in the curated implementation ontology, with implementation recommendations linked to the supporting implementation evidence and relevant implementation concepts wherever appropriate (\n<b>Page 16<\/b>, \u201cTo promote trustworthy AI, ImpleMATE will adopt an evidence-grounded approach\u2026\u201d). We have also expanded the description of the human-in-the-loop review process, clarifying that expert validation will occur throughout ontology development, implementation concept extraction, knowledge curation, and platform evaluation before AI-generated outputs are incorporated into the knowledge base or presented through the web application (\n<b>Page 16<\/b>, \u201contology updates and knowledge curation will remain subject to expert review through a human-in-the-loop process\u2026\u201d; \n<b>Page 17<\/b>, \u201cHuman-in-the-loop review will be incorporated throughout ontology development\u2026\u201d). Finally, we explicitly state that ImpleMATE is intended to support implementation planning and implementation decision support rather than clinical decision-making, and that all AI-generated outputs require appropriate human oversight (\n<b>Page 17<\/b>, \u201cImportantly, ImpleMATE is intended to support implementation decision-making rather than provide clinical diagnosis or direct patient management recommendations\u2026\u201d).   \n<b>4. Data governance and ethics need more concrete procedures<\/b> Governance principles are described, but practical details for using real-world project data are limited. \n<b>Must add:<\/b>&nbsp;consent\/ethics pathway, de-identification workflow, access control, audit logging, and data retention policies.  \n<b><u>Response:<\/u><\/b> We appreciate this suggestion. We have strengthened the description of data governance procedures in \n<b>Activity 1 (Ethical sharing of project data and responsible use of AI)<\/b>&nbsp;by providing additional detail regarding the use of real-world implementation project data. We now clarify that implementation project data will only be used following the relevant institutional ethics and governance approvals and participant consent where required. We also describe the de-identification workflow, role-based access controls, audit logging, and institutional data retention and disposal procedures (\n<b>Page 14<\/b>, \u201cImpleMATE will implement a secure data management framework that protects sensitive implementation project and health data\u2026\u201d). These additions complement the existing description of secure computing infrastructure and privacy safeguards (\n<b>Page 14<\/b>, \u201cImpleMATE will prioritise locally hosted AI models operating within secure computing environments\u2026\u201d).   \n<b>5.&nbsp;Reproducibility and sharing plan is missing<\/b> The paper states no associated data\/software. For an AI platform protocol, this weakens reproducibility. \n<b>Must add:<\/b>&nbsp;at least an intention to share the ontology, sample de-identified\/synthetic examples, and evaluation scripts (or provide controlled-access details).  \n<b><u>Response:<\/u><\/b> We thank the reviewer for this recommendation. We agree that transparency and reproducibility are important considerations for AI-enabled research. Therefore, we have added a statement to the \n<b>Discussion<\/b> outlining our intention to disseminate key research outputs arising from ImpleMATE, including the implementation ontology, de-identified or synthetic example datasets where appropriate, and documentation describing the AI evaluation framework and performance metrics. We also clarify that, where implementation project data or software components cannot be publicly released because of ethical, privacy, or intellectual property considerations, access will be managed through appropriate controlled-access arrangements in accordance with institutional governance requirements and ethics approvals (\n<b>Page 24<\/b>, \u201cTo support transparency and reproducibility\u2026\u201d).   \n<b>Minor comments<\/b> \u2022 Please check an affiliation line that appears incorrect (\u201cZurich, Swaziland\u201d).  \n<b><u>Response:<\/u><\/b> Thank you for identifying this error. The affiliation has been corrected to &quot;Zurich, Switzerland.&quot;   \u2022&nbsp;Improve consistency in terminology across sections (platform vs decision aid vs coaching tool).  \n<b><u>Response:<\/u><\/b> We appreciate this observation. We have reviewed the manuscript and standardised terminology throughout to consistently describe ImpleMATE as an AI-assisted implementation science platform comprising an AI-assisted decision aid and implementation coaching and data sharing platform.","URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3#referee-comment-16860","order":1,"name":"referee-comment-16860","label":"Referee Comment","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.5256\/f1000research.209793.r520122, Tuan D. Pham, Queen Mary University of London,, London, 18 Sep 2026, version 3, indexed","URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3#referee-response-520122","order":2,"name":"referee-response-520122","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.5256\/f1000research.208120.r440866, Omid Panahi, Centro Escolar University, Manila, Metro Manila, 17 Aug 2026, version 2, 3 approved with reservations","URL":"https:\/\/f1000research.com\/articles\/14-1184\/v2#referee-response-440866","order":3,"name":"referee-response-440866","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"<b>Chi Ho Chan<\/b>; \n<i>Posted: 01 Sep 2026<\/i>; We appreciate the reviewer\u2019s feedback that the manuscript has substantially improved. We believe these revisions provide a clearer and more reproducible specification of the planned AI pipeline, pilot evaluation, safety assessment, and future sharing arrangements.  \n<b>Some methodological issues should still be addressed to make the protocol sufficiently rigorous and reproducible. The most important remaining concern is that, although the workflow is now described in considerably greater detail, several operational aspects of the AI development and evaluation pipeline remain insufficiently specified for independent replication. The authors should provide, where feasible, more explicit information on the planned model architecture\/model selection, key hyperparameters or tuning strategy, prompt-development strategy for LLM components, exact train\/validation\/test procedures, prevention of data leakage, and the precise performance metrics and decision thresholds that will be used for extraction and recommendation tasks. These details are particularly important because the study's contribution depends substantially on the reliability and reproducibility of the proposed AI pipeline.<\/b>  \n<b><u>Response:<\/u><\/b> We thank the reviewer for this suggestion. We agree that further operational detail would strengthen the reproducibility of the proposed AI development and evaluation. We now specify the criteria that will inform model selection and provide the key hyperparameters, including learning rate, batch size, training epochs, etc., that will be evaluated during model development (Page 16 \u2013 Activity 2, \u201cFor the encoder-only concept extractor transformer model\u2026\u201d). The selection will be based on evaluation performance, computational feasibility, suitability for secure local deployment, and potential licensing. For the prompt development strategy, we added an iterative and documented process starting with defining task-specific instructions and output schemas to prompt refinement and selection (Pages 17,18 - Activity 2, \u201cPrompt development will follow an iterative, documented process\u2026\u201d) We also added more details about train\/validation\/test procedures and prevention of data leakage (Page 17 \u2013 Activity 2, \u201cThe training dataset will be used for model development and fine-tuning\u2026\u201d, \u201cThe split will be stratified by source type and implementation concept category\u2026\u201d, \u201c\u2026all documents from that project\/study will be assigned to the same partition to prevent information leakage\u2026\u201d). Regarding decision thresholds, for concept extraction tasks that produce confidence scores, thresholds for accepting extracted concepts will be selected using the validation dataset based on the prespecified performance metrics and subsequently fixed before evaluation on the independent test dataset (Page 17 \u2013 Activity 2, \u201cFor concept extraction tasks that generate confidence scores\u2026\u201d). For the recommendation tasks, because implementation recommendations are generative outputs rather than conventional classification predictions, a single numerical classification threshold is not directly applicable. Instead, recommendation outputs will be evaluated against traceability, and expert-review criteria (Page 17 \u2013 Activity 2, \u201cFor generative implementation recommendations\u2026 expert assessment of relevance and appropriateness.\u201d).   \n<b>The revised evaluation framework is a clear improvement, with specified outcomes, a mixed-methods pilot design, and quantitative and qualitative analyses. Nevertheless, the authors should further clarify the anticipated sample size or recruitment target, participant selection criteria, comparator or baseline condition where applicable, and how missing data and potential sources of bias will be handled. If a formal sample-size calculation is not appropriate for the planned pilot, this should be explicitly justified.<\/b>  \n<u><b>Response:<\/b><\/u> We thank the reviewer for this valuable suggestion. We have clarified the pilot evaluation design in Activity 4. We now specify an anticipated recruitment target of approximately 30 end users, purposively sampled to represent diverse user groups. Eligibility is defined as involvement in implementation planning, delivery, evaluation, or evidence translation (Page 22, \u201cThe pilot will recruit approximately 30 end users using purposive sampling to represent the intended user groups\u201d). We explicitly state that no formal sample size will be undertaken because the pilot is intended to assess feasibility, usability, technical performance, and preliminary signals of effectiveness rather than provide a definitive hypothesis test. The approximately 30-participant target is intended to provide representation across the principal user groups while remaining feasible for qualitative and task-based assessment. The comparator has also been clarified. Where feasible, participants will undertake comparable implementation tasks using both conventional approaches and ImpleMATE-supported approaches. Where a within-project comparison is not feasible, baseline or historical implementation data will be used as the comparator (Page 22, \u201cWhere feasible, participants will complete comparable tasks using both approaches\u2026\u201d). Missing-data procedures and potential biases are now described. Missingness will be summarised by outcome and participant group; analyses will use available data. AI performance outcomes will not be imputed when reference annotations or expert ratings are unavailable. We also identify and describe mitigation of different biases (Pages 22,23, \u201cMissing data and sources of bias will be documented\u2026\u201d).&nbsp;   \n<b>The manuscript also appropriately strengthens AI safety and human oversight. The evidence-grounded approach, expert review, and explicit distinction between implementation decision support and clinical decision-making are important strengths. The authors should nevertheless ensure that these safeguards are translated into measurable evaluation criteria, including how hallucinations, unsupported recommendations, and traceability failures will be detected and reported.<\/b>  \n<u><b>Response:<\/b><\/u> Thank you very much for this suggestion. We agree with the reviewer and have added specific procedures for these safeguards in Activity 2. Hallucination and unsupported recommendations are now defined as a factual statement or implementation recommendation that cannot be supported by or traced to the source material and curated ontology. Traceability completeness will be calculated as the proportion of factual claims or recommendations for which the system provides a valid link to the supporting evidence and\/or relevant ontology concept.&nbsp; A stratified sample of AI-generated outputs will be independently reviewed by two implementation science experts, with disagreements resolved through consensus or senior adjudication. We will report the number and proportion of hallucinations, unsupported recommendations, and traceability failures, with 95% confidence intervals where sample size permits. For the pilot decision-support evaluation, outputs must meet the evidence-support and traceability requirements and the prespecified expert-rating criteria to be considered acceptable; outputs failing these criteria will be flagged for review and excluded from the knowledge base until corrected and revalidated. (Page 18, \u201cAI safety outputs will be assessed and reported using prespecified measurable indicators\u2026\u201d)   \n<b>Finally, the manuscript should provide a clearer reproducibility and sharing plan. The previous review identified the absence of associated data\/software as a limitation and recommended sharing the ontology, de-identified or synthetic examples, and evaluation scripts where possible. Since this is a protocol and no dataset is currently provided, the dataset-accessibility question is appropriately considered Not Applicable; however, a concrete future data\/software sharing plan would substantially strengthen transparency and reproducibility.<\/b>  \n<u><b>Response:<\/b><\/u> We appreciate this suggestion to strengthen transparency and reproducibility. We have strengthened the reproducibility and sharing plan in the Discussion section. The planned outputs now include versioned releases of the implementation ontology and its documentation, annotation guidelines, data dictionaries, prompt templates, de-identified or synthetic example datasets where permissible, evaluation scripts, analysis code, and selected software components. Model specifications and configuration information will be documented to support replication using compatible locally hosted models. (Pages 25,26 \u201cTo support transparency and reproducibility\u2026\u201d)","URL":"https:\/\/f1000research.com\/articles\/14-1184\/v3#referee-comment-17063","order":4,"name":"referee-comment-17063","label":"Referee Comment","group":{"name":"article-reports","label":"Article Reports"}},{"value":"The project is part-funded by a UNSW Research Infrastructure Scheme: Supporting Collaborative Research 2025 and UNSW-UTS Trustworthy Digital Society; and supported by an MRFF Rapid Applied Research Translation grant (RARUR00125). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.","order":5,"name":"grant-information","label":"Grant Information"},{"value":"This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.","order":0,"name":"copyright-info","label":"Copyright"}]}}