{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,31]],"date-time":"2026-10-31T04:44:55Z","timestamp":1793421895105,"version":"4.5.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,2,25]],"date-time":"2021-02-25T00:00:00Z","timestamp":1614211200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2021,2,25]],"date-time":"2021-02-25T00:00:00Z","timestamp":1614211200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000024","name":"Canadian Institutes of Health Research","doi-asserted-by":"publisher","award":["201909PA0-169410"],"award-info":[{"award-number":["201909PA0-169410"]}],"id":[{"id":"10.13039\/501100000024","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001804","name":"Canada Research Chairs","doi-asserted-by":"publisher","award":["950-230706"],"award-info":[{"award-number":["950-230706"]}],"id":[{"id":"10.13039\/501100001804","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Res Involv Engagem"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Patient and public engagement (PPE) in research is growing internationally, and with it, the interest for its evaluation. In Canada, the Strategy for Patient-Oriented Research has generated national momentum and opportunities for greater PPE in research and health-system transformation. As is the case with most countries, the Canadian research community lacks a common evaluation framework for PPE, thus limiting our capacity to ensure integrity between principles and practices, learn across projects, identify common areas for improvement, and assess the impacts of engagement.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Objective<\/jats:title>\n                <jats:p>This project aims to build a national adaptable framework for the evaluation of PPE in research, by:\n<jats:list list-type=\"order\">\n                  \n                    \n                    <jats:list-item>\n                      <jats:p>Building consensus on common evaluation criteria and indicators for PPE in research;<\/jats:p>\n                    <\/jats:list-item>\n                  \n                  \n                    \n                    <jats:list-item>\n                      <jats:p>Defining recommendations to implement and adapt the framework to specific populations.<\/jats:p>\n                    <\/jats:list-item>\n                  \n                <\/jats:list><\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>Using a collaborative action-research approach, a national coalition of patient-oriented research leaders, (patient and community partners, engagement practitioners, researchers and health system leaders) will co-design the evaluation framework. We will develop core evaluation domains of the logic model by conducting a series of virtual consensus meetings using a nominal group technique with 50 patient partners and engagement practitioners, identified through 18 national research organizations. We will then conduct two Delphi rounds to prioritize process and impact indicators with 200 participants purposely recruited to include respondents from seldom-heard groups. Six expert working groups will define recommendations to implement and adapt the framework to research with specific populations, including Indigenous communities, immigrants, people with intellectual and physical disabilities, caregivers, and people with low literacy. Each step of framework development will be guided by an equity, diversity and inclusion approach in an effort to ensure that the participants engaged, the content produced, and the adaptation strategies proposed are relevant to diverse PPE.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Discussion<\/jats:title>\n                <jats:p>The potential contributions of this project are threefold: 1) support a national learning environment for engagement by offering a common blueprint for collaborative evaluation to the Canadian research community; 2) inform the international research community on potential (virtual) methodologies to build national consensus on common engagement evaluation frameworks; and 3) illustrate a shared attempt to engage patients and researchers in a strategic national initiative to strengthen evaluation capacity for PPE.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s40900-021-00255-4","type":"journal-article","created":{"date-parts":[[2021,2,25]],"date-time":"2021-02-25T10:03:44Z","timestamp":1614247424000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Developing a Canadian evaluation framework for patient and public engagement in research: study protocol"],"prefix":"10.1186","volume":"7","author":[{"given":"Audrey","family":"L\u2019Esp\u00e9rance","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nadia","family":"O\u2019Brien","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandre","family":"Gr\u00e9goire","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julia","family":"Abelson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carolyn","family":"Canfield","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claudio","family":"Del Grande","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maman Joyce","family":"Dogba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carol","family":"Fancott","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mary Anne","family":"Levasseur","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christine","family":"Loignon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Annette","family":"Majnemer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marie-Pascale","family":"Pomey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jananee","family":"Rasiah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jon","family":"Salsberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria","family":"Santana","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marie-Claude","family":"Tremblay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robin","family":"Urquhart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7824-8602","authenticated-orcid":false,"given":"Antoine","family":"Boivin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,2,25]]},"reference":[{"key":"255_CR1","series-title":"Canadian Policy Research Networks Ottawa","volume-title":"Assessing the impacts of public participation: concepts, evidence and policy implications","author":"J Abelson","year":"2006","unstructured":"Abelson, J., Gauvin, F.-P. Assessing the impacts of public participation: concepts, evidence and policy implications. Canadian Policy Research Networks Ottawa 2006."},{"key":"255_CR2","doi-asserted-by":"publisher","first-page":"61","DOI":"10.12927\/hcq.2018.25636","volume":"21","author":"J Abelson","year":"2018","unstructured":"Abelson J, Humphrey A, Syrowatka A, Bidonde J, Judd M. Evaluating patient, family and public engagement in health services improvement and system redesign. Healthc Q. 2018;21:61\u20137. https:\/\/doi.org\/10.12927\/hcq.2018.25636.","journal-title":"Healthc Q"},{"key":"255_CR3","doi-asserted-by":"publisher","first-page":"817","DOI":"10.1111\/hex.12378","volume":"19","author":"J Abelson","year":"2016","unstructured":"Abelson J, Li K, Wilson G, Shields K, Schneider C, Boesveld S. Supporting quality public and patient engagement in health system organizations: development and usability testing of the public and patient and engagement evaluation tool. Health Expect. 2016;19:817\u201327.","journal-title":"Health Expect"},{"key":"255_CR4","doi-asserted-by":"publisher","first-page":"1132","DOI":"10.1111\/hex.12949","volume":"22","author":"J Abelson","year":"2019","unstructured":"Abelson J, Tripp L, Kandasamy S, Burrows K, on behalf of the PPEET Implementation study team. Supporting the evaluation of public and patient engagement in health system organizations: results from an implementation research study. Health Expect. 2019;22:1132\u201343.","journal-title":"Health Expect"},{"key":"255_CR5","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1111\/j.1369-7625.2010.00660.x","volume":"15","author":"R Barber","year":"2012","unstructured":"Barber R, Boote J, Parry G, Cooper C, Yeeles P, Cook S. Can the impact of public involvement on research be evaluated? A mixed methods study. Health Expect. 2012;15:229\u201341.","journal-title":"Health Expect"},{"key":"255_CR6","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1017\/S1474746410000072","volume":"9","author":"S Becker","year":"2010","unstructured":"Becker S, Sempik J, Bryman A. Advocates, agnostics and adversaries: researchers\u2019 perceptions of service user involvement in social policy research. Soc Policy Soc. 2010;9:355\u201366. https:\/\/doi.org\/10.1017\/S1474746410000072.","journal-title":"Soc Policy Soc"},{"key":"255_CR7","doi-asserted-by":"publisher","first-page":"k5147","DOI":"10.1136\/bmj.k5147","volume":"363","author":"A Boivin","year":"2018","unstructured":"Boivin A, Richards T, Forsythe L, Gr\u00e9goire A, L\u2019Esp\u00e9rance A, Abelson J, et al. Evaluating patient and public involvement in research. BMJ. 2018a;363:k5147.","journal-title":"BMJ."},{"key":"255_CR8","doi-asserted-by":"publisher","first-page":"1075","DOI":"10.1111\/hex.1280430062858","volume":"21","author":"A Boivin","year":"2018","unstructured":"Boivin A, L\u2019Esp\u00e9rance A, Gauvin F-P, et al. Patient and public engagement in research and health system decision making: a systematic review of evaluation tools. Health Expect. 2018b;21:1075\u201384. https:\/\/doi.org\/10.1111\/hex.1280430062858.","journal-title":"Health Expect"},{"key":"255_CR9","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1111\/j.1369-7625.2012.00795.x22809132","volume":"17","author":"J Brett","year":"2014","unstructured":"Brett J, Staniszewska S, Mockford C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 2014;17:637\u201350. https:\/\/doi.org\/10.1111\/j.1369-7625.2012.00795.x22809132.","journal-title":"Health Expect"},{"key":"255_CR10","volume-title":"Strategy for patient-oriented research\u2014patient engagement framework","author":"Canadian Institutes of Health Research","year":"2014","unstructured":"Canadian Institutes of Health Research. Strategy for patient-oriented research\u2014patient engagement framework. 2014. http:\/\/www.cihr-irsc.gc.ca\/e\/48413.html Accessed 15 July 2019."},{"key":"255_CR11","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1186\/s40900-018-0111-5","volume":"4","author":"A-M Chudyk","year":"2018","unstructured":"Chudyk A-M, et al. Models and frameworks of patient engagement in health services research: a scoping review protocol. Res Involv Engagem. 2018;4:28.","journal-title":"Res Involv Engagem"},{"issue":"10","key":"255_CR12","doi-asserted-by":"publisher","first-page":"889","DOI":"10.15171\/ijhpm.2018.43","volume":"7","author":"V Dukhanin","year":"2018","unstructured":"Dukhanin V, Topazian R, DeCamp M. Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review. Int J Health Policy Manag. 2018;7(10):889\u2013903. https:\/\/doi.org\/10.15171\/ijhpm.2018.43.","journal-title":"Int J Health Policy Manag"},{"issue":"Special Issue","key":"255_CR13","doi-asserted-by":"publisher","first-page":"12","DOI":"10.12927\/hcq.2018.25642","volume":"21","author":"C Fancott","year":"2018","unstructured":"Fancott C, Baker GR, Judd M, Humphrey A, Morin A. Supporting Patient and Family Engagement for Healthcare Improvement: Reflections on \u2018Engagement-Capable Environments\u2019 in Pan-Canadian Learning Collaboratives. Healthcare Q. 2018;21(Special Issue):12\u201330. https:\/\/doi.org\/10.12927\/hcq.2018.25642.","journal-title":"Healthcare Q"},{"key":"255_CR14","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1111\/hex.1248628664563","volume":"20","author":"A Gibson","year":"2017","unstructured":"Gibson A, Welsman J, Britten N. Evaluating patient and public involvement in health research: from theoretical model to practical workshop. Health Expect. 2017;20:826\u201335. https:\/\/doi.org\/10.1111\/hex.1248628664563.","journal-title":"Health Expect"},{"key":"255_CR15","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1111\/hex.12888","volume":"22","author":"T Greenhalgh","year":"2019","unstructured":"Greenhalgh T, et al. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22:785\u2013801.","journal-title":"Health Expect"},{"key":"255_CR16","doi-asserted-by":"publisher","first-page":"396","DOI":"10.1111\/hex.12635","volume":"21","author":"CB Hamilton","year":"2018","unstructured":"Hamilton CB, et al. An empirically based conceptual framework for fostering meaningful patient engagement in research. Health Expect. 2018a;21:396\u2013406.","journal-title":"Health Expect"},{"issue":"11","key":"255_CR17","doi-asserted-by":"publisher","first-page":"e0206588","DOI":"10.1371\/journal.pone.0206588","volume":"13","author":"C Hamilton","year":"2018","unstructured":"Hamilton C, Hoens A, McQuitty S, McKinnon A, English K, Backman C, Azimi T, Khodarahmi N, Li L. Development and pre-testing of the patient engagement in research scale (PEIRS) to assess the quality of engagement from a patient perspective. PLoS One. 2018b;13(11):e0206588. https:\/\/doi.org\/10.1371\/journal.pone.0206588.","journal-title":"PLoS One"},{"key":"255_CR18","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1111\/hex.12873","volume":"22","author":"JD Harrison","year":"2019","unstructured":"Harrison JD, et al. Patient stakeholder engagement in research: a narrative review to describe foundational principles and best practice activities. Health Expect. 2019;22:307\u201316.","journal-title":"Health Expect"},{"key":"255_CR19","volume-title":"Diversity and inclusion: What\u2019s it about and why is it important for public involvement in research?","author":"INVOLVE","year":"2012","unstructured":"INVOLVE. Diversity and inclusion: What\u2019s it about and why is it important for public involvement in research? 2012. https:\/\/www.invo.org.uk\/posttypepublication\/diversity-and-inclusion-what\u2019s-it-about-and-why-is-it-important-for-public-involvement-in-research\/. Accessed 2 July 2020."},{"key":"255_CR20","doi-asserted-by":"publisher","first-page":"326","DOI":"10.1080\/14461242.2016.1149738","volume":"25","author":"A L'Esp\u00e9rance","year":"2016","unstructured":"L'Esp\u00e9rance A, Orsini M. Embodying experience and expertise: comparing mother and intended-mother activism in the cases of infertility and autism. Health Sociol Rev. 2016;25:326\u201340. https:\/\/doi.org\/10.1080\/14461242.2016.1149738.","journal-title":"Health Sociol Rev"},{"key":"255_CR21","doi-asserted-by":"publisher","first-page":"1057","DOI":"10.1007\/s11136-014-0845-y","volume":"24","author":"N Marlett","year":"2015","unstructured":"Marlett N, et al. Building new roles and relationships in research: a model of patient engagement research. Qual Life Res. 2015;24:1057\u201367.","journal-title":"Qual Life Res"},{"key":"255_CR22","unstructured":"Masters J. The History of Action Research. In: Hughes I, editor. Action Research Electronic Reader: The University of Sydney; 1995. http:\/\/www.behs.cchs.usyd.edu.au\/arow\/Reader\/rmasters.htm. Accessed 5 Nov 2019."},{"key":"255_CR23","doi-asserted-by":"publisher","first-page":"899","DOI":"10.1111\/hex.1268429654644","volume":"21","author":"E Mathie","year":"2018","unstructured":"Mathie E, Wythe H, Munday D, et al. Reciprocal relationships and the importance of feedback in patient and public involvement: a mixed methods study. Health Expect. 2018;21:899\u2013908. https:\/\/doi.org\/10.1111\/hex.1268429654644.","journal-title":"Health Expect"},{"key":"255_CR24","unstructured":"Moghaddam RZ, Brian PB, Fu, W-T.\u00a0The Structure of consensus building in online design discussions. 2011. 10p."},{"key":"255_CR25","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1186\/s40900-020-00217-2","volume":"6","author":"P Roche","year":"2020","unstructured":"Roche P, Shimmin C, Hickes S, et al. Valuing all voices: refining a trauma-informed, intersectional and critical reflexive framework for patient engagement in health research using a qualitative descriptive approach. Res Involv Engag. 2020;6:42.","journal-title":"Res Involv Engag"},{"key":"255_CR26","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1177\/0162243903259197","volume":"29","author":"G Rowe","year":"2004","unstructured":"Rowe G, Frewer LJ. Evaluating public-participation exercises: a research agenda. Sci Technol Hum Values. 2004;29:512\u201356.","journal-title":"Sci Technol Hum Values"},{"key":"255_CR27","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1186\/s12913-017-2463-1","volume":"17","author":"C Shimmin","year":"2017","unstructured":"Shimmin C, et al. Moving towards a more inclusive patient and public involvement in health research paradigm: the incorporation of a trauma-informed intersectional analysis. BMC Health Serv Res. 2017;17:539.","journal-title":"BMC Health Serv Res"},{"key":"255_CR28","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1186\/s40900-015-0008-529062495","volume":"1","author":"K Staley","year":"2015","unstructured":"Staley K. \u201cIs it worth doing?\u201d Measuring the impact of patient and public involvement in research. Res Involv Engag. 2015;1:6. https:\/\/doi.org\/10.1186\/s40900-015-0008-529062495.","journal-title":"Res Involv Engag"},{"key":"255_CR29","doi-asserted-by":"publisher","first-page":"628","DOI":"10.1111\/j.1470-6431.2011.01020","volume":"35","author":"S Staniszewska","year":"2011","unstructured":"Staniszewska S, Adebajo A, Barber R, et al. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int J Consum Stud. 2011;35:628\u201332. https:\/\/doi.org\/10.1111\/j.1470-6431.2011.01020.","journal-title":"Int J Consum Stud"},{"issue":"4","key":"255_CR30","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1002\/j.2168-9830.2006.tb00908.x","volume":"95","author":"K-H Tseng","year":"2006","unstructured":"Tseng K-H, Lou S-J, Ray Diez C, Yang H-J. Using online nominal group technique to implement knowledge transfer. J Eng Educ. 2006;95(4):335\u201345.","journal-title":"J Eng Educ."},{"key":"255_CR31","doi-asserted-by":"publisher","first-page":"337","DOI":"10.2105\/AJPH.62.3.337","volume":"62","author":"AH Van de Ven","year":"1972","unstructured":"Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62:337\u201342.","journal-title":"Am J Public Health"},{"key":"255_CR32","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1111\/hex.12951","volume":"21","author":"LE Vat","year":"2020","unstructured":"Vat LE, Finlay T, Jan Schuitmaker-Warnaar T, et al. Evaluating the \u201creturn on patient engagement initiatives\u201d in medicines research and development: a literature review. Health Expect. 2020;21:5\u201318. https:\/\/doi.org\/10.1111\/hex.12951.","journal-title":"Health Expect"}],"container-title":["Research Involvement and Engagement"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40900-021-00255-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s40900-021-00255-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40900-021-00255-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,2,25]],"date-time":"2021-02-25T10:12:03Z","timestamp":1614247923000},"score":1,"resource":{"primary":{"URL":"https:\/\/researchinvolvement.biomedcentral.com\/articles\/10.1186\/s40900-021-00255-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,25]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["255"],"URL":"https:\/\/doi.org\/10.1186\/s40900-021-00255-4","relation":{},"ISSN":["2056-7529"],"issn-type":[{"value":"2056-7529","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,25]]},"assertion":[{"value":"3 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 February 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 February 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This study has been approved by the CHUM research ethics committee. Informed consent will be obtained from all participants during the course of the study.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"10"}}