{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T10:45:35Z","timestamp":1777027535686,"version":"3.51.4"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"3","funder":[{"DOI":"10.13039\/100006093","name":"PCORI","doi-asserted-by":"publisher","award":["CDRN-1306-04819"],"award-info":[{"award-number":["CDRN-1306-04819"]}],"id":[{"id":"10.13039\/100006093","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,5,1]]},"abstract":"<jats:p>Objective: Little evidence exists about effective and scalable methods for meaningful stakeholder engagement in research. We explored patient\/caregiver experiences with a high-tech online engagement approach for patient-centered research prioritization, compared their experiences with those of professional stakeholders, and identified factors associated with favorable participant experiences.<\/jats:p>\n               <jats:p>Methods: We conducted 8 online modified-Delphi (OMD) panels. Panelists participated in 2 rating rounds with a statistical feedback\/online discussion round in between. Panels focused on weight management\/obesity, heart failure, and Kawasaki disease. We recruited a convenience sample of adults with any of the 3 conditions (or parents\/guardians of Kawasaki disease patients), clinicians, and researchers. Measures included self-reported willingness to use OMD again, the panelists\u2019 study participation and online discussion experiences, the system\u2019s perceived ease of use, and active engagement metrics.<\/jats:p>\n               <jats:p>Results: Out of 349 panelists, 292 (84%) completed the study. Of those, 46% were patients, 36% were clinicians, and 19% were researchers. In multivariate models, patients were not significantly more actively engaged (Odds ratio (OR)\u2009=\u20091.69, 95% confidence interval (CI), 0.94\u20133.05) but had more favorable study participation (\u03b2\u2009=\u20090.49; P\u2009\u2264\u2009.05) and online discussion (\u03b2\u2009=\u20090.18; P\u2009\u2264\u2009.05) experiences and were more willing to use OMD again (\u03b2\u2009=\u20090.36; P\u2009\u2264\u2009.05), compared to professional stakeholders. Positive perceptions of the OMD system\u2019s ease of use (\u03b2\u2009=\u20090.16; P\u2009\u2264\u2009.05) and favorable study participation (\u03b2\u2009=\u20090.26; P\u2009\u2264\u2009.05) and online discussion (\u03b2\u2009=\u20090.57; P\u2009\u2264\u2009.05) experiences were also associated with increased willingness to use OMD in the future. Active engagement was not associated with online experience indices or willingness to use OMD again.<\/jats:p>\n               <jats:p>Conclusion: Online approaches to engaging large numbers of stakeholders are a promising and efficient adjunct to in-person meetings.<\/jats:p>","DOI":"10.1093\/jamia\/ocw157","type":"journal-article","created":{"date-parts":[[2016,12,24]],"date-time":"2016-12-24T10:35:10Z","timestamp":1482575710000},"page":"537-543","source":"Crossref","is-referenced-by-count":32,"title":["Comparative analysis of stakeholder experiences with an online approach to prioritizing patient-centered research topics"],"prefix":"10.1093","volume":"24","author":[{"given":"Dmitry","family":"Khodyakov","sequence":"first","affiliation":[{"name":"RAND Corporation, Santa Monica, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sean","family":"Grant","sequence":"additional","affiliation":[{"name":"RAND Corporation, Santa Monica, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniella","family":"Meeker","sequence":"additional","affiliation":[{"name":"RAND Corporation, Santa Monica, CA, USA"},{"name":"University of Southern California, Los Angeles, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marika","family":"Booth","sequence":"additional","affiliation":[{"name":"RAND Corporation, Santa Monica, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nathaly","family":"Pacheco-Santivanez","sequence":"additional","affiliation":[{"name":"RAND Corporation, Santa Monica, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katherine K","family":"Kim","sequence":"additional","affiliation":[{"name":"University of California, Davis, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2016,12,19]]},"reference":[{"issue":"1","key":"2020110612435954100_ocw157-B1","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1186\/1472-6963-14-89","article-title":"Patient engagement in research: a systematic review","volume":"14","author":"Domecq","year":"2014","journal-title":"BMC Health Services Res."},{"issue":"8","key":"2020110612435954100_ocw157-B2","doi-asserted-by":"crossref","first-page":"985","DOI":"10.1007\/s11606-012-2037-1","article-title":"A new taxonomy for stakeholder engagement in patient-centered outcomes research","volume":"27","author":"Concannon","year":"2012","journal-title":"J General Int Med."},{"issue":"1","key":"2020110612435954100_ocw157-B3","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.healthpol.2009.11.007","article-title":"Public involvement at the design stage of primary health research: a narrative review of case examples","volume":"95","author":"Boote","year":"2010","journal-title":"Health Policy."},{"issue":"1","key":"2020110612435954100_ocw157-B4","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1007\/s11606-015-3450-z","article-title":"Patient and stakeholder engagement in the PCORI pilot projects: description and lessons learned","volume":"31","author":"Forsythe","year":"2016","journal-title":"J General Int Med."},{"issue":"2","key":"2020110612435954100_ocw157-B5","doi-asserted-by":"crossref","first-page":"133","DOI":"10.2217\/cer.14.79","article-title":"Evaluating patient and stakeholder engagement in research: moving from theory to practice","volume":"4","author":"Esmail","year":"2015","journal-title":"J Comparative Effectiveness Res."},{"key":"2020110612435954100_ocw157-B6","article-title":"Methods for Engaging Stakeholders To Identify and Prioritize Future Research Needs. Methods Future Research Needs Report No. 4. (Prepared by the Oregon Evidence-based Practice Center and the Vanderbilt Evidence-based Practice Center under Contract No. 290-2007-10057-I.) Rockville, MD: AHRQ Publication No. 11-EHC 044-EF. Agency for Healthcare Research and Quality. June 2011. Available at:","author":"O\u2019Haire"},{"issue":"4","key":"2020110612435954100_ocw157-B7","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1377\/hlthaff.2015.1112","article-title":"Understanding an informed public\u2019s views on the role of evidence in making health care decisions","volume":"35","author":"Carman","year":"2016","journal-title":"Health Affairs."},{"issue":"2","key":"2020110612435954100_ocw157-B8","doi-asserted-by":"crossref","first-page":"181","DOI":"10.2217\/cer.12.7","article-title":"Stakeholder participation in comparative effectiveness research: Defining a framework for effective engagement","volume":"1","author":"Deverka","year":"2012","journal-title":"J Comparative Effectiveness Res."},{"issue":"3","key":"2020110612435954100_ocw157-B9","first-page":"335","article-title":"Stakeholder engagement in patient-centered outcomes research: High-touch or high-tech? Expert Rev PharmacoecoOutcomes Res","volume":"14","author":"Lavallee","journal-title":"2014"},{"issue":"4","key":"2020110612435954100_ocw157-B10","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1111\/j.1369-7625.2010.00648.x","article-title":"Patients\u2019 and clinicians\u2019 research priorities","volume":"14","author":"Stewart","year":"2011","journal-title":"Health Expect."},{"key":"2020110612435954100_ocw157-B11","volume-title":"Innovative Methods in Stakeholder Engagement: An Environmental Scan","author":"Mallery","year":"2012"},{"issue":"1","key":"2020110612435954100_ocw157-B12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12913-015-0717-3","article-title":"Measuring organisational readiness for patient engagement (more): an international online Delphi consensus study","volume":"15","author":"Oostendorp","year":"2015","journal-title":"BMC Health Services Res."},{"issue":"3","key":"2020110612435954100_ocw157-B13","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1016\/j.amepre.2014.01.004","article-title":"Reducing the burden of suicide in the U.S.: The aspirational research goals of the National Action Alliance for Suicide Prevention Research Prioritization Task Force","volume":"47","author":"Claassen","year":"2014","journal-title":"Am J Prevent Med."},{"issue":"4","key":"2020110612435954100_ocw157-B14","doi-asserted-by":"crossref","first-page":"868","DOI":"10.1111\/hex.12383","article-title":"Collaborative learning framework for online stakeholder engagement","volume":"19","author":"Khodyakov","year":"2016","journal-title":"Health Expect."},{"issue":"8","key":"2020110612435954100_ocw157-B15","doi-asserted-by":"crossref","first-page":"1426","DOI":"10.1016\/j.techfore.2011.03.021","article-title":"ExpertLens: a system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge","volume":"78","author":"Dalal","year":"2011","journal-title":"Technol Forecasting Social Change."},{"issue":"1","key":"2020110612435954100_ocw157-B16","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1542\/peds.2014-4192","article-title":"Critical elements in the medical evaluation of suspected child physical abuse","volume":"136","author":"Campbell","year":"2015","journal-title":"Pediatrics."},{"issue":"5","key":"2020110612435954100_ocw157-B17","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1097\/NCN.0b013e3181fc3e59","article-title":"Use of a computer-mediated Delphi process to validate a mass casualty conceptual model","volume":"29","author":"Culley","year":"2011","journal-title":"Computers, Inform Nursing."},{"key":"2020110612435954100_ocw157-B18","article-title":"The Delphi method online: medical expert consensus via the Internet","volume-title":"Proceedings of the AMIA Symposium","author":"Cam","year":"2002"},{"issue":"1","key":"2020110612435954100_ocw157-B19","first-page":"6","article-title":"How can we recognize continuous quality improvement? Int J Qual Health Care","volume":"26","author":"Rubenstein","journal-title":"2014"},{"issue":"1","key":"2020110612435954100_ocw157-B20","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1186\/1471-2288-11-174","article-title":"Conducting online expert panels: a feasibility and experimental replicability study","volume":"11","author":"Khodyakov","year":"2011","journal-title":"BMC Med Res Methodol."},{"issue":"9","key":"2020110612435954100_ocw157-B21","doi-asserted-by":"crossref","first-page":"1548","DOI":"10.3899\/jrheum.141603","article-title":"Development of cardiovascular quality indicators for rheumatoid arthritis: results from an international expert panel using a novel online process","volume":"42","author":"Barber","year":"2015","journal-title":"J Rheumatol."},{"issue":"3","key":"2020110612435954100_ocw157-B22","doi-asserted-by":"crossref","first-page":"530","DOI":"10.3899\/jrheum.150839","article-title":"Development of system-level performance measures for evaluation of models of care for inflammatory arthritis in Canada","volume":"43","author":"Barber","year":"2016","journal-title":"J Rheumatol."},{"issue":"322","key":"2020110612435954100_ocw157-B23","first-page":"1","article-title":"Development of key performance indicators to evaluate centralized intake for patients with osteoarthritis and rheumatoid arthritis","volume":"17","author":"Barber","year":"2015","journal-title":"Arthritis ResTherapy."},{"key":"2020110612435954100_ocw157-B24","article-title":"Acceptability of an online modified-Delphi panel approach for developing health services performance measures: results from three panels on arthritis research","author":"Khodyakov","journal-title":"J Eval Clin Pract."},{"key":"2020110612435954100_ocw157-B25","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.trsl.2015.12.008","article-title":"On using ethical principles of community-engaged research in translational science","volume":"171","author":"Khodyakov","year":"2016","journal-title":"Trans Res."},{"key":"2020110612435954100_ocw157-B26","article-title":"Patient engagement in the process of planning and designing outpatient care improvements at the Veterans Administration healthcare system: Findings from an online expert panel","author":"Khodyakov","journal-title":"Health Expect."},{"issue":"4","key":"2020110612435954100_ocw157-B27","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1136\/amiajnl-2014-002751","article-title":"Pscanner: patient-centered scalable national network for effectiveness research","volume":"21","author":"Ohno-Machado","year":"2014","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612435954100_ocw157-B28","doi-asserted-by":"crossref","DOI":"10.7249\/RR1191","volume-title":"National Patient-Centered Clinical Research Network (PCORnet) Phase I","author":"Timbie","year":"2015"},{"issue":"4","key":"2020110612435954100_ocw157-B29","doi-asserted-by":"crossref","first-page":"359","DOI":"10.2217\/cer.12.36","article-title":"Facilitating comparative effectiveness research in cancer genomics: Evaluating stakeholder perceptions of the engagement process","volume":"1","author":"Deverka","year":"2012","journal-title":"J Comparative Effectiveness Res."},{"issue":"5","key":"2020110612435954100_ocw157-B30","first-page":"397","article-title":"Stakeholder engagement in comparative effectiveness research: how will we measure success? J Comparative Effectiveness Res","volume":"1","author":"Lavallee","journal-title":"2012"},{"issue":"1","key":"2020110612435954100_ocw157-B31","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1080\/014492996120373","article-title":"A model of group satisfaction in computer-mediated communication and face-to-face meetings","volume":"15","author":"Olaniran","year":"1996","journal-title":"Behav Inform Technol."},{"issue":"5","key":"2020110612435954100_ocw157-B32","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1287\/mnsc.29.5.530","article-title":"Development of a tool for measuring and analyzing computer user satisfaction","volume":"29","author":"Bailey","year":"1983","journal-title":"Manag Sci."},{"issue":"6","key":"2020110612435954100_ocw157-B33","doi-asserted-by":"crossref","first-page":"739","DOI":"10.1287\/mnsc.36.6.739","article-title":"User satisfaction with computer-mediated communication systems","volume":"36","author":"Hiltz","year":"1990","journal-title":"Manag Sci."},{"issue":"5","key":"2020110612435954100_ocw157-B34","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1177\/1461444807081230","article-title":"Democracy, deliberation and design: the case of online discussion forums","volume":"9","author":"Wright","year":"2007","journal-title":"New Media Soc."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/3\/537\/34148972\/ocw157.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/3\/537\/34148972\/ocw157.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:17:04Z","timestamp":1604686624000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/24\/3\/537\/2723171"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,12,19]]},"references-count":34,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2016,12,19]]},"published-print":{"date-parts":[[2017,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocw157","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,5]]},"published":{"date-parts":[[2016,12,19]]}}}