{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T22:07:32Z","timestamp":1784758052098,"version":"3.55.0"},"reference-count":54,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2021,2,23]],"date-time":"2021-02-23T00:00:00Z","timestamp":1614038400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,6,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Toolkits are an important knowledge translation strategy for implementing digital health. We studied how toolkits for the implementation and evaluation of digital health were developed, tested, and reported.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We conducted a systematic review of toolkits that had been used, field tested or evaluated in practice, and published in the English language from 2009 to July 2019. We searched several electronic literature sources to identify both peer-reviewed and gray literature, and records were screened as per systematic review conventions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Thirteen toolkits were eventually identified, all of which were developed in North America, Europe, or Australia. All reported their intended purpose, as well as their development process. Eight of the 13 toolkits involved a literature review, 3 did not, and 2 were unclear. Twelve reported an underlying conceptual framework, theory, or model: 3 cited the normalization process theory and 3 others cited the World Health Organization and International Telecommunication Union eHealth Strategy. Seven toolkits were reportedly evaluated, but details were unavailable. Forty-three toolkits were excluded for lack of field-testing.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Despite a plethora of published toolkits, few were tested, and even fewer were evaluated. Methodological rigor was of concern, as several did not include an underlying conceptual framework, literature review, or evaluation and refinement in real-world settings. Reporting was often inconsistent and unclear, and toolkits rarely reported being evaluated.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Greater attention needs to be paid to rigor and reporting when developing, evaluating, and reporting toolkits for implementing and evaluating digital health so that they can effectively function as a knowledge translation strategy.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab010","type":"journal-article","created":{"date-parts":[[2021,1,15]],"date-time":"2021-01-15T12:19:21Z","timestamp":1610713161000},"page":"1298-1307","source":"Crossref","is-referenced-by-count":29,"title":["Toolkits for implementing and evaluating digital health: A systematic review of rigor and reporting"],"prefix":"10.1093","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0081-2506","authenticated-orcid":false,"given":"Myron Anthony","family":"Godinho","sequence":"first","affiliation":[{"name":"WHO Collaborating Centre on eHealth (AUS-135), School of Population Health, UNSW Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sameera","family":"Ansari","sequence":"additional","affiliation":[{"name":"WHO Collaborating Centre on eHealth (AUS-135), School of Population Health, UNSW Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2164-0053","authenticated-orcid":false,"given":"Guan Nan","family":"Guo","sequence":"additional","affiliation":[{"name":"WHO Collaborating Centre on eHealth (AUS-135), School of Population Health, UNSW Sydney, New South Wales, Australia"},{"name":"National Drug and Alcohol Research Centre, University of New South Wales, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5989-3614","authenticated-orcid":false,"given":"Siaw-Teng","family":"Liaw","sequence":"additional","affiliation":[{"name":"WHO Collaborating Centre on eHealth (AUS-135), School of Population Health, UNSW Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,2,23]]},"reference":[{"key":"2021061318591444700_ocab010-B1","year":"2018"},{"issue":"1","key":"2021061318591444700_ocab010-B2","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1186\/s12911-014-0121-7","article-title":"Scoping review of toolkits as a knowledge translation strategy in health","volume":"14","author":"Barac","year":"2014","journal-title":"BMC Med Inform Decis Mak"},{"key":"2021061318591444700_ocab010-B3","year":"2012"},{"key":"2021061318591444700_ocab010-B4","year":"2015"},{"key":"2021061318591444700_ocab010-B5"},{"key":"2021061318591444700_ocab010-B6","year":"2019"},{"issue":"4","key":"2021061318591444700_ocab010-B7","doi-asserted-by":"crossref","first-page":"e006808","DOI":"10.1136\/bmjopen-2014-006808","article-title":"The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review","volume":"5","author":"Yamada","year":"2015","journal-title":"BMJ Open"},{"issue":"1","key":"2021061318591444700_ocab010-B8","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1186\/s12913-017-2413-y","article-title":"A qualitative study of clinic and community member perspectives on intervention toolkits: \u201cUnless the toolkit is used it won\u2019t help solve the problem","volume":"17","author":"Davis","year":"2017","journal-title":"BMC Health Serv Res"},{"key":"2021061318591444700_ocab010-B9","year":"2013"},{"key":"2021061318591444700_ocab010-B10"},{"key":"2021061318591444700_ocab010-B11","author":"Ansari","year":"2019"},{"key":"2021061318591444700_ocab010-B12","year":"2019"},{"key":"2021061318591444700_ocab010-B13","year":"2019"},{"key":"2021061318591444700_ocab010-B14","year":"2019"},{"issue":"2","key":"2021061318591444700_ocab010-B15","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1093\/pubmed\/fdy122","article-title":"Completeness of reporting in Indian qualitative public health research: a systematic review of 20 years of literature","volume":"41","author":"Godinho","year":"2019","journal-title":"J Public Health"},{"issue":"1","key":"2021061318591444700_ocab010-B16","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1186\/1472-6963-10-188","article-title":"Evaluation of the performance of routine information system management (PRISM) framework: evidence from Uganda","volume":"10","author":"Hotchkiss","year":"2010","journal-title":"BMC Health Serv Res"},{"key":"2021061318591444700_ocab010-B17","volume-title":"PRISM Case Studies: Strengthening and Evaluating RHIS","year":"2008"},{"issue":"3","key":"2021061318591444700_ocab010-B18","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1093\/heapol\/czp010","article-title":"PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems","volume":"24","author":"Aqil","year":"2009","journal-title":"Health Policy Plan"},{"issue":"1","key":"2021061318591444700_ocab010-B19","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1186\/1748-5908-6-122","article-title":"The e-health implementation toolkit: qualitative evaluation across four European countries","volume":"6","author":"MacFarlane","year":"2011","journal-title":"Implement Sci"},{"issue":"1","key":"2021061318591444700_ocab010-B20","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1186\/1472-6947-10-61","article-title":"Development and formative evaluation of the e-Health Implementation Toolkit (e-HIT)","volume":"10","author":"Murray","year":"2010","journal-title":"BMC Med Inform Decis Mak"},{"issue":"2","key":"2021061318591444700_ocab010-B21","article-title":"eHealth strategy development: a case study in Tanzania","volume":"2","author":"Darcy","year":"2014","journal-title":"J Health Inform Africa"},{"key":"2021061318591444700_ocab010-B22","author":"Ali","year":"2013"},{"key":"2021061318591444700_ocab010-B23","first-page":"246","article-title":"Towards National eHealth Implementation--a comparative study on WHO\/ITU National eHealth Strategy Toolkit in Iran","volume":"205","author":"Riazi","year":"2014","journal-title":"Stud Health Technol Inform"},{"key":"2021061318591444700_ocab010-B24","first-page":"913","article-title":"The WHO-ITU national eHealth strategy toolkit as an effective approach to national strategy development and implementation","volume":"192","author":"Hamilton","year":"2013","journal-title":"Stud Health Technol Inform"},{"key":"2021061318591444700_ocab010-B25","volume-title":"The MAPS Toolkit: mHealth Assessment and Planning for Scale","year":"2015"},{"issue":"1","key":"2021061318591444700_ocab010-B26","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1186\/s12992-018-0424-z","article-title":"Best practices in scaling digital health in low and middle income countries","volume":"14","author":"Labrique","year":"2018","journal-title":"Global Health"},{"issue":"3","key":"2021061318591444700_ocab010-B27","first-page":"379","article-title":"World health organization. First experiences with a WHO tool for assessing health information systems","volume":"2","author":"Verschuuren","year":"2016","journal-title":"Public Health Panor"},{"key":"2021061318591444700_ocab010-B28","volume-title":"Support Tool to Assess Health Information Systems and Develop and Strengthen Health Information Strategies","year":"2015"},{"key":"2021061318591444700_ocab010-B29","volume-title":"Assessing the National Health Information System: An Assessment Tool. Version 4.00","year":"2008"},{"key":"2021061318591444700_ocab010-B30","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/978-3-319-41652-6_11","volume-title":"Advances in Human Factors and Ergonomics in Healthcare","author":"McGee-Lennon","year":"2017"},{"key":"2021061318591444700_ocab010-B31","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.ijmedinf.2017.06.002","article-title":"The informatics capability maturity of integrated primary care centres in Australia","volume":"105","author":"Liaw","year":"2017","journal-title":"Int J Med Inform"},{"key":"2021061318591444700_ocab010-B32","volume-title":"Validating the Effectiveness of a Rapid Assessment Tool for Routine Health Information Systems","year":"2018"},{"key":"2021061318591444700_ocab010-B33","volume-title":"Routine Health Information System Rapid Assessment Tool: Implementation Guide","year":"2018"},{"issue":"5","key":"2021061318591444700_ocab010-B34","doi-asserted-by":"crossref","first-page":"e178","DOI":"10.2196\/jmir.8347","article-title":"Assessing competencies needed to engage with digital health services: development of the eHealth literacy assessment toolkit","volume":"20","author":"Karnoe","year":"2018","journal-title":"J Med Internet Res"},{"issue":"6","key":"2021061318591444700_ocab010-B35","doi-asserted-by":"crossref","first-page":"349","DOI":"10.5334\/ijic.2897","article-title":"Validation of the eHealth Literacy Assessment tool (eHLA)","volume":"16","author":"Knudsen","year":"2016","journal-title":"Int J Integr Care"},{"key":"2021061318591444700_ocab010-B36","volume-title":"Coordinating Digital Transformation: Ethiopia","year":"2019"},{"key":"2021061318591444700_ocab010-B37","volume-title":"Coordinating Digital Transformation: Nepal","year":"2019"},{"key":"2021061318591444700_ocab010-B38","volume-title":"Coordinating Digital Transformation: Tanzania","year":"2019"},{"key":"2021061318591444700_ocab010-B39","volume-title":"Coordinating Digital Transformation: Replication Guide","year":"2019"},{"key":"2021061318591444700_ocab010-B40","volume-title":"Coordinating Digital Transformation: Overview","year":"2019"},{"key":"2021061318591444700_ocab010-B41","year":"2019"},{"key":"2021061318591444700_ocab010-B42","volume-title":"Building a Strong and Interoperable Digital Health Information System for Uganda","year":"2018"},{"key":"2021061318591444700_ocab010-B43","volume-title":"Building a Strong and Interoperable Health Information System for Ghana","year":"2018"},{"key":"2021061318591444700_ocab010-B44","volume-title":"HIS Stages of Continuous Improvement Toolkit","year":"2017"},{"key":"2021061318591444700_ocab010-B45","volume-title":"Mapping a Path to Improve Uganda\u2019s Health Information System Using the Stages of Continuous Improvement Toolkit - Workshop Report","year":"2019"},{"key":"2021061318591444700_ocab010-B46","author":"Mechael","year":"2019"},{"key":"2021061318591444700_ocab010-B47","year":"2015"},{"issue":"10258","key":"2021061318591444700_ocab010-B48","doi-asserted-by":"crossref","first-page":"1250","DOI":"10.1016\/S0140-6736(20)30750-9","article-title":"Measuring universal health coverage based on an index of effective coverage of health services in 204 countries and territories, 1990\u20132019: a systematic analysis for the Global Burden of Disease Study 2019","volume":"396","author":"Lozano","year":"2020","journal-title":"Lancet"},{"issue":"10258","key":"2021061318591444700_ocab010-B49","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1016\/S0140-6736(20)31404-5","article-title":"Five insights from the global burden of disease Study 2019","volume":"396","author":"Murray","year":"2020","journal-title":"Lancet"},{"key":"2021061318591444700_ocab010-B50","author":"Marcelo","year":"2018"},{"key":"2021061318591444700_ocab010-B51","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1136\/bmjstel-2020-000606","article-title":"Serious Games\u2019 for unboxing Global Digital Health policymaking","volume":"6","author":"Godinho","year":"2020","journal-title":"BMJ Simul Technol Enhanc Learn"},{"key":"2021061318591444700_ocab010-B52","doi-asserted-by":"crossref","first-page":"205520762093011","DOI":"10.1177\/2055207620930118","article-title":"Community health alliances as social enterprises that digitally engage citizens and integrate services: A case study in Southwestern Sydney (protocol)","volume":"6","author":"Godinho","year":"2020","journal-title":"Digit Health"},{"key":"2021061318591444700_ocab010-B53","doi-asserted-by":"crossref","first-page":"104259","DOI":"10.1016\/j.ijmedinf.2020.104259","article-title":"mHealth for integrated people-centred health services in the Western Pacific: a systematic review","volume":"142","author":"Godinho","year":"2020","journal-title":"Int J Med Inform"},{"issue":"2","key":"2021061318591444700_ocab010-B54","doi-asserted-by":"crossref","first-page":"e165","DOI":"10.1002\/hsr2.165","article-title":"Reporting guideline checklists are not quality evaluation forms: they are guidance for writing","volume":"3","author":"Logullo","year":"2020","journal-title":"Health Sci Rep"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/6\/1298\/38615353\/ocab010.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/6\/1298\/38615353\/ocab010.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,6,13]],"date-time":"2021-06-13T18:59:49Z","timestamp":1623610789000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/6\/1298\/6146672"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,23]]},"references-count":54,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2021,2,23]]},"published-print":{"date-parts":[[2021,6,12]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab010","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,6,1]]},"published":{"date-parts":[[2021,2,23]]}}}