{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T17:20:52Z","timestamp":1777396852406,"version":"3.51.4"},"reference-count":123,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,8,27]],"date-time":"2020-08-27T00:00:00Z","timestamp":1598486400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,27]],"date-time":"2020-08-27T00:00:00Z","timestamp":1598486400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The field of digital health, and its meaning, has evolved rapidly over the last 20 years. For this article we followed the most recent definition provided by FDA in 2020. Emerging solutions offers tremendous potential to positively transform the healthcare sector. Despite the growing number of applications, however, the evolution of methodologies to perform timely, cost-effective and robust evaluations have not kept pace. It remains an industry-wide challenge to provide credible evidence, therefore, hindering wider adoption. Conventional methodologies, such as clinical trials, have seldom been applied and more pragmatic approaches are needed. In response, several academic centers such as researchers from the Institute of Global Health Innovation at Imperial College London have initiated a digital health clinical simulation test bed to explore new approaches for evidence gathering relevant to solution type and maturity. The aim of this article is to: (1) Review current research approaches and discuss their limitations; (2) Discuss challenges faced by different stakeholders in undertaking evaluations; and (3) Call for new approaches to facilitate the safe and responsible growth of the digital health sector.<\/jats:p>","DOI":"10.1038\/s41746-020-00314-2","type":"journal-article","created":{"date-parts":[[2020,8,27]],"date-time":"2020-08-27T10:04:31Z","timestamp":1598522671000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":280,"title":["Challenges for the evaluation of digital health solutions\u2014A call for innovative evidence generation approaches"],"prefix":"10.1038","volume":"3","author":[{"given":"Chaohui","family":"Guo","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1668-0672","authenticated-orcid":false,"given":"Hutan","family":"Ashrafian","sequence":"additional","affiliation":[]},{"given":"Saira","family":"Ghafur","sequence":"additional","affiliation":[]},{"given":"Gianluca","family":"Fontana","sequence":"additional","affiliation":[]},{"given":"Clarissa","family":"Gardner","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8717-0130","authenticated-orcid":false,"given":"Matthew","family":"Prime","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,8,27]]},"reference":[{"key":"314_CR1","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1097\/00004479-200004000-00003","volume":"23","author":"SR Frank","year":"2000","unstructured":"Frank, S. R. Digital health care\u2014the convergence of health care and the Internet. J. Ambul. Care Manag. 23, 8\u201317 (2000).","journal-title":"J. Ambul. Care Manag."},{"key":"314_CR2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-018-0076-7","volume":"2","author":"SC Mathews","year":"2019","unstructured":"Mathews, S. C. et al. Digital health: a path to validation. NPJ digital medicine 2, 1\u20139 (2019).","journal-title":"NPJ digital medicine"},{"key":"314_CR3","unstructured":"FDA. https:\/\/www.fda.gov\/medical-devices\/digital-health (2020)."},{"key":"314_CR4","unstructured":"IQVIA. IQVIA Institute for Human Data Science Study: Impact of Digital Health Grows as Innovation, Evidence and Adoption of Mobile Health Apps Accelerate. https:\/\/www.iqvia.com\/newsroom\/2017\/11\/impact-of-digital-health-grows-as-innovation-evidence-and-adoption-of-mobile-health-apps-accelerate\/ (2017)."},{"key":"314_CR5","unstructured":"NICE. https:\/\/www.nice.org.uk\/about\/what-we-do\/our-programmes\/evidence-standards-framework-for-digital-health-technologies (2019)."},{"key":"314_CR6","unstructured":"Rogers, E. M. Diffusion of innovations. Simon and Schuster (2010)."},{"key":"314_CR7","doi-asserted-by":"crossref","first-page":"1167","DOI":"10.1177\/0733464817732518","volume":"38","author":"C Ball","year":"2019","unstructured":"Ball, C. et al. The physical\u2013digital divide: exploring the social gap between digital natives and physical natives. J. Appl. Gerontol. 38, 1167\u20131184 (2019).","journal-title":"J. Appl. Gerontol."},{"key":"314_CR8","doi-asserted-by":"crossref","unstructured":"Francis, J. et al. Aging in the digital age: conceptualizing technology adoption and digital inequalities. In Ageing and digital technology, 35\u201349 (Springer, Singapore, 2019).","DOI":"10.1007\/978-981-13-3693-5_3"},{"key":"314_CR9","doi-asserted-by":"crossref","DOI":"10.2196\/jmir.5253","volume":"18","author":"S Peek","year":"2016","unstructured":"Peek, S. et al. What it takes to successfully implement technology for aging in place: focus groups with stakeholders. J. Med. Internet Res. 18, e98 (2016).","journal-title":"J. Med. Internet Res."},{"key":"314_CR10","doi-asserted-by":"crossref","first-page":"193","DOI":"10.2147\/CIA.S72399","volume":"10","author":"Y-H Wu","year":"2015","unstructured":"Wu, Y. -H. et al. Bridging the digital divide in older adults: a study from an initiative to inform older adults about new technologies. Clin. Interv. Aging 10, 193\u2013201 (2015).","journal-title":"Clin. Interv. Aging"},{"key":"314_CR11","unstructured":"FDA. https:\/\/www.fda.gov\/media\/98657\/download."},{"key":"314_CR12","doi-asserted-by":"crossref","unstructured":"Shuren et al. FDA regulation of mobile medical apps. JAMA, 320, 337\u2013338 (2018).","DOI":"10.1001\/jama.2018.8832"},{"key":"314_CR13","unstructured":"https:\/\/www.gov.uk\/government\/publications\/code-of-conduct-for-data-driven-health-and-care-technology\/initial-code-of-conduct-for-data-driven-health-and-care-technology."},{"key":"314_CR14","unstructured":"FDA. https:\/\/www.fda.gov\/medical-devices\/digital-health\/digital-health-software-precertification-pre-cert-program."},{"key":"314_CR15","doi-asserted-by":"crossref","first-page":"1385","DOI":"10.1097\/PRS.0b013e31819f25ff","volume":"123","author":"KC Chung","year":"2009","unstructured":"Chung, K. C. et al. Introducing evidence-based medicine to plastic and reconstructive surgery. Plast. Reconstr. Surg. 123, 1385 (2009).","journal-title":"Plast. Reconstr. Surg."},{"key":"314_CR16","doi-asserted-by":"crossref","first-page":"2234","DOI":"10.1097\/PRS.0b013e3181f44abc","volume":"126","author":"JW Song","year":"2010","unstructured":"Song, J. W. et al. Observational studies: cohort and case-control studies. Plast. Reconstr. Surg. 126, 2234 (2010).","journal-title":"Plast. Reconstr. Surg."},{"key":"314_CR17","doi-asserted-by":"crossref","first-page":"331","DOI":"10.6004\/jnccn.2018.7104","volume":"17","author":"PA Pawloski","year":"2019","unstructured":"Pawloski, P. A. et al. A systematic review of clinical decision support systems for clinical oncology practice. J. Natl. Compr. Canc. Netw. 17, 331\u2013338 (2019).","journal-title":"J. Natl. Compr. Canc. Netw."},{"key":"314_CR18","unstructured":"https:\/\/www.who.int\/reproductivehealth\/publications\/mhealth\/digital-health-interventions\/en\/. WHO (2016)."},{"key":"314_CR19","unstructured":"Ghafur, S. et al. A simulation test bed: the solution to the obstacles of evaluating the effectiveness of digital health interventions (in preparation)."},{"key":"314_CR20","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1258\/mi.2010.010004","volume":"16","author":"GP Cumming","year":"2010","unstructured":"Cumming, G. P. et al. Web-based survey on the effect of digital storytelling on empowering women to seek help for urogenital atrophy. Menopause Int. 16, 51\u201355 (2010).","journal-title":"Menopause Int."},{"key":"314_CR21","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.msard.2017.02.001","volume":"13","author":"L Lavorgna","year":"2017","unstructured":"Lavorgna, L. et al. Health-care disparities stemming from sexual orientation of Italian patients with Multiple Sclerosis: a cross-sectional web-based study. Mult. Scler. Relat. Disord. 13, 28\u201332 (2017).","journal-title":"Mult. Scler. Relat. Disord."},{"key":"314_CR22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.invent.2017.01.001","volume":"8","author":"N Topooco","year":"2017","unstructured":"Topooco, N. et al. Attitudes towards digital treatment for depression: a European stakeholder survey. Internet Interv. 8, 1\u20139 (2017).","journal-title":"Internet Interv."},{"key":"314_CR23","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1046\/j.1365-2702.2003.00662.x","volume":"12","author":"D Evans","year":"2003","unstructured":"Evans, D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J. Clin. Nurs. 12, 77\u201384 (2003).","journal-title":"J. Clin. Nurs."},{"key":"314_CR24","unstructured":"Schneiderheinze, H. et al. Development and usability analysis of a multimedia eConsent solution. Stud. Health Technol. Inform. In GMDS, 297\u2013303 (2019)."},{"key":"314_CR25","doi-asserted-by":"crossref","first-page":"e11222","DOI":"10.2196\/11222","volume":"5","author":"A Hardy","year":"2018","unstructured":"Hardy, A. et al. How inclusive, user-centered design research can improve psychological therapies for psychosis: development of SlowMo. JMIR Ment. Health 5, e11222 (2018).","journal-title":"JMIR Ment. Health"},{"key":"314_CR26","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.ijmedinf.2019.03.018","volume":"126","author":"I Maramba","year":"2019","unstructured":"Maramba, I. et al. Methods of usability testing in the development of eHealth applications: a scoping review. Int. J. Med. Inform. 126, 95\u2013104 (2019).","journal-title":"Int. J. Med. Inform."},{"key":"314_CR27","doi-asserted-by":"crossref","unstructured":"Molich, et al. Comparative usability evaluation. Behav. Inf. Technol. 23, 65\u201374 (2004).","DOI":"10.1080\/0144929032000173951"},{"key":"314_CR28","doi-asserted-by":"crossref","unstructured":"Zimmerman & Paschal. An exploratory usability evaluation of Colorado State University Libraries\u2019 digital collections and the Western Waters Digital Library Web sites. J. Acad. Librarianship. 35, 227\u2013240 (2009).","DOI":"10.1016\/j.acalib.2009.03.011"},{"key":"314_CR29","unstructured":"https:\/\/userfocus.co.uk\/articles\/strength-of-evidence.html."},{"key":"314_CR30","doi-asserted-by":"crossref","first-page":"379","DOI":"10.3758\/BF03195514","volume":"35","author":"L Faulkner","year":"2003","unstructured":"Faulkner, L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav. Res. Methods 35, 379\u2013383 (2003).","journal-title":"Behav. Res. Methods"},{"key":"314_CR31","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1136\/bmj.323.7303.42","volume":"323","author":"P J\u00fcni","year":"2001","unstructured":"J\u00fcni, P. et al. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 323, 42\u201346 (2001).","journal-title":"BMJ"},{"key":"314_CR32","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1016\/j.amepre.2013.03.017","volume":"45","author":"S Kumar","year":"2013","unstructured":"Kumar, S. et al. Mobile health technology evaluation: the mHealth evidence workshop. Am. J. Prev. Med. 45, 228\u2013236 (2013).","journal-title":"Am. J. Prev. Med."},{"key":"314_CR33","unstructured":"Baker, et al. Digital health: Smartphone-based monitoring of multiple sclerosis using Floodlight. Nature (2019)."},{"key":"314_CR34","doi-asserted-by":"crossref","unstructured":"Kohlberger, T. et al. Whole-slide image focus quality: automatic assessment and impact on AI cancer detection. J. Pathol. Inform. 10 (2019).","DOI":"10.4103\/jpi.jpi_11_19"},{"key":"314_CR35","doi-asserted-by":"crossref","first-page":"1159","DOI":"10.1016\/S0140-6736(05)71879-1","volume":"365","author":"A-W Chan","year":"2005","unstructured":"Chan, A.-W. & Altman, D. G. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 365, 1159\u20131162 (2005).","journal-title":"Lancet"},{"key":"314_CR36","doi-asserted-by":"crossref","unstructured":"Basch, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J. Clin. Oncol. 34, 557 (2016).","DOI":"10.1200\/JCO.2015.63.0830"},{"key":"314_CR37","doi-asserted-by":"crossref","unstructured":"Hershman, D. L. et al. Randomized trial of text messaging (TM) to reduce early discontinuation of aromatase inhibitor (AI) therapy in women with breast cancer: SWOG S1105. Oral presentation at: 2019 ASCO Annual Meeting (2019).","DOI":"10.1200\/JCO.2019.37.15_suppl.6516"},{"key":"314_CR38","doi-asserted-by":"crossref","unstructured":"Quinn, et al. WellDoc\u2122 Mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diab. Tech. Therap. 10, 160\u2013168 (2008).","DOI":"10.1089\/dia.2008.0283"},{"key":"314_CR39","unstructured":"Hammer, R. D. et al. Digital Tumor Board Solutions have significant impact on case preparation. JCO Clinical Cancer Informatics (forthcoming)."},{"key":"314_CR40","doi-asserted-by":"crossref","unstructured":"Greaney, et al. Study protocol for Young & Strong: a cluster randomized design to increase attention to unique issues faced by young women with newly diagnosed breast cancer. BMC Public Health. 15, 1\u201311 (2015).","DOI":"10.1186\/s12889-015-1346-9"},{"key":"314_CR41","doi-asserted-by":"crossref","unstructured":"Ohl, et al. Impact of availability of telehealth programs on documented HIV viral suppression: A Cluster-Randomized Program Evaluation in the Veterans Health Administration. Open Forum Infect Dis. 6, ofz206 (2019).","DOI":"10.1093\/ofid\/ofz206"},{"key":"314_CR42","doi-asserted-by":"crossref","unstructured":"Arnup, et al. The use of the cluster randomized crossover design in clinical trials: protocol for a systematic review. Syst. Rev. 3, 1\u20136 (2014).","DOI":"10.1186\/2046-4053-3-86"},{"key":"314_CR43","doi-asserted-by":"crossref","unstructured":"Eldridge, S. & Kerry, S. A Practical Guide to Cluster Randomised Trials in Health Services Research. Vol. 120 (John Wiley & Sons, 2012).","DOI":"10.1002\/9781119966241"},{"key":"314_CR44","doi-asserted-by":"crossref","first-page":"416","DOI":"10.2105\/AJPH.94.3.416","volume":"94","author":"A Donner","year":"2004","unstructured":"Donner, A. & Klar, N. Pitfalls of and controversies in cluster randomization trials. Am. J. Public Health 94, 416\u2013422 (2004).","journal-title":"Am. J. Public Health"},{"key":"314_CR45","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1093\/oxfordjournals.epirev.a017990","volume":"21","author":"ME Halloran","year":"1999","unstructured":"Halloran, M. E. et al. Design and interpretation of vaccine field studies. Epidemiol. Rev. 21, 73\u201388 (1999).","journal-title":"Epidemiol. Rev."},{"key":"314_CR46","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1093\/oxfordjournals.aje.a112592","volume":"108","author":"J Cornfield","year":"1978","unstructured":"Cornfield, J. Randomization by group: a formal analysis. Am. J. Epidemiol. 108, 100\u2013102 (1978).","journal-title":"Am. J. Epidemiol."},{"key":"314_CR47","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1136\/bmj.322.7282.355","volume":"322","author":"DJ Torgerson","year":"2001","unstructured":"Torgerson, D. J. Contamination in trials: is cluster randomisation the answer?. BMJ 322, 355\u2013357 (2001).","journal-title":"BMJ"},{"key":"314_CR48","doi-asserted-by":"crossref","unstructured":"Mazor, K. et al. Cluster Randomized Trials: opportunities and Barriers Identified by Leaders of Eight Health Plans. Med. Care. S29\u2013S37 (2007).","DOI":"10.1097\/MLR.0b013e31806728c4"},{"key":"314_CR49","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1037\/1082-989X.2.2.173","volume":"2","author":"SW Raudenbush","year":"1997","unstructured":"Raudenbush, S. W. Statistical analysis and optimal design for cluster randomized trials. Psychol. methods 2, 173 (1997).","journal-title":"Psychol. methods"},{"key":"314_CR50","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1093\/fampra\/17.2.192","volume":"17","author":"MK Campbell","year":"2000","unstructured":"Campbell, M. K. et al. Analysis of cluster randomized trials in primary care: a practical approach. Family Pract. 17, 192\u2013196 (2000).","journal-title":"Family Pract."},{"key":"314_CR51","doi-asserted-by":"crossref","first-page":"1220","DOI":"10.1037\/hea0000305","volume":"34","author":"P Klasnja","year":"2015","unstructured":"Klasnja, P. et al. Micro-Randomized Trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 34, 1220\u20131228 (2015).","journal-title":"Health Psychol."},{"key":"314_CR52","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1093\/abm\/kay067","volume":"53","author":"P Klasnja","year":"2018","unstructured":"Klasnja, P. et al. Efficacy of contextually tailored suggestions for physical activity: a Micro-randomized Optimization Trial of HeartSteps. Ann. Behav. Med. 53, 573\u2013582 (2018).","journal-title":"Behav. Med."},{"key":"314_CR53","unstructured":"Bolger & Laurenceau. Intensive Longitudinal Methods: An Introduction to Diary and Experience Sampling Research (Guilford Press, 2013)."},{"key":"314_CR54","doi-asserted-by":"crossref","unstructured":"Walls & Schafer. Models for intensive longitudinal data (Oxford University Press, 2006).","DOI":"10.1093\/acprof:oso\/9780195173444.001.0001"},{"key":"314_CR55","doi-asserted-by":"crossref","first-page":"8","DOI":"10.6030\/1939-067X-3.1.8","volume":"3","author":"SR Evans","year":"2010","unstructured":"Evans, S. R. Clinical trial structures. J. Exp. Stroke Transl. Med. 3, 8\u201318 (2010).","journal-title":"J. Exp. Stroke Transl. Med."},{"key":"314_CR56","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1016\/j.amepre.2009.02.002","volume":"36","author":"DJ Bowen","year":"2009","unstructured":"Bowen, D. J. et al. How we design feasibility studies. Am. J. Prev. Med. 36, 452\u2013457 (2009).","journal-title":"Am. J. Prev. Med."},{"key":"314_CR57","unstructured":"Dimitrov, D. M. & Rumrill, P. D. Pretest\u2013posttest Designs and Measurement of Change. (IOS Press, 2003)."},{"key":"314_CR58","doi-asserted-by":"crossref","first-page":"1246","DOI":"10.1016\/j.ijrobp.2011.09.056","volume":"83","author":"S Beriwal","year":"2012","unstructured":"Beriwal, S. et al. How effective are clinical pathways with and without online peer-review? An analysis of bone metastases pathway in a large, integrated National Cancer Institute-Designated Comprehensive Cancer Center Network. Int. J. Radiat. Oncol. Biol. Phys. 83, 1246\u20131251 (2012).","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"314_CR59","unstructured":"Bouaud, J. et al. Physicians\u2019 attitudes towards the advice of a guideline-based decision support system: a case study with OncoDoc2 in the Management of Breast Cancer Patients. Stud. Health Technol. Inform. 264\u2013269 (2015)."},{"key":"314_CR60","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1093\/annonc\/mdv032","volume":"26","author":"TO Mattsson","year":"2015","unstructured":"Mattsson, T. O. et al. Non-intercepted dose errors in prescribing anti-neoplastic treatment: a prospective, comparative cohort study. Ann. Oncol. 26, 981\u2013986 (2015).","journal-title":"Ann. Oncol."},{"key":"314_CR61","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1200\/JCO.2010.30.3909","volume":"29","author":"DL Berry","year":"2011","unstructured":"Berry, D. L. et al. Enhancing patient-provider communication with the electronic self-report assessment for cancer: a randomized trial. J. Clin. Oncol. 29, 1029\u20131035 (2011).","journal-title":"J. Clin. Oncol."},{"key":"314_CR62","first-page":"1","volume":"6","author":"E Caselli","year":"2016","unstructured":"Caselli, E. et al. Influence of sanitizing methods on healthcare-associated infections onset: a multicentre, randomized, controlled pre-post interventional study. J. Clin. Trials. 6, 1\u20136 (2016).","journal-title":"J. Clin. Trials."},{"key":"314_CR63","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1054\/JHSB.2001.0703","volume":"27","author":"S Sauerland","year":"2002","unstructured":"Sauerland, S. et al. Retrospective clinical studies in surgery: potentials and pitfalls. J. Hand Surg. Br. 27, 117\u2013121 (2002).","journal-title":"J. Hand Surg. Br."},{"key":"314_CR64","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1016\/j.annemergmed.2014.03.025","volume":"64","author":"AH Kaji","year":"2014","unstructured":"Kaji, A. H. et al. Looking through the retrospectoscope: reducing bias in emergency medicine chart review studies. Ann. Emerg. Med. 64, 292\u2013298 (2014).","journal-title":"Ann. Emerg. Med."},{"key":"314_CR65","first-page":"181","volume":"3","author":"C Tofthagen","year":"2012","unstructured":"Tofthagen, C. Threats to validity in retrospective studies. J. Adv. Pract. Oncol. 3, 181 (2012).","journal-title":"J. Adv. Pract. Oncol."},{"key":"314_CR66","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1093\/biostatistics\/kxn010","volume":"10","author":"S Geneletti","year":"2009","unstructured":"Geneletti, S. et al. Adjusting for selection bias in retrospective, case-control studies. Biostatistics 10, 17\u201331 (2009).","journal-title":"Biostatistics"},{"key":"314_CR67","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1093\/annonc\/mdx781","volume":"29","author":"SP Somashekhar","year":"2018","unstructured":"Somashekhar, S. P. et al. Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board. Ann. Oncol. 29, 418\u2013423 (2018).","journal-title":"Ann. Oncol."},{"key":"314_CR68","doi-asserted-by":"crossref","unstructured":"Graham, R. et al. Clinical Practice Guidelines We Can Trust: Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. (National Academies Press, 2011).","DOI":"10.17226\/13058"},{"key":"314_CR69","unstructured":"Goff, et al. ACC\/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology. American Heart Association Task Force on Practice Guidelines. Circulation 63, 2935\u20132959 (2014)."},{"key":"314_CR70","doi-asserted-by":"crossref","unstructured":"Rabar, S. et al. Guideline Development Group Risk assessment of fragility fractures: summary of NICE guidance. BMJ 345, p.e3698 (2012).","DOI":"10.1136\/bmj.e3698"},{"key":"314_CR71","doi-asserted-by":"crossref","unstructured":"Bouwmeester, W. et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 9, e1001221 (2012).","DOI":"10.1371\/journal.pmed.1001221"},{"key":"314_CR72","first-page":"211","volume":"131","author":"GS Collins","year":"2015","unstructured":"Collins, G. S. et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann. Intern. Med. 131, 211\u2013219 (2015).","journal-title":"Ann. Intern. Med."},{"key":"314_CR73","doi-asserted-by":"crossref","first-page":"i6460","DOI":"10.1136\/bmj.i6460","volume":"356","author":"TP Debray","year":"2017","unstructured":"Debray, T. P. et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ 356, i6460 (2017).","journal-title":"BMJ"},{"key":"314_CR74","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-018-0059-8","volume":"1","author":"J Shaw","year":"2018","unstructured":"Shaw, J. et al. Beyond \u201cimplementation\u201d: digital health innovation and service design. NPJ Digit. Med. 1, 1\u20135 (2018).","journal-title":"NPJ Digit. Med."},{"key":"314_CR75","doi-asserted-by":"crossref","unstructured":"Moxey, et al. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J. Am. Med. Inform. Associat. 17, 25\u201333 (2010).","DOI":"10.1197\/jamia.M3170"},{"key":"314_CR76","doi-asserted-by":"crossref","unstructured":"O\u2019Sullivan, et al. Decision time for clinical decision support systems. Clin. Med. 14, 338 (2014).","DOI":"10.7861\/clinmedicine.14-4-338"},{"key":"314_CR77","doi-asserted-by":"crossref","first-page":"872","DOI":"10.1161\/CIRCULATIONAHA.115.019902","volume":"133","author":"RJ Mentz","year":"2016","unstructured":"Mentz, R. J. et al. Good Clinical Practice Guidance and Pragmatic Clinical Trials: balancing the best of both worlds. Circulation 133, 872\u2013880 (2016).","journal-title":"Circulation"},{"key":"314_CR78","doi-asserted-by":"crossref","first-page":"454","DOI":"10.1056\/NEJMra1510059","volume":"375","author":"I Ford","year":"2016","unstructured":"Ford, I. et al. Pragmatic trials. N. Engl. J. Med. 375, 454\u2013463 (2016).","journal-title":"N. Engl. J. Med."},{"key":"314_CR79","doi-asserted-by":"crossref","first-page":"1568","DOI":"10.1002\/sim.7087","volume":"36","author":"KM Cunanan","year":"2017","unstructured":"Cunanan, K. M. et al. An efficient basket trial design. Stat. Med. 36, 1568\u20131579 (2017).","journal-title":"Stat. Med."},{"key":"314_CR80","unstructured":"Dahabreh, I. J. et al. Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment (2017)."},{"key":"314_CR81","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/S0010-4825(02)00012-4","volume":"32","author":"JG Anderson","year":"2002","unstructured":"Anderson, J. G. et al. Evaluation in health informatics: computer simulation. Comput. Biol. Med. 32, 151\u2013164 (2002).","journal-title":"Comput. Biol. Med."},{"key":"314_CR82","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/2110-5820-2-18","volume":"2","author":"Y Dong","year":"2012","unstructured":"Dong, Y. et al. Systems modeling and simulation applications for critical care medicine. Ann. Intensive Care 2, 1\u201310 (2012).","journal-title":"Ann. Intensive Care"},{"key":"314_CR83","doi-asserted-by":"crossref","unstructured":"Roberts, S. D. Tutorial on the simulation of healthcare systems. Proceedings of the 2011 winter simulation conference (wsc), 1403\u20131414 (2011).","DOI":"10.1109\/WSC.2011.6147860"},{"key":"314_CR84","doi-asserted-by":"crossref","unstructured":"Kushniruk A. et. al. From usability testing to clinical simulations: bringing context into the design and evaluation of usable and safe health information technologies. Contribution of the IMIA human factors engineering for healthcare informatics working group. Yearb. Med. Inform. 22, 78\u201385 (2013).","DOI":"10.1055\/s-0038-1638836"},{"key":"314_CR85","unstructured":"Kushniruk, A. W. et al. Low-cost rapid usability engineering: designing and customizing usable healthcare information systems. Healthc, Q. (2006)."},{"key":"314_CR86","unstructured":"Baylis, T. B. et al. Low-Cost Rapid Usability Testing for health information systems: is it worth the effort? Stud. Health Technol. Inform. (2012)."},{"key":"314_CR87","doi-asserted-by":"crossref","unstructured":"Yao, H. et al. Research and design on distributed remote simulation based on Web. In IEEE International Conference on Information Management and Engineering. pp. 522\u2013525 (2010).","DOI":"10.1109\/ICIME.2010.5477791"},{"key":"314_CR88","unstructured":"https:\/\/imperialbrc.nihr.ac.uk\/patients-public\/ppi-e-strategy\/."},{"key":"314_CR89","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/j.jbi.2015.02.002","volume":"54","author":"S Jensen","year":"2015","unstructured":"Jensen, S. et al. Clinical simulation: a method for development and evaluation of clinical information systems. J. Biomed. Inform. 54, 65\u201376 (2015).","journal-title":"J. Biomed. Inform."},{"key":"314_CR90","doi-asserted-by":"crossref","unstructured":"Li, et al. Integrating usability testing and think-aloud protocol analysis with \u201cnear-live\u201d clinical simulations in evaluating clinical decision support. Int. J. Med. Inform. 81, 761\u2013772 (2012).","DOI":"10.1016\/j.ijmedinf.2012.02.009"},{"key":"314_CR91","doi-asserted-by":"crossref","unstructured":"Mazur, et al. Association of the usability of electronic health records with cognitive workload and performance levels among physicians. JAMA Netw Open. 2, e191709\u2013e191709 (2019).","DOI":"10.1001\/jamanetworkopen.2019.1709"},{"key":"314_CR92","unstructured":"https:\/\/www.imperial.ac.uk\/global-health-innovation\/."},{"key":"314_CR93","doi-asserted-by":"crossref","unstructured":"Gardner, et al. A mixed methods study for the evaluation of a digital health solution for cancer multidisciplinary team meetings using simulation-based research methods. ASCO 2020 Annual Conference. American Society of Clinical Oncology. pp. e14063 (2020)","DOI":"10.1200\/JCO.2020.38.15_suppl.e14063"},{"key":"314_CR94","doi-asserted-by":"crossref","unstructured":"Khozin, et al. Real-world data for clinical evidence generation in oncology. J. Natl. Cancer Inst. 109, djx187 (2017).","DOI":"10.1093\/jnci\/djx187"},{"key":"314_CR95","doi-asserted-by":"crossref","unstructured":"Calabria, et al. Open triple therapy for chronic obstructive pulmonary disease: Patterns of prescription, exacerbations and healthcare costs from a large Italian claims database. Pulmon. Pharmacol. Therap. 61, 101904 (2020).","DOI":"10.1016\/j.pupt.2020.101904"},{"key":"314_CR96","doi-asserted-by":"crossref","unstructured":"Pal, et al. Real-world treatment patterns and adverse events in metastatic renal cell carcinoma from a large US claims database. BMC Cancer 19, 548 (2019).","DOI":"10.1186\/s12885-019-5716-z"},{"key":"314_CR97","unstructured":"http:\/\/www.pchalliance.org\/resources."},{"key":"314_CR98","unstructured":"https:\/\/ec.europa.eu\/docsroom\/documents\/17921\/attachments\/1\/translations."},{"key":"314_CR99","unstructured":"http:\/\/www.imdrf.org\/docs\/imdrf\/final\/technical\/imdrf-tech-170921-samd-n41-clinical-evaluation_1.pdf."},{"key":"314_CR100","unstructured":"https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/artificial-intelligence-and-machine-learning-software-medical-device."},{"key":"314_CR101","doi-asserted-by":"crossref","unstructured":"Hayden, et al. Evaluation of the quality of prognosis studies in systematic reviews. Ann. Intern. Med. 144, 427\u2013437 (2006).","DOI":"10.7326\/0003-4819-144-6-200603210-00010"},{"key":"314_CR102","doi-asserted-by":"crossref","unstructured":"Higgins, et al. The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials. BMJ 343, d5928 (2011).","DOI":"10.1136\/bmj.d5928"},{"key":"314_CR103","unstructured":"Sterne, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355 (2016)."},{"key":"314_CR104","doi-asserted-by":"crossref","unstructured":"Wolff, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann. Intern. Med. 170, 51\u201358 (2019).","DOI":"10.7326\/M18-1376"},{"key":"314_CR105","unstructured":"https:\/\/webstore.iec.ch\/preview\/info_iec62304%7bed1.0%7den_d.pdf. (2006)."},{"key":"314_CR106","unstructured":"ISO. https:\/\/www.iso.org\/standard\/52823.html."},{"key":"314_CR107","unstructured":"ISO. https:\/\/www.iso.org\/standard\/51344.html."},{"key":"314_CR108","unstructured":"ISO. https:\/\/www.iso.org\/standard\/57757.html."},{"key":"314_CR109","unstructured":"FDA. https:\/\/www.fda.gov\/regulatory-information\/selected-amendments-fdc-act\/21st-century-cures-act."},{"key":"314_CR110","unstructured":"https:\/\/www.eur-lex.europa.eu\/legal-content\/EN\/TXT\/PDF\/?uri=CELEX:32017R0746&from=DE."},{"key":"314_CR111","unstructured":"ISO. https:\/\/www.iso.org\/standard\/63553.html."},{"key":"314_CR112","unstructured":"ISO. https:\/\/www.iso.org\/standard\/74438.html."},{"key":"314_CR113","doi-asserted-by":"crossref","unstructured":"Denis, et al. Randomized Trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients. J. Natl. Cancer Inst. 109 (2017).","DOI":"10.1093\/jnci\/djx029"},{"key":"314_CR114","doi-asserted-by":"crossref","unstructured":"Connell, et al. Evaluation of a digitally-enabled care pathway for acute kidney injury management in hospital emergency admissions. Nature Digit. Med. 2, 1\u20139 (2019).","DOI":"10.1038\/s41746-019-0100-6"},{"key":"314_CR115","doi-asserted-by":"crossref","unstructured":"Palermo, et al. Mobile health intervention for self-management of adolescent chronic pain (WebMAP mobile): Protocol for a hybrid effectiveness-implementation cluster randomized controlled trial. Contemp. Clin. Trials. 74, 55\u201360 (2018).","DOI":"10.1016\/j.cct.2018.10.003"},{"key":"314_CR116","doi-asserted-by":"crossref","unstructured":"Frias, et al. Effectiveness of digital medicines to improve clinical outcomes in patients with uncontrolled hypertension and type 2 diabetes: Prospective, Open-Label, Cluster-Randomized Pilot Clinical Trial. J. Med. Internet Res. 19, e246 (2017).","DOI":"10.2196\/jmir.7833"},{"key":"314_CR117","first-page":"13","volume":"44","author":"MT Aziz","year":"2015","unstructured":"Aziz, M. T. et al. Reduction in chemotherapy order errors with computerised physician order entry and clinical decision support systems. Health Inf. Manag 44, 13\u201322 (2015).","journal-title":"Health Inf. Manag"},{"key":"314_CR118","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1200\/JOP.2011.000344","volume":"7","author":"AR Chen","year":"2011","unstructured":"Chen, A. R. et al. Computerized provider order entry in pediatric oncology: design, implementation, and outcomes. J Oncol Pract. 7, 218\u2013222 (2011).","journal-title":"J Oncol Pract."},{"key":"314_CR119","doi-asserted-by":"crossref","first-page":"4352","DOI":"10.1038\/s41467-018-06799-6","volume":"9","author":"J Wu","year":"2018","unstructured":"Wu, J. et al. Master clinical medical knowledge at certificated doctor-level with deep learning model. Nat. Commun. 9, 4352 (2018).","journal-title":"Nat. Commun."},{"key":"314_CR120","doi-asserted-by":"crossref","unstructured":"Ammenwerth, et al. Simulation studies for the evaluation of health information technologies: experiences and results. Health Inf. Manag. J. 41, 14\u201321 (2012).","DOI":"10.1177\/183335831204100202"},{"key":"314_CR121","doi-asserted-by":"crossref","unstructured":"Cheng, et al. Utility of a clinical decision support system in weight loss prediction after head and neck cancer radiotherapy. JCO Clin. Cancer Inform. 3, 1\u201311 (2018).","DOI":"10.1200\/CCI.18.00058"},{"key":"314_CR122","doi-asserted-by":"crossref","unstructured":"Palanica, et al. Do you understand the words that are comin outta my mouth? Voice assistant comprehension of medication names. Nat. Digit. Med. 2, 1\u20136 (2019).","DOI":"10.1038\/s41746-019-0133-x"},{"key":"314_CR123","doi-asserted-by":"crossref","first-page":"e142","DOI":"10.1200\/JOP.2011.000525","volume":"8","author":"H Seow","year":"2012","unstructured":"Seow, H. et al. Do high symptom scores trigger clinical actions? An audit after implementing electronic symptom screening. J. Oncol. Pract. 8, e142\u2013e148 (2012).","journal-title":"J. Oncol. Pract."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00314-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00314-2","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00314-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:30:53Z","timestamp":1670380253000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00314-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,27]]},"references-count":123,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["314"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-00314-2","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,27]]},"assertion":[{"value":"15 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 July 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 August 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"C.H.G. and M.P. are currently employees of Roche Diagnostics; H.A., S.G., G.F. and C.G. are employees of Imperial College London. H.A., S.G., G.F., C.G. receive infrastructure support provided by the NIHR Imperial Biomedical Research Center (BRC) at Imperial College London; S.G. is a co-founder of Psyma Ltd, a mobile app that allows clients to access psychological therapy through a live video call; M.P. is a co-founder of Open Medical Ltd, a cloud-based patient management platform.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"110"}}