{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T03:43:27Z","timestamp":1771299807883,"version":"3.50.1"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T00:00:00Z","timestamp":1626307200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T00:00:00Z","timestamp":1626307200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100004328","name":"Genentech","doi-asserted-by":"publisher","award":["G-68202"],"award-info":[{"award-number":["G-68202"]}],"id":[{"id":"10.13039\/100004328","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004328","name":"Genentech","doi-asserted-by":"publisher","award":["G-68202"],"award-info":[{"award-number":["G-68202"]}],"id":[{"id":"10.13039\/100004328","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>While digital health solutions continue to grow in number and in complexity, the ability for stakeholders in healthcare to easily discern quality lags far behind. This challenge is in part due to the lack of a transparent and standardized approach to validation. Evaluation of mobile health applications (apps) is further burdened by low barriers to development and direct-to-user marketing, leading to a crowded and confusing landscape. In this context, we investigated the pragmatic application of a previously described framework for digital health validation, the Digital Health Scorecard, in a cohort of 22 popular mobile health oncology apps. The apps evaluated using this framework performed poorly, scoring 49.4% across all evaluation criteria as a group. Performance across component domains varied considerably with cost scoring highest at 100%, usability at 56.7%, technical at 37.3%, and clinical at 15.9%. satisfaction of prospectively determined end-user requirements derived from patient, family, and clinician consensus scored 37.2%. While cost outperformed consistently and usability was adequate, the results also suggested that apps suffered from significant technical limitations, were of limited clinical value, and generally did not do what end users wanted. These large gaps further support the need for transparent and standardized evaluation to help all stakeholders in healthcare improve the quality of mobile health.<\/jats:p>","DOI":"10.1038\/s41746-021-00476-7","type":"journal-article","created":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T10:03:50Z","timestamp":1626343430000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":37,"title":["Mobile app validation: a digital health scorecard approach"],"prefix":"10.1038","volume":"4","author":[{"given":"Ramy","family":"Sedhom","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael J.","family":"McShea","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adam B.","family":"Cohen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonathan A.","family":"Webster","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1961-3360","authenticated-orcid":false,"given":"Simon C.","family":"Mathews","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,7,15]]},"reference":[{"key":"476_CR1","unstructured":"Pew Research Center. Mobile fact sheet. Available at: https:\/\/www.pewresearch.org\/internet\/fact-sheet\/mobile\/. Accessed September 22, 2020."},{"key":"476_CR2","doi-asserted-by":"publisher","first-page":"107","DOI":"10.2196\/mhealth.4924","volume":"3","author":"P Krebs","year":"2015","unstructured":"Krebs, P. & Duncan, D. T. Health app use among US mobile phone owners: a national survey. Jmir Mhealth Uhealth 3, 107\u2013119 (2015).","journal-title":"Jmir Mhealth Uhealth"},{"key":"476_CR3","doi-asserted-by":"crossref","unstructured":"Cohen, A. B., Dorsey, E. R., Mathews, S. C., Bates, D. W. & Safavi, K. npj Digit. Med. 3, Article number: 68 (2020).","DOI":"10.1038\/s41746-020-0276-9"},{"key":"476_CR4","doi-asserted-by":"publisher","first-page":"397","DOI":"10.1089\/tmj.2017.0112","volume":"24","author":"JL Larson","year":"2018","unstructured":"Larson, J. L., Rosen, A. B. & Wilson, F. A. The effect of telehealth interventions on quality of life of cancer patients: a systematic review and meta-analysis. Telemed. E-Health 24, 397\u2013405 (2018).","journal-title":"Telemed. E-Health"},{"key":"476_CR5","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1093\/europace\/eux299","volume":"20","author":"D Kotecha","year":"2018","unstructured":"Kotecha, D. et al. European Society of Cardiology smartphone and tablet applications for patients with atrial fibrillation and their health care providers. Europace 20, 225\u2013233 (2018).","journal-title":"Europace"},{"key":"476_CR6","doi-asserted-by":"publisher","first-page":"1660","DOI":"10.1038\/s41386-018-0030-z","volume":"43","author":"I Barnett","year":"2018","unstructured":"Barnett, I. et al. Relapse prediction in schizophrenia through digital phenotyping: a pilot study. Neuropsychopharmacology 43, 1660\u20131666 (2018).","journal-title":"Neuropsychopharmacology"},{"key":"476_CR7","doi-asserted-by":"crossref","unstructured":"Mathews, S. C. et al. Digital health: a path to validation. Npj Digit. Med. 2, Article number:38 (2019).","DOI":"10.1038\/s41746-019-0111-3"},{"key":"476_CR8","doi-asserted-by":"crossref","unstructured":"Mathews S. C. et al. Reply: Some considerations on digital health validation. npj Digit. Med. 2, Article number: 103 (2019).","DOI":"10.1038\/s41746-019-0176-z"},{"key":"476_CR9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0021-9","volume":"ume 1","author":"O Byambasuren","year":"2018","unstructured":"Byambasuren, O., Sanders, S., Beller, E. & Glasziou, P. Prescribable mHealth apps identified from an overview of systematic reviews. npj Digit. Med. ume 1, Article number: 12 (2018).","journal-title":"npj Digit. Med."},{"key":"476_CR10","doi-asserted-by":"publisher","first-page":"2310","DOI":"10.1377\/hlthaff.2016.0578","volume":"35","author":"K Singh","year":"2016","unstructured":"Singh, K. et al. Many mobile health apps target high-need, high-cost populations, but gaps remain. Health Aff. 35, 2310\u20132318 (2016).","journal-title":"Health Aff."},{"key":"476_CR11","doi-asserted-by":"publisher","first-page":"104001","DOI":"10.1016\/j.ijmedinf.2019.104001","volume":"133","author":"FA Ana","year":"2020","unstructured":"Ana, F. A. et al. Mobile applications in oncology: a systematic review of health science databases. Int. J. Med. Inform. 133, 104001 (2020).","journal-title":"Int. J. Med. Inform."},{"key":"476_CR12","doi-asserted-by":"publisher","first-page":"e27278","DOI":"10.1002\/pbc.27278","volume":"65","author":"JC Jupp","year":"2018","unstructured":"Jupp, J. C., Sultani, H., Cooper, C. A., Peterson, K. A. & Truong, T. H. Evaluation of mobile phone applications to support medication adherence and symptom management in oncology patients. Pediatr. Blood Cancer 65, e27278 (2018).","journal-title":"Pediatr. Blood Cancer"},{"key":"476_CR13","doi-asserted-by":"publisher","first-page":"e312","DOI":"10.2196\/jmir.6399","volume":"18","author":"KA Kessel","year":"2016","unstructured":"Kessel, K. A., Vogel, M. M., Schmidt-Graf, F. & Combs, S. E. Mobile apps in oncology: a survey on health care professionals\u2019 attitude toward telemedicine, mHealth, and oncological apps. J. Med. Internet Res. 18, e312 (2016).","journal-title":"J. Med. Internet Res."},{"key":"476_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1200\/CCI.17.00107","volume":"1","author":"CM Berkowitz","year":"2017","unstructured":"Berkowitz, C. M., Zullig, L. L., Koontz, B. F. & Smith, S. K. Prescribing an app? Oncology providers\u2019 views on mobile health apps for cancer care. JCO Clin. Cancer Inform. 1, 1\u20137 (2017).","journal-title":"JCO Clin. Cancer Inform."},{"key":"476_CR15","doi-asserted-by":"publisher","first-page":"e81","DOI":"10.2196\/mhealth.7689","volume":"5","author":"KA Kessel","year":"2017","unstructured":"Kessel, K. A. et al. Mobile health in oncology: a patient survey about app- assisted cancer care. JMIR mHealth uHealth 5, e81 (2017).","journal-title":"JMIR mHealth uHealth"},{"key":"476_CR16","doi-asserted-by":"crossref","unstructured":"Stoyanov, S. R. et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth 3, e27 (2015).","DOI":"10.2196\/mhealth.3422"},{"key":"476_CR17","doi-asserted-by":"publisher","first-page":"13","DOI":"10.24989\/dp.v1i2.1816","volume":"1","author":"T O\u2019Rourke","year":"2020","unstructured":"O\u2019Rourke, T., Pryss, R., Schlee, W. & Probst, T. Development of a multidimensional App-Quality Assessment Tool for Health-Related Apps (AQUA). Digit. Psychol. 1, 13\u201323 (2020).","journal-title":"Digit. Psychol."},{"key":"476_CR18","doi-asserted-by":"publisher","first-page":"e82","DOI":"10.2196\/jmir.7270","volume":"19","author":"A Baumel","year":"2017","unstructured":"Baumel, A., Faber, K., Mathur, N., Kane, J. M. & Muench, F. Enlight: a comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J. Med. Internet Res. 19, e82 (2017).","journal-title":"J. Med. Internet Res."},{"key":"476_CR19","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0268-9","volume":"ume 3","author":"DM Levine","year":"2020","unstructured":"Levine, D. M. et al. Design and testing of a mobile health application rating tool. npj Digit. Med. ume 3, Article number: 74 (2020).","journal-title":"npj Digit. Med."},{"key":"476_CR20","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1016\/j.amepre.2011.01.014","volume":"40","author":"SB Clauser","year":"2011","unstructured":"Clauser, S. B., Wagner, E. H., Bowles, E. J. A., Tuzzio, L. & Greene, S. M. Improving modern cancer care through information technology. Am. J. Prev. Med. 40, 198\u2013207 (2011).","journal-title":"Am. J. Prev. Med."},{"key":"476_CR21","unstructured":"IQVIA Institute. Global Oncology Trends 2018. Available at: https:\/\/www.iqvia.com\/institute\/reports\/global-oncology-trends. Accessed September 22, 2020."},{"key":"476_CR22","doi-asserted-by":"crossref","unstructured":"Atkins, D. et al. Grading quality of evidence and strength of recommendations. BMJ 328, Article 1490 (2004).","DOI":"10.1136\/bmj.328.7454.1490"},{"key":"476_CR23","unstructured":"World Wide Web Consortium (W3C) Web Accessibility Initiative (WAI). Web Content Accessibility Guidelines. Available at: https:\/\/www.w3.org\/WAI\/standards-guidelines\/wcag\/. Accessed September 22, 2020."},{"key":"476_CR24","unstructured":"Usability.gov Improving the User Experience. Available at: https:\/\/www.usability.gov\/how-to- and-tools\/methods\/system-usability-scale.html. Accessed February 10, 2021."},{"key":"476_CR25","unstructured":"HIMSS. 2019 Board Approved Xcertia Guidelines Issued on - HIMSS. Available at: https:\/\/www.himss.org\/sites\/hde\/files\/media\/file\/2020\/04\/17\/xcertia-guidelines-2019-final.pdf. Accessed September 22, 2020."},{"key":"476_CR26","unstructured":"Node.Health User Experience (UX) Measures. Available at: https:\/\/nodehealth.org\/ux-measures. Accessed September 22, 2020."},{"key":"476_CR27","unstructured":"Norton DWPD, Digital Context 2.0, Gifted Press, LLC, 2015, Chapter 1, and 3."},{"key":"476_CR28","unstructured":"Garrubba, M. J. C., Yap, G. & Melder, A. Design thinking methodology to improve the patient experience: an evidence snapshot. Centre for Clinical Effectiveness, Monash Innovation and Quality, Monash Health, Melbourne, Australia (2016)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00476-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00476-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00476-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T18:56:25Z","timestamp":1670093785000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00476-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,15]]},"references-count":28,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["476"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00476-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,15]]},"assertion":[{"value":"8 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 June 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 July 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"111"}}