{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T09:48:34Z","timestamp":1773827314281,"version":"3.50.1"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,5,21]],"date-time":"2020-05-21T00:00:00Z","timestamp":1590019200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,5,21]],"date-time":"2020-05-21T00:00:00Z","timestamp":1590019200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000905","name":"Commonwealth Fund","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000905","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Mobile health applications (\u201capps\u201d) have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps\u2019 potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with \u22654 stars and &lt;4-stars and then rated them with THESIS to examine the tool\u2019s interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96\u20133.09]), but especially poorly for privacy\/security (2.21 out of 5 [95% CI, 2.11\u20132.32]), interoperability (1.75 [95% CI, 1.59\u20131.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30\u20131.56]). Ratings using THESIS had fair interrater reliability (<jats:italic>\u03ba<\/jats:italic>\u2009=\u20090.3\u20130.6) and excellent scale reliability (<jats:italic>\u0251<\/jats:italic>\u2009=\u20090.85). Correlation with traditional star ratings was low (<jats:italic>r<\/jats:italic>\u2009=\u20090.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy\/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.<\/jats:p>","DOI":"10.1038\/s41746-020-0268-9","type":"journal-article","created":{"date-parts":[[2020,5,21]],"date-time":"2020-05-21T10:02:26Z","timestamp":1590055346000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":59,"title":["Design and testing of a mobile health application rating tool"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4337-9896","authenticated-orcid":false,"given":"David M.","family":"Levine","sequence":"first","affiliation":[]},{"given":"Zoe","family":"Co","sequence":"additional","affiliation":[]},{"given":"Lisa P.","family":"Newmark","sequence":"additional","affiliation":[]},{"given":"Alissa R.","family":"Groisser","sequence":"additional","affiliation":[]},{"given":"A. Jay","family":"Holmgren","sequence":"additional","affiliation":[]},{"given":"Jennifer S.","family":"Haas","sequence":"additional","affiliation":[]},{"given":"David W.","family":"Bates","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,5,21]]},"reference":[{"key":"268_CR1","unstructured":"The Growing Value of Digital Health. https:\/\/www.iqvia.com\/institute\/reports\/the-growing-value-of-digital-health (2017)."},{"key":"268_CR2","unstructured":"Demographics of Mobile Device Ownership and Adoption in the United States. Pew Research Center. http:\/\/www.pewinternet.org\/fact-sheet\/mobile\/ (2018)."},{"issue":"4","key":"268_CR3","doi-asserted-by":"publisher","first-page":"e101","DOI":"10.2196\/mhealth.4924","volume":"3","author":"P Krebs","year":"2015","unstructured":"Krebs, P. & Duncan, D. T. Health app use among US mobile phone owners: a national survey. JMIR mHealth uHealth. 3(4), e101 (2015).","journal-title":"JMIR mHealth uHealth."},{"issue":"4","key":"268_CR4","doi-asserted-by":"publisher","first-page":"e125","DOI":"10.2196\/jmir.5604","volume":"19","author":"JK Carroll","year":"2017","unstructured":"Carroll, J. K. et al. Who uses mobile phone health apps and does use matter? A secondary data analytics approach. J. Med. Internet Res. 19(4), e125 (2017).","journal-title":"J. Med. Internet Res."},{"issue":"1","key":"268_CR5","doi-asserted-by":"publisher","DOI":"10.1186\/s12966-016-0454-y","volume":"13","author":"S Schoeppe","year":"2016","unstructured":"Schoeppe, S. et al. Efficacy of interventions that use apps to improve diet, physical activity and sedentary behaviour: a systematic review. Int. J. Behav. Nutr. Phys. Act. 13(1), 127 (2016).","journal-title":"Int. J. Behav. Nutr. Phys. Act."},{"issue":"3","key":"268_CR6","doi-asserted-by":"publisher","first-page":"e4","DOI":"10.2196\/mhealth.6309","volume":"5","author":"BC Bonoto","year":"2017","unstructured":"Bonoto, B. C. et al. Efficacy of mobile apps to support the care of patients with diabetes mellitus: a systematic review and meta-analysis of randomized controlled trials. JMIR mHealth uHealth. 5(3), e4 (2017).","journal-title":"JMIR mHealth uHealth."},{"issue":"5","key":"268_CR7","doi-asserted-by":"publisher","first-page":"e97","DOI":"10.2196\/jmir.4883","volume":"18","author":"L Whitehead","year":"2016","unstructured":"Whitehead, L. & Seaton, P. The effectiveness of self-management mobile phone and tablet apps in long-term condition management: a systematic review. J. Med. Internet Res. 18(5), e97 (2016).","journal-title":"J. Med. Internet Res."},{"issue":"4","key":"268_CR8","doi-asserted-by":"publisher","first-page":"e136","DOI":"10.2196\/mhealth.6445","volume":"4","author":"K Singh","year":"2016","unstructured":"Singh, K. et al. Patient-facing mobile apps to treat high-need, high-cost populations: a scoping review. JMIR mHealth uHealth. 4(4), e136 (2016).","journal-title":"JMIR mHealth uHealth."},{"key":"268_CR9","unstructured":"23% of Users Abandon an App After One Use. Localytics. http:\/\/info.localytics.com\/blog\/23-of-users-abandon-an-app-after-one-use (2016)."},{"key":"268_CR10","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2018.6716.","author":"DM Levine","year":"2019","unstructured":"Levine, D. M., Landon, B. E. & Linder, J. A. Quality and experience of outpatient care in the United States for adults with or without primary care. JAMA Intern. Med. https:\/\/doi.org\/10.1001\/jamainternmed.2018.6716. (2019).","journal-title":"JAMA Intern. Med."},{"issue":"3","key":"268_CR11","doi-asserted-by":"publisher","first-page":"e35","DOI":"10.2196\/mhealth.6522","volume":"5","author":"Y Wu","year":"2017","unstructured":"Wu, Y. et al. Mobile app-based interventions to support diabetes self-management: a systematic review of randomized controlled trials to identify functions associated with glycemic efficacy. JMIR mHealth uHealth. 5(3), e35 (2017).","journal-title":"JMIR mHealth uHealth."},{"issue":"5","key":"268_CR12","doi-asserted-by":"publisher","first-page":"590","DOI":"10.1111\/den.12467","volume":"27","author":"V Lorenzo-Z\u00fa\u00f1iga","year":"2015","unstructured":"Lorenzo-Z\u00fa\u00f1iga, V., Moreno de Vega, V., Mar\u00edn, I., Barber\u00e1, M. & Boix, J. Improving the quality of colonoscopy bowel preparation using a smart phone application: a randomized trial. Dig. Endosc. 27(5), 590\u2013595 (2015).","journal-title":"Dig. Endosc."},{"issue":"21","key":"268_CR13","doi-asserted-by":"publisher","first-page":"2616","DOI":"10.1001\/jama.291.21.2616","volume":"291","author":"D Yach","year":"2004","unstructured":"Yach, D., Hawkes, C., Gould, C. L. & Hofman, K. J. The global burden of chronic diseases. JAMA 291(21), 2616 (2004).","journal-title":"JAMA"},{"issue":"14","key":"268_CR14","doi-asserted-by":"publisher","first-page":"1444","DOI":"10.1001\/jama.2018.0158","volume":"319","author":"AH Mokdad","year":"2018","unstructured":"Mokdad, A. H. et al. The state of US health. JAMA 319(14), 1444 (2018).","journal-title":"JAMA"},{"issue":"11","key":"268_CR15","doi-asserted-by":"publisher","first-page":"e0206702","DOI":"10.1371\/journal.pone.0206702","volume":"13","author":"S Chen","year":"2018","unstructured":"Chen, S., Kuhn, M., Prettner, K. & Bloom, D. E. The macroeconomic burden of noncommunicable diseases in the United States: estimates and projections. PLoS One 13(11), e0206702 (2018).","journal-title":"PLoS One"},{"key":"268_CR16","doi-asserted-by":"publisher","first-page":"f1811","DOI":"10.1136\/bmj.f1811","volume":"346","author":"M McCartney","year":"2013","unstructured":"McCartney, M. How do we know whether medical apps work? BMJ 346, f1811 (2013).","journal-title":"BMJ"},{"issue":"5","key":"268_CR17","doi-asserted-by":"publisher","first-page":"700","DOI":"10.1001\/jamainternmed.2016.0157","volume":"176","author":"TB Plante","year":"2016","unstructured":"Plante, T. B. et al. Validation of the instant blood pressure smartphone app. JAMA Intern. Med. 176(5), 700 (2016).","journal-title":"JAMA Intern. Med."},{"issue":"4","key":"268_CR18","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1001\/jamadermatol.2013.2382","volume":"149","author":"JA Wolf","year":"2013","unstructured":"Wolf, J. A. et al. Diagnostic inaccuracy of smartphone applications for melanoma detection. JAMA Dermatol. 149(4), 422 (2013).","journal-title":"JAMA Dermatol."},{"key":"268_CR19","unstructured":"FTC Fines Apps Firms for Claiming Their Technology Could Detect Melanoma. The Guardian. https:\/\/www.theguardian.com\/technology\/2015\/feb\/25\/ftc-fines-apps-detect-melanoma (2015)."},{"key":"268_CR20","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2016.0400","author":"AS Miner","year":"2016","unstructured":"Miner, A. S. et al. Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health. JAMA Intern. Med. https:\/\/doi.org\/10.1001\/jamainternmed.2016.0400 (2016).","journal-title":"JAMA Intern. Med."},{"key":"268_CR21","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-015-0451-z","author":"P Wicks","year":"2015","unstructured":"WicksP. & ChiauzziE. \u201cTrust but verify\u201d\u2014five approaches to ensure safe medical apps. BMC Med. https:\/\/doi.org\/10.1186\/s12916-015-0451-z (2015).","journal-title":"BMC Med."},{"key":"268_CR22","doi-asserted-by":"publisher","DOI":"10.1515\/dx-2018-0009","author":"ML Millenson","year":"2018","unstructured":"MillensonM. L., BaldwinJ. L., ZippererL. & SinghH. Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis. Diagnosis https:\/\/doi.org\/10.1515\/dx-2018-0009 (2018).","journal-title":"Diagnosis"},{"issue":"12","key":"268_CR23","doi-asserted-by":"publisher","first-page":"2310","DOI":"10.1377\/hlthaff.2016.0578","volume":"35","author":"K Singh","year":"2016","unstructured":"Singh, K. et al. Many mobile health apps target high-need, high-cost populations, but gaps remain. Health Aff. 35(12), 2310\u20132318 (2016).","journal-title":"Health Aff."},{"issue":"2","key":"268_CR24","doi-asserted-by":"publisher","first-page":"e72","DOI":"10.2196\/mhealth.5849","volume":"4","author":"SR Stoyanov","year":"2016","unstructured":"Stoyanov, S. R., Hides, L., Kavanagh, D. J. & Wilson, H. Development and validation of the user version of the Mobile Application Rating Scale (uMARS). JMIR mHealth uHealth. 4(2), e72 (2016).","journal-title":"JMIR mHealth uHealth."},{"key":"268_CR25","unstructured":"Medical App Checker: A Guide to Assessing Mobile Medical Apps. https:\/\/www.knmg.nl\/actualiteit-opinie\/nieuws\/nieuwsbericht\/medical-app-checker-a-guide-to-assessing-mobile-medical-apps.htm (2016)."},{"key":"268_CR26","unstructured":"App Evaluation Model. https:\/\/www.psychiatry.org\/psychiatrists\/practice\/mental-health-apps\/app-evaluation-model (2017)."},{"key":"268_CR27","unstructured":"Selecting a Mobile App: Evaluating the Usability of Medical Applications. http:\/\/www.himss.org\/selecting-mobile-app-evaluating-usability-medical-applications-0 (2012)."},{"issue":"3","key":"268_CR28","doi-asserted-by":"publisher","first-page":"e82","DOI":"10.2196\/jmir.7270","volume":"19","author":"A Baumel","year":"2017","unstructured":"Baumel, A. et al. A comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J. Med. Internet Res. 19(3), e82 (2017).","journal-title":"J. Med. Internet Res."},{"key":"268_CR29","unstructured":"App Guide. https:\/\/psyberguide.org\/apps\/ (2013)."},{"issue":"1","key":"268_CR30","doi-asserted-by":"publisher","first-page":"e11130","DOI":"10.2196\/11130","volume":"7","author":"FH McKay","year":"2019","unstructured":"McKay, F. H., Slykerman, S. & Dunn, M. The app behavior change scale: creation of a scale to assess the potential of apps to promote behavior change. JMIR mHealth uHealth. 7(1), e11130 (2019).","journal-title":"JMIR mHealth uHealth."},{"key":"268_CR31","unstructured":"Calidad App Salud. http:\/\/www.calidadappsalud.com\/en\/ (2012)."},{"key":"268_CR32","unstructured":"Assessment and Improvement of Practice. Good Practice Guidelines on Health Apps and Smart Devices (Mobile Health or MHealth). Saint-Denis La Plaine Cedex. https:\/\/www.has-sante.fr\/portail\/upload\/docs\/application\/pdf\/2017-03\/dir1\/good_practice_guidelines_on_health_apps_and_smart_devices_mobile_health_or_mhealth.pdf (2017)."},{"key":"268_CR33","unstructured":"Evaluation Methodology. Medappcare. https:\/\/www.medappcare.com\/en\/methodologie-devaluation\/ (2014)."},{"key":"268_CR34","unstructured":"ZTG AppCheck. https:\/\/appcheck.de\/kriterienkatalog (2019)."},{"key":"268_CR35","unstructured":"iPrescribeApps\u2014From Clinical Problem to Mobile Solution. https:\/\/www.iprescribeapps.com\/ (2016)."},{"key":"268_CR36","unstructured":"DMD Sant\u00e9. ApiAppS. https:\/\/projet-apiapps.org\/en\/partners\/dmd-sant\u00e9 (2012)."},{"key":"268_CR37","unstructured":"Groupe Pasteur Mutualite\u2014D\u00e9couvrez l\u2019application mobile GPM e-sant\u00e9. https:\/\/www.gpm.fr\/actualite.html?id=10093 (2019)."},{"key":"268_CR38","unstructured":"NHS Apps Library\u2014NHS Digital. https:\/\/digital.nhs.uk\/services\/nhs-apps-library (2019)."},{"key":"268_CR39","unstructured":"Methodology. myhealthapps.net. http:\/\/myhealthapps.net\/methodology (2015)."},{"key":"268_CR40","doi-asserted-by":"crossref","unstructured":"Xie. Z. & Zhu, S. AppWatcher: unveiling the underground market of trading mobile app reviews. In Proceedings of the 8th ACM Conference on Security & Privacy in Wireless and Mobile Networks\u2014WiSec\u201915. 1\u201311 (ACM Press, New York, 2015).","DOI":"10.1145\/2766498.2766510"},{"key":"268_CR41","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1016\/j.jss.2016.11.027","volume":"125","author":"N Genc-Nayebi","year":"2017","unstructured":"Genc-Nayebi, N. & Abran, A. A systematic literature review: opinion mining studies from mobile app store user reviews. J. Syst. Softw. 125, 207\u2013219 (2017).","journal-title":"J. Syst. Softw."},{"issue":"1","key":"268_CR42","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1177\/2374373516636681","volume":"3","author":"S Trzeciak","year":"2016","unstructured":"Trzeciak, S., Gaughan, J. P., Bosire, J. & Mazzarelli, A. J. Association between medicare summary star ratings for patient experience and clinical outcomes in US hospitals. J. Patient Exp. 3(1), 6\u20139 (2016).","journal-title":"J. Patient Exp."},{"key":"268_CR43","first-page":"E66","volume":"10","author":"RA Goodman","year":"2013","unstructured":"Goodman, R. A., Posner, S. F., Huang, E. S., Parekh, A. K. & Koh, H. K. Defining and measuring chronic conditions: imperatives for research, policy, program, and practice. Prev. Chronic Dis. 10, E66 (2013).","journal-title":"Prev. Chronic Dis."},{"issue":"4","key":"268_CR44","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1001\/jama.2018.8832","volume":"320","author":"J Shuren","year":"2018","unstructured":"Shuren, J., Patel, B. & Gottlieb, S. FDA regulation of mobile medical apps. JAMA 320(4), 337 (2018).","journal-title":"JAMA"},{"issue":"3","key":"268_CR45","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1159\/000494365","volume":"2","author":"AR Sheon","year":"2018","unstructured":"Sheon, A. R., Van Winkle, B., Solad, Y. & Atreja, A. An algorithm for digital medicine testing: A NODE. Health perspective intended to help emerging technology companies and healthcare systems navigate the trial and testing period prior to full-scale adoption. Digit. Biomark. 2(3), 139\u2013154 (2018).","journal-title":"Digit. Biomark."},{"issue":"1","key":"268_CR46","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1108\/09654280310459112","volume":"103","author":"J de Meyrick","year":"2013","unstructured":"de Meyrick, J. The Delphi method and health research. Health Educ. 103(1), 7\u201316 (2013).","journal-title":"Health Educ."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0268-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0268-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0268-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:15:15Z","timestamp":1670379315000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0268-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,21]]},"references-count":46,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["268"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0268-9","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,21]]},"assertion":[{"value":"20 June 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 March 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 May 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"D.M.L. is the PI of an investigator-initiated study in collaboration with the for-profit entity Biofourmis, Ltd. to refine a predictive analytics algorithm for home hospitalized patients. D.W.B. consults for EarlySense, which makes patient safety monitoring systems. He receives cash compensation from CDI (Negev), Ltd., which is a not-for-profit incubator for health IT startups. He receives equity from ValeraHealth, which makes software to help patients with chronic diseases. He receives equity from Clew which makes software to support clinical decision-making in intensive care. He receives equity from MDClone, which takes clinical data and produces deidentified versions of it. D.W.B.\u2019s financial interests have been reviewed by Brigham and Women\u2019s Hospital and Partners HealthCare in accordance with their institutional policies. All other authors have no disclosures.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"74"}}