{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T08:41:06Z","timestamp":1773477666850,"version":"3.50.1"},"reference-count":60,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,6,17]],"date-time":"2019-06-17T00:00:00Z","timestamp":1560729600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,6,17]],"date-time":"2019-06-17T00:00:00Z","timestamp":1560729600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Although &gt;10,000 behavioral health applications (\u201capps\u201d) are currently available on the Apple and Google Play marketplaces, they have been minimally evaluated or regulated and little is known about \u201creal world\u201d usage patterns. This investigation combined data from online behavioral health app rating frameworks and a mobile health market research firm to identify the most downloaded apps as well as determine rating and ranking concordance between frameworks. Findings demonstrated that the most commonly downloaded apps focus on relaxation, mindfulness, and meditation skills and that they often have notably discordant reviews across rating frameworks. Our results suggest that there is a growing need for: (1) standardized behavioral health app quality and effectiveness measures, (2) up-to-date behavioral health app guidance for clinicians and consumers, and (3) evidence-based apps that incorporate revealed consumer preferences.<\/jats:p>","DOI":"10.1038\/s41746-019-0129-6","type":"journal-article","created":{"date-parts":[[2019,6,17]],"date-time":"2019-06-17T10:02:40Z","timestamp":1560765760000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":166,"title":["By the numbers: ratings and utilization of behavioral health mobile applications"],"prefix":"10.1038","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4349-9644","authenticated-orcid":false,"given":"Andrew D.","family":"Carlo","sequence":"first","affiliation":[]},{"given":"Reza","family":"Hosseini Ghomi","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8746-171X","authenticated-orcid":false,"given":"Brenna N.","family":"Renn","sequence":"additional","affiliation":[]},{"given":"Patricia A.","family":"Are\u00e1n","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,6,17]]},"reference":[{"key":"129_CR1","unstructured":"Aitken, M. & Lyle, J. Patient Adoption of mHealth - Use, Evidence and Remaining Barriers to Mainstream Acceptance (IMS Institute for Healthcare Informatics, Parsippany, NJ, 2015)."},{"key":"129_CR2","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/j.cbpra.2018.01.002","volume":"25","author":"M Neary","year":"2018","unstructured":"Neary, M. & Schueller, S. M. State of the field of mental health apps. Cogn. Behav. Pract. 25, 531 (2018).","journal-title":"Cogn. Behav. Pract."},{"key":"129_CR3","unstructured":"Pohl, M. 325,000 mobile health apps available in 2017 \u2013 Android now the leading mHealth platform. https:\/\/research2guidance.com\/325000-mobile-health-apps-available-in-2017\/ (2017)."},{"key":"129_CR4","doi-asserted-by":"publisher","first-page":"734","DOI":"10.4088\/JCP.15com10619","volume":"77","author":"JB Torous","year":"2016","unstructured":"Torous, J. B., Chan, S. R., Yellowlees, P. M. & Boland, R. To use or not? Evaluating ASPECTS of smartphone apps and mobile technology for clinical care in psychiatry. J. Clin. Psychiatry 77, 734\u2013738 (2016).","journal-title":"J. Clin. Psychiatry"},{"key":"129_CR5","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1097\/NMD.0000000000000864","volume":"206","author":"J Torous","year":"2018","unstructured":"Torous, J. et al. The emerging imperative for a consensus approach toward the rating and clinical recommendation of mental health apps. J. Nerv. Ment. Dis. 206, 662\u2013666 (2018).","journal-title":"J. Nerv. Ment. Dis."},{"key":"129_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.cpr.2015.11.004","volume":"43","author":"A Kuester","year":"2016","unstructured":"Kuester, A., Niemeyer, H. & Knaevelsrud, C. Internet-based interventions for posttraumatic stress: a meta-analysis of randomized controlled trials. Clin. Psychol. Rev. 43, 1\u201316 (2016).","journal-title":"Clin. Psychol. Rev."},{"key":"129_CR7","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1176\/appi.ps.201600541","volume":"68","author":"DC Mohr","year":"2017","unstructured":"Mohr, D. C., Weingardt, K. R., Reddy, M. & Schueller, S. M. Three problems with current digital mental health research\u2026 and three things we can do about them. Psychiatr. Serv. 68, 427\u2013429 (2017).","journal-title":"Psychiatr. Serv."},{"key":"129_CR8","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1016\/j.cpr.2012.02.004","volume":"32","author":"D Richards","year":"2012","unstructured":"Richards, D. & Richardson, T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin. Psychol. Rev. 32, 329\u2013342 (2012).","journal-title":"Clin. Psychol. Rev."},{"key":"129_CR9","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/532020a","volume":"532","author":"E Anthes","year":"2016","unstructured":"Anthes, E. Pocket psychiatry - mobile mental-health apps have exploded onto the market, but few have been thoroughly tested. Nature 532, 20\u201323 (2016).","journal-title":"Nature"},{"key":"129_CR10","doi-asserted-by":"publisher","first-page":"h5627","DOI":"10.1136\/bmj.h5627","volume":"351","author":"S Gilbody","year":"2015","unstructured":"Gilbody, S. et al. Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. BMJ 351, h5627 (2015).","journal-title":"BMJ"},{"key":"129_CR11","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.6482","volume":"18","author":"PA Arean","year":"2016","unstructured":"Arean, P. A. et al. The use and effectiveness of mobile apps for depression: results from a fully remote clinical trial. J. Med. Internet Res. 18, e330 (2016).","journal-title":"J. Med. Internet Res."},{"key":"129_CR12","doi-asserted-by":"publisher","first-page":"793","DOI":"10.1093\/tbm\/ibx037","volume":"8","author":"A Baumel","year":"2018","unstructured":"Baumel, A. & Yom-Tov, E. Predicting user adherence to behavioral eHealth interventions in the real world: examining which aspects of intervention design matter most. Transl. Behav. Med 8, 793\u2013798 (2018).","journal-title":"Transl. Behav. Med"},{"key":"129_CR13","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2665","volume":"15","author":"LF Kohl","year":"2013","unstructured":"Kohl, L. F., Crutzen, R. & de Vries, N. K. Online prevention aimed at lifestyle behaviors: a systematic review of reviews. J. Med Internet Res 15, e146 (2013).","journal-title":"J. Med Internet Res"},{"key":"129_CR14","doi-asserted-by":"publisher","DOI":"10.2196\/mhealth.3196","volume":"2","author":"JA Lee","year":"2014","unstructured":"Lee, J. A. et al. Attitudes and preferences on the use of mobile health technology and health games for self-management: interviews with older adults on anticoagulation therapy. JMIR Mhealth Uhealth 2, e32 (2014).","journal-title":"JMIR Mhealth Uhealth"},{"key":"129_CR15","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1016\/j.cct.2015.07.007","volume":"45","author":"C Rosa","year":"2015","unstructured":"Rosa, C., Campbell, A. N., Miele, G. M., Brunner, M. & Winstanley, E. L. Using e-technologies in clinical trials. Conte. Clin. Trials 45, 41\u201354 (2015).","journal-title":"Conte. Clin. Trials"},{"key":"129_CR16","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/j.cbpra.2016.05.007","volume":"24","author":"J Torous","year":"2017","unstructured":"Torous, J., Levin, M. E., Ahern, D. K. & Oser, M. L. Cognitive behavioral mobile applications: clinical studies, marketplace overview, and research agenda. Cogn. Behav. Pract. 24, 215\u2013225 (2017).","journal-title":"Cogn. Behav. Pract."},{"issue":"1","key":"129_CR17","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1136\/ebmental-2018-300069","volume":"22","author":"Hannah Wisniewski","year":"2019","unstructured":"Wisniewski, H. et al. Understanding the quality, effectiveness and attributes of top- rated smartphone health apps. Evid Based Mental Health https:\/\/doi.org\/10.1136\/ebmental-2018-300069 (2019).","journal-title":"Evidence Based Mental Health"},{"key":"129_CR18","doi-asserted-by":"publisher","DOI":"10.2196\/mhealth.4924","volume":"3","author":"P Krebs","year":"2015","unstructured":"Krebs, P. & Duncan, D. T. Health app use among US mobile phone owners: a national survey. JMIR Mhealth Uhealth 3, e101 (2015).","journal-title":"JMIR Mhealth Uhealth"},{"key":"129_CR19","doi-asserted-by":"publisher","DOI":"10.2196\/mhealth.6827","volume":"5","author":"HY Huang","year":"2017","unstructured":"Huang, H. Y. & Bashir, M. Users\u2019 adoption of mental health apps: examining the impact of information cues. JMIR Mhealth Uhealth 5, e83 (2017).","journal-title":"JMIR Mhealth Uhealth"},{"key":"129_CR20","unstructured":"American Psychiatric Association (APA). App evaluation model. https:\/\/www.psychiatry.org\/psychiatrists\/practice\/mental-health-apps\/app-evaluation-model (2018)."},{"key":"129_CR21","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.7270","volume":"19","author":"A Baumel","year":"2017","unstructured":"Baumel, A., Faber, K., Mathur, N., Kane, J. M. & Muench, F. Enlight: a comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J. Med. Internet Res. 19, e82 (2017).","journal-title":"J. Med. Internet Res."},{"key":"129_CR22","unstructured":"Schueller, S. Should you trust mental health apps? http:\/\/www.theneuroethicsblog.com\/2018\/05\/should-you-trust-mental-health-apps.html (2018)."},{"key":"129_CR23","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1017\/S026646231800051X","volume":"34","author":"MR Moshi","year":"2018","unstructured":"Moshi, M. R., Tooher, R. & Merlin, T. Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review. Int. J. Technol. Assess. Health Care 34, 464\u2013475 (2018).","journal-title":"Int. J. Technol. Assess. Health Care"},{"key":"129_CR24","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1002\/wps.20592","volume":"18","author":"J Torous","year":"2019","unstructured":"Torous, J. et al. Towards a consensus around standards for smartphone apps and digital mental health. World Psychiatry 18, 97\u201398 (2019).","journal-title":"World Psychiatry"},{"key":"129_CR25","unstructured":"National Institute of Mental Health. Technology and the future of mental health treatment. https:\/\/www.nimh.nih.gov\/health\/topics\/technology-and-the-future-of-mental-health-treatment\/index.shtml (2018)."},{"key":"129_CR26","unstructured":"United States Food and Drug Administration. Digital Health Software Precertification (Pre-Cert) Program. https:\/\/www.fda.gov\/medicaldevices\/digitalhealth\/digitalhealthprecertprogram\/default.htm (2019)."},{"key":"129_CR27","unstructured":"World Health Organization. Classification of Digital Health Interventions v1.0. 1-20 (2018)."},{"key":"129_CR28","unstructured":"Peak-Brain Training. Peak-Brain Training. http:\/\/www.peak.net (2019)."},{"key":"129_CR29","unstructured":"Stem4 - stemming teenage mental illness; supporting teenage mental health. Calm Harm. https:\/\/stem4.org.uk (2019)."},{"key":"129_CR30","unstructured":"Lumosity. https:\/\/www.lumosity.com\/en\/ (2019)."},{"key":"129_CR31","unstructured":"Talkspace. https:\/\/www.talkspace.com (2019)."},{"key":"129_CR32","unstructured":"Happify. https:\/\/www.happify.com (2019)."},{"key":"129_CR33","unstructured":"Fabulous. https:\/\/thefabulous.co\/ (2019)."},{"key":"129_CR34","unstructured":"Daylio. https:\/\/daylio.webflow.io (2019)."},{"key":"129_CR35","unstructured":"Organization for the Review of Care and Health Applications (ORCHA). https:\/\/www.orcha.co.uk (2019)."},{"key":"129_CR36","unstructured":"Psyberguide. Psyberguide - A Project of One Mind. https:\/\/psyberguide.org (2019)."},{"key":"129_CR37","unstructured":"Mindtools.io. https:\/\/mindtools.io\/find-a-program\/ (2019)."},{"key":"129_CR38","unstructured":"Headspace. https:\/\/www.headspace.com (2019)."},{"key":"129_CR39","unstructured":"Calm. https:\/\/www.calm.com (2019)."},{"key":"129_CR40","unstructured":"7 Cups. 7 Cups of Tea. https:\/\/www.7cups.com (2019)."},{"key":"129_CR41","unstructured":"PrioriData. https:\/\/prioridata.com (2019)."},{"key":"129_CR42","doi-asserted-by":"crossref","unstructured":"Ng, M. M., Firth, J., Minen, M. & Torous, J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr. Serv. appips201800519 (2019).","DOI":"10.1176\/appi.ps.201800519"},{"key":"129_CR43","unstructured":"SimilarWeb. https:\/\/www.similarweb.com (2019)."},{"key":"129_CR44","doi-asserted-by":"publisher","first-page":"010502","DOI":"10.7189\/jogh.08.010502","volume":"8","author":"K Wazny","year":"2018","unstructured":"Wazny, K. Applications of crowdsourcing in health: an overview. J. Glob. Health 8, 010502 (2018).","journal-title":"J. Glob. Health"},{"key":"129_CR45","unstructured":"American Psychiatric Association (APA). App Evaluation Model. https:\/\/www.psychiatry.org\/psychiatrists\/practice\/mental-health-apps\/app-evaluation-model (2019)."},{"key":"129_CR46","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1377\/hlthaff.2018.05081","volume":"38","author":"K Safavi","year":"2019","unstructured":"Safavi, K., Mathews, S. C., Bates, D. W., Dorsey, E. R. & Cohen, A. B. Top-funded digital health companies and their impact on high-burden, high-cost conditions. Health Affairs (Millwood) 38, 115\u2013123 (2019).","journal-title":"Health Affairs (Millwood)"},{"key":"129_CR47","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2019.2542","volume":"2","author":"K Huckvale","year":"2019","unstructured":"Huckvale, K., Torous, J. & Larsen, M. E. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw. Open 2, e192542 (2019).","journal-title":"JAMA Netw. Open"},{"key":"129_CR48","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1097\/ACM.0000000000001205","volume":"92","author":"ER Dorsey","year":"2017","unstructured":"Dorsey, E. R. et al. The use of smartphones for health research. Acad. Med. 92, 157\u2013160 (2017).","journal-title":"Acad. Med."},{"key":"129_CR49","doi-asserted-by":"publisher","unstructured":"Renn, B. N., Hoeft, T. J., Lee, H. S., Bauer, A. M. & Are\u00e1n, P. A. Preference for in-person psychotherapy versus digital psychotherapy options for depression: survey of adults in the U.S. npj Digital Med. 2, https:\/\/doi.org\/10.1038\/s41746-019-0077-1 (2019).","DOI":"10.1038\/s41746-019-0077-1"},{"key":"129_CR50","unstructured":"RankedHealth. Curated health apps & devices - with a focus on clinical relevance, safety, and efficacy. http:\/\/www.rankedhealth.com (2019)."},{"key":"129_CR51","unstructured":"Anxiety and Depression Association of America. ADAA Reviewed Mental Health Apps. https:\/\/adaa.org\/finding-help\/mobile-apps (2019)."},{"issue":"4","key":"129_CR52","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/j.cbpra.2018.01.002","volume":"25","author":"Martha Neary","year":"2018","unstructured":"Neary, M. & Schueller, S. State of the field of mental health apps. Cognitive and Behavioral Practice. 25, 531\u2013537. https:\/\/doi.org\/10.1016\/j.cbpra.2018.01.002 (2018).","journal-title":"Cognitive and Behavioral Practice"},{"key":"129_CR53","unstructured":"IQVIA. AppScript. https:\/\/www.appscript.net\/ (2019)."},{"issue":"14","key":"129_CR54","doi-asserted-by":"publisher","first-page":"1856","DOI":"10.1001\/jama.285.14.1856","volume":"285","author":"Jack D. Edinger","year":"2001","unstructured":"Edinger, J. D., Wohlgemuth, W. K., Radtke, R. A., Marsh, G. R. & Quillian, R. E. Cognitive behavioral therapy for treatment of chronic primary insomnia. JAMA 285, https:\/\/doi.org\/10.1001\/jama.285.14.1856 (2001).","journal-title":"JAMA"},{"key":"129_CR55","doi-asserted-by":"crossref","unstructured":"Williams, G. & Mahmoud, A. Modeling user concerns in the app store: a case study on the rise and fall of yik yak. In 2018 IEEE 26th International Requirements Engineering Conference (RE) 64\u201375 (IEEE, 2018).","DOI":"10.1109\/RE.2018.00-51"},{"key":"129_CR56","doi-asserted-by":"publisher","unstructured":"Boudreau, K., Jeppesen, L. B. & Miric, M. Competing on free(mium): digital competition with network effects. SSRN Electron. J. https:\/\/doi.org\/10.2139\/ssrn.2984546 (2017).","DOI":"10.2139\/ssrn.2984546"},{"key":"129_CR57","doi-asserted-by":"publisher","DOI":"10.1111\/jems.12288","author":"S Comino","year":"2018","unstructured":"Comino, S., Manenti, F. M. & Mariuzzo, F. Updates management in mobile applications: iTunes vs Google Play. J. Econ. Manag. Strategy https:\/\/doi.org\/10.1111\/jems.12288 (2018).","journal-title":"J. Econ. Manag. Strategy"},{"key":"129_CR58","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1007\/s11135-014-0003-1","volume":"49","author":"R Falotico","year":"2014","unstructured":"Falotico, R. & Quatto, P. Fleiss\u2019 kappa statistic without paradoxes. Qual. Quant. 49, 463\u2013470 (2014).","journal-title":"Qual. Quant."},{"key":"129_CR59","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33, 159\u2013174 (1977).","journal-title":"Biometrics"},{"key":"129_CR60","unstructured":"Gamer, M. Package \u2018irr\u2019. https:\/\/cran.r-project.org\/web\/packages\/irr\/irr.pdf (2019)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0129-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0129-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0129-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T18:29:19Z","timestamp":1671301759000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0129-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,17]]},"references-count":60,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["129"],"URL":"https:\/\/doi.org\/10.1038\/s41746-019-0129-6","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,17]]},"assertion":[{"value":"22 February 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 May 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 June 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"54"}}