{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T11:10:12Z","timestamp":1772277012117,"version":"3.50.1"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T00:00:00Z","timestamp":1596067200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T00:00:00Z","timestamp":1596067200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>As use and availability of mobile health apps have increased, so too has the need for a thorough, accessible framework for app evaluation. The American Psychiatric Association\u2019s app evaluation model has emerged as a way to critically assess an app by considering accessibility, privacy and security, clinical foundation, engagement, and interoperability; however, there is no centralized database where users can view how various health apps perform when assessed via the APA model. In this perspective, we propose and outline our effort to translate the APA\u2019s model for the evaluation of health apps into a set of objective metrics that can be published online, making the framework actionable and accessible to a broad audience. The questions from the APA model were operationalized into 105 objective questions that are either binary or numeric. These questions serve as the foundation of an online database, where app evaluation consists of answering these 105 questions and can be crowdsourced. While the database has yet to be published and crowdsourced, initial internal testing demonstrated excellent interrater reliability. The database proposed here introduces a public and interactive approach to data collection that is guided by the APA model. The published product enables users to sort through the many mobile health apps and filter them according to individual preferences and priorities, making the ever-growing health app market more navigable.<\/jats:p>","DOI":"10.1038\/s41746-020-00312-4","type":"journal-article","created":{"date-parts":[[2020,7,30]],"date-time":"2020-07-30T10:24:33Z","timestamp":1596104673000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":128,"title":["Actionable health app evaluation: translating expert frameworks into objective metrics"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7540-5238","authenticated-orcid":false,"given":"Sarah","family":"Lagan","sequence":"first","affiliation":[]},{"given":"Patrick","family":"Aquino","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1103-8800","authenticated-orcid":false,"given":"Margaret R.","family":"Emerson","sequence":"additional","affiliation":[]},{"given":"Karen","family":"Fortuna","sequence":"additional","affiliation":[]},{"given":"Robert","family":"Walker","sequence":"additional","affiliation":[]},{"given":"John","family":"Torous","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,7,30]]},"reference":[{"key":"312_CR1","doi-asserted-by":"publisher","DOI":"10.1007\/s11920-019-0997-0","volume":"21","author":"J Rehm","year":"2019","unstructured":"Rehm, J. & Shield, K. D. Global burden of disease and the impact of mental and addictive disorders. Curr. Psychiatry Rep. 21, 10 (2019).","journal-title":"Curr. Psychiatry Rep."},{"key":"312_CR2","doi-asserted-by":"publisher","first-page":"947","DOI":"10.1007\/s11126-018-9592-5","volume":"89","author":"KL Fortuna","year":"2018","unstructured":"Fortuna, K. L. et al. Smartphone ownership, use, and willingness to use smartphones to provide peer-delivered services: results from a national online survey. Psychiatr. Quart. 89, 947\u2013956 (2018).","journal-title":"Psychiatr. Quart."},{"key":"312_CR3","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1001\/jamapsychiatry.2017.0262","volume":"74","author":"J Torous","year":"2017","unstructured":"Torous, J. & Roberts, L. W. Needed innovation in digital health and smartphone applications for mental health: transparency and trust. JAMA Psychiatry 74, 437\u2013438 (2017).","journal-title":"JAMA Psychiatry"},{"key":"312_CR4","doi-asserted-by":"crossref","unstructured":"Larsen, M. E., Nicholas, J. & Christensen, H. Quantifying app store dynamics: longitudinal tracking of mental health apps. JMIR mHealth uHealth, 4, e96 (2016).","DOI":"10.2196\/mhealth.6020"},{"key":"312_CR5","doi-asserted-by":"publisher","first-page":"1145","DOI":"10.1093\/cid\/cit455","volume":"57","author":"A Moodley","year":"2013","unstructured":"Moodley, A., Mangino, J. E. & Goff, D. A. Review of infectious diseases applications for iPhone\/iPad and Android: from pocket to patient. Clin. Infect. Dis. 57, 1145\u20131154 (2013).","journal-title":"Clin. Infect. Dis."},{"key":"312_CR6","doi-asserted-by":"publisher","DOI":"10.2196\/10141","volume":"20","author":"SM Schueller","year":"2018","unstructured":"Schueller, S. M., Neary, M., O\u2019Loughlin, K. & Adkins, E. C. Discovery of and interest in health apps among those with mental health needs: survey and focus group study. J. Med. Internet Res. 20, e10141 (2018).","journal-title":"J. Med. Internet Res."},{"key":"312_CR7","doi-asserted-by":"crossref","unstructured":"Singh, K. et al. Patient-facing mobile apps to treat high-need, high-cost populations: a scoping review. JMIR mHealth uHealth 4, e136 (2016).","DOI":"10.2196\/mhealth.6445"},{"key":"312_CR8","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1002\/wps.20472","volume":"16","author":"J Firth","year":"2017","unstructured":"Firth, J. et al. The efficacy of smartphone\u2010based mental health interventions for depressive symptoms: a meta\u2010analysis of randomized controlled trials. World Psychiatry 16, 287\u2013298 (2017).","journal-title":"World Psychiatry"},{"key":"312_CR9","unstructured":"Food and Drug Administration. Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff. https:\/\/www.fda.gov\/media\/80958\/download (2015)."},{"key":"312_CR10","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1017\/S026646231800051X","volume":"34","author":"MR Moshi","year":"2018","unstructured":"Moshi, M. R., Tooher, R. & Merlin, T. Suitability of current evaluation frameworks for use in the health technology assessment of mobile medical applications: a systematic review. Int. J. Technol. Assess. Health Care 34, 464\u2013475 (2018).","journal-title":"Int. J. Technol. Assess. Health Care"},{"key":"312_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-019-0129-6","volume":"2","author":"AD Carlo","year":"2019","unstructured":"Carlo, A. D., Ghomi, R. H., Renn, B. N. & Are\u00e1n, P. A. By the numbers: ratings and utilization of behavioral health mobile applications. NPJ Digital Med. 2, 1\u20138 (2019).","journal-title":"NPJ Digital Med."},{"key":"312_CR12","unstructured":"US Food and Drug Administration. Digital Health Innovation Action Plan: FDA Center for Devices and Radiological Health. https:\/\/www.fda.gov\/media\/106331\/download (2017)."},{"key":"312_CR13","unstructured":"American Psychiatric Association. App Evaluation Model. https:\/\/www.psychiatry.org\/psychiatrists\/practice\/mental-health-apps\/app-evaluation-model (2018)."},{"key":"312_CR14","unstructured":"NYC Well. App Library. https:\/\/nycwell.cityofnewyork.us\/en\/app-library\/ (2020)."},{"key":"312_CR15","doi-asserted-by":"publisher","first-page":"e52","DOI":"10.1016\/S2589-7500(19)30013-5","volume":"1","author":"P Henson","year":"2019","unstructured":"Henson, P., David, G., Albright, K. & Torous, J. Deriving a practical framework for the evaluation of health apps. Lancet Digital Health 1, e52\u2013e54 (2019).","journal-title":"Lancet Digital Health"},{"key":"312_CR16","doi-asserted-by":"publisher","first-page":"e27","DOI":"10.2196\/mhealth.3422","volume":"3","author":"SR Stoyanov","year":"2015","unstructured":"Stoyanov, S. R. et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth 3, e27 (2015).","journal-title":"JMIR mHealth uHealth"},{"key":"312_CR17","unstructured":"Health Care Information and Management Systems Society. mHIMSS App Usability Work Group (2012)."},{"key":"312_CR18","doi-asserted-by":"publisher","DOI":"10.2196\/13057","volume":"8","author":"P Llorens-Vernet","year":"2020","unstructured":"Llorens-Vernet, P. & Mir\u00f3, J. Standards for mobile health\u2013related apps: systematic review and development of a guide. JMIR mHealth uHealth 8, e13057 (2020).","journal-title":"JMIR mHealth uHealth"},{"key":"312_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12916-019-1447-x","volume":"17","author":"E Rodriguez-Villa","year":"2019","unstructured":"Rodriguez-Villa, E. & Torous, J. Regulating digital health technologies with transparency: the case for dynamic and multi-stakeholder evaluation. BMC Med. 17, 1\u20135 (2019).","journal-title":"BMC Med."},{"key":"312_CR20","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1136\/ebmental-2018-300069","volume":"22","author":"H Wisniewski","year":"2019","unstructured":"Wisniewski, H. et al. Understanding the quality, effectiveness and attributes of top-rated smartphone health apps. Evid.-Based Ment. Health 22, 4\u20139 (2019).","journal-title":"Evid.-Based Ment. Health"},{"key":"312_CR21","unstructured":"Mobile Fact Sheet. Pew Research Center. htttps:\/\/www.pewinternet.org\/fact-sheet\/mobile\/ (2018)."},{"key":"312_CR22","unstructured":"Pew Research Center. 10% of Americans Don\u2019t Use The Internet. Who are they? http:\/\/pewresearch.org\/fact-tank\/2019\/04\/22\/some-americans-don\u2019t-use-the-internet-who-are-they\/ (2019)."},{"key":"312_CR23","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1016\/j.jpsychires.2018.11.025","volume":"109","author":"L Hendrikoff","year":"2019","unstructured":"Hendrikoff, L. et al. Prospective acceptance of distinct mobile mental health features in psychiatric patients and mental health professionals. J. Psychiatr. Res. 109, 126\u2013132 (2019).","journal-title":"J. Psychiatr. Res."},{"key":"312_CR24","unstructured":"Auxier, B. & Turner, E. Americans and privacy: concerned, confused and feeling lack of control over their personal information. Pew Research Center: Internet, Science and Tech. https:\/\/www.pewresearch.org\/internet\/2019\/11\/15\/americans-and-privacy-concerned-confused-and-feeling-lack-of-control-over-their-personal-information\/ (2019)."},{"key":"312_CR25","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.4581","volume":"17","author":"J Nicholas","year":"2015","unstructured":"Nicholas, J., Larsen, M. E., Proudfoot, J. & Christensen, H. Mobile apps for bipolar disorder: a systematic review of features and content quality. J. Med. Internet Res. 17, e198 (2015).","journal-title":"J. Med. Internet Res."},{"key":"312_CR26","doi-asserted-by":"publisher","first-page":"e192542","DOI":"10.1001\/jamanetworkopen.2019.2542","volume":"2","author":"K Huckvale","year":"2019","unstructured":"Huckvale, K., Torous, J. & Larsen, M. E. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw. Open 2, e192542\u2013e192542 (2019).","journal-title":"JAMA Netw. Open"},{"key":"312_CR27","unstructured":"Thompson, S. One Nation, Tracked: An Investigation into the Smartphone Tracking Industry. The New York Times. https:\/\/www.nytimes.com\/interactive\/2019\/12\/19\/opinion\/location-tracking-cell-phone.html (2019)."},{"key":"312_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-019-0093-1","volume":"2","author":"ME Larsen","year":"2019","unstructured":"Larsen, M. E. et al. Using science to sell apps: evaluation of mental health app store quality claims. NPJ Digital Med. 2, 1\u20136 (2019).","journal-title":"NPJ Digital Med."},{"key":"312_CR29","doi-asserted-by":"crossref","unstructured":"Weisel, K et al. Standalone smartphone apps for mental health\u2014a systematic review and meta-analysis. npj Digital Med. 2, 1\u201310 (2019).","DOI":"10.1038\/s41746-019-0188-8"},{"key":"312_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12916-019-1461-z","volume":"17","author":"L Martinengo","year":"2019","unstructured":"Martinengo, L. et al. Suicide prevention and depression apps\u2019 suicide risk assessment and management: a systematic assessment of adherence to clinical guidelines. BMC Med. 17, 1\u201312 (2019).","journal-title":"BMC Med."},{"key":"312_CR31","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/S2215-0366(15)00565-9","volume":"3","author":"J Torous","year":"2016","unstructured":"Torous, J. & Firth, J. The digital placebo effect: mobile mental health meets clinical psychiatry. Lancet Psychiatry 3, 100\u2013102 (2016).","journal-title":"Lancet Psychiatry"},{"key":"312_CR32","doi-asserted-by":"publisher","DOI":"10.2196\/14567","volume":"21","author":"A Baumel","year":"2019","unstructured":"Baumel, A., Muench, F., Edan, S. & Kane, J. M. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J. Med. Internet Res. 21, e14567 (2019).","journal-title":"J. Med. Internet Res."},{"key":"312_CR33","doi-asserted-by":"crossref","unstructured":"Schueller, S. M. Person-activity fit in positive psychological intervention. In The Wiley Blackwell Handbook of Positive Psychological Interventions (eds. Parks, A. C. & Schueller, S. M.) (Wiley Blackwell, 2014).","DOI":"10.1002\/9781118315927.ch22"},{"key":"312_CR34","doi-asserted-by":"publisher","first-page":"141","DOI":"10.3389\/fpsyt.2018.00141","volume":"9","author":"Y Khazaal","year":"2018","unstructured":"Khazaal, Y., Favrod, J., Sort, A., Borgeat, F. & Bouchard, S. Computers and games for mental health and well-being. Front. Psychiatry 9, 141 (2018).","journal-title":"Front. Psychiatry"},{"key":"312_CR35","first-page":"597","volume":"42","author":"S Chan","year":"2019","unstructured":"Chan, S., Li, L., Torous, J., Gratzer, D. & Yellowlees, P. M. Review and implementation of self-help and automated tools in mental health care. Psychiatr. Clin. 42, 597\u2013609 (2019).","journal-title":"Psychiatr. Clin."},{"key":"312_CR36","doi-asserted-by":"publisher","DOI":"10.2196\/10035","volume":"20","author":"A Pung","year":"2018","unstructured":"Pung, A., Fletcher, S. L. & Gunn, J. M. Mobile app use by primary care patients to manage their depressive symptoms: qualitative study. J. Med. Internet Res. 20, e10035 (2018).","journal-title":"J. Med. Internet Res."},{"key":"312_CR37","doi-asserted-by":"publisher","DOI":"10.2196\/12656","volume":"6","author":"T Fleming","year":"2019","unstructured":"Fleming, T. et al. The importance of user segmentation for designing digital therapy for adolescent mental health: findings from scoping processes. JMIR Mental Health 6, e12656 (2019).","journal-title":"JMIR Mental Health"},{"key":"312_CR38","doi-asserted-by":"publisher","DOI":"10.2196\/10652","volume":"5","author":"E Carpenter-Song","year":"2018","unstructured":"Carpenter-Song, E., Noel, V. A., Acquilano, S. C. & Drake, R. E. Real-world technology use among people with mental illnesses: qualitative study. JMIR Mental Health 5, e10652 (2018).","journal-title":"JMIR Mental Health"},{"key":"312_CR39","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1002\/wps.20673","volume":"18","author":"J Linardon","year":"2019","unstructured":"Linardon, J., Cuijpers, P., Carlbring, P., Messer, M. & Fuller\u2010Tyszkiewicz, M. The efficacy of app\u2010supported smartphone interventions for mental health problems: a meta\u2010analysis of randomized controlled trials. World Psychiatry 18, 325\u2013336 (2019).","journal-title":"World Psychiatry"},{"key":"312_CR40","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh, M. L. Interrater reliability: the kappa statistic. Biochemia Med. 22, 276\u2013282 (2012).","journal-title":"Biochemia Med."},{"key":"312_CR41","doi-asserted-by":"publisher","DOI":"10.2196\/15321","volume":"7","author":"C Qu","year":"2020","unstructured":"Qu, C., Sas, C., Roquet, C. D. & Doherty, G. Functionality of top-rated mobile apps for depression: systematic search and evaluation. JMIR Mental Health 7, e15321 (2020).","journal-title":"JMIR Mental Health"},{"key":"312_CR42","unstructured":"National Institute for Health Care and Excellence. Behaviour Change: Digital and Mobile Health Interventions. Draft for consultation. https:\/\/www.nice.org.uk\/guidance\/GID-NG10101\/documents\/draft-guideline (2020)."},{"key":"312_CR43","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s13187-017-1293-5","volume":"34","author":"H Kim","year":"2019","unstructured":"Kim, H. et al. Mobile health application and e-health literacy: opportunities and concerns for cancer patients and caregivers. J. Cancer Educ. 34, 3\u20138 (2019).","journal-title":"J. Cancer Educ."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00312-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00312-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00312-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:25:56Z","timestamp":1670379956000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-00312-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,30]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["312"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-00312-4","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,30]]},"assertion":[{"value":"20 February 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 July 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 July 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"J.T. declares unrelated research support from Otuska. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"100"}}