{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T05:08:33Z","timestamp":1775797713544,"version":"3.50.1"},"reference-count":73,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T00:00:00Z","timestamp":1702598400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T00:00:00Z","timestamp":1702598400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Among the thousands of eHealth tools available, the vast majority do not get past pilot phases because they cannot prove value, and only a few have been systematically assessed. Although multiple eHealth assessment frameworks have been developed, these efforts face multiple challenges. This study aimed to address some of these challenges by validating and refining an initial list of 55 assessment criteria based on previous frameworks through a two-round modified Delphi process with in-between rounds of interviews. The expert panel (\n                    <jats:italic>n<\/jats:italic>\n                    \u2009=\u200957) included participants from 18 countries and 9 concerned parties. A consensus was reached on 46 criteria that were classified into foundational and contextual criteria. The 36 foundational criteria focus on evaluating the eHealth tool itself and were grouped into nine clusters: technical aspects, clinical utility and safety, usability and human centricity, functionality, content, data management, endorsement, maintenance, and developer. The 10 contextual criteria focus on evaluating the factors that vary depending on the context the tool is being evaluated for and were grouped into seven clusters: data-protection compliance, safety regulatory compliance, interoperability and data integration, cultural requirements, affordability, cost-benefit, and implementability. The classification of criteria into foundational and contextual helps us assess not only the quality of an isolated tool, but also its potential fit in a specific setting. Criteria subscales may be particularly relevant when determining the strengths and weaknesses of the tool being evaluated. This granularity enables different concerned parties to make informed decisions about which tools to consider according to their specific needs and priorities.\n                  <\/jats:p>","DOI":"10.1038\/s41746-023-00982-w","type":"journal-article","created":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T06:02:16Z","timestamp":1702620136000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["A sociotechnical framework to assess patient-facing eHealth tools: results of a modified Delphi process"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8817-0148","authenticated-orcid":false,"given":"Christine","family":"Jacob","sequence":"first","affiliation":[]},{"given":"Johan","family":"Lindeque","sequence":"additional","affiliation":[]},{"given":"Roman","family":"M\u00fcller","sequence":"additional","affiliation":[]},{"given":"Alexander","family":"Klein","sequence":"additional","affiliation":[]},{"given":"Thomas","family":"Metcalfe","sequence":"additional","affiliation":[]},{"given":"Samantha L.","family":"Connolly","sequence":"additional","affiliation":[]},{"given":"Florian","family":"Koerber","sequence":"additional","affiliation":[]},{"given":"Roma","family":"Maguire","sequence":"additional","affiliation":[]},{"given":"Fabrice","family":"Denis","sequence":"additional","affiliation":[]},{"given":"Sabina C.","family":"Heuss","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2897-0389","authenticated-orcid":false,"given":"Marc K.","family":"Peter","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,12,15]]},"reference":[{"key":"982_CR1","unstructured":"World Health Organization. eHealth. http:\/\/www.emro.who.int\/health-topics\/ehealth\/. 2022."},{"key":"982_CR2","doi-asserted-by":"publisher","first-page":"e18655","DOI":"10.2196\/18655","volume":"22","author":"O Schmalz","year":"2020","unstructured":"Schmalz, O. et al. Digital Monitoring and Management of Patients With Advanced or Metastatic Non-Small Cell Lung Cancer Treated With Cancer Immunotherapy and Its Impact on Quality of Clinical Care: Interview and Survey Study Among Health Care Professionals and Patients. J. Med. Internet Res. 22, e18655 (2020).","journal-title":"J. Med. Internet Res."},{"key":"982_CR3","doi-asserted-by":"publisher","first-page":"e20203","DOI":"10.2196\/20203","volume":"8","author":"C Jacob","year":"2020","unstructured":"Jacob, C., Sanchez-Vazquez, A. & Ivory, C. Factors Impacting Clinicians\u2019 Adoption of a Clinical Photo Documentation App and its Implications for Clinical Workflows and Quality of Care: Qualitative Case Study. JMIR Mhealth Uhealth 8, e20203 (2020).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR4","doi-asserted-by":"publisher","first-page":"e13555","DOI":"10.2196\/13555","volume":"7","author":"C Jacob","year":"2019","unstructured":"Jacob, C., Sanchez-Vazquez, A. & Ivory, C. Clinicians\u2019 Role in the Adoption of an Oncology Decision Support App in Europe and Its Implications for Organizational Practices: Qualitative Case Study. JMIR Mhealth Uhealth 7, e13555 (2019).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR5","doi-asserted-by":"publisher","first-page":"e30549","DOI":"10.2196\/30549","volume":"23","author":"F Denis","year":"2021","unstructured":"Denis, F. & Krakowski, I. How Should Oncologists Choose an Electronic Patient-Reported Outcome System for Remote Monitoring of Patients With Cancer? J. Med. Internet Res. 23, e30549 (2021).","journal-title":"J. Med. Internet Res."},{"key":"982_CR6","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1200\/JCO.2015.63.0830","volume":"34","author":"E Basch","year":"2016","unstructured":"Basch, E. et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J. Clin. Oncol. 34, 557\u201365 (2016).","journal-title":"J. Clin. Oncol."},{"key":"982_CR7","doi-asserted-by":"publisher","first-page":"e42401","DOI":"10.2196\/42401","volume":"25","author":"R van Kessel","year":"2023","unstructured":"van Kessel, R., Kyriopoulos, I., Wong, B. L. H. & Mossialos, E. The Effect of the COVID-19 Pandemic on Digital Health\u2013Seeking Behavior: Big Data Interrupted Time-Series Analysis of Google Trends. J. Med. Internet Res. 25, e42401 (2023).","journal-title":"J. Med. Internet Res."},{"key":"982_CR8","doi-asserted-by":"publisher","first-page":"e46694","DOI":"10.2196\/46694","volume":"25","author":"C Perrin Franck","year":"2023","unstructured":"Perrin Franck, C. et al. iCHECK-DH: Guidelines and Checklist for the Reporting on Digital Health Implementations. J. Med. Internet Res. 25, e46694 (2023).","journal-title":"J. Med. Internet Res."},{"key":"982_CR9","doi-asserted-by":"publisher","first-page":"S41","DOI":"10.9745\/GHSP-D-18-00167","volume":"6","author":"L-A Long","year":"2018","unstructured":"Long, L.-A., Pariyo, G. & Kallander, K. Digital Technologies for Health Workforce Development in Low- and Middle-Income Countries: A Scoping Review. Glob. Health Sci. Pr. 6, S41\u2013S48 (2018).","journal-title":"Glob. Health Sci. Pr."},{"key":"982_CR10","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/j.cbpra.2018.01.002","volume":"25","author":"M Neary","year":"2018","unstructured":"Neary, M. & Schueller, S. M. State of the Field of Mental Health Apps. Cogn. Behav. Pr. 25, 531\u2013537 (2018).","journal-title":"Cogn. Behav. Pr."},{"key":"982_CR11","doi-asserted-by":"publisher","first-page":"205520762110536","DOI":"10.1177\/20552076211053690","volume":"7","author":"M Neary","year":"2021","unstructured":"Neary, M. et al. A process for reviewing mental health apps: Using the One Mind PsyberGuide Credibility Rating System. Digit Health 7, 205520762110536 (2021).","journal-title":"Digit Health"},{"key":"982_CR12","doi-asserted-by":"publisher","first-page":"745","DOI":"10.48048\/wjst.2020.6482","volume":"17","author":"W Hongsanun","year":"2020","unstructured":"Hongsanun, W. & Insuk, S. Quality Assessment Criteria for Mobile Health Apps: A Systematic Review. Walailak J. Sci. Technol. 17, 745\u2013759 (2020).","journal-title":"Walailak J. Sci. Technol."},{"key":"982_CR13","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1038\/s41746-021-00476-7","volume":"4","author":"R Sedhom","year":"2021","unstructured":"Sedhom, R., McShea, M. J., Cohen, A. B., Webster, J. A. & Mathews, S. C. Mobile app validation: a digital health scorecard approach. NPJ Digit Med. 4, 111 (2021).","journal-title":"NPJ Digit Med."},{"key":"982_CR14","doi-asserted-by":"publisher","first-page":"e42679","DOI":"10.2196\/42679","volume":"11","author":"RM Bernard","year":"2023","unstructured":"Bernard, R. M. et al. Mobile Health Self-management Support for Spinal Cord Injury: Systematic Literature Review. JMIR Mhealth Uhealth 11, e42679 (2023).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR15","doi-asserted-by":"publisher","first-page":"1125","DOI":"10.1016\/j.sapharm.2018.01.006","volume":"14","author":"EE Ali","year":"2018","unstructured":"Ali, E. E., Teo, A. K. S., Goh, S. X. L., Chew, L. & Yap, K. Y.-L. MedAd-AppQ: A quality assessment tool for medication adherence apps on iOS and android platforms. Res. Soc. Adm. Pharm. 14, 1125\u20131133 (2018).","journal-title":"Res. Soc. Adm. Pharm."},{"key":"982_CR16","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1038\/s41746-019-0111-3","volume":"2","author":"SC Mathews","year":"2019","unstructured":"Mathews, S. C. et al. Digital health: a path to validation. NPJ Digit Med 2, 38 (2019).","journal-title":"NPJ Digit Med"},{"key":"982_CR17","doi-asserted-by":"publisher","first-page":"e10414","DOI":"10.2196\/10414","volume":"6","author":"RS Larson","year":"2018","unstructured":"Larson, R. S. A Path to Better-Quality mHealth Apps. JMIR Mhealth Uhealth 6, e10414 (2018).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR18","doi-asserted-by":"publisher","first-page":"e45143","DOI":"10.2196\/45143","volume":"10","author":"C Jacob","year":"2023","unstructured":"Jacob, C. et al. Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis. JMIR Hum. Factors 10, e45143 (2023).","journal-title":"JMIR Hum. Factors"},{"key":"982_CR19","unstructured":"Mathews, S. The Digital Health Scorecard (DHS). John Hopkins Medicine. https:\/\/www.hopkinsmedicine.org\/news\/articles\/2019\/11\/the-digital-health-scorecard (2019)."},{"key":"982_CR20","doi-asserted-by":"publisher","first-page":"e15433","DOI":"10.2196\/15433","volume":"9","author":"A Muro-Culebras","year":"2021","unstructured":"Muro-Culebras, A. et al. Tools for Evaluating the Content, Efficacy, and Usability of Mobile Health Apps According to the Consensus-Based Standards for the Selection of Health Measurement Instruments: Systematic Review. JMIR Mhealth Uhealth 9, e15433 (2021).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR21","doi-asserted-by":"publisher","first-page":"205520762110186","DOI":"10.1177\/20552076211018617","volume":"7","author":"H Unsworth","year":"2021","unstructured":"Unsworth, H. et al. The NICE Evidence Standards Framework for digital health and care technologies \u2013 Developing and maintaining an innovative evidence framework with global impact. Digit Health 7, 205520762110186 (2021).","journal-title":"Digit Health"},{"key":"982_CR22","doi-asserted-by":"publisher","first-page":"e23479","DOI":"10.2196\/23479","volume":"23","author":"KP Fadahunsi","year":"2021","unstructured":"Fadahunsi, K. P. et al. Information Quality Frameworks for Digital Health Technologies: Systematic Review. J. Med. Internet Res. 23, e23479 (2021).","journal-title":"J. Med. Internet Res."},{"key":"982_CR23","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1017\/S026646232000015X","volume":"36","author":"C Vis","year":"2020","unstructured":"Vis, C., B\u00fchrmann, L., Riper, H. & Ossebaard, H. C. Health technology assessment frameworks for eHealth: A systematic review. Int. J. Technol. Assess. Health Care 36, 204\u2013216 (2020).","journal-title":"Int. J. Technol. Assess. Health Care"},{"key":"982_CR24","doi-asserted-by":"publisher","first-page":"e82","DOI":"10.2196\/jmir.7270","volume":"19","author":"A Baumel","year":"2017","unstructured":"Baumel, A., Faber, K., Mathur, N., Kane, J. M. & Muench, F. Enlight: A Comprehensive Quality and Therapeutic Potential Evaluation Tool for Mobile and Web-Based eHealth Interventions. J. Med. Internet Res. 19, e82 (2017).","journal-title":"J. Med. Internet Res."},{"key":"982_CR25","doi-asserted-by":"publisher","first-page":"e19898","DOI":"10.2196\/19898","volume":"22","author":"R de la Vega","year":"2020","unstructured":"de la Vega, R., Ritterband, L. & Palermo, T. M. Assessing Digital Health Implementation for a Pediatric Chronic Pain Intervention: Comparing the RE-AIM and BIT Frameworks Against Real-World Trial Data and Recommendations for Future Studies. J. Med. Internet Res. 22, e19898 (2020).","journal-title":"J. Med. Internet Res."},{"key":"982_CR26","doi-asserted-by":"publisher","first-page":"620","DOI":"10.1016\/j.jneb.2018.02.002","volume":"50","author":"KN DiFilippo","year":"2018","unstructured":"DiFilippo, K. N., Huang, W.-H. D. & Chapman-Novakofski, K. M. Mobile Apps for the Dietary Approaches to Stop Hypertension (DASH): App Quality Evaluation. J. Nutr. Educ. Behav. 50, 620\u2013625 (2018).","journal-title":"J. Nutr. Educ. Behav."},{"key":"982_CR27","doi-asserted-by":"publisher","first-page":"53","DOI":"10.4018\/IJEBR.2017070104","volume":"13","author":"M Naghizadeh","year":"2017","unstructured":"Naghizadeh, M., Movahhed, M. K. & Jedari, Y. G. Quality Assessment Framework for Mobile Health Systems. Int. J. E-Bus. Res. 13, 53\u201366 (2017).","journal-title":"Int. J. E-Bus. Res."},{"key":"982_CR28","doi-asserted-by":"publisher","first-page":"e60","DOI":"10.2196\/mhealth.7291","volume":"5","author":"M Bradway","year":"2017","unstructured":"Bradway, M. et al. mHealth Assessment: Conceptualization of a Global Framework. JMIR Mhealth Uhealth 5, e60 (2017).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR29","doi-asserted-by":"publisher","first-page":"e20482","DOI":"10.2196\/20482","volume":"8","author":"N Alon","year":"2020","unstructured":"Alon, N., Stern, A. D. & Torous, J. Assessing the Food and Drug Administration\u2019s Risk-Based Framework for Software Precertification With Top Health Apps in the United States: Quality Improvement Study. JMIR Mhealth Uhealth 8, e20482 (2020).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR30","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-015-0303-x","volume":"13","author":"K Huckvale","year":"2015","unstructured":"Huckvale, K., Morrison, C., Ouyang, J., Ghaghda, A. & Car, J. The evolution of mobile apps for asthma: an updated systematic assessment of content and tools. BMC Med 13, 58 (2015).","journal-title":"BMC Med"},{"key":"982_CR31","doi-asserted-by":"publisher","first-page":"e185","DOI":"10.2196\/mhealth.9217","volume":"6","author":"L Hutton","year":"2018","unstructured":"Hutton, L. et al. Assessing the Privacy of mHealth Apps for Self-Tracking: Heuristic Evaluation Approach. JMIR Mhealth Uhealth 6, e185 (2018).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR32","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1007\/s12553-018-0216-8","volume":"8","author":"KM Scott","year":"2018","unstructured":"Scott, K. M., Richards, D. & Londos, G. Assessment criteria for parents to determine the trustworthiness of maternal and child health apps: a pilot study. Health Technol. 8, 63\u201370 (2018).","journal-title":"Health Technol."},{"key":"982_CR33","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1016\/j.jacbts.2016.06.003","volume":"1","author":"GA Van Norman","year":"2016","unstructured":"Van Norman, G. A. Drugs and Devices: Comparison of European and U.S. Approval Processes. JACC Basic Transl. Sci. 1, 399\u2013412 (2016).","journal-title":"JACC Basic Transl. Sci."},{"key":"982_CR34","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/s41746-023-00862-3","volume":"6","author":"M Uncovska","year":"2023","unstructured":"Uncovska, M., Freitag, B., Meister, S. & Fehring, L. Rating analysis and BERTopic modeling of consumer versus regulated mHealth app reviews in Germany. NPJ Digit Med. 6, 115 (2023).","journal-title":"NPJ Digit Med."},{"key":"982_CR35","doi-asserted-by":"publisher","first-page":"100663","DOI":"10.1016\/j.hlpt.2022.100663","volume":"11","author":"JD Iqbal","year":"2022","unstructured":"Iqbal, J. D. & Biller-Andorno, N. The regulatory gap in digital health and alternative pathways to bridge it. Health Policy Technol. 11, 100663 (2022).","journal-title":"Health Policy Technol."},{"key":"982_CR36","doi-asserted-by":"publisher","unstructured":"Leonardi, P. Methodological Guidelines for the Study of Materiality and Affordances. In Routledge Companion to Qualitative Research in Organization Studies 1 Edition. (eds. Mir R., Jain S.). (New York, Routledge, Routledge, 2017). https:\/\/doi.org\/10.4324\/9781315686103 (2017).","DOI":"10.4324\/9781315686103"},{"key":"982_CR37","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-023-02241-1","volume":"12","author":"M Sasseville","year":"2023","unstructured":"Sasseville, M. et al. The impact of technology systems and level of support in digital mental health interventions: a secondary meta-analysis. Syst. Rev. 12, 78 (2023).","journal-title":"Syst. Rev."},{"key":"982_CR38","unstructured":"FHNW - University of Applied Sciences Northwestern Switzerland. Assessing quality and impact of eHealth tools \u2013 a Practical Toolbox. https:\/\/ehealth-criteria-toolbox.net\/ (2022)."},{"key":"982_CR39","doi-asserted-by":"publisher","first-page":"e27","DOI":"10.2196\/mhealth.3422","volume":"3","author":"SR Stoyanov","year":"2015","unstructured":"Stoyanov, S. R. et al. Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps. JMIR Mhealth Uhealth 3, e27 (2015).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR40","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1038\/s41746-023-00836-5","volume":"6","author":"J Silberman","year":"2023","unstructured":"Silberman, J. et al. Rigorous and rapid evidence assessment in digital health with the evidence DEFINED framework. NPJ Digit Med 6, 101 (2023).","journal-title":"NPJ Digit Med"},{"key":"982_CR41","doi-asserted-by":"publisher","first-page":"e18072","DOI":"10.2196\/18072","volume":"8","author":"C Jacob","year":"2020","unstructured":"Jacob, C., Sanchez-Vazquez, A. & Ivory, C. Understanding Clinicians\u2019 Adoption of Mobile Health Tools: A Qualitative Review of the Most Used Frameworks. JMIR Mhealth Uhealth 8, e18072 (2020).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR42","doi-asserted-by":"publisher","first-page":"e15935","DOI":"10.2196\/15935","volume":"8","author":"C Jacob","year":"2020","unstructured":"Jacob, C., Sanchez-Vazquez, A. & Ivory, C. Social, Organizational, and Technological Factors Impacting Clinicians\u2019 Adoption of Mobile Health Tools: Systematic Literature Review. JMIR Mhealth Uhealth 8, e15935 (2020).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR43","doi-asserted-by":"publisher","first-page":"e36284","DOI":"10.2196\/36284","volume":"10","author":"C Jacob","year":"2022","unstructured":"Jacob, C., Sezgin, E., Sanchez-Vazquez, A. & Ivory, C. Sociotechnical Factors Affecting Patients\u2019 Adoption of Mobile Health Tools: Systematic Literature Review and Narrative Synthesis. JMIR Mhealth Uhealth 10, e36284 (2022).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR44","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1037\/amp0000895","volume":"77","author":"SL Connolly","year":"2022","unstructured":"Connolly, S. L. et al. Patient and provider predictors of telemental health use prior to and during the COVID-19 pandemic within the Department of Veterans Affairs. Am. Psychologist 77, 249\u2013261 (2022).","journal-title":"Am. Psychologist"},{"key":"982_CR45","doi-asserted-by":"publisher","first-page":"e33012","DOI":"10.2196\/33012","volume":"9","author":"F Dahlhausen","year":"2021","unstructured":"Dahlhausen, F. et al. Physicians\u2019 Attitudes Toward Prescribable mHealth Apps and Implications for Adoption in Germany: Mixed Methods Study. JMIR Mhealth Uhealth 9, e33012 (2021).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR46","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-023-09227-8","volume":"23","author":"C Varsi","year":"2023","unstructured":"Varsi, C., Andersen, L. F., Koksvik, G. T., Severinsen, F. & Paulsen, M. M. Intervention-related, contextual and personal factors affecting the implementation of an evidence-based digital system for prevention and treatment of malnutrition in elderly institutionalized patients: a qualitative study. BMC Health Serv. Res. 23, 245 (2023).","journal-title":"BMC Health Serv. Res."},{"key":"982_CR47","doi-asserted-by":"publisher","first-page":"e47898","DOI":"10.2196\/47898","volume":"10","author":"J Torous","year":"2023","unstructured":"Torous, J., Benson, N. M., Myrick, K. & Eysenbach, G. Focusing on Digital Research Priorities for Advancing the Access and Quality of Mental Health. JMIR Ment. Health 10, e47898 (2023).","journal-title":"JMIR Ment. Health"},{"key":"982_CR48","doi-asserted-by":"publisher","first-page":"100379","DOI":"10.1016\/j.invent.2021.100379","volume":"24","author":"AE Roberts","year":"2021","unstructured":"Roberts, A. E. et al. Evaluating the quality and safety of health-related apps and e-tools: Adapting the Mobile App Rating Scale and developing a quality assurance protocol. Internet Inter. 24, 100379 (2021).","journal-title":"Internet Inter."},{"key":"982_CR49","doi-asserted-by":"publisher","first-page":"518","DOI":"10.1002\/da.22654","volume":"34","author":"M Sucala","year":"2017","unstructured":"Sucala, M. et al. Anxiety: There is an app for that. A systematic review of anxiety apps. Depress Anxiety 34, 518\u2013525 (2017).","journal-title":"Depress Anxiety"},{"key":"982_CR50","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1038\/s41746-019-0129-6","volume":"2","author":"AD Carlo","year":"2019","unstructured":"Carlo, A. D., Hosseini Ghomi, R., Renn, B. N. & Are\u00e1n, P. A. By the numbers: ratings and utilization of behavioral health mobile applications. NPJ Digit Med 2, 54 (2019).","journal-title":"NPJ Digit Med"},{"key":"982_CR51","doi-asserted-by":"publisher","first-page":"2042","DOI":"10.1177\/1460458219896492","volume":"26","author":"F Alqahtani","year":"2020","unstructured":"Alqahtani, F. & Orji, R. Insights from user reviews to improve mental health apps. Health Inform. J. 26, 2042\u20132066 (2020).","journal-title":"Health Inform. J."},{"key":"982_CR52","doi-asserted-by":"publisher","first-page":"e047001","DOI":"10.1136\/bmjopen-2020-047001","volume":"11","author":"S Lagan","year":"2021","unstructured":"Lagan, S., Sandler, L. & Torous, J. Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open 11, e047001 (2021).","journal-title":"BMJ Open"},{"key":"982_CR53","doi-asserted-by":"publisher","unstructured":"Cajander, \u00c5. & Gr\u00fcnloh, C. Electronic Health Records Are More Than a Work Tool. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems New York, NY, USA, ACM, p. 1\u201313. https:\/\/doi.org\/10.1145\/3290605.3300865 (2019).","DOI":"10.1145\/3290605.3300865"},{"key":"982_CR54","doi-asserted-by":"publisher","first-page":"e41481","DOI":"10.2196\/41481","volume":"9","author":"C Jacob","year":"2022","unstructured":"Jacob, C., Bourke, S. & Heuss, S. From Testers to Cocreators\u2014the Value of and Approaches to Successful Patient Engagement in the Development of eHealth Solutions: Qualitative Expert Interview Study. JMIR Hum. Factors 9, e41481 (2022).","journal-title":"JMIR Hum. Factors"},{"key":"982_CR55","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1177\/0091270004273680","volume":"45","author":"CM Smith","year":"2005","unstructured":"Smith, C. M. Origin and Uses of \u201cPrimum Non Nocere\u201d -Above All, Do No Harm! J. Clin. Pharmacol. 45, 371\u2013377 (2005).","journal-title":"J. Clin. Pharmacol."},{"key":"982_CR56","unstructured":"STAT News. \u2018Hard to catch up\u2019: FDA commissioner on regulating new digital health tools. https:\/\/www.statnews.com\/2023\/05\/09\/fda-ai-digital-health-care\/. (2023)."},{"key":"982_CR57","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1002\/jtr.709","volume":"11","author":"HM Donohoe","year":"2009","unstructured":"Donohoe, H. M. & Needham, R. D. Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. Int. J. Tour. Res. 11, 415\u2013437 (2009).","journal-title":"Int. J. Tour. Res."},{"key":"982_CR58","doi-asserted-by":"publisher","first-page":"32","DOI":"10.7748\/ns1999.07.13.45.32.c2650","volume":"13","author":"N Bowles","year":"1999","unstructured":"Bowles, N. The Delphi technique. Nurs. Stand. 13, 32\u201336 (1999).","journal-title":"Nurs. Stand."},{"key":"982_CR59","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1136\/ebnurs-2020-103303","volume":"23","author":"D Barrett","year":"2020","unstructured":"Barrett, D. & Heale, R. What are Delphi studies? Evid. Based Nurs. 23, 68\u201369 (2020).","journal-title":"Evid. Based Nurs."},{"key":"982_CR60","doi-asserted-by":"publisher","unstructured":"Grime, M. M., Wright, G. Delphi Method. Wiley StatsRef: Statistics Reference Online Wiley; p. 1\u20136. https:\/\/doi.org\/10.1002\/9781118445112.stat07879 (2016).","DOI":"10.1002\/9781118445112.stat07879"},{"key":"982_CR61","doi-asserted-by":"publisher","first-page":"116","DOI":"10.5662\/wjm.v11.i4.116","volume":"11","author":"P Nasa","year":"2021","unstructured":"Nasa, P., Jain, R. & Juneja, D. Delphi methodology in healthcare research: How to decide its appropriateness. World J. Methodol. 11, 116\u2013129 (2021).","journal-title":"World J. Methodol."},{"key":"982_CR62","doi-asserted-by":"publisher","first-page":"684","DOI":"10.1177\/0269216317690685","volume":"31","author":"S J\u00fcnger","year":"2017","unstructured":"J\u00fcnger, S., Payne, S. A., Brine, J., Radbruch, L. & Brearley, S. G. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat. Med 31, 684\u2013706 (2017).","journal-title":"Palliat. Med"},{"key":"982_CR63","doi-asserted-by":"publisher","unstructured":"Niederberger, M. & Spranger, J. Delphi Technique in Health Sciences: A Map. Front Public Health. 8, 22. https:\/\/doi.org\/10.3389\/fpubh.2020.00457. (2020).","DOI":"10.3389\/fpubh.2020.00457"},{"key":"982_CR64","doi-asserted-by":"publisher","first-page":"e1000097","DOI":"10.1371\/journal.pmed.1000097","volume":"6","author":"D Moher","year":"2009","unstructured":"Moher, D., Liberati, A., Tetzlaff, J. & Altman, D. G. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6, e1000097 (2009).","journal-title":"PLoS Med"},{"key":"982_CR65","doi-asserted-by":"crossref","unstructured":"Higgins, J. et al. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. (Wiley, 2019).","DOI":"10.1002\/9781119536604"},{"key":"982_CR66","doi-asserted-by":"publisher","unstructured":"Braun, V. & Clarke, V. Successful Qualitative Research: A Practical Guide For Beginners. SAGE Publications Ltd; https:\/\/doi.org\/10.1002\/jmr.2361 (2013).","DOI":"10.1002\/jmr.2361"},{"key":"982_CR67","doi-asserted-by":"publisher","first-page":"26152","DOI":"10.3402\/qhw.v9.26152","volume":"9","author":"V Braun","year":"2014","unstructured":"Braun, V. & Clarke, V. What can \u201cthematic analysis\u201d offer health and wellbeing researchers? Int J. Qual. Stud. Health Well-Being 9, 26152 (2014).","journal-title":"Int J. Qual. Stud. Health Well-Being"},{"key":"982_CR68","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","volume":"3","author":"V Braun","year":"2006","unstructured":"Braun, V. & Clarke, V. Using thematic analysis in psychology. Qual. Res Psychol. 3, 77\u2013101 (2006).","journal-title":"Qual. Res Psychol."},{"key":"982_CR69","doi-asserted-by":"publisher","first-page":"e10016","DOI":"10.2196\/10016","volume":"6","author":"J Zelmer","year":"2018","unstructured":"Zelmer, J. et al. An Assessment Framework for e-Mental Health Apps in Canada: Results of a Modified Delphi Process. JMIR Mhealth Uhealth 6, e10016 (2018).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR70","doi-asserted-by":"publisher","first-page":"e16899","DOI":"10.2196\/16899","volume":"8","author":"N Robles","year":"2020","unstructured":"Robles, N. et al. Evaluation Criteria for Weight Management Apps: Validation Using a Modified Delphi Process. JMIR Mhealth Uhealth 8, e16899 (2020).","journal-title":"JMIR Mhealth Uhealth"},{"key":"982_CR71","unstructured":"Creswell, J. & Poth, C. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. (SAGE Publications, Inc, 2017)."},{"key":"982_CR72","unstructured":"R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria URL https:\/\/wwwR-project.org\/ (2022)."},{"key":"982_CR73","doi-asserted-by":"crossref","unstructured":"Wickham H. ggplot2: Elegant Graphics for Data Analysis. (New York, Springer-Verlag, 2016).","DOI":"10.1007\/978-3-319-24277-4"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00982-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00982-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00982-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,15]],"date-time":"2023-12-15T07:03:16Z","timestamp":1702623796000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00982-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,15]]},"references-count":73,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["982"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00982-w","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-3252229\/v1","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,15]]},"assertion":[{"value":"10 August 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 November 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 December 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"T.M. and A.K. are employees and shareholders of F. Hoffmann-La Roche AG. T.M. is a shareholder of multiple eHealth related companies. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"232"}}