{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T21:51:12Z","timestamp":1791496272807,"version":"4.3.4"},"reference-count":30,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Hum Factors"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n In 2019, Germany introduced a unique regulatory framework for digital therapeutics (DTx), termed digital health applications (DiGAs) in Germany, with the goal of integrating evidence-based DTx into statutory health care. DTx are eligible for reimbursement by statutory health insurance if manufacturers demonstrate positive health care effects, such as improved health status or health literacy, in a controlled study setting. Although regulatory evaluation primarily relies on manufacturer-conducted studies, these studies do not fully capture how users experience and use DiGAs in everyday life.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The study aims to systematically characterize user experiences with reimbursable DiGAs by analyzing the sentiment and thematic content in publicly available app store reviews at scale to identify recurring patterns of positive and negative feedback across regulatory-relevant quality dimensions.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n All DiGAs with publicly available mobile app reviews were identified via the official German Federal Institute for Drugs and Medical Devices (BfArM) directory. User reviews were extracted from the German Apple App Store and Google Play Store using a tailored Python script. Reviews were then processed using GPT-4o, which extracted up to 5 core statements per review and assigned each a sentiment label (positive, neutral, or negative) and one of ten predefined thematic categories to each statement: content, technology, cost and reimbursement, login and registration, prescription and approval, user experience and design, support, tracking and documentation, effectiveness, and overall impression, all derived from the regulatory and quality criteria applicable to DiGA certification. Classification quality was assessed through manual validation, in which each of the 3 authors independently reviewed one-third of all model outputs.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n In total, 44 mobile DiGAs were included and analyzed. After data extraction and cleansing, the final dataset comprised 4328 reviews containing at least one interpretable statement, resulting in 9439 interpretable statements. A systematic validation of the automated classification demonstrated exceptionally high model performance, with 99% accuracy for sentiment classification (\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores of 1.00 for positive and 0.99 for negative categories) and 95% accuracy for category classification (average\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.95). While the categories overall impression (1233\/1431, 86.2% positive) and effectiveness (1440\/1609, 89.5% positive) received particularly positive feedback, users commented most negatively on the login and registration process (263\/280, 93.9% negative) and technology-related aspects (563\/668, 84.3% negative).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n User feedback on reimbursable DiGAs is predominantly positive, particularly regarding perceived effectiveness, overall impression, and content. However, recurring criticism of login and registration processes, technical reliability, and prescription and approval procedures reveals persistent barriers to access and use. As many of these aspects are prerequisites rather than peripheral convenience issues, they should be treated as essential implementation requirements. Analyzed at scale with large language model-based methods, publicly available app store reviews can complement formal evaluation by providing user-centered real-world evidence for postmarket monitoring and regulatory quality improvement.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/92415","type":"journal-article","created":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T21:23:44Z","timestamp":1791494624000},"page":"e92415-e92415","source":"Crossref","is-referenced-by-count":0,"title":["What Users Say About Reimbursable Digital Therapeutics in Germany: Large-Scale App Store Review Analysis Using a Large Language Model"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2554-4381","authenticated-orcid":false,"given":"Benjamin","family":"Kinast","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8021-0151","authenticated-orcid":false,"given":"Henrik","family":"Rohde","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1748-1563","authenticated-orcid":false,"given":"Bj\u00f6rn","family":"Schreiweis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8349-6798","authenticated-orcid":false,"given":"Hannes","family":"Ulrich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,8]]},"reference":[{"key":"R1","unstructured":"Driving the digital transformation of Germany\u2019s healthcare system for the good of patients. Federal Ministry of Health. URL: https:\/\/www.bundesgesundheitsministerium.de\/en\/digital-healthcare-act [Accessed 11-08-2025]"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01137-1","article-title":"The three-year evolution of Germany\u2019s Digital Therapeutics reimbursement program and its path forward","volume":"7","author":"Schmidt","journal-title":"NPJ Digit Med"},{"key":"R3","unstructured":"Sozialgesetzbuch (SGB) F\u00fcnftes Buch (V) - Gesetzliche Krankenversicherung - (Artikel 1 des Gesetzes v. 20. Dezember 1988, BGBl. I S. 2477)\u00a7 139e Verzeichnis f\u00fcr digitale Gesundheitsanwendungen; Verordnungserm\u00e4chtigung [Article in German]. Bundesministerium der Justiz und f\u00fcr Verbraucherschutz, Bundesamt f\u00fcr Justiz. URL: https:\/\/www.gesetze-im-internet.de\/sgb_5\/__139e.html [Accessed 30-06-2026]"},{"key":"R4","unstructured":"DiGA-Verzeichnis [Article in German]. Bundesinstitut f\u00fcr Arzneimittel und Medizinprodukte. URL: https:\/\/diga.bfarm.de\/de\/verzeichnis [Accessed 15-03-2025]"},{"key":"R5","unstructured":"Verordnung \u00fcber das Verfahren und die Anforderungen zur Pr\u00fcfung der Erstattungsf\u00e4higkeit digitaler Gesundheitsanwendungen in der gesetzlichen Krankenversicherung (Digitale Gesundheitsanwendungen-Verordnung - DiGAV)\u00a7 8 Begriff der positiven Versorgungseffekte [Article in German]. Bundesministerium der Justiz und f\u00fcr Verbraucherschutz, Bundesamt f\u00fcr Justiz. 2020. URL: https:\/\/www.gesetze-im-internet.de\/digav\/__8.html [Accessed 10-05-2025]"},{"key":"R6","unstructured":"Bericht nach \u00a7 139e Absatz 10 SGB V zur Nutzung, Akzeptanz und Wirkung digitaler Gesundheitsanwendungen \u2013 DiGA-Bericht 2024 [Report in German]. GKV-Spitzenverband; 2025. URL: https:\/\/www.gkv-spitzenverband.de\/media\/dokumente\/krankenversicherung_1\/telematik\/digitales\/2024_DiGA-Bericht_final.pdf[Accessed 20-09-2026]"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-023-09287-w","article-title":"Evidence requirements of permanently listed digital health applications (DiGA) and their implementation in the German DiGA directory: an analysis","volume":"23","author":"M\u00e4der","journal-title":"BMC Health Serv Res"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/54518","article-title":"Key considerations for designing clinical studies to evaluate digital health solutions","volume":"26","author":"Bolinger","journal-title":"J Med Internet Res"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00314-2","article-title":"Challenges for the evaluation of digital health solutions-a call for innovative evidence generation approaches","volume":"3","author":"Guo","journal-title":"NPJ Digit Med"},{"issue":"3","key":"R10","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1007\/s12599-024-00870-x","article-title":"Reimagining digital health","volume":"66","author":"Sunyaev","journal-title":"Bus Inf Syst Eng"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s00103-023-03804-2","article-title":"Digitale Gesundheitsanwendungen (DiGA) im Spannungsfeld von Fortschritt und Kritik : Diskussionsbeitrag der Fachgruppe \u201eDigital Health\u201c der Gesellschaft f\u00fcr Informatik e. V [Article in German]","volume":"67","author":"Schlieter","journal-title":"Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","first-page":"100848","DOI":"10.1016\/j.hlpt.2024.100848","article-title":"Apps don\u2019t work for patients who don\u2019t use them: towards frameworks for digital therapeutics adherence","volume":"13","author":"Schwartz","journal-title":"Health Policy Technol"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-025-12656-2","article-title":"From awareness to integration: a qualitative interview study on the impact of digital therapeutics on physicians\u2019 practices in Germany","volume":"25","author":"Kendziorra","journal-title":"BMC Health Serv Res"},{"issue":"7","key":"R14","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10222-6","article-title":"A large scale analysis of mHealth app user reviews","volume":"27","author":"Haggag","journal-title":"Empir Softw Eng"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1145\/2414536.2414577","article-title":"A preliminary analysis of mobile app user reviews","author":"Vasa","journal-title":"Proc 24th Aust Comput-Human Interact Conf"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/j.zefq.2022.09.008","article-title":"Wie belastbar sind Studien der aktuell dauerhaft aufgenommenen digitalen Gesundheitsanwendungen (DiGA)? Methodische Qualit\u00e4t der Studien zum Nachweis positiver Versorgungseffekte von DiGA [Article in German]","volume":"175","author":"Kolominsky-Rabas","journal-title":"Z Evid Fortbild Qual Gesundhwes"},{"key":"R17","doi-asserted-by":"crossref","unstructured":"Xiaozhou L Zheying Z Kostas S . Mobile app evolution analysis based on user reviews. In: Fujita H Herrera-Viedma E , editors. New Trends in Intelligent Software Methodologies, Tools and Techniques. IOS Press; 2018:773-786. [doi: 10.3233\/978-1-61499-900-3-773]","DOI":"10.3233\/978-1-61499-900-3-773"},{"key":"R18","unstructured":"Das Fast-Track-Verfahren f\u00fcr digitale Gesundheitsanwendungen (DiGA) nach \u00a7 139e SGB V [Report in German]. Bundesinstitut f\u00fcr Arzneimittel und Medizinprodukte (BfArM); Dec 2023. URL: https:\/\/www.bfarm.de\/SharedDocs\/Downloads\/DE\/Medizinprodukte\/diga_leitfaden.pdf?__blob=publicationFile [Accessed 07-07-2025]"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00862-3","article-title":"Rating analysis and BERTopic modeling of consumer versus regulated mHealth app reviews in Germany","volume":"6","author":"Uncovska","journal-title":"NPJ Digit Med"},{"issue":"5","key":"R20","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.84710","article-title":"Sentiment and thematic analysis of user reviews for FDA-cleared prescription digital therapeutics: a mixed-methods real-world evidence study","volume":"17","author":"Lakhan","journal-title":"Cureus"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","first-page":"2042","DOI":"10.1177\/1460458219896492","article-title":"Insights from user reviews to improve mental health apps","volume":"26","author":"Alqahtani","journal-title":"Health Informatics J"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s40547-024-00143-4","article-title":"Sentiment analysis in the age of generative AI","volume":"11","author":"Krugmann","journal-title":"Cust Need and Solut"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1136\/bmjment-2025-301762","article-title":"Development and evaluation of prompts for a large language model to screen titles and abstracts in a living systematic review","volume":"28","author":"Homiar","journal-title":"BMJ Ment Health"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.220007","article-title":"Performance of multiple pretrained BERT models to automate and accelerate data annotation for large datasets","volume":"4","author":"Tejani","journal-title":"Radiol Artif Intell"},{"issue":"7321","key":"R25","doi-asserted-by":"publisher","first-page":"1103","DOI":"10.1136\/bmj.323.7321.1103","article-title":"Ethical issues in qualitative research on internet communities","volume":"323","author":"Eysenbach","journal-title":"BMJ"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2013.6636712","article-title":"User feedback in the appstore: an empirical study","author":"Pagano","journal-title":"21st IEEE Int Requirements Eng Conf"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1177\/20552076241249604","article-title":"Critical success factors for creating sustainable digital health applications: a systematic review of the German case","volume":"10","author":"Schramm","journal-title":"Digit Healt"},{"key":"R28","unstructured":"Hull E Jackson K Dick J . Requirements Engineering. 2nd ed. Springer; 2005. ISBN: 978-1-84628-075-7"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"3316","DOI":"10.1007\/s10664-019-09706-9","article-title":"Towards understanding and detecting fake reviews in app stores","volume":"24","author":"Martens","journal-title":"Empir Software Eng"},{"key":"R30","unstructured":"ISO\/IEC\/IEEE International Standard - systems and software engineering -- life cycle processes -- requirements engineering. International Organization for Standardization (ISO), International Electrotechnical Commission (IEC), and Institute of Electrical and Electronics Engineers (IEEE); Nov 2018:1. URL: https:\/\/www.iso.org\/standard\/72089.html [Accessed 16-09-2026]"}],"container-title":["JMIR Human Factors"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T21:23:45Z","timestamp":1791494625000},"score":1,"resource":{"primary":{"URL":"https:\/\/humanfactors.jmir.org\/2026\/1\/e92415"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,8]]},"references-count":30,"URL":"https:\/\/doi.org\/10.2196\/92415","relation":{},"ISSN":["2292-9495"],"issn-type":[{"value":"2292-9495","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,8]]},"article-number":"v13i1e92415"}}