{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:45:33Z","timestamp":1785545133523,"version":"3.56.0"},"reference-count":34,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Understanding and improving patient care is pivotal for health care providers. With increasing volumes of the Friends and Family Test (FFT) data in England, manual analysis of this patient feedback poses challenges for many health care organizations. This underscores the importance of automated text analysis, particularly in predicting sentiments and themes in real time.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n Leveraging machine learning and natural language processing, this study explores the utility of a supervised algorithm to systematically test and refine the algorithm\u2019s cross-contextual performance in diverse health care settings, addressing variations in population characteristics, geographical locations, and care settings, ultimately driving improvements based on patient feedback.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n The text analytics algorithm initially developed in a large acute trust in London was further tested in 9 health care organizations with diverse care settings across England. These trusts varied in technical capacity and resource, population demographics, and FFT free text datasets. Testing and validation of the algorithm were performed, including manual coding of a subset of retrospective comments. Technical infrastructure, including coding environments and packages for algorithm testing and deployment, was optimized. The algorithm was iteratively trained using bag of words from anonymized data, tailored to accommodate contextual variations, and tested for change in algorithm performance while simultaneously rectifying issues identified.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The algorithm demonstrated satisfactory overall accuracy (>75%) in predicting themes and sentiments embedded within free-text responses across a variety of care settings and population demographics. While the algorithm yielded strong and reusable models in relatively stable environments, such as adult inpatient care settings, the initial accuracy was notably lower in organizations providing services such as pediatrics and mental health. However, the accuracy of our algorithm significantly improved when individual trust coding templates were applied. Thematic saturation was reached after the fifth organization was recruited, and no further coding was required for the last 4 organizations. Subsequently, a framework and pipeline for deployment of the algorithm were developed to provide a standardized approach for implementation and analysis of FFT free text, ensuring ease of use.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study represents a significant step forward in leveraging free-text FFT data for valuable insights in diverse health care settings through the testing and development of a robust supervised learning text analytics algorithm. The disparity in some care settings was anticipated, given that the lexicon and phraseology used was inherently different from those prevalent in adult inpatient care (where the algorithm was developed). However, these challenges were addressed with further coding and testing. This approach enhanced the accuracy and reliability of the algorithm, encouraged inter- and intraorganizational collaboration, and shared learning.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/60900","type":"journal-article","created":{"date-parts":[[2025,4,11]],"date-time":"2025-04-11T17:05:06Z","timestamp":1744391106000},"page":"e60900-e60900","source":"Crossref","is-referenced-by-count":0,"title":["Leveraging AI to Drive Timely Improvements in Patient Experience Feedback: Algorithm Validation"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4434-1785","authenticated-orcid":false,"given":"Mustafa","family":"Khanbhai","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5772-8029","authenticated-orcid":false,"given":"Catalina","family":"Carenzo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2435-8638","authenticated-orcid":false,"given":"Sarindi","family":"Aryasinghe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6490-7272","authenticated-orcid":false,"given":"David","family":"Manton","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5509-4580","authenticated-orcid":false,"given":"Erik","family":"Mayer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,7,10]]},"reference":[{"issue":"6","key":"R1","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1136\/bmjqs-2016-005223","article-title":"Can we use patient-reported feedback to drive change? The challenges of using patient-reported feedback and how they might be addressed","volume":"26","author":"Flott","journal-title":"BMJ Qual Saf"},{"key":"R2","unstructured":"The Friends and Family Test. NHS England; 2014. URL: https:\/\/www.england.nhs.uk\/fft\/ [Accessed 24-05-2025]"},{"issue":"8","key":"R3","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-011907","article-title":"Systematic review of approaches to using patient experience data for quality improvement in healthcare settings","volume":"6","author":"Gleeson","journal-title":"BMJ Open"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.1697","article-title":"Analyzing patients satisfaction level for medical services using Twitter data","volume":"10","author":"Usman","journal-title":"PeerJ Comput Sci"},{"key":"R5","unstructured":"Using the Friends and Family Test to improve patient experience. NHS England and NHS Improvement; 2019. URL: https:\/\/www.england.nhs.uk\/wp-content\/uploads\/2019\/09\/using-the-fft-to-improve-patient-experience-guidance-v2.pdf [Accessed 24-05-2025]"},{"issue":"3","key":"R6","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1136\/bmjqs-2012-001527","article-title":"Harnessing the cloud of patient experience: using social media to detect poor quality healthcare","volume":"22","author":"Greaves","journal-title":"BMJ Qual Saf"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2021.104642","article-title":"Using natural language processing to understand, facilitate and maintain continuity in patient experience across transitions of care","volume":"157","author":"Khanbhai","journal-title":"Int J Med Inform"},{"key":"R8","unstructured":"Devlin J Chang M Lee K Toutanova K . Editor BERT: pre-training of deep bidirectional transformers for language understanding. Presented at: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Jun 2-7, 2019; Minneapolis, Minnesota."},{"issue":"3","key":"R9","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1016\/j.ipha.2024.03.002","article-title":"Exploring the competence of ChatGPT for customer and patient service management","volume":"2","author":"Haleem","journal-title":"Intell Pharm"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2020-100262","article-title":"Applying natural language processing and machine learning techniques to patient experience feedback: a systematic review","volume":"28","author":"Khanbhai","journal-title":"BMJ Health Care Inform"},{"key":"R11","unstructured":"Sakai H Lam S Mikaeili M Bosire J Jovin F . Large language models for patient comments multi-label classification. arXiv. Preprint posted online on Oct 31, 2024. [doi: 10.48550\/arXiv.2410.23528]"},{"issue":"7","key":"R12","doi-asserted-by":"publisher","first-page":"153","DOI":"10.12700\/APH.20.7.2023.7.9","article-title":"Analyzing narratives of patient experiences: a BERT topic modeling approach","volume":"20","author":"Osv\u00e1th","journal-title":"Acta Polytech Hung"},{"key":"R13","doi-asserted-by":"crossref","unstructured":"Strubell E Ganesh A McCallum A . Energy and policy considerations for deep learning in NLP. arXiv. Preprint posted online on Jun 5, 2019. [doi: 10.48550\/arXiv.1906.02243]","DOI":"10.18653\/v1\/P19-1355"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/27887","article-title":"Enriching the value of patient experience feedback: web-based dashboard development using co-design and heuristic evaluation","volume":"9","author":"Khanbhai","journal-title":"JMIR Hum Factors"},{"key":"R15","unstructured":"Enston C . The power of the written word. NHS England. 2021. URL: https:\/\/www.england.nhs.uk\/blog\/the-power-of-the-written-word [Accessed 09-05-2025]"},{"issue":"2825","key":"R16","doi-asserted-by":"publisher","first-page":"30","DOI":"10.5555\/1953048.2078195","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","journal-title":"J Mach Learn Res"},{"key":"R17","unstructured":"NHS National Quality Board. NHS patient experience framework. Department of Health; 2011. URL: https:\/\/assets.publishing.service.gov.uk\/media\/5a7c62b740f0b62aff6c14e9\/dh_132788.pdf [Accessed 09-05-2025]"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1108\/eb026526","article-title":"A statistical interpretation of term specificity and its application in retrieval","volume":"28","author":"Jones","journal-title":"J Doc"},{"issue":"1\u20132","key":"R19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1561\/1500000011","article-title":"Opinion mining and sentiment analysis","volume":"2","author":"Pang","journal-title":"FNT in Information Retrieval"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-023-10393-8","article-title":"A review of semi-supervised learning for text classification","author":"Duarte","journal-title":"Artif Intell Rev"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104336","article-title":"Sentiment analysis of clinical narratives: a scoping review","volume":"140","author":"Denecke","journal-title":"J Biomed Inform"},{"issue":"9 Pt B","key":"R22","doi-asserted-by":"publisher","first-page":"1248","DOI":"10.1016\/j.jacr.2019.05.032","article-title":"Collecting data to facilitate change","volume":"16","author":"Towbin","journal-title":"J Am Coll Radiol"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1007\/10_2021_193","article-title":"FAIR data infrastructure","volume":"182","author":"Neumann","journal-title":"Adv Biochem Eng Biotechnol"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"28","DOI":"10.35680\/2372-0247.1003","article-title":"Evaluation and measurement of patient experience","volume":"1","author":"LaVela","journal-title":"Patient Exp J"},{"key":"R25","unstructured":"Garrisson M . The role of volunteer in improving patient experience. The Beryl Institute. 2016. URL: https:\/\/theberylinstitute.org\/product\/the-role-of-the-volunteer-in-improving-patient-experience\/ [Accessed 24-05-2025]"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/resprot.3433","article-title":"Collecting and analyzing patient experiences of health care from social media","volume":"4","author":"Rastegar-Mojarad","journal-title":"JMIR Res Protoc"},{"issue":"4","key":"R27","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1002\/jhm.2533","article-title":"Real-time patient experience surveys of hospitalized medical patients","volume":"11","author":"Indovina","journal-title":"J Hosp Med"},{"issue":"6","key":"R28","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1136\/bmjqs-2015-004309","article-title":"Measuring patient-perceived quality of care in US hospitals using Twitter","volume":"25","author":"Hawkins","journal-title":"BMJ Qual Saf"},{"issue":"10","key":"R29","doi-asserted-by":"publisher","first-page":"838","DOI":"10.1136\/bmjqs-2014-002875","article-title":"Tweets about hospital quality: a mixed methods study","volume":"23","author":"Greaves","journal-title":"BMJ Qual Saf"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1097\/QMH.0b013e3182417fc4","article-title":"Feasibility of real-time satisfaction surveys through automated analysis of patients\u2019 unstructured comments and sentiments","volume":"21","author":"Alemi","journal-title":"Qual Manag Health Care"},{"issue":"2","key":"R31","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1055\/s-0040-1708049","article-title":"Natural language processing to extract meaningful information from patient experience feedback","volume":"11","author":"Nawab","journal-title":"Appl Clin Inform"},{"issue":"8","key":"R32","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1136\/bmjqs-2015-004063","article-title":"Development and testing of a text-mining approach to analyse patients\u2019 comments on their experiences of colorectal cancer care","volume":"25","author":"Wagland","journal-title":"BMJ Qual Saf"},{"issue":"2","key":"R33","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.ijmedinf.2014.10.001","article-title":"Dashboards for improving patient care: review of the literature","volume":"84","author":"Dowding","journal-title":"Int J Med Inform"},{"key":"R34","unstructured":"Pereira P . What gets measured gets done. or does it? The Health Foundation. 2014. URL: https:\/\/www.health.org.uk\/blogs\/what-gets-measured-gets-done-or-does-it [Accessed 09-05-2025]"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e60900","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:31:36Z","timestamp":1785537096000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e60900"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,10]]},"references-count":34,"URL":"https:\/\/doi.org\/10.2196\/60900","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,10]]},"article-number":"v13i6e60900"}}