{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T06:32:19Z","timestamp":1775111539776,"version":"3.50.1"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T00:00:00Z","timestamp":1657843200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T00:00:00Z","timestamp":1657843200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Evaluating patients\u2019 experiences is essential when incorporating the patients\u2019 perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-022-01923-5","type":"journal-article","created":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T10:04:24Z","timestamp":1657879464000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":46,"title":["Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)"],"prefix":"10.1186","volume":"22","author":[{"given":"Marieke M.","family":"van Buchem","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Olaf M.","family":"Neve","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ilse M. J.","family":"Kant","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ewout W.","family":"Steyerberg","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hileen","family":"Boosman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Erik F.","family":"Hensen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,7,15]]},"reference":[{"key":"1923_CR1","doi-asserted-by":"publisher","first-page":"221","DOI":"10.2147\/PROM.S274015","volume":"11","author":"CM Bastemeijer","year":"2020","unstructured":"Bastemeijer CM, Boosman H, Zandbelt L, Timman R, de Boer D, Hazelzet JA. Patient experience monitor (PEM): the development of new short-form picker experience Questionnaires for hospital patients with a wide range of literacy levels<\/p>. Patient Relat Outcome Meas. 2020;11:221\u201330.","journal-title":"Patient Relat Outcome Meas"},{"key":"1923_CR2","unstructured":"Medicine I of. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001; Available from: https:\/\/www.nap.edu\/catalog\/10027\/crossing-the-quality-chasm-a-new-health-system-for-the"},{"issue":"1","key":"1923_CR3","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2020-100262","volume":"28","author":"M Khanbhai","year":"2021","unstructured":"Khanbhai M, Anyadi P, Symons J, Flott K, Darzi A, Mayer E. Applying natural language processing and machine learning techniques to patient experience feedback: a systematic review. Bmj Heal Care Inform. 2021;28(1): e100262.","journal-title":"Bmj Heal Care Inform"},{"issue":"5","key":"1923_CR4","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1093\/intqhc\/mzs039","volume":"24","author":"E Riiskjaer","year":"2012","unstructured":"Riiskjaer E, Ammentorp J, Kofoed PE. The value of open-ended questions in surveys on patient experience: number of comments and perceived usefulness from a hospital perspective. Int J Qual Health C. 2012;24(5):509\u201316.","journal-title":"Int J Qual Health C"},{"issue":"1","key":"1923_CR5","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1097\/QMH.0b013e3182417fc4","volume":"21","author":"F Alemi","year":"2012","unstructured":"Alemi F, Torii M, Clementz L, Aron DC. Feasibility of real-time satisfaction surveys through automated analysis of patients\u2019 unstructured comments and sentiments. Qual Manag Health Ca. 2012;21(1):9\u201319.","journal-title":"Qual Manag Health Ca"},{"key":"1923_CR6","doi-asserted-by":"crossref","unstructured":"Anjum A, Zhao X, Bahja M, Lycett M. Identifying patient experience from online resources via sentiment analysis and topic modelling. Proc 3rd Ieee Acm Int Conf Big Data Comput Appl Technologies. 2016;94\u20139.","DOI":"10.1145\/3006299.3006335"},{"issue":"4","key":"1923_CR7","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.9162","volume":"6","author":"J Jones","year":"2018","unstructured":"Jones J, Pradhan M, Hosseini M, Kulanthaivel A, Hosseini M. Novel approach to cluster patient-generated data into actionable topics: case study of a web-based breast cancer forum. JMIR Med Inform. 2018;6(4): e45.","journal-title":"JMIR Med Inform"},{"key":"1923_CR8","doi-asserted-by":"publisher","first-page":"S10","DOI":"10.1016\/S0140-6736(13)60366-9","volume":"380","author":"F Greaves","year":"2012","unstructured":"Greaves F, Ramirez-Cano D, Millett C, Darzi A, Donaldson L. Machine learning and sentiment analysis of unstructured free-text information about patient experience online. Lancet. 2012;380:S10.","journal-title":"Lancet"},{"issue":"4","key":"1923_CR9","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1377\/hlthaff.2015.1030","volume":"35","author":"BL Ranard","year":"2017","unstructured":"Ranard BL, Werner RM, Antanavicius T, Schwartz HA, Smith RJ, Meisel ZF, et al. Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care. Health Affair. 2017;35(4):697\u2013705.","journal-title":"Health Affair"},{"issue":"1","key":"1923_CR10","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1186\/s12911-020-1104-5","volume":"20","author":"SA Cammel","year":"2020","unstructured":"Cammel SA, Vos MSD, van Soest D, Hettne KM, Boer F, Steyerberg EW, et al. How to automatically turn patient experience free-text responses into actionable insights: a natural language programming (NLP) approach. Bmc Med Inform Decis. 2020;20(1):97.","journal-title":"Bmc Med Inform Decis"},{"key":"1923_CR11","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2021.104642","volume":"157","author":"M Khanbhai","year":"2022","unstructured":"Khanbhai M, Warren L, Symons J, Flott K, Harrison-White S, Manton D, et al. Using natural language processing to understand, facilitate and maintain continuity in patient experience across transitions of care. Int J Med Inform. 2022;157: 104642.","journal-title":"Int J Med Inform"},{"issue":"4","key":"1923_CR12","doi-asserted-by":"publisher","first-page":"330","DOI":"10.2106\/JBJS.18.00695","volume":"101","author":"ME Menendez","year":"2019","unstructured":"Menendez ME, Shaker J, Lawler SM, Ring D, Jawa A. Negative patient-experience comments after total shoulder arthroplasty. J Bone Joint Surg. 2019;101(4):330\u20137.","journal-title":"J Bone Joint Surg"},{"issue":"23","key":"1923_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3310\/hsdr07230","volume":"7","author":"C Rivas","year":"2019","unstructured":"Rivas C, Tkacz D, Antao L, Mentzakis E, Gordon M, Anstee S, et al. Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study. Heal Serv Deliv Res. 2019;7(23):1\u2013160.","journal-title":"Heal Serv Deliv Res"},{"issue":"02","key":"1923_CR14","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1055\/s-0040-1708049","volume":"11","author":"K Nawab","year":"2020","unstructured":"Nawab K, Ramsey G, Schreiber R. Natural language processing to extract meaningful information from patient experience feedback. Appl Clin Inform. 2020;11(02):242\u201352.","journal-title":"Appl Clin Inform"},{"key":"1923_CR15","unstructured":"Doing-Harris K, Mowery DL, Daniels C, Chapman WW, Conway M. Understanding patient satisfaction with received healthcare services: A natural language processing approach. In: AMIA annual symposium proceedings. 2017."},{"issue":"Suppl 1","key":"1923_CR16","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s13326-019-0215-3","volume":"10","author":"I Spasi\u0107","year":"2019","unstructured":"Spasi\u0107 I, Owen D, Smith A, Button K. KLOSURE: closing in on open\u2013ended patient questionnaires with text mining. J Biomed Semant. 2019;10(Suppl 1):24.","journal-title":"J Biomed Semant"},{"issue":"10","key":"1923_CR17","doi-asserted-by":"publisher","first-page":"953","DOI":"10.1111\/j.1525-1497.2005.0178.x","volume":"20","author":"K Davis","year":"2005","unstructured":"Davis K, Schoenbaum SC, Audet AM. A 2020 vision of patient-centered primary care. J Gen Intern Med. 2005;20(10):953\u20137.","journal-title":"J Gen Intern Med"},{"issue":"1","key":"1923_CR18","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1177\/0194599817695800","volume":"157","author":"G Soulier","year":"2017","unstructured":"Soulier G, van Leeuwen BM, Putter H, Jansen JC, Malessy MJA, van Benthem PPG, et al. Quality of life in 807 patients with vestibular schwannoma: comparing treatment modalities. Otolaryngology Head Neck Surg. 2017;157(1):92\u20138.","journal-title":"Otolaryngology Head Neck Surg"},{"issue":"8","key":"1923_CR19","doi-asserted-by":"publisher","first-page":"2197","DOI":"10.1007\/s11136-021-02822-4","volume":"30","author":"JJ Gagnier","year":"2021","unstructured":"Gagnier JJ, Lai J, Mokkink LB, Terwee CB. COSMIN reporting guideline for studies on measurement properties of patient-reported outcome measures. Qual Life Res. 2021;30(8):2197\u2013218.","journal-title":"Qual Life Res"},{"key":"1923_CR20","unstructured":"Face H. BERT [Internet]. [cited 2021 Dec 14]. Available from: https:\/\/huggingface.co\/docs\/transformers\/model_doc\/bert#transformers.BertForSequenceClassification"},{"key":"1923_CR21","unstructured":"Norvig P. How to Write a Spelling Corrector [Internet]. 2016 [cited 2021 Nov 21]. Available from: https:\/\/norvig.com\/spell-correct.html"},{"key":"1923_CR22","unstructured":"Seal M, Rodriguez T. CyHunSpell [Internet]. 2021. Available from: https:\/\/pypi.org\/project\/cyhunspell\/"},{"key":"1923_CR23","doi-asserted-by":"crossref","unstructured":"Qi P, Zhang Y, Zhang Y, Bolton J, Manning CD. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In: Proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations [Internet]. 2020. Available from: https:\/\/nlp.stanford.edu\/pubs\/qi2020stanza.pdf","DOI":"10.18653\/v1\/2020.acl-demos.14"},{"key":"1923_CR24","unstructured":"Tulkens S, Emmery C, Daelemans W. Evaluating unsupervised dutch word embeddings as a linguistic resource. In: Proceedings of the tenth international conference on language resources and evaluation (LREC 2016). European language resources association (ELRA); 2016."},{"key":"1923_CR25","unstructured":"Sch\u00e4fer R, Bildhauer F. Building Large Corpora from the Web Using a New Efficient Tool Chain. Piperidis\"] [\"Nicoletta Calzolari (Conference Chair) and Khalid Choukri and Thierry Declerck and Mehmet U\u011fur Do\u011fan and Bente Maegaard and Joseph Mariani and Asuncion Moreno and Jan Odijk and Stelios, editor. 23AD;486\u201393. Available from: http:\/\/rolandschaefer.net\/?p=70"},{"key":"1923_CR26","unstructured":"Sch\u00e4fer R. Processing and querying large web corpora with the COW14 architecture. Witt\"] [\"Piotr Ba\u0144ski and Hanno Biber and Evelyn Breiteneder and Marc Kupietz and Harald L\u00fcngen and Andreas, editor. 2015; Available from: http:\/\/rolandschaefer.net\/?p=749"},{"key":"1923_CR27","unstructured":"Europe C of. Common European Framework of Reference for Languages: Learning, teaching, assessment \u2013 Companion volume [Internet]. Strasbourg: Council of Europe Publishing; 2020. Available from: www.coe.int\/lang-cefr"},{"key":"1923_CR28","doi-asserted-by":"publisher","DOI":"10.2196\/17984","author":"I Spasic","year":"2020","unstructured":"Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform. 2020. https:\/\/doi.org\/10.2196\/17984.","journal-title":"JMIR Med Inform"},{"key":"1923_CR29","first-page":"101","volume":"58","author":"SM Jim\u00e9nez-Zafra","year":"2017","unstructured":"Jim\u00e9nez-Zafra SM, Mart\u00edn-Valdivia MT, Maks I, Izquierdo R. Analysis of patient satisfaction in Dutch and Spanish online reviews. Procesamiento del Lenguaje Natural. 2017;58:101\u20138.","journal-title":"Procesamiento del Lenguaje Natural"},{"issue":"8","key":"1923_CR30","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1136\/bmjqs-2015-004063","volume":"25","author":"R Wagland","year":"2016","unstructured":"Wagland R, Recio-Saucedo A, Simon M, Bracher M, Hunt K, Foster C, et al. Development and testing of a text-mining approach to analyse patients\u2019 comments on their experiences of colorectal cancer care. Bmj Qual Saf. 2016;25(8):604.","journal-title":"Bmj Qual Saf"},{"issue":"3","key":"1923_CR31","doi-asserted-by":"publisher","first-page":"15","DOI":"10.35680\/2372-0247.1234","volume":"4","author":"AS Gallan","year":"2017","unstructured":"Gallan AS, Girju M, Girju R. Perfect ratings with negative comments: learning from contradictory patient survey responses. Patient Exp J. 2017;4(3):15\u201328.","journal-title":"Patient Exp J"},{"key":"1923_CR32","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2020.113304","volume":"135","author":"J Kazmaier","year":"2020","unstructured":"Kazmaier J, van Vuuren JH. A generic framework for sentiment analysis: leveraging opinion-bearing data to inform decision making. Decis Support Syst. 2020;135: 113304.","journal-title":"Decis Support Syst"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01923-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-022-01923-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01923-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,29]],"date-time":"2024-09-29T05:46:32Z","timestamp":1727588792000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-022-01923-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,15]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["1923"],"URL":"https:\/\/doi.org\/10.1186\/s12911-022-01923-5","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,15]]},"assertion":[{"value":"4 April 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 June 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 July 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The medical ethics committee from Leiden Den Haag Delft (METC LDD) waived the need for approval for this study. The reference number of the waiver is N19.121.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"183"}}