{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T04:23:56Z","timestamp":1771647836677,"version":"3.50.1"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,5,27]],"date-time":"2020-05-27T00:00:00Z","timestamp":1590537600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,5,27]],"date-time":"2020-05-27T00:00:00Z","timestamp":1590537600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>Patient experience surveys often include free-text responses. Analysis of these responses is time-consuming and often underutilized. This study examined whether Natural Language Processing (NLP) techniques could provide a data-driven, hospital-independent solution to indicate points for quality improvement.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Methods<\/jats:title>\n<jats:p>This retrospective study used routinely collected patient experience data from two hospitals. A data-driven NLP approach was used. Free-text responses were categorized into topics, subtopics (i.e. n-grams) and labelled with a sentiment score. The indicator \u2018impact\u2019, combining sentiment and frequency, was calculated to reveal topics to improve, monitor or celebrate. The topic modelling architecture was tested on data from a second hospital to examine whether the architecture is transferable to another hospital.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>A total of 38,664 survey responses from the first hospital resulted in 127 topics and 294 n-grams. The indicator \u2018impact\u2019 revealed n-grams to celebrate (15.3%), improve (8.8%), and monitor (16.7%). For hospital 2, a similar percentage of free-text responses could be labelled with a topic and n-grams. Between-hospitals, most topics (69.7%) were similar, but 32.2% of topics for hospital 1 and 29.0% of topics for hospital 2 were unique.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusions<\/jats:title>\n<jats:p>In both hospitals, NLP techniques could be used to categorize patient experience free-text responses into topics, sentiment labels and to define priorities for improvement. The model\u2019s architecture was shown to be hospital-specific as it was able to discover new topics for the second hospital. These methods should be considered for future patient experience analyses to make better use of this valuable source of information.<\/jats:p>\n<\/jats:sec>","DOI":"10.1186\/s12911-020-1104-5","type":"journal-article","created":{"date-parts":[[2020,5,27]],"date-time":"2020-05-27T09:02:48Z","timestamp":1590570168000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":35,"title":["How to automatically turn patient experience free-text responses into actionable insights: a natural language programming (NLP) approach"],"prefix":"10.1186","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6213-7147","authenticated-orcid":false,"given":"Simone A.","family":"Cammel","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marit S.","family":"De Vos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daphne","family":"van Soest","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kristina M.","family":"Hettne","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fred","family":"Boer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ewout W.","family":"Steyerberg","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hileen","family":"Boosman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,5,27]]},"reference":[{"key":"1104_CR1","unstructured":"Hamming, J. F., H. Boosman, and P. J. de Mheen Marang-van. \"The Association Between Complications, Incidents, and Patient Experience: Retrospective Linkage of Routine Patient Experience Surveys and Safety Data.\" Journal of patient safety (2019)."},{"issue":"6","key":"1104_CR2","doi-asserted-by":"publisher","first-page":"e015726","DOI":"10.1136\/bmjopen-2016-015726","volume":"7","author":"M Cunningham","year":"2017","unstructured":"Cunningham M, Wells M. Qualitative analysis of 6961 free-text comments from the first National Cancer Patient Experience Survey in Scotland. BMJ Open. 2017;7(6):e015726.","journal-title":"BMJ Open"},{"key":"1104_CR3","volume-title":"Supervised topic models","author":"DM Blei","year":"2010","unstructured":"Blei DM, McAuliffe JD. Supervised topic models; 2010."},{"key":"1104_CR4","volume-title":"Topic modeling and Latent Dirichlet Allocation (LDA) in Python","author":"S Li","year":"2018","unstructured":"Li S. Topic modeling and Latent Dirichlet Allocation (LDA) in Python; 2018."},{"key":"1104_CR5","volume-title":"Sentiment analysis model to emphasize the impact of online reviews in Healthcare industry, vol. 41","author":"AM Abirami","year":"2017","unstructured":"Abirami AM, Askarunisa A. Sentiment analysis model to emphasize the impact of online reviews in Healthcare industry, vol. 41; 2017."},{"issue":"585","key":"1104_CR6","doi-asserted-by":"publisher","first-page":"157","DOI":"10.3399\/bjgp11X567072","volume":"61","author":"A Edwards","year":"2011","unstructured":"Edwards A, Evans R, White P, Elwyn G. Experiencing patient-experience surveys: a qualitative study of the accounts of GPs. Br J Gen Pract. 2011;61(585):157\u201366.","journal-title":"Br J Gen Pract"},{"issue":"3","key":"1104_CR7","doi-asserted-by":"publisher","first-page":"15","DOI":"10.35680\/2372-0247.1234","volume":"4","author":"AS Gallan","year":"2017","unstructured":"Gallan AS, Girju M, Girju R. Perfect ratings with negative comments: learning from contradictory patient survey responses. Patient Exp J. 2017;4(3):15\u201328.","journal-title":"Patient Exp J"},{"key":"1104_CR8","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1145\/3006299.3006335","volume-title":"Proceedings of the 3rd IEEE\/ACM International Conference on Big Data Computing, Applications and Technologies","author":"M Bahja","year":"2016","unstructured":"Bahja M, Lycett M. Identifying patient experience from online resources via sentiment analysis and topic modelling. In: Proceedings of the 3rd IEEE\/ACM International Conference on Big Data Computing, Applications and Technologies; 2016. p. 94\u20139."},{"issue":"9","key":"1104_CR9","doi-asserted-by":"publisher","first-page":"e011830","DOI":"10.1136\/bmjopen-2016-011830","volume":"6","author":"M Bracher","year":"2016","unstructured":"Bracher M, Corner DJ, Wagland R. Exploring experiences of cancer care in Wales: a thematic analysis of free-text responses to the 2013 Wales Cancer Patient Experience Survey (WCPES). BMJ Open. 2016;6(9):e011830.","journal-title":"BMJ Open"},{"key":"1104_CR10","unstructured":"Esuli, Andrea, Alejandro Moreo, and Fabrizio Sebastiani. \"Building Automated Survey Coders via Interactive Machine Learning.\" arXiv preprint arXiv:1903.12110 (2019)."},{"issue":"3","key":"1104_CR11","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1080\/10580530.2015.1044342","volume":"32","author":"P Varanasi","year":"2015","unstructured":"Varanasi P, Tanniru M. Seeking intelligence from patient experience using text mining: analysis of emergency department data. Inf Syst Manag. 2015;32(3):220\u20138.","journal-title":"Inf Syst Manag"},{"issue":"3","key":"1104_CR12","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1016\/j.jnn.2017.09.003","volume":"24","author":"E Ainley","year":"2018","unstructured":"Ainley E, King J, K\u00e4sbauer S, Cooper R. A framework analysis of free-text data from the neonatal survey 2014. J Neonatal Nurs. 2018;24(3):163\u20138.","journal-title":"J Neonatal Nurs"},{"key":"1104_CR13","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718520","volume-title":"Twitterrank: Finding Topic-Sensitive Influential Twitterers","author":"J Weng","year":"2010","unstructured":"Weng J, Lim E-P, Jiang J, Qi Z. Twitterrank: Finding Topic-Sensitive Influential Twitterers; 2010."},{"key":"1104_CR14","volume-title":"Automatic Classification of Unstructured Blog Text, vol. 05","author":"MK Dalal","year":"2013","unstructured":"Dalal MK, Zaveri M. Automatic Classification of Unstructured Blog Text, vol. 05; 2013."},{"key":"1104_CR15","volume-title":"Sentiment Analysis: Concept, Analysis and Applications","author":"S Gupta","year":"2018","unstructured":"Gupta S. Sentiment Analysis: Concept, Analysis and Applications; 2018."},{"key":"1104_CR16","volume-title":"Use of Sentiment Analysis for Capturing Patient Experience From Free-Text Comments Posted Online, vol. 15","author":"F Greaves","year":"2013","unstructured":"Greaves F, Ramirez-Cano D, Millett C, Darzi A, Donaldson L. Use of Sentiment Analysis for Capturing Patient Experience From Free-Text Comments Posted Online, vol. 15; 2013."},{"key":"1104_CR17","volume-title":"How to Write a Spelling Corrector","author":"P Norvig","year":"2016","unstructured":"P. Norvig, How to Write a Spelling Corrector. 2016. [Online]. Available: https:\/\/norvig.com\/spell-correct.html."},{"key":"1104_CR18","unstructured":"J. Words!, \u201cDictionaries.\u201d [Online]. Available: http:\/\/www.gwicks.net\/dictionaries.htm. Accessed May 2019."},{"key":"1104_CR19","first-page":"640","volume-title":"Proceedings of the AMIA annual fall symposium","author":"KA Spackman","year":"1997","unstructured":"Spackman KA, Campbell KE, C\u00f4t\u00e9 RA. SNOMED RT: a reference terminology for health care. In: Proceedings of the AMIA annual fall symposium; 1997. p. 640."},{"key":"1104_CR20","unstructured":"Van Rossum, G., & Drake, F. L. (2009). Python 3 Reference Manual. Scotts Valley, CA: CreateSpace; 2019."},{"key":"1104_CR21","unstructured":"Bird, Steven, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. \" O'Reilly Media, Inc.\", 2009."},{"key":"1104_CR22","volume-title":"Guide to NumPy, 2nd ed","author":"TE Oliphant","year":"2015","unstructured":"Oliphant TE. Guide to NumPy, 2nd ed. USA: CreateSpace Independent Publishing Platform; 2015."},{"key":"1104_CR23","unstructured":"Pedregosa, Fabian, et al. \"Scikit-learn: Machine learning in Python.\" the Journal of machine Learning research 12 (2011): 2825-2830."},{"issue":"3","key":"1104_CR24","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/MCSE.2007.55","volume":"9","author":"JD Hunter","year":"2007","unstructured":"Hunter JD. Matplotlib: a 2D graphics environment. Comput Sci Eng. 2007;9(3):90\u20135.","journal-title":"Comput Sci Eng"},{"key":"1104_CR25","first-page":"556","volume-title":"Advances in neural information processing systems","author":"DD Lee","year":"2001","unstructured":"Lee DD, Seung HS. Algorithms for non-negative matrix factorization. In: Advances in neural information processing systems; 2001. p. 556\u201362."},{"issue":"4","key":"1104_CR26","doi-asserted-by":"publisher","first-page":"459","DOI":"10.1177\/0894439314555329","volume":"33","author":"D O\u2019Callaghan","year":"2014","unstructured":"O\u2019Callaghan D, Greene D, Conway M, Carthy J, Cunningham P. Down the (White) rabbit hole: the extreme right and online recommender systems. Soc Sci Comput Rev. 2014;33(4):459\u201378.","journal-title":"Soc Sci Comput Rev"},{"key":"1104_CR27","first-page":"100","volume-title":"Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics","author":"D Newman","year":"2010","unstructured":"Newman D, Lau JH, Grieser K, Baldwin T. Automatic evaluation of topic coherence. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics; 2010. p. 100\u20138."},{"key":"1104_CR28","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1109\/ICDM.2007.86","volume-title":"Seventh IEEE International Conference on Data Mining (ICDM 2007)","author":"X Wang","year":"2007","unstructured":"Wang X, McCallum A, Wei X. Topical N-Grams: Phrase and Topic Discovery, with an Application to Information Retrieval. In: Seventh IEEE International Conference on Data Mining (ICDM 2007); 2007. p. 697\u2013702."},{"key":"1104_CR29","first-page":"161175","volume-title":"Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval","author":"WB Cavnar","year":"1994","unstructured":"Cavnar WB, Trenkle JM. N-gram-based text categorization. In: Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval; 1994. p. 161175."},{"key":"1104_CR30","volume-title":"FuzzyWuzzy String Matching","author":"A Cohen","year":"2011","unstructured":"A. Cohen, FuzzyWuzzy String Matching. 2011. [Online]. Available: https:\/\/chairnerd.seatgeek.com\/fuzzywuzzy-fuzzy-string-matching-in-python\/."},{"issue":"3","key":"1104_CR31","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276\u201382.","journal-title":"Biochem Med"},{"issue":"1","key":"1104_CR32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ejor.2015.12.023","volume":"253","author":"T Aven","year":"2016","unstructured":"Aven T. Risk assessment and risk management: review of recent advances on their foundation. Eur J Oper Res. 2016;253(1):1\u201313.","journal-title":"Eur J Oper Res"},{"issue":"8","key":"1104_CR33","doi-asserted-by":"publisher","first-page":"604 LP","DOI":"10.1136\/bmjqs-2015-004063","volume":"25","author":"R Wagland","year":"2016","unstructured":"Wagland R, et al. Development and testing of a text-mining approach to analyse patients\u2019 comments on their experiences of colorectal cancer care. BMJ Qual Saf. 2016;25(8):604 LP\u2013614.","journal-title":"BMJ Qual Saf"},{"key":"1104_CR34","first-page":"191","volume-title":"Use of Sentiment Mining and Online NMF for Topic Modeling Through the Analysis of Patients Online Unstructured Comments: International Conference, ICSH 2018, Wuhan, China, July 1\u20133, 2018, Proceedings","author":"A Shah","year":"2018","unstructured":"Shah A, Yan X, Shah S, Khan S. Use of Sentiment Mining and Online NMF for Topic Modeling Through the Analysis of Patients Online Unstructured Comments: International Conference, ICSH 2018, Wuhan, China, July 1\u20133, 2018, Proceedings; 2018. p. 191\u2013203."},{"issue":"1","key":"1104_CR35","first-page":"7","volume":"6","author":"JV Carrera-Trejo","year":"2015","unstructured":"Carrera-Trejo JV, Sidorov G, Miranda-Jim\u00e9nez S, Moreno Ibarra M, Cadena Mart\u00ednez R. Latent Dirichlet allocation complement in the vector space model for multi-label text classification. Int J Comb Optim Probl Informatics. 2015;6(1):7\u201319.","journal-title":"Int J Comb Optim Probl Informatics"},{"issue":"7584","key":"1104_CR36","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1136\/bmj.39071.441609.80","volume":"334","author":"C Vincent","year":"2007","unstructured":"Vincent C. Incident reporting and patient safety. BMJ. 2007;334(7584):51.","journal-title":"BMJ"},{"issue":"9","key":"1104_CR37","doi-asserted-by":"publisher","first-page":"758 LP","DOI":"10.1136\/bmjqs-2017-007463","volume":"27","author":"MS de Vos","year":"2018","unstructured":"de Vos MS, Hamming JF, Marang-van de Mheen PJ. The problem with using patient complaints for improvement. BMJ Qual Saf. 2018;27(9):758 LP\u2013762.","journal-title":"BMJ Qual Saf"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1104-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-020-1104-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1104-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,26]],"date-time":"2021-05-26T23:24:56Z","timestamp":1622071496000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-1104-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,27]]},"references-count":37,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1104"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-1104-5","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,27]]},"assertion":[{"value":"13 November 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 April 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 May 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The requirement for ethical approval was waived by the local Ethics Committee (METC Leiden-Den Hag-Delft) (#N19.121) based on the Dutch Medical Research Involving Human Subject Act.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"97"}}