{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T21:37:03Z","timestamp":1774733823985,"version":"3.50.1"},"reference-count":48,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>Evaluating patients\u2019 experience and satisfaction often calls for analyses of free-text data. Language and domain-specific information extraction can reduce costly manual preprocessing and enable the analysis of extensive collections of experience-based narratives. The research aims were to (1) elicit free-text narratives about experiences with health services of international students in Poland, (2) develop domain- and language-specific algorithms for the extraction of information relevant for the evaluation of quality and safety of health services, and (3) test the performance of information extraction algorithms\u2019 on questions about the patients\u2019 experiences with health services. The materials were free-text narratives about health clinic encounters produced by English-speaking foreigners recalling their experiences (n = 104) in healthcare facilities in Poland. A linguistic analysis of the text collection led to constructing a semantic\u2013syntactic lexicon and a set of lexical-syntactic frames. These were further used to develop rule-based information extraction algorithms in the form of Python scripts. The extraction algorithms generated text classifications according to predefined queries. In addition, the narratives were classified by human readers. The algorithm-based and the human readers\u2019 classifications were highly correlated and significant (p &lt; 0.01), indicating an excellent performance of the automatic query algorithms. The study results demonstrate that domain-specific and language-specific information extraction from free-text narratives can be used as an efficient and low-cost method for evaluating patient experiences and satisfaction with health services and built into software solutions for the quality evaluation in health care.<\/jats:p>","DOI":"10.3390\/ijerph191610182","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T21:23:56Z","timestamp":1660771436000},"page":"10182","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Evaluating Patients\u2019 Experiences with Healthcare Services: Extracting Domain and Language-Specific Information from Free-Text Narratives"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1121-0049","authenticated-orcid":false,"given":"Barbara","family":"Jacennik","sequence":"first","affiliation":[{"name":"Polish Telemedicine and eHealth Society, 03-728 Warsaw, Poland"}]},{"given":"Emilia","family":"Zawadzka-Gosk","sequence":"additional","affiliation":[{"name":"Multimedia Department, Polish-Japanese Academy of Information Technology, 02-008 Warsaw, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3644-1022","authenticated-orcid":false,"given":"Joaquim Paulo","family":"Moreira","sequence":"additional","affiliation":[{"name":"International Healthcare Management Research and Development Center (IHM-RDC), Shandong Provincial Qianfoshan Hospital, Jinan 250014, China"},{"name":"Gestao em Saude, Atlantica Instituto Universitario, 2730-036 Oeiras, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2602-1128","authenticated-orcid":false,"given":"Wojciech Micha\u0142","family":"Glinkowski","sequence":"additional","affiliation":[{"name":"Polish Telemedicine and eHealth Society, 03-728 Warsaw, Poland"},{"name":"Center of Excellence \u201cTeleOrto\u201d for Telediagnostics and Treatment of Disorders and Injuries of the Locomotor System, Department of Medical Informatics and Telemedicine, Medical University of Warsaw, 00-581 Warsaw, Poland"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,17]]},"reference":[{"key":"ref_1","unstructured":"Fette, G., Ertl, M., W\u00f6rner, A., Kluegl, P., St\u00f6rk, S., and Puppe, F. (2012). Information Extraction from Unstructured Electronic Health Records and Integration into a Data Warehouse. Lecture Notes in Informatics (LNI), Proceedings\u2014Series of the Gesellschaft fur Informatik (GI), Gesellschaft f\u00fcr Informatik."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"H\u00e4rk\u00e4nen, M., Paananen, J., Murrells, T., Rafferty, A.M., and Franklin, B.D. (2019). Identifying Risks Areas Related to Medication Administrations\u2014Text Mining Analysis Using Free-Text Descriptions of Incident Reports. BMC Health Serv. Res., 19.","DOI":"10.1186\/s12913-019-4597-9"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.jbi.2017.12.016","article-title":"Automatic Information Extraction from Unstructured Mammography Reports Using Distributed Semantics","volume":"78","author":"Gupta","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Toyabe, S.I. (2012). Detecting Inpatient Falls by Using Natural Language Processing of Electronic Medical Records. BMC Health Serv. Res., 12.","DOI":"10.1186\/1472-6963-12-448"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical Information Extraction Applications: A Literature Review","volume":"77","author":"Wang","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"923","DOI":"10.1016\/j.jbi.2009.07.007","article-title":"Rule-Based Information Extraction from Patients\u2019 Clinical Data","volume":"42","author":"Mykowiecka","year":"2009","journal-title":"J. Biomed. Inform."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Lorenzoni, G., Rampazzo, R., Buratin, A., Berchialla, P., and Gregori, D. (2021). Does the Integration of Pre-Coded Information with Narratives Improve in-Hospital Falls\u2019 Surveillance?. Appl. Sci., 11.","DOI":"10.3390\/app11104406"},{"key":"ref_8","unstructured":"De Silva, D. (2013). Evidence Scan: Measuring the Patient Experience, The Health Foundation."},{"key":"ref_9","first-page":"1","article-title":"Poland: Health System Review","volume":"21","author":"Sowada","year":"2019","journal-title":"Health Syst. Transit."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Fiorio, C.V., Gorli, M., and Verzillo, S. (2018). Evaluating Organizational Change in Health Care: The Patient-Centered Hospital Model. BMC Health Serv. Res., 18.","DOI":"10.1186\/s12913-018-2877-4"},{"key":"ref_11","first-page":"176","article-title":"What Is Narrative-Based Medicine? Narrative-Based Medicine 1","volume":"64","author":"Zaharias","year":"2018","journal-title":"Can. Fam. Physician"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"e011220","DOI":"10.1136\/bmjopen-2016-011220","article-title":"Research Studies on Patients\u2019 Illness Experience Using the Narrative Medicine Approach: A Systematic Review","volume":"6","author":"Fioretti","year":"2016","journal-title":"BMJ Open"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e031568","DOI":"10.1136\/bmjopen-2019-031568","article-title":"Content and Outcomes of Narrative Medicine Programmes: A Systematic Review of the Literature through 2019","volume":"10","author":"Remein","year":"2020","journal-title":"BMJ Open"},{"key":"ref_14","unstructured":"(2020, December 31). Patient, Family and Carer\u2019s Experience Strategy. Available online: https:\/\/www.elht.nhs.uk\/application\/files\/6115\/2086\/9299\/PatientFamily_and_Carers_Experience_Strategy0_8.pdf."},{"key":"ref_15","unstructured":"NSW Agency for Clinical Innovation, Australia (2020, December 31). Collecting Patient & Carer Stories, Available online: https:\/\/aci.health.nsw.gov.au\/resources\/patient-experience\/collect-stories\/collect_stories\/Guide_to_collect_patient_carer_stories.pdf."},{"key":"ref_16","unstructured":"(2020, December 31). Using Stories to Improve Patient, Carer and Staff Experiences and Outcomes. Available online: https:\/\/www.england.nhs.uk\/6cs\/wp-content\/uploads\/sites\/25\/2015\/09\/scht-storytelling-toolkit.pdf."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Jacennik, B. (2022). On Digital Health Research Priorities: From Telemedicine to Telehealth. Int. Healthc. Rev., 1.","DOI":"10.56226\/ihr.v1i1.13"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1016\/j.ijhm.2005.10.002","article-title":"Text Mining a Decade of Progress in Hospitality Human Resource Management Research: Identifying Emerging Thematic Development","volume":"26","author":"Singh","year":"2007","journal-title":"Int. J. Hosp. Manag."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Vankka, J., Myllykoski, H., Peltonen, T., and Riippa, K. (2019, January 22\u201325). Sentiment Analysis of Finnish Customer Reviews. Proceedings of the 2019 6th International Conference on Social Networks Analysis, Management and Security, SNAMS 2019, Granada, Spain.","DOI":"10.1109\/SNAMS.2019.8931724"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"35318","DOI":"10.1109\/ACCESS.2020.2974983","article-title":"Text Mining of Open-Ended Questions in Self-Assessment of University Teachers: An LDA Topic Modeling Approach","volume":"8","author":"Gonzalez","year":"2020","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e40","DOI":"10.1093\/jamia\/ocw097","article-title":"Congestive Heart Failure Information Extraction Framework for Automated Treatment Performance Measures Assessment","volume":"24","author":"Meystre","year":"2017","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Achilonu, O.J., Olago, V., Singh, E., Eijkemans, R.M.J.C., Nimako, G., and Musenge, E. (2021). A Text Mining Approach in the Classification of Free-Text Cancer Pathology Reports from the South African National Health Laboratory Services. Information, 12.","DOI":"10.3390\/info12110451"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1186\/s40537-019-0254-8","article-title":"An Analytical Study of Information Extraction from Unstructured and Multidimensional Big Data","volume":"6","author":"Adnan","year":"2019","journal-title":"J. Big Data"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1109\/MIS.2002.1039833","article-title":"Mining Open Answers in Questionnaire Data","volume":"17","author":"Yamanishi","year":"2002","journal-title":"IEEE Intell. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.knosys.2008.06.003","article-title":"Mining Fuzzy Association Rules from Questionnaire Data","volume":"22","author":"Chen","year":"2009","journal-title":"Knowl.-Based Syst."},{"key":"ref_26","unstructured":"H\u00e4m\u00e4l\u00e4inen, W., Joy, M., Berger, F., and Huttunen, S. (2018). Clustering Students\u2019 Open-Ended Questionnaire Answers. arXiv."},{"key":"ref_27","unstructured":"Hirasawa, S., Shih, F.-Y., and Yang, W.-T. (2007, January 7\u201310). Student Questionnaire Analyses for Class Management by Text Mining Both in Japanese and in Chinese. Proceedings of the 2007 IEEE International Conference on Systems, Man and Cybernetics, Montreal, Que."},{"key":"ref_28","first-page":"1","article-title":"Bayesian analysis in natural language processing","volume":"9","author":"Cohen","year":"2016","journal-title":"Synth. Lect. Hum. Lang. Technol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s13326-019-0215-3","article-title":"KLOSURE: Closing in on Open\u2013Ended Patient Questionnaires with Text Mining","volume":"10","author":"Owen","year":"2019","journal-title":"J. Biomed. Semant."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"330","DOI":"10.2106\/JBJS.18.00695","article-title":"Negative Patient-Experience Comments After Total Shoulder Arthroplasty","volume":"101","author":"Menendez","year":"2019","journal-title":"JBJS"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Xiong, W., Wu, L., Alleva, F., Droppo, J., Huang, X., and Stolcke, A. (2017, January 15\u201320). The Microsoft 2017 Conversational Speech Recognition System. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.","DOI":"10.1109\/ICASSP.2018.8461870"},{"key":"ref_32","unstructured":"Jacennik, B., and Zwierzchowski, T. (2011). Text Analysis (AT). Program for Quantitative Content Analysis of Text Collections, Version 0.6.8.0, University of Finance and Management in Warsaw."},{"key":"ref_33","unstructured":"Jacennik, B. (2014, January 17). AT program applications to content analysis of Internet texts on health, disease and treatment. Presented at Health Communication Conference, Krak\u00f3w, Poland."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1109\/MCI.2014.2307227","article-title":"Jumping NLP curves: A review of natural language processing research","volume":"9","author":"Cambria","year":"2014","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1109\/MCI.2018.2840738","article-title":"Recent trends in deep learning based natural language processing","volume":"13","author":"Young","year":"2018","journal-title":"IEEE Comput. Intell. Mag."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1055\/s-0040-1702001","article-title":"Medical Information Extraction in the Age of Deep Learning","volume":"29","author":"Hahn","year":"2020","journal-title":"Yearb. Med. Inform."},{"key":"ref_37","unstructured":"Marquez, L., and Salgado, J.G. (2022, January 01). Machine Learning and Natural Language Processing. Available online: https:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.31.3498."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Nagarhalli, T.P., Vaze, V., and Rana, N.K. (2021, January 4\u20136). Impact of machine learning in natural language processing: A review. Proceedings of the 2021 Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV), Tirunelveli, India.","DOI":"10.1109\/ICICV50876.2021.9388380"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Choi, J.D., Tetreault, J., and Stent, A. (2015, January 26\u201331). It Depends: Dependency Parser Comparison Using A Web-Based Evaluation Tool. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.","DOI":"10.3115\/v1\/P15-1038"},{"key":"ref_40","unstructured":"McKinney, W. (2012). Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython, O\u2019Reilly Media, Inc."},{"key":"ref_41","unstructured":"Srinivasa-Desikan, B. (2018). Natural Language Processing and Computational Linguistics: A Practical Guide to Text Analysis with Python, Gensim, spaCy, and Keras, Packt Publishing Ltd."},{"key":"ref_42","unstructured":"Final Code (2022, January 01). COLAB. Available online: http:\/\/bit.do\/fMfBe."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1006\/enfo.2001.0061","article-title":"Detecting Trends Using Spearman\u2019s Rank Correlation Coefficient","volume":"2","author":"Gauthier","year":"2001","journal-title":"Environ. Forensics"},{"key":"ref_44","unstructured":"Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics, SAGE Publications Ltd."},{"key":"ref_45","first-page":"2081","article-title":"How Important Is Syntactic Parsing Accuracy?","volume":"52","author":"Vilares","year":"2017","journal-title":"Empir. Eval. Rule-Based Sentim. Anal."},{"key":"ref_46","first-page":"9","article-title":"Language Models Are Unsupervised Multitask Learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_47","unstructured":"Devlin, J., Chang, M.-W., Lee, K., Google, K.T., and Language, A.I. (2018). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1016\/S1473-3099(20)30120-1","article-title":"An interactive web-based dashboard to track COVID-19 in real time","volume":"20","author":"Dong","year":"2020","journal-title":"Lancet Infect. Dis."}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/16\/10182\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:10:57Z","timestamp":1760141457000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/16\/10182"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,17]]},"references-count":48,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["ijerph191610182"],"URL":"https:\/\/doi.org\/10.3390\/ijerph191610182","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,17]]}}}