{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T11:08:34Z","timestamp":1788952114516,"version":"build-2803163510"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T00:00:00Z","timestamp":1788912000000},"content-version":"vor","delay-in-days":40,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005416","name":"Norwegian Research Council","doi-asserted-by":"publisher","award":["331770"],"award-info":[{"award-number":["331770"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Automated sentiment analysis can be used to analyse free-text patient experience comments, but in health services research, it requires technical performance and methodological validity. Traditional models lack contextual understanding, whereas masked language models (MLMs) enable nuanced sentiment classification. This study is the first to fine-tune an MLM (NorBERT3large) on human-annotated data and evaluate the validity of four-category sentiment classifications across survey and online feedback in general practice.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective(s)<\/jats:title>\n                    <jats:p>To evaluate a fine-tuned MLM against human-annotated sentiment labels, compare its performance with traditional methods and a large language model, and assess convergent and known-groups validity using survey and online comments.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>NorBERT3large was fine-tuned on human-annotated free-text survey comments. Performance was evaluated using F1-scores. The model classified unseen online data. Construct validity was assessed per COSMIN guidelines through correlations with patient experience scores and subgroup analyses.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The fine-tuned NorBERT3large achieved high F1-scores on held-out survey test data and outperformed traditional models used in prior patient experience research, as well as a zero-shot large language model reference. On unseen online reviews, performance was high in the random sample and strongest for mixed and positive sentiment in the balanced evaluation set, while the rare neutral comments were difficult to classify reliably. Construct validity was supported by expected correlations with patient experience scores and known subgroup differences.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Beyond good classification performance, the findings support automated sentiment analysis as a valid patient experience measure in general practice. The sentiment classifications may complement traditional patient experience measures and help identify areas for quality improvement.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/fampra\/cmag071","type":"journal-article","created":{"date-parts":[[2026,8,22]],"date-time":"2026-08-22T11:41:35Z","timestamp":1787398895000},"source":"Crossref","is-referenced-by-count":0,"title":["Exploiting survey and online patient experience comments in general practice: validating a fine-tuned language model for automatic sentiment analysis"],"prefix":"10.1093","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7503-2048","authenticated-orcid":false,"given":"Rebecka Maria","family":"Norman","sequence":"first","affiliation":[{"name":"Division for Health Services, Norwegian Institute of Public Health , N-0213 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lilja Charlotte","family":"Storset","sequence":"additional","affiliation":[{"name":"Department of Informatics, Section for Machine Learning, Language Technology Group, University of Oslo , N-0316 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Petter","family":"M\u00e6hlum","sequence":"additional","affiliation":[{"name":"Department of Informatics, Section for Machine Learning, Language Technology Group, University of Oslo , N-0316 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6191-416X","authenticated-orcid":false,"given":"Hilde Hestad","family":"Iversen","sequence":"additional","affiliation":[{"name":"Division for Health Services, Norwegian Institute of Public Health , N-0213 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elma","family":"Jelin","sequence":"additional","affiliation":[{"name":"Division for Health Services, Norwegian Institute of Public Health , N-0213 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erik","family":"Velldal","sequence":"additional","affiliation":[{"name":"Department of Informatics, Section for Machine Learning, Language Technology Group, University of Oslo , N-0316 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lilja","family":"\u00d8vrelid","sequence":"additional","affiliation":[{"name":"Department of Informatics, Section for Machine Learning, Language Technology Group, University of Oslo , N-0316 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4261-5455","authenticated-orcid":false,"given":"\u00d8yvind","family":"Bjertn\u00e6s","sequence":"additional","affiliation":[{"name":"Division for Health Services, Norwegian Institute of Public Health , N-0213 Oslo ,","place":["Norway"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,9,9]]},"reference":[{"key":"2026090905504356500_cmag071-B1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.g6034","article-title":"Understanding high and low patient experience scores in primary care: analysis of patients\u2019 survey data for general practices and individual doctors","volume":"349","author":"Roberts","year":"2014","journal-title":"BMJ"},{"key":"2026090905504356500_cmag071-B2","doi-asserted-by":"publisher","first-page":"S28","DOI":"10.1097\/MLR.0b013e31826cbc0d","article-title":"Psychometric properties of the Consumer Assessment of Healthcare Providers and Systems (CAHPS\u00ae) clinician and group adult visit survey","volume":"50 Suppl","author":"Dyer","year":"2012","journal-title":"Med Care"},{"key":"2026090905504356500_cmag071-B3","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2020-100262","article-title":"Applying natural language processing and machine learning techniques to patient experience feedback: a systematic review","volume":"28","author":"Khanbhai","year":"2021","journal-title":"BMJ Health Care Inform"},{"key":"2026090905504356500_cmag071-B4","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1111\/1468-0009.12374","article-title":"What words convey: the potential for patient narratives to inform quality improvement","volume":"97","author":"Grob","year":"2019","journal-title":"Milbank Q"},{"key":"2026090905504356500_cmag071-B5","first-page":"8","volume-title":"It's Difficult to Be Neutral \u2013 Human and LLM-Based Sentiment Annotation of Patient Comments. In: Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC-COLING 2024.","author":"M\u00e6hlum","year":"2024"},{"key":"2026090905504356500_cmag071-B6","author":"Liu","year":"2015"},{"key":"2026090905504356500_cmag071-B7","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2025-101631","article-title":"From words to action? A scoping review on automatic sentiment analysis of patient experience comments from online sources and surveys","volume":"32","author":"Jelin","year":"2025","journal-title":"BMJ Health Care Inform"},{"key":"2026090905504356500_cmag071-B8","doi-asserted-by":"publisher","author":"Devlin","DOI":"10.18653\/v1\/N19-1423"},{"key":"2026090905504356500_cmag071-B9","author":"Vaswani"},{"key":"2026090905504356500_cmag071-B10","doi-asserted-by":"publisher","first-page":"6179","DOI":"10.1038\/s41598-025-89996-w","article-title":"Analyzing patient perspectives with large language models: a cross-sectional study of sentiment and thematic classification on exception from informed consent","volume":"15","author":"Kornblith","year":"2025","journal-title":"Sci Rep"},{"key":"2026090905504356500_cmag071-B11","doi-asserted-by":"publisher","first-page":"167","DOI":"10.3390\/bdcc8120167","article-title":"Aspect-based sentiment analysis of patient feedback using large language models","volume":"8","author":"Alkhnbashi","year":"2024","journal-title":"Big Data Cogn Comput"},{"key":"2026090905504356500_cmag071-B12","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1186\/s12911-022-01923-5","article-title":"Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)","volume":"22","author":"van Buchem","year":"2022","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026090905504356500_cmag071-B13","doi-asserted-by":"publisher","first-page":"153","DOI":"10.12700\/APH.20.7.2023.7.9","article-title":"Analyzing narratives of patient experiences: a BERT topic modeling approach","volume":"20","author":"Osv\u00e1th","year":"2023","journal-title":"Acta Polytech Hung"},{"key":"2026090905504356500_cmag071-B14","author":"Norman"},{"key":"2026090905504356500_cmag071-B15","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1186\/s12875-024-02495-1","article-title":"Multimorbidity and patient experience with general practice: a national cross-sectional survey in Norway","volume":"25","author":"Norman","year":"2024","journal-title":"BMC Prim Care"},{"key":"2026090905504356500_cmag071-B16","doi-asserted-by":"publisher","DOI":"10.2196\/38932","article-title":"Web-based public ratings of general practitioners in Norway: validation study","volume":"7","author":"Bjertn\u00e6s","year":"2023","journal-title":"JMIR Form Res"},{"key":"2026090905504356500_cmag071-B17","doi-asserted-by":"publisher","first-page":"897","DOI":"10.1186\/s12913-022-08307-5","article-title":"Reliability and validity of the psychiatric inpatient patient experience questionnaire \u2013 continuous electronic measurement (PIPEQ-CEM)","volume":"22","author":"Iversen","year":"2022","journal-title":"BMC Health Serv Res"},{"key":"2026090905504356500_cmag071-B18","first-page":"5025","volume-title":"Proceedings of the Twelfth Language Resources and Evaluation Conference","author":"\u00d8vrelid","year":"2020"},{"key":"2026090905504356500_cmag071-B19","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem Med (Zagreb)"},{"key":"2026090905504356500_cmag071-B20","first-page":"618","volume-title":"Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)","author":"Samuel","year":"2023"},{"key":"2026090905504356500_cmag071-B21","volume-title":"Exploiting Patient Comments","author":"LTG Oslo","year":"."},{"key":"2026090905504356500_cmag071-B22","volume-title":"The Truth of the F-Measure","author":"Sasaki","year":"2007"},{"key":"2026090905504356500_cmag071-B23","volume-title":"NorMistral-11B-thinking. Hugging Face","author":"norallm"},{"key":"2026090905504356500_cmag071-B24","volume-title":"COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs): user manual. Version 1.0","author":"Mokkink","year":"2018"},{"key":"2026090905504356500_cmag071-B25","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2023.1174154","article-title":"A sentiment analysis on online psychiatrist reviews to identify clinical attributes of psychiatrists that shape the therapeutic alliance","volume":"14","author":"Park","year":"2023","journal-title":"Front Psychiatry"},{"key":"2026090905504356500_cmag071-B26","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1007\/s43390-021-00419-y","article-title":"What are patients saying about you online? A sentiment analysis of online written reviews on scoliosis research society surgeons","volume":"10","author":"Tang","year":"2022","journal-title":"Spine Deform"},{"key":"2026090905504356500_cmag071-B27","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1017\/S1478951523001001","article-title":"Development of a model and method for hospice quality assessment from natural language processing (NLP) analysis of online caregiver reviews","volume":"22","author":"Hotchkiss","year":"2024","journal-title":"Palliat Support Care"},{"key":"2026090905504356500_cmag071-B28","doi-asserted-by":"publisher","DOI":"10.2196\/14455","article-title":"Gender, soft skills, and patient experience in online physician reviews: a large-scale text analysis","volume":"22","author":"Dunivin","year":"2020","journal-title":"J Med Internet Res"},{"key":"2026090905504356500_cmag071-B29","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1136\/bmjqs-2019-009456","article-title":"The value of Facebook in nation-wide hospital quality assessment: a national mixed-methods study in Norway","volume":"29","author":"Bjertnaes","year":"2020","journal-title":"BMJ Qual Saf"},{"key":"2026090905504356500_cmag071-B30","volume-title":"Den norske fastlegetjenesten 2023\u20132024: En forskningsbasert baselinem\u00e5ling av pasienterfaringer p\u00e5 kommuneniv\u00e5 [The Norwegian General Practitioner Service 2023\u20132024: A Research-Based Baseline Measurement of Patient Experiences at the Municipal Level]. Rapport","author":"Norman","year":"2025"},{"key":"2026090905504356500_cmag071-B31","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2017-021161","article-title":"Continuity of care with doctors-a matter of life and death? A systematic review of continuity of care and mortality","volume":"8","author":"Pereira Gray","year":"2018","journal-title":"BMJ Open"},{"key":"2026090905504356500_cmag071-B32","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1111\/j.1525-1497.2005.40135.x","article-title":"Continuity of care and other determinants of patient satisfaction with primary care","volume":"20","author":"Fan","year":"2005","journal-title":"J Gen Intern Med"},{"key":"2026090905504356500_cmag071-B33","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1186\/s12875-020-01251-5","article-title":"Continuity of care is associated with satisfaction with local health care services","volume":"21","author":"Lautamatti","year":"2020","journal-title":"BMC Fam Pract"},{"key":"2026090905504356500_cmag071-B34","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1177\/1757913916634136","article-title":"Determinants of patient satisfaction: a systematic review","volume":"137","author":"Batbaatar","year":"2017","journal-title":"Perspect Public Health"},{"key":"2026090905504356500_cmag071-B35","doi-asserted-by":"publisher","first-page":"275","DOI":"10.52403\/ijhsr.20231038","article-title":"Enhancing patient experience by automating and transforming free text into actionable consumer insights: a natural language processing (NLP) approach","volume":"13","author":"Vashishtha","year":"2023","journal-title":"Int J Health Sci Res"},{"key":"2026090905504356500_cmag071-B36","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1177\/1355819619844540","article-title":"Online patient feedback: a cross-sectional survey of the attitudes and experiences of United Kingdom health care professionals","volume":"24","author":"Atherton","year":"2019","journal-title":"J Health Serv Res Policy"},{"key":"2026090905504356500_cmag071-B37","doi-asserted-by":"publisher","first-page":"e276","DOI":"10.2196\/jmir.4989","article-title":"General practitioners\u2019 concerns about online patient feedback: findings from a descriptive exploratory qualitative study in England","volume":"17","author":"Patel","year":"2015","journal-title":"J Med Internet Res"},{"key":"2026090905504356500_cmag071-B38"}],"container-title":["Family Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/fampra\/article-pdf\/43\/5\/cmag071\/71005392\/cmag071.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/fampra\/article-pdf\/43\/5\/cmag071\/71005392\/cmag071.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T10:11:31Z","timestamp":1788948691000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/fampra\/article\/doi\/10.1093\/fampra\/cmag071\/8788707"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,31]]},"references-count":38,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,7,31]]}},"URL":"https:\/\/doi.org\/10.1093\/fampra\/cmag071","relation":{},"ISSN":["0263-2136","1460-2229"],"issn-type":[{"value":"0263-2136","type":"print"},{"value":"1460-2229","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,10]]},"published":{"date-parts":[[2026,7,31]]},"article-number":"cmag071"}}