{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T23:31:03Z","timestamp":1776209463274,"version":"3.50.1"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2025,2,25]],"date-time":"2025-02-25T00:00:00Z","timestamp":1740441600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,5,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>This study assesses the abilities of 2 large language models (LLMs), GPT-4 and BioMistral 7B, in responding to patient queries, particularly concerning rare diseases, and compares their performance with that of physicians.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>A total of 103 patient queries and corresponding physician answers were extracted from EXABO, a question-answering forum dedicated to rare respiratory diseases. The responses provided by physicians and generated by LLMs were ranked on a Likert scale by a panel of 4 experts based on 4 key quality criteria for health communication: correctness, comprehensibility, relevance, and empathy.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The performance of generative pretrained transformer 4 (GPT-4) was significantly better than the performance of the physicians and BioMistral 7B. While the overall ranking considers GPT-4\u2019s responses to be mostly correct, comprehensive, relevant, and emphatic, the responses provided by BioMistral 7B were only partially correct and empathetic. The responses given by physicians rank in between. The experts concur that an LLM could lighten the load for physicians, rigorous validation is considered essential to guarantee dependability and efficacy.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Open-source models such as BioMistral 7B offer the advantage of privacy by running locally in health-care settings. GPT-4, on the other hand, demonstrates proficiency in communication and knowledge depth. However, challenges persist, including the management of response variability, the balancing of comprehensibility with medical accuracy, and the assurance of consistent performance across different languages.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>The performance of GPT-4 underscores the potential of LLMs in facilitating physician-patient communication. However, it is imperative that these systems are handled with care, as erroneous responses have the potential to cause harm without the requisite validation procedures.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf034","type":"journal-article","created":{"date-parts":[[2025,2,25]],"date-time":"2025-02-25T17:17:47Z","timestamp":1740503867000},"page":"775-783","source":"Crossref","is-referenced-by-count":6,"title":["MedBot vs RealDoc: efficacy of large language modeling in physician-patient communication for rare diseases"],"prefix":"10.1093","volume":"32","author":[{"given":"Magdalena T","family":"Weber","sequence":"first","affiliation":[{"name":"Institute of Medical Informatics, University Medicine Frankfurt, Goethe University Frankfurt , Frankfurt 60590,","place":["Germany"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5240-8484","authenticated-orcid":false,"given":"Richard","family":"Noll","sequence":"additional","affiliation":[{"name":"Institute of Medical Informatics, University Medicine Frankfurt, Goethe University Frankfurt , Frankfurt 60590,","place":["Germany"]}]},{"given":"Alexandra","family":"Marchl","sequence":"additional","affiliation":[{"name":"Institute of Medical Informatics, University Medicine Frankfurt, Goethe University Frankfurt , Frankfurt 60590,","place":["Germany"]}]},{"given":"Carlo","family":"Facchinello","sequence":"additional","affiliation":[{"name":"Santagostino Medical Center , Bologna 40138,","place":["Italy"]}]},{"given":"Achim","family":"Gr\u00fcnewaldt","sequence":"additional","affiliation":[{"name":"Department of Respiratory Medicine and Allergology, University Medicine Frankfurt, Goethe University Frankfurt , Frankfurt 60590,","place":["Germany"]}]},{"given":"Christian","family":"H\u00fcgel","sequence":"additional","affiliation":[{"name":"HELIOS Dr Horst Schmidt Kliniken Wiesbaden, Klinik f\u00fcr Pneumologie , Wiesbaden 65199,","place":["Germany"]}]},{"given":"Khader","family":"Musleh","sequence":"additional","affiliation":[{"name":"Department of Respiratory Medicine and Allergology, University Medicine Frankfurt, Goethe University Frankfurt , Frankfurt 60590,","place":["Germany"]}]},{"given":"Thomas O F","family":"Wagner","sequence":"additional","affiliation":[{"name":"European Reference Network for Rare Respiratory Diseases (ERN-LUNG), University Medicine Frankfurt , Frankfurt 60590,","place":["Germany"]}]},{"given":"Holger","family":"Storf","sequence":"additional","affiliation":[{"name":"Institute of Medical Informatics, University Medicine Frankfurt, Goethe University Frankfurt , Frankfurt 60590,","place":["Germany"]}]},{"given":"Jannik","family":"Schaaf","sequence":"additional","affiliation":[{"name":"Institute of Medical Informatics, University Medicine Frankfurt, Goethe University Frankfurt , Frankfurt 60590,","place":["Germany"]}]}],"member":"286","published-online":{"date-parts":[[2025,2,25]]},"reference":[{"key":"2025042203522377700_ocaf034-B1","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2025042203522377700_ocaf034-B2","author":"Achiam","year":"2024"},{"key":"2025042203522377700_ocaf034-B3","doi-asserted-by":"publisher","first-page":"e44658","DOI":"10.7759\/cureus.44658","article-title":"Reimagining healthcare: unleashing the power of artificial intelligence in medicine","volume":"15","author":"Iqbal","year":"2023","journal-title":"Cureus"},{"key":"2025042203522377700_ocaf034-B4","doi-asserted-by":"publisher","first-page":"102642","DOI":"10.1016\/j.ijinfomgt.2023.102642","article-title":"Opinion paper: \u201cSo what if ChatGPT wrote it?\u201d Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy","volume":"71","author":"Dwivedi","year":"2023","journal-title":"Int J Inf Manag"},{"key":"2025042203522377700_ocaf034-B5","doi-asserted-by":"publisher","first-page":"109713","DOI":"10.1016\/j.isci.2024.109713","article-title":"The application of large language models in medicine: a scoping review","volume":"27","author":"Meng","year":"2024","journal-title":"iScience"},{"key":"2025042203522377700_ocaf034-B6","doi-asserted-by":"publisher","first-page":"ooad079","DOI":"10.1093\/jamiaopen\/ooad079","article-title":"How can artificial intelligence decrease cognitive and work burden for front line practitioners?","volume":"6","author":"Gandhi","year":"2023","journal-title":"JAMIA Open"},{"key":"2025042203522377700_ocaf034-B7","author":"Wen"},{"key":"2025042203522377700_ocaf034-B8","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1186\/s13000-024-01464-7","article-title":"Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic medicine with a focus on digital pathology\u2014a recent scoping review","volume":"19","author":"Ullah","year":"2024","journal-title":"Diagn Pathol"},{"key":"2025042203522377700_ocaf034-B9","doi-asserted-by":"publisher","first-page":"e2336997","DOI":"10.1001\/jamanetworkopen.2023.36997","article-title":"Large language model-based chatbot vs surgeon-generated informed consent documentation for common procedures","volume":"6","author":"Decker","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2025042203522377700_ocaf034-B10","doi-asserted-by":"publisher","first-page":"104770","DOI":"10.1016\/j.ebiom.2023.104770","article-title":"Benchmarking large language models\u2019 performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard","volume":"95","author":"Lim","year":"2023","journal-title":"eBioMedicine"},{"key":"2025042203522377700_ocaf034-B11","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"2025042203522377700_ocaf034-B12","doi-asserted-by":"publisher","first-page":"e2330320","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions","volume":"6","author":"Bernstein","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2025042203522377700_ocaf034-B13","doi-asserted-by":"publisher","first-page":"e54706","DOI":"10.2196\/54706","article-title":"Physician versus large language model chatbot responses to web-based questions from autistic patients in Chinese: cross-sectional comparative analysis","volume":"26","author":"He","year":"2024","journal-title":"J Med Internet Res"},{"key":"2025042203522377700_ocaf034-B14","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1186\/s13023-021-01769-6","article-title":"Diagnostic precision and identification of rare diseases is dependent on distance of residence relative to tertiary medical facilities","volume":"16","author":"Walter","year":"2021","journal-title":"Orphanet J Rare Dis"},{"key":"2025042203522377700_ocaf034-B15","doi-asserted-by":"publisher","first-page":"4732","DOI":"10.3390\/ijerph20064732","article-title":"Enhancing equitable access to rare disease diagnosis and treatment around the world: a review of evidence, policies, and challenges","volume":"20","author":"Adachi","year":"2023","journal-title":"Int J Environ Res Public Health"},{"key":"2025042203522377700_ocaf034-B16","author":"Exabo"},{"key":"2025042203522377700_ocaf034-B17","first-page":"236","article-title":"Conception of an expert advisory board for the European Reference Network for rare respiratory diseases","volume":"247","author":"Walther","year":"2018","journal-title":"Stud Health Technol Inform"},{"key":"2025042203522377700_ocaf034-B18"},{"key":"2025042203522377700_ocaf034-B19","first-page":"101","author":"Weber","year":"2024"},{"key":"2025042203522377700_ocaf034-B20","author":"Labrak","year":"2024"},{"key":"2025042203522377700_ocaf034-B21","year":"2024"},{"key":"2025042203522377700_ocaf034-B22","author":"Orphanet"},{"key":"2025042203522377700_ocaf034-B23","author":"Hu","year":"2021"},{"key":"2025042203522377700_ocaf034-B24","year":"2024"},{"key":"2025042203522377700_ocaf034-B25","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1136\/bmj.311.6997.109","article-title":"Rigour and qualitative research","volume":"311","author":"Mays","year":"1995","journal-title":"BMJ"},{"key":"2025042203522377700_ocaf034-B26","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1002\/ece3.926","article-title":"What is an expert? A systems perspective on expertise","volume":"4","author":"Caley","year":"2014","journal-title":"Ecol Evol"},{"key":"2025042203522377700_ocaf034-B27"},{"key":"2025042203522377700_ocaf034-B28","doi-asserted-by":"publisher","first-page":"396","DOI":"10.9734\/BJAST\/2015\/14975","article-title":"Likert scale: explored and explained","volume":"7","author":"Joshi","year":"2015","journal-title":"Br J Appl Sci Technol"},{"key":"2025042203522377700_ocaf034-B29","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1111\/cgf.14521","article-title":"Effective use of Likert scales in visualization evaluations: a systematic review","volume":"41","author":"South","year":"2022","journal-title":"Comput Graph Forum"},{"key":"2025042203522377700_ocaf034-B30","author":"Kruskal","journal-title":"J Am Stat Assoc"},{"key":"2025042203522377700_ocaf034-B31","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"J Chiropr Med"},{"key":"2025042203522377700_ocaf034-B32"},{"key":"2025042203522377700_ocaf034-B33"},{"key":"2025042203522377700_ocaf034-B34"},{"key":"2025042203522377700_ocaf034-B35"},{"key":"2025042203522377700_ocaf034-B36","doi-asserted-by":"publisher","first-page":"886","DOI":"10.1186\/s12913-023-09869-8","article-title":"Quality communication can improve patient-centred health outcomes among older patients: a rapid review","volume":"23","author":"Sharkiya","year":"2023","journal-title":"BMC Health Serv Res"},{"key":"2025042203522377700_ocaf034-B37","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1016\/j.zefq.2020.07.007","article-title":"Communicating test results in a comprehensible manner: a randomized controlled trial of word usage in doctor-patient communication","volume":"156-157","author":"Auschra","year":"2020","journal-title":"Z Evid Fortbild Qual Gesundhwes"},{"key":"2025042203522377700_ocaf034-B38","doi-asserted-by":"publisher","first-page":"385","DOI":"10.4300\/JGME-D-13-00072.1","article-title":"\u201cBest practice\u201d for patient-centered communication: a narrative review","volume":"5","author":"King","year":"2013","journal-title":"J Grad Med Educ"},{"key":"2025042203522377700_ocaf034-B39","doi-asserted-by":"publisher","first-page":"26","DOI":"10.3390\/healthcare8010026","article-title":"The role of empathy in health and social care professionals","volume":"8","author":"Moudatsou","year":"2020","journal-title":"Healthcare (Basel)"},{"key":"2025042203522377700_ocaf034-B40","doi-asserted-by":"publisher","first-page":"571","DOI":"10.1016\/j.icte.2023.02.007","article-title":"Digitization of healthcare sector: a study on privacy and security concerns","volume":"9","author":"Paul","year":"2023","journal-title":"ICT Express"},{"key":"2025042203522377700_ocaf034-B41","author":"Minaee","year":"2024"},{"key":"2025042203522377700_ocaf034-B42"},{"key":"2025042203522377700_ocaf034-B43","author":"Williams","year":"2024"},{"key":"2025042203522377700_ocaf034-B44","doi-asserted-by":"publisher","first-page":"1495582","DOI":"10.3389\/fmed.2024.1495582","article-title":"Why we need to be careful with LLMs in medicine","volume":"11","author":"B\u00e9lisle-Pipon","year":"2024","journal-title":"Front Med (Lausanne)"},{"key":"2025042203522377700_ocaf034-B45","doi-asserted-by":"publisher","first-page":"e39089","DOI":"10.2196\/39089","article-title":"Online physician\u2013patient interaction and patient satisfaction: empirical study of the internet hospital service","volume":"25","author":"Wu","year":"2023","journal-title":"J Med Internet Res"},{"key":"2025042203522377700_ocaf034-B46","author":"Lewis"},{"key":"2025042203522377700_ocaf034-B47","author":"Gao","year":"2024"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/5\/775\/62167464\/ocaf034.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/5\/775\/62167464\/ocaf034.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,22]],"date-time":"2025-04-22T07:52:31Z","timestamp":1745308351000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/5\/775\/8042190"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,25]]},"references-count":47,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2025,2,25]]},"published-print":{"date-parts":[[2025,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf034","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,5]]},"published":{"date-parts":[[2025,2,25]]}}}