{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T16:40:50Z","timestamp":1791909650709,"version":"4.3.4"},"reference-count":32,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T00:00:00Z","timestamp":1778457600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate and compare the performance of four publicly available large language models\u2014ChatGPT, Gemini, Copilot, and Grok\u2014in answering medical questions related to Multiple Sclerosis, focusing on accuracy, transparency, and clinical actionability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Four publicly available large language models (ChatGPT, Gemini, Grok, and Copilot) were selected based on accessibility and their ability to respond to medical questions. A total of 25 questions\u2014five for each of the five key domains (diagnosis, treatment, prevention, disease control, and disease management)\u2014were developed. The responses generated by the models were evaluated using the DISCERN-AI and NLAT-AI assessment tools.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The evaluation of four AI chatbots\u2014ChatGPT, Gemini, Copilot, and Grok\u2014on multiple sclerosis (MS) content revealed clear differences in quality and consistency. According to DISCERN-AI criteria, Gemini achieved the highest overall quality, excelling in relevance, transparency, balance, and acknowledgment of uncertainty. Grok ranked second, showing generally balanced results with slightly lower scores than Gemini. ChatGPT exhibited strong yet uneven performance, with particular weaknesses in content addressing vulnerable populations. Copilot demonstrated the weakest overall performance, with consistently lower scores across nearly all criteria.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Gemini demonstrated the strongest and most consistent performance across all domains, followed by Grok with slightly lower but balanced results. ChatGPT showed strong yet uneven outcomes, with weaknesses in addressing vulnerable populations. Copilot ranked lowest, consistently underperforming across metrics. These findings highlight significant differences among large language models in generating accurate and clinically relevant responses for multiple sclerosis, underscoring the importance of considering each model\u2019s strengths and limitations in healthcare applications.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0346445","type":"journal-article","created":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T17:38:15Z","timestamp":1778521095000},"page":"e0346445","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Evaluation of the performance of large language models in responding to medical questions related to multiple sclerosis: A case study of large language models including ChatGPT, Gemini, Grok and Copilot"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5631-539X","authenticated-orcid":true,"given":"Meisam","family":"Dastani","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad Shayan","family":"Sajjadi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bassem","family":"Yamout","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Melika","family":"Arab Bafrani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8929-9923","authenticated-orcid":true,"given":"Amirreza","family":"Nasirzadeh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,5,11]]},"reference":[{"issue":"3","key":"pone.0346445.ref001","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3641289","article-title":"A Survey on Evaluation of Large Language Models","volume":"15","author":"Y Chang","year":"2024","journal-title":"ACM Trans Intell Syst Technol"},{"issue":"8","key":"pone.0346445.ref002","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"AJ Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"pone.0346445.ref003","article-title":"Cognitive Function in Multiple Sclerosis Patients Based on Age, Gender, and Education Level","author":"E Hassanshahi","year":"2020","journal-title":"ACTA"},{"issue":"9","key":"pone.0346445.ref004","doi-asserted-by":"crossref","first-page":"992","DOI":"10.1016\/j.ajic.2024.03.016","article-title":"Utilizing natural language processing and large language models in the diagnosis and prediction of infectious diseases: A systematic review","volume":"52","author":"M Omar","year":"2024","journal-title":"Am J Infect Control"},{"issue":"5","key":"pone.0346445.ref005","doi-asserted-by":"crossref","first-page":"1570","DOI":"10.1002\/jvc2.459","article-title":"Usefulness of the large language model ChatGPT (GPT\u20104) as a diagnostic tool and information source in dermatology","volume":"3","author":"JPS Nielsen","year":"2024","journal-title":"JEADV Clinical Practice"},{"issue":"1","key":"pone.0346445.ref006","doi-asserted-by":"crossref","first-page":"17118","DOI":"10.1038\/s41598-024-67429-4","article-title":"A chatbot based question and answer system for the auxiliary diagnosis of chronic diseases based on large language model","volume":"14","author":"S Zhang","year":"2024","journal-title":"Sci Rep"},{"issue":"2","key":"pone.0346445.ref007","doi-asserted-by":"crossref","first-page":"633","DOI":"10.14245\/ns.2448098.049","article-title":"Analyzing Large Language Models\u2019 Responses to Common Lumbar Spine Fusion Surgery Questions: A Comparison Between ChatGPT and Bard","volume":"21","author":"SP Lang","year":"2024","journal-title":"Neurospine"},{"issue":"11","key":"pone.0346445.ref008","article-title":"ChatGPT in Answering Queries Related to Lifestyle-Related Diseases and Disorders","volume":"15","author":"H Mondal","year":"2023","journal-title":"Cureus"},{"issue":"3","key":"pone.0346445.ref009","doi-asserted-by":"crossref","first-page":"14604582251381269","DOI":"10.1177\/14604582251381269","article-title":"Performance of artificial intelligence large language models (Copilot and Gemini) compared to human experts in healthcare policy making: A mixed-methods cross-sectional study","volume":"31","author":"M Khosravi","year":"2025","journal-title":"Health Informatics J"},{"issue":"7","key":"pone.0346445.ref010","doi-asserted-by":"crossref","first-page":"219","DOI":"10.3390\/fi16070219","article-title":"A Systematic Review and Comprehensive Analysis of Pioneering AI Chatbot Models from Education to Healthcare: ChatGPT, Bard, Llama, Ernie and Grok","volume":"16","author":"K Wangsa","year":"2024","journal-title":"Future Internet"},{"issue":"6","key":"pone.0346445.ref011","doi-asserted-by":"crossref","first-page":"1790","DOI":"10.1007\/s11695-023-06603-5","article-title":"Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery","volume":"33","author":"JS Samaan","year":"2023","journal-title":"Obes Surg"},{"issue":"2","key":"pone.0346445.ref012","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1007\/s41666-023-00155-0","article-title":"Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models","volume":"8","author":"C Shyr","year":"2024","journal-title":"J Healthc Inform Res"},{"issue":"3","key":"pone.0346445.ref013","doi-asserted-by":"crossref","DOI":"10.1111\/bpa.13207","article-title":"Evaluating the performance of large language models: ChatGPT and Google Bard in generating differential diagnoses in clinicopathological conferences of neurodegenerative disorders","volume":"34","author":"S Koga","year":"2024","journal-title":"Brain Pathol"},{"issue":"2","key":"pone.0346445.ref014","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s00296-023-05464-6","article-title":"Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4","volume":"44","author":"M Krusche","year":"2024","journal-title":"Rheumatol Int"},{"issue":"7","key":"pone.0346445.ref015","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1111\/jerd.13046","article-title":"Implications of large language models such as ChatGPT for dental medicine","volume":"35","author":"F Eggmann","year":"2023","journal-title":"J Esthet Restor Dent"},{"issue":"8","key":"pone.0346445.ref016","article-title":"Large Language Models in Hematology Case Solving: A Comparative Study of ChatGPT-3.5, Google Bard, and Microsoft Bing","volume":"15","author":"A Kumari","year":"2023","journal-title":"Cureus"},{"key":"pone.0346445.ref017","doi-asserted-by":"crossref","DOI":"10.2196\/55939","article-title":"Evaluating the Efficacy of ChatGPT as a Patient Education Tool in Prostate Cancer: Multimetric Assessment","volume":"26","author":"D Gibson","year":"2024","journal-title":"J Med Internet Res"},{"issue":"1","key":"pone.0346445.ref018","doi-asserted-by":"crossref","first-page":"18004","DOI":"10.1038\/s41598-025-03074-9","article-title":"Large language models\u2019 capabilities in responding to tuberculosis medical questions: testing ChatGPT, Gemini, and Copilot","volume":"15","author":"M Dastani","year":"2025","journal-title":"Sci Rep"},{"issue":"2","key":"pone.0346445.ref019","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1017\/jrc.2020.17","article-title":"Information and resources important to the quality of life of people living with multiple sclerosis","volume":"26","author":"M Bishop","year":"2020","journal-title":"Aust J Rehabil Couns"},{"issue":"4","key":"pone.0346445.ref020","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1111\/hex.12482","article-title":"Multiple sclerosis patients need and want information on exercise promotion from healthcare providers: a qualitative study","volume":"20","author":"YC Learmonth","year":"2017","journal-title":"Health Expect"},{"key":"pone.0346445.ref021","doi-asserted-by":"crossref","first-page":"51","DOI":"10.2147\/PPA.S344690","article-title":"Information-Seeking Strategies of People with Multiple Sclerosis in Spain: The INFOSEEK-MS Study","volume":"16","author":"Y Higueras","year":"2022","journal-title":"Patient Prefer Adherence"},{"key":"pone.0346445.ref022","doi-asserted-by":"crossref","DOI":"10.2196\/50638","article-title":"Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial","volume":"25","author":"B Mesk\u00f3","year":"2023","journal-title":"J Med Internet Res"},{"key":"pone.0346445.ref023","doi-asserted-by":"crossref","first-page":"1373843","DOI":"10.3389\/fsurg.2024.1373843","article-title":"Evaluating ChatGPT-4\u2019s performance as a digital health advisor for otosclerosis surgery","volume":"11","author":"S Sahin","year":"2024","journal-title":"Front Surg"},{"issue":"1","key":"pone.0346445.ref024","article-title":"ChatGPT as a patient education tool in colorectal cancer-An in-depth assessment of efficacy, quality and readability","volume":"27","author":"AHY Siu","year":"2025","journal-title":"Colorectal Dis"},{"issue":"4","key":"pone.0346445.ref025","article-title":"Evaluating the accuracy and reliability of large language models (ChatGPT, Claude, DeepSeek, Gemini, Grok, and Le Chat) in answering item-analyzed multiple-choice questions on blood physiology","volume":"17","author":"M Agarwal","year":"2025","journal-title":"Cureus"},{"issue":"1","key":"pone.0346445.ref026","doi-asserted-by":"crossref","first-page":"1712","DOI":"10.1038\/s41598-024-83575-1","article-title":"Comparison of the performances between ChatGPT and Gemini in answering questions on viral hepatitis","volume":"15","author":"M Sahin Ozdemir","year":"2025","journal-title":"Sci Rep"},{"issue":"4","key":"pone.0346445.ref027","article-title":"Accuracy of ChatGPT in Neurolocalization","volume":"16","author":"WF Dabbas","year":"2024","journal-title":"Cureus"},{"issue":"1","key":"pone.0346445.ref028","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1186\/s12245-025-00949-6","article-title":"Comparative performance of ChatGPT, Gemini, and final-year emergency medicine clerkship students in answering multiple-choice questions: implications for the use of AI in medical education","volume":"18","author":"SN Al-Thani","year":"2025","journal-title":"Int J Emerg Med"},{"issue":"2","key":"pone.0346445.ref029","article-title":"Assessment of ChatGPT\u2019s performance on neurology written board examination questions","volume":"5","author":"TC Chen","year":"2023","journal-title":"BMJ Neurol Open"},{"key":"pone.0346445.ref030","doi-asserted-by":"crossref","first-page":"1495378","DOI":"10.3389\/fmed.2025.1495378","article-title":"Artificial intelligence in healthcare education: evaluating the accuracy of ChatGPT, Copilot, and Google Gemini in cardiovascular pharmacology","volume":"12","author":"IM Salman","year":"2025","journal-title":"Front Med (Lausanne)"},{"key":"pone.0346445.ref031","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2024.clinicalnlp-1.3","article-title":"Gemini goes to med school: exploring the capabilities of multimodal large language models on medical challenge problems & hallucinations","author":"A Pal","year":"2024","journal-title":"Proceedings of the 6th Clinical Natural Language Processing Workshop"},{"key":"pone.0346445.ref032","doi-asserted-by":"crossref","first-page":"1392","DOI":"10.1177\/13524585241277376","article-title":"Integrating large language models in care, research, and education in multiple sclerosis management","volume":"30","author":"H Inojosa","year":"2024","journal-title":"Mult Scler"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0346445","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T17:38:20Z","timestamp":1778521100000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0346445"}},"subtitle":[],"editor":[{"given":"Stephen R.","family":"Milford","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,5,11]]},"references-count":32,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,5,11]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0346445","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,11]]}}}