{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T17:01:19Z","timestamp":1787763679521,"version":"build-2784847793"},"reference-count":63,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2025,5,1]],"date-time":"2025-05-01T00:00:00Z","timestamp":1746057600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"},{"start":{"date-parts":[[2025,5,1]],"date-time":"2025-05-01T00:00:00Z","timestamp":1746057600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/501100012456","name":"National Social Science Fund of China","doi-asserted-by":"publisher","award":["No. 23BTQ086"],"award-info":[{"award-number":["No. 23BTQ086"]}],"id":[{"id":"10.13039\/501100012456","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["DIGITAL HEALTH"],"published-print":{"date-parts":[[2025,5]]},"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate the performance of generative AI tools, specifically Ernie Bot and ChatGPT, in supporting online medical consultations in China, focusing on their accuracy, safety, and empathy, and to assess their potential role in addressing the supply-demand gap in the healthcare system.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We collected 233 trigeminal neuralgia consultations from a Chinese medical platform, including patient questions and doctor replies. Each question was input into ChatGPT-3.5 and Ernie Bot with role-specific prompts to generate large language models (LLMs) responses. Four blinded raters\u2014two doctors and two patients\u2014evaluated all responses using DISCERN and a modified PEMAT. Lexical, syntactic, and semantic analyses were conducted, with Spearman correlations assessing links between linguistic features and perceived quality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>While doctors led in reliability, Ernie Bot scored highest overall, especially in empathy and clarity, likely due to stylistic choices rather than true understanding. Despite their fluency, LLMs remain prone to factual errors. Text analysis showed distinct linguistic patterns, with several features significantly correlated with perceived quality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>LLMs demonstrate strengths in perceived empathy and clarity but fall short in clinical accuracy and depth when addressing complex cases. Although they outperform doctors in communication-related aspects, their limitations in high-risk decision-making remain evident. As such, LLMs hold promise as adjunct tools for non-urgent consultations, but further refinement is needed to meet the standards of precise and personalized healthcare delivery.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/20552076251388140","type":"journal-article","created":{"date-parts":[[2025,10,23]],"date-time":"2025-10-23T18:41:31Z","timestamp":1761244891000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":4,"title":["Comparing large language models and human doctors in symptom-driven online medical consultations: A case study on trigeminal neuralgia"],"prefix":"10.1177","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-8719-0397","authenticated-orcid":false,"given":"Liantan","family":"Duan","sequence":"first","affiliation":[{"name":"Shandong University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7148-8275","authenticated-orcid":false,"given":"Zhong","family":"Yao","sequence":"additional","affiliation":[{"name":"Qilu Hospital of Shandong University"},{"name":"Cheeloo College of Medicine, Shandong University, Jinan, China"},{"name":"Shandong University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7327-5403","authenticated-orcid":false,"given":"Xiaoyu","family":"Li","sequence":"additional","affiliation":[{"name":"Shandong University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-3701-0945","authenticated-orcid":false,"given":"Yifei","family":"Wu","sequence":"additional","affiliation":[{"name":"Shandong University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1720-4993","authenticated-orcid":false,"given":"Dongfang","family":"Sheng","sequence":"additional","affiliation":[{"name":"Shandong University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2025,10,23]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph17051520"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-020-4992-2"},{"key":"e_1_3_3_4_2","unstructured":"http:\/\/www.gov.cn\/lianbo\/bumen\/202408\/content_6971248.htm"},{"key":"e_1_3_3_5_2","unstructured":"https:\/\/www.nhsa.gov.cn\/art\/2024\/7\/25\/art_7_13340.html"},{"key":"e_1_3_3_6_2","unstructured":"https:\/\/www3.cnnic.cn\/n4\/2025\/0117\/c88-11229.html"},{"key":"e_1_3_3_7_2","unstructured":"https:\/\/www.iimedia.cn\/c1061\/102861.html"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.2196\/37042"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.nurpra.2020.09.013"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.2196\/12126"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12553-018-0273-z"},{"key":"e_1_3_3_12_2","first-page":"25","article-title":"The rise of artificial intelligence in healthcare applications","volume":"1","author":"Bohr A","year":"2020","unstructured":"Bohr A, Memarzadeh K. The rise of artificial intelligence in healthcare applications. Artif Intell Healthc 2020; 1: 25\u201360.","journal-title":"Artif Intell Healthc"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2023.105173"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02332-5"},{"key":"e_1_3_3_15_2","first-page":"1","article-title":"Evaluating large language models as agents in the clinic","volume":"7","author":"Mehandru N","year":"2024","unstructured":"Mehandru N, Miao BY, Almaraz ER, et al. Evaluating large language models as agents in the clinic. npj Dig Med 2024; 7: 1\u20133.","journal-title":"npj Dig Med"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.clineuro.2025.108986"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2025.1546706"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-025-08866-7"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare12080825"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.2196\/37685"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10439-023-03329-4"},{"key":"e_1_3_3_23_2","first-page":"113","article-title":"The ethics of AI in health care: a mapping review","volume":"260","author":"Morley J","year":"2020","unstructured":"Morley J, Machado CCV, Burr C, et al. The ethics of AI in health care: a mapping review. SSRN Electr J 2020; 260: 113\u2013172.","journal-title":"SSRN Electr J"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1093\/jcr\/ucz013"},{"key":"e_1_3_3_25_2","article-title":"Exploring the ethical challenges of conversational AI in mental health care: scoping review","volume":"12","author":"Meadi MR","year":"2025","unstructured":"Meadi MR, Sillekens T, Metselaar S, et al. Exploring the ethical challenges of conversational AI in mental health care: scoping review. JMIR Ment Health 2025; 12: e60432\u20132.","journal-title":"JMIR Ment Health"},{"key":"e_1_3_3_26_2","first-page":"295","article-title":"Ethical and legal challenges of artificial intelligence-driven healthcare","volume":"1","author":"Gerke S","year":"2020","unstructured":"Gerke S, Minssen T, Cohen G. Ethical and legal challenges of artificial intelligence-driven healthcare. Artif Intell Healthc 2020; 1: 295\u2013336.","journal-title":"Artif Intell Healthc"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1177\/0333102416687280"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41572-024-00523-z"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12875-025-02763-8"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.3389\/fsurg.2023.1310414"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"e_1_3_3_32_2","unstructured":"Nori H King N McKinney SM et al. Capabilities of GPT-4 on Medical Challenge Problems. 2023 Mar 20;"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.2196\/50882"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1097\/NNE.0000000000001566"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1111\/1475-6773.14230"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1080\/08164622.2023.2298812"},{"key":"e_1_3_3_37_2","doi-asserted-by":"crossref","unstructured":"Miaschi A Brunato D Dell\u2019Orletta F et al. What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity. 2021.","DOI":"10.18653\/v1\/2021.deelio-1.5"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1093\/ije\/dyq191"},{"key":"e_1_3_3_39_2","unstructured":"Zhang Y Li Y Cui L et al. Siren\u2019s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv.org. 2023."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203774441"},{"key":"e_1_3_3_41_2","volume-title":"Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020;Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Bender EM","unstructured":"Bender EM, Koller A. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020;Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics."},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.54.101601.145041"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0073791"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.2196\/54706"},{"key":"e_1_3_3_45_2","first-page":"2627","volume-title":"Better to ask in English: cross-lingual evaluation of large language models for healthcare queriesProceedings of the ACM Web Conference 2022","author":"Jin Y","unstructured":"Jin Y, Chandra M, Verma G, et al. Better to ask in English: cross-lingual evaluation of large language models for healthcare queries. In: Proceedings of the ACM Web Conference 2022, 2024 May 8, pp.2627\u20132638."},{"key":"e_1_3_3_46_2","unstructured":"Agarwal V Jin Y Chandra M et al. MedHalu: Hallucinations in Responses to Healthcare Queries by Large"},{"key":"e_1_3_3_47_2","unstructured":"Language Models. arXiv (Cornell University). 2024."},{"key":"e_1_3_3_48_2","doi-asserted-by":"crossref","unstructured":"Gilson A Safranek C Huang T et al. How Does ChatGPT Perform on the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. MedRxiv 2022 Dec 26.","DOI":"10.1101\/2022.12.23.22283901"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1002\/art.42737"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asmr.2022.10.004"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1186\/s13018-025-05955-1"},{"key":"e_1_3_3_52_2","first-page":"17","article-title":"Emotional intelligence of large language models","volume":"1","author":"Wang X","year":"2023","unstructured":"Wang X, Li X, Yin Z, et al. Emotional intelligence of large language models. J Pacific Rim Psychol 2023; 1: 17.","journal-title":"J Pacific Rim Psychol"},{"key":"e_1_3_3_53_2","doi-asserted-by":"crossref","unstructured":"Williams CYK Subramanian CR Ali SS et al. Physician- and Large Language Model-Generated Hospital Discharge Summaries: a Blinded Comparative Quality and Safety Study. medRxiv. Cold Spring Harbor Laboratory 2024 Sep 30.","DOI":"10.1101\/2024.09.29.24314562"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-025-03726-3"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jrp.2010.04.001"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-81052-3"},{"key":"e_1_3_3_57_2","unstructured":"Wang X Wang Z Liu J et al. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. arXiv (Cornell University). 2023 Jan 1."},{"key":"e_1_3_3_58_2","article-title":"FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare","volume":"388","author":"Lekadir K","year":"2025","unstructured":"Lekadir K, Frangi AF, Porras AR, et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. Br Med J 2025; 388: e081554.","journal-title":"Br Med J"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.2196\/52597"},{"key":"e_1_3_3_60_2","doi-asserted-by":"crossref","unstructured":"Lee YK Suh J Zhan H et al. Large Language Models Produce Responses Perceived to be Empathic. arXiv (Cornell University). 2024 Mar 26.","DOI":"10.1109\/ACII63134.2024.00012"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2017.7797"},{"key":"e_1_3_3_62_2","article-title":"Neurosurgical interventions for the treatment of classical trigeminal neuralgia","volume":"9","author":"Zakrzewska JM","year":"2011","unstructured":"Zakrzewska JM, Akram H. Neurosurgical interventions for the treatment of classical trigeminal neuralgia. Cochrane Database Syst Rev 2011; 9: CD007312.","journal-title":"Cochrane Database Syst Rev"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1214\/09-SS057"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-018-0272-7"}],"container-title":["DIGITAL HEALTH"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20552076251388140","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/20552076251388140","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20552076251388140","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T17:54:56Z","timestamp":1777398896000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/20552076251388140"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5]]},"references-count":63,"alternative-id":["10.1177\/20552076251388140"],"URL":"https:\/\/doi.org\/10.1177\/20552076251388140","relation":{},"ISSN":["2055-2076","2055-2076"],"issn-type":[{"value":"2055-2076","type":"print"},{"value":"2055-2076","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5]]},"article-number":"20552076251388140"}}