{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T16:34:13Z","timestamp":1786120453405,"version":"build-2736575974"},"reference-count":43,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) are increasingly applied in clinical decision support, yet their diagnostic performance in Chinese-language settings and under realistic clinical workflows remains unclear. In particular, how LLMs perform across diseases with different prevalence and under stepwise diagnostic processes has not been well characterized.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the diagnostic capabilities of LLMs for common diseases and rare diseases using clinical vignettes within a hypothetico-deductive framework and to identify their potential and limitations for clinical diagnosis.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We evaluated 4 Chinese LLMs (Doubao 1.5, DeepSeek-V3, Kimi K1.5, and Leftdoctor GPT 3.5) using 56 clinical cases (28 chronic obstructive pulmonary disease [COPD], and 28 relapsing polychondritis [RP]) sourced from the China Clinical Case Results Database (March 31-April 14, 2025). Patient information was provided incrementally, starting with the initial medical history, followed by physical examination, and laboratory results. Evaluation metrics included top-3 accuracy (RTop3D), top-1 accuracy (RTopD), final diagnostic accuracy (RFA), and mean reciprocal rank (MRR). Statistical analysis was performed using generalized estimating equations (GEE), Friedman tests, and Wilcoxon signed-rank tests with Bonferroni correction. In addition, a qualitative analysis was conducted to characterize recurrent patterns of diagnostic errors.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n LLMs demonstrated significantly higher diagnostic accuracy for COPD compared to RP across all metrics (\n P<\/jats:italic>\n <.001). Diagnostic accuracy improved after additional clinical information was provided, with the improvement mainly observed in RP cases. In RP, diagnostic accuracy increased from 32.14% to 71.43% for DeepSeek and from 35.71% to 78.57% for Doubao, whereas COPD accuracy remained consistently high across all diagnostic stages (82.14%\u201092.86%). For COPD, ranking performance was high and comparable among all models (MRR range: 0.82\u20100.89;\n P<\/jats:italic>\n =.71). In RP, diagnostic performance differed significantly among models (MRR range: 0.10\u20100.39;\n P<\/jats:italic>\n <.001). Qualitative analysis showed that COPD errors were mainly related to a failure to recognize specific features, whereas RP errors involved more diverse patterns, particularly the neglect of negative evidence and the failure to recognize specific features.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Chinese LLMs demonstrated relatively strong diagnostic performance for common diseases such as COPD, but lower and less stable performance for rare diseases such as RP. Additional clinical information improved diagnostic accuracy primarily in RP cases, although differences between models remained evident under diagnostically complex conditions. Error patterns in RP cases suggest that current LLMs remain limited in their ability to integrate complex clinical information and exclusionary findings. Careful evaluation and appropriate clinical oversight remain important for their application in clinical practice.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/89963","type":"journal-article","created":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T20:52:38Z","timestamp":1786049558000},"page":"e89963-e89963","source":"Crossref","is-referenced-by-count":0,"title":["Stepwise Diagnostic Evaluation of Chinese Large Language Models: Comparative Study of Common and Rare Diseases"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-4053-4412","authenticated-orcid":false,"given":"Jiayi","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7560-5212","authenticated-orcid":false,"given":"Jiao","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8147-2736","authenticated-orcid":false,"given":"Rui","family":"Guo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,6]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","DOI":"10.1002\/imm3.70011","article-title":"Artificial intelligence revolutionizes anti\u2010infective drug discovery: from target identification to lead optimization","volume":"1","author":"Li","journal-title":"iMetaMed"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.2196\/59069","article-title":"Revolutionizing health care: the transformative impact of large language models in medicine","volume":"27","author":"Zhang","journal-title":"J Med Internet Res"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/48996","article-title":"Automated paper screening for clinical reviews using large language models: data analysis study","volume":"26","author":"Guo","journal-title":"J Med Internet Res"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.2196\/54706","article-title":"Physician versus large language model chatbot responses to web-based questions from autistic patients in Chinese: cross-sectional comparative analysis","volume":"26","author":"He","journal-title":"J Med Internet Res"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.2196\/48568","article-title":"Utility of ChatGPT in clinical practice","volume":"25","author":"Liu","journal-title":"J Med Internet Res"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2023.1237704","article-title":"Overview of chatbots with special emphasis on artificial intelligence-enabled ChatGPT in medical science","volume":"6","author":"Chakraborty","journal-title":"Front Artif Intell"},{"issue":"2","key":"R7","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1016\/j.annemergmed.2024.01.039","article-title":"Harnessing the power of generative AI for clinical summaries: perspectives from emergency physicians","volume":"84","author":"Barak-Corren","journal-title":"Ann Emerg Med"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2024.115724","article-title":"Psychiatrists\u2019 experiences and opinions of generative artificial intelligence in mental healthcare: an online mixed methods survey","volume":"333","author":"Blease","journal-title":"Psychiatry Res"},{"issue":"2","key":"R9","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1007\/s00296-023-05464-6","article-title":"Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4","volume":"44","author":"Krusche","journal-title":"Rheumatol Int"},{"key":"R10","doi-asserted-by":"crossref","unstructured":"Balogh EP Miller BT Ball JR . Improving Diagnosis in Health Care. National Academies Press (US); 2015. [doi: 10.17226\/21794]","DOI":"10.17226\/21794"},{"issue":"7972","key":"R11","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-46411-8","article-title":"Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks","volume":"15","author":"Sandmann","journal-title":"Nat Commun"},{"key":"R13","unstructured":"Nadeau D Kroutikov M McNeil K Baribeau S . Benchmarking Llama2, mistral, gemma and GPT for factuality, toxicity. arXiv. Preprint posted online on Apr 15, 2024. [doi: 10.48550\/arXiv.2404.09785]"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajem.2024.07.043","article-title":"The accuracy of Gemini, GPT-4, and GPT-4o in ECG analysis: a comparison with cardiologists and emergency medicine specialists","volume":"84","author":"G\u00fcnay","journal-title":"Am J Emerg Med"},{"issue":"11","key":"R15","doi-asserted-by":"publisher","first-page":"1231","DOI":"10.1007\/s11604-024-01619-y","article-title":"Diagnostic performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in \u201cDiagnosis Please\u201d cases","volume":"42","author":"Sonoda","journal-title":"Jpn J Radiol"},{"issue":"6","key":"R16","doi-asserted-by":"publisher","first-page":"2655","DOI":"10.1007\/s00431-024-05527-1","article-title":"Screening\/diagnosis of pediatric endocrine disorders through the artificial intelligence model in different language settings","volume":"183","author":"Ying","journal-title":"Eur J Pediatr"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/63924","article-title":"Performance of artificial intelligence chatbots on ultrasound examinations: cross-sectional comparative analysis","volume":"13","author":"Zhang","journal-title":"JMIR Med Inform"},{"issue":"9","key":"R18","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.45043","article-title":"ChatGPT conquers the Saudi medical licensing exam: exploring the accuracy of artificial intelligence in medical knowledge assessment and implications for modern medical education","volume":"15","author":"Aljindan","journal-title":"Cureus"},{"issue":"4","key":"R19","doi-asserted-by":"publisher","first-page":"745","DOI":"10.1007\/s10439-023-03318-7","article-title":"How does ChatGPT perform on the Italian residency admission national exam compared to 15,869 medical graduates?","volume":"52","author":"Alessandri Bonetti","journal-title":"Ann Biomed Eng"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2023-100815","article-title":"ChatGPT in Iranian medical licensing examination: evaluating the diagnostic accuracy and decision-making capabilities of an AI-based model","volume":"30","author":"Ebrahimian","journal-title":"BMJ Health Care Inform"},{"issue":"12","key":"R21","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000397","article-title":"How does ChatGPT-4 preform on non-English national medical licensing examination? An evaluation in Chinese language","volume":"2","author":"Fang","journal-title":"PLOS Digit Health"},{"issue":"2","key":"R22","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1097\/ACM.0000000000005549","article-title":"Examining ChatGPT performance on USMLE sample items and implications for assessment","volume":"99","author":"Yaneva","journal-title":"Acad Med"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1016\/j.csbj.2024.12.013","article-title":"Evaluating ChatGPT, Gemini and other large language models (LLMs) in orthopaedic diagnostics: a prospective clinical study","volume":"28","author":"Pagano","journal-title":"Comput Struct Biotechnol J"},{"issue":"10","key":"R24","doi-asserted-by":"publisher","first-page":"2886","DOI":"10.1038\/s41591-024-03139-8","article-title":"Integrated image-based deep learning and language models for primary diabetes care","volume":"30","author":"Li","journal-title":"Nat Med"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1186\/s13023-025-03656-w","article-title":"Comparative analysis of large language models on rare disease identification","volume":"20","author":"Ao","journal-title":"Orphanet J Rare Dis"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1016\/j.jaci.2025.02.004","article-title":"Evaluating large language model performance to support the diagnosis and management of patients with primary immune disorders","volume":"156","author":"Rider","journal-title":"J Allergy Clin Immunol"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.jtauto.2023.100213","article-title":"Accuracy of generative artificial intelligence models in differential diagnoses of familial Mediterranean fever and deficiency of Interleukin-1 receptor antagonist","volume":"7","author":"Pillai","journal-title":"J Transl Autoimmun"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-64827-6","article-title":"OpenMedLM: prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models","volume":"14","author":"Maharjan","journal-title":"Sci Rep"},{"issue":"24","key":"R29","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2426153122","article-title":"Human-AI collectives most accurately diagnose clinical vignettes","volume":"122","author":"Z\u00f6ller","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"10131","key":"R30","doi-asserted-by":"publisher","first-page":"1706","DOI":"10.1016\/S0140-6736(18)30841-9","article-title":"Prevalence and risk factors of chronic obstructive pulmonary disease in China (the China pulmonary health [CPH] study): a national cross-sectional study","volume":"391","author":"Wang","journal-title":"Lancet"},{"issue":"5","key":"R31","doi-asserted-by":"publisher","DOI":"10.1002\/acr2.70027","article-title":"Clinical manifestations and treatment in patients with relapsing polychondritis: a multicenter observational cohort study","volume":"7","author":"Yang","journal-title":"ACR Open Rheumatol"},{"issue":"10","key":"R32","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics15101221","article-title":"Diagnostic performance of publicly available large language models in corneal diseases: a comparison with human specialists","volume":"15","author":"Jiao","journal-title":"Diagnostics (Basel)"},{"issue":"9","key":"R33","doi-asserted-by":"publisher","DOI":"10.2196\/29839","article-title":"Application of artificial intelligence in community-based primary health care: systematic scoping review and critical appraisal","volume":"23","author":"Abbasgholizadeh Rahimi","journal-title":"J Med Internet Res"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajem.2025.04.028","article-title":"AI-assisted blood gas interpretation: a comparative study with an emergency physician","volume":"94","author":"G\u00fcn","journal-title":"Am J Emerg Med"},{"issue":"5","key":"R35","doi-asserted-by":"publisher","first-page":"39131700","DOI":"10.1056\/aioa2300151","article-title":"Comparative evaluation of LLMs in clinical oncology","volume":"1","author":"Rydzewski","journal-title":"NEJM AI"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","first-page":"39426","DOI":"10.1038\/s41598-025-22940-0","article-title":"Limitations of large language models in clinical problem-solving arising from inflexible reasoning","volume":"15","author":"Kim","journal-title":"Sci Rep"},{"issue":"5","key":"R37","doi-asserted-by":"publisher","DOI":"10.1016\/j.isci.2024.109713","article-title":"The application of large language models in medicine: a scoping review","volume":"27","author":"Meng","journal-title":"iScience"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2022.104227","article-title":"Deep learning for rare disease: a scoping review","volume":"135","author":"Lee","journal-title":"J Biomed Inform"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-14-67","article-title":"A Bayesian framework for estimating the incremental value of a diagnostic test in the absence of a gold standard","volume":"14","author":"Ling","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-67998-6","article-title":"Bayesian teaching enables probabilistic reasoning in large language models","volume":"17","author":"Qiu","journal-title":"Nat Commun"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-64769-1","article-title":"Quantifying the reasoning abilities of LLMs on clinical cases","volume":"16","author":"Qiu","journal-title":"Nat Commun"},{"key":"R42","unstructured":"Bommasani R Hudson DA Adeli E et al. On the opportunities and risks of foundation models. arXiv. Preprint posted online on Aug 16, 2021. [doi: 10.48550\/arXiv.2108.07258]"},{"issue":"1","key":"R43","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1002\/mdr2.70042","article-title":"Large language models in cardiovascular imaging: current applications and future prospects","volume":"2","author":"Yuan","journal-title":"Med Research"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T20:52:44Z","timestamp":1786049564000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e89963"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,6]]},"references-count":43,"URL":"https:\/\/doi.org\/10.2196\/89963","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,6]]},"article-number":"v28i2e89963"}}