{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T17:35:37Z","timestamp":1790962537116,"version":"4.1.0"},"reference-count":30,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Generative artificial intelligence (AI) systems are increasingly deployed in clinical pharmacy; yet, systematic evaluation of their efficacy, limitations, and risks across diverse practice scenarios remains limited.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to quantitatively evaluate and compare the performance of 8 mainstream generative AI systems across 4 core clinical pharmacy scenarios\u2014medication consultation, medication education, prescription review, and case analysis with pharmaceutical care\u2014using a multidimensional framework.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Forty-eight clinically validated questions were selected via stratified sampling from real-world sources (eg, hospital consultations, clinical case banks, and national pharmacist training databases). Three researchers simultaneously tested 8 different generative AI systems (ERNIE Bot, Doubao, Kimi, Qwen, GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet, and DeepSeek-R1) using standardized prompts within a single day (February 20, 2025). A double-blind scoring design was used, with 6 experienced clinical pharmacists (\u22655 years experience) evaluating the AI responses across 6 dimensions: accuracy, rigor, applicability, logical coherence, conciseness, and universality, scored 0\u201010 per predefined criteria (eg, \u22123 for inaccuracy and \u22122 for incomplete rigor). Statistical analysis used one-way ANOVA with Tukey Honestly Significant Difference (HSD) post hoc testing and intraclass correlation coefficients (ICC) for interrater reliability (2-way random model). Qualitative thematic analysis identified recurrent errors and limitations.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n DeepSeek-R1 (DeepSeek) achieved the highest overall performance (mean composite score: medication consultation 9.4, SD 1.0; case analysis 9.3, SD 1.0), significantly outperforming others in complex tasks (\n P<\/jats:italic>\n <.05). Critical limitations were observed across models, including high-risk decision errors\u201475% omitted critical contraindications (eg, ethambutol in optic neuritis) and a lack of localization\u201490% erroneously recommended macrolides for drug-resistant\n Mycoplasma pneumoniae<\/jats:italic>\n (China\u2019s high-resistance setting), while only DeepSeek-R1 aligned with updated American Academy of Pediatrics (AAP) guidelines for pediatric doxycycline. Complex reasoning deficits: only Claude-3.5-Sonnet detected a gender-diagnosis contradiction (prostatic hyperplasia in female); no model identified diazepam\u2019s 7-day prescription limit. Interrater consistency was lowest for conciseness in case analysis (ICC=0.70), reflecting evaluator disagreement on complex outputs. ERNIE Bot (Baidu) consistently underperformed (case analysis: 6.8, SD 1.5;\n P<\/jats:italic>\n <.001 vs DeepSeek-R1).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n While generative AI shows promise as a pharmacist assistance tool, significant limitations\u2014including high-risk errors (eg, contraindication omissions), inadequate localization, and complex reasoning gaps\u2014preclude autonomous clinical decision-making. Performance stratification highlights DeepSeek-R1\u2019s current advantage, but all systems require optimization in dynamic knowledge updating, complex scenario reasoning, and output interpretability. Future deployment must prioritize human oversight (human-AI co-review), ethical safeguards, and continuous evaluation frameworks.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/76128","type":"journal-article","created":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T09:15:09Z","timestamp":1750151709000},"page":"e76128-e76128","source":"Crossref","is-referenced-by-count":17,"title":["Comparative Analysis of Generative Artificial Intelligence Systems in Solving Clinical Pharmacy Problems: Mixed Methods Study"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-4974-132X","authenticated-orcid":false,"given":"Lulu","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6943-485X","authenticated-orcid":false,"given":"Pengqiang","family":"Du","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9934-291X","authenticated-orcid":false,"given":"Xiaojing","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-7520-7531","authenticated-orcid":false,"given":"Hongwei","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9358-8549","authenticated-orcid":false,"given":"Ming","family":"Ni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8787-7579","authenticated-orcid":false,"given":"Meng","family":"Yan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7948-7073","authenticated-orcid":false,"given":"Aifeng","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,7,24]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.2147\/IPRP.S466748","article-title":"Advancing pharmacy practice: the role of intelligence-driven pharmacy practice and the emergence of pharmacointelligence","volume":"13","author":"Hatem","journal-title":"Integr Pharm Res Pract"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","journal-title":"Nat Med"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2147\/DHPS.S425858","article-title":"Evaluating the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, Bing AI, and Bard against conventional drug-drug interactions clinical tools","volume":"15","author":"Al-Ashwal","journal-title":"Drug Healthc Patient Saf"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2024.105474","article-title":"Generative artificial intelligence in healthcare: a scoping review on benefits, challenges and applications","volume":"188","author":"Moulaei","journal-title":"Int J Med Inform"},{"issue":"7","key":"R5","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.42150","article-title":"ChatGPT and artificial intelligence in transplantation research: is it always correct?","volume":"15","author":"Rawashdeh","journal-title":"Cureus"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.xcrm.2023.101356","article-title":"Artificial intelligence, ChatGPT, and other large language models for social determinants of health: current state and future directions","volume":"5","author":"Ong","journal-title":"Cell Rep Med"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.2196\/49655","article-title":"Implementing AI in hospitals to achieve a learning health system: systematic review of current enablers and barriers","volume":"26","author":"Kamel Rahimi","journal-title":"J Med Internet Res"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1111\/bcp.15896","article-title":"Evaluating the performance of ChatGPT in clinical pharmacy: a comparative study of ChatGPT and clinical pharmacists","volume":"90","author":"Huang","journal-title":"Br J Clin Pharmacol"},{"issue":"2","key":"R9","doi-asserted-by":"publisher","first-page":"e137","DOI":"10.1016\/S2589-7500(21)00229-6","article-title":"Key use cases for artificial intelligence to reduce the frequency of adverse drug events: a scoping review","volume":"4","author":"Syrowatka","journal-title":"Lancet Digit Health"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/57162","article-title":"Evaluation of AI-driven LabTest checker for diagnostic accuracy and safety: prospective cohort study","volume":"12","author":"Szumilas","journal-title":"JMIR Med Inform"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2147\/JMDH.S482757","article-title":"Artificial intelligence, the digital surgeon: unravelling its emerging footprint in healthcare - the narrative review","volume":"17","author":"Shang","journal-title":"J Multidiscip Healthc"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.79221","article-title":"DeepSeek in healthcare: revealing opportunities and steering challenges of a new open-source artificial intelligence frontier","volume":"17","author":"Temsah","journal-title":"Cureus"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1055\/s-0043-1768726","article-title":"Exploring the latest highlights in medical natural language processing across multiple languages: a survey","volume":"32","author":"Shaitarova","journal-title":"Yearb Med Inform"},{"issue":"4","key":"R14","doi-asserted-by":"publisher","DOI":"10.1002\/oto2.70048","article-title":"Enhancing multilingual patient education: ChatGPT\u2019s accuracy and readability for SSNHL queries in English and Spanish","volume":"8","author":"Ajit-Roger","journal-title":"OTO Open"},{"issue":"50","key":"R15","doi-asserted-by":"publisher","DOI":"10.1097\/MD.0000000000036671","article-title":"Ethical implications of AI and robotics in healthcare: a review","volume":"102","author":"Elendu","journal-title":"Medicine (Baltimore)"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1177\/00187208251326795","article-title":"Beyond binary decisions: evaluating the effects of AI error type on trust and performance in AI-assisted tasks","volume":"2025","author":"Kim","journal-title":"Hum Factors"},{"issue":"3","key":"R17","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1111\/1468-0009.12504","article-title":"Artificial intelligence and liability in medicine: balancing safety and innovation","volume":"99","author":"Maliha","journal-title":"Milbank Q"},{"issue":"4","key":"R18","doi-asserted-by":"publisher","first-page":"251","DOI":"10.2471\/BLT.19.237487","article-title":"Artificial intelligence in health care: accountability and safety","volume":"98","author":"Habli","journal-title":"Bull World Health Organ"},{"issue":"2","key":"R19","first-page":"80","volume":"26","author":"Tessler","journal-title":"Isr Med Assoc J"},{"issue":"11","key":"R20","doi-asserted-by":"publisher","first-page":"1051","DOI":"10.1177\/00220345241271160","article-title":"Trustworthy artificial intelligence in dentistry: learnings from the EU AI Act","volume":"103","author":"Ducret","journal-title":"J Dent Res"},{"issue":"2","key":"R21","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s40264-023-01376-3","article-title":"The unseen hand: AI-based prescribing decision support tools and the evaluation of drug safety and effectiveness","volume":"47","author":"Dickinson","journal-title":"Drug Saf"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1016\/j.arr.2024.102497","article-title":"AI-driven innovations in Alzheimer\u2019s disease: integrating early diagnosis, personalized treatment, and prognostic modelling","volume":"101","author":"Kale","journal-title":"Ageing Res Rev"},{"issue":"9","key":"R23","doi-asserted-by":"publisher","DOI":"10.3390\/jcm14093078","article-title":"Management of burns: multi-center assessment comparing AI models and experienced plastic surgeons","volume":"14","author":"Marcaccini","journal-title":"J Clin Med"},{"issue":"5","key":"R24","doi-asserted-by":"publisher","first-page":"2623","DOI":"10.1109\/TCBB.2021.3103595","article-title":"A multimodal framework for improving in silico drug repositioning with the prior knowledge from knowledge graphs","volume":"19","author":"Xiong","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-023-04698-z","article-title":"Revolutionizing healthcare: the role of artificial intelligence in clinical practice","volume":"23","author":"Alowais","journal-title":"BMC Med Educ"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02554-8","article-title":"Enhancing post-traumatic stress disorder patient assessment: leveraging natural language processing for research of domain criteria identification using electronic medical records","volume":"24","author":"Miranda","journal-title":"BMC Med Inform Decis Mak"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.3389\/fphar.2019.00680","article-title":"Development of a complex intervention to improve adherence to antidiabetic medication in older people using an anthropomorphic virtual assistant software","volume":"10","author":"F\u00e9lix","journal-title":"Front Pharmacol"},{"issue":"2","key":"R28","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1515\/dx-2017-0045","article-title":"Diagnostic accuracy in Family Medicine residents using a clinical decision support system (DXplain): a randomized-controlled trial","volume":"5","author":"Martinez-Franco","journal-title":"Diagnosis (Berl)"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","DOI":"10.1111\/tct.70040","article-title":"The role of AI in reshaping medical education: opportunities and challenges","volume":"22","author":"Ali","journal-title":"Clin Teach"},{"issue":"5","key":"R30","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.59747","article-title":"Artificial intelligence and healthcare simulation: the shifting landscape of medical education","volume":"16","author":"Hamilton","journal-title":"Cureus"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e76128","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:07:33Z","timestamp":1782842853000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e76128"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,24]]},"references-count":30,"URL":"https:\/\/doi.org\/10.2196\/76128","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,24]]},"article-number":"v13i1e76128"}}