{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T04:50:48Z","timestamp":1793163048654,"version":"4.5.0"},"reference-count":45,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Despite the transformative potential of artificial intelligence (AI)\u2013based chatbots in medicine, their implementation is hindered by data privacy and security concerns. DeepSeek offers a conceivable solution through its capability for local offline operations. However, as of 2025, it remains unclear whether DeepSeek can achieve an accuracy comparable to that of conventional, cloud-based AI chatbots.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aims to evaluate whether DeepSeek, an AI-based chatbot capable of offline operation, achieves answer accuracy on medical multiple-choice questions (MCQs) comparable to that of leading chatbots (ie, ChatGPT and Gemini) on German medical MCQs, thereby assessing its potential as a privacy-preserving alternative for clinical use.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A total of 200 interdisciplinary MCQs from the German Progress Test Medicine were administered to ChatGPT (GPT-o3-mini), DeepSeek (DeepSeek-R1), and Gemini (Gemini 2.0 Flash). Accuracy was defined as the proportion of correctly solved questions. Overall differences among the 3 models were tested with the Cochran Q test, while pairwise comparisons were conducted using the McNemar test. Subgroup analyses were performed by medical domain (Fisher exact test) and question length (Wilcoxon rank-sum test). An a priori power analysis indicated a minimum sample size of 195 questions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>All 3 chatbots surpassed the conventional passing threshold of 60%, with accuracies of 96% (192\/200) for DeepSeek, 94% (188\/200) for Gemini, and 92.5% (185\/200) for ChatGPT. The overall difference among models was not statistically significant (P=.10) nor were pairwise comparisons. However, incorrect responses were significantly associated with longer question length for DeepSeek (P=.049) and ChatGPT (P=.04) but not for Gemini. No significant differences in performance were observed across clinical versus preclinical domains or medical specialties (all P&gt;.05).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Overall, DeepSeek demonstrates outstanding performance on German medical MCQs comparable to the widely used chatbots ChatGPT and Gemini. Similar to ChatGPT, DeepSeek\u2019s performance declined with increasing question length, highlighting verbosity as a persistent challenge for large language models. While DeepSeek\u2019s offline capability and lower operational costs are advantageous, its safe and reliable application in clinical contexts requires further investigation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/77357","type":"journal-article","created":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T20:55:05Z","timestamp":1766091305000},"page":"e77357","source":"Crossref","is-referenced-by-count":10,"title":["Performance of DeepSeek-R1, ChatGPT (GPT-o3-mini), and Gemini 2.0 Flash on German Medical Multiple-Choice Questions: Comparative Evaluation"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8411-8799","authenticated-orcid":false,"given":"Annika","family":"Meyer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6380-158X","authenticated-orcid":false,"given":"Yassin","family":"Karay","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5237-961X","authenticated-orcid":false,"given":"Andrea U","family":"Steinbicker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6588-720X","authenticated-orcid":false,"given":"Thomas","family":"Streichert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4046-0234","authenticated-orcid":false,"given":"Remco","family":"Overbeek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,12,18]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000866"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.4103\/jfmpc.jfmpc_440_19"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMe2206291"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.2196\/50965"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.2196\/48568"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1515\/labmed-2024-0034"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1515\/cclm-2024-0246"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2023.1838"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.79221"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2501.12948"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2025.104791"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.1093\/nsr\/nwaf044"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1002\/ksa.12628"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-025-00229-6"},{"key":"ref16","unstructured":"Progress Test Medizin2025-11-18https:\/\/ptm-dashboard.charite.de\/"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2018.1472370"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1007\/s40670-024-02176-9"},{"key":"ref19","unstructured":"R Core TeamR: a language and environment for statistical computingR Foundation for Statistical Computing20222025-11-18https:\/\/www.r-project.org\/"},{"key":"ref20","unstructured":"ChampelySEkstromCDalgaardPGillJWeibelzahlSAnandkumarAFordCVolcicRDe RosarioHpwr: basic functions for power analysisThe Comprehensive R Archive Network20202025-11-18https:\/\/cran.r-project.org\/web\/packages\/pwr\/pwr.pdf"},{"key":"ref21","unstructured":"ChanCLeeperTJBeckerJSchochDrio: a Swiss-army knife for data I\/OThe Comprehensive R Archive Network2025-11-18https:\/\/cran.r-project.org\/web\/packages\/rio\/readme\/README.html"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.21105\/JOSS.01686"},{"key":"ref23","unstructured":"SjobergDDLarmarangeJCurryMLaveryJWhitingKZaborECgtsummary: presentation-ready data summary and analytic result tablesDaniel D. Sjoberg2025-11-18https:\/\/www.danieldsjoberg.com\/gtsummary\/\/reference\/gtsummary-package.html"},{"key":"ref24","unstructured":"KassambaraArstatix: pipe-friendly framework for basic statistical testsThe Comprehensive R Archive Network20252025-11-18https:\/\/cloud.r-project.org\/web\/packages\/rstatix\/index.html"},{"key":"ref25","unstructured":"LarmarangeJLudeckeDWickhamHBojanowskiMBriatteFlabelled: manipulating labelled dataThe Comprehensive R Archive Network2025-11-18https:\/\/cran.r-project.org\/web\/packages\/labelled\/index.html"},{"key":"ref26","unstructured":"KaplanJfastDummies: fast creation of dummy (binary) columns and rows from categorical variablesThe Comprehensive R Archive Network20257222025-11-18https:\/\/cran.r-project.org\/web\/packages\/fastDummies\/fastDummies.pdf"},{"key":"ref27","unstructured":"NeuwirthERColorBrewer: ColorBrewer palettesThe Comprehensive R Archive Network20222025-11-18https:\/\/cran.r-project.org\/web\/packages\/RColorBrewer\/index.html"},{"key":"ref28","unstructured":"WilkeCOcowplot: streamlined plot theme and plot annotations for ggplot2The Comprehensive R Archive Network2025772025-11-18https:\/\/cran.r-project.org\/web\/packages\/cowplot\/index.html"},{"key":"ref29","unstructured":"K\u00f6lnUEthikkommission der medizinischen fakult\u00e4t der Universit\u00e4t zu K\u00f6lnUniversit\u00e4t zu K\u00f6ln2025-09-29https:\/\/medfak.uni-koeln.de\/forschung-transfer\/translation-i-klinische-forschung\/ethikkommission"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-024-07421-0"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1186\/s40348-023-00160-8"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1016\/j.tox.2019.152254"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.64924"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1080\/10872981.2023.2220920"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.2196\/60807"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1007\/s10439-023-03338-3"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1136\/bjo-2025-327360"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1016\/j.aopr.2025.05.001"},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.1101\/2025.01.27.25321169"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-025-00330-w"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-025-02159-2"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2501.18438"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.1145\/3630106.3659005"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.2196\/56930"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.07377"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e77357","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:09:30Z","timestamp":1782842970000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e77357"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,18]]},"references-count":45,"URL":"https:\/\/doi.org\/10.2196\/77357","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.77357","asserted-by":"object"}]},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,18]]}}}