{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T07:18:07Z","timestamp":1790666287302,"version":"4.1.0"},"reference-count":23,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"\n Background<\/jats:title>\n Large language models (LLMs) are rapidly incorporated into medical education and examination preparation; yet, most benchmarking evidence is derived from English-language material. Whether frontier commercial models and Brazilian Portuguese domain-specialized systems perform equivalently on high-stakes Brazilian medical examinations remains unclear.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to quantify and compare the performance of 9 frontier commercial LLMs and 1 Brazilian Portuguese domain-specialized system (Charcot, Voa Health) on 2026 Brazilian National Medical Education Examination (Exame Nacional de Avalia\u00e7\u00e3o da Forma\u00e7\u00e3o M\u00e9dica [ENAMED] 2026) and to describe the patterns of systematic between-model error as complementary quality signal.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n All 100 items of ENAMED 2026 (99 valid after annulment) were administered to 10 frontier-panel models across 5 independent runs under identical Portuguese prompts (temperature=0; top-p=.95). Commercial models were accessed through a unified OpenRouter client layer (DeepSeek provider-pinned). The primary outcome was mean accuracy against the preliminary key; the secondary outcomes were convergence error (CE), normalized mean response time (NMRT), and intermodel agreement. Accuracy was analyzed with Shapiro-Wilk, Levene, Kruskal-Wallis (\u03b52), Dunn-Holm post hoc, and a binomial generalized linear mixed model with question and run random intercepts. NMRT excluded Charcot (different stack) and Grok 4 (latency outlier). A total of 7 open-weight and small language models were assessed as a small language model (SLM) substudy.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Frontier-panel accuracy ranged from 73.74% (365\/495) for GPT-4o-mini to 96.97% (480\/495) for Charcot. Accuracy was nonnormal (Shapiro-Wilk, W=0.82; P<.001) with homogeneous variance (Levene P=.26). Kruskal-Wallis showed large between-model differences (H9=47.65; P<.001; \u03b52=0.97). Dunn-Holm flagged 8 of 45 pairs: Charcot was separable from GPT-4o-mini, DeepSeek v3.2-exp, and Grok 4, but not from the top frontier cluster. The generalized linear mixed model preserved the ranking (all comparators odds ratio<1 vs Charcot; upper CI<1 except GPT-5). A total of 9 items met the default CE criterion; item 77 showed 10-of-10 convergence, later confirmed by Instituto Nacional de Estudos e Pesquisas Educacionais An\u00edsio Teixeira rectification, and sensitivity analysis preserved the CE set (5-20 items). Intermodel agreement was high (Fleiss \u03ba=0.852; Krippendorff \u03b1=.852). Among 8 retained commercial models, NMRT correlated positively with accuracy (Spearman \u03c1=0.74; P=.04); these results are interpreted as descriptive of an orchestration-level latency\u2013accuracy law. SLM accuracy ranged from 47.47% (Gemma 3 4B) to 82.22% (GPT-OSS 120B), with lower agreement (Fleiss \u03ba=0.508) and 17 CE items. The best SLM-panel model lagged every frontier-panel model except GPT-4o-mini, with a 12-15 percentage-point gap against the top cluster.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n On ENAMED 2026, a Brazilian Portuguese domain-specialized system ranked first, indistinguishable from frontier commercial cluster and above subfrontier and open-weight systems. Charcot\u2019s architecture is not publicly disclosed; these findings should be interpreted as comparative black-box evidence of performance and not as mechanistic evidence of specialization. CE was stable, and it prospectively flagged 1 rectified item, supporting its use as a quality assurance screen.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/89839","type":"journal-article","created":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T14:10:06Z","timestamp":1778249406000},"page":"e89839","source":"Crossref","is-referenced-by-count":1,"title":["Performance of Large Language Models on the Brazilian National Medical Education Examination: Comparative Benchmark Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9214-691X","authenticated-orcid":false,"given":"Francys de Luca","family":"Fernandes da Silva","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1016-3319","authenticated-orcid":false,"given":"Eduardo Augusto","family":"Roeder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8649-6494","authenticated-orcid":false,"given":"Jo\u00e3o Victor","family":"Bruneti Severino","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0815-3288","authenticated-orcid":false,"given":"Matheus","family":"Nespolo Berger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6271-6862","authenticated-orcid":false,"given":"Pedro Angelo","family":"Basei de Paula","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1151-9652","authenticated-orcid":false,"given":"Davi","family":"Ferreira","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8261-5258","authenticated-orcid":false,"given":"Maria","family":"Han Veiga","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2983-3968","authenticated-orcid":false,"given":"Thyago Proen\u00e7a","family":"de Moraes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6057-0350","authenticated-orcid":false,"given":"Gustavo","family":"Lenci Marques","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,5,29]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.2196\/75849"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-06309-x"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.2196\/66114"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2024.102938"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1136\/bmjhci-2024-101195"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.3390\/app15137134"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.36660\/ijcs.20240231"},{"key":"ref10","unstructured":"Instituto Nacional de Estudos e Pesquisas Educacionais An\u00edsio Teixeira (INEP)Exame Nacional de Avalia\u00e7\u00e3o da Forma\u00e7\u00e3o M\u00e9dica (Enamed)2026-05-15https:\/\/www.gov.br\/pt-br\/servicos\/fazer-o-exame-nacional-de-avaliacao-da-formacao-medica"},{"key":"ref11","unstructured":"OpenRouter2026-05-15https:\/\/openrouter.ai"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1145\/3697010"},{"key":"ref13","first-page":"1","author":"Atil, B","journal-title":"ArXiv. Preprint posted online on August 6, 2024"},{"key":"ref14","first-page":"1","author":"Song, Y","journal-title":"ArXiv. Preprint posted online on July 15, 2024"},{"issue":"21","key":"ref15","first-page":"19","volume":"1","author":"Tomczak, M","year":"2014","journal-title":"Trends Sport Sci"},{"key":"ref16","author":"Miller, E","journal-title":"ArXiv. Preprint posted online on November 1, 2024"},{"key":"ref17","unstructured":"High-risk pregnancy manualBrazil Ministry of Health20222026-05-25https:\/\/bvsms.saude.gov.br\/bvs\/publicacoes\/manual_gestacao_alto_risco.pdf"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01789-7"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1800"},{"key":"ref20","first-page":"1","author":"Moore, K","journal-title":"ArXiv. Preprint posted online on September 6, 2024"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.52202\/075280-3275"},{"key":"ref22","author":"Arcuschin, I","journal-title":"ArXiv. Preprint posted online on June 17, 2025"},{"key":"ref23","first-page":"1","author":"Blackwell, RE","journal-title":"ArXiv. Preprint posted online on October 4, 2024"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e89839","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T15:20:11Z","timestamp":1780068011000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e89839"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,29]]},"references-count":23,"URL":"https:\/\/doi.org\/10.2196\/89839","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.89839","asserted-by":"object"}]},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,29]]}}}