{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T16:51:05Z","timestamp":1791391865257,"version":"4.3.3"},"reference-count":17,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Independent audits of medical large language models have concentrated on models that answer unsafely rather than those that decline safe questions. In June 2026, Anthropic released Claude Fable 5 with a safeguard that detects requests related to cybersecurity, biology, or chemistry and reroutes them to a fallback model, Claude Opus 4.8. The developer states that this safeguard is deliberately conservative and affects fewer than 5% of sessions. Whether that figure holds for consumer health questions, many containing dense biomedical vocabulary, was unknown.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study measured how often Claude Fable 5 routed consumer health questions to its fallback model and whether routing varied by clinical domain and framing.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n In this observational, point-in-time audit, the first 500 unique prompts in alphabetical order from the HealthSearchQA benchmark (3173 prompts) were each entered once into Claude Fable 5 through the web interface, in a new session with no system prompt and default settings, from June 9 to 12, 2026. Two reviewers independently coded each response as routed to fallback upfront or not routed upfront, with a third adjudicating disagreements. Midgeneration truncation due to a safety interruption was recorded separately. Prompts were labeled by clinical domain, question type, and sensitivity. Rates were reported with Wilson 95% CIs and compared using chi-square tests. Gemini 2.5 Flash served as an independent comparator.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Reviewers agreed on 474 of 500 responses (94.8%; Cohen \u03ba=0.90; 95% CI 0.86-0.94). Fable 5 routed 243 (48.6%; 95% CI 44.2%-53.0%) prompts to the fallback. Of the 257 not routed upfront, 236 (91.8%; 95% CI 87.8%-94.6%) were fully answered and 21 (8.2%; 95% CI 5.4%-12.2%) were truncated midgeneration. Fallback routing varied by clinical domain (\n \u03c7<\/jats:italic>\n 2<\/jats:sup>\n 10<\/jats:sub>\n =59.5;\n P<\/jats:italic>\n <.001), from 88.9% (32\/36; 95% CI 74.7%-95.6%) for oncology and 79.3% (23\/29; 95% CI 61.6%-90.2%) for reproductive and obstetric prompts to 26.3% (5\/19; 95% CI 11.8%-48.8%) for mental and behavioral health. It also varied by question type (\n \u03c7<\/jats:italic>\n 2<\/jats:sup>\n 3<\/jats:sub>\n =61.9;\n P<\/jats:italic>\n <.001): 63.1% (101\/160; 95% CI 55.4%-70.2%) for prognosis or severity, 49.6% (137\/276; 95% CI 43.8%-55.5%) for definition, and 1.8% (1\/55; 95% CI 0.3%-9.6%) for diagnosis or treatment. Gemini answered 89.7% (218\/243; 95% CI 85.3%-92.9%) of routed prompts and declined 19% (95\/500; 95% CI 15.8%-22.7%) overall. Standardization to the full 3173-prompt question-type and domain compositions yielded fallback rates of 48.8% and 47.5%, respectively.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Claude Fable 5 routed nearly half of these benchmark consumer health prompts away from the primary model, and routing was associated with clinical domain, question category, and disease vocabulary. Because benignness was not independently adjudicated, wording covaried with clinical content, and the fallback model\u2019s subsequent answer was not recorded, these findings describe routing and truncation rather than user-facing refusal or a causal classifier feature. Fallback routing is a measurable safety property that audits should report alongside answer quality.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/104856","type":"journal-article","created":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T15:30:05Z","timestamp":1786462205000},"page":"e104856-e104856","source":"Crossref","is-referenced-by-count":0,"title":["Safety-Filter Fallback on Consumer Health Questions During the Initial Claude Fable 5 Deployment: Observational Study"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7678-633X","authenticated-orcid":false,"given":"Yosef","family":"Adiniaev","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0438-0827","authenticated-orcid":false,"given":"Mahmud","family":"Omar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7242-1328","authenticated-orcid":false,"given":"Yiftach","family":"Barash","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7074-8909","authenticated-orcid":false,"given":"Olga R","family":"Brook","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7542-8608","authenticated-orcid":false,"given":"Alon","family":"Gorenshtein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4567-3108","authenticated-orcid":false,"given":"Eyal","family":"Klang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,7]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"100949","DOI":"10.1016\/j.landig.2025.100949","article-title":"Mapping the susceptibility of large language models to medical misinformation across clinical notes and social media: a cross-sectional benchmarking analysis","volume":"8","author":"Omar","journal-title":"Lancet Digit Health"},{"issue":"6","key":"R2","doi-asserted-by":"publisher","first-page":"1873","DOI":"10.1038\/s41591-025-03626-6","article-title":"Sociodemographic biases in medical decision making by large language models","volume":"31","author":"Omar","journal-title":"Nat Med"},{"key":"R3","doi-asserted-by":"crossref","unstructured":"R\u00f6ttger P Kirk H Vidgen B Attanasio G Bianchi F Hovy D . XSTest: a test suite for identifying exaggerated safety behaviours in large language models. In: Duh K Gomez H Bethard S , editors. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics; 2024:5377-5400. [doi: 10.18653\/v1\/2024.naacl-long.301]","DOI":"10.18653\/v1\/2024.naacl-long.301"},{"key":"R4","unstructured":"Arora RK Wei J Soskin Hicks R et al. HealthBench: evaluating large language models towards improved human health. arXiv. Preprint posted online on May 13, 2025. [doi: 10.48550\/arXiv.2505.08775]"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","DOI":"10.7326\/aimcc.2024.1260","article-title":"A case of bromism influenced by use of artificial intelligence","volume":"4","author":"Eichenberger","journal-title":"Ann Intern Med Clin Cases"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2026.1779200","article-title":"Preparedness for generative AI adoption among Chinese cancer survivors: a multi-center cross-sectional survey study","volume":"14","author":"Wang","journal-title":"Front Public Health"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocn.2025.111097","article-title":"Performance evaluation of ChatGPT-4.0 and Gemini on image-based neurosurgery board practice questions: a comparative analysis","volume":"134","author":"McNulty","journal-title":"J Clin Neurosci"},{"key":"R8","unstructured":"Claude Fable 5 and Claude Mythos 5. Anthropic. 2026. URL: https:\/\/www.anthropic.com\/news\/claude-fable-5-mythos-5 [Accessed 13-06-2026]"},{"issue":"7972","key":"R9","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/17187","article-title":"Prevalence of health misinformation on social media: systematic review","volume":"23","author":"Suarez-Lledo","journal-title":"J Med Internet Res"},{"key":"R11","unstructured":"Ethics and governance of artificial intelligence for health: guidance on large multi-modal models. World Health Organization. 2024. URL: https:\/\/www.who.int\/publications\/i\/item\/9789240084759 [Accessed 13-06-2026]"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-026-02584-8","article-title":"New model, old risks: sociodemographic bias and adversarial hallucinations vulnerability in GPT-5","volume":"9","author":"Omar","journal-title":"NPJ Digit Med"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.bas.2026.105997","article-title":"AI at the sella turcica: multi-model large language model evaluation in pituitary adenomas","volume":"6","author":"Aliyeva","journal-title":"Brain Spine"},{"issue":"4","key":"R14","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1080\/09286586.2026.2651197","article-title":"Evaluating three large language models in ophthalmology education: a comparative study of accuracy and readability across 442 questions","volume":"33","author":"\u00c7akmak","journal-title":"Ophthalmic Epidemiol"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1007\/s00405-026-10245-5","article-title":"\u201cMELMA\u201d in otolaryngology: medical evaluation of large language model answers. Clinician-rated scoring (MELMA-Q) and web-based auditing (MELMA-W) novel tools for AI assessment","author":"Ahmadov","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"R16","unstructured":"Redeploying Fable 5. Anthropic; 2026. URL: https:\/\/www.anthropic.com\/news\/redeploying-fable-5 [Accessed 08-08-2026]"},{"key":"R17","unstructured":"Statement on the US government directive to suspend access to Fable 5 and Mythos 5. Anthropic. 2026. URL: https:\/\/www.anthropic.com\/news\/fable-mythos-access [Accessed 13-06-2026]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T16:00:30Z","timestamp":1791388830000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e104856"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,7]]},"references-count":17,"URL":"https:\/\/doi.org\/10.2196\/104856","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,7]]},"article-number":"v5i1e104856"}}