{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T22:10:10Z","timestamp":1790115010323,"version":"4.0.1"},"reference-count":30,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Res Protoc"],"abstract":"\n Background<\/jats:title>\n Most clinical reasoning evaluations in large language models (LLMs) score only the final answer, usually multiple-choice accuracy, which explains little about model reasoning. Two developments stress this gap: reasoning-optimized models are now common, and several expose an explicit extended thinking control. Whether that deliberation improves the reasoning process remains untested with a validated instrument.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This protocol introduces and aims to validate the Rapid Evaluation Assessment of Clinical Reasoning Tool (REACT)\u2013AI, a Behaviorally Anchored Rating Scale (BARS) with 13 subdomains for process-oriented assessment of AI clinical reasoning, that is, the quality of the externalized reasoning rather than its faithfulness to the model\u2019s internal computation. It also builds a conflict-of-interest\u2013controlled LLM-as-judge pipeline and tests whether extended thinking improves reasoning quality.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This 2-phase, prospective, comparative study has a within-model thinking-mode factor. Six flagship models are run in standard and extended thinking or reasoning modes, giving 12 conditions: 4 providers are toggled within the same model, and 2 pair a standard model with a dedicated reasoning model. In phase 1, 5 standardized urgent care vignettes are answered under all 12 conditions, 3 runs per condition (180 AI outputs), and by an independent expert clinician panel (n=5) and senior and junior medical students. Every output is scored by dual-blind human raters and, in parallel, by LLM judges, with no model judging its own family. Acceptance criteria for deploying the judge at scale are a weighted \u03ba of at least 0.60 and an intraclass correlation coefficient above 0.75. In phase 2, the validated pipeline scores 3600 AI outputs. A self-correction turn and a paired Gulf English condition run alongside and are reported separately, with the AI metacognition assessment rubric and disinformation generation rate as secondary instruments.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n As of June 2026, ethics approval was obtained (ERSC_2025_6124), and the study is registered on the Open Science Framework under embargo. Responses to the 5 phase 1 vignettes were collected from senior and junior medical students between January and June 2026; 173 scripts were received and remain sealed, unopened, and unscored. No AI outputs have been generated, and the expert panel remains unrecruited. Following the June 1, 2026, model-version lock, AI generation and scoring will begin in September 2026, with phase 1 calibration through December 2026 and phase 2 from January to April 2027; results are expected in winter 2027. The study will report the validated instrument, judge calibration against human experts, self-preference bias by model family, and the effect of extended thinking on reflection and metacognition, including null findings.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n REACT-AI and its judge pipeline are built to be reusable. They close the gap between accuracy benchmarks and genuine reasoning assessment and provide a template for studying reasoning mode effects.<\/jats:p>\n <\/jats:sec>\n \n Trial Registration<\/jats:title>\n OSF Registries osf.io\/jep5t; https:\/\/osf.io\/jep5t<\/jats:p>\n <\/jats:sec>\n \n International Registered Report Identifier (IRRID)<\/jats:title>\n DERR1-10.2196\/103220<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/103220","type":"journal-article","created":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T21:55:10Z","timestamp":1790114110000},"page":"e103220","source":"Crossref","is-referenced-by-count":0,"title":["Process-Oriented, Behaviorally Anchored Assessment of Clinical Reasoning in Large Language Models and the Effect of Extended Thinking: Protocol for a Prospective, Multigroup, Comparative Study"],"prefix":"10.2196","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2704-8931","authenticated-orcid":false,"given":"Adnan","family":"Agha","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2784-5762","authenticated-orcid":false,"given":"Muhammad","family":"Jalil","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9335-9208","authenticated-orcid":false,"given":"Eram","family":"Anwar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1447-9836","authenticated-orcid":false,"given":"Omran","family":"Bakoush","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,22]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.94563"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01963-x"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.26021"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.13375"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0000000000002618"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2018.4385"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1097\/acm.0000000000003027"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.40969"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-025-04190-9"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2026.4003"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-64769-1"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.22514\/jofph.2026.029"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1037\/h0047060"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-022-07513-5"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2020"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2411.15594"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-025-04151-2"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2604.25933"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2604.06996"},{"key":"ref21","first-page":"368","author":"Rouillard, A","year":"2026","journal-title":"Artificial Intelligence in Medicine: 24th International Conference, AIME"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1101\/2025.04.22.25326219"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03425-5"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-1034-x"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.m3210"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2024-087469"},{"key":"ref27","first-page":"10162","author":"Yan, Z","year":"2026","journal-title":"Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.2307\/257876"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1037\/a0016476"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-025-03626-6"}],"container-title":["JMIR Research Protocols"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e103220","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T21:55:12Z","timestamp":1790114112000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e103220"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,22]]},"references-count":30,"URL":"https:\/\/doi.org\/10.2196\/103220","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.103220","asserted-by":"object"}]},"ISSN":["1929-0748"],"issn-type":[{"value":"1929-0748","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,22]]}}}