{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T09:35:55Z","timestamp":1793352955850,"version":"4.5.0"},"reference-count":19,"publisher":"BMJ","issue":"5","license":[{"start":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T00:00:00Z","timestamp":1778630400000},"content-version":"unspecified","delay-in-days":12,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2026,4,1]]},"published-print":{"date-parts":[[2026,5]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To assess and compare the performance of four contemporary frontier large language models (LLMs)\u2014GPT-5.2 (OpenAI), Gemini 3 Pro (Google DeepMind), Claude Sonnet 4.6 (Anthropic) and Grok 4.1 (xAI)\u2014on a simulated Fellowship of The Royal College of Surgeons Urology (FRCS(Urol)) Part A examination, evaluating overall accuracy, subspecialty-level performance, output consistency and response time.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Design<\/jats:title>\n                    <jats:p>Controlled comparative evaluation study using a standardised simulation framework with repeated independent testing runs per model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Setting<\/jats:title>\n                    <jats:p>All models were accessed via their respective consumer-facing interfaces. No clinical setting or patient data were involved. Testing was conducted under uniform conditions with conversational memory disabled across all sessions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Participants<\/jats:title>\n                    <jats:p>Four large language models were evaluated. No human participants were involved. Models were selected to represent the current frontier of publicly accessible LLMs from four distinct commercial developers. No models were excluded following selection.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Interventions<\/jats:title>\n                    <jats:p>Each model was presented with 240 FRCS (Urol) Part A single best answer questions, mapped to the Joint Committee on Intercollegiate Examinations' Urology Syllabus Blueprint (2023). A standardised prompt was delivered at the start of each session. Each model completed five independent examination runs. No fine-tuning or system-level modification was applied to any model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Primary and secondary outcome measures<\/jats:title>\n                    <jats:p>The primary outcome was overall examination accuracy for each model, benchmarked against an indicative pass threshold for the FRCS (Urol) Part A examination. Secondary outcomes were performance across 18 individual urology subspecialty topics; response time reported as mean total and per-question elapsed time; and consistency of performance quantified by SD and 95% CIs derived from a sequential Monte Carlo sampling procedure. All outcomes were prospectively planned and fully measured as specified.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Three of four models exceeded the indicative 74% pass threshold: Gemini 3 Pro (82.4%\u00b10.9%; 95%\u2009CI 81.3 to 83.6%), Claude Sonnet 4.6 (79.3%\u00b11.1%; 95%\u2009CI 77.9 to 80.6%) and GPT-5.2 (76.1%\u00b12.4%; 95%\u2009CI 73.1 to 79.1%). Grok 4.1 failed (70.4%\u00b10.6%; 95%\u2009CI 69.6 to 71.2%), with its entire CI below 74%. All models completed the assessment in under 3\u2009min. Strong performance was observed in research methodology (90\u201398%)\u2009and andrology (92\u201398%), with the weakest results in paediatric urology (38.7\u201354.7%)\u2009and testicular cancer (48.2\u201367.3%). Substantial within-model output instability was identified across several domains, most notably GPT-5.2 in female urology (SD\u00b122.8%) and anatomy (SD\u00b114.2%).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Three of four frontier LLMs achieved scores consistent with passing the FRCS (Urol) Part A examination, representing a substantial advance since ChatGPT-3.5. Aggregate accuracy alone, however, obscures important subspecialty weaknesses and output instability. LLMs should be regarded as adjunctive revision aids rather than authoritative knowledge sources and always used alongside expert-led teaching. Future work should evaluate performance on Part B and viva-style assessments.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjopen-2025-108775","type":"journal-article","created":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T14:26:12Z","timestamp":1778682372000},"page":"e108775","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":2,"title":["Performance of large language models (GPT-5.2, Gemini 3 Pro, Claude Sonnet 4.6 and Grok 4.1) on the Fellowship of The Royal College of Surgeons Urology Part A examination"],"prefix":"10.1136","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-9991-8075","authenticated-orcid":false,"given":"Abdul Rhaman","family":"Kafagi","sequence":"first","affiliation":[{"name":"Faculty of Biology Medicine and Health, The University of Manchester, Manchester, England, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-5813-708X","authenticated-orcid":false,"given":"Nazih","family":"Atassi","sequence":"additional","affiliation":[{"name":"Faculty of Biology Medicine and Health, The University of Manchester, Manchester, England, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abdul-Hadi","family":"Kafagi","sequence":"additional","affiliation":[{"name":"Department of Urology, Wythenshawe Hospital, Manchester University Hospitals NHS Foundation Trust, Manchester, England, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,5,13]]},"reference":[{"key":"2026051307253283000_16.5.e108775.1","doi-asserted-by":"crossref","unstructured":"Dai Z , Yang Z , Yang Y , et al . Transformer-xl: attentive language models beyond a fixed-length context. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; Stroudsburg, PA, USA, Florence, Italy. doi:10.18653\/v1\/P19-1285 Available: https:\/\/aclanthology.org\/P19-1","DOI":"10.18653\/v1\/P19-1285"},{"key":"2026051307253283000_16.5.e108775.2","doi-asserted-by":"crossref","DOI":"10.1186\/s12909-024-05944-8","article-title":"Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis","volume":"24","author":"Jin","year":"2024","journal-title":"BMC Med Educ"},{"key":"2026051307253283000_16.5.e108775.3","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1080\/0142159X.2023.2249588","article-title":"ChatGPT-4: An assessment of an upgraded artificial intelligence chatbot in the United States Medical Licensing Examination","volume":"46","author":"Mihalache","year":"2024","journal-title":"Med Teach"},{"key":"2026051307253283000_16.5.e108775.4","unstructured":"Evaluation of ChatGPT Performance on Emergency Medicine Board Examination Questions: Observational Study. JMIR Preprints. Available: https:\/\/preprints.jmir.org\/preprint\/67696 [Accessed 3 Mar 2026]."},{"key":"2026051307253283000_16.5.e108775.5","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1177\/20514158241262081","article-title":"ChatGPT sitting for FRCS Urology examination: Will artificial intelligence get certified?","volume":"18","author":"Desouky","year":"2025","journal-title":"J Clin Urol"},{"key":"2026051307253283000_16.5.e108775.6","article-title":"Evaluating the Accuracy and Reliability of Large Language Models (ChatGPT, Claude, DeepSeek, Gemini, Grok, and Le Chat) in Answering Item-Analyzed Multiple-Choice Questions on Blood Physiology","volume":"17","author":"Agarwal","year":"2025","journal-title":"Cureus"},{"key":"2026051307253283000_16.5.e108775.7","unstructured":"OpenAI . GPT-5 system card. 2026. Available: https:\/\/openai.com\/index\/gpt-5-system-card\/ [Accessed 3 Mar 2026]."},{"key":"2026051307253283000_16.5.e108775.8","unstructured":"Pichai S . A new era of intelligence with Gemini 3. Google. 2025. Available: https:\/\/blog.google\/products-and-platforms\/products\/gemini\/gemini-3\/ [Accessed 3 Mar 2026]."},{"key":"2026051307253283000_16.5.e108775.9","unstructured":"Anthropic . System card: claude sonnet 4.6. 2025. Available: https:\/\/blog.google\/products-and-platforms\/products\/gemini\/gemini-3\/ [Accessed 3 Mar 2026]."},{"key":"2026051307253283000_16.5.e108775.10","unstructured":"xAI . Grok 4.1 model card. Available: https:\/\/data.x.ai\/2025-11-17-grok-4-1-model-card.pdf [Accessed 3 Mar 2026]."},{"key":"2026051307253283000_16.5.e108775.11","unstructured":"The Royal College of Surgeons of Edinburgh . Joint committee on intercollegiate examinations\u2019 urology syllabus blueprint. Available: https:\/\/www.jcie.org.uk\/content\/content.aspx?ID=21 [Accessed 3 Mar 2026]."},{"key":"2026051307253283000_16.5.e108775.12","doi-asserted-by":"crossref","unstructured":"Shyr C , Ren B , Hsu C-Y , et al . A statistical framework for evaluating the repeatability and reproducibility of large language models. medRxiv 2025.:2025.08.06.25333170. doi:10.1101\/2025.08.06.25333170","DOI":"10.1101\/2025.08.06.25333170"},{"key":"2026051307253283000_16.5.e108775.13","doi-asserted-by":"crossref","unstructured":"Christodoulopoulos C , Chakraborty T , Rose C , eds. Findings of the Association for Computational Linguistics: EMNLP 2025. Suzhou, China: Association for Computational Linguistics. In: 2025 doi:10.18653\/v1\/2025.findings-emnlp","DOI":"10.18653\/v1\/2025.findings-emnlp"},{"key":"2026051307253283000_16.5.e108775.14","doi-asserted-by":"crossref","unstructured":"Horovicz M , Goldshmidt R . TokenSHAP: interpreting large language models with monte carlo shapley value estimation. Proceedings of the 1st Workshop on NLP for Science (NLP4Science); Miami, FL, USA, 2024 doi:10.18653\/v1\/2024.nlp4science-1.1 Available: https:\/\/aclanthology.org\/2024.nlp4science-1","DOI":"10.18653\/v1\/2024.nlp4science-1.1"},{"key":"2026051307253283000_16.5.e108775.15","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-025-31010-4","article-title":"Benchmarking large language models on the United States medical licensing examination for clinical reasoning and medical licensing scenarios","volume":"16","author":"Siam","year":"2025","journal-title":"Sci Rep"},{"key":"2026051307253283000_16.5.e108775.16","doi-asserted-by":"crossref","unstructured":"Wang S , Hu M , Li Q , et al . Capabilities of gpt\u20115 on multimodal medical reasoning. Vancouver, Canada, doi:10.1117\/12.3086794","DOI":"10.1117\/12.3086794"},{"key":"2026051307253283000_16.5.e108775.17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3697010","article-title":"An Empirical Study of the Non-Determinism of ChatGPT in Code Generation","volume":"34","author":"Ouyang","year":"2025","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"2026051307253283000_16.5.e108775.18","doi-asserted-by":"crossref","DOI":"10.2196\/53559","article-title":"The Temperature Feature of ChatGPT: Modifying Creativity for Clinical Research","volume":"11","author":"Davis","year":"2024","journal-title":"JMIR Hum Factors"},{"key":"2026051307253283000_16.5.e108775.19","article-title":"Piloting Temperature-Driven Variability in Emergency Diagnostic Accuracy Using a Leading Large Language Model","volume":"17","author":"Jarrett","year":"2025","journal-title":"Cureus"}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2025-108775","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T14:26:24Z","timestamp":1778682384000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2025-108775"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5]]},"references-count":19,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,5,13]]},"published-print":{"date-parts":[[2026,5]]}},"alternative-id":["10.1136\/bmjopen-2025-108775"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2025-108775","relation":{},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5]]}}}