{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T05:38:44Z","timestamp":1792820324966,"version":"4.5.0"},"reference-count":18,"publisher":"SAGE Publications","issue":"11","license":[{"start":{"date-parts":[[2026,4,19]],"date-time":"2026-04-19T00:00:00Z","timestamp":1776556800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The American Surgeon\u2122"],"published-print":{"date-parts":[[2026,11]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) have demonstrated strong performance on general medical and surgical examinations; however, their capacity to accurately interpret and apply subspecialty clinical practice guidelines remains incompletely characterized.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To evaluate and compare the accuracy and consistency of two contemporary LLMs\u2014Google Gemini and OpenEvidence\u2014using multiple-choice questions (MCQs) derived directly from the 2022 American Society of Colon and Rectal Surgeons (ASCRS) Clinical Practice Guidelines for anorectal abscess, fistula-in-ano, and rectovaginal fistula.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      Thirty guideline-based MCQs were developed and independently validated by surgeon reviewers. Each question was presented to both models under identical conditions without additional prompting. Accuracy was calculated with 95% confidence intervals and compared against chance performance (p\n                      <jats:sub>0<\/jats:sub>\n                      = .25). Inter-model agreement was assessed using Cohen\u2019s kappa coefficient.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Both Gemini and OpenEvidence correctly answered 29 of 30 questions (96.7%; 95% CI, 0.83-0.999), significantly exceeding chance performance (\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      &lt; .0001 for both models). Both models missed the same question, yielding perfect inter-model agreement (\u03ba = 1.0).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>In this focused pilot study restricted to ASCRS anorectal disease guidelines, both LLMs demonstrated near-perfect and statistically equivalent accuracy. These findings suggest that contemporary LLMs can accurately apply subspecialty surgical guidelines within a narrow domain, though broader, multi-guideline evaluations are required before generalization.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/00031348261443342","type":"journal-article","created":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T06:32:22Z","timestamp":1776666742000},"page":"2859-2862","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Artificial Intelligence in Surgical Education: A Pilot Study Using ASCRS Guideline-Derived Questions"],"prefix":"10.1177","volume":"92","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-3122-2150","authenticated-orcid":false,"given":"Shivam","family":"Pandya","sequence":"first","affiliation":[{"name":"Los Robles Regional Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tyler","family":"Wilson","sequence":"additional","affiliation":[{"name":"Los Robles Regional Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryan","family":"Meyer","sequence":"additional","affiliation":[{"name":"Los Robles Regional Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8397-9186","authenticated-orcid":false,"given":"Tamir E.","family":"Bresler","sequence":"additional","affiliation":[{"name":"Los Robles Regional Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manabu","family":"Fujita","sequence":"additional","affiliation":[{"name":"Los Robles Regional Medical Center"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,4,19]]},"reference":[{"issue":"2","key":"e_1_3_4_2_2","first-page":"e0000198","article-title":"Performance of ChatGPT on the United States medical licensing examination","volume":"2","author":"Kung TH","year":"2023","unstructured":"Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on the United States medical licensing examination. PLOS Digit Health. 2023;2(2):e0000198.","journal-title":"PLOS Digit Health"},{"key":"e_1_3_4_3_2","unstructured":"Nori H King N McKinney SM et al. Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. 2023."},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.4174\/astr.2023.104.5.269"},{"issue":"4","key":"e_1_3_4_6_2","first-page":"e199","article-title":"Artificial intelligence performance on American board of thoracic surgery examinations","volume":"166","author":"Khalpey Z","year":"2023","unstructured":"Khalpey Z, Al-Shakhshir T, Khosya A, et al. Artificial intelligence performance on American board of thoracic surgery examinations. J Thorac Cardiovasc Surg. 2023;166(4):e199-e207.","journal-title":"J Thorac Cardiovasc Surg"},{"issue":"10","key":"e_1_3_4_7_2","first-page":"1956","article-title":"Large language models in orthopaedic surgery education","volume":"481","author":"Nawari AM","year":"2023","unstructured":"Nawari AM, Maassen A, Laitano E, et al. Large language models in orthopaedic surgery education. Clin Orthop Relat Res. 2023;481(10):1956-1965.","journal-title":"Clin Orthop Relat Res"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1097\/DCR.0000000000002473"},{"key":"e_1_3_4_9_2","volume-title":"Clinical Practice Guidelines we can Trust","author":"Institute of Medicine","year":"2011","unstructured":"Institute of Medicine. Clinical Practice Guidelines we can Trust. Washington, DC: National Academies Press; 2011."},{"issue":"8","key":"e_1_3_4_10_2","first-page":"e232938","article-title":"The role of clinical practice guidelines in value-based care","volume":"4","author":"Gliklich RE","year":"2023","unstructured":"Gliklich RE, Leavy MB, Dreyer NA. The role of clinical practice guidelines in value-based care. JAMA Health Forum. 2023;4(8):e232938.","journal-title":"JAMA Health Forum"},{"issue":"1","key":"e_1_3_4_11_2","first-page":"15","article-title":"Methodological challenges in evaluating large language models in medicine","volume":"30","author":"Chen Y","year":"2024","unstructured":"Chen Y, Ding L, Guo J, et al. Methodological challenges in evaluating large language models in medicine. Nat Med. 2024;30(1):15-18.","journal-title":"Nat Med"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.5317da47"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1009213286"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/26.4.404"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"issue":"2","key":"e_1_3_4_16_2","first-page":"237","article-title":"AtlasGPT: a domain-specific large language model for neurosurgery","volume":"94","author":"Soman L","year":"2024","unstructured":"Soman L, Mohanty A, Reddy V, et al. AtlasGPT: a domain-specific large language model for neurosurgery. Neurosurg. 2024;94(2):237-245.","journal-title":"Neurosurg"},{"issue":"1","key":"e_1_3_4_17_2","first-page":"1","article-title":"Artificial intelligence and the future of surgery","volume":"275","author":"Hashmi ZG","year":"2022","unstructured":"Hashmi ZG, Kaji AH, Nathens AB. Artificial intelligence and the future of surgery. Ann Surg. 2022;275(1):1-7.","journal-title":"Ann Surg"},{"issue":"15","key":"e_1_3_4_18_2","first-page":"1443","article-title":"Artificial intelligence hallucinations and patient safety","volume":"330","author":"Tsioutis I","year":"2023","unstructured":"Tsioutis I, Gkegkes ID. Artificial intelligence hallucinations and patient safety. JAMA. 2023;330(15):1443-1444.","journal-title":"JAMA"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1426-2"}],"container-title":["The American Surgeon\u2122"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00031348261443342","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/00031348261443342","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00031348261443342","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T05:10:36Z","timestamp":1792818636000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/00031348261443342"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,19]]},"references-count":18,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2026,11]]}},"alternative-id":["10.1177\/00031348261443342"],"URL":"https:\/\/doi.org\/10.1177\/00031348261443342","relation":{},"ISSN":["0003-1348","1555-9823"],"issn-type":[{"value":"0003-1348","type":"print"},{"value":"1555-9823","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,19]]}}}