{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T23:39:34Z","timestamp":1792193974924,"version":"4.4.0"},"reference-count":18,"publisher":"Radiological Society of North America (RSNA)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Radiology"],"published-print":{"date-parts":[[2024,5,1]]},"abstract":"<jats:p>Default GPT-3.5 and GPT-4 were reliably accurate over three attempts at answering radiology board\u2013style multiple-choice questions but had poor repeatability and robustness and were frequently overconfident, limiting usability without domain-specific optimization.<\/jats:p>","DOI":"10.1148\/radiol.232715","type":"journal-article","created":{"date-parts":[[2024,5,21]],"date-time":"2024-05-21T09:46:25Z","timestamp":1716284785000},"source":"Crossref","is-referenced-by-count":82,"title":["Evaluation of Reliability, Repeatability, Robustness, and Confidence                     of GPT-3.5 and GPT-4 on a Radiology Board\u2013style                     Examination"],"prefix":"10.1148","volume":"311","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6603-7621","authenticated-orcid":false,"given":"Satheesh","family":"Krishna","sequence":"first","affiliation":[{"name":"From the University Medical Imaging Toronto, Joint Department of                     Medical Imaging, University Health Network, Mount Sinai Hospital and                     Women\u2019s College Hospital, University of Toronto, Toronto General                     Hospital, 200 Elizabeth St, Toronto, ON, Canada M5G 24C (S.K., R. Bleakney, R.                     Bhayana); Department of Medical Imaging, University of Toronto, Toronto, ON,                     Canada (S.K., R. Bleakney, R. Bhayana); and Department of Family Medicine,                     University of Ottawa, Ottawa, ON, Canada (N.B.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4966-6578","authenticated-orcid":false,"given":"Nishaant","family":"Bhambra","sequence":"additional","affiliation":[{"name":"From the University Medical Imaging Toronto, Joint Department of                     Medical Imaging, University Health Network, Mount Sinai Hospital and                     Women\u2019s College Hospital, University of Toronto, Toronto General                     Hospital, 200 Elizabeth St, Toronto, ON, Canada M5G 24C (S.K., R. Bleakney, R.                     Bhayana); Department of Medical Imaging, University of Toronto, Toronto, ON,                     Canada (S.K., R. Bleakney, R. Bhayana); and Department of Family Medicine,                     University of Ottawa, Ottawa, ON, Canada (N.B.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4493-2676","authenticated-orcid":false,"given":"Robert","family":"Bleakney","sequence":"additional","affiliation":[{"name":"From the University Medical Imaging Toronto, Joint Department of                     Medical Imaging, University Health Network, Mount Sinai Hospital and                     Women\u2019s College Hospital, University of Toronto, Toronto General                     Hospital, 200 Elizabeth St, Toronto, ON, Canada M5G 24C (S.K., R. Bleakney, R.                     Bhayana); Department of Medical Imaging, University of Toronto, Toronto, ON,                     Canada (S.K., R. Bleakney, R. Bhayana); and Department of Family Medicine,                     University of Ottawa, Ottawa, ON, Canada (N.B.)."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8352-7953","authenticated-orcid":false,"given":"Rajesh","family":"Bhayana","sequence":"additional","affiliation":[{"name":"From the University Medical Imaging Toronto, Joint Department of                     Medical Imaging, University Health Network, Mount Sinai Hospital and                     Women\u2019s College Hospital, University of Toronto, Toronto General                     Hospital, 200 Elizabeth St, Toronto, ON, Canada M5G 24C (S.K., R. Bleakney, R.                     Bhayana); Department of Medical Imaging, University of Toronto, Toronto, ON,                     Canada (S.K., R. Bleakney, R. Bhayana); and Department of Family Medicine,                     University of Ottawa, Ottawa, ON, Canada (N.B.)."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1577","reference":[{"key":"r2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"r3","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232756"},{"key":"r4","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230582"},{"key":"r5","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.230987"},{"key":"r6","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacr.2023.05.003"},{"key":"r7","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2023.1838"},{"key":"r8","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.231040"},{"key":"r9","unstructured":"Zhou K, Jurafsky D, Hashimoto T. Navigating                         the Grey Area: Expressions of Overconfidence and Uncertainty in Language                         Models. arXiv 2302.13439 [preprint]                             https:\/\/arxiv.org\/abs\/2302.13439. Published February 26,                         2023. Accessed September 29, 2023."},{"key":"r12","doi-asserted-by":"publisher","DOI":"10.1086\/459563"},{"key":"r13","unstructured":"Anderson LW, Krathwohl DR. A                         taxonomy for learning, teaching, and assessing: A revision of Bloom\u2019s                         taxonomy of educational objectives: complete                         edition. Addison Wesley                         Longman, 2001."},{"key":"r14","unstructured":"Casper S, Bailey L. Examples                         of Prompts that Make GPT-4 Output                             Falsehoods. https:\/\/www.alignmentforum.org\/posts\/iNvWCTpyEd4zTqjjv\/examples-of-prompts-that-make-gpt-4-output-falsehoods.                         Published July 22, 2023. Accessed September 29,                     2023."},{"key":"r15","doi-asserted-by":"publisher","DOI":"10.11613\/BM.2012.031"},{"key":"r16","unstructured":"Chen L, Zaharia M, Zou J. How                         is ChatGPT\u2019s behavior changing over                         time? arXiv 2307.09009 [preprint] https:\/\/arxiv.org\/abs\/2307.09009. Published July 18, 2023.                         Accessed September 29, 2023."},{"key":"r17","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232411"},{"key":"r18","doi-asserted-by":"publisher","DOI":"10.1177\/08465371231222229"},{"key":"r19","unstructured":"Jiang AQ, Sablayrolles A, Mensch A, et                             al. Mistral                         7B. arXiv 2310.06825 [preprint] https:\/\/arxiv.org\/abs\/2310.06825. Published October 10,                         2023. Accessed January 10, 2024."},{"key":"r21","doi-asserted-by":"publisher","DOI":"10.1002\/acp.3320"},{"key":"r22","unstructured":"Madaan A, Tandon N, Gupta P, et                             al. Self-Refine: Iterative                         Refinement with Self-Feedback. arXiv                         2303.17651 [preprint] https:\/\/arxiv.org\/abs\/2303.17651. Published March 30, 2023.                         Accessed September 29, 2023."}],"container-title":["Radiology"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/pubs.rsna.org\/doi\/pdf\/10.1148\/radiol.232715","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,24]],"date-time":"2025-01-24T16:37:05Z","timestamp":1737736625000},"score":1,"resource":{"primary":{"URL":"http:\/\/pubs.rsna.org\/doi\/10.1148\/radiol.232715"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,1]]},"references-count":18,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,5,1]]}},"alternative-id":["10.1148\/radiol.232715"],"URL":"https:\/\/doi.org\/10.1148\/radiol.232715","relation":{},"ISSN":["0033-8419","1527-1315"],"issn-type":[{"value":"0033-8419","type":"print"},{"value":"1527-1315","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,1]]},"article-number":"e232715"}}