{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,11]],"date-time":"2026-11-11T01:20:33Z","timestamp":1794360033374,"version":"4.9.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,2,25]],"date-time":"2025-02-25T00:00:00Z","timestamp":1740441600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,2,25]],"date-time":"2025-02-25T00:00:00Z","timestamp":1740441600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Educ"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Recent advancements in generative artificial intelligence (AI) have opened new avenues in educational methodologies, particularly in medical education. This study seeks to assess whether generative AI might be useful in addressing the depletion of assessment question banks, a challenge intensified during the Covid-era due to the prevalence of open-book examinations, and to augment the pool of formative assessment opportunities available to students. While many recent publications have sought to ascertain whether AI can achieve a passing standard in existing examinations, this study investigates the potential for AI to generate the exam itself.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Summary of work<\/jats:title>\n                    <jats:p>This research utilized a commercially available AI large language model (LLM), OpenAI GPT-4, to generate 220 single best answer (SBA) questions, adhering to Medical Schools Council Assessment Alliance guidelines the and a selection of Learning Outcomes (LOs) of the Scottish Graduate-Entry Medicine (ScotGEM) program. All questions were assessed by an expert panel for accuracy and quality. A total of 50 AI-generated and 50 human-authored questions were used to create two 50-item formative SBA examinations for Year 1 and Year 2 ScotGEM students. Each exam, delivered via the Speedwell eSystem, comprised 25 AI-generated and 25 human-authored questions presented in random order. Students completed the online, closed-book exams on personal devices under exam conditions that reflected summative examinations. The performance of both AI-generated and human-authored questions was evaluated, focusing on facility and discrimination index as key metrics.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Summary of results<\/jats:title>\n                    <jats:p>The screening process revealed that 69% of AI-generated SBAs were fit for inclusion in the examinations with little or no modifications required. Modifications, when necessary, were predominantly due to reasons such as the inclusion of \"all of the above\" options, usage of American English spellings, and non-alphabetized answer choices. 31% of questions were rejected for inclusion in the examinations, due to factual inaccuracies and non-alignment with students\u2019 learning. When included in an examination, post hoc statistical analysis indicated no significant difference in performance between the AI- and human- authored questions in terms of facility and discrimination index.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion and conclusion<\/jats:title>\n                    <jats:p>The outcomes of this study suggest that AI LLMs can generate SBA questions that are in line with best-practice guidelines and specific LOs. However, a robust quality assurance process is necessary to ensure that erroneous questions are identified and rejected. The insights gained from this research provide a foundation for further investigation into refining AI prompts, aiming for a more reliable generation of curriculum-aligned questions. LLMs show significant potential in supplementing traditional methods of question generation in medical education. This approach offers a viable solution to rapidly replenish and diversify assessment resources in medical curricula, marking a step forward in the intersection of AI and education.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12909-025-06881-w","type":"journal-article","created":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T23:58:07Z","timestamp":1740441487000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Quality assurance and validity of AI-generated single best answer questions"],"prefix":"10.1186","volume":"25","author":[{"given":"Ayla","family":"Ahmed","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ellen","family":"Kerr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"O\u2019Malley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,2,25]]},"reference":[{"issue":"23","key":"6881_CR1","doi-asserted-by":"publisher","first-page":"8410","DOI":"10.1073\/pnas.1319030111","volume":"111","author":"S Freeman","year":"2014","unstructured":"Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A. 2014;111(23):8410\u20135.","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"6018","key":"6881_CR2","doi-asserted-by":"publisher","first-page":"772","DOI":"10.1126\/science.1199327","volume":"331","author":"JD Karpicke","year":"2011","unstructured":"Karpicke JD, Blunt JR. Retrieval practice produces more learning than elaborative studying with concept mapping. Science. 2011;331(6018):772\u20135.","journal-title":"Science"},{"issue":"7","key":"6881_CR3","doi-asserted-by":"publisher","first-page":"784","DOI":"10.1080\/09658211.2013.831454","volume":"22","author":"MA Smith","year":"2014","unstructured":"Smith MA, Karpicke JD. Retrieval practice with short-answer, multiple-choice, and hybrid tests. Memory. 2014;22(7):784\u2013802.","journal-title":"Memory"},{"issue":"3","key":"6881_CR4","doi-asserted-by":"publisher","first-page":"118","DOI":"10.4103\/0019-5359.95934","volume":"64","author":"AM Mujeeb","year":"2010","unstructured":"Mujeeb AM, Pardeshi ML, Ghongane BB. Comparative assessment of multiple choice questions versus short essay questions in pharmacology examinations. Indian J Med Sci. 2010;64(3):118\u201324.","journal-title":"Indian J Med Sci"},{"issue":"1","key":"6881_CR5","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1111\/teth.12318","volume":"19","author":"MH Bassett","year":"2016","unstructured":"Bassett MH. Teaching Critical Thinking without (Much) Writing: Multiple-Choice and Metacognition. Teach Theol Relig. 2016;19(1):20\u201340.","journal-title":"Teach Theol Relig"},{"issue":"1","key":"6881_CR6","first-page":"39","volume":"5","author":"MU Khan","year":"2011","unstructured":"Khan MU, Aljarallah BM. Evaluation of Modified Essay Questions (MEQ) and Multiple Choice Questions (MCQ) as a tool for Assessing the Cognitive Skills of Undergraduate Medical Students. Int J Health Sci (Qassim). 2011;5(1):39\u201343.","journal-title":"Int J Health Sci (Qassim)"},{"key":"6881_CR7","unstructured":"Medical Schools Council Assessment Alliance. Applied Knowledge Test Style Guide (v2). 2022."},{"issue":"1","key":"6881_CR8","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1186\/s12909-024-05239-y","volume":"24","author":"Y Artsi","year":"2024","unstructured":"Artsi Y, Sorin V, Konen E, Glicksberg BS, Nadkarni G, Klang E. Large language models for generating medical examinations: systematic review. BMC Med Educ. 2024;24(1):354.","journal-title":"BMC Med Educ"},{"issue":"Suppl 1","key":"6881_CR9","doi-asserted-by":"publisher","first-page":"S85","DOI":"10.1016\/j.mjafi.2020.11.007","volume":"77","author":"D Kumar","year":"2021","unstructured":"Kumar D, Jaipurkar R, Shekhar A, Sikri G, Srinivas V. Item analysis of multiple choice questions: A quality assurance test for an assessment tool. Med J Armed Forces India. 2021;77(Suppl 1):S85\u20139.","journal-title":"Med J Armed Forces India"},{"issue":"2","key":"6881_CR10","doi-asserted-by":"publisher","first-page":"67","DOI":"10.47102\/annals-acadmedsg.V35N2p67","volume":"35","author":"SM Sim","year":"2006","unstructured":"Sim SM, Rasiah RI. Relationship between item difficulty and discrimination indices in true\/false-type multiple choice questions of a para-clinical multidisciplinary paper. Ann Acad Med Singap. 2006;35(2):67\u201371.","journal-title":"Ann Acad Med Singap"},{"issue":"5","key":"6881_CR11","doi-asserted-by":"publisher","first-page":"654","DOI":"10.1016\/j.ejvs.2017.07.012","volume":"54","author":"PA Coughlin","year":"2017","unstructured":"Coughlin PA, Featherstone CR. How to write a high quality Multiple Choice Question (MCQ): a guide for clinicians. Eur J Vasc Endovasc Surg. 2017;54(5):654\u20138.","journal-title":"Eur J Vasc Endovasc Surg"},{"issue":"1","key":"6881_CR12","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1186\/s12909-016-0773-3","volume":"16","author":"BR Rush","year":"2016","unstructured":"Rush BR, Rankin DC, White BJ. The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Med Educ. 2016;16(1):250.","journal-title":"BMC Med Educ"},{"issue":"30","key":"6881_CR13","doi-asserted-by":"publisher","first-page":"e2305016120","DOI":"10.1073\/pnas.2305016120","volume":"120","author":"F Gilardi","year":"2023","unstructured":"Gilardi F, Alizadeh M, Kubli M. ChatGPT outperforms crowd workers for text-annotation tasks. Proc Natl Acad Sci U S A. 2023;120(30):e2305016120.","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"2","key":"6881_CR14","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1007\/s40593-018-00172-w","volume":"29","author":"J Leo","year":"2019","unstructured":"Leo J, Kurdi G, Matentzoglu N, Parsia B, Sattler U, Forge S, et al. Ontology-based generation of medical, multi-term MCQs. Int J Artif Intell Educ. 2019;29(2):145\u201388.","journal-title":"Int J Artif Intell Educ"},{"issue":"9","key":"6881_CR15","doi-asserted-by":"publisher","first-page":"604","DOI":"10.47102\/annals-acadmedsg.V35N9p604","volume":"35","author":"E Palmer","year":"2006","unstructured":"Palmer E, Devitt P. Constructing multiple choice questions as a method for learning. Ann Acad Med Singap. 2006;35(9):604\u20138.","journal-title":"Ann Acad Med Singap"},{"key":"6881_CR16","doi-asserted-by":"publisher","first-page":"238212052096525","DOI":"10.1177\/2382120520965255","volume":"7","author":"AM Monaghan","year":"2020","unstructured":"Monaghan AM. Medical Teaching and Assessment in the Era of COVID-19. J Med Educ Curric Dev. 2020;7:238212052096525.","journal-title":"J Med Educ Curric Dev"},{"key":"6881_CR17","doi-asserted-by":"crossref","unstructured":"Plevris V, Papazafeiropoulos G, Jim\u00e9nez Rios A. Chatbots Put to the Test in Math and Logic Problems: A Comparison and Assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. Ai. 2023;4(4):949\u201369.","DOI":"10.3390\/ai4040048"},{"key":"6881_CR18","doi-asserted-by":"crossref","unstructured":"Gordon M, Daniel M, Ajiboye A, Uraiby H, Xu NY, Bartlett R, et al. A scoping review of artificial intelligence in medical education: BEME Guide No. 84. Med Teach. 2024;46(4):446\u201370.","DOI":"10.1080\/0142159X.2024.2314198"},{"key":"6881_CR19","doi-asserted-by":"publisher","unstructured":"Aster A, Laupichler MC, Rockwell-Kollmann T, Masala G, Bala E, Raupach T. ChatGPT and other large language models in medical education \u2014 scoping literature review. Medical Science Educator. 2024. https:\/\/doi.org\/10.1007\/s40670-024-02206-6.","DOI":"10.1007\/s40670-024-02206-6"},{"issue":"2","key":"6881_CR20","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","volume":"2","author":"TH Kung","year":"2023","unstructured":"Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2): e0000198.","journal-title":"PLOS Digit Health"},{"issue":"1","key":"6881_CR21","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1037\/h0057123","volume":"30","author":"TL Kelley","year":"1939","unstructured":"Kelley TL. The selection of upper and lower groups for the validation of test items. J Educ Psychol. 1939;30(1):17\u201324.","journal-title":"J Educ Psychol"},{"key":"6881_CR22","doi-asserted-by":"publisher","unstructured":"Bedi S, Fleming SL, Chiang C-C, Morse K, Kumar A, Patel B, et al. QUEST-AI: a system for question generation, verification, and refinement using AI for USMLE-Style Exams. medRxiv. 2024. https:\/\/doi.org\/10.1101\/2023.04.25.23288588.","DOI":"10.1101\/2023.04.25.23288588"},{"issue":"1","key":"6881_CR23","doi-asserted-by":"publisher","first-page":"772","DOI":"10.1186\/s12909-023-04752-w","volume":"23","author":"E Klang","year":"2023","unstructured":"Klang E, Portugez S, Gross R, Kassif LR, Brenner A, Gilboa M, et al. Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4. BMC Med Educ. 2023;23(1):772.","journal-title":"BMC Med Educ."},{"issue":"5","key":"6881_CR24","doi-asserted-by":"publisher","first-page":"508","DOI":"10.1097\/ACM.0000000000005626","volume":"99","author":"MC Laupichler","year":"2024","unstructured":"Laupichler MC, Rother JF, Grunwald Kadow IC, Ahmadi S, Raupach T. Large language models in medical education: comparing ChatGPT- to human-generated exam questions. Acad Med. 2024;99(5):508\u201312.","journal-title":"Acad Med"},{"issue":"11","key":"6881_CR25","doi-asserted-by":"publisher","first-page":"1224","DOI":"10.1080\/0142159X.2023.2249239","volume":"45","author":"M Zuckerman","year":"2023","unstructured":"Zuckerman M, Flood R, Tan RJB, Kelp N, Ecker DJ, Menke J, et al. ChatGPT for assessment writing. Med Teach. 2023;45(11):1224\u20137.","journal-title":"Med Teach"},{"issue":"1189","key":"6881_CR26","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1093\/postmj\/qgae065","volume":"100","author":"YS Kiyak","year":"2024","unstructured":"Kiyak YS, Emekli E. ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review. Postgrad Med J. 2024;100(1189):858\u201365.","journal-title":"Postgrad Med J"},{"key":"6881_CR27","doi-asserted-by":"publisher","unstructured":"Co\u015fkun \u00d6, K\u0131yak YS, Budako\u011flu I\u0130. ChatGPT to generate clinical vignettes for teaching and multiple-choice questions for assessment: a randomized controlled experiment. Med Teach. 2025;47(2):268\u201374. https:\/\/doi.org\/10.1080\/0142159X.2024.2327477.","DOI":"10.1080\/0142159X.2024.2327477"},{"key":"6881_CR28","doi-asserted-by":"publisher","first-page":"e58275","DOI":"10.2196\/58275","volume":"3","author":"A O'Malley","year":"2024","unstructured":"O\u2019Malley A, Veenhuizen M, Ahmed A. Ensuring appropriate representation in artificial intelligence-generated medical imagery: protocol for a methodological approach to address skin tone bias. JMIR AI. 2024;3:e58275.","journal-title":"JMIR AI"},{"key":"6881_CR29","doi-asserted-by":"publisher","unstructured":"Fan BE, Chow M, Winkler S. Artificial Intelligence-Generated Facial Images for Medical Education. Med Sci Educ. 2023;34(1):5\u20137. 2023. https:\/\/doi.org\/10.1007\/s40670-023-01942-5.","DOI":"10.1007\/s40670-023-01942-5"},{"issue":"1","key":"6881_CR30","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1001\/jamasurg.2023.5695","volume":"159","author":"R Ali","year":"2024","unstructured":"Ali R, Tang OY, Connolly ID, Abdulrazeq HF, Mirza FN, Lim RK, et al. Demographic representation in 3 leading artificial intelligence text-to-image generators. JAMA Surg. 2024;159(1):87\u201395.","journal-title":"JAMA Surg"},{"key":"6881_CR31","doi-asserted-by":"publisher","unstructured":"M'gadzah SAT, O'Malley A Does a complex prompt alter the diagnostic accuracy of common ophthalmological conditions by GPT-4? JMIR Preprints. https:\/\/doi.org\/10.2196\/preprints.64986.","DOI":"10.2196\/preprints.64986"}],"container-title":["BMC Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12909-025-06881-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12909-025-06881-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12909-025-06881-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T23:58:11Z","timestamp":1740441491000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmededuc.biomedcentral.com\/articles\/10.1186\/s12909-025-06881-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,25]]},"references-count":31,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["6881"],"URL":"https:\/\/doi.org\/10.1186\/s12909-025-06881-w","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-5666975\/v1","asserted-by":"object"}]},"ISSN":["1472-6920"],"issn-type":[{"value":"1472-6920","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,25]]},"assertion":[{"value":"18 December 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 February 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 February 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Ethical approval was awarded on 19 Oct 2024 by the School of Medicine Ethics Committee at University of St Andrews (Reference number MD17293). Since this study does not involve patients a clinical trial number was not applicable. All students received information about the study before attempting the exam, and a consent form was required to be completed. Students\u2019 exam responses were only used in this study if they provided consent. These provisions aligned our study with the Declaration of Helsinki. As there was a dependent relationship between the researcher (i.e. teachers\/assessors on the ScotGEM programme) and the students, it was made clear that withholding consent would not disadvantage the student and that they would be able to attempt the exams as normal.Students\u2019 individual responses were not anonymised to provide them with feedback after the conclusion of the exam; however, the identities of the students were not presented to the research team for the purposes of the post-hoc analysis. Participants were provided with their induvial exam feedback (privately) and the general findings of this study (during a whole-class briefing session). Students\u2019 responses and were stored securely on University cloud storage (OneDrive) and only accessible by the research team.Since this study is not a clinical trial we have not reported a clinical trial number.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"300"}}