{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,10]],"date-time":"2026-08-10T21:28:48Z","timestamp":1786397328855,"version":"build-2736575974"},"reference-count":22,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Generative artificial intelligence (GenAI) is increasingly used to draft multiple-choice questions (MCQs) for health professions education, but much evidence concerns raw model outputs, expert ratings, or item difficulty alone. Educators edit GenAI drafts before use, and whether such items are psychometrically ready for postgraduate assessment remains unclear.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to compare human-edited GenAI-assisted and educator-crafted MCQs for postgraduate Family Medicine Applied Knowledge Test\u2013level assessment, examining difficulty, discrimination, reliability, distractor functioning, and participant perceptions.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We conducted a blinded cross-sectional, within-participant comparative psychometric evaluation in Singapore. Sixty best-of-five single-best-answer MCQs were evaluated, 30 human-edited GenAI-assisted items and 30 educator-crafted items, topic-matched across postgraduate FM domains and randomized across 2 assessment sets. Eligible participants were postgraduate doctors enrolled in FM residency or postgraduate family medicine programs, preparing for the Applied Knowledge Test, and blinded to item origin; incomplete paired responses were excluded. Outcomes included paired total scores, score correlation and agreement, Kuder-Richardson Formula 20 reliability, item difficulty index, corrected point-biserial discrimination, distractor functioning, and perceived difficulty, clarity, and relevance. Analyses used paired-sample tests, Pearson correlation, Fisher exact tests, and item-level psychometric statistics, with \u03b1=.05 and Bonferroni correction within comparison families.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Of 74 participants, 73 completed both item sets and were included in the analysis. The final sample comprised 36 graduate diploma in FM trainees, 5 MMed FM trainees, and 32 FM residents. Paired-sample testing showed lower scores on GenAI-assisted than educator-crafted items (mean 19.12, SD 2.83 vs mean 21.10, SD 3.42 out of 30; mean difference \u22121.97, 95% CI \u22122.72 to \u22121.23;\n P<\/jats:italic>\n <.001; Cohen\n d<\/jats:italic>\n =0.62), indicating that GenAI-assisted items were not easier. Scores were positively correlated (\n r<\/jats:italic>\n =0.49, 95% CI 0.30\u20100.64;\n P<\/jats:italic>\n <.001), but Bland-Altman analysis indicated limited agreement. Kuder-Richardson Formula 20 reliability was lower for GenAI-assisted items (0.38 vs 0.60). Mean difficulty index did not differ significantly (0.64 vs 0.70; mean difference \u22120.07, 95% CI \u22120.19 to 0.06;\n P<\/jats:italic>\n =.29), and more GenAI-assisted items fell within the acceptable difficulty range (18\/30, 60.0% vs 13\/30, 43.3%). However, mean corrected point-biserial discrimination was lower for GenAI-assisted items (0.09 vs 0.18; mean difference \u22120.08, 95% CI \u22120.16 to \u22120.01;\n P<\/jats:italic>\n =.04), and negative discrimination was more common (6\/30, 20% vs 3\/30, 10%). GenAI-assisted items also had more nonfunctioning and negatively discriminating distractors, although these differences were not statistically significant. Participant ratings of perceived difficulty, clarity, and practice relevance did not differ by origin.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Human-edited GenAI-assisted MCQs can achieve plausible difficulty, but difficulty and surface acceptability did not ensure assessment readiness. Using trainee response data, this study extends work on raw outputs or expert opinion. GenAI should be used as a drafting adjunct within educator-led workflows prioritizing key verification, distractor engineering, pilot testing, empirical item analysis, and repair before item-bank or summative use.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/100179","type":"journal-article","created":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T11:15:05Z","timestamp":1784546105000},"page":"e100179-e100179","source":"Crossref","is-referenced-by-count":0,"title":["Human-Edited Generative AI-Assisted Multiple-Choice Questions in Postgraduate Family Medicine: Blinded Cross-Sectional Comparative Psychometric Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-3377-1973","authenticated-orcid":false,"given":"Kai Ping","family":"Sze","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2965-8993","authenticated-orcid":false,"given":"Jia Qing","family":"Lim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9102-9981","authenticated-orcid":false,"given":"Yng Miin","family":"Loke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7983-7249","authenticated-orcid":false,"given":"Teck Yong Gabriel","family":"Ding","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6252-798X","authenticated-orcid":false,"given":"Wei Chieh","family":"Wee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6866-3298","authenticated-orcid":false,"given":"Tongyuan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-4631-4937","authenticated-orcid":false,"given":"Jingkai Lawrence","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3556-643X","authenticated-orcid":false,"given":"Vivek","family":"Bansal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3424-8915","authenticated-orcid":false,"given":"Qi Wei","family":"Fong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9398-7656","authenticated-orcid":false,"given":"Minyang","family":"Chow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,10]]},"reference":[{"key":"R1","unstructured":"Division of Graduate Medical Studies, Yong Loo Lin School of Medicine, National University of Singapore. Family Medicine Applied Knowledge Test Handbook. Division of Graduate Medical Studies, Yong Loo Lin School of Medicine, National University of Singapore; 2025. URL: https:\/\/medicine.nus.edu.sg\/dgms\/wp-content\/uploads\/sites\/30\/2022\/02\/FM-AKT-Handbook.pdf [Accessed 08-02-2026]"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-06796-6","article-title":"AI versus human-generated multiple-choice questions for medical education: a cohort study in a high-stakes examination","volume":"25","author":"Law","journal-title":"BMC Med Educ"},{"issue":"12","key":"R3","doi-asserted-by":"publisher","first-page":"1961","DOI":"10.1080\/0142159X.2025.2505122","article-title":"GPT-4 versus human authors in clinically complex MCQ creation: a blinded analysis of item quality","volume":"47","author":"Wu","journal-title":"Med Teach"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02313-7","article-title":"Psychometric properties and detectability of GPT-4o-generated multiple-choice questions compared with human-authored items across imaging specialties","volume":"9","author":"Linde","journal-title":"NPJ Digit Med"},{"issue":"12","key":"R5","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0001156","article-title":"Evaluating ChatGPT-4 in the development of family medicine residency examinations","volume":"4","author":"Chaudhari","journal-title":"PLOS Digit Health"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-28492-7","article-title":"Benchmarking ChatGPT-generated multiple-choice questions against faculty-authored items in dental education","volume":"15","author":"Kiyani","journal-title":"Sci Rep"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-025-07528-6","article-title":"Comparison of AI-generated and clinician-designed multiple-choice questions in emergency medicine exam: a psychometric analysis","volume":"25","author":"Kaya","journal-title":"BMC Med Educ"},{"issue":"1189","key":"R8","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1093\/postmj\/qgae065","article-title":"ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature review","volume":"100","author":"K\u0131yak","journal-title":"Postgrad Med J"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0340277","article-title":"The use of large language models in generating multiple choice questions for health professions education: a systematic review and network meta-analysis","volume":"21","author":"Riehm","journal-title":"PLoS ONE"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"42096","DOI":"10.1038\/s41598-025-26036-7","article-title":"The pitfalls of multiple-choice questions in generative AI and medical education","volume":"15","author":"Singh","journal-title":"Sci Rep"},{"key":"R11","unstructured":"GPT-51: a smarter, more conversational ChatGPT. OpenAI. Nov 2025. URL: https:\/\/openai.com\/index\/gpt-5-1\/ [Accessed 18-11-2025]"},{"key":"R12","doi-asserted-by":"crossref","unstructured":"Salkind NJ . Encyclopedia of Research Design. SAGE Publications; 2010. [doi: 10.4135\/9781412961288.n205]","DOI":"10.4135\/9781412961288"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1080\/14739879.2012.11494065","article-title":"Testing the test: an analysis of the MRCGP Applied Knowledge Test as an assessment tool","volume":"23","author":"Metcalfe","journal-title":"Educ Prim Care"},{"issue":"4","key":"R14","doi-asserted-by":"publisher","first-page":"1045","DOI":"10.1152\/advan.00197.2025","article-title":"Faculty versus artificial intelligence chatbot: a comparative analysis of multiple-choice question quality in physiology","volume":"49","author":"Dhanvijay","journal-title":"Adv Physiol Educ"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1111\/jedm.12000","article-title":"Validating the interpretations and uses of test scores","volume":"50","author":"Kane","journal-title":"J Educ Meas"},{"issue":"6","key":"R16","doi-asserted-by":"publisher","first-page":"560","DOI":"10.1111\/medu.12678","article-title":"A contemporary approach to validity arguments: a practical guide to Kane\u2019s framework","volume":"49","author":"Cook","journal-title":"Med Educ"},{"issue":"2","key":"R17","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1007\/s10459-004-4019-5","article-title":"The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education","volume":"10","author":"Downing","journal-title":"Adv Health Sci Educ Theory Pract"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-016-0773-3","article-title":"The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value","volume":"16","author":"Rush","journal-title":"BMC Med Educ"},{"issue":"3","key":"R19","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1207\/S15324818AME1503_5","article-title":"A review of multiple-choice item-writing guidelines for classroom assessment","volume":"15","author":"Haladyna","journal-title":"Appl Meas Educ"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-9-40","article-title":"An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis","volume":"9","author":"Tarrant","journal-title":"BMC Med Educ"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","first-page":"52","DOI":"10.36834\/cmej.77596","article-title":"The impact of systematically repairing multiple choice questions with low discrimination on assessment reliability: an interrupted time series analysis","volume":"15","author":"Desy","journal-title":"Can Med Ed J"},{"issue":"10","key":"R22","doi-asserted-by":"publisher","first-page":"959","DOI":"10.1111\/j.1365-2923.2008.03124.x","article-title":"Test-enhanced learning in medical education","volume":"42","author":"Larsen","journal-title":"Med Educ"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,10]],"date-time":"2026-08-10T20:52:37Z","timestamp":1786395157000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e100179"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,10]]},"references-count":22,"URL":"https:\/\/doi.org\/10.2196\/100179","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,10]]},"article-number":"v12i1e100179"}}