{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:44:16Z","timestamp":1785545056639,"version":"3.56.0"},"reference-count":34,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Educ"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) have emerged as promising tools in medical education due to their ability to understand, generate, and reason with natural language. Their ability to simulate expert reasoning suggests a potential for supporting quality control in assessment design. In this study, the use of LLMs in identifying ambiguous or poorly constructed exam items in critical care academic assessments was evaluated.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The study aimed to develop automated ambiguity and quality scores to objectively assess individual questions and entire exam components.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We analyzed 264 questions from academic exams conducted over 3 academic years (2023-2025) at the Medical School of Universit\u00e9 C\u00f4te d\u2019Azur. Questions were drawn from 4 docimological formats: progressive clinical cases (PCC), mini-PCC, key feature problems, and isolated question sequences (IQS). Each element was submitted to 4 LLMs (ChatGPT, Gemini Pro, Le Chat, and DeepSeek) without prompt engineering. Performance was evaluated using the official correction key. We applied 4 binary diagnostic tags based on model agreement and self-reported ambiguity: ambiguity, low performance, incoherence, and subjective ambiguity. These tags generated a composite ambiguity score and contributed to a weighted quality score for each exam component.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n LLMs achieved mean scores in the same range as students, with no significant differences across academic years and significantly higher performance on the mini-PCC and IQS formats (\n P<\/jats:italic>\n =.049 and\n P<\/jats:italic>\n =.04, respectively). IQS items had the highest ambiguity scores (54 items received a score of 2 in both 2023 and 2024, and 53 items retained the same score). Tag patterns revealed frequent issues with ambiguity and inconsistency. Quality scores varied across academic years. IQS predominantly showed moderate ambiguity (score 2), with occasional instances of strong signals. There was no significant difference in quality based on author specialty or seniority (\n P<\/jats:italic>\n =.08 and\n P<\/jats:italic>\n =.44, respectively).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n In this pilot study, LLMs may offer a preliminary framework to proactively detect ambiguous exam questions and estimate the overall quality of an exam. Integrating these tools into the assessment design process could potentially reduce the need for postexam corrections and may help improve fairness and clarity in medical evaluations.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/82702","type":"journal-article","created":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T16:15:06Z","timestamp":1776701706000},"page":"e82702-e82702","source":"Crossref","is-referenced-by-count":0,"title":["Ambiguity Detection in Medical Exams via Large Language Models: Retrospective Cross-Sectional Pilot Study"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1101-8153","authenticated-orcid":false,"given":"Romain","family":"Lombardi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6147-9201","authenticated-orcid":false,"given":"Alexandre","family":"Destere","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2681-8260","authenticated-orcid":false,"given":"Jean","family":"Dellamonica","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6591-6966","authenticated-orcid":false,"given":"Alexandre O","family":"G\u00e9rard","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3379-4065","authenticated-orcid":false,"given":"Mathieu","family":"Jozwiak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,5,26]]},"reference":[{"issue":"2","key":"R1","doi-asserted-by":"publisher","first-page":"210","DOI":"10.7326\/M23-2772","article-title":"Large language models in medicine: the potentials and pitfalls: a narrative review","volume":"177","author":"Omiye","journal-title":"Ann Intern Med"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1016\/j.plrev.2024.10.013","article-title":"LLMs and generative agent-based models for complex systems research","volume":"51","author":"Lu","journal-title":"Phys Life Rev"},{"key":"R3","doi-asserted-by":"crossref","unstructured":"Ouyang L Wu J Jiang X et al. Training language models to follow instructions with human feedback. Presented at: 36th Conference on Neural Information Processing Systems (NeurIPS 2022); Nov 28 to Dec 9, 2022; New Orleans, Louisiana, USA. [doi: 10.52202\/068431-2011]","DOI":"10.52202\/068431-2011"},{"key":"R4","unstructured":"Gemini TeamAnil R Borgeaud S et al. Gemini: a family of highly capable multimodal models. arXiv. Preprint posted online on Dec 19, 2023. [doi: 10.48550\/ARXIV.2312.11805]"},{"key":"R5","unstructured":"DeepSeek-AILiu A Feng B et al. DeepSeek-V2: a strong, economical, and efficient mixture-of-experts language model. arXiv. Preprint posted online on May 7, 2024. [doi: 10.48550\/ARXIV.2405.04434]"},{"key":"R6","unstructured":"Jiang AQ Sablayrolles A Mensch A et al. Mistral 7B. arXiv. Preprint posted online on Oct 10, 2023. [doi: 10.48550\/ARXIV.2310.06825]"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"1565","DOI":"10.1007\/s00405-023-08337-7","article-title":"How ChatGPT works: a mini review","volume":"281","author":"Briganti","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"R8","unstructured":"Wu Y Schuster M Chen Z et al. Google\u2019s neural machine translation system: bridging the gap between human and machine translation. arXiv. Preprint posted online on Sep 26, 2016. [doi: 10.48550\/ARXIV.1609.08144]"},{"key":"R9","unstructured":"Chen M Tworek J Jun H et al. Evaluating large language models trained on code. arXiv. Preprint posted online on Jul 7, 2021. [doi: 10.48550\/ARXIV.2107.03374]"},{"issue":"8","key":"R10","doi-asserted-by":"publisher","first-page":"928","DOI":"10.1001\/jamasurg.2024.1621","article-title":"Large language model capabilities in perioperative risk prediction and prognostication","volume":"159","author":"Chung","journal-title":"JAMA Surg"},{"issue":"8","key":"R11","doi-asserted-by":"publisher","first-page":"1018","DOI":"10.1007\/s00134-023-07128-2","article-title":"Assessing the usefulness of a large language model to query and summarize unstructured medical notes in intensive care","volume":"49","author":"Madden","journal-title":"Intensive Care Med"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajem.2024.12.032","article-title":"Use of a large language model (LLM) for ambulance dispatch and triage","volume":"89","author":"Shekhar","journal-title":"Am J Emerg Med"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1177\/08850666241267871","article-title":"Evaluating the appropriateness, consistency, and readability of ChatGPT in critical care recommendations","volume":"40","author":"Balta","journal-title":"J Intensive Care Med"},{"issue":"3","key":"R14","doi-asserted-by":"publisher","first-page":"776","DOI":"10.1093\/jamia\/ocad252","article-title":"Harnessing the potential of large language models in medical education: promise and pitfalls","volume":"31","author":"Ben\u00edtez","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-02954-4","article-title":"A systematic review of large language model (LLM) evaluations in clinical medicine","volume":"25","author":"Shool","journal-title":"BMC Med Inform Decis Mak"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2024.02.016","article-title":"The operating and anesthetic reference assistant (OARA): a fine-tuned large language model for resident teaching","volume":"234","author":"Guthrie","journal-title":"Am J Surg"},{"issue":"7","key":"R17","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0307383","article-title":"Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians","volume":"19","author":"Hadi","journal-title":"PLoS One"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.2196\/59435","article-title":"Application of large language models in medical training evaluation-using ChatGPT as a standardized patient: multimetric assessment","volume":"27","author":"Wang","journal-title":"J Med Internet Res"},{"issue":"3","key":"R19","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-080558","article-title":"Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework","volume":"14","author":"Maitland","journal-title":"BMJ Open"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.1002\/bcp.70137","article-title":"Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: from exam takers to exam shapers","volume":"10","author":"G\u00e9rard","journal-title":"Br J Clin Pharmacol"},{"issue":"5","key":"R21","doi-asserted-by":"publisher","first-page":"302","DOI":"10.1016\/j.revmed.2020.11.006","article-title":"Progressive clinical case-based multiple-choice questions: an innovative way to evaluate and rank undergraduate medical students","volume":"42","author":"Cohen Aubart","journal-title":"Rev Med Interne"},{"key":"R22","unstructured":"Code de la sant\u00e9 publique [Article in French]. L\u00e9gifrance (R\u00c9PUBLIQUE FRAN\u00c7AISE Libert\u00e9, \u00c9galit\u00e9, Fraternit\u00e9). URL: https:\/\/www.legifrance.gouv.fr\/loda\/id\/LEGISCTA000006190263 [Accessed 15-05-2026]"},{"key":"R23","unstructured":"Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95\/46\/EC (General Data Protection Regulation) (text with EEA relevance). European Union; 2016. URL: https:\/\/eur-lex.europa.eu\/eli\/reg\/2016\/679\/oj\/eng [Accessed 15-05-2026]"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-03074-9","article-title":"Large language models\u2019 capabilities in responding to tuberculosis medical questions: testing ChatGPT, Gemini, and Copilot","volume":"15","author":"Dastani","journal-title":"Sci Rep"},{"issue":"12","key":"R25","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.6370","article-title":"Developing a beginner\u2019s guide to writing a clinical case report: a pilot evaluation by junior doctors","volume":"11","author":"Oyibo","journal-title":"Cureus"},{"issue":"12","key":"R26","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.3701","article-title":"A medical article publishing club for junior doctors: a quantitative and qualitative analysis","volume":"10","author":"Oyibo","journal-title":"Cureus"},{"issue":"9","key":"R27","doi-asserted-by":"publisher","first-page":"974","DOI":"10.1111\/j.1365-2929.2004.01916.x","article-title":"Different written assessment methods: What can be said about their strengths and weaknesses?","volume":"38","author":"Schuwirth","journal-title":"Med Educ"},{"issue":"1096","key":"R28","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1136\/postgradmedj-2015-133893","article-title":"Single best answer question\u2013writing tips for clinicians","volume":"93","author":"Walsh","journal-title":"Postgrad Med J"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Rezigalla AA . Item analysis: concept and application. In: Firstenberg MS Stawicki SP , editors. Medical Education for the 21st Century. IntechOpen; 2022:1-17. [doi: 10.5772\/intechopen.100138]","DOI":"10.5772\/intechopen.100138"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-022-03844-3","article-title":"Evaluation of the quality of multiple-choice questions according to the students\u2019 academic level","volume":"22","author":"I\u00f1arrairaegui","journal-title":"BMC Med Educ"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2018.00149","article-title":"Best practices for developing and validating scales for health, social, and behavioral research: a primer","volume":"6","author":"Boateng","journal-title":"Front Public Health"},{"key":"R32","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E . Capabilities of GPT-4 on medical challenge problems. arXiv. Preprint posted online on Mar 20, 2023. [doi: 10.48550\/ARXIV.2303.13375]"},{"issue":"3","key":"R33","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","journal-title":"Nat Med"},{"issue":"7972","key":"R34","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature New Biol"}],"container-title":["JMIR Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e82702","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:26:18Z","timestamp":1785536778000},"score":1,"resource":{"primary":{"URL":"https:\/\/mededu.jmir.org\/2026\/1\/e82702"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,26]]},"references-count":34,"URL":"https:\/\/doi.org\/10.2196\/82702","relation":{},"ISSN":["2369-3762"],"issn-type":[{"value":"2369-3762","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,26]]},"article-number":"v12i2e82702"}}