{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T05:34:28Z","timestamp":1789796068682,"version":"4.0.1"},"reference-count":35,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T00:00:00Z","timestamp":1753142400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: Otolaryngologists are increasingly using head and neck ultrasound (HNUS). Determining whether a practitioner of HNUS has achieved adequate theoretical knowledge remains a challenge. This study assesses the performance of two large language models (LLMs) in generating multiple-choice questions (MCQs) for head and neck ultrasound, compared with MCQs generated by physicians. Methods: Physicians and LLMs (ChatGPT, GPT4o, and Google Gemini, Gemini Advanced) created a total of 90 MCQs that covered the topics of lymph nodes, thyroid, and salivary glands. Experts in HNUS additionally evaluated all physician-drafted MCQs using a Delphi-like process. The MCQs were assessed by an international panel of experts in HNUS, who were blinded to the source of the questions. Using a Likert scale, the evaluation was based on an overall assessment including six assessment criteria: clarity, relevance, suitability, quality of distractors, adequate rationale of the answer, and an assessment of the level of difficulty. Results: Four experts in the clinical field of HNUS assessed the 90 MCQs. No significant differences were observed between the two LLMs. Physician-drafted questions (n = 30) had significant differences with Google Gemini in terms of relevance, suitability, and adequate rationale of the answer, but only significant differences in terms of suitability compared with ChatGPT. Compared to MCQ items (n = 16) validated by medical experts, LLM-constructed MCQ items scored significantly lower across all criteria. The difficulty level of the MCQs was the same. Conclusions: Our study demonstrates that both LLMs could be used to generate MCQ items with a quality comparable to drafts from physicians. However, the quality of LLM-generated MCQ items was still significantly lower than MCQs validated by ultrasound experts. LLMs are therefore cost-effective to generate a quick draft for MCQ items that afterward should be validated by experts before being used for assessment purposes. In this way, the value of LLM is not the elimination of humans, but rather vastly superior time management.<\/jats:p>","DOI":"10.3390\/diagnostics15151848","type":"journal-article","created":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T15:00:42Z","timestamp":1753196442000},"page":"1848","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Evaluation of Multiple-Choice Tests in Head and Neck Ultrasound Created by Physicians and Large Language Models"],"prefix":"10.3390","volume":"15","author":[{"given":"Jacob P. S.","family":"Nielsen","sequence":"first","affiliation":[{"name":"Department of Otorhinolaryngology, Head and Neck Surgery and Audiology, Copenhagen University Hospital (Rigshospitalet), 2100 Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"August Krogh","family":"Mikkelsen","sequence":"additional","affiliation":[{"name":"Department of Otorhinolaryngology, Head and Neck Surgery and Audiology, Copenhagen University Hospital (Rigshospitalet), 2100 Copenhagen, Denmark"},{"name":"Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4402-0287","authenticated-orcid":false,"given":"Julian","family":"Kuenzel","sequence":"additional","affiliation":[{"name":"Department of Otorhinolaryngology, Head and Neck Surgery, University Hospital Regensburg, 93053 Regensburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9551-6082","authenticated-orcid":false,"given":"Merry E.","family":"Sebelik","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology-Head and Neck Surgery, Emory University School of Medicine, Atlanta, GA 30332, USA"},{"name":"The Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gitta","family":"Madani","sequence":"additional","affiliation":[{"name":"Imperial College Healthcare NHS Trust, London W6 8RF, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tsung-Lin","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Otolaryngology, National Taiwan University Hospital, Taipei 100225, Taiwan"},{"name":"Graduate Institute of Clinical Medicine, National Taiwan University College of Medicine, Taipei 100233, Taiwan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3267-3560","authenticated-orcid":false,"given":"Tobias","family":"Todsen","sequence":"additional","affiliation":[{"name":"Department of Otorhinolaryngology, Head and Neck Surgery and Audiology, Copenhagen University Hospital (Rigshospitalet), 2100 Copenhagen, Denmark"},{"name":"Department of Clinical Medicine, University of Copenhagen, 2100 Copenhagen, Denmark"},{"name":"CAMES\u2014Copenhagen Academy for Medical Education and Simulation, Capital Region of Denmark, 2100 Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"e1201","DOI":"10.1002\/lio2.1201","article-title":"Head and neck ultrasound training improves the diagnostic performance of otolaryngology residents","volume":"9","author":"Warm","year":"2024","journal-title":"Laryngoscope Investig. Otolaryngol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"107073","DOI":"10.1016\/j.oraloncology.2024.107073","article-title":"Surgeon-performed intraoperative transoral ultrasound improves the detection of human papillomavirus-positive head and neck cancers of unknown primary","volume":"159","author":"Lomholt","year":"2024","journal-title":"Oral Oncol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kaltoft, M., Hahn, C.H., Wessman, M., Hansen, M.L., Agander, T.K., Makouei, F., Wessel, I., and Todsen, T. (2024). Intraoral Ultrasound versus MRI for Depth of Invasion Measurement in Oral Tongue Squamous Cell Carcinoma: A Prospective Diagnostic Accuracy Study. Cancers, 16.","DOI":"10.3390\/cancers16030637"},{"key":"ref_4","first-page":"11","article-title":"Surgeon-performed ultrasonography Collecting validity evidence for assessment of abdominal and head & neck ultrasonography skills","volume":"64","author":"Todsen","year":"2017","journal-title":"Dan. Med. J."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"E29","DOI":"10.1055\/a-1922-6778","article-title":"Head and Neck Ultrasound\u2014EFSUMB Training Recommendations for the Practice of Medical Ultrasound in Europe","volume":"8","author":"Todsen","year":"2022","journal-title":"Ultrasound Int. Open"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1346","DOI":"10.1002\/lary.26841","article-title":"Competency-based assessment in surgeon-performed head and neck ultrasonography: A validity study","volume":"128","author":"Todsen","year":"2018","journal-title":"Laryngoscope"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Weimer, J.M., Rink, M., M\u00fcller, L., Dirks, K., Ille, C., Bozzato, A., Sproll, C., Weimer, A.M., Neubert, C., and Buggenhagen, H. (2023). Development and Integration of DOPS as Formative Tests in Head and Neck Ultrasound Education: Proof of Concept Study for Exploration of Perceptions. Diagnostics, 13.","DOI":"10.3390\/diagnostics13040661"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1080\/0142159X.2018.1505035","article-title":"Plus \u00e7a change, plus c\u2019est pareil: Making a continued case for the use of MCQs in medical education","volume":"41","author":"Pugh","year":"2019","journal-title":"Med. Teach."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1136\/bmj.326.7390.643","article-title":"ABC of learning and teaching in medicine Written assessment","volume":"326","author":"Schuwirth","year":"2003","journal-title":"BMJ"},{"key":"ref_10","unstructured":"(2025, January 03). United States Medical Licensing Examination (USMLE). Available online: https:\/\/www.usmle.org\/."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"24","DOI":"10.34172\/rdme.2022.024","article-title":"Assessment of multiple-choice questions by item analysis for medical students\u2019 examinations","volume":"11","author":"Nojomi","year":"2022","journal-title":"Res. Dev. Med. Educ."},{"key":"ref_12","unstructured":"Van Der, W.J., Cees, L., and Glas, A.W. (2010). Statistics for Social and Behavioral Sciences Elements of Adaptive Testing, Springer."},{"key":"ref_13","unstructured":"(2024, May 20). ChatGPT by Open AI. Available online: https:\/\/chatgpt.com\/."},{"key":"ref_14","unstructured":"(2024, May 20). Gemini by Google. Available online: https:\/\/gemini.google.com\/app."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1570","DOI":"10.1002\/jvc2.459","article-title":"Usefulness of the large language model ChatGPT (GPT-4) as a diagnostic tool and information source in dermatology","volume":"3","author":"Nielsen","year":"2024","journal-title":"JEADV Clin. Pract."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"5001","DOI":"10.1007\/s00405-024-08746-2","article-title":"Reliability of large language models for advanced head and neck malignancies management: A comparison between ChatGPT 4 and Gemini Advanced","volume":"281","author":"Lorenzi","year":"2024","journal-title":"Eur. Arch. Oto-Rhino-Laryngol."},{"key":"ref_17","first-page":"e40977","article-title":"Analysing the Applicability of ChatGPT, Bard, and Bing to Generate Reasoning-Based Multiple-Choice Questions in Medical Physiology","volume":"15","author":"Agarwal","year":"2023","journal-title":"Cureus"},{"key":"ref_18","first-page":"268","article-title":"ChatGPT to generate clinical vignettes for teaching and multiple-choice questions for assessment: A randomized controlled experiment","volume":"47","year":"2024","journal-title":"Med. Teach."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Cheung, B.H.H., Lau, G.K.K., Wong, G.T.C., Lee, E.Y.P., Kulkarni, D., Seow, C.S., Wong, R., and Co, M.T. (2023). ChatGPT versus human in generating medical graduate exam multiple choice questions\u2014A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom). PLoS ONE, 18.","DOI":"10.1371\/journal.pone.0290691"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"858","DOI":"10.1093\/postmj\/qgae065","article-title":"ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: A literature review","volume":"100","author":"Emekli","year":"2024","journal-title":"Postgrad. Med. J."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yudkowsky, R., Park, Y.S., and Downing, S.M. (2019). Assessment in Health Professions Education, Routledge.","DOI":"10.4324\/9781315166902"},{"key":"ref_22","unstructured":"Smulders, S. (2024, December 05). 15+ Rules For Crafting Effective GPT Chat Prompts. Available online: https:\/\/expandi.io\/blog\/chat-gpt-rules\/."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ahuja, A.T., and Evans, R.M. (2000). Practical Head and Neck Ultrasound, Cambridge University Press.","DOI":"10.1017\/CBO9781139878388"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"(2014). AIUM Practice Guideline for the Performance of Ultrasound Examinations of the Head and Neck. J. Ultrasound Med., 33, 366\u2013382.","DOI":"10.7863\/ultra.33.2.366"},{"key":"ref_25","unstructured":"Orloff, L.A. (2017). Head and Neck Ultrasonography: Essential and Extended Applications, Plural Publishing Inc.. [2nd ed.]."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Haladyna, T.M., and Rodriguez, M.C. (2013). Developing and Validating Test Items, Routledge.","DOI":"10.4324\/9780203850381"},{"key":"ref_27","unstructured":"Swanson, D.B. (2024, December 15). Constructing Written Test Questions for the Basic and Clinical Sciences. Available online: https:\/\/www.researchgate.net\/publication\/242759434."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Tolsgaard, M.G., Todsen, T., Sorensen, J.L., Ringsted, C., Lorentzen, T., Ottesen, B., and Tabor, A. (2013). International Multispecialty Consensus on How to Evaluate Ultrasound Competence: A Delphi Consensus Survey. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0057687"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Mikkelsen, A.K., Kuenzel, J., Orloff, L.A., Sebelik, M.E., McQueen, A., Madani, G., Konge, L., Yang, T.L., Melchiors, J., and Weimer, J. (2025). Development and validation of a multiple-choice test for head and neck ultrasound certification. Eur. Arch. Oto-Rhino-Laryngol.","DOI":"10.1007\/s00405-025-09533-3"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"3872","DOI":"10.1016\/j.acra.2024.06.046","article-title":"Large Language Models as Tools to Generate Radiology Board-Style Multiple-Choice Questions","volume":"31","author":"Mistry","year":"2024","journal-title":"Acad Radiol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1016\/j.jbi.2008.08.010","article-title":"Research electronic data capture (REDCap)-A metadata-driven methodology and workflow process for providing translational research informatics support","volume":"42","author":"Harris","year":"2009","journal-title":"J. Biomed. Inform."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Chen, L., Zaharia, M., and Zou, J. (2024). How Is ChatGPT\u2019s Behavior Changing Over Time?. Harv. Data Sci. Rev., 6.","DOI":"10.1162\/99608f92.5317da47"},{"key":"ref_33","unstructured":"U.S. Copyright Office (2025, February 15). Copyright and Artificial Intelligence, Part 2 Copyrightability Report, Available online: www.copyright.gov."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Cross, J.L., Choma, M.A., and Onofrey, J.A. (2024). Bias in medical AI: Implications for clinical decision-making. PLoS Digit. Health, 3.","DOI":"10.1371\/journal.pdig.0000651"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"20552076251348850","DOI":"10.1177\/20552076251348850","article-title":"Answering real-world clinical questions using large language model, retrieval-augmented generation, and agentic systems","volume":"11","author":"Low","year":"2025","journal-title":"Digit. Health"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/15\/1848\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:14:07Z","timestamp":1760033647000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/15\/1848"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,22]]},"references-count":35,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["diagnostics15151848"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics15151848","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,22]]}}}