{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T09:35:58Z","timestamp":1793352958384,"version":"4.5.0"},"reference-count":50,"publisher":"Georg Thieme Verlag KG","issue":"04","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Background Large language models (LLMs) are rapidly transforming medical education, yet their performance in Allergy\/Immunology remains insufficiently characterized. Furthermore, concerns regarding accuracy, consistency, and sensitivity to input format persist.<\/jats:p>\n                  <jats:p>Objectives This study aimed to evaluate and compare the accuracy and response consistency of three leading LLMs\u2014ChatGPT-5, Gemini 2.5, and Grok 4\u2014on Allergy\/Immunology United States Medical Licensing Examination (USMLE) Step 1-style questions under different prompt conditions.<\/jats:p>\n                  <jats:p>Methods Thirty-five USMLE Step 1-style questions were selected. Questions were presented to each model in two formats: single-question prompts and a combined prompt containing all questions. Fifteen trials were conducted for each format per model. Performance was assessed using mean accuracy, and variability was measured using Shannon entropy. Mixed-effects models tested the effects of model, prompt condition, and question difficulty.<\/jats:p>\n                  <jats:p>Results Overall accuracy differed significantly (p &lt; 0.001), with Gemini (80.7%) and Grok (80.5%) achieving higher mean scores than ChatGPT (74.3%). Single-item prompts yielded superior performance, with Grok (93.1%) and Gemini (90.9%) demonstrating the highest accuracy. Transitioning to a combined prompt significantly reduced accuracy for all models. Accuracy also decreased with increasing question difficulty for all models. Grok demonstrated superior reliability, maintaining the lowest overall response entropy, whereas ChatGPT exhibited the highest variability.<\/jats:p>\n                  <jats:p>Conclusion On Allergy\/Immunology Step 1-style questions, Gemini and Grok demonstrated higher accuracy than ChatGPT, although their overall accuracies remained approximately 81%. Grok offered the most consistent performance. All models demonstrated substantial sensitivity to prompt complexity and inherent performance limitations. These findings underscore the importance of prompt optimization and support the supplementary role of these models in medical education.<\/jats:p>","DOI":"10.1055\/a-2946-7393","type":"journal-article","created":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T23:02:32Z","timestamp":1787958152000},"page":"754-762","source":"Crossref","is-referenced-by-count":1,"title":["A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy\/Immunology Questions: Evaluating Correctness and Consistency"],"prefix":"10.1055","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2197-2365","authenticated-orcid":false,"given":"Moshe","family":"Carroll","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Montefiore Medical Center, New York, United States","place":["Bronx"],"department":["Department of Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sabrina","family":"Kentis","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Montefiore Medical Center, New York, United States","place":["Bronx"],"department":["Department of Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hannah","family":"Kareff","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Montefiore Medical Center, New York, United States","place":["Bronx"],"department":["Department of Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clyde","family":"Schechter","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, New York, United States","place":["Bronx"],"department":["Department of Family and Social Medicine"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6560-7418","authenticated-orcid":false,"given":"Sunit","family":"Jariwala","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Montefiore Medical Center, New York, United States","place":["Bronx"],"department":["Department of Medicine, Division of Allergy\/Immunology"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"194","published-online":{"date-parts":[[2026,9,7]]},"reference":[{"issue":"03","key":"ref1","first-page":"e80852","article-title":"Artificial intelligence in medical education: transforming learning and practice","volume":"17","author":"A Sriram","year":"2025","journal-title":"Cureus"},{"issue":"01","key":"ref2","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1186\/s12909-025-07744-0","article-title":"Integrating artificial intelligence into medical education: a narrative systematic review of current applications, challenges, and future directions","volume":"25","author":"Z Ahsan","year":"2025","journal-title":"BMC Med Educ"},{"issue":"03","key":"ref4","first-page":"e79878","article-title":"Applications of artificial intelligence in medical education: a systematic review","volume":"17","author":"E Hallquist","year":"2025","journal-title":"Cureus"},{"issue":"01","key":"ref6","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1186\/s12909-025-06704-y","article-title":"Medical students\u2019 attitudes toward AI in education: perception, effectiveness, and its credibility","volume":"25","author":"A Sami","year":"2025","journal-title":"BMC Med Educ"},{"key":"ref7","doi-asserted-by":"crossref","first-page":"1517116","DOI":"10.3389\/feduc.2025.1517116","article-title":"Swedish medical students\u2019 attitudes toward artificial intelligence and effects on career plans: a survey","volume":"10","author":"N Neittaanm\u00e4ki","year":"2025","journal-title":"Front Educ"},{"issue":"04","key":"ref8","first-page":"e37432","article-title":"Exploring the boundaries of reality: investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references","volume":"15","author":"S A Athaluri","year":"2023","journal-title":"Cureus"},{"issue":"02","key":"ref9","first-page":"e35179","article-title":"Artificial hallucinations in ChatGPT: implications in scientific writing","volume":"15","author":"H Alkaissi","year":"2023","journal-title":"Cureus"},{"issue":"06","key":"ref10","doi-asserted-by":"crossref","first-page":"887","DOI":"10.3390\/healthcare11060887","article-title":"ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns","volume":"11","author":"M Sallam","year":"2023","journal-title":"Healthcare"},{"issue":"01","key":"ref11","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1186\/s41239-025-00563-9","article-title":"Ethical considerations for teaching with artificial intelligence: a scoping review in medical education settings","volume":"22","author":"A Itani","year":"2025","journal-title":"Int J Educ Technol High Educ"},{"key":"ref12","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1007\/s40688-025-00572-5","article-title":"AI integration or domestication? Adoption patterns and ethical concerns among Ohio school psychologists","volume":"30","author":"A B Lockwood","year":"2026","journal-title":"Contemp Sch Psychol"},{"issue":"09","key":"ref13","doi-asserted-by":"crossref","first-page":"e0000604","DOI":"10.1371\/journal.pdig.0000604","article-title":"Performance of publicly available large language models on internal medicine board-style questions","volume":"3","author":"C Tarabanis","year":"2024","journal-title":"PLoS Digit Health"},{"key":"ref14","doi-asserted-by":"crossref","first-page":"e45312","DOI":"10.2196\/45312","article-title":"How does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment","volume":"9","author":"A Gilson","year":"2023","journal-title":"JMIR Med Educ"},{"issue":"02","key":"ref15","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1097\/ACM.0000000000005549","article-title":"Examining ChatGPT performance on USMLE sample items and implications for assessment","volume":"99","author":"V Yaneva","year":"2024","journal-title":"Acad Med"},{"issue":"02","key":"ref16","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"T H Kung","year":"2023","journal-title":"PLoS Digit Health"},{"issue":"03","key":"ref17","first-page":"e55991","article-title":"Comparing the performance of popular large language models on the National Board of medical examiners sample questions","volume":"16","author":"A Abbas","year":"2024","journal-title":"Cureus"},{"issue":"05","key":"ref18","doi-asserted-by":"crossref","first-page":"1049","DOI":"10.1055\/a-2405-0138","article-title":"ChatGPT performs worse on USMLE-style ethics questions compared to medical knowledge questions","volume":"15","author":"T Danehy","year":"2024","journal-title":"Appl Clin Inform"},{"issue":"01","key":"ref19","doi-asserted-by":"crossref","first-page":"35","DOI":"10.2215\/CJN.0000000000000330","article-title":"Performance of ChatGPT on nephrology test questions","volume":"19","author":"J Miao","year":"2024","journal-title":"Clin J Am Soc Nephrol"},{"issue":"05","key":"ref20","doi-asserted-by":"crossref","first-page":"e230582","DOI":"10.1148\/radiol.230582","article-title":"Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations","volume":"307","author":"R Bhayana","year":"2023","journal-title":"Radiology"},{"issue":"01","key":"ref22","doi-asserted-by":"crossref","first-page":"1507","DOI":"10.1186\/s12909-025-07624-7","article-title":"Comparison of accuracy and consistency of AI language models when answering standardised dental MCQs","volume":"25","author":"A F Alshammari","year":"2025","journal-title":"BMC Med Educ"},{"key":"ref23","doi-asserted-by":"crossref","first-page":"225","DOI":"10.21911\/aai.2024.606","article-title":"Performance of large language models in medical exams: a review focusing on allergy, immunology, and related fields","volume":"x","author":"B Dumanoglu","year":"2024","journal-title":"Asthma Allergy Immunol"},{"issue":"02","key":"ref24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.61872\/sdj-2024-06-01","article-title":"ChatGPT\u2019s performance in dentistry and allergyimmunology assessments: a comparative study","volume":"134","author":"A Fuchs","year":"2023","journal-title":"Swiss Dent J"},{"key":"ref26","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"C E Shannon","year":"1948","journal-title":"Bell Syst Tech J"},{"key":"ref27","first-page":"60","article-title":"Assessing the performance of widely used large language models across medical disciplines using USMLE-style exam questions: an in-depth evaluation","volume":"12","author":"Z S \u00d6zler","year":"2025","journal-title":"Turk Med Stud J"},{"key":"ref28","doi-asserted-by":"crossref","first-page":"e63430","DOI":"10.2196\/63430","article-title":"ChatGPT-4 Omni performance in USMLE disciplines and clinical skills: comparative analysis","volume":"10","author":"B T Bicknell","year":"2024","journal-title":"JMIR Med Educ"},{"issue":"01","key":"ref29","doi-asserted-by":"crossref","first-page":"1387","DOI":"10.1038\/s41598-025-31010-4","article-title":"Benchmarking large language models on the United States medical licensing examination for clinical reasoning and medical licensing scenarios","volume":"16","author":"M K Siam","year":"2025","journal-title":"Sci Rep"},{"issue":"04","key":"ref30","doi-asserted-by":"crossref","first-page":"e256359","DOI":"10.1001\/jamanetworkopen.2025.6359","article-title":"Semantic clinical artificial intelligence vs native large language model performance on the USMLE","volume":"8","author":"P L Elkin","year":"2025","journal-title":"JAMA Netw Open"},{"issue":"01","key":"ref31","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1186\/s12909-025-07009-w","article-title":"Accuracy of LLMs in medical education: evidence from a concordance test with medical teacher","volume":"25","author":"V Mishra","year":"2025","journal-title":"BMC Med Educ"},{"issue":"01","key":"ref32","first-page":"e101101","article-title":"A comparative evaluation of GPT-4 turbo and Gemini-pro in medical licensing exams: enhancing artificial intelligence\u2019s role in medical education","volume":"18","author":"T T Sussan","year":"2026","journal-title":"Cureus"},{"issue":"01","key":"ref33","doi-asserted-by":"crossref","first-page":"1372","DOI":"10.1186\/s12909-024-06309-x","article-title":"Performance of ChatGPT and bard on the medical licensing examinations varies across different cultures: a comparison study","volume":"24","author":"Y Chen","year":"2024","journal-title":"BMC Med Educ"},{"issue":"10","key":"ref34","first-page":"e94300","article-title":"Performance of ChatGPT and large language models on medical licensing exams worldwide: a systematic review and network meta-analysis with meta-regression","volume":"17","author":"A Kasagga","year":"2025","journal-title":"Cureus"},{"issue":"08","key":"ref35","doi-asserted-by":"crossref","first-page":"867","DOI":"10.1097\/ACM.0000000000005242","article-title":"ChatGPT and the future of medical education","volume":"98","author":"S Feng","year":"2023","journal-title":"Acad Med"},{"issue":"06","key":"ref36","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1001\/jamainternmed.2023.1835","article-title":"How chatbots and large language model artificial intelligence systems will reshape modern medicine: fountain of creativity or Pandora\u2019s box?","volume":"183","author":"R Li","year":"2023","journal-title":"JAMA Intern Med"},{"issue":"13","key":"ref37","doi-asserted-by":"crossref","first-page":"e494","DOI":"10.1093\/ajhp\/zxaf168","article-title":"Evaluation of large language models\u2019 ability to identify clinically relevant drug-drug interactions and generate high-quality clinical pharmacotherapy recommendations","volume":"83","author":"A Chase","year":"2026","journal-title":"Am J Health Syst Pharm"},{"key":"ref38","doi-asserted-by":"crossref","first-page":"e72153","DOI":"10.2196\/72153","article-title":"Clinical large language model evaluation by expert review (CLEVER): framework development and validation","volume":"4","author":"V Kocaman","year":"2025","journal-title":"JMIR AI"},{"key":"ref39","doi-asserted-by":"crossref","first-page":"2400429","DOI":"10.1002\/aisy.202400429","article-title":"Large language model-based chatbots in higher education","volume":"7","author":"D Yigci","year":"2024","journal-title":"Adv Intell Syst"},{"key":"ref40","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: how language models use long contexts","volume":"12","author":"N F Liu","year":"2024","journal-title":"Trans Assoc Comput Linguist"},{"issue":"01","key":"ref41","first-page":"e101139","article-title":"Large language models for data extraction from unstructured and semi-structured electronic health records: a multiple model performance evaluation. BMJ health care","volume":"32","author":"V Ntinopoulos","year":"2025","journal-title":"Inform"},{"key":"ref42","doi-asserted-by":"crossref","first-page":"e50638","DOI":"10.2196\/50638","article-title":"Prompt engineering as an important emerging skill for medical professionals: tutorial","volume":"25","author":"B Mesk\u00f3","year":"2023","journal-title":"J Med Internet Res"},{"key":"ref43","doi-asserted-by":"crossref","first-page":"e60501","DOI":"10.2196\/60501","article-title":"Prompt engineering paradigms for medical applications: scoping review","volume":"26","author":"J Zaghir","year":"2024","journal-title":"J Med Internet Res"},{"key":"ref44","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.imed.2025.03.002","article-title":"Evaluating large language models and agents in healthcare: key challenges in clinical applications","volume":"5","author":"X Chen","year":"2025","journal-title":"Intell Med"},{"issue":"11","key":"ref45","doi-asserted-by":"crossref","first-page":"e0311937","DOI":"10.1371\/journal.pone.0311937","article-title":"Assessing the accuracy and consistency of answers by ChatGPT to questions regarding carbon monoxide poisoning","volume":"19","author":"J Qiu","year":"2024","journal-title":"PLoS One"},{"issue":"07","key":"ref46","doi-asserted-by":"crossref","first-page":"1030","DOI":"10.1253\/circj.CJ-23-0308","article-title":"Evaluation of the accuracy of ChatGPT in answering clinical questions on the Japanese Society of Hypertension Guidelines","volume":"87","author":"K Kusunose","year":"2023","journal-title":"Circ J"},{"issue":"8017","key":"ref47","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1038\/s41586-024-07421-0","article-title":"Detecting hallucinations in large language models using semantic entropy","volume":"630","author":"S Farquhar","year":"2024","journal-title":"Nature"},{"issue":"01","key":"ref48","doi-asserted-by":"crossref","first-page":"1506","DOI":"10.1038\/s41598-024-85003-w","article-title":"Evaluation of LLMs accuracy and consistency in the registered dietitian exam through prompt engineering and knowledge retrieval","volume":"15","author":"I Azimi","year":"2025","journal-title":"Sci Rep"},{"key":"ref49","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1097\/NR9.0000000000000063","article-title":"Evaluation of accuracies of large language models in answering clinical questions related to Mediterranean diet on cardiodiabesity","volume":"3","author":"C Chen","year":"2024","journal-title":"Interdiscip Nurs Res"},{"key":"ref51","doi-asserted-by":"crossref","first-page":"104512","DOI":"10.1016\/j.ebiom.2023.104512","article-title":"Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine","volume":"90","author":"S Harrer","year":"2023","journal-title":"EBioMedicine"},{"issue":"07","key":"ref52","doi-asserted-by":"crossref","first-page":"1493","DOI":"10.1093\/jamia\/ocae101","article-title":"A taxonomy for advancing systematic error analysis in multi-site electronic health record-based clinical concept extraction","volume":"31","author":"S Fu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"issue":"04","key":"ref54","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1021\/acscentsci.4c01935","article-title":"Leveraging prompt engineering in large language models for accelerating chemical research","volume":"11","author":"F Luo","year":"2025","journal-title":"ACS Cent Sci"},{"key":"ref55","first-page":"2006","volume-title":"Large language models sensitivity to the order of options in multiple-choice questions","author":"P Pezeshkpour","year":"2024"},{"issue":"04","key":"ref56","doi-asserted-by":"crossref","first-page":"e264003","DOI":"10.1001\/jamanetworkopen.2026.4003","article-title":"Large language model performance and clinical reasoning tasks","volume":"9","author":"A S Rao","year":"2026","journal-title":"JAMA Netw Open"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/a-2946-7393.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T23:12:01Z","timestamp":1788822721000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/a-2946-7393"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":50,"journal-issue":{"issue":"04","published-online":{"date-parts":[[2026,8,14]]},"published-print":{"date-parts":[[2026,8]]}},"URL":"https:\/\/doi.org\/10.1055\/a-2946-7393","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,8,28]]},"published":{"date-parts":[[2026,8]]}}}