{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T18:13:24Z","timestamp":1790878404479,"version":"4.1.0"},"reference-count":37,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) are increasingly being explored for clinical decision support, but their performance in audiology has not been systematically benchmarked using clinically grounded case materials and rubric-based safety evaluations.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to develop and evaluate AUDIOLOGYBENCH, a 3-tier benchmark for characterizing frontier LLM capability in clinical audiology along (1) curated domain knowledge, (2) literature-derived evidence, and (3) clinical reasoning under multimodal case input, with an explicit human audit of the automated adjudicator on the primary end point.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n The benchmark comprises 3139 objective items from educational resources, 3175 research article\u2013derived items from peer-reviewed articles published between 2015 and 2025, and 67 multimodal clinical case studies graded against a standardized A-F rubric with 6 prespecified critical-error types that cap scores at D or F. Eight models were evaluated on the educational objective items: 4 frontier multimodal models (Gemini 2.5 Pro, Grok 4, OpenAI O3, and Claude Sonnet 4 Thinking) were evaluated on the research article\u2013derived items, and on 804 case study evaluations. Adjudication used Gemini 2.5 Pro (objective and research-derived items) and Claude Opus 4.5 (case studies). The case study adjudicator was independently audited against PhD-level audiologist consensus on blinded subsamples, supplemented by a post-stratified human-calibrated sensitivity analysis.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n A striking task-type dissociation emerged on case studies: clinical recommendations (Q3) achieved a mean score of 89.74 (SD 13.92, 95% CI 88.07\u201091.41), a 98.1% (263\/268) pass rate, and no dangerous recommendations; audiometric numerical interpretation (Q1) achieved a mean score of 67.89 (SD 18.47, 95% CI 65.68\u201070.10), with a 35.4% (95\/268) critical-error rate; and differential diagnosis (Q2) achieved a mean score of 67.79 (SD 15.33, 95% CI 65.95\u201069.63). Question type, not model selection, dominated performance (eta-squared_H=0.333 vs 0.001; rank biserial\n r<\/jats:italic>\n \u22650.679). Interreviewer reliability between audiologists was high (quadratic-weighted \u03ba of 0.78 and 0.85 across the 80-item and 50-item audits, respectively). When 2 audiologists regraded all 80 model Q1 responses with the diagnostic images available, the adjudicator\u2019s per-item Q1 labels diverged from human judgment (\u03ba=0.05; overflagging; sensitivity: 19\/26, 73%; positive predictive value: 19\/53, 36%), yet its reweighted Q1 critical-error rate (36.2%) was broadly consistent with the image-grounded human estimates (28%\u201034%), suggesting no systematic inflation of the headline rate. The principal Q3>{Q1, Q2} ranking was preserved under post-stratified human calibration. Web-style multiple-choice items showed ceiling effects (>95% accuracy); short-answer prompts remained challenging (best 30%).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Current frontier LLMs show strong recommendation generation but substantial limitations in audiometric numerical interpretation that are shared across models and that an automated adjudicator partially miscalibrated at the per-item level. AUDIOLOGYBENCH characterizes capability boundaries rather than certifying clinical readiness. Deployment of LLM-assisted audiology workflows requires structured human verification of all numerical findings and awareness of fabrication and severity misclassification failure modes documented here.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/94755","type":"journal-article","created":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T21:23:26Z","timestamp":1789161806000},"page":"e94755-e94755","source":"Crossref","is-referenced-by-count":1,"title":["Evaluating Large Language Models in Clinical Audiology (AUDIOLOGYBENCH): Benchmark Development and Validation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6531-3305","authenticated-orcid":false,"given":"Linkai","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2160-5485","authenticated-orcid":false,"given":"Changgeng","family":"Mo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9851-3713","authenticated-orcid":false,"given":"Haoshuai","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5382-3011","authenticated-orcid":false,"given":"Hanlin","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-1989-4925","authenticated-orcid":false,"given":"Congxi","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6356-9604","authenticated-orcid":false,"given":"Shangqiguo","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9467-8303","authenticated-orcid":false,"given":"Varsha M","family":"Athreya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0126-2371","authenticated-orcid":false,"given":"Matthew B","family":"Fitzgerald","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8420-9554","authenticated-orcid":false,"given":"Shan X","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,11]]},"reference":[{"key":"R1","unstructured":"Saab K Tu T Weng WH et al. Capabilities of gemini models in medicine. arXiv. Preprint posted online on Apr 29, 2024. URL: https:\/\/arxiv.org\/abs\/2404.18416 [Accessed 11-08-2026]"},{"key":"R2","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E . Capabilities of GPT-4 on medical challenge problems. arXiv. Preprint posted online on Mar 20, 2023. URL: https:\/\/arxiv.org\/abs\/2303.13375 [Accessed 11-08-2026]"},{"key":"R3","unstructured":"Augmented intelligence in medicine. American Medical Association. 2025. URL: https:\/\/www.ama-assn.org\/practice-management\/digital-health\/augmented-intelligence-medicine [Accessed 11-08-2026]"},{"issue":"14","key":"R4","doi-asserted-by":"publisher","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? A large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","journal-title":"Appl Sci"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"10","key":"R6","doi-asserted-by":"publisher","first-page":"840","DOI":"10.1038\/s42256-021-00394-z","article-title":"Harnessing the power of artificial intelligence to transform hearing healthcare and research","volume":"3","author":"Lesica","journal-title":"Nat Mach Intell"},{"key":"R7","unstructured":"Pal A Umapathi LK Sankarasubbu M . MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. arXiv. Preprint posted online on Mar 27, 2022. [doi: 10.48550\/arXiv.2203.14371]"},{"key":"R8","unstructured":"Arora RK Wei J Hicks RS et al. HealthBench: evaluating large language models towards improved human health. arXiv. Preprint posted online on May 13, 2025. URL: https:\/\/arxiv.org\/abs\/2505.08775 [Accessed 11-08-2026]"},{"key":"R9","unstructured":"Bedi S Cui H Fuentes M Unell A Wornow M Banda JM et al. MedHELM: holistic evaluation of large language models for medical tasks. arXiv. Preprint posted online on May 26, 2025. URL: https:\/\/arxiv.org\/abs\/2505.23802 [Accessed 11-08-2026]"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/76947","article-title":"Evaluating large language models in ophthalmology: systematic review","volume":"27","author":"Zhang","journal-title":"J Med Internet Res"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1557920","article-title":"Swedish Medical LLM Benchmark: development and evaluation of a framework for assessing large language models in the Swedish medical domain","volume":"8","author":"Mo\u00ebll","journal-title":"Front Artif Intell"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.2196\/55595","article-title":"Exploring the performance of ChatGPT-4 in the Taiwan audiologist qualification examination: preliminary observational study highlighting the potential of AI chatbots in hearing care","volume":"10","author":"Wang","journal-title":"JMIR Med Educ"},{"issue":"4","key":"R13","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.1007\/s00405-025-09932-6","article-title":"Diagnostic interpretation of pure tone audiograms by multimodal LLMs: a comparative study of ChatGPT-5.0 and Gemini 2.5","volume":"283","author":"Kara\u00e7ayl\u0131","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Zheng L Chiang WL Sheng Y et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. Presented at: 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks; Dec 10-16, 2023; New Orleans, LA. [doi: 10.52202\/075280-2020]","DOI":"10.52202\/075280-2020"},{"key":"R15","unstructured":"AudiologyQA. GitHub. URL: https:\/\/github.com\/pc9413\/AudiologyQA [Accessed 15-08-2026]"},{"issue":"1","key":"R16","first-page":"19","volume":"21","author":"Tomczak","journal-title":"Trends Sport Sci"},{"issue":"2","key":"R17","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1037\/\/0033-2909.86.2.420","article-title":"Intraclass correlations: uses in assessing rater reliability","volume":"86","author":"Shrout","journal-title":"Psychol Bull"},{"issue":"5","key":"R18","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1016\/0895-4356(93)90018-v","article-title":"Bias, prevalence and kappa","volume":"46","author":"Byrt","journal-title":"J Clin Epidemiol"},{"issue":"6","key":"R19","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1016\/0895-4356(90)90158-l","article-title":"High agreement but low kappa: I. The problems of two paradoxes","volume":"43","author":"Feinstein","journal-title":"J Clin Epidemiol"},{"issue":"6","key":"R20","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1016\/0895-4356(90)90159-m","article-title":"High agreement but low kappa: II. Resolving the paradoxes","volume":"43","author":"Cicchetti","journal-title":"J Clin Epidemiol"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1093\/ptj\/85.3.257","article-title":"The kappa statistic in reliability studies: use, interpretation, and sample size requirements","volume":"85","author":"Sim","journal-title":"Phys Ther"},{"key":"R22","unstructured":"Mirzadeh I Alizadeh K Shahrokhi H Tuzel O Bengio S Farajtabar M . GSM-symbolic: understanding the limitations of mathematical reasoning in large language models. arXiv. Preprint posted online on Oct 7, 2024. URL: https:\/\/arxiv.org\/abs\/2410.05229 [Accessed 11-08-2026]"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01021-3","article-title":"Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support","volume":"5","author":"Omar","journal-title":"Commun Med (Lond)"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1016\/j.ebiom.2023.104512","article-title":"Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine","volume":"90","author":"Harrer","journal-title":"EBioMedicine"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.2196\/75849","article-title":"Harm reduction strategies for thoughtful use of large language models in the medical domain: perspectives for patients and clinicians","volume":"27","author":"Mo\u00ebll","journal-title":"J Med Internet Res"},{"issue":"10","key":"R26","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.40969","article-title":"Large language model influence on diagnostic reasoning: a randomized clinical trial","volume":"7","author":"Goh","journal-title":"JAMA Netw Open"},{"issue":"2","key":"R27","doi-asserted-by":"publisher","first-page":"155","DOI":"10.5465\/amle.2025.0029","article-title":"Leveling up or leveling down? the impact of generative AI on student performance in business schools","volume":"25","author":"Bergenholtz","journal-title":"Acad Manag Learn Educ"},{"key":"R28","unstructured":"Guest O Suarez M M\u00fcller B et al. Poster: against the uncritical adoption of \u201cAI\u201d technologies in academia. Zenodo. Preprint posted online on Sep 13, 2025. [doi: 10.5281\/zenodo.17065099]"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R30","unstructured":"Wataoka K Takahashi T Ri R . Self-preference bias in LLM-as-a-judge. arXiv. Preprint posted online on Oct 29, 2024. URL: https:\/\/arxiv.org\/abs\/2410.21819 [Accessed 11-08-2026]"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Panickssery A Bowman S Feng S . LLM evaluators recognize and favor their own generations. Presented at: Advances in Neural Information Processing Systems 37; Dec 10-15, 2024; Vancouver, BC. URL: http:\/\/www.proceedings.com\/79017.html [Accessed 11-08-2026] [doi: 10.52202\/079017-2197]","DOI":"10.52202\/079017-2197"},{"key":"R32","unstructured":"Ye J Wang Y Huang Y et al. Justice or prejudice? Quantifying biases in LLM-as-a-judge. arXiv. Preprint posted online on Oct 3, 2024. URL: https:\/\/arxiv.org\/abs\/2410.02736 [Accessed 11-08-2026]"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","first-page":"78","DOI":"10.3766\/jaaa.18.1.7","article-title":"Qualind: a method for assessing the accuracy of automated tests","volume":"18","author":"Margolis","journal-title":"J Am Acad Audiol"},{"issue":"9","key":"R34","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-020-01627-1","article-title":"AutoAudio: deep learning for automatic audiogram interpretation","volume":"44","author":"Crowson","journal-title":"J Med Syst"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-020-60898-3","article-title":"Data-driven audiogram classification for mobile audiometry","volume":"10","author":"Charih","journal-title":"Sci Rep"},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Atari M Xue MJ Park PS Blasi DE Henrich J . Which humans? PsyArXiv. Preprint posted online on Sep 22, 2023. [doi: 10.31234\/osf.io\/5b26t]","DOI":"10.31234\/osf.io\/5b26t"},{"key":"R37","doi-asserted-by":"crossref","unstructured":"Luccioni AS Jernite Y Strubell E . Power hungry processing: watts driving the cost of AI deployment? Presented at: FAccT \u201924: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency; Jun 3-6, 2024:85-99; [doi: 10.1145\/3630106.3658542]","DOI":"10.1145\/3630106.3658542"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T21:23:32Z","timestamp":1789161812000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e94755"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,11]]},"references-count":37,"URL":"https:\/\/doi.org\/10.2196\/94755","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,11]]},"article-number":"v28i14e94755"}}