{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,20]],"date-time":"2026-06-20T04:17:57Z","timestamp":1781929077229,"version":"3.54.5"},"reference-count":64,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T00:00:00Z","timestamp":1773532800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T00:00:00Z","timestamp":1777852800000},"content-version":"vor","delay-in-days":50,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["81871455"],"award-info":[{"award-number":["81871455"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Zhejiang Provincial Natural Science Foundation of China","award":["LY22H180001"],"award-info":[{"award-number":["LY22H180001"]}]},{"name":"Municipal Natural Science Foundation of Beijing of China","award":["7222306"],"award-info":[{"award-number":["7222306"]}]},{"DOI":"10.13039\/501100001809","name":"he National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["72574006"],"award-info":[{"award-number":["72574006"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>No LLMs (Large Language Models) have yet been evaluated for understanding picture reports. Pure-tone audiograms, the gold standard for hearing loss assessment, are technical and often incomprehensible to patients without specialist interpretation. We conducted a blinded, multicenter evaluation of eight LLMs across diagnostic, interpretive, and recommendation tasks using 140 audiogram reports, assessed by clinicians and lay reviewers. The study revealed that DeepSeek-V3 achieved the highest diagnostic accuracy (severity: 67.00% ; type: 54.00%), R1 proved most suitable for general readership (FKGL: 6.41). The general public perceived significant benefits from all models in comprehension and emotional support, with Gemini 2.0 Flash\/Thinking scoring higher. Challenges remain in understanding pathological mechanisms and controlling hallucinations. While current general-purpose LLMs cannot replace the diagnostic capabilities of physicians, they may serve as effective auxiliary tools for translating specialized audiogram data into structured, patient-accessible interpretations, with particular relevance for populations facing limited access to hearing-care services.<\/jats:p>","DOI":"10.1038\/s41746-026-02537-1","type":"journal-article","created":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T03:15:33Z","timestamp":1773544533000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A multicenter multifunctional assessment of large language models in pure-tone audiogram interpretation for patients"],"prefix":"10.1038","volume":"9","author":[{"given":"Jun","family":"Liang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mengyao","family":"Xing","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peng","family":"Xiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guixuan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qichuan","family":"Fang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tingting","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhengyang","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"XueMing","family":"Leng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiuke","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoyi","family":"Jiao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenghua","family":"Tian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianbo","family":"Lei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,3,15]]},"reference":[{"key":"2537_CR1","first-page":"e1","volume":"66","author":"KA Alshehri","year":"2019","unstructured":"Alshehri, K. A., Alqulayti, W. M., Yaghmoor, B. E. & Alem, H. Public awareness of ear health and hearing loss in Jeddah, Saudi Arabia. South Afr. J. Commun. Disord. 66, e1\u2013e6 (2019).","journal-title":"South Afr. J. Commun. Disord."},{"key":"2537_CR2","doi-asserted-by":"publisher","first-page":"157","DOI":"10.5604\/12321966.1227650","volume":"24","author":"MD G\u0142owacka","year":"2017","unstructured":"G\u0142owacka, M. D. et al. Knowledge of parents from urban and rural areas vs. prevention methods of hearing loss threats seen as challenges for public health. Ann. Agric. Environ. Med. 24, 157\u2013161 (2017).","journal-title":"Ann. Agric. Environ. Med."},{"key":"2537_CR3","doi-asserted-by":"publisher","first-page":"594","DOI":"10.1111\/1753-6405.12133","volume":"37","author":"D Ferndale","year":"2013","unstructured":"Ferndale, D., Watson, B. & Munro, L. Hearing loss as a public health matter-why not everyone wants their deafness or hearing loss cured. Aust. N. Z. J. Public Health 37, 594\u2013595 (2013).","journal-title":"Aust. N. Z. J. Public Health"},{"key":"2537_CR4","unstructured":"Sensory Functions, Disability and Rehabilitation (SDR). World Report on Hearing. https:\/\/www.who.int\/publications\/i\/item\/9789240020481 (2021)."},{"key":"2537_CR5","doi-asserted-by":"publisher","first-page":"996","DOI":"10.1016\/S0140-6736(21)00516-X","volume":"397","author":"GBD 2019 Hearing Loss Collaborators","year":"2021","unstructured":"GBD 2019 Hearing Loss Collaborators Hearing loss prevalence and years lived with disability, 1990-2019: findings from the Global Burden of Disease Study 2019. Lancet 397, 996\u20131009 (2021).","journal-title":"Lancet"},{"key":"2537_CR6","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-57944-9","volume":"14","author":"SY Lee","year":"2024","unstructured":"Lee, S. Y., Seo, H. W., Jung, S. M., Lee, S. H. & Chung, J. H. Assessing the accuracy and reliability of application-based audiometry for hearing evaluation. Sci. Rep. 14, 7359 (2024).","journal-title":"Sci. Rep."},{"key":"2537_CR7","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1007\/s11060-019-03326-5","volume":"145","author":"J Greenberg","year":"2019","unstructured":"Greenberg, J. et al. Cultivating resiliency in patients with neurofibromatosis 2 who are deafened or have severe hearing loss: a live\u2011video randomized control trial. J. Neurooncol. 145, 561\u2013569 (2019).","journal-title":"J. Neurooncol."},{"key":"2537_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1097\/AUD.0000000000000687","volume":"40","author":"SG Curhan","year":"2019","unstructured":"Curhan, S. G. WHO world hearing forum: guest editorial. Ear and hearing care: a global public health priority. Ear Hear. 40, 1\u20132 (2019).","journal-title":"Ear Hear."},{"key":"2537_CR9","doi-asserted-by":"publisher","first-page":"e2242972","DOI":"10.1001\/jamanetworkopen.2022.42972","volume":"5","author":"R Gotlieb","year":"2022","unstructured":"Gotlieb, R. et al. Accuracy in patient understanding of common medical phrases. JAMA Netw. Open 5, e2242972 (2022).","journal-title":"JAMA Netw. Open"},{"key":"2537_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(23)00254-6","volume":"6","author":"The Lancet Digital Health, null","year":"2024","unstructured":"The Lancet Digital Health, null Large language models: a new chapter in digital health. Lancet Digit. Health 6, e1 (2024).","journal-title":"Lancet Digit. Health"},{"key":"2537_CR11","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1038\/s41591-024-03456-y","volume":"31","author":"E Goh","year":"2025","unstructured":"Goh, E. et al. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat. Med. 31, 1233\u20131238 (2025).","journal-title":"Nat. Med."},{"key":"2537_CR12","doi-asserted-by":"publisher","first-page":"1477898","DOI":"10.3389\/fmed.2024.1477898","volume":"11","author":"S Aydin","year":"2024","unstructured":"Aydin, S., Karabacak, M., Vlachos, V. & Margetis, K. Large language models in patient education: a scoping review of applications in medicine. Front. Med. 11, 1477898 (2024).","journal-title":"Front. Med."},{"key":"2537_CR13","doi-asserted-by":"publisher","DOI":"10.2196\/53297","volume":"26","author":"L Masanneck","year":"2024","unstructured":"Masanneck, L. et al. Triage performance across large language models, ChatGPT, and untrained doctors in emergency medicine: comparative study. J. Med. Internet Res. 26, e53297 (2024).","journal-title":"J. Med. Internet Res."},{"key":"2537_CR14","doi-asserted-by":"publisher","first-page":"e248895","DOI":"10.1001\/jamanetworkopen.2024.8895","volume":"7","author":"CYK Williams","year":"2024","unstructured":"Williams, C. Y. K. et al. Use of a large language model to assess clinical acuity of adults in the emergency department. JAMA Netw. Open 7, e248895 (2024).","journal-title":"JAMA Netw. Open"},{"key":"2537_CR15","doi-asserted-by":"publisher","first-page":"e2460637","DOI":"10.1001\/jamanetworkopen.2024.60637","volume":"8","author":"MJ Duggan","year":"2025","unstructured":"Duggan, M. J. et al. Clinician experiences with ambient scribe technology to assist with documentation burden and efficiency. JAMA Netw. Open 8, e2460637 (2025).","journal-title":"JAMA Netw. Open"},{"key":"2537_CR16","doi-asserted-by":"publisher","first-page":"942","DOI":"10.1038\/s41591-024-02888-w","volume":"30","author":"K Roberts","year":"2024","unstructured":"Roberts, K. Large language models for reducing clinicians\u2019 documentation burden. Nat. Med. 30, 942\u2013943 (2024).","journal-title":"Nat. Med."},{"key":"2537_CR17","doi-asserted-by":"publisher","first-page":"e248857","DOI":"10.1001\/jamanetworkopen.2024.8857","volume":"7","author":"AB Friedman","year":"2024","unstructured":"Friedman, A. B., Delgado, M. K. & Weissman, G. E. Artificial intelligence for emergency care triage-much promise, but still much to learn. JAMA Netw. Open 7, e248857 (2024).","journal-title":"JAMA Netw. Open"},{"key":"2537_CR18","doi-asserted-by":"publisher","first-page":"808","DOI":"10.1007\/s11547-023-01651-4","volume":"128","author":"CA Mallio","year":"2023","unstructured":"Mallio, C. A., Sertorio, A. C., Bernetti, C. & Beomonte Zobel, B. Large language models for structured reporting in radiology: performance of GPT-4, ChatGPT-3.5, Perplexity and Bing. Radiol. Med. 128, 808\u2013812 (2023).","journal-title":"Radiol. Med."},{"key":"2537_CR19","doi-asserted-by":"publisher","DOI":"10.2196\/51926","volume":"26","author":"X Liu","year":"2024","unstructured":"Liu, X. et al. Uncovering language disparity of ChatGPT on retinal vascular disease classification: cross-sectional study. J. Med. Internet Res. 26, e51926 (2024).","journal-title":"J. Med. Internet Res."},{"key":"2537_CR20","doi-asserted-by":"publisher","first-page":"e2417641","DOI":"10.1001\/jamanetworkopen.2024.17641","volume":"7","author":"JB Longwell","year":"2024","unstructured":"Longwell, J. B. et al. Performance of large language models on medical oncology examination questions. JAMA Netw. Open 7, e2417641 (2024).","journal-title":"JAMA Netw. Open"},{"key":"2537_CR21","doi-asserted-by":"crossref","unstructured":"Ullah, E., Parwani, A., Baig, M. M. & Singh, R. Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic medicine with a focus on digital pathology - a recent scoping review. Diagn. Pathol. 19, 43 (2024).","DOI":"10.1186\/s13000-024-01464-7"},{"key":"2537_CR22","doi-asserted-by":"publisher","DOI":"10.2196\/64544","volume":"9","author":"D Lee","year":"2025","unstructured":"Lee, D. et al. Using large language models to automate data extraction from surgical pathology reports: retrospective cohort study. JMIR Form. Res. 9, e64544 (2025).","journal-title":"JMIR Form. Res."},{"key":"2537_CR23","doi-asserted-by":"publisher","unstructured":"Li, Z. et al. Leveraging large language models for NLG evaluation: advances and challenges. in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (eds Al-Onaizan, Y., Bansal, M. & Chen, Y.-N.) 16028\u201316045 (Association for Computational Linguistics, Miami, Florida, USA, 2024). https:\/\/doi.org\/10.18653\/v1\/2024.emnlp-main.896.","DOI":"10.18653\/v1\/2024.emnlp-main.896"},{"key":"2537_CR24","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data. Biometrics 33, 159\u2013174 (1977).","journal-title":"Biometrics"},{"key":"2537_CR25","doi-asserted-by":"publisher","DOI":"10.1111\/bpa.13207","volume":"34","author":"S Koga","year":"2024","unstructured":"Koga, S., Martin, N. B. & Dickson, D. W. Evaluating the performance of large language models: ChatGPT and Google Bard in generating differential diagnoses in clinicopathological conferences of neurodegenerative disorders. Brain Pathol. Zur. Switz. 34, e13207 (2024).","journal-title":"Brain Pathol. Zur. Switz."},{"key":"2537_CR26","doi-asserted-by":"publisher","unstructured":"Mikhail, D. et al. DeepSeek-R1 vs OpenAI o1 for ophthalmic diagnoses and management plans. JAMA Ophthalmol. e252918, https:\/\/doi.org\/10.1001\/jamaophthalmol.2025.2918 (2025).","DOI":"10.1001\/jamaophthalmol.2025.2918"},{"key":"2537_CR27","doi-asserted-by":"publisher","DOI":"10.2214\/AJR.24.31696","volume":"223","author":"JE Lee","year":"2024","unstructured":"Lee, J. E. et al. Lung cancer staging using chest CT and FDG PET\/CT free-text reports: comparison among three ChatGPT large language models and six human readers of varying experience. Am. J. Roentgenol. 223, e2431696 (2024).","journal-title":"Am. J. Roentgenol."},{"key":"2537_CR28","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.240273","volume":"312","author":"PS Suh","year":"2024","unstructured":"Suh, P. S. et al. Comparing diagnostic accuracy of radiologists versus GPT-4V and Gemini Pro Vision using image inputs from diagnosis please cases. Radiology 312, e240273 (2024).","journal-title":"Radiology"},{"key":"2537_CR29","doi-asserted-by":"crossref","unstructured":"Zhu, Q. et al. How well do multi-modal LLMs interpret CT scans? An auto-evaluation framework for analyses. J. Biomed. Inform. 168, 104864 (2025).","DOI":"10.1016\/j.jbi.2025.104864"},{"key":"2537_CR30","doi-asserted-by":"publisher","first-page":"e2412767","DOI":"10.1001\/jamanetworkopen.2024.12767","volume":"7","author":"E Steimetz","year":"2024","unstructured":"Steimetz, E. et al. Use of artificial intelligence chatbots in interpretation of pathology reports. JAMA Netw. Open 7, e2412767 (2024).","journal-title":"JAMA Netw. Open"},{"key":"2537_CR31","doi-asserted-by":"publisher","first-page":"108013","DOI":"10.1016\/j.cmpb.2024.108013","volume":"245","author":"J Li","year":"2024","unstructured":"Li, J., Dada, A., Puladi, B., Kleesiek, J. & Egger, J. ChatGPT in healthcare: a taxonomy and systematic review. Comput. Methods Prog. Biomed. 245, 108013 (2024).","journal-title":"Comput. Methods Prog. Biomed."},{"key":"2537_CR32","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1007\/s11912-025-01649-3","volume":"27","author":"Z Zhou","year":"2025","unstructured":"Zhou, Z. et al. ChatGPT in oncology diagnosis and treatment: applications, legal and ethical challenges. Curr. Oncol. Rep. 27, 336\u2013354 (2025).","journal-title":"Curr. Oncol. Rep."},{"key":"2537_CR33","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/s44178-025-00174-4","volume":"4","author":"A Mundinger","year":"2025","unstructured":"Mundinger, A. Artificial intelligence in breast oncology. Holist. Integr. Oncol. 4, 33 (2025).","journal-title":"Holist. Integr. Oncol."},{"key":"2537_CR34","unstructured":"Clear & Simple | National Institutes of Health (NIH). https:\/\/www.nih.gov\/institutes-nih\/nih-office-director\/office-communications-public-liaison\/clear-communication\/clear-simple."},{"key":"2537_CR35","doi-asserted-by":"publisher","first-page":"237437352199884","DOI":"10.1177\/2374373521998847","volume":"8","author":"MK Rooney","year":"2021","unstructured":"Rooney, M. K. et al. Readability of patient education materials from high-impact medical journals: a 20-year analysis. J. Patient Exp. 8, 2374373521998847 (2021).","journal-title":"J. Patient Exp."},{"key":"2537_CR36","doi-asserted-by":"publisher","unstructured":"Ho, B., Hong, E. M. & Benson, B. E. Assessing and improving the effectiveness of online patient education materials on essential vocal tremor: a comprehensive evaluation. J. Voice https:\/\/doi.org\/10.1016\/j.jvoice.2024.02.021 (2024).","DOI":"10.1016\/j.jvoice.2024.02.021"},{"key":"2537_CR37","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.csbj.2023.11.058","volume":"24","author":"A Su\u00e1rez","year":"2024","unstructured":"Su\u00e1rez, A. et al. Beyond the Scalpel: assessing ChatGPT\u2019s potential as an auxiliary intelligent virtual assistant in oral surgery. Comput. Struct. Biotechnol. J. 24, 46\u201352 (2024).","journal-title":"Comput. Struct. Biotechnol. J."},{"key":"2537_CR38","doi-asserted-by":"publisher","DOI":"10.2196\/56655","volume":"26","author":"Z He","year":"2024","unstructured":"He, Z. et al. Quality of answers of generative large language models versus peer users for interpreting laboratory test results for lay patients: evaluation study. J. Med. Internet Res. 26, e56655 (2024).","journal-title":"J. Med. Internet Res."},{"key":"2537_CR39","doi-asserted-by":"publisher","DOI":"10.2196\/52597","volume":"26","author":"V Sorin","year":"2024","unstructured":"Sorin, V. et al. Large language models and empathy: systematic review. J. Med. Internet Res. 26, e52597 (2024).","journal-title":"J. Med. Internet Res."},{"key":"2537_CR40","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1001\/jamaoto.2023.3509","volume":"150","author":"ML Cantuaria","year":"2024","unstructured":"Cantuaria, M. L. et al. Hearing loss, hearing aid use, and risk of dementia in older adults. JAMA Otolaryngol. Head. Neck Surg. 150, 157\u2013164 (2024).","journal-title":"JAMA Otolaryngol. Head. Neck Surg."},{"key":"2537_CR41","first-page":"2451","volume":"49","author":"Y Xian","year":"2022","unstructured":"Xian, Y., Gao, J., Chen, H. et al. Meta-analysis of the prevalence of hearing loss among the elderly in China. Mod. Prev. Med. 49, 2451\u20132458 (2022).","journal-title":"Mod. Prev. Med."},{"key":"2537_CR42","doi-asserted-by":"publisher","first-page":"e083305","DOI":"10.1136\/bmj-2024-083305","volume":"390","author":"CHART Collaborative","year":"2025","unstructured":"CHART Collaborative Reporting guidelines for chatbot health advice studies: explanation and elaboration for the Chatbot Assessment Reporting Tool (CHART). BMJ 390, e083305 (2025).","journal-title":"BMJ"},{"key":"2537_CR43","unstructured":"M. Xing, T. Zhou & Q. Fang OSF | Evaluation of the diagnostic, interpretive and advisory capabilities of LLM in pure-tone hearing reports: a multicenter cross-sectional study. https:\/\/osf.io\/bvjgt\/."},{"key":"2537_CR44","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-52415-1","volume":"15","author":"CYK Williams","year":"2024","unstructured":"Williams, C. Y. K., Miao, B. Y., Kornblith, A. E. & Butte, A. J. Evaluating the use of large language models to provide clinical recommendations in the emergency department. Nat. Commun. 15, 8236 (2024).","journal-title":"Nat. Commun."},{"key":"2537_CR45","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","volume":"333","author":"S Bedi","year":"2025","unstructured":"Bedi, S. et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA 333, 319\u2013328 (2025).","journal-title":"JAMA"},{"key":"2537_CR46","doi-asserted-by":"publisher","first-page":"e2457879","DOI":"10.1001\/jamanetworkopen.2024.57879","volume":"8","author":"B Huo","year":"2025","unstructured":"Huo, B. et al. Large language models for chatbot health advice studies: a systematic review. JAMA Netw. Open 8, e2457879 (2025).","journal-title":"JAMA Netw. Open"},{"key":"2537_CR47","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41591-024-03328-5","volume":"31","author":"S Johri","year":"2025","unstructured":"Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat. Med. 31, 77\u201386 (2025).","journal-title":"Nat. Med."},{"key":"2537_CR48","doi-asserted-by":"publisher","first-page":"2546","DOI":"10.1038\/s41591-025-03727-2","volume":"31","author":"S Sandmann","year":"2025","unstructured":"Sandmann, S. et al. Benchmark evaluation of DeepSeek large language models in clinical decision-making. Nat. Med. 31, 2546\u20132549 (2025).","journal-title":"Nat. Med."},{"key":"2537_CR49","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-02954-4","volume":"25","author":"S Shool","year":"2025","unstructured":"Shool, S. et al. A systematic review of large language model (LLM) evaluations in clinical medicine. BMC Med. Inform. Decis. Mak. 25, 117 (2025).","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"2537_CR50","doi-asserted-by":"publisher","first-page":"258","DOI":"10.1038\/s41746-024-01258-7","volume":"7","author":"TYC Tam","year":"2024","unstructured":"Tam, T. Y. C. et al. A framework for human evaluation of large language models in healthcare derived from literature review. NPJ Digit. Med. 7, 258 (2024).","journal-title":"NPJ Digit. Med."},{"key":"2537_CR51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3641289","volume":"15","author":"Y Chang","year":"2024","unstructured":"Chang, Y. et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 15, 1\u201345 (2024).","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"2537_CR52","unstructured":"IBM Corp. Downloading IBM SPSS Statistics 29.0.2.0. https:\/\/www.ibm.com\/support\/pages\/downloading-ibm-spss-statistics-29020."},{"key":"2537_CR53","unstructured":"Flesch R. A new readability yardstick. https:\/\/psycnet.apa.org\/record\/1949-01274-001 (1948)."},{"key":"2537_CR54","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1037\/h0076540","volume":"60","author":"M Coleman","year":"1975","unstructured":"Coleman, M. & Liau, T. L. A computer readability formula designed for machine scoring. J. Appl. Psychol. 60, 283\u2013284 (1975).","journal-title":"J. Appl. Psychol."},{"key":"2537_CR55","unstructured":"Fry E. Fry\u2019s readability graph: clarifications, validity, and extension to Level 17 on JSTOR. https:\/\/www.jstor.org\/stable\/40018802 (1977)."},{"key":"2537_CR56","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1038\/s41586-024-07421-0","volume":"630","author":"S Farquhar","year":"2024","unstructured":"Farquhar, S., Kossen, J., Kuhn, L. & Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature 630, 625\u2013630 (2024).","journal-title":"Nature"},{"key":"2537_CR57","first-page":"e52365","volume":"16","author":"FH Bazzari","year":"2024","unstructured":"Bazzari, F. H. & Bazzari, A. H. Utilizing ChatGPT in telepharmacy. Cureus 16, e52365 (2024).","journal-title":"Cureus"},{"key":"2537_CR58","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).","journal-title":"Nature"},{"key":"2537_CR59","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1007\/s00405-023-08282-5","volume":"281","author":"JR Lechien","year":"2024","unstructured":"Lechien, J. R., Georgescu, B. M., Hans, S. & Chiesa-Estomba, C. M. ChatGPT performance in laryngology and head and neck surgery: a clinical case-series. Eur. Arch. Otorhinolaryngol. 281, 319\u2013333 (2024).","journal-title":"Eur. Arch. Otorhinolaryngol."},{"key":"2537_CR60","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-52853-3","volume":"14","author":"W Peng","year":"2024","unstructured":"Peng, W. et al. Evaluating AI in medicine: a comparative analysis of expert and ChatGPT responses to colorectal cancer questions. Sci. Rep. 14, 2840 (2024).","journal-title":"Sci. Rep."},{"key":"2537_CR61","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2025.1530442","volume":"7","author":"J Chen","year":"2025","unstructured":"Chen, J. et al. A comparative analysis of large language models on clinical questions for autoimmune diseases. Front. Digit. Health 7, 1530442 (2025).","journal-title":"Front. Digit. Health"},{"key":"2537_CR62","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02709-7","volume":"24","author":"J Lee","year":"2024","unstructured":"Lee, J., Park, S., Shin, J. & Cho, B. Analyzing evaluation methods for large language models in the medical field: a scoping review. BMC Med. Inform. Decis. Mak. 24, 366 (2024).","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"2537_CR63","doi-asserted-by":"publisher","first-page":"105219","DOI":"10.1016\/j.ijmedinf.2023.105219","volume":"179","author":"JY Yun","year":"2023","unstructured":"Yun, J. Y., Kim, D. J., Lee, N. & Kim, E. K. A comprehensive evaluation of ChatGPT consultation quality for augmentation mammoplasty: a comparative analysis between plastic surgeons and laypersons. Int. J. Med. Inf. 179, 105219 (2023).","journal-title":"Int. J. Med. Inf."},{"key":"2537_CR64","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-51531-8","volume":"14","author":"J Choi","year":"2024","unstructured":"Choi, J. et al. Availability of ChatGPT to provide medical information for patients with kidney cancer. Sci. Rep. 14, 1542 (2024).","journal-title":"Sci. Rep."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02537-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02537-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02537-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T12:03:15Z","timestamp":1777896195000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02537-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,15]]},"references-count":64,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["2537"],"URL":"https:\/\/doi.org\/10.1038\/s41746-026-02537-1","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,15]]},"assertion":[{"value":"14 December 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 March 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 March 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"348"}}