{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T01:35:10Z","timestamp":1791509710252,"version":"4.3.4"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T00:00:00Z","timestamp":1779062400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T00:00:00Z","timestamp":1786060800000},"content-version":"vor","delay-in-days":81,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["72474004"],"award-info":[{"award-number":["72474004"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100017003","name":"Peking University Health Science Center","doi-asserted-by":"publisher","award":["BMU2021YJ010"],"award-info":[{"award-number":["BMU2021YJ010"]}],"id":[{"id":"10.13039\/501100017003","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Large language models (LLMs) are increasingly applied in healthcare, yet their evaluation relies predominantly on static benchmarks that are costly, contamination-prone, and lack calibrated measurement properties for fine-grained performance tracking. We developed and validated a computerized adaptive testing (CAT) framework grounded in item response theory to enable scalable, psychometrically rigorous assessment of standardized medical knowledge in LLMs. A two-phase study comprising Monte Carlo simulations and empirical evaluation of 38 LLMs was conducted between July and September 2025. The CAT protocol achieved a near-perfect correlation with full-bank results (\n                    <jats:italic>r<\/jats:italic>\n                    \u2009=\u20090.988) using only 1.3% of items. Evaluation time decreased from 6.85\u2009hours to 8.4\u2009min per model, and token usage dropped from 1.77 million to 0.03 million. Model rankings were fully preserved (Spearman\u2019s\n                    <jats:italic>\u03c1<\/jats:italic>\n                    \u2009=\u20091.0). At current API pricing, per-model evaluation costs fell from approximately $1,475 to under $5. This adaptive methodology serves as an essential pre- screening and continuous monitoring tool under a standardized testing protocol. Crucially, it is not a substitute for real-world clinical validation or safety-oriented prospective studies; rather, it enables developers and healthcare institutions to implement rigorous, high-frequency, evidence-based evaluation of foundational knowledge prior to more resource-intensive downstream testing.\n                  <\/jats:p>","DOI":"10.1038\/s41746-026-02671-w","type":"journal-article","created":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T17:48:39Z","timestamp":1779126519000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking"],"prefix":"10.1038","volume":"9","author":[{"given":"Tianpeng","family":"Zheng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhehan","family":"Jiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiayi","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shicong","family":"Feng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,18]]},"reference":[{"key":"2671_CR1","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02459-6","volume":"24","author":"YJ Park","year":"2024","unstructured":"Park, Y. J. et al. Assessing the research landscape and clinical utility of large language models: a scoping review. BMC Med. Inform. Decis. Mak. 24, 72 (2024).","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"2671_CR2","doi-asserted-by":"crossref","unstructured":"Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl. Sci. 11, 6421 (2021).","DOI":"10.3390\/app11146421"},{"key":"2671_CR3","unstructured":"Hendrycks, D. et al. Measuring massive multitask language understanding. Proc. Int. Conf. Learn. Represent. (2021)."},{"key":"2671_CR4","first-page":"17709","volume":"38","author":"Y Cai","year":"2024","unstructured":"Cai, Y. et al. MedBench: a large-scale Chinese benchmark for evaluating medical large language models. Proc. AAAI Conf. Artif. Intell. 38, 17709\u201317717 (2024).","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"2671_CR5","doi-asserted-by":"publisher","first-page":"1866","DOI":"10.1001\/jama.2025.6571","volume":"333","author":"D Zeng","year":"2025","unstructured":"Zeng, D., Qin, Y., Sheng, B. & Wong, T. Y. DeepSeek\u2019s \u201clow-cost\u201d adoption across China\u2019s hospital systems: too fast, too soon?. JAMA 333, 1866\u20131869 (2025).","journal-title":"JAMA"},{"key":"2671_CR6","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1111\/nyas.15007","volume":"1525","author":"R Bommasani","year":"2023","unstructured":"Bommasani, R., Liang, P. & Lee, T. Holistic evaluation of language models. Ann. N. Y. Acad. Sci. 1525, 140\u2013146 (2023).","journal-title":"Ann. N. Y. Acad. Sci."},{"key":"2671_CR7","unstructured":"Hugging Face. Open LLM leaderboard. https:\/\/huggingface.co\/spaces\/HuggingFaceH4\/open_llm_leaderboard (accessed 2023)."},{"key":"2671_CR8","unstructured":"Srivastava, A. et al. Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. (2023)."},{"key":"2671_CR9","first-page":"248","volume":"174","author":"A Pal","year":"2022","unstructured":"Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. Proc. Mach. Learn. Res. 174, 248\u2013260 (2022).","journal-title":"Proc. Mach. Learn. Res."},{"key":"2671_CR10","doi-asserted-by":"crossref","unstructured":"Pal, A., Umapathi, L. K. & Sankarasubbu, M. Med-HALT: medical domain hallucination test for large language models. In Proc. Conf. Comput. Nat. Lang. Learn. 314\u2013334 (2023).","DOI":"10.18653\/v1\/2023.conll-1.21"},{"key":"2671_CR11","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172\u2013180 (2023).","journal-title":"Nature"},{"key":"2671_CR12","doi-asserted-by":"publisher","unstructured":"Ye, H., Jin, J., Xie, Y., Zhang, X. & Song, G. Large language model psychometrics: a systematic review of evaluation, validation, and enhancement. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2505.08245 (2025).","DOI":"10.48550\/arXiv.2505.08245"},{"key":"2671_CR13","first-page":"82483","volume":"267","author":"Y Zhuang","year":"2025","unstructured":"Zhuang, Y. et al. Position: AI evaluation should learn from how we test humans. Proc. Mach. Learn. Res. 267, 82483\u201382508 (2025).","journal-title":"Proc. Mach. Learn. Res."},{"key":"2671_CR14","first-page":"68","volume":"13","author":"A Gyamfi","year":"2023","unstructured":"Gyamfi, A. & Acquaye, R. Parameters and models of item response theory (IRT): a review of literature. Acta Educ. Gen. 13, 68\u201378 (2023).","journal-title":"Acta Educ. Gen."},{"key":"2671_CR15","doi-asserted-by":"publisher","first-page":"462","DOI":"10.5334\/pme.855","volume":"12","author":"L Xu","year":"2023","unstructured":"Xu, L., Jiang, Z., Han, Y., Liang, H. & Ouyang, J. Developing computerized adaptive testing for a national health professionals exam: an attempt from psychometric simulations. Perspect. Med. Educ. 12, 462\u2013471 (2023).","journal-title":"Perspect. Med. Educ."},{"key":"2671_CR16","unstructured":"Polo, F. M. et al. Tinybenchmarks: evaluating LLMs with fewer examples. In Proc. Int. Conf. Mach. Learn. (2024)."},{"key":"2671_CR17","doi-asserted-by":"crossref","unstructured":"Zhuang, Y. et al. Survey of computerized adaptive testing: a machine learning perspective. IEEE Trans. Pattern Anal. Mach. Intell. (2026).","DOI":"10.1109\/TPAMI.2026.3672850"},{"key":"2671_CR18","doi-asserted-by":"publisher","DOI":"10.2196\/45312","volume":"9","author":"A Gilson","year":"2023","unstructured":"Gilson, A. et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Med. Educ. 9, e45312 (2023).","journal-title":"JMIR Med. Educ."},{"key":"2671_CR19","doi-asserted-by":"crossref","unstructured":"Zhang, M. et al. LLMEval-Med: a real-world clinical benchmark for medical LLMs with physician validation. Findings Assoc. Comput. Linguist. EMNLP 4888\u20134914 (2025).","DOI":"10.18653\/v1\/2025.findings-emnlp.263"},{"key":"2671_CR20","doi-asserted-by":"publisher","first-page":"e0001141","DOI":"10.1371\/journal.pdig.0001141","volume":"4","author":"M Griot","year":"2025","unstructured":"Griot, M., Vanderdonckt, J. & Yuksel, D. Implementation of large language models in electronic health records. PLOS Digit. Health 4, e0001141 (2025).","journal-title":"PLOS Digit. Health"},{"key":"2671_CR21","doi-asserted-by":"publisher","unstructured":"Gu, Y. et al. The illusion of readiness in health AI. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2509.18234 (2025).","DOI":"10.48550\/arXiv.2509.18234"},{"key":"2671_CR22","doi-asserted-by":"crossref","unstructured":"Griot, M., Vanderdonckt, J., Yuksel, D. & Hemptinne, C. Pattern recognition or medical knowledge? The problem with multiple-choice questions in medicine. Proc. Annu. Meet. Assoc. Comput. Linguist. 5321\u20135341 (2025).","DOI":"10.18653\/v1\/2025.acl-long.266"},{"key":"2671_CR23","doi-asserted-by":"crossref","unstructured":"Bedi, S. et al. Testing and evaluation of health care applications of large language models. JAMA. 333, 319\u2013328 (2025)..","DOI":"10.1001\/jama.2024.21700"},{"key":"2671_CR24","doi-asserted-by":"crossref","unstructured":"Embretson, S. E. & Reise, S. P. Item response theory for psychologists. (Lawrence Erlbaum Associates, 2000).","DOI":"10.1037\/10519-153"},{"key":"2671_CR25","doi-asserted-by":"crossref","unstructured":"Hambleton, R. K. & Swaminathan, H. Estimation of ability. in Item Response Theory: Principles and Applications 85\u2013106 (Springer, 1985).","DOI":"10.1007\/978-94-017-1988-9"},{"key":"2671_CR26","unstructured":"Mullis, I. V. S., Martin, M. O., Kennedy, A. M., Trong, K. L. & Sainsbury, M. PIRLS 2011 Assessment Framework (TIMSS & PIRLS International Study Center, 2009)."},{"key":"2671_CR27","unstructured":"OECD. PISA 2022 technical report. https:\/\/www.oecd.org\/en\/publications\/pisa-2022-technical-report_01820d6d-en.html (2024)."},{"key":"2671_CR28","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1207\/s15324818ame1301_2","volume":"13","author":"WJ van der Linden","year":"2000","unstructured":"van der Linden, W. J. & Glas, C. A. W. Capitalization on item calibration error in adaptive testing. Appl. Meas. Educ. 13, 35\u201353 (2000).","journal-title":"Appl. Meas. Educ."},{"key":"2671_CR29","doi-asserted-by":"publisher","first-page":"e6","DOI":"10.1017\/prp.2019.6","volume":"13","author":"Y Wu","year":"2019","unstructured":"Wu, Y., Cai, Y. & Tu, D. A computerized adaptive testing advancing the measurement of subjective well-being. J. Pac. Rim Psychol. 13, e6 (2019).","journal-title":"J. Pac. Rim Psychol."},{"key":"2671_CR30","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1177\/014662168200600405","volume":"6","author":"RD Bock","year":"1982","unstructured":"Bock, R. D. & Mislevy, R. J. Adaptive EAP estimation of ability in a microcomputer environment. Appl. Psychol. Meas. 6, 431\u2013444 (1982).","journal-title":"Appl. Psychol. Meas."},{"key":"2671_CR31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11336-014-9401-5","volume":"80","author":"H-H Chang","year":"2015","unstructured":"Chang, H.-H. Psychometrics behind computerized adaptive testing. Psychometrika 80, 1\u201320 (2015).","journal-title":"Psychometrika"},{"key":"2671_CR32","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-026-36627-7","volume":"16","author":"T Zheng","year":"2026","unstructured":"Zheng, T., Liu, J., Feng, S. & Jiang, Z. Fine-grained evaluation of large language models in medicine using non-parametric cognitive diagnostic modeling. Sci. Rep. 16, 6460 (2026).","journal-title":"Sci. Rep."},{"key":"2671_CR33","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1007\/s11336-015-9443-3","volume":"81","author":"D Magis","year":"2016","unstructured":"Magis, D. Efficient standard error formulas of ability estimators with dichotomous item response models. Psychometrika 81, 184\u2013200 (2016).","journal-title":"Psychometrika"},{"key":"2671_CR34","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1177\/0146621611407482","volume":"35","author":"D Magis","year":"2011","unstructured":"Magis, D. & Ra\u00eeche, G. CatR: an R package for computerized adaptive testing. Appl. Psychol. Meas. 35, 576\u2013577 (2011).","journal-title":"Appl. Psychol. Meas."},{"key":"2671_CR35","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-4-23","volume":"4","author":"SP Coderre","year":"2004","unstructured":"Coderre, S. P., Harasym, P., Mandin, H. & Fick, G. The impact of two multiple-choice question formats on the problem-solving strategies used by novices and experts. BMC Med. Educ. 4, 23 (2004).","journal-title":"BMC Med. Educ."},{"key":"2671_CR36","doi-asserted-by":"publisher","first-page":"709","DOI":"10.4300\/JGME-D-14-00176.1","volume":"6","author":"BS Heist","year":"2014","unstructured":"Heist, B. S., Gonzalo, J. D., Durning, S., Torre, D. & Elnicki, D. M. Exploring clinical reasoning strategies and test-taking behaviors during clinical vignette style multiple-choice examinations: a mixed methods study. J. Grad. Med. Educ. 6, 709\u2013714 (2014).","journal-title":"J. Grad. Med. Educ."},{"key":"2671_CR37","doi-asserted-by":"publisher","first-page":"1075","DOI":"10.1111\/medu.13367","volume":"51","author":"LT Surry","year":"2017","unstructured":"Surry, L. T., Torre, D. & Durning, S. J. Exploring examinee behaviours as validity evidence for multiple-choice question examinations. Med. Educ. 51, 1075\u20131085 (2017).","journal-title":"Med. Educ."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02671-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02671-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02671-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T11:03:33Z","timestamp":1786100613000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02671-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,18]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["2671"],"URL":"https:\/\/doi.org\/10.1038\/s41746-026-02671-w","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,18]]},"assertion":[{"value":"28 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 April 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"All authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Ethics"}}],"article-number":"604"}}