{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T21:15:45Z","timestamp":1782854145484,"version":"3.54.5"},"reference-count":29,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption.<\/jats:p>","DOI":"10.2196\/70901","type":"journal-article","created":{"date-parts":[[2025,5,26]],"date-time":"2025-05-26T17:47:58Z","timestamp":1748281678000},"page":"e70901-e70901","source":"Crossref","is-referenced-by-count":4,"title":["Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics"],"prefix":"10.2196","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2470-4278","authenticated-orcid":false,"given":"Luning","family":"Sun","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4732-7305","authenticated-orcid":false,"given":"Christopher","family":"Gibbons","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9746-7632","authenticated-orcid":false,"given":"Jos\u00e9","family":"Hern\u00e1ndez-Orallo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5768-1095","authenticated-orcid":false,"given":"Xiting","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6464-2326","authenticated-orcid":false,"given":"Liming","family":"Jiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0174-3212","authenticated-orcid":false,"given":"David","family":"Stillwell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3281-9574","authenticated-orcid":false,"given":"Fang","family":"Luo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-3257-3077","authenticated-orcid":false,"given":"Xing","family":"Xie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,5,26]]},"reference":[{"issue":"7956","key":"R1","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","journal-title":"Nature New Biol"},{"issue":"4","key":"R2","doi-asserted-by":"publisher","first-page":"e179","DOI":"10.1016\/S2589-7500(23)00048-1","article-title":"Using ChatGPT to write patient clinic letters","volume":"5","author":"Ali","journal-title":"Lancet Digit Health"},{"key":"R3","unstructured":"Tu T Palepu A Schaekermann M Saab K Freyberg J Tanno R et al. Towards conversational diagnostic AI. 2024. URL: http:\/\/arxiv.org\/abs\/2401.05654 [doi: 10.48550\/arXiv.2401.05654]"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","journal-title":"NPJ Digit Med"},{"issue":"7972","key":"R5","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature New Biol"},{"key":"R6","unstructured":"Kasai J Kasai Y Sakaguchi K Yamada Y Radev D . Evaluating GPT-4 and chatgpt on japanese medical licensing examinations. 2023. URL: http:\/\/arxiv.org\/abs\/2303.18027 [doi: 10.48550\/arXiv.2303.18027]"},{"key":"R7","doi-asserted-by":"crossref","unstructured":"Hern\u00e1ndez-Orallo J . The Measure of All Minds: Evaluating Natural and Artificial Intelligence. Cambridge University Press; 2017. [doi: 10.1017\/9781316594179] ISBN: 9781316594179","DOI":"10.1017\/9781316594179"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-02481-y","article-title":"Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during COVID-19","volume":"11","author":"Duckworth","journal-title":"Sci Rep"},{"key":"R9","unstructured":"Yuan L Chen Y Cui G Gao H Zou F Cheng X et al. Revisiting out-of-distribution robustness in NLP: benchmark, analysis, and llms evaluations. NeurIPS. Preprint posted online on 2023. URL: http:\/\/arxiv.org\/abs\/2306.04618 [doi: 10.48550\/arXiv.2306.04618]"},{"key":"R10","unstructured":"Zhang X Li J Chu W Hai J Xu R Yang Y et al. On the out-of-distribution generalization of multimodal large language models. 2024. URL: http:\/\/arxiv.org\/abs\/2402.06599 [doi: 10.48550\/arXiv.2402.06599]"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01208-3","article-title":"Evaluating multimodal AI in medical diagnostics","author":"Kaczmarczyk","journal-title":"NPJ Digit Med"},{"key":"R12","unstructured":"Sclar M Choi Y Tsvetkov Y Suhr A . Quantifying language models\u2019 sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting. 2023. URL: http:\/\/arxiv.org\/abs\/2310.11324 [doi: 10.48550\/arXiv.2310.11324]"},{"key":"R13","unstructured":"Deng C Zhao Y Tang X Gerstein M Cohan A . Investigating data contamination in modern benchmarks for large language models. 2023. URL: http:\/\/arxiv.org\/abs\/2311.09783 [doi: 10.48550\/arXiv.2311.09783]"},{"key":"R14","unstructured":"Balloccu S Schmidtov\u00e1 P Lango M Du\u0161ek O . Repeat: data contamination and evaluation malpractices in closed-source llms. 2024. URL: http:\/\/arxiv.org\/abs\/2402.03927 [doi: 10.48550\/arXiv.2402.03927]"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01344-w","article-title":"The path forward for large language models in medicine is open","volume":"7","author":"Riedemann","journal-title":"NPJ Digit Med"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Rust J Golombok S . Modern Psychometrics: The Science of Psychological Assessment. Routledge; 2014. [doi: 10.4324\/9781315787527] ISBN: 9781315787527","DOI":"10.4324\/9781315787527"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1037\/1040-3590.4.1.26","article-title":"The development of markers for the big-five factor structure","volume":"4","author":"Goldberg","journal-title":"Psychol Assess"},{"key":"R18","unstructured":"Burnell R Hao H Conway ARA Orallo JH . Revealing the structure of language model capabilities. 2023. URL: http:\/\/arxiv.org\/abs\/2306.10062 [doi: 10.48550\/arXiv.2306.10062]"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1111\/nyas.15007","article-title":"Holistic evaluation of language models","volume":"1525","author":"Bommasani","journal-title":"Ann N Y Acad Sci"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1111\/1475-6773.14016","article-title":"A systematic review of research on empathy in health care","volume":"58","author":"Nembhard","journal-title":"Health Serv Res"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.2196\/52597","article-title":"Large language models and empathy: systematic review","volume":"26","author":"Sorin","journal-title":"J Med Internet Res"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01083-y","article-title":"Evaluating large language models as agents in the clinic","volume":"7","author":"Mehandru","journal-title":"NPJ Digit Med"},{"key":"R23","unstructured":"Wang X Jiang L Hernandez-Orallo J Sun L Stillwell D Luo F et al. Evaluating general-purpose AI with psychometrics. URL: http:\/\/arxiv.org\/abs\/2310.16379 [doi: 10.48550\/arXiv.2310.16379]"},{"key":"R24","doi-asserted-by":"crossref","unstructured":"Embretson SE Reise SP . Item Response Theory. Psychology Press; 2013. [doi: 10.4324\/9781410605269] ISBN: 9781410605269","DOI":"10.4324\/9781410605269"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.clinpsy.032408.153553","article-title":"Item response theory and clinical measurement","volume":"5","author":"Reise","journal-title":"Annu Rev Clin Psychol"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.09.004","article-title":"Item response theory in AI: Analysing machine learning classifiers at the instance level","volume":"271","author":"Mart\u00ednez-Plumed","journal-title":"Artif Intell"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Gierl MJ Haladyna TM . Automatic item generation: theory and practice. Routledge; 2013. [doi: 10.4324\/9780203803912]","DOI":"10.4324\/9780203803912"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.2196\/48633","article-title":"Barriers to and facilitators of artificial intelligence adoption in health care: scoping review","volume":"11","author":"Hassan","journal-title":"JMIR Hum Factors"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/53616","article-title":"Benefits and risks of AI in health care: narrative review","volume":"13","author":"Chustecki","journal-title":"Interact J Med Res"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2025\/1\/e70901","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:59:55Z","timestamp":1782842395000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2025\/1\/e70901"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,26]]},"references-count":29,"URL":"https:\/\/doi.org\/10.2196\/70901","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,26]]},"article-number":"v27i7956e70901"}}