{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T16:02:16Z","timestamp":1792598536627,"version":"4.5.0"},"reference-count":34,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The proliferation of both general purpose and health care\u2013specific large language models (LLMs) has intensified the challenge of effectively evaluating and comparing them. Data contamination plagues the validity of public benchmarks, self-preference distorts LLM-as-a-judge approaches, and there is a gap between the tasks used to test models and those used in clinical practice.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>In response, we propose CLEVER (Clinical Large Language Model Evaluation\u2013Expert Review), a methodology for blind, randomized, preference-based evaluation by practicing medical doctors on specific tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We demonstrate the methodology by comparing GPT-4o (OpenAI) against 2 health care\u2013specific LLMs, with 8 billion and 70 billion parameters, over 3 tasks: clinical text summarization, clinical information extraction, and question answering on biomedical research.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Medical doctors prefer the medical model\u2013small LLM trained by John Snow Labs over GPT-4o 45% to 92% more often on the dimensions of factuality, clinical relevance, and conciseness.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The models show comparable performance on open-ended medical question answering, suggesting that health care\u2013specific LLMs can outperform much larger general purpose LLMs in tasks that require understanding of clinical context. We test the validity of CLEVER evaluations by conducting interannotator agreement, interclass correlation, and washout period analysis.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/72153","type":"journal-article","created":{"date-parts":[[2025,11,2]],"date-time":"2025-11-02T21:20:07Z","timestamp":1762118407000},"page":"e72153-e72153","source":"Crossref","is-referenced-by-count":30,"title":["Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation"],"prefix":"10.2196","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0065-6478","authenticated-orcid":false,"given":"Veysel","family":"Kocaman","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7824-6389","authenticated-orcid":false,"given":"Mustafa Aytu\u011f","family":"Kaya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1004-8839","authenticated-orcid":false,"given":"Andrei Marian","family":"Feier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2782-5478","authenticated-orcid":false,"given":"David","family":"Talby","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,12,4]]},"reference":[{"key":"R1","unstructured":"Hu EJ Shen Y Wallis P et al. LoRA: Low-Rank Adaptation of large language models. arXiv. Preprint posted online on Oct 16, 2021. [doi: 10.48550\/arXiv.2106.09685]"},{"key":"R2","unstructured":"Zhao J et al. LoRA land: 310 fine-tuned LLMs that rival GPT-4, a technical report. arXiv. Preprint posted online on Apr 29, 2024. [doi: 10.48550\/arXiv.2405.00732]"},{"key":"R3","unstructured":"Schoelkopf H . A deep dive on LLM evaluation. Parlance Labs. 2024. URL: https:\/\/parlance-labs.com\/education\/evals\/schoelkopf.html [Accessed 13-11-2025]"},{"key":"R4","unstructured":"Biderman S et al. Lessons from the trenches on reproducible evaluation of language models. arXiv. Preprint posted online on May 29, 2024. [doi: 10.48550\/arXiv.2405.14782]"},{"key":"R5","doi-asserted-by":"crossref","unstructured":"Badshah S Sajjad H . Reference-guided verdict: LLMs-as-judges in automatic evaluation of free-form text. arXiv. Preprint posted online on Nov 10, 2025. [doi: 10.48550\/arXiv.2408.09235]","DOI":"10.18653\/v1\/2025.winlp-main.37"},{"key":"R6","unstructured":"Hendrycks D et al. Measuring Massive Multitask Language Understanding. arXiv. Preprint posted online on Jan 12, 2021. [doi: 10.48550\/arXiv.2009.03300]"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104286","article-title":"DR.BENCH: Diagnostic Reasoning Benchmark for clinical natural language processing","volume":"138","author":"Gao","journal-title":"J Biomed Inform"},{"key":"R8","unstructured":"Dada A et al. CLUE: a clinical language understanding evaluation for LLMs. arXiv. Preprint posted online on Oct 12, 2025. [doi: 10.48550\/arXiv.2404.04067]"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models","volume":"333","author":"Bedi","journal-title":"JAMA"},{"key":"R10","unstructured":"Ni S et al. Training on the benchmark is not all you need. arXiv. Preprint posted online on Feb 28, 2025. [doi: 10.48550\/arXiv.2409.01790]"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Deng C Zhao Y Tang X Gerstein M Cohan A . Investigating data contamination in modern benchmarks for large language models. Presented at: 2024 Conference of the North American Chapter of the Association for Computational Linguistics; Jun 16-21, 2024; Mexico City, Mexico. URL: https:\/\/aclanthology.org\/2024.naacl-long [Accessed 14-11-2025] [doi: 10.18653\/v1\/2024.naacl-long.482]","DOI":"10.18653\/v1\/2024.naacl-long.482"},{"key":"R12","unstructured":"Oren Y et al. Proving test set contamination in black box language models. arXiv. Preprint posted online on Nov 24, 2023. [doi: 10.48550\/arXiv.2310.17623]"},{"key":"R13","unstructured":"Xu R et al. Benchmarking benchmark leakage in large language models. arXiv. Preprint posted online on Apr 29, 2024. [doi: 10.48550\/arXiv.2404.18824]"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Goodhart CA . Problems of monetary management: the UK experience. In: Monetary Theory and Practice: The UK Experience. Macmillan Education UK; 1984:91-121. [doi: 10.1007\/978-1-349-17295-5_4]","DOI":"10.1007\/978-1-349-17295-5_4"},{"key":"R15","unstructured":"OpenAI. GPT-4o system card. arXiv. Preprint posted online on Oct 25, 2024. [doi: 10.48550\/arXiv.2410.21276]"},{"key":"R16","unstructured":"Measuring the benefits of healthcare specific large language models. NLP Summit. 2024. URL: https:\/\/www.nlpsummit.org\/measuring-the-benefits-of-healthcare-specific-%20large-language-models\/ [Accessed 14-11-2025]"},{"key":"R17","unstructured":"Avnat E et al. Performance of large language models in numerical vs semantic medical knowledge: benchmarking on evidence-based Q&As. arXiv. Preprint posted online on Jul 24, 2024. [doi: 10.48550\/arXiv.2406.03855]"},{"key":"R18","unstructured":"Wang C et al. Survey on factuality in large language models: knowledge, retrieval and domain-specificity. arXiv. Preprint posted online on Dec 16, 2023. [doi: 10.48550\/arXiv.2310.07521]"},{"key":"R19","doi-asserted-by":"crossref","unstructured":"Wang Y Wang M Manzoor MA et al. Factuality of large language models: a survey. Presented at: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing; Nov 12-16, 2024:19519-19529; Miami, Florida, USA. URL: https:\/\/aclanthology.org\/2024.emnlp-main [Accessed 14-11-2025] [doi: 10.18653\/v1\/2024.emnlp-main.1088]","DOI":"10.18653\/v1\/2024.emnlp-main.1088"},{"key":"R20","unstructured":"Chen S et al. FELM: benchmarking factuality evaluation of large language models. arXiv. Preprint posted online on Nov 28, 2023. [doi: 10.48550\/arXiv.2310.00741]"},{"key":"R21","doi-asserted-by":"crossref","unstructured":"Iqbal H Wang Y Wang M et al. OpenFactCheck: a unified framework for factuality evaluation of LLMs. Presented at: 2024 Conference on Empirical Methods in Natural Language Processing; Nov 12-16, 2024; Miami, Florida, USA. URL: https:\/\/aclanthology.org\/2024.emnlp-demo [Accessed 14-11-2025] [doi: 10.18653\/v1\/2024.emnlp-demo.23]","DOI":"10.18653\/v1\/2024.emnlp-demo.23"},{"key":"R22","unstructured":"Saito K et al. Verbosity bias in preference labeling by large language models. Preprint posted online on Oct 16, 2023. [doi: 10.48550\/arXiv.2310.10076]"},{"issue":"7","key":"R23","doi-asserted-by":"publisher","first-page":"3009","DOI":"10.1007\/s00330-017-5161-9","article-title":"Structured vs narrative reporting of pelvic MRI for fibroids: clarity and impact on treatment planning","volume":"28","author":"Franconeri","journal-title":"Eur Radiol"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2021.109954","article-title":"Structured reporting of CT scans of patients with trauma leads to faster, more detailed diagnoses: an experimental study","volume":"144","author":"Jorg","journal-title":"Eur J Radiol"},{"issue":"9","key":"R25","doi-asserted-by":"publisher","first-page":"3702","DOI":"10.1007\/s00330-018-5340-3","article-title":"Structured reporting adds clinical value in primary CT staging of diffuse large B-cell lymphoma","volume":"28","author":"Schoeppe","journal-title":"Eur Radiol"},{"issue":"4","key":"R26","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","journal-title":"Nat Med"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Abacha AB et al. Overview of the MEDIQA-chat 2023 shared tasks on the summarization & generation of doctor-patient conversations. Presented at: 5th Clinical Natural Language Processing Workshop; Jul 14, 2023; Toronto, Canada. [doi: 10.18653\/v1\/2023.clinicalnlp-1.52]","DOI":"10.18653\/v1\/2023.clinicalnlp-1.52"},{"key":"R28","unstructured":"Liu Y et al. Aligning with human judgement: the role of pairwise preference in large language model evaluators. arXiv. Preprint posted online on Jan 17, 2025. [doi: 10.48550\/arXiv.2403.16950]"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1037\/\/0033-2909.86.2.420","article-title":"Intraclass correlations: uses in assessing rater reliability","volume":"86","author":"Shrout","journal-title":"Psychol Bull"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-018-0550-6","article-title":"Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales","volume":"18","author":"Bobak","journal-title":"BMC Med Res Methodol"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Panickssery A Bowman SR Feng S . LLM evaluators recognize and favor their own generations. arXiv. Preprint posted online on Apr 15, 2024. [doi: 10.48550\/arXiv.2404.13076]","DOI":"10.52202\/079017-2197"},{"key":"R32","unstructured":"Wataoka K Takahashi T . Self- preference bias in LLM-as-a-judge. arXiv. Preprint posted online on Jun 21, 2025. [doi: 10.48550\/arXiv.2410.21819]"},{"key":"R33","unstructured":"Zheng L . Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv. Preprint posted online on Dec 24, 2023. [doi: 10.48550\/arXiv.2306.05685]"},{"key":"R34","unstructured":"Clinical Large Language Model Evaluation \u2013 CLEVER. John Snow Labs. URL: https:\/\/www.johnsnowlabs.com\/marketplace\/clinical-large-language-model-evaluation-clever\/ [Accessed 28-11-2025]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e72153","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:02:24Z","timestamp":1782842544000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2025\/1\/e72153"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,4]]},"references-count":34,"URL":"https:\/\/doi.org\/10.2196\/72153","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,4]]},"article-number":"v4i4e72153"}}