{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,29]],"date-time":"2026-10-29T23:57:23Z","timestamp":1793318243757,"version":"4.5.0"},"reference-count":32,"publisher":"Walter de Gruyter GmbH","issue":"11","license":[{"start":{"date-parts":[[2025,8,14]],"date-time":"2025-08-14T00:00:00Z","timestamp":1755129600000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"OpenAI\u2019s Researcher Access Program"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,10,27]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec id=\"j_cclm-2025-0647_abs_001\">\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Large language models (LLMs), such as OpenAI\u2019s GPT-4o, have demonstrated considerable promise in transforming clinical decision support systems. In this study, we focused on a single but crucial task of clinical decision-making: laboratory test ordering.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"j_cclm-2025-0647_abs_002\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We evaluated the self-consistency and performance of GPT-4o as a laboratory test recommender for 15 simulated clinical cases of different complexities across primary and emergency care settings. Through two prompting strategies\u00a0\u2013\u00a0zero-shot and chain-of-thought\u00a0\u2013\u00a0the model\u2019s recommendations were evaluated against expert consensus-derived gold-standard laboratory test orders categorized into essential and conditional test orders.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"j_cclm-2025-0647_abs_003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We found that GPT-4o exhibited high self-consistency across repeated prompts, surpassing the consistency observed among individual expert orders in the earliest round of consensus. Precision was moderate to high for both prompting strategies (68\u201382\u202f%), although relatively lower recall (41\u201351\u202f%) highlighted a risk of underutilization. A detailed analysis of false negatives (FNs) and false positives (FPs) could explain some gaps in recall and precision. Notably, variability in recommendations centered primarily on conditional tests, reflecting the broader diagnostic uncertainty that can arise in diverse clinical contexts. Our analysis revealed that neither prompting strategy, case complexity, nor clinical context significantly affected GPT-4o\u2032s performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"j_cclm-2025-0647_abs_004\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This work underscores the promise of LLMs in optimizing laboratory test ordering while identifying gaps for enhancing their alignment with clinical practice. Future research should focus on real-world implementation, integrating clinician feedback, and ensuring alignment with local test menus and guidelines to improve both performance and trust in LLM-driven clinical decision support.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1515\/cclm-2025-0647","type":"journal-article","created":{"date-parts":[[2025,8,13]],"date-time":"2025-08-13T17:13:33Z","timestamp":1755105213000},"page":"2186-2197","source":"Crossref","is-referenced-by-count":7,"title":["Evaluating large language models as clinical laboratory test recommenders in primary and emergency care: a crucial step in clinical decision making"],"prefix":"10.1515","volume":"63","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7797-1655","authenticated-orcid":false,"given":"Ahmed Medhat","family":"Zayed","sequence":"first","affiliation":[{"name":"Department of Public Health and Primary Care, Faculty of Medicine , 26657 KU Leuven , Leuven , Belgium"},{"name":"Laboratory Medicine Department , Menoufia University National Liver Institute , Shebin El-Kom , Egypt"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8528-5719","authenticated-orcid":false,"given":"Glynis","family":"Frans","sequence":"additional","affiliation":[{"name":"Department of Laboratory Medicine , University Hospitals Leuven , Leuven , Belgium"},{"name":"Department of Microbiology, Immunology and Transplantation, Clinical and Diagnostic Immunology , KU Leuven , Leuven , Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicolas","family":"Delvaux","sequence":"additional","affiliation":[{"name":"Department of Public Health and Primary Care, Faculty of Medicine , 26657 KU Leuven , Leuven , Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"374","published-online":{"date-parts":[[2025,8,14]]},"reference":[{"key":"2025103114402211077_j_cclm-2025-0647_ref_001","unstructured":"Freedman, DB. Towards better test utilization\u00a0\u2013\u00a0strategies to improve physician ordering and their impact on patient outcomes. EJIFCC 2015;26:15\u201330."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_002","unstructured":"The Lewin Group, Inc. The value of diagnostics innovation, adoption and diffusion into health care 2005. [Internet] [cited 2023 Apr 3]; Available from: https:\/\/www.lewin.com\/content\/dam\/Lewin\/Resources\/Site_Sections\/Publications\/ValueofDiagnostics.pdf."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_003","doi-asserted-by":"crossref","unstructured":"Wians, FH. Clinical laboratory tests: which, why, and what do the results mean? Lab Med 2009;40:105\u201313. https:\/\/doi.org\/10.1309\/lm404l0hhutwwudd.","DOI":"10.1309\/LM404L0HHUTWWUDD"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_004","doi-asserted-by":"crossref","unstructured":"Zhi, M, Ding, EL, Theisen-Toupal, J, Whelan, J, Arnaout, R. The landscape of inappropriate laboratory testing: a 15-year meta-analysis. PLoS One 2013;8:e78962. https:\/\/doi.org\/10.1371\/journal.pone.0078962.","DOI":"10.1371\/journal.pone.0078962"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_005","doi-asserted-by":"crossref","unstructured":"Hickner, J, Thompson, PJ, Wilkinson, T, Epner, P, Shaheen, M, Pollock, AM, et al.. Primary care physicians\u2019 challenges in ordering clinical laboratory tests and interpreting results. J\u00a0Am Board Fam Med 2014;27:268\u201374. https:\/\/doi.org\/10.3122\/jabfm.2014.02.130104.","DOI":"10.3122\/jabfm.2014.02.130104"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_006","doi-asserted-by":"crossref","unstructured":"Lam, JH, Pickles, K, Stanaway, FF, Bell, KJL. Why Clinicians overtest: development of a thematic framework. BMC Health Serv Res 2020;20:1011. https:\/\/doi.org\/10.1186\/s12913-020-05844-9.","DOI":"10.1186\/s12913-020-05844-9"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_007","doi-asserted-by":"crossref","unstructured":"Delvaux, N, Van Thienen, K, Heselmans, A, de Velde, SV, Ramaekers, D, Aertgeerts, B. The effects of computerized clinical decision support systems on laboratory test ordering: a systematic review. Arch Pathol Lab Med 2017;141:585\u201395. https:\/\/doi.org\/10.5858\/arpa.2016-0115-ra.","DOI":"10.5858\/arpa.2016-0115-RA"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_008","doi-asserted-by":"crossref","unstructured":"Delvaux, N, Piessens, V, Burghgraeve, TD, Mamouris, P, Vaes, B, Stichele, RV, et al.. Clinical decision support improves the appropriateness of laboratory test ordering in primary care without increasing diagnostic error: the ELMO cluster randomized trial. Implement Sci 2020;15:100. https:\/\/doi.org\/10.1186\/s13012-020-01059-y.","DOI":"10.1186\/s13012-020-01059-y"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_009","doi-asserted-by":"crossref","unstructured":"Shortliffe, EH, Sep\u00falveda, MJ. Clinical decision support in the era of artificial intelligence. JAMA 2018;320:2199\u2013200. https:\/\/doi.org\/10.1001\/jama.2018.17163.","DOI":"10.1001\/jama.2018.17163"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_010","unstructured":"Open AI, Achiam, J, Adler, S, Agarwal, S, Ahmad, L, Akkaya, I, et al.. GPT-4 technical report 2023. [Internet] [cited 2024 Jan 30]; Available from: http:\/\/arxiv.org\/abs\/2303.08774."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_011","unstructured":"Gemini Team Google, Anil, R, Borgeaud, S, Alayrac, JB, Yu, J, Soricut, R, et al.. Gemini: a family of highly capable multimodal models 2024. [Internet] [cited 2024 Dec 24]; Available from: http:\/\/arxiv.org\/abs\/2312.11805."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_012","doi-asserted-by":"crossref","unstructured":"Singhal, K, Azizi, S, Tu, T, Mahdavi, SS, Wei, J, Chung, HW, et al.. Large language models encode clinical knowledge. Nature 2023;620:1\u20139. https:\/\/doi.org\/10.1038\/s41586-023-06291-2.","DOI":"10.1038\/s41586-023-06291-2"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_013","doi-asserted-by":"crossref","unstructured":"Cadamuro, J, Cabitza, F, Debeljak, Z, Bruyne, SD, Frans, G, Perez, SM, et al.. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI). Clin Chem Lab Med 2023;61:1158\u201366. https:\/\/doi.org\/10.1515\/cclm-2023-0355.","DOI":"10.1515\/cclm-2023-0355"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_014","doi-asserted-by":"crossref","unstructured":"Goh, E, Gallo, R, Hom, J, Strong, E, Weng, Y, Kerman, H, et al.. Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open 2024;7:e2440969. https:\/\/doi.org\/10.1001\/jamanetworkopen.2024.40969.","DOI":"10.1001\/jamanetworkopen.2024.40969"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_015","unstructured":"Singhal, K, Tu, T, Gottweis, J, Sayres, R, Wulczyn, E, Hou, L, et al.. Towards expert-level medical question answering with large language models 2023. [Internet] [cited 2024 Dec 24]; Available from: http:\/\/arxiv.org\/abs\/2305.09617."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_016","doi-asserted-by":"crossref","unstructured":"Savage, T, Nayak, A, Gallo, R, Rangan, E, Chen, JH. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digit Med 2024;7:1\u20137. https:\/\/doi.org\/10.1038\/s41746-024-01010-1.","DOI":"10.1038\/s41746-024-01010-1"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_017","unstructured":"Saab, K, Tu, T, Weng, WH, Tanno, R, Stutz, D, Wulczyn, E, et al.. Capabilities of gemini models in medicine [internet] 2024. [cited 2024 Apr 30]; Available from: http:\/\/arxiv.org\/abs\/2404.18416."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_018","doi-asserted-by":"crossref","unstructured":"Perlis, RH, Fihn, SD. Evaluating the application of large language models in clinical research contexts. JAMA Netw Open 2023;6:e2335924. https:\/\/doi.org\/10.1001\/jamanetworkopen.2023.35924.","DOI":"10.1001\/jamanetworkopen.2023.35924"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_019","unstructured":"Wei, J, Bosma, M, Zhao, V, Guu, K, Yu, AW, Lester, B, et al.. Finetuned language models are zero-shot learners 2022. [Internet] [cited 2024 Dec 25]; Available from: https:\/\/openreview.net\/forum?id=gEZrGCozdqR."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_020","unstructured":"Wei, J, Wang, X, Schuurmans, D, Bosma, M, Ichter, B, Xia, F, et al.. Chain-of-Thought prompting elicits reasoning in large language models 2023. [Internet] [cited 2024 Apr 18]; Available from: http:\/\/arxiv.org\/abs\/2201.11903."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_021","doi-asserted-by":"crossref","unstructured":"Dalkey, N, Helmer, O. An experimental application of the Delphi method to the use of experts. Manag Sci 1963;9:458\u201367. https:\/\/doi.org\/10.1287\/mnsc.9.3.458.","DOI":"10.1287\/mnsc.9.3.458"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_022","unstructured":"Laboboeken\u00a0\u2013\u00a0UZ leuven [Internet]. [cited 2025 Feb 13]; Available from: https:\/\/laboboeken.nexuzhealth.com\/toc\/pboek\/internet\/GHB"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_023","unstructured":"Vectara. Leaderboard comparing LLM performance at producing hallucinations when summarizing short documents. GitHub. [Internet] [cited 2024 Dec 24];Available from: https:\/\/github.com\/vectara\/hallucination-leaderboard."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_024","unstructured":"GPT-4o system card. [Internet]. [cited 2025 Feb 18];Available from: https:\/\/openai.com\/index\/gpt-4o-system-card\/."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_025","unstructured":"OpenAI platform - assistants API overview. [Internet]. [cited 2024 Dec 24]; Available from: https:\/\/platform.openai.com\/docs\/assistants\/overview."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_026","unstructured":"Prompt Engineering Guide. DAIR.AI. Prompt engineering guide\u00a0\u2013\u00a0LLM settings 2024. [Internet] [cited 2024 Dec 25]; Available from: https:\/\/www.promptingguide.ai\/introduction\/settings."},{"key":"2025103114402211077_j_cclm-2025-0647_ref_027","doi-asserted-by":"crossref","unstructured":"Savage, T, Wang, J, Gallo, R, Boukil, A, Patel, V, Safavi-Naini, SAA, et al.. Large language model uncertainty proxies: discrimination and calibration for medical diagnosis and treatment. J\u00a0Am Med Inf Assoc 2025;32:139\u201349. https:\/\/doi.org\/10.1093\/jamia\/ocae254.","DOI":"10.1093\/jamia\/ocae254"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_028","doi-asserted-by":"crossref","unstructured":"Arvidsson, R, Gunnarsson, R, Entezarjou, A, Sundemo, D, Wikberg, C. ChatGPT (GPT-4) versus doctors on complex cases of the Swedish family medicine specialist examination: an observational comparative study. BMJ Open 2024;14:e086148. https:\/\/doi.org\/10.1136\/bmjopen-2024-086148.","DOI":"10.1136\/bmjopen-2024-086148"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_029","doi-asserted-by":"crossref","unstructured":"Gallo, RJ, Baiocchi, M, Savage, TR, Chen, JH. Establishing best practices in large language model research: an application to repeat prompting. J\u00a0Am Med Inf Assoc 2024;31:ocae294.","DOI":"10.1093\/jamia\/ocae294"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_030","doi-asserted-by":"crossref","unstructured":"Wang, L, Chen, X, Deng, X, Wen, H, You, M, Liu, W, et al.. Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. NPJ Digit Med 2024;7:1\u20139. https:\/\/doi.org\/10.1038\/s41746-024-01029-4.","DOI":"10.1038\/s41746-024-01029-4"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_031","doi-asserted-by":"crossref","unstructured":"Wang, G, Sun, Z, Gong, Z, Ye, S, Chen, Y, Zhao, Y, et al.. Do advanced language models eliminate the need for prompt engineering in software engineering? 2024. [Internet] [cited 2025 Jan 22]; Available from: http:\/\/arxiv.org\/abs\/2411.02093.","DOI":"10.1145\/3771933"},{"key":"2025103114402211077_j_cclm-2025-0647_ref_032","doi-asserted-by":"crossref","unstructured":"Bedi, S, Liu, Y, Orr-Ewing, L, Dash, D, Koyejo, S, Callahan, A, et al.. Testing and evaluation of health care applications of large language models: a systematic review. J\u00a0Am Med Assoc 2025;333:319\u201328. https:\/\/doi.org\/10.1001\/jama.2024.21700.","DOI":"10.1001\/jama.2024.21700"}],"container-title":["Clinical Chemistry and Laboratory Medicine (CCLM)"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/cclm-2025-0647\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/cclm-2025-0647\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T14:40:36Z","timestamp":1761921636000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.degruyterbrill.com\/document\/doi\/10.1515\/cclm-2025-0647\/html"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,14]]},"references-count":32,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,6,26]]},"published-print":{"date-parts":[[2025,10,27]]}},"alternative-id":["10.1515\/cclm-2025-0647"],"URL":"https:\/\/doi.org\/10.1515\/cclm-2025-0647","relation":{},"ISSN":["1434-6621","1437-4331"],"issn-type":[{"value":"1434-6621","type":"print"},{"value":"1437-4331","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,14]]}}}