{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,14]],"date-time":"2026-11-14T21:05:37Z","timestamp":1794690337280,"version":"4.9.1"},"reference-count":22,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,2,10]],"date-time":"2025-02-10T00:00:00Z","timestamp":1739145600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,2,10]],"date-time":"2025-02-10T00:00:00Z","timestamp":1739145600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Crit Care"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) show increasing potential for their use in healthcare for administrative support and clinical decision making. However, reports on their performance in critical care medicine is lacking.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>This study evaluated five LLMs (GPT-4o, GPT-4o-mini, GPT-3.5-turbo, Mistral Large 2407 and Llama 3.1 70B) on 1181 multiple choice questions (MCQs) from the gotheextramile.com database, a comprehensive database of critical care questions at European Diploma in Intensive Care examination level. Their performance was compared to random guessing and 350 human physicians on a 77-MCQ practice test. Metrics included accuracy, consistency, and domain-specific performance. Costs, as a proxy for energy consumption, were also analyzed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      GPT-4o achieved the highest accuracy at 93.3%, followed by Llama 3.1 70B (87.5%), Mistral Large 2407 (87.9%), GPT-4o-mini (83.0%), and GPT-3.5-turbo (72.7%). Random guessing yielded 41.5% (\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.001). On the practice test, all models surpassed human physicians, scoring 89.0%, 80.9%, 84.4%, 80.3%, and 66.5%, respectively, compared to 42.7% for random guessing (\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.001) and 61.9% for the human physicians. However, in contrast to the other evaluated LLMs (\n                      <jats:italic>p<\/jats:italic>\n                      \u2009&lt;\u20090.001), GPT-3.5-turbo\u2019s performance did not significantly outperform physicians (\n                      <jats:italic>p<\/jats:italic>\n                      \u2009=\u20090.196). Despite high overall consistency, all models gave consistently incorrect answers. The most expensive model was GPT-4o, costing over 25 times more than the least expensive model, GPT-4o-mini.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LLMs exhibit exceptional accuracy and consistency, with four outperforming human physicians on a European-level practice exam. GPT-4o led in performance but raised concerns about energy consumption. Despite their potential in critical care, all models produced consistently incorrect answers, highlighting the need for more thorough and ongoing evaluations to guide responsible implementation in clinical settings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s13054-025-05302-0","type":"journal-article","created":{"date-parts":[[2025,2,10]],"date-time":"2025-02-10T04:27:41Z","timestamp":1739161661000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":43,"title":["Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study"],"prefix":"10.1186","volume":"29","author":[{"given":"Jessica D.","family":"Workum","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bas W. S.","family":"Volkers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Davy","family":"van de Sande","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sumesh","family":"Arora","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Goeijenbier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diederik","family":"Gommers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michel E.","family":"van Genderen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,2,10]]},"reference":[{"key":"5302_CR1","doi-asserted-by":"crossref","unstructured":"Schoonbeek RC, Workum JD, Schuit SCE, Doornberg JN, Van Der Laan TP, Bootsma-Robroeks CMHHT. Completeness, correctness and conciseness of physician-written versus large language model generated patient summaries integrated in electronic health records [preprint]. SSRN [Internet]. 2024; Available from: https:\/\/ssrn.com\/abstract=4835935","DOI":"10.2139\/ssrn.4835935"},{"issue":"1","key":"5302_CR2","doi-asserted-by":"publisher","first-page":"2050","DOI":"10.1038\/s41467-024-46411-8","volume":"15","author":"S Sandmann","year":"2024","unstructured":"Sandmann S, Riepenhausen S, Plagwitz L, Varghese J. Systematic analysis of ChatGPT, google search and llama 2 for clinical decision support tasks. Nat Commun. 2024;15(1):2050.","journal-title":"Nat Commun"},{"key":"5302_CR3","doi-asserted-by":"publisher","first-page":"928","DOI":"10.1001\/jamasurg.2024.1621","volume":"159","author":"P Chung","year":"2024","unstructured":"Chung P, Fong CT, Walters AM, Aghaeepour N, Yetisgen M, O\u2019Reilly-Shah VN. Large language model capabilities in perioperative risk prediction and prognostication. JAMA Surg. 2024;159:928\u201337.","journal-title":"JAMA Surg"},{"issue":"2","key":"5302_CR4","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1177\/08850666241267871","volume":"40","author":"KY Balta","year":"2025","unstructured":"Balta KY, Javidan AP, Walser E, Arntfield R, Prager R. Evaluating the appropriateness, consistency, and readability of ChatGPT in critical care recommendations. J Intensive Care Med. 2025;40(2):184\u201390.","journal-title":"J Intensive Care Med"},{"key":"5302_CR5","unstructured":"Chen H, Fang Z, Singla Y, Dredze M. Benchmarking large language models on answering and explaining challenging medical questions. ArXiv [Internet]. 2024; Available from: http:\/\/arxiv.org\/abs\/2402.18060"},{"key":"5302_CR6","unstructured":"Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of GPT-4 on medical challenge problems. ArXiv [Internet]. 2023; Available from: http:\/\/arxiv.org\/abs\/2303.13375"},{"issue":"2","key":"5302_CR7","doi-asserted-by":"publisher","first-page":"AIdbp2300092","DOI":"10.1056\/AIdbp2300092","volume":"1","author":"S Wu","year":"2024","unstructured":"Wu S, Koo M, Blum L, Black A, Kao L, Fei Z, et al. Benchmarking open-source large language models, GPT-4 and Claude 2 on multiple-choice questions in nephrology. NEJM AI. 2024;1(2):AIdbp2300092.","journal-title":"NEJM AI"},{"issue":"5","key":"5302_CR8","doi-asserted-by":"publisher","first-page":"aloa2300151","DOI":"10.1056\/AIoa2300151","volume":"1","author":"NR Rydzewski","year":"2024","unstructured":"Rydzewski NR, Dinakaran D, Zhao SG, Ruppin E, Turkbey B, Citrin DE, Patel KR. Comparative evaluation of LLMs in clinical oncology. NEJM AI. 2024;1(5):aloa2300151.","journal-title":"NEJM AI"},{"key":"5302_CR9","doi-asserted-by":"publisher","first-page":"e2417641","DOI":"10.1001\/jamanetworkopen.2024.17641","volume":"7","author":"JB Longwell","year":"2024","unstructured":"Longwell JB, Hirsch I, Binder F, Gonzalez Conchas GA, Mau D, Jang R, et al. Performance of large language models on medical oncology examination questions. JAMA Netw Open. 2024;7:e2417641.","journal-title":"JAMA Netw Open"},{"key":"5302_CR10","doi-asserted-by":"publisher","first-page":"722","DOI":"10.1093\/ced\/llad197","volume":"49","author":"L Passby","year":"2024","unstructured":"Passby L, Jenko N, Wernham A. Performance of ChatGPT on specialty certificate examination in dermatology multiple-choice questions. Clin Exp Dermatol. 2024;49:722\u20137.","journal-title":"Clin Exp Dermatol"},{"key":"5302_CR11","doi-asserted-by":"publisher","unstructured":"Ali S, Shahab O, Al Shabeeb R, Ladak F, Yang JO, Nadkarni G, et al. General purpose large language models match human performance on gastroenterology board exam self-assessments. medRxiv [Internet]. Available from: https:\/\/doi.org\/10.1101\/2023.09.21.23295918","DOI":"10.1101\/2023.09.21.23295918"},{"issue":"3","key":"5302_CR12","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1177\/1751143717746566","volume":"19","author":"FR James","year":"2018","unstructured":"James FR, Power N, Laha S. Decision-making in intensive care medicine\u2013a review. J Intensive Care Soc. 2018;19(3):247\u201358.","journal-title":"J Intensive Care Soc"},{"key":"5302_CR13","doi-asserted-by":"publisher","unstructured":"Gallifant J, Afshar M, Ameen S, Aphinyanaphongs Y, Chen S, Cacciamani G, et al. The TRIPOD-LLM statement: a targeted guideline for reporting large language models use 2 3 affiliations. medRxiv [Internet]. Available from: https:\/\/doi.org\/10.1101\/2024.07.24.24310930","DOI":"10.1101\/2024.07.24.24310930"},{"key":"5302_CR14","unstructured":"Arora S. gotheextramile.com [Internet]. Available from: https:\/\/gotheextramile.com\/"},{"key":"5302_CR15","doi-asserted-by":"crossref","unstructured":"Liu M, Okuhara T, Dai Z, Huang W, Okada H, Furukawa E, et al. Performance of advanced large language models (GPT-4o, GPT-4, Gemini 1.5 Pro, Claude 3 Opus) on Japanese medical licensing examination: a comparative study. medRxiv [Internet]. 2024;2024.07.09.24310129. Available from: http:\/\/medrxiv.org\/content\/early\/2024\/07\/09\/2024.07.09.24310129.abstract","DOI":"10.1101\/2024.07.09.24310129"},{"key":"5302_CR16","unstructured":"Johannes Husom E, Goknil A, Khin Shar L, Sen S. The price of prompting: profiling energy use in large language models inference. ArXiv [Internet]. 2024 [cited 2024 Nov 20]; Available from: https:\/\/github.com\/ejhusom\/MELODI"},{"key":"5302_CR17","doi-asserted-by":"crossref","unstructured":"Luccioni S, Jernite Y, Strubell E. Power hungry processing: watts driving the cost of AI deployment? 2024 ACM conference on fairness, accountability, and transparency, FAccT 2024. Association for Computing Machinery, Inc; 2024. pp. 85\u201399.","DOI":"10.1145\/3630106.3658542"},{"key":"5302_CR18","doi-asserted-by":"crossref","unstructured":"Samsi S, Zhao D, McDonald J, Li B, Michaleas A, Jones M, et al. From words to watts: benchmarking the energy costs of large language model inference. ArXiv. 2023","DOI":"10.1109\/HPEC58863.2023.10363447"},{"key":"5302_CR19","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1001\/jamainternmed.2024.0295","volume":"184","author":"S Cabral","year":"2024","unstructured":"Cabral S, Restrepo D, Kanjee Z, Wilson P, Crowe B, Abdulnour RE, et al. Clinical reasoning of a generative artificial intelligence model compared with physicians. JAMA Intern Med. 2024;184:581\u20133.","journal-title":"JAMA Intern Med"},{"issue":"1","key":"5302_CR20","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/s41746-024-01010-1","volume":"7","author":"T Savage","year":"2024","unstructured":"Savage T, Nayak A, Gallo R, Rangan E, Chen JH. Diagnostic reasoning prompts reveal the potential for large language model interpretability in medicine. NPJ Digit Med. 2024;7(1):20.","journal-title":"NPJ Digit Med"},{"key":"5302_CR21","doi-asserted-by":"publisher","first-page":"642","DOI":"10.1038\/s41467-024-55628-6","volume":"16","author":"M Griot","year":"2025","unstructured":"Griot M, Hemptinne C, Vanderdonckt J, Yuksel D. Large language models lack essential metacognition for reliable medical reasoning. Nat Commun. 2025;16:642.","journal-title":"Nat Commun"},{"issue":"12","key":"5302_CR22","doi-asserted-by":"publisher","first-page":"Alcs2400639","DOI":"10.1056\/AIcs2400639","volume":"1","author":"J Wang","year":"2024","unstructured":"Wang J, Redelmeier DA. Cognitive biases and artificial intelligence. NEJM AI. 2024;1(12):Alcs2400639. https:\/\/doi.org\/10.1056\/AIcs2400639.","journal-title":"NEJM AI"}],"container-title":["Critical Care"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13054-025-05302-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13054-025-05302-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13054-025-05302-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,10]],"date-time":"2025-02-10T04:27:50Z","timestamp":1739161670000},"score":1,"resource":{"primary":{"URL":"https:\/\/ccforum.biomedcentral.com\/articles\/10.1186\/s13054-025-05302-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,10]]},"references-count":22,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["5302"],"URL":"https:\/\/doi.org\/10.1186\/s13054-025-05302-0","relation":{},"ISSN":["1364-8535"],"issn-type":[{"value":"1364-8535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,10]]},"assertion":[{"value":"19 December 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 January 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 February 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Ethics approval was not required for this study as it did not involve human participants, patient data, or any animal subjects and consisted of review of existing data and literature.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval statement"}},{"value":"During the writing process of the manuscript, some authors utilized GPT-4o (OpenAI) for linguistic purposes. All authors reviewed and edited the content as needed and take full responsibility for the content of the publication.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declaration of generative AI and AI-assisted technologies in the writing process"}},{"value":"Sumesh Arora is director of gotheextramile.com and its companion App (Critical Care MCQ) and responsible of hosting and developing the dataset of critical care questions used in this study. The other authors have no conflicts of interest to declare.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"72"}}