{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,1]],"date-time":"2026-11-01T01:45:07Z","timestamp":1793497507733,"version":"4.5.0"},"reference-count":39,"publisher":"American Medical Association (AMA)","issue":"7","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"\n Importance<\/jats:title>\n Artificial intelligence (AI) presents transformative opportunities to address the increasing challenges faced by health care systems globally. Particularly, in data-rich environments, such as intensive care units (ICUs), AI could assist in enhancing clinical decision-making, streamline workflows, and improve patient outcomes. Despite these promising applications, the practical implementation of AI in clinical settings remains limited.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To systematically evaluate AI system operationalization in the ICU, focusing on the AI field\u2019s progress over time, technical maturity, and risk of bias.<\/jats:p>\n <\/jats:sec>\n \n Evidence Review<\/jats:title>\n In this systematic review, 5 databases (Embase, MEDLINE ALL, Web of Science Core Collection, Cochrane Central Register of Controlled Trials, and Google Scholar) were searched for studies published from July 28, 2020, to June 10, 2024. Eligible studies evaluated AI applications designed for use within ICUs for adults (aged \u226516 years) and used data collected during ICU stays. Two reviewers independently screened titles and abstracts, with a third reviewer resolving disagreements. Data extraction included AI application aims, dataset origins, technology readiness level (TRL) categorization, and the use of reporting standards. Risk of bias was assessed using the PROBAST (Prediction Model Study Risk of Bias Assessment Tool).<\/jats:p>\n <\/jats:sec>\n \n Findings<\/jats:title>\n Of 17\u202f401 screened records, 1263 studies met the inclusion criteria. A total of 936 studies (74% of all studies) were classified as TRL 4 or below, indicating early-stage development or initial validation. Among these, 447 (37%) used internal datasets, 562 (46%) used MIMIC (Medical Information Mart for Intensive Care) datasets (I-IV), and 78 (6%) used the open-source eICU Collaborative Research Database. External validation (TRL 5) was achieved by 24% of studies. Only 25 (2%) progressed to clinical integration (TRL\u22656), with no studies reaching full implementation (TRL 9). Although approximately half of generative AI models reached a higher TRL (14 [47%] with TRL 5), none reached clinical integration. Additionally, only 207 studies (16%) referenced reporting standards, with adherence modestly increasing from 14% in 2021 to 23% in 2024. High risk of bias was identified in 581 of 1103 studies (53%), primarily due to methodologic shortcomings in the analysis domain.<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n Despite substantial growth in AI research within intensive care medicine in recent years, the transition from development to clinical implementation still remains limited and has made little progress over time. A paradigm shift is urgently required in the medical literature\u2014one that moves beyond retrospective validation toward the operationalization and prospective testing of AI for tangible clinical impact.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2025.22866","type":"journal-article","created":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T15:31:11Z","timestamp":1753284671000},"page":"e2522866","source":"Crossref","is-referenced-by-count":81,"title":["Operationalization of Artificial Intelligence Applications in the Intensive Care Unit"],"prefix":"10.1001","volume":"8","author":[{"given":"Willemijn E. M.","family":"Berkhout","sequence":"first","affiliation":[{"name":"Department of Adult Intensive Care, Erasmus MC University Medical Center, Rotterdam, the Netherlands"},{"name":"Erasmus MC Datahub, University Center Rotterdam, Rotterdam, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julia J.","family":"van Wijngaarden","sequence":"additional","affiliation":[{"name":"Department of Adult Intensive Care, Erasmus MC University Medical Center, Rotterdam, the Netherlands"},{"name":"Erasmus MC Datahub, University Center Rotterdam, Rotterdam, the Netherlands"},{"name":"Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jessica D.","family":"Workum","sequence":"additional","affiliation":[{"name":"Department of Adult Intensive Care, Erasmus MC University Medical Center, Rotterdam, the Netherlands"},{"name":"Erasmus MC Datahub, University Center Rotterdam, Rotterdam, the Netherlands"},{"name":"Department of Intensive Care, Elisabeth-TweeSteden Hospital, Tilburg, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Davy","family":"van de Sande","sequence":"additional","affiliation":[{"name":"Department of Adult Intensive Care, Erasmus MC University Medical Center, Rotterdam, the Netherlands"},{"name":"Erasmus MC Datahub, University Center Rotterdam, Rotterdam, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Denise E.","family":"Hilling","sequence":"additional","affiliation":[{"name":"Erasmus MC Datahub, University Center Rotterdam, Rotterdam, the Netherlands"},{"name":"Department of Surgical Oncology and Gastrointestinal Surgery, Erasmus MC Cancer Center, University Medical Center Rotterdam, Rotterdam, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian","family":"Jung","sequence":"additional","affiliation":[{"name":"Cardiovascular Research Institute D\u00fcsseldorf (CARID), Medical Faculty and University Hospital of D\u00fcsseldorf, Heinrich-Heine University D\u00fcsseldorf, D\u00fcsseldorf, Germany"},{"name":"Department of Cardiology, Pulmonology, and Vascular Medicine, Heinrich-Heine-University D\u00fcsseldorf, D\u00fcsseldorf, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Geert","family":"Meyfroidt","sequence":"additional","affiliation":[{"name":"Department of Cellular and Molecular Medicine, Laboratory of Intensive Care Medicine, KU Leuven, Leuven, Belgium"},{"name":"Department of Intensive Care Medicine, University Hospitals Leuven, Leuven, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diederik","family":"Gommers","sequence":"additional","affiliation":[{"name":"Department of Adult Intensive Care, Erasmus MC University Medical Center, Rotterdam, the Netherlands"},{"name":"Erasmus MC Datahub, University Center Rotterdam, Rotterdam, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefan N. R.","family":"Buijsman","sequence":"additional","affiliation":[{"name":"Faculty of Technology, Policy, and Management, Delft University of Technology, Delft, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michel E.","family":"van Genderen","sequence":"additional","affiliation":[{"name":"Department of Adult Intensive Care, Erasmus MC University Medical Center, Rotterdam, the Netherlands"},{"name":"Erasmus MC Datahub, University Center Rotterdam, Rotterdam, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2025,7,23]]},"reference":[{"issue":"11","key":"zoi250666r1","doi-asserted-by":"publisher","first-page":"738","DOI":"10.1136\/bmjqs-2023-016686","article-title":"Health services under pressure: a scoping review and development of a taxonomy of adaptive strategies.","volume":"33","author":"Page","year":"2024","journal-title":"BMJ Qual Saf"},{"issue":"6","key":"zoi250666r2","doi-asserted-by":"publisher","first-page":"951","DOI":"10.3390\/jpm13060951","article-title":"A review of the role of artificial intelligence in healthcare.","volume":"13","author":"Al Kuwaiti","year":"2023","journal-title":"J Pers Med"},{"issue":"7","key":"zoi250666r3","doi-asserted-by":"publisher","first-page":"750","DOI":"10.1007\/s00134-021-06446-7","article-title":"Moving from bytes to bedside: a systematic review on the use of artificial intelligence in the intensive care unit.","volume":"47","author":"van de Sande","year":"2021","journal-title":"Intensive Care Med"},{"issue":"1","key":"zoi250666r4","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1038\/s41746-024-01270-x","article-title":"A scoping review of reporting gaps in FDA-approved AI medical devices.","volume":"7","author":"Muralidharan","year":"2024","journal-title":"NPJ Digit Med"},{"issue":"11","key":"zoi250666r5","doi-asserted-by":"publisher","DOI":"10.1056\/AIe2400881","article-title":"We need more randomized clinical trials of AI.","volume":"1","author":"Ouyang","year":"2024","journal-title":"NEJM AI"},{"issue":"9","key":"zoi250666r6","doi-asserted-by":"publisher","first-page":"818","DOI":"10.1001\/jama.2023.15481","article-title":"AI in medicine: JAMA\u2019s focus on clinical outcomes, patient-centered care, quality, and equity.","volume":"330","author":"Khera","year":"2023","journal-title":"JAMA"},{"issue":"1","key":"zoi250666r7","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1186\/s44247-024-00135-3","article-title":"Artificial intelligence integration in healthcare: perspectives and trends in a survey of U.S. health system leaders.","volume":"2","author":"Guleria","year":"2024","journal-title":"BMC Digit Health"},{"key":"zoi250666r8","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2024.105377","article-title":"A multinational study on artificial intelligence adoption: clinical implementers\u2019 perspectives.","volume":"184","author":"Marco-Ruiz","year":"2024","journal-title":"Int J Med Inform"},{"issue":"1","key":"zoi250666r9","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1186\/s13054-024-04860-z","article-title":"Use of artificial intelligence in critical care: opportunities and obstacles.","volume":"28","author":"Pinsky","year":"2024","journal-title":"Crit Care"},{"issue":"1","key":"zoi250666r10","doi-asserted-by":"publisher","first-page":"701","DOI":"10.1186\/s12913-024-11112-x","article-title":"Understanding the integration of artificial intelligence in healthcare organisations and systems through the NASSS framework: a qualitative study in a leading Canadian academic centre.","volume":"24","author":"Alami","year":"2024","journal-title":"BMC Health Serv Res"},{"issue":"1","key":"zoi250666r11","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1186\/2046-4053-2-36","article-title":"Are systematic reviews up-to-date at the time of publication?","volume":"2","author":"Beller","year":"2013","journal-title":"Syst Rev"},{"key":"zoi250666r13","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1016\/j.jclinepi.2017.08.010","article-title":"Living systematic review, 1: introduction: the why, what, when, and how.","volume":"91","author":"Elliott","year":"2017","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"zoi250666r14","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1186\/s13643-023-02325-y","article-title":"How to update a living systematic review and keep it alive during a pandemic: a practical guide.","volume":"12","author":"Heron","year":"2023","journal-title":"Syst Rev"},{"issue":"71","key":"zoi250666r15","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"issue":"1","key":"zoi250666r17","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1186\/s13643-016-0384-4","article-title":"Rayyan: a web and mobile app for systematic reviews.","volume":"5","author":"Ouzzani","year":"2016","journal-title":"Syst Rev"},{"issue":"7","key":"zoi250666r18","doi-asserted-by":"publisher","first-page":"1486","DOI":"10.1007\/s00134-020-06045-y","article-title":"Machine learning in intensive care medicine: ready for take-off?","volume":"46","author":"Fleuren","year":"2020","journal-title":"Intensive Care Med"},{"issue":"1","key":"zoi250666r19","doi-asserted-by":"publisher","first-page":"W1","DOI":"10.7326\/M18-1377","article-title":"PROBAST: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration.","volume":"170","author":"Moons","year":"2019","journal-title":"Ann Intern Med"},{"issue":"1","key":"zoi250666r20","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1038\/s41746-023-00832-9","article-title":"Perspectives on validation of clinical predictive algorithms.","volume":"6","author":"de Hond","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"10","key":"zoi250666r21","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.46454","article-title":"A systematic review of the barriers to the implementation of artificial intelligence in healthcare.","volume":"15","author":"Ahmed","year":"2023","journal-title":"Cureus"},{"issue":"5","key":"zoi250666r22","doi-asserted-by":"publisher","first-page":"e367","DOI":"10.1016\/S2589-7500(24)00047-5","article-title":"Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review.","volume":"6","author":"Han","year":"2024","journal-title":"Lancet Digit Health"},{"issue":"1","key":"zoi250666r23","doi-asserted-by":"publisher","DOI":"10.1056\/AIpc2300038","article-title":"Development pipeline and geographic representation of trials for artificial intelligence\/machine learning\u2013enabled medical devices (2010 to 2023).","volume":"1","author":"Serra-Burriel","year":"2024","journal-title":"NEJM AI"},{"issue":"11","key":"zoi250666r24","doi-asserted-by":"publisher","first-page":"e848","DOI":"10.1016\/S2589-7500(24)00143-2","article-title":"Generative artificial intelligence and ethical considerations in health care: a scoping review and ethics checklist.","volume":"6","author":"Ning","year":"2024","journal-title":"Lancet Digit Health"},{"issue":"1","key":"zoi250666r25","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1038\/s41746-024-01064-1","article-title":"To warrant clinical adoption AI models require a multi-faceted implementation evaluation.","volume":"7","author":"van de Sande","year":"2024","journal-title":"NPJ Digit Med"},{"issue":"8","key":"zoi250666r26","doi-asserted-by":"publisher","DOI":"10.1056\/AIp2400223","article-title":"A call for artificial intelligence implementation science centers to evaluate clinical effectiveness.","volume":"1","author":"Longhurst","year":"2024","journal-title":"NEJM AI"},{"issue":"1","key":"zoi250666r27","doi-asserted-by":"publisher","first-page":"e64","DOI":"10.1016\/S2589-7500(24)00224-3","article-title":"Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations.","volume":"7","author":"Alderman","year":"2025","journal-title":"Lancet Digit Health"},{"issue":"3","key":"zoi250666r28","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1001\/jama.2023.26930","article-title":"A nationwide network of health AI assurance laboratories.","volume":"331","author":"Shah","year":"2024","journal-title":"JAMA"},{"issue":"17","key":"zoi250666r29","doi-asserted-by":"publisher","first-page":"1483","DOI":"10.1001\/jama.2025.1335","article-title":"Moving toward implementation of responsible artificial intelligence in health care: the European TRAIN Initiative.","volume":"333","author":"van Genderen","year":"2025","journal-title":"JAMA"},{"issue":"4","key":"zoi250666r30","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.4622","article-title":"Algorithmovigilance: advancing methods to analyze and monitor artificial intelligence-driven health care for effectiveness and equity.","volume":"4","author":"Embi","year":"2021","journal-title":"JAMA Netw Open"},{"issue":"1","key":"zoi250666r31","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models.","volume":"31","author":"Gallifant","year":"2025","journal-title":"Nat Med"},{"issue":"4","key":"zoi250666r32","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review.","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"issue":"7","key":"zoi250666r33","doi-asserted-by":"publisher","DOI":"10.1056\/AIra2400038","article-title":"Medical ethics of large language models in medicine.","volume":"1","author":"Ong","year":"2024","journal-title":"NEJM AI"},{"issue":"10","key":"zoi250666r34","doi-asserted-by":"publisher","DOI":"10.1136\/bmjgh-2022-009378","article-title":"Past, present and future of living systematic review: a bibliometrics analysis.","volume":"7","author":"Zheng","year":"2022","journal-title":"BMJ Glob Health"},{"key":"zoi250666r35","doi-asserted-by":"publisher","first-page":"m1328","DOI":"10.1136\/bmj.m1328","article-title":"Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal.","volume":"369","author":"Wynants","year":"2020","journal-title":"BMJ"},{"issue":"2","key":"zoi250666r36","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pmed.1001603","article-title":"Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.","volume":"11","author":"Elliott","year":"2014","journal-title":"PLoS Med"},{"issue":"4","key":"zoi250666r37","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.3909","article-title":"Comparison of methods to reduce bias from clinical prediction models of postpartum depression.","volume":"4","author":"Park","year":"2021","journal-title":"JAMA Netw Open"},{"key":"zoi250666r38","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-025-03609-7","article-title":"Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial.","author":"Rossetti","year":"2025","journal-title":"Nat Med"},{"issue":"2281","key":"zoi250666r39","doi-asserted-by":"publisher","first-page":"n2281","DOI":"10.1136\/bmj.n2281","article-title":"Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review.","volume":"375","author":"Andaur Navarro","year":"2021","journal-title":"BMJ"},{"key":"zoi250666r16","first-page":"295","author":"Gerke","year":"2020"},{"key":"zoi250666r12","unstructured":"Stanford university Human-Centered Artificial Intelligence. The AI index report: measuring trends in AI. 2024. Accessed June 3, 2025. https:\/\/hai.stanford.edu\/ai-index\/2025-ai-index-report"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2836754\/berkhout_2025_oi_250666_1752524356.7958.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,23]],"date-time":"2025-07-23T15:31:12Z","timestamp":1753284672000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2836754"}},"subtitle":["A Systematic Review"],"short-title":[],"issued":{"date-parts":[[2025,7,23]]},"references-count":39,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,7,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2025.22866","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,23]]}}}