{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T05:44:52Z","timestamp":1790401492917,"version":"4.1.0"},"reference-count":22,"publisher":"BMJ","issue":"12","license":[{"start":{"date-parts":[[2025,12,23]],"date-time":"2025-12-23T00:00:00Z","timestamp":1766448000000},"content-version":"unspecified","delay-in-days":22,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"publisher","award":["R35GM155262"],"award-info":[{"award-number":["R35GM155262"]}],"id":[{"id":"10.13039\/100000057","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2025,12,8]]},"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Few artificial intelligence (AI) clinical decision support systems (CDSSs) are ever evaluated in practice. Although some signal of clinical effectiveness may be needed to justify AI deployment and testing, such data are typically unavailable in early-stage research. This conundrum is especially relevant in the intensive care unit (ICU), where conditions like sepsis and acute respiratory distress syndrome (ARDS) require high-stakes decisions. Our group developed the AI ventilator assistant (AVA), a novel AI CDSS for patients with sepsis ARDS receiving invasive mechanical ventilation. But the promising results of predictive performance estimates are not sufficient to assess AVA\u2019s clinical safety and appropriateness prior to future evaluation and deployment. Therefore, we propose a Clinician Turing Test as a novel validation approach to determine whether clinicians can distinguish AVA-generated treatment recommendations from those enacted by real human clinicians. If AVA\u2019s recommendations are consistently indistinguishable from those of real clinicians, thereby \u2018passing\u2019 this Turing test, this would provide a strong preclinical signal of safety and appropriateness.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods and analysis<\/jats:title>\n                    <jats:p>This multisite, randomised, electronic, vignette-based Phase 1b study will use a Clinician Turing Test design. We aim to recruit 350 critical care clinicians, including physicians and advanced practice providers from six US hospitals. Participants will review nine clinical vignettes of patients with sepsis and ARDS derived from the Molecular Epidemiology of Severe Sepsis in the ICU cohort and an associated profile of a suggested treatment plan. For each participant\u2013vignette combination, the source of the treatment profile will be randomly assigned (AI-generated by AVA vs the actually enacted treatment from real human clinicians) in a 1:1 allocation. The primary endpoint is the participants\u2019 accuracy in identifying whether a treatment profile was AI-generated or human-generated, assessed using equivalence testing through a mixed-effects logistic regression model with random effects for participants and vignettes. Secondarily, a fitted binary classifier will assess discrimination ability using the C-statistic. Secondary endpoints include clinicians\u2019 perceptions of the safety and appropriateness of the treatment profiles, confidence in distinguishing AI-generated and human-generated recommendations, interest in AI CDSSs for sepsis and ventilator management and the time to complete the survey. This novel Phase 1b design provides preliminary but essential information about an AI CDSS\u2019s clinical appropriateness without the risk or cost of actual deployment, thereby informing decisions about future clinical implementation and evaluation in real clinical environments.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Ethics and dissemination<\/jats:title>\n                    <jats:p>This protocol was approved by the Institutional Review Board of the University of Pennsylvania (Protocol #858201). Results are expected in 2026 and will be submitted for publication in peer-reviewed journals and presented at scientific conferences.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Trial registration number<\/jats:title>\n                    <jats:p>\n                      <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"clintrialgov\" xlink:href=\"NCT07025096\">NCT07025096<\/jats:ext-link>\n                      .\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjopen-2025-106757","type":"journal-article","created":{"date-parts":[[2025,12,25]],"date-time":"2025-12-25T06:10:34Z","timestamp":1766643034000},"page":"e106757","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":3,"title":["Evaluating AI-based comprehensive clinical decision support for sepsis and ARDS: protocol for a Clinician Turing Test"],"prefix":"10.1136","volume":"15","clinical-trial-number":[{"clinical-trial-number":"nct07025096","registry":"10.18810\/clinical-trials-gov"},{"clinical-trial-number":"nct07025096","registry":"10.18810\/clinical-trials-gov"}],"author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2470-8209","authenticated-orcid":false,"given":"Antonia","family":"Angeli Gazola","sequence":"first","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6324-8140","authenticated-orcid":false,"given":"Nicholas S","family":"Bishop","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin E","family":"Schmid","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7007-4908","authenticated-orcid":false,"given":"Romain","family":"Pirracchio","sequence":"additional","affiliation":[{"name":"Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas S","family":"Valley","sequence":"additional","affiliation":[{"name":"Division of Pulmonary and Critical Care, Department of Medicine, University of Michigan, Ann Arbor, Michigan, USA"},{"name":"Center for Bioethics and Social Sciences in Medicine, University of Michigan, Ann Arbor, Michigan, USA"},{"name":"Institute for Healthcare Policy and Innovation, University of Michigan, Ann Arbor, Michigan, USA"},{"name":"Department of Veterans Affairs, VA Ann Arbor Healthcare System, Ann Arbor, Michigan, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sivasubramanium V","family":"Bhavani","sequence":"additional","affiliation":[{"name":"Department of Medicine, Emory University, Atlanta, Georgia, USA"},{"name":"Emory Critical Care Center, Atlanta, Georgia, USA"},{"name":"Division of Pulmonary, Allergy, Critical Care and Sleep Medicine, Emory University School of Medicine, Atlanta, Georgia, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dustin C","family":"Krutsinger","sequence":"additional","affiliation":[{"name":"Divisions of Pulmonary, Critical Care, and Sleep Medicine, University of Nebraska Medical Center, Omaha, Nebraska, USA"},{"name":"Department of Anesthesiology, University of Nebraska Medical Center, Omaha, Nebraska, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heather M","family":"Giannini","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA"},{"name":"Virtua Pulmonology and Sleep Medicine, Virtua Health Inc, Marlton, New Jersey, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yingying","family":"Lu","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lyle H","family":"Ungar","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science (CIS), University of Pennsylvania, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nuala J","family":"Meyer","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA"},{"name":"Translational Lung Biology, Lung Biology Institute, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Meeta Prasad","family":"Kerlin","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"},{"name":"Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA"},{"name":"Leonard Davis Institute of Health Economics, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9588-3819","authenticated-orcid":false,"given":"Gary E","family":"Weissman","sequence":"additional","affiliation":[{"name":"Palliative and Advanced Illness Research (PAIR) Center, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA"},{"name":"Department of Medicine, University of Pennsylvania Division of Pulmonary Allergy and Critical Care, Philadelphia, Pennsylvania, USA"},{"name":"Leonard Davis Institute of Health Economics, Philadelphia, Pennsylvania, USA"},{"name":"Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2025,12,24]]},"reference":[{"key":"2026010703482736000_15.12.e106757.1","doi-asserted-by":"publisher","DOI":"10.1007\/s00134-020-06045-y"},{"key":"2026010703482736000_15.12.e106757.2","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2022.33946","article-title":"Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review","volume":"5","author":"Plana","year":"2022","journal-title":"JAMA Netw Open"},{"key":"2026010703482736000_15.12.e106757.3","doi-asserted-by":"crossref","DOI":"10.1056\/AIp2400223","article-title":"A Call for Artificial Intelligence Implementation Science Centers to Evaluate Clinical Effectiveness","volume":"1","author":"Longhurst","year":"2024","journal-title":"NEJM AI"},{"key":"2026010703482736000_15.12.e106757.4","doi-asserted-by":"publisher","DOI":"10.1093\/mind\/LIX.236.433"},{"key":"2026010703482736000_15.12.e106757.5","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2313925121","article-title":"A Turing test of whether AI chatbots are behaviorally similar to humans","volume":"121","author":"Mei","year":"2024","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2026010703482736000_15.12.e106757.6","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1038\/s43018-020-0041-7","article-title":"A Turing test for artificial intelligence in cancer","volume":"1","year":"2020","journal-title":"Nat Cancer"},{"key":"2026010703482736000_15.12.e106757.7","first-page":"1","article-title":"Turing test-inspired method for analysis of biases prevalent in artificial intelligence-based medical imaging","volume":"2","author":"Tripathi","year":"2022","journal-title":"AI Ethics"},{"key":"2026010703482736000_15.12.e106757.8","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0290773","article-title":"ChatGPT- versus human-generated answers to frequently asked questions about diabetes: A Turing test-inspired survey among employees of a Danish diabetes center","volume":"18","author":"Hulman","year":"2023","journal-title":"PLoS One"},{"key":"2026010703482736000_15.12.e106757.9","doi-asserted-by":"crossref","unstructured":"Weissman GE , Bishop NS , Ungar LH , et al . Training and validation of a pandemic clinical decision support model for ventilators using collective intelligence and imitation learning. American Thoracic Society 2024 International Conference, May 17-22, 2024 - San Diego, CA; May 2024:A5057. doi:10.1164\/ajrccm-conference.2024.209.1_MeetingAbstracts.A5057 Available: https:\/\/www.atsjournals.org\/doi\/book\/10.1164\/ajrccm-conference.2024.C22","DOI":"10.1164\/ajrccm-conference.2024.209.1_MeetingAbstracts.A5057"},{"key":"2026010703482736000_15.12.e106757.10","doi-asserted-by":"crossref","DOI":"10.1002\/lrh2.10439","article-title":"The Complete Inpatient Record Using Comprehensive Electronic Data (CIRCE) project: A team-based approach to clinically validated, research-ready electronic health record data","volume":"9","author":"Schneider","year":"2025","journal-title":"Learn Health Syst"},{"key":"2026010703482736000_15.12.e106757.11","doi-asserted-by":"crossref","DOI":"10.1186\/s13054-016-1244-2","article-title":"Low Plasma Levels of Adiponectin Do Not Explain Acute Respiratory Distress Syndrome Risk: a Prospective Cohort Study of Patients with Severe Sepsis","volume":"20","author":"Palakshappa","year":"2016","journal-title":"Crit Care"},{"key":"2026010703482736000_15.12.e106757.12","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2288-11-94","article-title":"Simulation methods to estimate design power: an overview for applied research","volume":"11","author":"Arnold","year":"2011","journal-title":"BMC Med Res Methodol"},{"key":"2026010703482736000_15.12.e106757.13","doi-asserted-by":"publisher","DOI":"10.1016\/j.bbmt.2008.10.004"},{"key":"2026010703482736000_15.12.e106757.14","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1177\/2515245918770963","article-title":"Equivalence Testing for Psychological Research: A Tutorial","volume":"1","author":"Lakens","year":"2018","journal-title":"Advances in Methods and Practices in Psychological Science"},{"key":"2026010703482736000_15.12.e106757.15","unstructured":"R Core Team . R Foundation for Statistical Computing Vienna (Austria); R: A language and environment for statistical computing, 2021. Available: https:\/\/www.R-project.org\/"},{"key":"2026010703482736000_15.12.e106757.16","doi-asserted-by":"crossref","first-page":"274","DOI":"10.7326\/M22-2998","article-title":"Chess Lessons: Harnessing Collective Human Intelligence and Imitation Learning to Support Clinical Decisions","volume":"176","author":"Weissman","year":"2023","journal-title":"Ann Intern Med"},{"key":"2026010703482736000_15.12.e106757.17","doi-asserted-by":"crossref","unstructured":"Hulman A , Dollerup OL , Mortensen JF , et al . Study protocol: ChatGPT-generated versus expert-written answers to frequently asked questions about diabetes - an e-survey among all employees of a Danish diabetes center. Figshare 2023. doi:10.1101\/2023.02.13.23285745","DOI":"10.1101\/2023.02.13.23285745"},{"key":"2026010703482736000_15.12.e106757.18","doi-asserted-by":"crossref","first-page":"100496","DOI":"10.1016\/j.chbr.2024.100496","article-title":"Turing tests in chess: An experiment revealing the role of human subjectivity","volume":"16","author":"Eisma","year":"2024","journal-title":"Computers in Human Behavior Reports"},{"key":"2026010703482736000_15.12.e106757.19","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0279349","article-title":"Evaluating diagnostic content of AI-generated chest radiography: A multi-center visual Turing test","volume":"18","author":"Myong","year":"2023","journal-title":"PLoS One"},{"key":"2026010703482736000_15.12.e106757.20","article-title":"A multi-Task Learning based applicable AI model simultaneously predicts stage, histology, grade and LNM for cervical cancer before surgery","volume":"24","author":"Wang","year":"2023","journal-title":"BMC Womens Health"},{"key":"2026010703482736000_15.12.e106757.21","article-title":"A blind randomized validated convolutional neural network for auto-segmentation of clinical target volume in rectal cancer patients receiving neoadjuvant radiotherapy","volume":"12","author":"Wu","year":"2023","journal-title":"Cancer Med"},{"key":"2026010703482736000_15.12.e106757.22","doi-asserted-by":"crossref","DOI":"10.2196\/46939","article-title":"Putting ChatGPT\u2019s Medical Advice to the (Turing) Test: Survey Study","volume":"9","author":"Nov","year":"2023","journal-title":"JMIR Med Educ"}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2025-106757","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,7]],"date-time":"2026-01-07T11:51:56Z","timestamp":1767786716000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2025-106757"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12]]},"references-count":22,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12,24]]},"published-print":{"date-parts":[[2025,12]]}},"alternative-id":["10.1136\/bmjopen-2025-106757"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2025-106757","relation":{},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12]]}}}