{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T00:03:04Z","timestamp":1790121784676,"version":"4.0.1"},"reference-count":35,"publisher":"Georg Thieme Verlag KG","issue":"04","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Objective Assess inter-rater agreement on clinician- and clinical decision support (CDS)-driven diuretic titrations and risk classifications to benchmark a semi-autonomous CDS (OTTO-FM).<\/jats:p>\n                  <jats:p>Methods Secondary analysis of prospectively collected porcine data modeling postoperative fluid overload. Three pediatric cardiac intensivists rated items (reasonable\/unsure\/unreasonable) for clinician-driven (human phase; 29 items) and CDS-driven titrations with risk classification (CDS phase; 44 items). Inter-rater agreement was estimated with ordinal-weighted Gwet\u2019s AC2.<\/jats:p>\n                  <jats:p>Results Dosing inter-rater agreement was similar between phases (human AC2: 0.79 [0.61\u20130.92]; CDS: 0.83 [0.68\u20130.92]), though unanimity was reached on only 19 of 29 (65.5%) human-phase and 31 of 44 (70.5%) CDS-phase items. Risk-label inter-rater agreement stratified sharply by label: high-risk, 0.92 (0.74\u20130.95) versus low-risk, 0.54 (0.28\u20130.75). Disagreement also differed in structure between tasks: 19 of 20 nonunanimous risk items were minority dissent (one rater against a two-rater majority), including 16 of 17 among low-risk items, whereas 7 of 13 nonunanimous CDS dosing items were three-way splits.<\/jats:p>\n                  <jats:p>Conclusion Inter-rater agreement on dosing was substantial and similar across phases, but not unanimous, suggesting that perfect reasonableness consensus with CDS may be unrealistic. Agreement on risk classification was lower, and the structure of disagreement differed between CDS dose and risk. Empirical inter-rater agreement baselines may contextualize CDS evaluation, and these differences may have implications for the design of semi-autonomous CDS if confirmed in larger studies.<\/jats:p>","DOI":"10.1055\/a-2956-4559","type":"journal-article","created":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T23:21:32Z","timestamp":1790119292000},"page":"817-824","source":"Crossref","is-referenced-by-count":0,"title":["Measuring Expert Inter-Rater Agreement with a Semi-Automated Clinical Decision Support System: Who Agrees with What?"],"prefix":"10.1055","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-6313-9230","authenticated-orcid":false,"given":"Yasmin","family":"Benhalim","sequence":"additional","affiliation":[{"name":"University of Michigan Medical School, Michigan, United States","place":["Ann Arbor"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew W.","family":"Hodgman","sequence":"additional","affiliation":[{"name":"University of Michigan Medical School, Michigan, United States","place":["Ann Arbor"],"department":["Department of Computational Medicine and Bioinformatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John R.","family":"Charpie","sequence":"additional","affiliation":[{"name":"University of Michigan Medical School, Michigan, United States","place":["Ann Arbor"],"department":["Department of Pediatrics, Division of Cardiology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabe E.","family":"Owens","sequence":"additional","affiliation":[{"name":"University of Michigan Medical School, Michigan, United States","place":["Ann Arbor"],"department":["Department of Pediatrics, Division of Cardiology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ranjit","family":"Aiyagari","sequence":"additional","affiliation":[{"name":"University of Michigan Medical School, Michigan, United States","place":["Ann Arbor"],"department":["Department of Pediatrics, Division of Cardiology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6367-2865","authenticated-orcid":false,"given":"Daniel E.","family":"Ehrmann","sequence":"additional","affiliation":[{"name":"University of Michigan Medical School, Michigan, United States","place":["Ann Arbor"],"department":["Department of Pediatrics, Division of Cardiology"]},{"name":"Congenital Heart Center at C.S. Mott Children\u2019s Hospital, Michigan, United States","place":["Ann Arbor"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"194","published-online":{"date-parts":[[2026,9,22]]},"reference":[{"issue":"08","key":"ref1","doi-asserted-by":"crossref","first-page":"e075239","DOI":"10.1136\/bmjopen-2023-075239","article-title":"Decision aids in the ICU: a scoping review","volume":"13","author":"Y Lei","year":"2023","journal-title":"BMJ Open"},{"issue":"17","key":"ref2","doi-asserted-by":"crossref","first-page":"1564","DOI":"10.1056\/NEJMp2404691","article-title":"Compared with what? Measuring AI against the health care we have","volume":"391","author":"I S Kohane","year":"2024","journal-title":"N Engl J Med"},{"issue":"01","key":"ref3","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1038\/s41746-025-01764-2","article-title":"Clinician-informed XAI evaluation checklist with metrics (CLIX-M) for AI-powered clinical decision support systems","volume":"8","author":"A Brankovic","year":"2025","journal-title":"NPJ Digit Med"},{"issue":"01","key":"ref4","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1186\/s13054-016-1262-0","article-title":"Development of an algorithm to aid triage decisions for intensive care unit admission: a clinical vignette and retrospective cohort study","volume":"20","author":"J GR Ramos","year":"2016","journal-title":"Crit Care"},{"issue":"03","key":"ref5","doi-asserted-by":"crossref","first-page":"406.e3","DOI":"10.1016\/j.jpainsymman.2018.06.003","article-title":"Inter-rater agreement of intensivists evaluating the goal concordance of preference-sensitive ICU interventions","volume":"56","author":"A E Turnbull","year":"2018","journal-title":"J Pain Symptom Manag"},{"issue":"01","key":"ref6","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1186\/2110-5820-4-2","article-title":"Mechanical ventilation and clinical practice heterogeneity in intensive care units: a multicenter case-vignette study","volume":"4","author":"Y L Nguyen","year":"2014","journal-title":"Ann Intensive Care"},{"issue":"02","key":"ref7","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1055\/a-2272-6184","article-title":"Usability testing of situation awareness clinical decision support in the intensive care unit","volume":"15","author":"M J Molloy","year":"2024","journal-title":"Appl Clin Inform"},{"issue":"03","key":"ref8","doi-asserted-by":"crossref","first-page":"033001","DOI":"10.1088\/2516-1091\/ac6d36","article-title":"Physiological closed-loop control in critical care: opportunities for innovations","volume":"4","author":"J O Hahn","year":"2022","journal-title":"Prog Biomed Eng"},{"issue":"03","key":"ref9","doi-asserted-by":"crossref","first-page":"516","DOI":"10.1055\/a-2535-5823","article-title":"Sharing a hybrid electronic health record + fast healthcare interoperability resources clinical decision support across health systems: automating smoking cessation for pediatric caregivers","volume":"16","author":"S N Saleh","year":"2025","journal-title":"Appl Clin Inform"},{"issue":"03","key":"ref11","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1177\/0018720810376055","article-title":"Complacency and bias in human use of automation: an attentional integration","volume":"52","author":"R Parasuraman","year":"2010","journal-title":"Hum Factors"},{"issue":"18","key":"ref12","doi-asserted-by":"crossref","first-page":"1707","DOI":"10.1056\/NEJMoa1907863","article-title":"Six-month randomized, multicenter trial of closed-loop control in type 1 diabetes","volume":"381","author":"S A Brown","year":"2019","journal-title":"N Engl J Med"},{"issue":"05","key":"ref13","doi-asserted-by":"crossref","first-page":"R196","DOI":"10.1186\/cc12890","article-title":"Feasibility study on full closed-loop control ventilation (IntelliVent-ASV) in ICU patients with acute respiratory failure: a prospective observational comparative study","volume":"17","author":"J M Arnal","year":"2013","journal-title":"Crit Care"},{"issue":"09","key":"ref15","doi-asserted-by":"crossref","first-page":"3129","DOI":"10.1007\/s00467-023-05929-7","article-title":"Early postoperative weight-based fluid overload is associated with worse outcomes after neonatal cardiac surgery","volume":"38","author":"K L Brandewie","year":"2023","journal-title":"Pediatr Nephrol"},{"issue":"01","key":"ref16","doi-asserted-by":"crossref","first-page":"14175","DOI":"10.1038\/s41598-019-50481-w","article-title":"Alkaline phosphatase treatment of acute kidney injury in an infant piglet model of cardiopulmonary bypass with deep hypothermic circulatory arrest","volume":"9","author":"J A Davidson","year":"2019","journal-title":"Sci Rep"},{"issue":"06","key":"ref17","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1177\/02676591241309824","article-title":"Recovery model for minimally invasive central cannulation, cardiopulmonary bypass, and cardioplegic arrest in quadrupeds","volume":"40","author":"M D Johnson","year":"2025","journal-title":"Perfusion"},{"issue":"01","key":"ref18","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1186\/s40635-022-00459-2","article-title":"Closed-loop automated critical care as proof-of-concept study for resuscitation in a swine model of ischemia-reperfusion injury","volume":"10","author":"N TP Patel","year":"2022","journal-title":"Intensive Care Med Exp"},{"issue":"03","key":"ref19","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1097\/MAT.0000000000000058","article-title":"A novel rotary pulsatile flow pump for cardiopulmonary bypass","volume":"60","author":"N R Teman","year":"2014","journal-title":"ASAIO J"},{"issue":"03","key":"ref20","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1097\/ALN.0000000000002581","article-title":"Automated titration of vasopressor infusion using a closed-loop controller: in vivo feasibility study using a swine model","volume":"130","author":"A Joosten","year":"2019","journal-title":"Anesthesiology"},{"issue":"04","key":"ref21","doi-asserted-by":"crossref","first-page":"474","DOI":"10.1097\/MAT.0000000000000228","article-title":"Successful porcine renal transplantation after 60 minutes of donor warm ischemia: extracorporeal perfusion and thrombolytics","volume":"61","author":"D S Demos","year":"2015","journal-title":"ASAIO J"},{"issue":"203","key":"ref22","doi-asserted-by":"publisher","DOI":"10.3791\/65967","article-title":"Adult and pediatric porcine model of acute volume overload","author":"B D Alvis","year":"2024","journal-title":"J Vis Exp"},{"issue":"01","key":"ref23","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1109\/JBHI.2022.3211765","article-title":"A novel tropical geometry-based interpretable machine learning method: pilot application to delivery of advanced heart failure therapies","volume":"27","author":"H Yao","year":"2023","journal-title":"IEEE J Biomed Health Inform"},{"issue":"11","key":"ref24","doi-asserted-by":"crossref","first-page":"e0295016","DOI":"10.1371\/journal.pone.0295016","article-title":"Predicting need for heart failure advanced therapies using an interpretable tropical geometry-based fuzzy neural network","volume":"18","author":"Y Zhang","year":"2023","journal-title":"PLoS One"},{"issue":"10","key":"ref25","doi-asserted-by":"crossref","first-page":"E424","DOI":"10.1016\/S0735-1097(13)60424-6","article-title":"Vasoactive inotropic score (VIS) is associated with outcome after infant cardiac surgery: a report of the pediatric cardiac critical care consortium (PC4)","volume":"61","author":"M Gaies","year":"2013","journal-title":"J Am Coll Cardiol"},{"issue":"02","key":"ref26","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1007\/s10157-019-01805-7","article-title":"Kidney disease: improving global outcomes in neonates with acute kidney injury after cardiac surgery","volume":"24","author":"K Ueno","year":"2020","journal-title":"Clin Exp Nephrol"},{"key":"ref28","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"K L Gwet","year":"2008","journal-title":"Br J Math Stat Psychol"},{"key":"ref30","volume-title":"R: A Language and Environment for Statistical Computing","author":"R Core Team","year":"2025"},{"issue":"06","key":"ref31","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1016\/0895-4356(90)90159-M","article-title":"High agreement but low kappa: II. Resolving the paradoxes","volume":"43","author":"D V Cicchetti","year":"1990","journal-title":"J Clin Epidemiol"},{"issue":"03","key":"ref32","doi-asserted-by":"crossref","first-page":"890","DOI":"10.2307\/2532810","article-title":"An introduction to the bootstrap","volume":"50","author":"S T Buckland","year":"1994","journal-title":"Biometrics"},{"issue":"01","key":"ref33","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"J R Landis","year":"1977","journal-title":"Biometrics"},{"issue":"04","key":"ref34","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit","volume":"70","author":"J Cohen","year":"1968","journal-title":"Psychol Bull"},{"issue":"03","key":"ref35","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1080\/00029238.1971.11080840","article-title":"A new procedure for assessing reliability of scoring EEG sleep recordings","volume":"11","author":"D V Cicchetti","year":"1971","journal-title":"Am J EEG Technol"},{"issue":"02","key":"ref36","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1016\/j.jbi.2007.09.003","article-title":"Grand challenges in clinical decision support","volume":"41","author":"D F Sittig","year":"2008","journal-title":"J Biomed Inform"},{"issue":"08","key":"ref37","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"A Wong","year":"2021","journal-title":"JAMA Intern Med"},{"issue":"03","key":"ref38","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1136\/amiajnl-2013-001813","article-title":"Overrides of medication-related clinical decision support alerts in outpatients","volume":"21","author":"K C Nanji","year":"2014","journal-title":"J Am Med Inform Assoc"},{"issue":"01","key":"ref39","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1257\/pol.20150421","article-title":"Physician beliefs and patient preferences: a new look at regional variation in health care spending","volume":"11","author":"D Cutler","year":"2019","journal-title":"Am Econ J Econ Policy"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/a-2956-4559.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T23:21:38Z","timestamp":1790119298000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/a-2956-4559"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":35,"journal-issue":{"issue":"04","published-online":{"date-parts":[[2026,8,14]]},"published-print":{"date-parts":[[2026,8]]}},"URL":"https:\/\/doi.org\/10.1055\/a-2956-4559","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8]]}}}