{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T16:06:11Z","timestamp":1790697971858,"version":"4.1.0"},"reference-count":46,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T00:00:00Z","timestamp":1790640000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Clin. Med.","JCM"],"abstract":"<jats:p>Background\/Objectives: Catheter-appropriateness audits report a single rate without a measure of its reliability. We measured how far an emergency department (ED) estimate depends on the clinician applying the checklist, and which items carry that dependence. Methods: We enrolled 117 consecutive adults catheterized in a tertiary ED between 21 January and 21 May 2026. Three emergency physicians independently classified every patient against a ten-item checklist drawn from the 2009 CDC\/HICPAC guideline and the Ann Arbor criteria. A designated senior rater served as comparator, a clinical judgment rather than a criterion standard. We computed standalone rates, per-item Fleiss\u2019 \u03ba and Gwet\u2019s AC1, patient-level agreement, and a post hoc counterfactual item-replacement analysis. Results: Ten patients (8.5%, 95% CI 4.7\u201315.0) had no valid indication according to the senior rater. Auditing alone, the two index raters would have reported 0.9% and 25.6%. Pooled agreement over 1170 patient-by-indication decisions was substantial by \u03ba (0.630 and 0.707) and almost perfect by AC1 (0.885 and 0.925); excluding the complementary item changed neither. Patient-level agreement was lower (three-rater \u03ba 0.199, AC1 0.792). Disagreement concentrated in hourly urine-output monitoring (AC1 0.617) and neurogenic bladder, endorsed in 44 patients by one rater and none by another (AC1 0.668). That item accounted for 48% of one rater\u2019s excess endorsements; replacing that rater\u2019s item-5 ratings with the senior rater\u2019s narrowed the between-rater range from 30-fold to 6-fold. Conclusions: In this cohort the audit rate was rater-dependent, and the dependence concentrated in identifiable checklist items. Item-level operational definitions and prospective calibration should be evaluated as strategies for improving audit reliability, and per-item agreement should be reported with a prevalence-robust coefficient alongside \u03ba.<\/jats:p>","DOI":"10.3390\/jcm15197574","type":"journal-article","created":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T15:14:37Z","timestamp":1790694877000},"page":"7574","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Rater Dependence in Emergency Department Catheter-Appropriateness Audits: A Prospective Three-Rater Study of Item-Level Reliability"],"prefix":"10.3390","volume":"15","author":[{"given":"\u00d6mer","family":"\u00c7etin","sequence":"first","affiliation":[{"name":"Department of Emergency Medicine, Faculty of Medicine, Bolu Abant \u0130zzet Baysal University, 14030 Bolu, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8722-5402","authenticated-orcid":false,"given":"Faruk","family":"Dan\u0131\u015f","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Faculty of Medicine, Bolu Abant \u0130zzet Baysal University, 14030 Bolu, T\u00fcrkiye"},{"name":"Department of Emergency Medicine, Bolu \u0130zzet Baysal Training and Research Hospital, 14030 Bolu, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1016\/j.idc.2024.07.006","article-title":"Urinary Catheter-Associated Infections","volume":"38","author":"Kidder","year":"2024","journal-title":"Infect. Dis. Clin. N. Am."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1016\/j.idc.2021.08.003","article-title":"Urinary Tract Infections","volume":"35","author":"Chenoweth","year":"2021","journal-title":"Infect. Dis. Clin. N. Am."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1086\/651091","article-title":"Guideline for Prevention of Catheter-Associated Urinary Tract Infections 2009","volume":"31","author":"Gould","year":"2010","journal-title":"Infect. Control Hosp. Epidemiol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"185","DOI":"10.4103\/tjem.tjem_164_24","article-title":"The efficacy and application of tranexamic acid in emergency medicine: Emergency Medicine Association of T\u00fcrkiye clinical policy guideline","volume":"24","author":"Aksel","year":"2024","journal-title":"Turk. J. Emerg. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1111\/acem.12334","article-title":"Urinary Catheter Use and Appropriateness in U.S. Emergency Departments, 1995\u20132010","volume":"21","author":"Schuur","year":"2014","journal-title":"Acad. Emerg. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1557","DOI":"10.1001\/jama.2022.14378","article-title":"Trends in Urinary Catheter Use by Indication in US Emergency Departments, 2002\u20132019","volume":"328","author":"Schuur","year":"2022","journal-title":"JAMA"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.medcli.2024.09.003","article-title":"Appropriateness of urinary catheterization in the hospital setting: A multicenter observational study","volume":"164","author":"Urrutia","year":"2025","journal-title":"Med. Clin."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1017\/ice.2017.256","article-title":"Reducing Inappropriate Urinary Catheter Use in the Emergency Department: Comparing Two Collaborative Structures","volume":"39","author":"Greene","year":"2017","journal-title":"Infect. Control Hosp. Epidemiol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"864","DOI":"10.1016\/S1473-3099(19)30709-1","article-title":"De-implementation strategy to reduce inappropriate use of intravenous and urinary catheters (RICAT): A multicentre, prospective, interrupted time-series and before and after study","volume":"20","author":"Laan","year":"2020","journal-title":"Lancet Infect. Dis."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Parker, V., Giles, M., Graham, L., Suthers, B., Watts, W., O\u2019Brien, T., and Searles, A. (2017). Avoiding inappropriate urinary catheter use and catheter-associated urinary tract infection (CAUTI): A pre-post control intervention study. BMC Health Serv. Res., 17.","DOI":"10.1186\/s12913-017-2268-2"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Papanikolopoulou, A., Maltezou, H.C., Stoupis, A., Kalimeri, D., Pavli, A., Boufidou, F., Karalexi, M., Pantazis, N., Pantos, C., and Tountas, Y. (2022). Catheter-Associated Urinary Tract Infections, Bacteremia, and Infection Control Interventions in a Hospital: A Six-Year Time-Series Study. J. Clin. Med., 11.","DOI":"10.3390\/jcm11185418"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"307","DOI":"10.14744\/nci.2024.94557","article-title":"Predicting mortality in necrotizing fasciitis: Retrospective evaluation of 69 cases","volume":"12","author":"Kudu","year":"2025","journal-title":"North. Clin. Istanb."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/2036-7902-5-S1-S1","article-title":"Sources of error in emergency ultrasonography","volume":"5","author":"Pinto","year":"2013","journal-title":"Crit. Ultrasound J."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1888","DOI":"10.1056\/NEJM199806253382607","article-title":"The Reproducibility of a Method to Identify the Overuse and Underuse of Medical Procedures","volume":"338","author":"Shekelle","year":"1998","journal-title":"N. Engl. J. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1136\/qhc.11.2.125","article-title":"Quality assessment for three common conditions in primary care: Validity and reliability of review criteria developed by expert panels for angina, asthma and type 2 diabetes","volume":"11","author":"Campbell","year":"2002","journal-title":"Qual. Saf. Health Care"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Mafra, A.C.C.N., Miraglia, J.L., Colugnati, F.A.B., Padilha, G.S.L., Tadeucci, R.R.S., Almeida, E.H.P., and Bracco, M.M. (2020). A prospective evaluation of inter-rater agreement of routine medical records audits at a large general hospital in S\u00e3o Paulo, Brazil. BMC Health Serv. Res., 20.","DOI":"10.1186\/s12913-020-05495-w"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1007\/s43678-025-00937-y","article-title":"Validating the falls decision rule: Optimizing head CT use in older adults with ground-level falls","volume":"27","author":"Kudu","year":"2025","journal-title":"CJEM"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1016\/S0196-0644(99)70309-4","article-title":"Methodologic Standards for the Development of Clinical Decision Rules in Emergency Medicine","volume":"33","author":"Stiell","year":"1999","journal-title":"Ann. Emerg. Med."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1016\/0895-4356(90)90158-L","article-title":"High agreement but low Kappa: I. The problems of two paradoxes","volume":"43","author":"Feinstein","year":"1990","journal-title":"J. Clin. Epidemiol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1016\/j.jhsa.2024.01.006","article-title":"The Kappa Paradox Explained","volume":"49","author":"Derksen","year":"2024","journal-title":"J. Hand Surg. Am."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","year":"2008","journal-title":"Br. J. Math. Stat. Psychol."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhao, X., Feng, G.C., Ao, S.H., and Liu, P.L. (2022). Interrater reliability estimators tested against true interrater reliabilities. BMC Med. Res. Methodol., 22.","DOI":"10.1186\/s12874-022-01707-5"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1016\/j.jclinepi.2007.11.008","article-title":"The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies","volume":"61","author":"Altman","year":"2008","journal-title":"J. Clin. Epidemiol."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gragnano, F., Artana, C., Carrara, G., Sacco, A., Montalto, C., Mennuni, M., Zuin, M., D\u2019Ascenzo, F., Nevo, F.D., and De Sio, V. (2026). Diagnostic and therapeutic inertia for LDL cholesterol management of consecutively verified patients with acute myocardial infarction. Eur. J. Intern. Med., 106990. Online ahead of print.","DOI":"10.1016\/j.ejim.2026.106990"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.7326\/M14-1304","article-title":"The Ann Arbor Criteria for Appropriate Urinary Catheter Use in Hospitalized Medical Patients: Results Obtained by Using the RAND\/UCLA Appropriateness Method","volume":"162","author":"Meddings","year":"2015","journal-title":"Ann. Intern. Med."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1002\/(SICI)1097-0258(19980115)17:1<101::AID-SIM727>3.0.CO;2-E","article-title":"Sample size and optimal designs for reliability studies","volume":"17","author":"Walter","year":"1998","journal-title":"Stat. Med."},{"key":"ref_27","unstructured":"R Core Team (2026). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.jclinepi.2010.03.002","article-title":"Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed","volume":"64","author":"Kottner","year":"2011","journal-title":"J. Clin. Epidemiol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The Measurement of Observer Agreement for Categorical Data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: The kappa statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem. Med."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Chen, Z., Tian, F., Wan, X., Zhou, F., and Wu, F. (2026). Potentially Inappropriate Medication Use in Older Adults: Concordance Among the Chinese and AGS\/Beers and STOPP Criteria. J. Clin. Med., 15.","DOI":"10.3390\/jcm15155826"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/s40292-025-00705-0","article-title":"Discrepancies Between Physician-Perceived and Calculated Cardiovascular Risk in Primary Prevention: Implications for LDL-C Target Achievement and Appropriate Lipid-Lowering Therapy","volume":"32","author":"Cesaro","year":"2025","journal-title":"High Blood Press. Cardiovasc. Prev."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Kaiser, I., Pfahlberg, A.B., Mathes, S., Uter, W., Diehl, K., Steeb, T., Heppt, M.V., and Gefeller, O. (2023). Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training. J. Clin. Med., 12.","DOI":"10.3390\/jcm12051976"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Dolotova, D.D., Solominova, T.A., Polunina, N.A., Blagosklonova, E.R., Plyusova, N.S., Ramazanov, G.R., Muslimov, R.S., Solominov, M.V., and Gavrilov, A.V. (2026). Inter-Rater Reliability of Subarachnoid Hemorrhage Radiological Grading Scales: A Systematic Review and Meta-Analysis. J. Clin. Med., 15.","DOI":"10.3390\/jcm15082899"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Hu, F.W., Shih, H.I., Hsu, H.C., Chen, C.H., and Chang, C.M. (2018). Dynamic changes in the appropriateness of urinary catheter use among hospitalized older patients in the emergency department. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0193905"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"e19982","DOI":"10.1016\/j.heliyon.2023.e19982","article-title":"Usability of EtCO2 values in the decision to terminate resuscitation by integrating them into the TOR rule (an extended TOR rule): A preliminary analysis","volume":"9","author":"Kudu","year":"2023","journal-title":"Heliyon"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e42361","DOI":"10.1097\/MD.0000000000042361","article-title":"The value of optic nerve sheath measurement in patients presenting to the emergency department with hypertensive crisis","volume":"104","author":"Kudu","year":"2025","journal-title":"Medicine"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1373\/clinchem.2007.096032","article-title":"Bias in Sensitivity and Specificity Caused by Data-Driven Selection of Optimal Cutoff Values: Mechanisms, Magnitude, and Solutions","volume":"54","author":"Leeflang","year":"2008","journal-title":"Clin. Chem."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1002\/pst.1659","article-title":"The disagreeable behaviour of the kappa statistic","volume":"14","author":"Flight","year":"2014","journal-title":"Pharm. Stat."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Konstantinidis, M., Le, L.W., and Gao, X. (2022). An Empirical Comparative Assessment of Inter-Rater Agreement of Binary Outcomes and Multiple Raters. Symmetry, 14.","DOI":"10.3390\/sym14020262"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1016\/j.jclinepi.2021.09.004","article-title":"Kappa and AC1\/2 statistics: Beyond the paradox","volume":"142","author":"Minozzi","year":"2022","journal-title":"J. Clin. Epidemiol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1136\/qhc.11.4.358","article-title":"Research methods used in developing and applying quality indicators in primary care","volume":"11","author":"Campbell","year":"2002","journal-title":"Qual. Saf. Health Care"},{"key":"ref_43","first-page":"e50818","article-title":"Comparative Analysis of Perfusion Index and End-Tidal Carbon Dioxide in Cardiac Arrest Patients: Implications for Hemodynamic Monitoring and Resuscitation Outcomes","volume":"15","author":"Kudu","year":"2023","journal-title":"Cureus"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"336","DOI":"10.24938\/kutfd.1543770","article-title":"Assessing the prognostic value of the hemoglobin-to-red cell distribution width ratio in emergency department patients with acute coronary syndrome","volume":"26","author":"Kudu","year":"2024","journal-title":"K\u0131r\u0131kkale \u00dcniversitesi T\u0131p Fak\u00fcltesi Derg."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1925","DOI":"10.1093\/eurheartj\/ehu207","article-title":"Towards better clinical prediction models: Seven steps for development and an ABCD for validation","volume":"35","author":"Steyerberg","year":"2014","journal-title":"Eur. Heart J."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"No\u017cewski, J., Bondarczuk, R., Ho\u0142ody, G., Kitt, M., Mu\u0107ka, N., and Religioni, U. (2025). Barriers to Improving Pain Management in the Emergency Department: Lessons from a Lean-Driven Quality Improvement Initiative. J. Clin. Med., 14.","DOI":"10.3390\/jcm14134566"}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/19\/7574\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T15:30:07Z","timestamp":1790695807000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/19\/7574"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,29]]},"references-count":46,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["jcm15197574"],"URL":"https:\/\/doi.org\/10.3390\/jcm15197574","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,29]]}}}