{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,23]],"date-time":"2025-09-23T13:25:28Z","timestamp":1758633928847,"version":"3.44.0"},"reference-count":35,"publisher":"BMJ","issue":"8","content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Qual Saf"],"accepted":{"date-parts":[[2014,1,8]]},"published-print":{"date-parts":[[2014,8]]},"abstract":"<jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To assess the inter-rater reliability (IRR) of two novel observation tools for measuring surgical safety checklist performance and teamwork.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Summary background<\/jats:title>\n                  <jats:p>Data surgical safety checklists can promote adherence to standards of care and improve teamwork in the operating room. Their use has been associated with reductions in mortality and other postoperative complications. However, checklist effectiveness depends on how well they are performed.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Authors from the Safe Surgery 2015 initiative developed a pair of novel observation tools through literature review, expert consultation and end-user testing. In one South Carolina hospital participating in the initiative, two observers jointly attended 50 surgical cases and independently rated surgical teams using both tools. We used descriptive statistics to measure checklist performance and teamwork at the hospital. We assessed IRR by measuring percent agreement, Cohen's \u03ba, and weighted \u03ba scores.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The overall percent agreement and \u03ba between the two observers was 93% and 0.74 (95% CI 0.66 to 0.79), respectively, for the Checklist Coaching Tool and 86% and 0.84 (95% CI 0.77 to 0.90) for the Surgical Teamwork Tool. Percent agreement for individual sections of both tools was 79% or higher. Additionally, \u03ba scores for six of eight sections on the Checklist Coaching Tool and for two of five domains on the Surgical Teamwork Tool achieved the desired 0.7 threshold. However, teamwork scores were high and variation was limited. There were no significant changes in the percent agreement or \u03ba scores between the first 10 and last 10 cases observed.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Both tools demonstrated substantial IRR and required limited training to use. These instruments may be used to observe checklist performance and teamwork in the operating room. However, further refinement and calibration of observer expectations, particularly in rating teamwork, could improve the utility of the tools.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1136\/bmjqs-2013-002446","type":"journal-article","created":{"date-parts":[[2014,2,5]],"date-time":"2014-02-05T00:19:24Z","timestamp":1391559564000},"page":"639-650","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":19,"title":["The Surgical Safety Checklist and Teamwork Coaching Tools: a study of inter-rater reliability"],"prefix":"10.1136","volume":"23","author":[{"given":"Lyen C","family":"Huang","sequence":"first","affiliation":[{"name":"Ariadne Labs: a joint center for health system innovation at the Brigham and Women's Hospital and Harvard School of Public Health, Boston, Massachusetts, USA"},{"name":"Department of Surgery, Stanford University School of Medicine, Stanford, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dante","family":"Conley","sequence":"additional","affiliation":[{"name":"Department of Health Policy and Management, Harvard School of Public Health, Boston, Massachusetts, USA"},{"name":"Tanana Valley Clinic, Fairbanks, Alaska, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stu","family":"Lipsitz","sequence":"additional","affiliation":[{"name":"Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher C","family":"Wright","sequence":"additional","affiliation":[{"name":"Greenville Health System, Greenville, South Carolina, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thomas W","family":"Diller","sequence":"additional","affiliation":[{"name":"Greenville Health System, Greenville, South Carolina, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lizabeth","family":"Edmondson","sequence":"additional","affiliation":[{"name":"Ariadne Labs: a joint center for health system innovation at the Brigham and Women's Hospital and Harvard School of Public Health, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William R","family":"Berry","sequence":"additional","affiliation":[{"name":"Ariadne Labs: a joint center for health system innovation at the Brigham and Women's Hospital and Harvard School of Public Health, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sara J","family":"Singer","sequence":"additional","affiliation":[{"name":"Department of Health Policy and Management, Harvard School of Public Health, Boston, Massachusetts, USA"},{"name":"Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA"},{"name":"Mongan Institute for Health Policy, Massachusetts General Hospital, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"239","published-online":{"date-parts":[[2014,2,4]]},"reference":[{"key":"2025090410375777000_23.8.639.1","first-page":"695","article-title":"The effect of the WHO surgical safety checklist on complication rate and communication","volume":"109","author":"Fudickar","year":"2012","journal-title":"Dtsch Arztebl Int"},{"key":"2025090410375777000_23.8.639.2","doi-asserted-by":"crossref","unstructured":"Treadwell JR Lucas S Tsou AY . Surgical checklists: a systematic review of impacts and implementation. BMJ Qual Saf 2014;23:299\u2013318.","DOI":"10.1136\/bmjqs-2012-001797"},{"key":"2025090410375777000_23.8.639.3","doi-asserted-by":"publisher","DOI":"10.1136\/bmjqs.2009.040022"},{"key":"2025090410375777000_23.8.639.4","doi-asserted-by":"publisher","DOI":"10.1111\/j.1399-6576.2011.02525.x"},{"key":"2025090410375777000_23.8.639.5","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMsa0810119"},{"key":"2025090410375777000_23.8.639.6","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMsa0911535"},{"key":"2025090410375777000_23.8.639.7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jamcollsurg.2012.07.015"},{"key":"2025090410375777000_23.8.639.8","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.0b013e31823779ae"},{"key":"2025090410375777000_23.8.639.9","first-page":"48","article-title":"Compliance and quality in administration of a surgical safety checklist in a tertiary New Zealand hospital","volume":"124","author":"Vogts","year":"2011","journal-title":"N Z Med J"},{"key":"2025090410375777000_23.8.639.10","doi-asserted-by":"publisher","DOI":"10.1136\/bmjqs-2011-000094"},{"key":"2025090410375777000_23.8.639.11","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1016\/j.surg.2012.05.034","article-title":"Implementing a surgical checklist: more than checking a box","volume":"152","author":"Levy","year":"2012","journal-title":"Surgeon"},{"key":"2025090410375777000_23.8.639.12","first-page":"287","article-title":"Paradoxical effects of interprofessional briefings on OR team performance","volume":"10","author":"Whyte","year":"2008","journal-title":"Cogn Technol Work"},{"key":"2025090410375777000_23.8.639.13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jamcollsurg.2011.01.052"},{"key":"2025090410375777000_23.8.639.14","doi-asserted-by":"publisher","DOI":"10.1111\/pan.12018"},{"key":"2025090410375777000_23.8.639.15","doi-asserted-by":"crossref","unstructured":"Valentine MA Nembhard IM Edmondson AC . Measuring Teamwork in Health Care Settings: A Review of Survey Instruments . Harvard Business School Working Paper #11-116. 2011.","DOI":"10.2139\/ssrn.1838538"},{"key":"2025090410375777000_23.8.639.16","unstructured":"Weick KE Sutcliffe KM . Managing the unexpected: assuring high performance in an age of complexity. San Francisco, CA: Jossey-Bass, 2001."},{"key":"2025090410375777000_23.8.639.17","doi-asserted-by":"publisher","DOI":"10.1016\/S1479-666X(06)80084-3"},{"key":"2025090410375777000_23.8.639.18","doi-asserted-by":"publisher","DOI":"10.1136\/bmjqs.2010.041913"},{"key":"2025090410375777000_23.8.639.19","doi-asserted-by":"publisher","DOI":"10.1136\/qshc.2007.024760"},{"key":"2025090410375777000_23.8.639.20","doi-asserted-by":"crossref","unstructured":"Mazzocco K Petitti DB Fong KT . Surgical team behaviors and patient outcomes. 2009;197:678\u201385.","DOI":"10.1016\/j.amjsurg.2008.03.002"},{"key":"2025090410375777000_23.8.639.21","doi-asserted-by":"publisher","DOI":"10.1016\/j.jamcollsurg.2006.10.018"},{"key":"2025090410375777000_23.8.639.22","doi-asserted-by":"publisher","DOI":"10.1136\/qshc.2004.009936"},{"key":"2025090410375777000_23.8.639.23","doi-asserted-by":"crossref","unstructured":"Fleiss JL Levin B Paik MC . Statistical methods for rates and proportions. 3rd edn. New York, NY: John Wiley & sons, 2003.","DOI":"10.1002\/0471445428"},{"key":"2025090410375777000_23.8.639.24","doi-asserted-by":"publisher","DOI":"10.1007\/BF02294066"},{"key":"2025090410375777000_23.8.639.25","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90158-L"},{"key":"2025090410375777000_23.8.639.26","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"2025090410375777000_23.8.639.27","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2010.03.002"},{"key":"2025090410375777000_23.8.639.28","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.0b013e318279560b"},{"key":"2025090410375777000_23.8.639.29","doi-asserted-by":"publisher","DOI":"10.1111\/j.1445-2197.2008.04833.x"},{"key":"2025090410375777000_23.8.639.30","doi-asserted-by":"publisher","DOI":"10.1007\/s00268-007-9320-z"},{"key":"2025090410375777000_23.8.639.31","doi-asserted-by":"publisher","DOI":"10.1093\/bja\/aeg112"},{"key":"2025090410375777000_23.8.639.32","doi-asserted-by":"crossref","unstructured":"Johnston FM Tergas AI Bennett JL . Measuring briefing and checklist compliance in surgery: a tool for quality improvement. Am J Med Qual 2013. [Epub ahead of print]. doi:10.1177\/1062860613509402","DOI":"10.1177\/1062860613509402"},{"key":"2025090410375777000_23.8.639.33","doi-asserted-by":"publisher","DOI":"10.1136\/qshc.2005.015388"},{"key":"2025090410375777000_23.8.639.34","unstructured":"Baker DP Dismukes RK . A gold standards approach to training instructors to evaluate crew performance. NASA Technical Memorandum . 2003."},{"key":"2025090410375777000_23.8.639.35","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.0b013e31824a9a02"}],"container-title":["BMJ Quality &amp; Safety"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjqs-2013-002446","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,4]],"date-time":"2025-09-04T17:38:07Z","timestamp":1757007487000},"score":1,"resource":{"primary":{"URL":"https:\/\/qualitysafety.bmj.com\/lookup\/doi\/10.1136\/bmjqs-2013-002446"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,2,4]]},"references-count":35,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2014,7,14]]},"published-print":{"date-parts":[[2014,8]]}},"alternative-id":["10.1136\/bmjqs-2013-002446"],"URL":"https:\/\/doi.org\/10.1136\/bmjqs-2013-002446","relation":{},"ISSN":["2044-5415","2044-5423"],"issn-type":[{"type":"print","value":"2044-5415"},{"type":"electronic","value":"2044-5423"}],"subject":[],"published":{"date-parts":[[2014,2,4]]}}}