{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T17:31:51Z","timestamp":1792603911119,"version":"4.5.0"},"reference-count":20,"publisher":"Wiley","issue":"7-8","license":[{"start":{"date-parts":[[2012,9,3]],"date-time":"2012-09-03T00:00:00Z","timestamp":1346630400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["ANZ Journal of Surgery"],"published-print":{"date-parts":[[2014,7]]},"abstract":"Abstract<\/jats:title>Background<\/jats:title>The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA<\/jats:styled-content>, UK<\/jats:styled-content>, A<\/jats:styled-content>ustralia and N<\/jats:styled-content>ew Z<\/jats:styled-content>ealand.<\/jats:p><\/jats:sec>Methods<\/jats:title>We use data from 19 applicants for higher surgical training in 2008 at the R<\/jats:styled-content>oyal C<\/jats:styled-content>ollege of S<\/jats:styled-content>urgeons in I<\/jats:styled-content>reland to compare: (i) the objective structured assessment of technical skills (OSATS<\/jats:styled-content>) method; and (ii) a procedure\u2010specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons.<\/jats:p><\/jats:sec>Results<\/jats:title>The overall interrater reliability (IRR<\/jats:styled-content>) of the OSATS <\/jats:styled-content>assessment as determined by a correlation coefficient was 0.507 (P<\/jats:styled-content><\/jats:italic> < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR<\/jats:styled-content>. The checklist's overall IRR<\/jats:styled-content> was 0.89. Individually, only five (26%) of the OSATS<\/jats:styled-content> assessments reached the 0.8 level of IRR<\/jats:styled-content> in contrast to 18 (95%) of the checklist assessments.<\/jats:p><\/jats:sec>Discussion<\/jats:title>We propose binary procedure\u2010based assessment checklists as more reliable assessment instruments with more robust reproducibility.<\/jats:p><\/jats:sec>","DOI":"10.1111\/j.1445-2197.2012.06236.x","type":"journal-article","created":{"date-parts":[[2012,9,4]],"date-time":"2012-09-04T00:11:49Z","timestamp":1346717509000},"page":"568-573","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":45,"title":["Objective structured assessment of technical skills and checklist scales reliability compared for high stakes assessments"],"prefix":"10.1111","volume":"84","author":[{"given":"Anthony G.","family":"Gallagher","sequence":"first","affiliation":[{"name":"School of Medicine University College Cork Cork Ireland"},{"name":"National Surgical Training Centre Royal College of Surgeons in Ireland Dublin Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gerald C.","family":"O'Sullivan","sequence":"additional","affiliation":[{"name":"Cork Cancer Research Centre Biosciences Institute, University College Cork Cork Ireland"},{"name":"National Surgical Training Centre Royal College of Surgeons in Ireland Dublin Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gerald","family":"Leonard","sequence":"additional","affiliation":[{"name":"Otolaryngology \u2013 Head and Neck Surgery University of Connecticut Health Center Farmington Connecticut USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brendan P.","family":"Bunting","sequence":"additional","affiliation":[{"name":"School of Psychology University of Ulster Derry Northern Ireland UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kieran J.","family":"McGlade","sequence":"additional","affiliation":[{"name":"School of Medicine, Dentistry and Biomedical Sciences Queen's University Belfast Northern Ireland UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2012,9,3]]},"reference":[{"key":"e_1_2_6_2_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1365-2168.1997.02502.x"},{"key":"e_1_2_6_3_1","unstructured":"American Educational Research Association American Psychological Association & National Council on Measurement in Education.Standards for educational and psychological tests 1999."},{"key":"e_1_2_6_4_1","volume-title":"Individual Differences","author":"Cooper C","year":"2002"},{"key":"e_1_2_6_5_1","first-page":"137","article-title":"A 2nd\u2010order factor analysis of the reliability and validity of the 11 plus examination in Northern Ireland","volume":"18","author":"Bunting B","year":"1987","journal-title":"Econ. Soc. Rev."},{"key":"e_1_2_6_6_1","unstructured":"Oxford English Dictionary. Oxford: Oxford University Press 2004."},{"key":"e_1_2_6_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00464-003-0035-4"},{"key":"e_1_2_6_8_1","doi-asserted-by":"publisher","DOI":"10.1097\/PRS.0b013e31816b19bc"},{"key":"e_1_2_6_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0002-9610(98)00327-4"},{"key":"e_1_2_6_10_1","doi-asserted-by":"publisher","DOI":"10.1097\/01.sla.0000234808.85789.6a"},{"key":"e_1_2_6_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejcts.2005.03.012"},{"key":"e_1_2_6_12_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139168342"},{"key":"e_1_2_6_13_1","volume-title":"Behavior Modification in Applied Settings","author":"Kazdin AE","year":"1994"},{"key":"e_1_2_6_14_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1445-2197.2008.04439.x"},{"key":"e_1_2_6_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjps.2008.06.054"},{"key":"e_1_2_6_16_1","volume-title":"A Competency Based Curriculum for Specialist Training in Trauma and Orthopaedics","author":"Pitts D","year":"2006"},{"key":"e_1_2_6_17_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.b1802"},{"key":"e_1_2_6_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2006.06.050"},{"key":"e_1_2_6_19_1","doi-asserted-by":"publisher","DOI":"10.1097\/00000658-200210000-00008"},{"key":"e_1_2_6_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jamcollsurg.2008.05.007"},{"key":"e_1_2_6_21_1","doi-asserted-by":"publisher","DOI":"10.1001\/archotol.130.10.1145"}],"container-title":["ANZ Journal of Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1111%2Fj.1445-2197.2012.06236.x","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/j.1445-2197.2012.06236.x","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,5]],"date-time":"2023-10-05T12:20:39Z","timestamp":1696508439000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/j.1445-2197.2012.06236.x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,9,3]]},"references-count":20,"journal-issue":{"issue":"7-8","published-print":{"date-parts":[[2014,7]]}},"alternative-id":["10.1111\/j.1445-2197.2012.06236.x"],"URL":"https:\/\/doi.org\/10.1111\/j.1445-2197.2012.06236.x","archive":["Portico"],"relation":{},"ISSN":["1445-1433","1445-2197"],"issn-type":[{"value":"1445-1433","type":"print"},{"value":"1445-2197","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,9,3]]},"assertion":[{"value":"2012-04-27","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-09-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}