{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T11:42:52Z","timestamp":1793014972579,"version":"4.5.0"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T00:00:00Z","timestamp":1646092800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T00:00:00Z","timestamp":1646092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Adv in Health Sci Educ"],"published-print":{"date-parts":[[2022,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Variation in examiner stringency is a recognised problem in many standardised summative assessments of performance such as the OSCE. The stated strength of the OSCE is that such error might largely balance out over the exam as a whole. This study uses linear mixed models to estimate the impact of different factors (examiner, station, candidate and exam) on station-level total domain score and, separately, on a single global grade. The exam data is from 442 separate administrations of an 18 station OSCE for international medical graduates who want to work in the National Health Service in the UK. We find that variation due to examiner is approximately twice as large for domain scores as it is for grades (16% vs. 8%), with smaller residual variance in the former (67% vs. 76%). Combined estimates of exam-level (relative) reliability across all data are 0.75 and 0.69 for domains scores and grades respectively. The correlation between two separate estimates of stringency for individual examiners (one for grades and one for domain scores) is relatively high (r=0.76) implying that examiners are generally quite consistent in their stringency between these two assessments of performance. Cluster analysis indicates that examiners fall into two broad groups characterised as hawks or doves on both measures. At the exam level, correcting for examiner stringency produces systematically lower cut-scores under borderline regression standard setting than using the raw marks. In turn, such a correction would produce higher pass rates\u2014although meaningful direct comparisons are challenging to make. As in other studies, this work shows that OSCEs and other standardised performance assessments are subject to substantial variation in examiner stringency, and require sufficient domain sampling to ensure quality of pass\/fail decision-making is at least adequate. More, perhaps qualitative, work is needed to understand better how examiners might score similarly (or differently) between the awarding of station-level domain scores and global grades. The issue of the potential systematic bias of borderline regression evidenced for the first time here, with sources of error producing cut-scores higher than they should be, also needs more investigation.<\/jats:p>","DOI":"10.1007\/s10459-022-10096-9","type":"journal-article","created":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T14:02:38Z","timestamp":1646143358000},"page":"457-473","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Pass\/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomes"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1161-5938","authenticated-orcid":false,"given":"Matt","family":"Homer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,3,1]]},"reference":[{"key":"10096_CR1","unstructured":"Bacher, J., Wenzig, K., & Vogler, M. (2004). SPSS TwoStep Cluster\u2014a first evaluation (Vol. 2004\u20132). N\u00fcrnberg: Universit\u00e4t Erlangen-N\u00fcrnberg, Wirtschafts- und Sozialwissenschaftliche Fakult\u00e4t, Sozialwissenschaftliches Institut Lehrstuhl f\u00fcr Soziologie"},{"issue":"1","key":"10096_CR2","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1111\/j.1743-498X.2012.00607.x","volume":"10","author":"I Bartman","year":"2013","unstructured":"Bartman, I., Smee, S., & Roy, M. (2013). A method for identifying extreme OSCE examiners. The Clinical Teacher, 10(1), 27\u201331. https:\/\/doi.org\/10.1111\/j.1743-498X.2012.00607.x","journal-title":"The Clinical Teacher"},{"issue":"1","key":"10096_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v067.i01","volume":"67","author":"D Bates","year":"2015","unstructured":"Bates, D., M\u00e4chler, M., Bolker, B., & Walker, S. (2015). Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1), 1\u201348. https:\/\/doi.org\/10.18637\/jss.v067.i01","journal-title":"Journal of Statistical Software"},{"issue":"2","key":"10096_CR4","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1080\/01421590078526","volume":"22","author":"MF Ben-David","year":"2000","unstructured":"Ben-David, M. F. (2000). AMEE Guide No. 18: Standard setting in student assessment. Medical Teacher, 22(2), 120\u2013130. https:\/\/doi.org\/10.1080\/01421590078526","journal-title":"Medical Teacher"},{"issue":"11","key":"10096_CR5","doi-asserted-by":"publisher","first-page":"960","DOI":"10.3109\/0142159X.2012.703791","volume":"34","author":"R Bloch","year":"2012","unstructured":"Bloch, R., & Norman, G. (2012). Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68. Medical Teacher, 34(11), 960\u2013992. https:\/\/doi.org\/10.3109\/0142159X.2012.703791","journal-title":"Medical Teacher"},{"key":"10096_CR6","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3456-0","volume-title":"Generalizability Theory","author":"RL Brennan","year":"2001","unstructured":"Brennan, R. L. (2001). Generalizability Theory (1st ed.). New York: Springer","edition":"1"},{"key":"10096_CR7","unstructured":"General Medical Council (2020a). What is the PLAB 2 exam? What is the PLAB 2 exam? https:\/\/www.gmc-uk.org\/registration-and-licensing\/join-the-register\/plab\/plab-2-guide\/what-is-the-plab-2-exam. Accessed 7 May 2020"},{"key":"10096_CR8","unstructured":"General Medical Council (2020b). PLAB (Professional and Linguistic Assessments Board). Professional and Linguistic Assessments Board. https:\/\/www.gmc-uk.org\/registration-and-licensing\/join-the-register\/plab. Accessed 7 May 2020"},{"key":"10096_CR9","doi-asserted-by":"publisher","unstructured":"Harasym, P., Woloschuk, W., & Cunning, L. (2008). Undesired variance due to examiner stringency\/leniency effect in communication skill scores assessed in OSCEs. Advances in health sciences education: theory and practice. https:\/\/doi.org\/10.1007\/s10459-007-9068-0","DOI":"10.1007\/s10459-007-9068-0"},{"key":"10096_CR10","unstructured":"Harden, R., Lilley, P., & Patricio, M. (2015). The Definitive Guide to the OSCE: The Objective Structured Clinical Examination as a performance assessment., 1e (1 edition.). Edinburgh; New York: Churchill Livingstone"},{"issue":"8","key":"10096_CR11","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1111\/j.1365-2923.2008.03103.x","volume":"42","author":"R Hays","year":"2008","unstructured":"Hays, R., Gupta, T. S., & Veitch, J. (2008). The practical value of the standard error of measurement in borderline pass\/fail decisions. Medical Education, 42(8), 810\u2013815. https:\/\/doi.org\/10.1111\/j.1365-2923.2008.03103.x","journal-title":"Medical Education"},{"key":"10096_CR12","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-020-09990-x","author":"M Homer","year":"2020","unstructured":"Homer, M. (2020). Re-conceptualising and accounting for examiner (cut-score) stringency in a \u2018high frequency, small cohort\u2019 performance test. Advances in Health Sciences Education. https:\/\/doi.org\/10.1007\/s10459-020-09990-x","journal-title":"Advances in Health Sciences Education"},{"issue":"0","key":"10096_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/0142159X.2019.1681388","volume":"0","author":"M Homer","year":"2019","unstructured":"Homer, M., Fuller, R., Hallam, J., & Pell, G. (2019). Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can \u2018work.\u2019. Medical Teacher, 0(0), 1\u201310. https:\/\/doi.org\/10.1080\/0142159X.2019.1681388","journal-title":"Medical Teacher"},{"key":"10096_CR14","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2020.1781072","author":"M Homer","year":"2020","unstructured":"Homer, M., Fuller, R., Hallam, J., & Pell, G. (2020). Shining a spotlight on scoring in the OSCE: checklists and item weighting. Medical Teacher. https:\/\/doi.org\/10.1080\/0142159X.2020.1781072","journal-title":"Medical Teacher"},{"issue":"0","key":"10096_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/0142159X.2020.1856353","volume":"0","author":"M Homer","year":"2020","unstructured":"Homer, M., & Russell, J. (2020). Conjunctive standards in OSCEs: The why and the how of number of stations passed criteria. Medical Teacher, 0(0), 1\u201314. https:\/\/doi.org\/10.1080\/0142159X.2020.1856353","journal-title":"Medical Teacher"},{"issue":"2","key":"10096_CR16","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1111\/medu.12621","volume":"49","author":"JS Ilgen","year":"2015","unstructured":"Ilgen, J. S., Ma, I. W. Y., Hatala, R., & Cook, D. A. (2015). A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Medical Education, 49(2), 161\u2013173. https:\/\/doi.org\/10.1111\/medu.12621","journal-title":"Medical Education"},{"issue":"3","key":"10096_CR17","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1027\/1614-2241\/a000149","volume":"14","author":"Z Jiang","year":"2018","unstructured":"Jiang, Z. (2018). Using the Linear Mixed-Effect Model Framework to Estimate Generalizability Variance Components in R. Methodology, 14(3), 133\u2013142. https:\/\/doi.org\/10.1027\/1614-2241\/a000149","journal-title":"Methodology"},{"key":"10096_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2021.01.018","author":"K Jogerst","year":"2021","unstructured":"Jogerst, K., Eurboonyanun, C., Park, Y. S., Cassidy, D., McKinley, S., Hamdi, I., et al. (2021). Implementation of the ACS\/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory. Am J Surg. https:\/\/doi.org\/10.1016\/j.amjsurg.2021.01.018","journal-title":"Am J Surg"},{"issue":"9","key":"10096_CR19","doi-asserted-by":"publisher","first-page":"e1447","DOI":"10.3109\/0142159X.2013.818635","volume":"35","author":"KZ Khan","year":"2013","unstructured":"Khan, K. Z., Gaunt, K., Ramachandran, S., & Pushkar, P. (2013). The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration. Medical Teacher, 35(9), e1447\u20131463. https:\/\/doi.org\/10.3109\/0142159X.2013.818635","journal-title":"Medical Teacher"},{"issue":"2","key":"10096_CR20","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1046\/j.1365-2923.2003.01429.x","volume":"37","author":"A Kramer","year":"2003","unstructured":"Kramer, A., Muijtjens, A., Jansen, K., D\u00fcsman, H., Tan, L., & van der Vleuten, C. (2003). Comparison of a rational and an empirical standard setting procedure for an OSCE. Objective structured clinical examinations. Medical Education, 37(2), 132\u2013139","journal-title":"Medical Education"},{"issue":"2","key":"10096_CR21","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1177\/1094428105284919","volume":"9","author":"CE Lance","year":"2006","unstructured":"Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say? Organizational Research Methods, 9(2), 202\u2013220. https:\/\/doi.org\/10.1177\/1094428105284919","journal-title":"Organizational Research Methods"},{"issue":"3","key":"10096_CR22","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1111\/medu.14357","volume":"55","author":"BS Malau-Aduli","year":"2021","unstructured":"Malau-Aduli, B. S., Hays, R. B., D\u2019Souza, K., Smith, A. M., Jones, K., Turner, R., et al. (2021). Examiners\u2019 decision-making processes in observation-based clinical examinations. Medical Education, 55(3), 344\u2013353. https:\/\/doi.org\/10.1111\/medu.14357","journal-title":"Medical Education"},{"issue":"2","key":"10096_CR23","doi-asserted-by":"publisher","first-page":"97","DOI":"10.3109\/0142159X.2013.853119","volume":"36","author":"DW McKinley","year":"2014","unstructured":"McKinley, D. W., & Norcini, J. J. (2014). How to set standards on performance-based examinations: AMEE Guide No. 85. Medical Teacher, 36(2), 97\u2013110. https:\/\/doi.org\/10.3109\/0142159X.2013.853119","journal-title":"Medical Teacher"},{"issue":"1","key":"10096_CR24","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1186\/1472-6920-6-42","volume":"6","author":"I McManus","year":"2006","unstructured":"McManus, I., Thompson, M., & Mollon, J. (2006). Assessment of examiner leniency and stringency (\u2018hawk-dove effect\u2019) in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Medical Education, 6(1), 42. https:\/\/doi.org\/10.1186\/1472-6920-6-42","journal-title":"BMC Medical Education"},{"issue":"11","key":"10096_CR25","doi-asserted-by":"publisher","first-page":"2074","DOI":"10.1002\/sim.8086","volume":"38","author":"TP Morris","year":"2019","unstructured":"Morris, T. P., White, I. R., & Crowther, M. J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11), 2074\u20132102. https:\/\/doi.org\/10.1002\/sim.8086","journal-title":"Statistics in Medicine"},{"issue":"7","key":"10096_CR26","doi-asserted-by":"publisher","first-page":"618","DOI":"10.1111\/j.1365-2929.2006.02511.x","volume":"40","author":"G Norman","year":"2006","unstructured":"Norman, G., Bordage, G., Page, G., & Keane, D. (2006). How specific is case specificity? Medical Education, 40(7), 618\u2013623. https:\/\/doi.org\/10.1111\/j.1365-2929.2006.02511.x","journal-title":"Medical Education"},{"key":"10096_CR27","unstructured":"Norusis, M. (2011). IBM SPSS Statistics 19 Guide to Data Analysis (1st ed.). Boston, MA: Addison Wesley; 1st edition. https:\/\/www.goodreads.com\/work\/best_book\/18037003-ibm-spss-statistics-19-guide-to-data-analysis. Accessed 28 April 2021"},{"key":"10096_CR28","doi-asserted-by":"crossref","unstructured":"Park, Y. S. (2019). Chapter 3: Reliability. In R. Yudkowsky (Ed.), Assessment in Health Professions Education (2nd edition., pp.\u00a033\u201350). New York: Routledge","DOI":"10.4324\/9781315166902-3"},{"issue":"10","key":"10096_CR29","doi-asserted-by":"publisher","first-page":"802","DOI":"10.3109\/0142159X.2010.507716","volume":"32","author":"G Pell","year":"2010","unstructured":"Pell, G., Fuller, R., Homer, M., & Roberts, T. (2010). How to measure the quality of the OSCE: A review of metrics - AMEE guide no. 49. Medical Teacher, 32(10), 802\u2013811. https:\/\/doi.org\/10.3109\/0142159X.2010.507716","journal-title":"Medical Teacher"},{"issue":"0","key":"10096_CR30","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/0142159X.2021.1948519","volume":"0","author":"SA Santen","year":"2021","unstructured":"Santen, S. A., Ryan, M., Helou, M. A., Richards, A., Perera, R. A., Haley, K., et al. (2021). Building reliable and generalizable clerkship competency assessments: Impact of \u2018hawk-dove\u2019 correction. Medical Teacher, 0(0), 1\u20137. https:\/\/doi.org\/10.1080\/0142159X.2021.1948519","journal-title":"Medical Teacher"},{"issue":"2","key":"10096_CR31","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1007\/s10459-012-9370-3","volume":"18","author":"W Tavares","year":"2013","unstructured":"Tavares, W., & Eva, K. W. (2013). Exploring the Impact of Mental Workload on Rater-Based Assessments. Advances in Health Sciences Education, 18(2), 291\u2013303. https:\/\/doi.org\/10.1007\/s10459-012-9370-3","journal-title":"Advances in Health Sciences Education"},{"issue":"1","key":"10096_CR32","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1111\/test.12087","volume":"38","author":"D Trafimow","year":"2016","unstructured":"Trafimow, D. (2016). The attenuation of correlation coefficients: a statistical literacy issue. Teaching Statistics, 38(1), 25\u201328. https:\/\/doi.org\/10.1111\/test.12087","journal-title":"Teaching Statistics"},{"issue":"0","key":"10096_CR33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/0142159X.2019.1652260","volume":"0","author":"TJ Wood","year":"2019","unstructured":"Wood, T. J., & Pugh, D. (2019). Are rating scales really better than checklists for measuring increasing levels of expertise? Medical Teacher, 0(0), 1\u20136. https:\/\/doi.org\/10.1080\/0142159X.2019.1652260","journal-title":"Medical Teacher"},{"key":"10096_CR34","doi-asserted-by":"publisher","unstructured":"Yeates, P., Cope, N., Hawarden, A., Bradshaw, H., McCray, G., & Homer, M. (2018). Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs. Medical Education, 0(0), https:\/\/doi.org\/10.1111\/medu.13783","DOI":"10.1111\/medu.13783"},{"key":"10096_CR35","doi-asserted-by":"publisher","unstructured":"Yeates, P., Moult, A., Cope, N., McCray, G., Xilas, E., Lovelock, T., et al. (2021). Measuring the Effect of Examiner Variability in a Multiple-Circuit Objective Structured Clinical Examination (OSCE). Academic Medicine, Publish Ahead of Print. https:\/\/doi.org\/10.1097\/ACM.0000000000004028","DOI":"10.1097\/ACM.0000000000004028"},{"issue":"1","key":"10096_CR36","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1080\/0142159X.2017.1248916","volume":"39","author":"P Yeates","year":"2017","unstructured":"Yeates, P., & Sebok-Syer, S. S. (2017). Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students\u2019 scores using Many Facet Rasch Modeling. Medical Teacher, 39(1), 92\u201399. https:\/\/doi.org\/10.1080\/0142159X.2017.1248916","journal-title":"Medical Teacher"}],"container-title":["Advances in Health Sciences Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10459-022-10096-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10459-022-10096-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10459-022-10096-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,18]],"date-time":"2022-05-18T10:29:24Z","timestamp":1652869764000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10459-022-10096-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,1]]},"references-count":36,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,5]]}},"alternative-id":["10096"],"URL":"https:\/\/doi.org\/10.1007\/s10459-022-10096-9","relation":{},"ISSN":["1382-4996","1573-1677"],"issn-type":[{"value":"1382-4996","type":"print"},{"value":"1573-1677","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,1]]},"assertion":[{"value":"20 July 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 January 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 March 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"When taking PLAB examinations, candidates consent to their examination data being used for research purposes. The General Medical Council in the UK gave permission for the analysis of the anonymized data used in this paper. This research was not externally funded.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics and funding"}}]}}