{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T09:33:40Z","timestamp":1793612020039,"version":"4.5.0"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,10,16]],"date-time":"2023-10-16T00:00:00Z","timestamp":1697414400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,16]],"date-time":"2023-10-16T00:00:00Z","timestamp":1697414400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Adv in Health Sci Educ"],"published-print":{"date-parts":[[2024,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Quantitative measures of systematic differences in OSCE scoring across examiners (often termed examiner stringency) can threaten the validity of examination outcomes. Such effects are usually conceptualised and operationalised based solely on checklist\/domain scores in a station, and global grades are not often used in this type of analysis. In this work, a large candidate-level exam dataset is analysed to develop a more sophisticated understanding of examiner stringency. Station scores are modelled based on global grades\u2014with each candidate, station and examiner allowed to vary in their ability\/stringency\/difficulty in the modelling. In addition, examiners are also allowed to vary in how they discriminate across grades\u2014to our knowledge, this is the first time this has been investigated. Results show that examiners contribute strongly to variance in scoring in two distinct ways\u2014via the traditional conception of score stringency (34% of score variance), but also in how they discriminate in scoring across grades (7%). As one might expect, candidate and station account only for a small amount of score variance at the station-level once candidate grades are accounted for (3% and 2% respectively) with the remainder being residual (54%). Investigation of impacts on station-level candidate pass\/fail decisions suggest that examiner differential stringency effects combine to give false positive (candidates passing in error) and false negative (failing in error) rates in stations of around 5% each but at the exam-level this reduces to 0.4% and 3.3% respectively. This work adds to our understanding of examiner behaviour by demonstrating that examiners can vary in qualitatively different ways in their judgments. For institutions, it emphasises the key message that it is important to sample widely from the examiner pool via sufficient stations to ensure OSCE-level decisions are sufficiently defensible. It also suggests that examiner training should include discussion of global grading, and the combined effect of scoring and grading on candidate outcomes.<\/jats:p>","DOI":"10.1007\/s10459-023-10289-w","type":"journal-article","created":{"date-parts":[[2023,10,16]],"date-time":"2023-10-16T08:01:54Z","timestamp":1697443314000},"page":"919-934","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Towards a more nuanced conceptualisation of differential examiner stringency in OSCEs"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1161-5938","authenticated-orcid":false,"given":"Matt","family":"Homer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,10,16]]},"reference":[{"issue":"1","key":"10289_CR1","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1111\/j.1743-498X.2012.00607.x","volume":"10","author":"I Bartman","year":"2013","unstructured":"Bartman, I., Smee, S., & Roy, M. (2013). A method for identifying extreme OSCE examiners. The Clinical Teacher, 10(1), 27\u201331. https:\/\/doi.org\/10.1111\/j.1743-498X.2012.00607.x","journal-title":"The Clinical Teacher"},{"issue":"1","key":"10289_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v067.i01","volume":"67","author":"D Bates","year":"2015","unstructured":"Bates, D., M\u00e4chler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1\u201348. https:\/\/doi.org\/10.18637\/jss.v067.i01","journal-title":"Journal of Statistical Software"},{"issue":"2","key":"10289_CR3","doi-asserted-by":"publisher","first-page":"1051","DOI":"10.1007\/s11135-018-0802-x","volume":"53","author":"A Bell","year":"2019","unstructured":"Bell, A., Fairbrother, M., & Jones, K. (2019). Fixed and random effects models: Making an informed choice. Quality & Quantity, 53(2), 1051\u20131074. https:\/\/doi.org\/10.1007\/s11135-018-0802-x","journal-title":"Quality & Quantity"},{"key":"10289_CR4","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"1988","unstructured":"Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Routledge."},{"issue":"6","key":"10289_CR5","doi-asserted-by":"publisher","first-page":"560","DOI":"10.1111\/medu.12678","volume":"49","author":"DA Cook","year":"2015","unstructured":"Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane\u2019s framework. Medical Education, 49(6), 560\u2013575. https:\/\/doi.org\/10.1111\/medu.12678","journal-title":"Medical Education"},{"key":"10289_CR6","unstructured":"Crowson, M. (2020). Multilevel modeling in R using lme4 package (video). Youtube. https:\/\/www.youtube.com\/watch?v=8r9bUKUVecc"},{"key":"10289_CR7","unstructured":"General Medical Council. (2020a). What is the PLAB 2 exam? What is the PLAB 2 exam? Retrieved May 7, 2020, fromhttps:\/\/www.gmc-uk.org\/registration-and-licensing\/join-the-register\/plab\/plab-2-guide\/what-is-the-plab-2-exam"},{"key":"10289_CR8","unstructured":"General Medical Council. (2020b). PLAB (Professional and Linguistic Assessments Board). Professional and Linguistic Assessments Board. Retrieved May 7, 2020, fromhttps:\/\/www.gmc-uk.org\/registration-and-licensing\/join-the-register\/plab"},{"key":"10289_CR9","unstructured":"General Medical Council. (2022). PLAB reports. Retrieved February 28, 2023, fromhttps:\/\/www.gmc-uk.org\/registration-and-licensing\/join-the-register\/plab\/plab-reports"},{"key":"10289_CR10","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-007-9068-0","author":"P Harasym","year":"2008","unstructured":"Harasym, P., Woloschuk, W., & Cunning, L. (2008). Undesired variance due to examiner stringency\/leniency effect in communication skill scores assessed in OSCEs. Advances in Health Sciences Education: Theory and Practice. https:\/\/doi.org\/10.1007\/s10459-007-9068-0","journal-title":"Advances in Health Sciences Education: Theory and Practice"},{"key":"10289_CR11","doi-asserted-by":"publisher","unstructured":"Hatala, R., Cook, D. A., Brydges, R., & Hawkins, R. (2015). Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): A systematic review of validity evidence. Advances in Health Sciences Education: Theory and Practice. https:\/\/doi.org\/10.1007\/s10459-015-9593-1","DOI":"10.1007\/s10459-015-9593-1"},{"issue":"8","key":"10289_CR12","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1111\/j.1365-2923.2008.03103.x","volume":"42","author":"R Hays","year":"2008","unstructured":"Hays, R., Gupta, T. S., & Veitch, J. (2008). The practical value of the standard error of measurement in borderline pass\/fail decisions. Medical Education, 42(8), 810\u2013815. https:\/\/doi.org\/10.1111\/j.1365-2923.2008.03103.x","journal-title":"Medical Education"},{"issue":"7","key":"10289_CR13","doi-asserted-by":"publisher","first-page":"564","DOI":"10.3109\/0142159X.2013.789134","volume":"35","author":"B Hodges","year":"2013","unstructured":"Hodges, B. (2013). Assessment in the post-psychometric era: Learning to love the subjective and collective. Medical Teacher, 35(7), 564\u2013568. https:\/\/doi.org\/10.3109\/0142159X.2013.789134","journal-title":"Medical Teacher"},{"key":"10289_CR14","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-020-09990-x","author":"M Homer","year":"2020","unstructured":"Homer, M. (2020). Re-conceptualising and accounting for examiner (cut-score) stringency in a \u2018high frequency, small cohort\u2019 performance test. Advances in Health Sciences Education. https:\/\/doi.org\/10.1007\/s10459-020-09990-x","journal-title":"Advances in Health Sciences Education"},{"key":"10289_CR15","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-022-10096-9","author":"M Homer","year":"2022","unstructured":"Homer, M. (2022). Pass\/fail decisions and standards: The impact of differential examiner stringency on OSCE outcomes. Advances in Health Sciences Education. https:\/\/doi.org\/10.1007\/s10459-022-10096-9","journal-title":"Advances in Health Sciences Education"},{"key":"10289_CR16","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2023.2197138","author":"Matt Homer","year":"2023","unstructured":"Homer, M. (2023). Setting defensible minimum-stations-passed standards in OSCE-type assessments. Medical Teacher. https:\/\/doi.org\/10.1080\/0142159X.2023.2197138","journal-title":"Medical Teacher"},{"key":"10289_CR17","unstructured":"IBM Corp. (2021). IBM SPSS Statistics for Windows, Version 28.0. IBM Corp."},{"issue":"2","key":"10289_CR18","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1111\/medu.12621","volume":"49","author":"JS Ilgen","year":"2015","unstructured":"Ilgen, J. S., Ma, I. W. Y., Hatala, R., & Cook, D. A. (2015). A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Medical Education, 49(2), 161\u2013173. https:\/\/doi.org\/10.1111\/medu.12621","journal-title":"Medical Education"},{"key":"10289_CR19","doi-asserted-by":"publisher","unstructured":"Khan, K. Z., Gaunt, K., Ramachandran, S., & Pushkar, P. (2013). The Objective Structured Clinical Examination (OSCE): AMEE Guide No. 81. Part II: organisation & administration. Medical Teacher, 35(9), e1447\u2013e1463. https:\/\/doi.org\/10.3109\/0142159X.2013.818635","DOI":"10.3109\/0142159X.2013.818635"},{"issue":"2","key":"10289_CR20","first-page":"132","volume":"37","author":"A Kramer","year":"2003","unstructured":"Kramer, A., Muijtjens, A., Jansen, K., D\u00fcsman, H., Tan, L., & van der Vleuten, C. (2003). Comparison of a rational and an empirical standard setting procedure for an OSCE. Objective Structured Clinical Examinations. Medical Education, 37(2), 132\u2013139.","journal-title":"Objective Structured Clinical Examinations. Medical Education"},{"issue":"2","key":"10289_CR21","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1111\/j.1745-3984.1995.tb00462.x","volume":"32","author":"SA Livingston","year":"1995","unstructured":"Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32(2), 179\u2013197. https:\/\/doi.org\/10.1111\/j.1745-3984.1995.tb00462.x","journal-title":"Journal of Educational Measurement"},{"key":"10289_CR22","doi-asserted-by":"publisher","unstructured":"McKinley, D. W., & Norcini, J. J. (2014). How to set standards on performance-based examinations: AMEE Guide No. 85. Medical Teacher, 36(2), 97\u2013110. https:\/\/doi.org\/10.3109\/0142159X.2013.853119","DOI":"10.3109\/0142159X.2013.853119"},{"issue":"1","key":"10289_CR23","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1186\/1472-6920-6-42","volume":"6","author":"I McManus","year":"2006","unstructured":"McManus, I., Thompson, M., & Mollon, J. (2006). Assessment of examiner leniency and stringency ('hawk-dove effect\u2019) in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Medical Education, 6(1), 42. https:\/\/doi.org\/10.1186\/1472-6920-6-42","journal-title":"BMC Medical Education"},{"key":"10289_CR24","volume-title":"Introduction to linear regression analysis","author":"DC Montgomery","year":"2012","unstructured":"Montgomery, D. C., Peck, E. A., & Vining, G. G. (2012). Introduction to linear regression analysis (5th ed.). Wiley-Blackwell.","edition":"5"},{"issue":"11","key":"10289_CR25","doi-asserted-by":"publisher","first-page":"2074","DOI":"10.1002\/sim.8086","volume":"38","author":"TP Morris","year":"2019","unstructured":"Morris, T. P., White, I. R., & Crowther, M. J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11), 2074\u20132102. https:\/\/doi.org\/10.1002\/sim.8086","journal-title":"Statistics in Medicine"},{"key":"10289_CR26","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2012.00322","author":"K Nimon","year":"2012","unstructured":"Nimon, K. (2012). Statistical assumptions of substantive analyses across the general linear model: A mini-review. Frontiers in Psychology. https:\/\/doi.org\/10.3389\/fpsyg.2012.00322","journal-title":"Frontiers in Psychology"},{"issue":"7","key":"10289_CR27","doi-asserted-by":"publisher","first-page":"618","DOI":"10.1111\/j.1365-2929.2006.02511.x","volume":"40","author":"G Norman","year":"2006","unstructured":"Norman, G., Bordage, G., Page, G., & Keane, D. (2006). How specific is case specificity? Medical Education, 40(7), 618\u2013623. https:\/\/doi.org\/10.1111\/j.1365-2929.2006.02511.x","journal-title":"Medical Education"},{"key":"10289_CR28","doi-asserted-by":"crossref","unstructured":"Osterlind, S. J., & Everson, H. T. (2009). Differential item functioning. SAGE.","DOI":"10.4135\/9781412993913"},{"issue":"2","key":"10289_CR29","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1111\/medu.14039","volume":"54","author":"J Pearce","year":"2020","unstructured":"Pearce, J. (2020). In defence of constructivist, utility-driven psychometrics for the \u2018post-psychometric era.\u2019 Medical Education, 54(2), 99\u2013102. https:\/\/doi.org\/10.1111\/medu.14039","journal-title":"Medical Education"},{"key":"10289_CR30","doi-asserted-by":"publisher","unstructured":"Pell, G., Fuller, R., Homer, M., & Roberts, T. (2010). How to measure the quality of the OSCE: A review of metrics\u2014AMEE guide no. 49. Medical Teacher, 32(10), 802\u2013811. https:\/\/doi.org\/10.3109\/0142159X.2010.507716","DOI":"10.3109\/0142159X.2010.507716"},{"issue":"1","key":"10289_CR31","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1007\/s10459-017-9771-4","volume":"23","author":"SK Schauber","year":"2018","unstructured":"Schauber, S. K., Hecht, M., & Nouns, Z. M. (2018). Why assessment in medical education needs a solid foundation in modern test theory. Advances in Health Sciences Education: Theory and Practice, 23(1), 217\u2013232. https:\/\/doi.org\/10.1007\/s10459-017-9771-4","journal-title":"Advances in Health Sciences Education: Theory and Practice"},{"issue":"5","key":"10289_CR32","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1002\/pits.20234","volume":"44","author":"B Thompson","year":"2007","unstructured":"Thompson, B. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools, 44(5), 423\u2013432. https:\/\/doi.org\/10.1002\/pits.20234","journal-title":"Psychology in the Schools"},{"key":"10289_CR33","doi-asserted-by":"publisher","DOI":"10.1080\/0142159X.2022.2031943","author":"N Valentine","year":"2022","unstructured":"Valentine, N., Durning, S. J., Shanahan, E. M., van der Vleuten, C., & Schuwirth, L. (2022). The pursuit of fairness in assessment: Looking beyond the objective. Medical Teacher. https:\/\/doi.org\/10.1080\/0142159X.2022.2031943","journal-title":"Medical Teacher"},{"issue":"1","key":"10289_CR34","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/s10459-022-10139-1","volume":"28","author":"WYA Wong","year":"2023","unstructured":"Wong, W. Y. A., Thistlethwaite, J., Moni, K., & Roberts, C. (2023). Using cultural historical activity theory to reflect on the sociocultural complexities in OSCE examiners\u2019 judgements. Advances in Health Sciences Education, 28(1), 27\u201346. https:\/\/doi.org\/10.1007\/s10459-022-10139-1","journal-title":"Advances in Health Sciences Education"},{"key":"10289_CR35","doi-asserted-by":"publisher","DOI":"10.1111\/medu.13783","author":"P Yeates","year":"2018","unstructured":"Yeates, P., Cope, N., Hawarden, A., Bradshaw, H., McCray, G., & Homer, M. (2018). Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs. Medical Education. https:\/\/doi.org\/10.1111\/medu.13783","journal-title":"Medical Education"},{"key":"10289_CR36","doi-asserted-by":"publisher","unstructured":"Yeates, P., Moult, A., Cope, N., McCray, G., Xilas, E., Lovelock, T., Vaughan, N., Daw, D., Fuller, R., & McKinley, R. K. (2021). Measuring the effect of examiner variability in a multiple-circuit Objective Structured Clinical Examination (OSCE). Academic Medicine. https:\/\/doi.org\/10.1097\/ACM.0000000000004028","DOI":"10.1097\/ACM.0000000000004028"},{"issue":"11","key":"10289_CR37","doi-asserted-by":"publisher","first-page":"1250","DOI":"10.1080\/0142159X.2020.1801997","volume":"42","author":"P Yeates","year":"2020","unstructured":"Yeates, P., Moult, A., Lefroy, J., Walsh-House, J., Clews, L., McKinley, R., & Fuller, R. (2020). Understanding and developing procedures for video-based assessment in medical education. Medical Teacher, 42(11), 1250\u20131260. https:\/\/doi.org\/10.1080\/0142159X.2020.1801997","journal-title":"Medical Teacher"},{"issue":"3","key":"10289_CR38","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1007\/s10459-012-9372-1","volume":"18","author":"P Yeates","year":"2013","unstructured":"Yeates, P., O\u2019Neill, P., Mann, K., & Eva, K. (2013). Seeing the same thing differently. Advances in Health Sciences Education, 18(3), 325\u2013341. https:\/\/doi.org\/10.1007\/s10459-012-9372-1","journal-title":"Advances in Health Sciences Education"}],"container-title":["Advances in Health Sciences Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10459-023-10289-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10459-023-10289-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10459-023-10289-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T07:15:48Z","timestamp":1719386148000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10459-023-10289-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,16]]},"references-count":38,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,7]]}},"alternative-id":["10289"],"URL":"https:\/\/doi.org\/10.1007\/s10459-023-10289-w","relation":{},"ISSN":["1382-4996","1573-1677"],"issn-type":[{"value":"1382-4996","type":"print"},{"value":"1573-1677","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,16]]},"assertion":[{"value":"28 March 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 September 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.\u00a0However,\u00a0the author does\u00a0act\u00a0as a paid assessment consultant to the PLAB2 panel on an ad hoc basis.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"When taking PLAB examinations, candidates consent to their examination data being used for research purposes. The General Medical Council in the UK gave permission for the analysis of the anonymized data used in this paper.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}