{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T16:06:07Z","timestamp":1792080367732,"version":"4.4.0"},"reference-count":103,"publisher":"Wiley","issue":"8","license":[{"start":{"date-parts":[[2013,7,10]],"date-time":"2013-07-10T00:00:00Z","timestamp":1373414400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["asmepublications.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Education"],"published-print":{"date-parts":[[2013,8]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>The purpose of this study was to use meta\u2010analysis to establish which of the information available to the resident selection committee is associated with resident or doctor performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      Multiple electronic databases were searched to 4 September 2012. Two reviewers independently selected studies that met the present inclusion criteria and extracted data in duplicate; disagreement was resolved by consensus. Risk for bias was assessed using a customised bias assessment tool. Measures of association were converted to a common effect size (\n                      <jats:styled-content style=\"fixed-case\">H<\/jats:styled-content>\n                      edges' g). Meta\u2010analysis was performed using the random\u2010effects model for each selection strategy and all outcomes without pooling. Sensitivity analysis for each selection strategy\u2013outcome pair was performed with pooling of effect size.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Eighty studies involving a total of 41\u00a0704 participants were included in the meta\u2010analysis. Seventeen different selection strategies and 17 outcomes were assessed across these studies. The strongest positive associations referred to examination\u2010based selection strategies, such as the US Medical Licensing Examination (\n                      <jats:styled-content style=\"fixed-case\">USMLE<\/jats:styled-content>\n                      ) Step 1, and examination\u2010based outcomes, such as scores on in\u2010training examinations. Moderate positive associations were present for medical school marks and both examination\u2010based and subjective outcomes. Minimal or no associations were seen for the selection tools represented by interviews, reference letters and deans' letters.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Standardised examination performance and medical school grades show the strongest associations with current measures of doctor performance. Deans' letters, reference letters and interviews all show a lower than expected strength of association given the relative value often assigned to them during resident doctor selection. Objective selection strategies are potentially the most useful to residency selection committees based on current evaluative methods. However, reports in the literature of validated long\u2010term doctor performance outcomes are scant.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/medu.12234","type":"journal-article","created":{"date-parts":[[2013,7,10]],"date-time":"2013-07-10T00:53:15Z","timestamp":1373417595000},"page":"790-800","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":95,"title":["Associations between residency selection strategies and doctor performance: a meta\u2010analysis"],"prefix":"10.1111","volume":"47","author":[{"given":"Stephanie","family":"Kenny","sequence":"first","affiliation":[{"name":"Department of Medical Imaging Ottawa Hospital University of Ottawa Ottawa Ontario Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"McInnes","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging Ottawa Hospital University of Ottawa Ottawa Ontario Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vivek","family":"Singh","sequence":"additional","affiliation":[{"name":"Department of Medical Imaging Ottawa Hospital University of Ottawa Ottawa Ontario Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2013,7,10]]},"reference":[{"key":"e_1_2_6_2_1","first-page":"195","article-title":"The residency application abyss: insights and advice","volume":"84","author":"Olson DP","year":"2011","journal-title":"Yale J Biol Med"},{"key":"e_1_2_6_3_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-199207000-00011"},{"key":"e_1_2_6_4_1","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1097\/00001888-199901001-00018","article-title":"Programme directors\u2019 responses to a survey on variables used to select residents in a time of change","volume":"74","author":"Wagoner NE","year":"1999","journal-title":"Acad Med"},{"key":"e_1_2_6_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaad.2010.07.034"},{"key":"e_1_2_6_6_1","doi-asserted-by":"publisher","DOI":"10.1080\/01421590600622723"},{"key":"e_1_2_6_7_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.290.9.1183"},{"key":"e_1_2_6_8_1","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0b013e3181970c6b"},{"key":"e_1_2_6_9_1","unstructured":"Association of American Medical Colleges.Curriculum report: honours policy \u2013 curriculum inventory and reports (CIR) \u2013 initiatives.https:\/\/www.aamc.org\/download\/183226\/data\/honorspolicy.pdf. [Accessed 23 November 2012.]"},{"key":"e_1_2_6_10_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.283.15.2008"},{"key":"e_1_2_6_11_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.b2700"},{"key":"e_1_2_6_12_1","doi-asserted-by":"publisher","DOI":"10.1093\/jpepsy\/jsp004"},{"issue":"2","key":"e_1_2_6_13_1","first-page":"249","article-title":"Assessing agreement on classification tasks: the kappa statistic","volume":"22","author":"Carletta J","year":"1996","journal-title":"Comput Linguist"},{"key":"e_1_2_6_14_1","unstructured":"ViswanathanM AnsariMT BerkmanNDet\u00a0al.Assessing the risk of bias of individual studies in systematic reviews of health care interventions. Agency for Healthcare Research and Quality Methods Guide for Comparative Effectiveness Reviews. AHRQ Publication No. 12\u2010EHC047\u2010EF. 2012.www.effectivehealthcare.ahrq.gov\/[Accessed 22 September 2012.]"},{"key":"e_1_2_6_15_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.315.7109.629"},{"key":"e_1_2_6_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/0197-2456(86)90046-2"},{"key":"e_1_2_6_17_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.327.7414.557"},{"key":"e_1_2_6_18_1","first-page":"278","volume-title":"Cochrane Handbook for Systematic Reviews of Interventions","author":"Deeks JJ","year":"2008"},{"key":"e_1_2_6_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1076-6332(00)80580-2"},{"key":"e_1_2_6_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsurg.2011.07.007"},{"key":"e_1_2_6_21_1","doi-asserted-by":"publisher","DOI":"10.1097\/00004424-199205000-00013"},{"key":"e_1_2_6_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2004.03.003"},{"key":"e_1_2_6_23_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-200510001-00009"},{"key":"e_1_2_6_24_1","doi-asserted-by":"publisher","DOI":"10.7205\/MILMED.172.6.640"},{"key":"e_1_2_6_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2923.2011.04080.x"},{"key":"e_1_2_6_26_1","doi-asserted-by":"publisher","DOI":"10.1177\/0194599810396604"},{"key":"e_1_2_6_27_1","doi-asserted-by":"publisher","DOI":"10.1006\/jsre.2000.5924"},{"key":"e_1_2_6_28_1","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.C.01184"},{"key":"e_1_2_6_29_1","doi-asserted-by":"publisher","DOI":"10.1001\/archpedi.154.3.256"},{"key":"e_1_2_6_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1076-6332(03)80189-7"},{"key":"e_1_2_6_31_1","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0b013e3181e1a85d"},{"key":"e_1_2_6_32_1","first-page":"189","article-title":"Validity coefficients of clinical competence on NBME Part III examination","volume":"60","author":"Calhoun JG","year":"1985","journal-title":"J Med Educ"},{"key":"e_1_2_6_33_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-200010001-00023"},{"key":"e_1_2_6_34_1","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0b013e3181cece3d"},{"key":"e_1_2_6_35_1","doi-asserted-by":"publisher","DOI":"10.1097\/01.SMJ.0000157560.75496.CB"},{"key":"e_1_2_6_36_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2923.2009.03605.x"},{"key":"e_1_2_6_37_1","doi-asserted-by":"publisher","DOI":"10.1001\/archoto.2012.1374"},{"issue":"7","key":"e_1_2_6_38_1","first-page":"590","article-title":"The dean's letter of recommendation and internship performance","volume":"51","author":"Clemente M","year":"1976","journal-title":"J Med Educ"},{"key":"e_1_2_6_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsurg.2009.12.006"},{"key":"e_1_2_6_40_1","doi-asserted-by":"publisher","DOI":"10.4065\/mcp.2010.0655"},{"key":"e_1_2_6_41_1","doi-asserted-by":"publisher","DOI":"10.1177\/016327878801100108"},{"key":"e_1_2_6_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jamcollsurg.2005.12.006"},{"issue":"6","key":"e_1_2_6_43_1","first-page":"715","article-title":"Evaluation of a system designed to enhance the resident selection process","volume":"109","author":"DaRosa DA","year":"1991","journal-title":"Surgery"},{"key":"e_1_2_6_44_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ap.29.1.69"},{"key":"e_1_2_6_45_1","doi-asserted-by":"publisher","DOI":"10.1001\/archsurg.2010.177"},{"key":"e_1_2_6_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11999-010-1327-3"},{"key":"e_1_2_6_47_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ap.29.5.443"},{"key":"e_1_2_6_48_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ap.32.6.498"},{"issue":"2","key":"e_1_2_6_49_1","first-page":"270","article-title":"Resident selection: applicant selection criteria compared with performance","volume":"92","author":"Erlandson EE","year":"1982","journal-title":"Surgery"},{"key":"e_1_2_6_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaad.2009.12.051"},{"key":"e_1_2_6_51_1","doi-asserted-by":"publisher","DOI":"10.1097\/01.PHM.0000083688.23340.EA"},{"issue":"1","key":"e_1_2_6_52_1","first-page":"80","article-title":"Overall student performance as a measure of internship performance","volume":"74","author":"Gardner B","year":"1973","journal-title":"Surgery"},{"key":"e_1_2_6_53_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-198908000-00013"},{"key":"e_1_2_6_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-007-0405-z"},{"key":"e_1_2_6_55_1","doi-asserted-by":"publisher","DOI":"10.1213\/ANE.0b013e3181fcfacd"},{"key":"e_1_2_6_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1076-6332(00)80575-9"},{"issue":"5","key":"e_1_2_6_57_1","first-page":"379","article-title":"Pre\u2010medical and medical school performance in predicting first\u2010year residency performance","volume":"62","author":"Gunzburger LK","year":"1987","journal-title":"J Med Educ"},{"key":"e_1_2_6_58_1","doi-asserted-by":"publisher","DOI":"10.1197\/j.aem.2005.01.002"},{"key":"e_1_2_6_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11999-009-0939-y"},{"key":"e_1_2_6_60_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-199302000-00035"},{"key":"e_1_2_6_61_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2923.2007.02841.x"},{"key":"e_1_2_6_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2010.03.056"},{"key":"e_1_2_6_63_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15328015TLM1301_3"},{"key":"e_1_2_6_64_1","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-9-42"},{"issue":"10","key":"e_1_2_6_65_1","first-page":"759","article-title":"Efficacy of cognitive\/non\u2010cognitive measures in predicting resident\u2010doctor performance","volume":"54","author":"Keck JW","year":"1979","journal-title":"J Med Educ"},{"key":"e_1_2_6_66_1","doi-asserted-by":"publisher","DOI":"10.1097\/00007611-198612000-00020"},{"key":"e_1_2_6_67_1","first-page":"302","article-title":"Resident recruitment: what are good predictors for performance during paediatric residency training?","volume":"85","author":"Khongphatthanayothin A","year":"2002","journal-title":"J Med Assoc Thai"},{"issue":"5","key":"e_1_2_6_68_1","first-page":"425","article-title":"The applicant interview as a predictor of resident performance","volume":"59","author":"Komives E","year":"1984","journal-title":"J Med Educ"},{"key":"e_1_2_6_69_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-199304000-00018"},{"key":"e_1_2_6_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ambp.2007.01.002"},{"key":"e_1_2_6_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-008-0641-x"},{"key":"e_1_2_6_72_1","doi-asserted-by":"publisher","DOI":"10.1001\/archsurg.143.7.647"},{"key":"e_1_2_6_73_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1986.03380240061029"},{"key":"e_1_2_6_74_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15328015tlm1804_6"},{"key":"e_1_2_6_75_1","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6920-2-7"},{"key":"e_1_2_6_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0002-9610(96)00389-3"},{"key":"e_1_2_6_77_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-199311000-00015"},{"key":"e_1_2_6_78_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-009-9158-2"},{"key":"e_1_2_6_79_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.acra.2011.11.017"},{"issue":"1","key":"e_1_2_6_80_1","first-page":"47","article-title":"Correlation between housestaff performance on the United States Medical Licensing Examination and standardised patient encounters","volume":"72","author":"Rifkin WD","year":"2005","journal-title":"Mt Sinai J Med"},{"key":"e_1_2_6_81_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ap.34.2.128"},{"key":"e_1_2_6_82_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02599990"},{"key":"e_1_2_6_83_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0002-9378(85)80242-8"},{"key":"e_1_2_6_84_1","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.H.01243"},{"key":"e_1_2_6_85_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsurg.2010.06.010"},{"key":"e_1_2_6_86_1","doi-asserted-by":"publisher","DOI":"10.4300\/JGME-D-09-00101.1"},{"key":"e_1_2_6_87_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajog.2011.02.019"},{"key":"e_1_2_6_88_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-199109000-00024"},{"key":"e_1_2_6_89_1","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0b013e3181b37fd2"},{"key":"e_1_2_6_90_1","doi-asserted-by":"publisher","DOI":"10.1097\/00001888-200505000-00019"},{"key":"e_1_2_6_91_1","doi-asserted-by":"publisher","DOI":"10.1097\/BLO.0b013e31805d7eda"},{"key":"e_1_2_6_92_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jemermed.2008.04.010"},{"key":"e_1_2_6_93_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsurg.2010.05.002"},{"issue":"7","key":"e_1_2_6_94_1","first-page":"572","article-title":"Using ratings of resident competence to evaluate NBME examination passing standards","volume":"62","author":"Turner BJ","year":"1987","journal-title":"J Med Educ"},{"key":"e_1_2_6_95_1","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1097\/01.blo.0000224042.84839.44","article-title":"A quantitative composite scoring tool for orthopaedic residency screening and selection","volume":"44","author":"Turner NS","year":"2006","journal-title":"Clin Orthop Relat Res"},{"issue":"7","key":"e_1_2_6_96_1","first-page":"591","article-title":"Predictors of success in an anaesthesiology residency","volume":"61","author":"Warrick SS","year":"1986","journal-title":"J Med Educ"},{"issue":"12","key":"e_1_2_6_97_1","first-page":"967","article-title":"A prospective evaluation of performance during the first year of the medical residency","volume":"59","author":"Weiss ST","year":"1984","journal-title":"J Med Educ"},{"issue":"5","key":"e_1_2_6_98_1","first-page":"356","article-title":"Medical school achievements as predictors of residency performance","volume":"63","author":"Yindra KJ","year":"1988","journal-title":"J Med Educ"},{"key":"e_1_2_6_99_1","doi-asserted-by":"publisher","DOI":"10.4300\/JGME-D-09-00098.1"},{"key":"e_1_2_6_100_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMra054784"},{"key":"e_1_2_6_101_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.287.2.226"},{"key":"e_1_2_6_102_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(03)14849-0"},{"key":"e_1_2_6_103_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2923.2010.03760.x"},{"key":"e_1_2_6_104_1","doi-asserted-by":"publisher","DOI":"10.1108\/00483480110380154"}],"container-title":["Medical Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1111%2Fmedu.12234","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/medu.12234","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T15:26:13Z","timestamp":1762269973000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmepublications.onlinelibrary.wiley.com\/doi\/10.1111\/medu.12234"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,7,10]]},"references-count":103,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2013,8]]}},"alternative-id":["10.1111\/medu.12234"],"URL":"https:\/\/doi.org\/10.1111\/medu.12234","archive":["Portico"],"relation":{},"ISSN":["0308-0110","1365-2923"],"issn-type":[{"value":"0308-0110","type":"print"},{"value":"1365-2923","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,7,10]]},"assertion":[{"value":"2012-11-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-03-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-07-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}