{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T17:01:48Z","timestamp":1779987708611,"version":"3.53.1"},"reference-count":8,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"6","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:sec>\n                    <jats:title>BACKGROUND:<\/jats:title>\n                    <jats:p>Standardized letters of recommendation were introduced to capture more meaningful fellowship applicant evaluations. There are concerns that this letter may introduce subjectivity to assessments traditionally measured by objective metrics, such as board scores and publications.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>OBJECTIVE:<\/jats:title>\n                    <jats:p>This study aimed to evaluate the influence of individual application elements on ranking by referee.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>DESIGN:<\/jats:title>\n                    <jats:p>Collected data included applicant demographics and objective assessments of academic performance, including in-service examination percentiles, US Medical Licensing Examination Step 1 and Step 2 scores, and publication count. Summary statistics and ORs from repeated-measures logistic regression models using the method of generalized estimating equations are reported.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>SETTINGS:<\/jats:title>\n                    <jats:p>Single academic institution.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>MAIN OUTCOME MEASURES:<\/jats:title>\n                    <jats:p>To determine whether objective applicant parameters are associated with being subjectively ranked No. 1 by the referee.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>RESULTS:<\/jats:title>\n                    <jats:p>\n                      Among 302 standardized letter of recommendation forms, 51.3% of the applicants were ranked No. 1 by the referee. Applicants ranked No. 1 had higher junior and senior in-training examination percentiles (60.3 vs 50.9,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0.0002 and 59.5 vs 48.0,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      &lt; 0.0001, respectively), Step 1 scores (235 vs 231,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0.0287), and average number of publications (5.9 vs 3.5,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0.0011). For each unit rise of in-training percentile and publication number, the OR for being ranked No. 1 improved by 1.03 (\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0.002) and 1.08 (\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0.013), respectively. Ranks lower than No. 1 did not demonstrate the expected decline in objective performance metrics with decreasing rank, except for Step 1 percentile (\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      \u2264 0.001). Technical ability was the strongest predictor of No. 1 ranking.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>LIMITATIONS:<\/jats:title>\n                    <jats:p>Analyses are limited to elements included in standardized letters.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>CONCLUSIONS:<\/jats:title>\n                    <jats:p>\n                      Standardized letters retain the relationship between objective performance metrics and subjective ratings for applicants ranked No. 1 by referee. However, this relationship is lost when applicants are ranked lower than first place. Our findings provide valuable insights and inform best practices for the use of standardized letters of recommendation in the context of fellowship selection. See\n                      <jats:bold>Video Abstract<\/jats:bold>\n                      .\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>EVALUACI\u00d3N CR\u00cdTICA DE LAS CARTAS DE RECOMENDACI\u00d3N ESTANDARIZADAS PARA BECAS DE INVESTIGACI\u00d3N EN CIRUG\u00cdA COLORRECTAL UTILIZANDO CRITERIOS OBJETIVOS<\/jats:title>\n                    <jats:p>\n                      <jats:bold>ANTECEDENTES:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>Se introdujo la carta de recomendaci\u00f3n estandarizada para obtener evaluaciones m\u00e1s significativas de los solicitantes de becas. Existe la preocupaci\u00f3n de que esta carta pueda introducir subjetividad en las evaluaciones que tradicionalmente se miden con m\u00e9tricas objetivas, como las puntuaciones de los ex\u00e1menes y las publicaciones.<\/jats:p>\n                    <jats:p>\n                      <jats:bold>OBJETIVO:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>Evaluar la influencia de los elementos individuales de la solicitud en la clasificaci\u00f3n por parte del evaluador.<\/jats:p>\n                    <jats:p>\n                      <jats:bold>DISE\u00d1O:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>Los datos recopilados incluyeron datos demogr\u00e1ficos de los solicitantes y evaluaciones objetivas del rendimiento acad\u00e9mico, incluyendo percentiles de ex\u00e1menes en servicio, puntuaciones de los pasos 1 y 2, y n\u00famero de publicaciones. Se presentan estad\u00edsticas resumidas y odds ratios de modelos de regresi\u00f3n log\u00edstica de medidas repetidas utilizando el m\u00e9todo de ecuaciones de estimaci\u00f3n generalizadas.<\/jats:p>\n                    <jats:p>\n                      <jats:bold>ENTORNO:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>Se analizaron las cartas estandarizadas enviadas a una sola instituci\u00f3n durante el ciclo de solicitudes de 2019.<\/jats:p>\n                    <jats:p>\n                      <jats:bold>PRINCIPALES MEDIDAS DE RESULTADOS:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>Determinar si los par\u00e1metros objetivos de los solicitantes est\u00e1n asociados con la clasificaci\u00f3n subjetiva como n\u00famero 1 por parte del \u00e1rbitro.<\/jats:p>\n                    <jats:p>\n                      <jats:bold>RESULTADOS:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>\n                      Entre los 302 formularios LOR estandarizados, el 51,3 % de los solicitantes fueron clasificados en primer lugar por el evaluador. Los solicitantes clasificados en primer lugar obtuvieron percentiles m\u00e1s altos en los ex\u00e1menes de formaci\u00f3n de primer y segundo a\u00f1o (60,3 frente a 50,9,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0,0002 y 59,5 frente a 48,0,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      &lt; 0,0001, respectivamente), puntuaciones en el Step 1 (235 frente a 231,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0,0287) y un mayor n\u00famero medio de publicaciones (5,9 frente a 3,5,\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0,0011). Por cada unidad de aumento en el percentil de formaci\u00f3n y el n\u00famero de publicaciones, la raz\u00f3n de probabilidades de obtener la clasificaci\u00f3n n.\u00ba 1 mejor\u00f3 en 1,03 (\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0,002) y 1,08 (\n                      <jats:italic toggle=\"yes\">p<\/jats:italic>\n                      = 0,013), respectivamente. Las clasificaciones inferiores a la n.\u00ba 1 no mostraron la disminuci\u00f3n esperada en las m\u00e9tricas de rendimiento objetivo al descender la clasificaci\u00f3n, excepto el percentil del paso 1 (\n                      <jats:italic toggle=\"yes\">p \u2264<\/jats:italic>\n                      0,001). La capacidad t\u00e9cnica fue el predictor m\u00e1s fuerte de la clasificaci\u00f3n en el puesto n.\u00ba 1.\n                    <\/jats:p>\n                    <jats:p>\n                      <jats:bold>LIMITACIONES:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>Los an\u00e1lisis se limitaron a los elementos incluidos en las cartas estandarizadas.<\/jats:p>\n                    <jats:p>\n                      <jats:bold>CONCLUSIONES:<\/jats:bold>\n                    <\/jats:p>\n                    <jats:p>\n                      Las cartas estandarizadas mantienen la relaci\u00f3n entre las m\u00e9tricas de rendimiento objetivo y las calificaciones subjetivas de los candidatos clasificados en el puesto n.\u00ba 1 por el evaluador. Sin embargo, esta relaci\u00f3n se pierde cuando los candidatos se clasifican por debajo del primer puesto. Nuestros hallazgos proporcionan informaci\u00f3n valiosa y sirven de base para las mejores pr\u00e1cticas en el uso de cartas de recomendaci\u00f3n estandarizadas en el contexto de la selecci\u00f3n de becas.\n                      <jats:italic toggle=\"yes\">(AI-generated translation<\/jats:italic>\n                      )\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1097\/dcr.0000000000004091","type":"journal-article","created":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T15:00:08Z","timestamp":1769612408000},"page":"1766-1771","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":0,"title":["A Critical Appraisal of Colorectal Surgery Fellowship Standardized Letters of Recommendations Using Objective Criteria"],"prefix":"10.1097","volume":"69","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3063-6267","authenticated-orcid":false,"given":"Shan L.","family":"Kalmeta","sequence":"first","affiliation":[{"name":"Department of Surgery, Oregon Health and Science University, Portland, Oregon"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wendelyn M.","family":"Oslock","sequence":"additional","affiliation":[{"name":"Department of Surgery, University of Alabama, Birmingham, Alabama"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahmoud H.","family":"Abdel-Rasoul","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, The Ohio State University College of Medicine, Columbus, Ohio"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa R.","family":"Coleman","sequence":"additional","affiliation":[{"name":"Department of Surbery, Inova-Fairfax Hospital, Washington, DC"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sabrena F.","family":"Noria","sequence":"additional","affiliation":[{"name":"Department of Surgery, The Ohio State University Wexner Medical Center, Columbus, Ohio"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew F.","family":"Kalady","sequence":"additional","affiliation":[{"name":"Department of Surgery, The Ohio State University Wexner Medical Center, Columbus, Ohio"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2530-8362","authenticated-orcid":false,"given":"Syed G.","family":"Husain","sequence":"additional","affiliation":[{"name":"Department of Surgery, The Ohio State University Wexner Medical Center, Columbus, Ohio"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2026,1,28]]},"reference":[{"key":"R2-20260528","doi-asserted-by":"crossref","first-page":"e121","DOI":"10.1016\/j.jsurg.2020.06.036","article-title":"Best practices in letters of recommendation for general surgery residency: results of expert stakeholder focus groups.","volume":"77","author":"Naples","year":"2020","journal-title":"J Surg Educ"},{"key":"R3-20260528","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1016\/j.jsurg.2020.07.030","article-title":"Colon and rectal surgery residency selection criteria: a national program director survey.","volume":"78","author":"Lam","year":"2021","journal-title":"J Surg Educ"},{"key":"R4-20260528","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1097\/DCR.0000000000000525","article-title":"Current state of colorectal surgery training: a survey of program directors, current and recently matched colorectal residents, and recent colorectal graduates.","volume":"59","author":"Bailey","year":"2016","journal-title":"Dis Colon Rectum"},{"key":"R5-20260528","doi-asserted-by":"crossref","first-page":"e20.00023","DOI":"10.2106\/JBJS.OA.20.00023","article-title":"Race- and gender-based differences in descriptions of applicants in the letters of recommendation for orthopaedic surgery residency.","volume":"5","author":"Powers","year":"2020","journal-title":"JB JS Open Access"},{"key":"R6-20260528","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1176\/appi.ap.35.5.342","article-title":"Letters of recommendation in residency training: what do they really mean?","volume":"35","author":"MacLean","year":"2011","journal-title":"Acad Psychiatry"},{"key":"R7-20260528","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1067\/S0022-3476(03)00255-5","article-title":"Improving the letter of recommendation for pediatric residency applicants: an idea whose time has come?","volume":"143","author":"Morgenstern","year":"2003","journal-title":"J Pediatr"},{"key":"R8-20260528","doi-asserted-by":"crossref","first-page":"261","DOI":"10.4300\/JGME-D-17-00712.1","article-title":"Dear program director: deciphering letters of recommendation.","volume":"10","author":"Saudek","year":"2018","journal-title":"J Grad Med Educ"},{"key":"R9-20260528","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/S0149-7944(01)00538-4","article-title":"The content and value of letters of recommendation in the resident candidate evaluative process.","volume":"59","author":"Fortune","year":"2002","journal-title":"Curr Surg"}],"container-title":["Diseases of the Colon &amp; Rectum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.lww.com\/10.1097\/DCR.0000000000004091","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T16:02:26Z","timestamp":1779984146000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.lww.com\/10.1097\/DCR.0000000000004091"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,28]]},"references-count":8,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.1097\/dcr.0000000000004091","relation":{},"ISSN":["0012-3706","1530-0358"],"issn-type":[{"value":"0012-3706","type":"print"},{"value":"1530-0358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,28]]}}}