{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T15:32:44Z","timestamp":1772119964526,"version":"3.50.1"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,3,17]],"date-time":"2021-03-17T00:00:00Z","timestamp":1615939200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,17]],"date-time":"2021-03-17T00:00:00Z","timestamp":1615939200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Eng Ethics"],"published-print":{"date-parts":[[2021,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The primary goal of the peer review of research grant proposals is to evaluate their quality for the funding agency. An important secondary goal is to provide constructive feedback to applicants for their resubmissions. However, little is known about whether review feedback achieves this goal. In this paper, we present a multi-methods analysis of responses from grant applicants regarding their perceptions of the effectiveness and appropriateness of peer review feedback they received from grant submissions. Overall, 56\u201360% of applicants determined the feedback to be appropriate (fair, well-written, and well-informed), although their judgments were more favorable if their recent application was funded. Importantly, independent of funding success, women found the feedback better written than men, and more white applicants found the feedback to be fair than non-white applicants. Also, perceptions of a variety of biases were specifically reported in respondents\u2019 feedback. Less than 40% of applicants found the feedback to be very useful in informing their research and improving grantsmanship and future submissions. Further, negative perceptions of the appropriateness of review feedback were positively correlated with more negative perceptions of feedback usefulness. Importantly, respondents suggested that highly competitive funding pay-lines and poor inter-panel reliability limited the usefulness of review feedback. Overall, these results suggest that more effort is needed to ensure that appropriate and useful feedback is provided to all applicants, bolstering the equity of the review process and likely improving the quality of resubmitted proposals.<\/jats:p>","DOI":"10.1007\/s11948-021-00295-9","type":"journal-article","created":{"date-parts":[[2021,3,17]],"date-time":"2021-03-17T15:03:07Z","timestamp":1615993387000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Grant Review Feedback: Appropriateness and Usefulness"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6043-2153","authenticated-orcid":false,"given":"Stephen A.","family":"Gallo","sequence":"first","affiliation":[]},{"given":"Karen B.","family":"Schmaling","sequence":"additional","affiliation":[]},{"given":"Lisa A.","family":"Thompson","sequence":"additional","affiliation":[]},{"given":"Scott R.","family":"Glisson","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,3,17]]},"reference":[{"issue":"1\u20132","key":"295_CR1","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1023\/A:1018768729602","volume":"38","author":"S Beyer","year":"1998","unstructured":"Beyer, S. (1998). Gender differences in self-perception and negative recall biases. Sex Roles: A Journal of Research, 38(1\u20132), 103\u2013133.","journal-title":"Sex Roles: A Journal of Research"},{"issue":"1","key":"295_CR2","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1177\/0146167219845921","volume":"46","author":"M Biernat","year":"2020","unstructured":"Biernat, M., Carnes, M., Filut, A., & Kaatz, A. (2020). Gender, race, and grant reviews: Translating and responding to research feedback. Personality and Social Psychology Bulletin, 46(1), 140\u2013154.","journal-title":"Personality and Social Psychology Bulletin"},{"key":"295_CR3","unstructured":"Boss J. M. & Eckert S.H. (2003). Academic scientists at work: I Can't believe they didn't like it!: Part II---grant proposals. https:\/\/www.sciencemag.org\/careers\/2003\/12\/academic-scientists-work-i-cant-believe-they-didnt-it-part-ii-grant-proposals. Last accessed April 2020."},{"issue":"4","key":"295_CR4","doi-asserted-by":"publisher","first-page":"556","DOI":"10.1097\/ACM.0000000000001025","volume":"91","author":"JE Boyington","year":"2016","unstructured":"Boyington, J. E., Antman, M. D., Patel, K. C., & Lauer, M. S. (2016). Towards independence: Resubmission rate of unfunded national heart, lung, and blood institute R01 research grant applications among early stage investigators. Academic Medicine: Journal of the Association of American Medical Colleges, 91(4), 556.","journal-title":"Academic Medicine: Journal of the Association of American Medical Colleges"},{"issue":"2","key":"295_CR6","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1073\/pnas.1418761112","volume":"112","author":"RJ Daniels","year":"2015","unstructured":"Daniels, R. J. (2015). A generation at risk: Young investigators and the future of the biomedical workforce. Proceedings of the National Academy of Sciences, 112(2), 313\u2013318.","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"23","key":"295_CR7","doi-asserted-by":"publisher","first-page":"eaaz4868","DOI":"10.1126\/sciadv.aaz4868","volume":"6","author":"EA Erosheva","year":"2020","unstructured":"Erosheva, E. A., Grant, S., Chen, M. C., Lindner, M. D., Nakamura, R. K., & Lee, C. J. (2020). NIH peer review: Criterion scores completely account for racial disparities in overall impact scores. Science Advances, 6(23), eaaz4868.","journal-title":"Science Advances"},{"issue":"2","key":"295_CR9","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1007\/s10669-018-9677-6","volume":"38","author":"S Gallo","year":"2018","unstructured":"Gallo, S., Thompson, L., Schmaling, K., & Glisson, S. (2018). Risk evaluation in peer review of grant applications. Environment Systems and Decisions, 38(2), 216\u2013229.","journal-title":"Environment Systems and Decisions"},{"key":"295_CR8","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/s41073-020-00093-0","volume":"5","author":"SA Gallo","year":"2020","unstructured":"Gallo, S. A., Schmaling, K. B., Thompson, L. A., & Glisson, S. R. (2020a). Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion. Research Integrity and Peer Review, 5, 7. https:\/\/doi.org\/10.1186\/s41073-020-00093-0","journal-title":"Research Integrity and Peer Review"},{"key":"295_CR10","doi-asserted-by":"publisher","first-page":"761","DOI":"10.1007\/s11948-019-00123-1","volume":"26","author":"SA Gallo","year":"2020","unstructured":"Gallo, S. A., Thompson, L. A., Schmaling, K. B., & Glisson, S. R. (2020b). The participation and motivations of grant peer reviewers: A comprehensive survey. Science and Engineering Ethics, 26, 761\u2013782. https:\/\/doi.org\/10.1007\/s11948-019-00123-1","journal-title":"Science and Engineering Ethics"},{"issue":"8","key":"295_CR11","doi-asserted-by":"publisher","first-page":"1098","DOI":"10.1097\/ACM.0000000000001278","volume":"91","author":"DK Ginther","year":"2016","unstructured":"Ginther, D. K., Kahn, S., & Schaffer, W. T. (2016). Gender, race\/ethnicity, and National Institutes of Health R01 research awards: Is there evidence of a double bind for women of color? Academic Medicine: Journal of the Association of American Medical Colleges, 91(8), 1098.","journal-title":"Academic Medicine: Journal of the Association of American Medical Colleges"},{"issue":"6045","key":"295_CR12","doi-asserted-by":"publisher","first-page":"1015","DOI":"10.1126\/science.1196783","volume":"333","author":"DK Ginther","year":"2011","unstructured":"Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. (2011). Race, ethnicity, and NIH research awards. Science, 333(6045), 1015\u20131019.","journal-title":"Science"},{"key":"295_CR13","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1093\/biosci\/bix034","volume":"67","author":"R Gropp","year":"2017","unstructured":"Gropp, R., Glisson, S., Gallo, S., & Thompson, L. (2017). Peer review: A system under stress. BioScience, 67, 407\u2013410.","journal-title":"BioScience"},{"issue":"12","key":"295_CR14","doi-asserted-by":"publisher","first-page":"e1002010","DOI":"10.1371\/journal.pbio.1002010","volume":"12","author":"D Gurwitz","year":"2014","unstructured":"Gurwitz, D., Milanesi, E., & Koenig, T. (2014). Grant application review: The case of transparency. PLoS Biology, 12(12), e1002010. https:\/\/doi.org\/10.1371\/journal.pbio.1002010","journal-title":"PLoS Biology"},{"issue":"9","key":"295_CR15","doi-asserted-by":"publisher","first-page":"e0199648","DOI":"10.1371\/journal.pone.0199648","volume":"13","author":"PA Haggerty","year":"2018","unstructured":"Haggerty, P. A., & Fenton, M. J. (2018). Outcomes of early NIH-funded investigators: Experience of the National Institute of Allergy and Infectious Diseases. PloS ONE, 13(9), e0199648. https:\/\/doi.org\/10.1371\/journal.pone.0199648","journal-title":"PloS ONE"},{"issue":"31","key":"295_CR16","doi-asserted-by":"publisher","first-page":"7943","DOI":"10.1073\/pnas.1800615115","volume":"115","author":"LA Hechtman","year":"2018","unstructured":"Hechtman, L. A., Moore, N. P., Schulkey, C. E., Miklos, A. C., Calcagno, A. M., Aragon, R., & Greenberg, J. H. (2018). NIH funding longevity by gender. Proceedings of the National Academy of Sciences, 115(31), 7943\u20137948. https:\/\/doi.org\/10.1073\/pnas.1800615115","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"17","key":"295_CR17","doi-asserted-by":"publisher","first-page":"9284","DOI":"10.1073\/pnas.1915378117","volume":"117","author":"B Hofstra","year":"2020","unstructured":"Hofstra, B., Kulkarni, V. V., Galvez, S. M. N., He, B., Jurafsky, D., & McFarland, D. A. (2020). The diversity-innovation paradox in science. Proceedings of the National Academy of Sciences, 117(17), 9284\u20139291.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"295_CR18","unstructured":"Irwin, D., S. A. Gallo, & S. R. Glisson. (2013). Opinion: Learning from peer review. The Scientist http:\/\/www.the-scientist.com. Last accessed April 2020."},{"key":"295_CR19","unstructured":"Lauer, M. (2016). Are you on the fence about whether to resubmit? https:\/\/nexus.od.nih.gov\/all\/2016\/10\/28\/are-you-on-the-fence-about-whether-to-resubmit\/. Last accessed April 2020."},{"key":"295_CR20","unstructured":"Lauer, M. (2017). Resubmissions revisited: Funded resubmission applications and their initial peer review scores. https:\/\/nexus.od.nih.gov\/all\/2017\/02\/17\/resubmissions-revisited-funded-resubmission-applications-and-their-initial-peer-review-scores\/. Last accessed April 2020."},{"issue":"1","key":"295_CR22","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1002\/asi.22784","volume":"64","author":"CJ Lee","year":"2013","unstructured":"Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2\u201317.","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"295_CR23","unstructured":"Mayo, M. (2016). The gender gap in feedback and self-perception harvard business review. https:\/\/hbr.org\/2016\/08\/the-gender-gap-in-feedback-and-self-perception. Last accessed April 2020."},{"key":"295_CR24","doi-asserted-by":"crossref","unstructured":"Mervis, J. (2016). In effort to understand continuing racial disparities, NIH to test for bias in study sections. Science. https:\/\/www.sciencemag.org\/news\/2016\/06\/effort-understand-continuing-racial-disparities-nih-test-bias-study-sections#. Last accessed April 2020.","DOI":"10.1126\/science.aaf5776"},{"key":"295_CR26","unstructured":"NIAID. (2020). Revise and resubmit an application. https:\/\/www.niaid.nih.gov\/grants-contracts\/revise-resubmit-application. Last accessed April 2020."},{"key":"295_CR27","unstructured":"NIH. (2012a). Enhancing peer review survey results report_2012 https:\/\/enhancing-peer-review.nih.gov\/docs\/Enhancing_Peer_Review_Report_2012.pdf. Last accessed November 2018."},{"key":"295_CR28","unstructured":"NIH. (2012b). Draft report of the advisory committee to the director working group on diversity in the biomedical research workforce. https:\/\/acd.od.nih.gov\/documents\/reports\/DiversityBiomedicalResearchWorkforceReport.pdf. Last accessed April 2020."},{"key":"295_CR29","unstructured":"NIH. (2017). Enhancing peer review survey results report https:\/\/grants.nih.gov\/grants\/peer\/files\/Final-Phase-III-Enhancing-Peer-Review-Report.pdf. Last accessed April 2020"},{"key":"295_CR30","unstructured":"NIH. (2018). Peer review https:\/\/grants.nih.gov\/grants\/peer-review.htm. Last accessed April 2020."},{"key":"295_CR31","unstructured":"NIH. (2019). NIH peer review: Grants and cooperative agreements. https:\/\/grants.nih.gov\/grants\/peerreview22713webv2.pdf. Last accessed April 2020."},{"key":"295_CR32","unstructured":"NIH. (2020). Frequently asked questions. Applicant guidance: Next steps. https:\/\/grants.nih.gov\/faqs#\/next-steps.htm?anchor=question53104. Last accessed April 2020."},{"key":"295_CR5","unstructured":"NIH CSR. (2020). CSR insider's guide to NIH peer review for reviewers. https:\/\/public.csr.nih.gov\/ForReviewers\/MeetingOverview\/InsidersGuideReviewers. Last accessed October 2020"},{"issue":"1","key":"295_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/reseval\/rvw025","volume":"26","author":"EL Pier","year":"2017","unstructured":"Pier, E. L., Raclaw, J., Kaatz, A., Brauer, M., Carnes, M., Nathan, M. J., & Ford, C. E. (2017). \u2018Your comments are meaner than your score\u2019: Score calibration talk influences intra-and inter-panel variability during scientific grant peer review. Research Evaluation, 26(1), 1\u201314.","journal-title":"Research Evaluation"},{"issue":"12","key":"295_CR33","doi-asserted-by":"publisher","first-page":"2952","DOI":"10.1073\/pnas.1714379115","volume":"115","author":"EL Pier","year":"2018","unstructured":"Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018). Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences, 115(12), 2952\u20132957.","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"11\u201312","key":"295_CR35","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1007\/BF00289805","volume":"21","author":"TA Roberts","year":"1989","unstructured":"Roberts, T. A., & Nolen-Hoeksema, S. (1989). Sex differences in reactions to evaluative feedback. Sex Roles, 21(11\u201312), 725\u2013747.","journal-title":"Sex Roles"},{"key":"295_CR36","unstructured":"Sutcivni J. (2017). Revising and resubmitting rejected proposals. https:\/\/www.giving.temple.edu\/s\/705\/images\/editor_documents\/giving\/revising_rejected_proposals_april_2017.pdf?sessionid=21ba9f7f-3807-44ed-9721-4e0bc54831a0&cc=1. Last accessed April 2020."},{"issue":"16","key":"295_CR38","doi-asserted-by":"publisher","first-page":"E489","DOI":"10.1503\/cmaj.170901","volume":"190","author":"R Tamblyn","year":"2018","unstructured":"Tamblyn, R., Girard, N., Qian, C. J., & Hanley, J. (2018). Assessment of potential bias in research grant peer review in Canada. CMAJ, 190(16), E489\u2013E499.","journal-title":"CMAJ"},{"key":"295_CR39","unstructured":"Ware, M. (2008). Peer review: Benefits, perceptions and alternatives. London, UK: Publishing Research Consortium. http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.214.9676&rep=rep1&type=pdf. Last accessed April 2020."}],"container-title":["Science and Engineering Ethics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11948-021-00295-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11948-021-00295-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11948-021-00295-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,28]],"date-time":"2021-04-28T13:06:20Z","timestamp":1619615180000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11948-021-00295-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,17]]},"references-count":36,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,4]]}},"alternative-id":["295"],"URL":"https:\/\/doi.org\/10.1007\/s11948-021-00295-9","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.11.24.396192","asserted-by":"object"}]},"ISSN":["1353-3452","1471-5546"],"issn-type":[{"value":"1353-3452","type":"print"},{"value":"1471-5546","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,17]]},"assertion":[{"value":"27 May 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 February 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 March 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors declare they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"The study was reviewed by the Washington State University Office of Research Assurances (Assurance# FWA00002946) and granted an exemption from IRB review consistent with 45 CFR 46.101(b)(2).","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Participants were free to choose whether or not to participate in the survey and consented by their participation.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}],"article-number":"18"}}