{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T10:27:59Z","timestamp":1770978479274,"version":"3.50.1"},"reference-count":54,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2009,4,17]],"date-time":"2009-04-17T00:00:00Z","timestamp":1239926400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2009,12]]},"DOI":"10.1007\/s11192-008-2220-7","type":"journal-article","created":{"date-parts":[[2009,4,17]],"date-time":"2009-04-17T14:56:11Z","timestamp":1239980171000},"page":"789-809","source":"Crossref","is-referenced-by-count":61,"title":["Peer review of grant applications in biology and medicine. Reliability, fairness, and validity"],"prefix":"10.1007","volume":"81","author":[{"given":"Martin","family":"Reinhart","sequence":"first","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2009,4,17]]},"reference":[{"issue":"5","key":"2220_CR1","first-page":"507","volume":"13","author":"P. W. Armstrong","year":"1997","unstructured":"Armstrong, P. W., Caverson, M. M., Adams, L., Taylor, M., Olley, P. M. (1997), Evaluation of the heart and stroke foundation of canada research scholarship program: Research productivity and impact, Canadian Journal of Cardiology, 13(5): 507\u2013516.","journal-title":"Canadian Journal of Cardiology"},{"issue":"5","key":"2220_CR2","doi-asserted-by":"crossref","first-page":"631","DOI":"10.2307\/2095599","volume":"52","author":"V. Bakanic","year":"1987","unstructured":"Bakanic, V., Mcphail, C., Simon, R. J. (1987), The manuscript review and decision-making process, American Sociological Review, 52(5): 631\u2013642.","journal-title":"American Sociological Review"},{"key":"2220_CR3","first-page":"211","volume-title":"Universit\u00e4t auf dem Pr\u00fcfstand. Konzepte und Befunde der Hochschulforschung","author":"L. Bornmann","year":"2003","unstructured":"Bornmann, L., Daniel, H. D. (2003), Begutachtung durch Fachkollegen in der Wissenschaft. Stand der Forschung zur Reliabilit\u00e4t, Fairness und Validit\u00e4t des Peer-Review-Verfahrens. In: Schwarz, S., Teichler, U. (Eds), Universit\u00e4t auf dem Pr\u00fcfstand. Konzepte und Befunde der Hochschulforschung. Campus, Frankfurt am Main, pp. 211\u2013230."},{"issue":"2","key":"2220_CR4","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1007\/s11192-005-0214-2","volume":"63","author":"L. Bornmann","year":"2005","unstructured":"Bornmann, L., Daniel, H. D. (2005), Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees\u2019 decisions, Scientometrics, 63(2): 297\u2013320.","journal-title":"Scientometrics"},{"issue":"1","key":"2220_CR5","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1017\/S0140525X00065717","volume":"14","author":"R. F. Bornstein","year":"1991","unstructured":"Bornstein, R. F. (1991), The predictive validity of peer review: A neglected issue, Behavioral and Brain Sciences, 14(1): 138\u20139.","journal-title":"Behavioral and Brain Sciences"},{"issue":"3","key":"2220_CR6","first-page":"17","volume":"3","author":"T. Brody","year":"2007","unstructured":"Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S., Swan, A. (2007), Incentivizing the open access research web: Publication-archiving, data-archiving and scientometrics, CTWatch Quarterly, 3(3): 17\u201318.","journal-title":"CTWatch Quarterly"},{"key":"2220_CR7","volume-title":"The Consequences of Unfunded NIH Applications for the Investigator and His Research","author":"G. M. Carter","year":"1978","unstructured":"Carter, G. M. (1978), The Consequences of Unfunded NIH Applications for the Investigator and His Research, Rand Corporation, Santa Monica."},{"key":"2220_CR8","volume-title":"Peer Review, Citations, and Biomedical Research Policy: NIH Grants to Medical School Faculty","author":"G. M. Carter","year":"1974","unstructured":"Carter, G. M. (1974), Peer Review, Citations, and Biomedical Research Policy: NIH Grants to Medical School Faculty, Rand Corporation, Santa Monica."},{"key":"2220_CR9","volume-title":"A Citation Study of the NIH Peer Review Process","author":"G. M. Carter","year":"1978","unstructured":"Carter, G. M. (1978), A Citation Study of the NIH Peer Review Process, Rand Corporation, Santa Monica."},{"issue":"2","key":"2220_CR10","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1177\/0013164494054002017","volume":"54","author":"G. B. Chapman","year":"1994","unstructured":"Chapman, G. B., Mccauley, C. (1994), Predictive validity of quality ratings of National Science Foundation graduate fellows, Educational and Psychological Measurement, 54(2): 428\u2013438.","journal-title":"Educational and Psychological Measurement"},{"issue":"1","key":"2220_CR11","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1017\/S0140525X00065675","volume":"14","author":"D. V. Cicchetti","year":"1991","unstructured":"Cicchetti, D. V. (1991A), The reliability of peer review for manuscript and grant submissions: A crossdisciplinary investigation, Behavioral and Brain Sciences, 14(1): 119\u2013135.","journal-title":"Behavioral and Brain Sciences"},{"key":"2220_CR12","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1017\/S0140525X00066024","volume":"14","author":"D. V. Cicchetti","year":"1991","unstructured":"Cicchetti, D. V. (1991B), Reflections from the peer-review mirror, Behavioral and Brain Sciences, 14: 167\u2013186.","journal-title":"Behavioral and Brain Sciences"},{"issue":"2","key":"2220_CR13","first-page":"127","volume":"86","author":"D. V. Cicchetti","year":"1981","unstructured":"Cicchetti, D. V., Sparrow, S. A. (1981), Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86(2): 127\u2013137.","journal-title":"American Journal of Mental Deficiency"},{"issue":"1","key":"2220_CR14","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1023\/A:1005609624130","volume":"47","author":"L. E. Claveria","year":"2000","unstructured":"Claveria, L. E., Guallar, E., Cami, J., Conde, J., Pastor, R., Ricoy, J. R., Rodriguez, E., Ruizpalomo, F., Munoz, E. (2000), Does peer review predict the performance of research projects in health sciences? Scientometrics, 47(1): 11\u201323.","journal-title":"Scientometrics"},{"key":"2220_CR15","volume-title":"Peer Review in the National Science Foundation: Phase Two of a Study","author":"J. R. Cole","year":"1981","unstructured":"Cole, J. R., Cole, S. (1981), Peer Review in the National Science Foundation: Phase Two of a Study, National Academy Press, Washington D.C."},{"key":"2220_CR16","unstructured":"Cole, S., Fiorentine, R. (1991), Discrimination against women in science: The confusion of outcome with process, In: H. Zuckerman, J. R. Cole, J. T. Bruer (Eds), The Outer Circle: Women in the Scientific Community, Yale University Press, 205\u2013226."},{"key":"2220_CR17","volume-title":"Peer Review in the National Science Foundation: Phase One of a Study","author":"S. Cole","year":"1978","unstructured":"Cole, S., Rubin, L., Cole, J. R. (1978), Peer Review in the National Science Foundation: Phase One of a Study, National Academy Press, Washington D.C."},{"key":"2220_CR18","doi-asserted-by":"crossref","DOI":"10.1002\/3527602208","volume-title":"Guardians of Science","author":"H. D. Daniel","year":"1993","unstructured":"Daniel, H. D. (1993), Guardians of Science, VCH, New York."},{"key":"2220_CR19","doi-asserted-by":"crossref","unstructured":"Demicheli, V., Pietrantonj, C. (2004), Peer review for improving the quality of grant applications, The Cochrane Library, 4.","DOI":"10.1002\/14651858.MR000003"},{"key":"2220_CR20","unstructured":"Deutsche Forschungsgemeinschaft (2007), Jahresbericht 2006, Bonn."},{"issue":"5","key":"2220_CR21","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1177\/030631299029005004","volume":"29","author":"L. Dirk","year":"1999","unstructured":"Dirk, L. (1999), A measure of originality: The elements of science, Social Studies of Science, 29(5): 765\u2013776.","journal-title":"Social Studies of Science"},{"key":"2220_CR22","volume-title":"Citation Indexing \u2014 Its Theory and Application in Science, Technology, and Humanities","author":"E. Garfield","year":"1979","unstructured":"Garfield, E. (1979), Citation Indexing \u2014 Its Theory and Application in Science, Technology, and Humanities, John Wiley and Sons, New York."},{"issue":"2","key":"2220_CR23","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1177\/000312240406900203","volume":"69","author":"J. Guetzkow","year":"2004","unstructured":"Guetzkow, J., Lamont, M., Mallard, G. (2004), What is originality in the humanities and the social sciences? American Sociological Review, 69(2): 190\u2013212.","journal-title":"American Sociological Review"},{"key":"2220_CR24","unstructured":"Harnad, S. (1983), Peer Commentary on Peer Review: A Case Study in Scientific Quality Control. Cambridge University Press."},{"issue":"3","key":"2220_CR25","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1177\/016224398501000307","volume":"10","author":"S. Harnad","year":"1985","unstructured":"Harnad, S. (1985), Rational disagreement in peer review, Science, Technology, & Human Values, 10(3): 55\u201362.","journal-title":"Science, Technology, & Human Values"},{"key":"2220_CR26","volume-title":"Begutachtung in der Forschungsf\u00f6rderung \u2014 Die Argumente der Gutachter in der deutschen Forschungsgemeinschaft","author":"I. Hartmann","year":"1990","unstructured":"Hartmann, I. (1990), Begutachtung in der Forschungsf\u00f6rderung \u2014 Die Argumente der Gutachter in der deutschen Forschungsgemeinschaft. Rita G. Fischer, Frankfurt am Main."},{"issue":"5","key":"2220_CR27","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1007\/BF02020704","volume":"19","author":"I. Hartmann","year":"1990","unstructured":"Hartmann, I., Neidhardt, F. (1990), Peer review at the Deutsche Forschungsgemeinschaft, Scientometrics, 19(5): 419\u2013425.","journal-title":"Scientometrics"},{"issue":"1","key":"2220_CR28","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/BF02017752","volume":"27","author":"S. Hemlin","year":"1993","unstructured":"Hemlin, S. (1993), Scientific quality in the eyes of the scientist. A questionnaire study, Scientometrics, 27(1): 3\u201318.","journal-title":"Scientometrics"},{"issue":"1","key":"2220_CR29","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1515\/zfsoz-2004-0104","volume":"33","author":"S. Hirschauer","year":"2004","unstructured":"Hirschauer, S. (2004), Peer Review Verfahren auf dem Pr\u00fcfstand: Zum Soziologiedefizit der Wissenschaftsevaluation, Zeitschrift f\u00fcr Soziologie, 33(1): 62\u201383.","journal-title":"Zeitschrift f\u00fcr Soziologie"},{"key":"2220_CR30","volume-title":"Der SNF im Spiegel der Forschenden","author":"H. Hoffmann","year":"2002","unstructured":"Hoffmann, H., Joye, D., Kuhn, F., Metral, G. (2002), Der SNF im Spiegel der Forschenden. SIDOS, Neuch\u00e2tel."},{"key":"2220_CR31","doi-asserted-by":"crossref","DOI":"10.1002\/0471722146","volume-title":"Applied Logistic Regression","author":"D. W. Hosmer","year":"2000","unstructured":"Hosmer, D. W., Lemeshow, S. (2000), Applied Logistic Regression, John Wiley and sons, New York."},{"issue":"5","key":"2220_CR32","first-page":"256","volume":"20","author":"L. Howard","year":"1999","unstructured":"Howard, L., Wilkinson, G. (1999), Peer review and editorial decision-making, Neuroendocrinology Letters, 20(5): 256\u2013260.","journal-title":"Neuroendocrinology Letters"},{"key":"2220_CR33","volume-title":"Die Herstellung von Gewissheit: Firmenkredite und Risikoanalyse in Mitteleuropa","author":"H. Kalthoff","year":"1999","unstructured":"Kalthoff, H. (1999), Die Herstellung von Gewissheit: Firmenkredite und Risikoanalyse in Mitteleuropa. Frankfurter Institut f\u00fcr Transformationsstudien, Europa-Universit\u00e4t Viadrina."},{"key":"2220_CR34","unstructured":"Kelle, U., Prein, G., Bird, K. (1995), Computer-Aided Qualitative Data Analysis: Theory, Methods and Practice, Sage Publications."},{"key":"2220_CR35","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1017\/CBO9781139171434.009","volume":"4","author":"I. Lakatos","year":"1970","unstructured":"Lakatos, I. (1970), Falsification and the methodology of scientific research programmes, Criticism and the Growth of Knowledge, 4: 91\u2013195.","journal-title":"Criticism and the Growth of Knowledge"},{"issue":"6","key":"2220_CR36","doi-asserted-by":"crossref","first-page":"820","DOI":"10.1177\/030631201031006002","volume":"31","author":"L. Langfeldt","year":"2001","unstructured":"Langfeldt, L. (2001), The decision-making constraints and processes of grant peer review, and their effects on the review outcome, Social Studies of Science, 31(6): 820\u2013841.","journal-title":"Social Studies of Science"},{"key":"2220_CR37","doi-asserted-by":"crossref","unstructured":"Longino, H. E. (2002), The Fate of Knowledge, Princeton University Press.","DOI":"10.1515\/9780691187013"},{"key":"2220_CR38","volume-title":"Computer-Aided Qualitative Data Analysis","author":"M. Lonkila","year":"1995","unstructured":"Lonkila, M. (1995), Grounded theory as an emerging paradigm for CAQDAS. In: U. Kelle, G. Prein, K. Bird (Eds), Computer-Aided Qualitative Data Analysis. London: Sage."},{"key":"2220_CR39","unstructured":"Merton, R. K. (1973), The normative structure of science. In: The Sociology of Science, University of Chicago Press, pp. 267\u2013279."},{"key":"2220_CR40","unstructured":"National Science Foundation (n.d.), US NSF \u2014 Budget, Retrieved August 6, 2008, from http:\/\/www.nsf.gov\/about\/budget\/ ."},{"key":"2220_CR41","doi-asserted-by":"crossref","unstructured":"Neidhardt, F. (1988), Selbststeuerung in der Forschungsf\u00f6rderung: Das Gutachterwesen der DFG, Westdeutscher Verlag.","DOI":"10.1007\/978-3-322-87530-3"},{"issue":"2","key":"2220_CR42","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1629\/09155","volume":"9","author":"C. Oppenheim","year":"1996","unstructured":"Oppenheim, C. (1996), Do citations count? Citation indexing and the Research Assessment Exercise (RAE), Serials: The Journal for the Serials Community, 9(2): 155\u2013161.","journal-title":"Serials: The Journal for the Serials Community"},{"issue":"2","key":"2220_CR43","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1017\/S0140525X00011183","volume":"5","author":"D. P. Peters","year":"1982","unstructured":"Peters, D. P., Ceci, S. J. (1982), Peer-review practices of psychological journals: The fate of published articles, submitted again, Behavioral and Brain Sciences, 5(2): 187\u2013195.","journal-title":"Behavioral and Brain Sciences"},{"key":"2220_CR44","unstructured":"Reinhart, M. (forthcoming), Peer review and quality criteria in science funding: access point versus boundary organization."},{"key":"2220_CR45","unstructured":"Reinhart, M. (2006), Peer Review, Retrieved June 26, 2008, from http:\/\/www.forschungsinfo.de\/iq\/agora\/Peer%20Review\/peer_review.html."},{"key":"2220_CR46","first-page":"27","volume":"1","author":"M. Reinahrt","year":"2006","unstructured":"Reinahrt, M., Sirtes, D. (2006), Wieviel Intransparenz ist f\u00fcr Entscheidungen \u00fcber exzellente Wissenschaft notwendig? IfQ Working Paper, 1: 27\u201336.","journal-title":"IfQ Working Paper"},{"key":"2220_CR47","unstructured":"Schweizerischer Nationalfonds. (2002), Stiftungsurkunde \/ Statuten, Retrieved August 14, 2007, from http:\/\/www.snf.ch\/SiteCollectionDocuments\/por_org_statuten_d.pdf ."},{"key":"2220_CR48","unstructured":"Schweizerischer Nationalfonds. (2007), Jahresbericht 2006, Bern."},{"key":"2220_CR49","volume-title":"The Correlation Between Rae Ratings and Citation Counts In Psychology","author":"A. Smith","year":"2002","unstructured":"Smith, A., Eysenck, M. (2002), The Correlation Between Rae Ratings and Citation Counts In Psychology, Department of Psychology, Royal Holloway, University of London, UK."},{"key":"2220_CR50","doi-asserted-by":"crossref","unstructured":"Solomon, M. (2001), Social Empiricism, MIT Press.","DOI":"10.7551\/mitpress\/6296.001.0001"},{"key":"2220_CR51","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1017\/S0140525X00065985","volume":"14","author":"L. J. Stricker","year":"1991","unstructured":"Stricker, L. J. (1991), Disagreement among journal reviewers: No cause for undue alarm, Behavioral and Brain Sciences, 14: 163\u2013164.","journal-title":"Behavioral and Brain Sciences"},{"key":"2220_CR52","first-page":"288","volume":"24","author":"T. Strulik","year":"2007","unstructured":"Strulik, T. (2007), Evaluationen in der Wirtschaft \u2014 Rating-Agenturen und das Management des Beobachtetwerdens, Leviathan Sonderheft, 24: 288\u2013314.","journal-title":"Leviathan Sonderheft"},{"key":"2220_CR53","unstructured":"Weller, A. C. (2001), Editorial Peer Review: Its Strengths and Weaknesses, Information Today."},{"key":"2220_CR54","first-page":"14","volume-title":"Peer Review in Health Sciences","author":"F. Wood","year":"1999","unstructured":"Wood, F., Wessely, S. (1999), Peer review of grant applications: A systematic review. In: F. Godlee, T. Jefferson (Eds), Peer Review in Health Sciences. BMJ Books, London, 14\u201331."}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-008-2220-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11192-008-2220-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-008-2220-7","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,6,1]],"date-time":"2019-06-01T09:26:55Z","timestamp":1559381215000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11192-008-2220-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,4,17]]},"references-count":54,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2009,12]]}},"alternative-id":["2220"],"URL":"https:\/\/doi.org\/10.1007\/s11192-008-2220-7","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2009,4,17]]}}}