{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T04:10:58Z","timestamp":1768277458458,"version":"3.49.0"},"reference-count":72,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,7,11]],"date-time":"2019-07-11T00:00:00Z","timestamp":1562803200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"},{"start":{"date-parts":[[2019,7,11]],"date-time":"2019-07-11T00:00:00Z","timestamp":1562803200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2019,10]]},"DOI":"10.1007\/s11192-019-03184-y","type":"journal-article","created":{"date-parts":[[2019,7,11]],"date-time":"2019-07-11T13:14:11Z","timestamp":1562850851000},"page":"537-554","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":41,"title":["Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?"],"prefix":"10.1007","volume":"121","author":[{"given":"Giovanni","family":"Abramo","sequence":"first","affiliation":[]},{"given":"Ciriaco Andrea","family":"D\u2019Angelo","sequence":"additional","affiliation":[]},{"given":"Emanuela","family":"Reale","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,7,11]]},"reference":[{"issue":"3","key":"3184_CR1","doi-asserted-by":"publisher","first-page":"590","DOI":"10.1016\/j.joi.2018.05.001","volume":"12","author":"G Abramo","year":"2018","unstructured":"Abramo, G. (2018). Revisiting the scientometric conceptualization of impact and its measurement. Journal of Informetrics, 12(3), 590\u2013597.","journal-title":"Journal of Informetrics"},{"issue":"4","key":"3184_CR2","doi-asserted-by":"publisher","first-page":"659","DOI":"10.1016\/j.joi.2011.06.004","volume":"5","author":"G Abramo","year":"2011","unstructured":"Abramo, G., Cicero, T., & D\u2019Angelo, C. A. (2011a). Assessing the varying level of impact measurement accuracy as a function of the citation window length. Journal of Informetrics, 5(4), 659\u2013667.","journal-title":"Journal of Informetrics"},{"issue":"2","key":"3184_CR3","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.joi.2011.11.007","volume":"6","author":"G Abramo","year":"2012","unstructured":"Abramo, G., Cicero, T., & D\u2019Angelo, C. A. (2012). The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems. Journal of Informetrics, 6(2), 155\u2013168.","journal-title":"Journal of Informetrics"},{"issue":"1","key":"3184_CR4","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1007\/s11192-012-0875-6","volume":"95","author":"G Abramo","year":"2013","unstructured":"Abramo, G., Cicero, T., & D\u2019Angelo, C. A. (2013a). National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case. Scientometrics, 95(1), 311\u2013324.","journal-title":"Scientometrics"},{"issue":"3","key":"3184_CR5","doi-asserted-by":"publisher","first-page":"2053","DOI":"10.1007\/s11192-016-2153-5","volume":"109","author":"G Abramo","year":"2016","unstructured":"Abramo, G., & D\u2019Angelo, C. A. (2016). Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014). Scientometrics, 109(3), 2053\u20132065.","journal-title":"Scientometrics"},{"issue":"3","key":"3184_CR6","doi-asserted-by":"publisher","first-page":"929","DOI":"10.1007\/s11192-011-0459-x","volume":"89","author":"G Abramo","year":"2011","unstructured":"Abramo, G., D\u2019Angelo, C. A., & Di Costa, F. (2011b). National research assessment exercises: a comparison of peer review and bibliometrics rankings. Scientometrics, 89(3), 929\u2013941.","journal-title":"Scientometrics"},{"issue":"1","key":"3184_CR7","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1016\/j.joi.2018.11.003","volume":"13","author":"G Abramo","year":"2019","unstructured":"Abramo, G., D\u2019Angelo, C. A., & Felici, G. (2019). Predicting long-term publication impact through a combination of early citations and journal impact factor. Journal of Informetrics, 13(1), 32\u201349.","journal-title":"Journal of Informetrics"},{"issue":"6","key":"3184_CR8","first-page":"761","volume":"42","author":"G Abramo","year":"2015","unstructured":"Abramo, G., D\u2019Angelo, C. A., & Rosati, F. (2015). The determinants of academic career advancement: evidence from Italy. Science and Public Policy, 42(6), 761\u2013774.","journal-title":"Science and Public Policy"},{"issue":"1","key":"3184_CR9","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1093\/reseval\/rvs031","volume":"22","author":"G Abramo","year":"2013","unstructured":"Abramo, G., D\u2019Angelo, C. A., & Viel, F. (2013b). Selecting competent referees to assess research projects proposals: a study of referees\u2019 registers. Research Evaluation, 22(1), 41\u201351.","journal-title":"Research Evaluation"},{"key":"3184_CR10","doi-asserted-by":"crossref","unstructured":"Aksnes, D.W., Langfeldt, L., & Wouters, P. (2019). Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open, January\u2013March, 1\u201317.","DOI":"10.1177\/2158244019829575"},{"issue":"1","key":"3184_CR11","doi-asserted-by":"publisher","first-page":"33","DOI":"10.3152\/147154404781776563","volume":"13","author":"DW Aksnes","year":"2004","unstructured":"Aksnes, D. W., & Taxt, R. E. (2004). Peer reviews and bibliometric indicators: A comparative study at Norvegian University. Research Evaluation, 13(1), 33\u201341.","journal-title":"Research Evaluation"},{"key":"3184_CR12","unstructured":"Alf\u00f2, M., Benedetto, S., Malgarini, M., & Scipione, S. (2017). On the use of bibliometric information for assessing articles quality: an analysis based on the third Italian research evaluation exercise. In 2017 STI conference, Paris."},{"issue":"6","key":"3184_CR13","doi-asserted-by":"publisher","first-page":"e5910","DOI":"10.1371\/journal.pone.0005910","volume":"4","author":"L Allen","year":"2009","unstructured":"Allen, L., Jones, C., Dolby, K., Lynn, D., & Walport, M. (2009). Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS ONE, 4(6), e5910.","journal-title":"PLoS ONE"},{"issue":"3","key":"3184_CR14","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1093\/reseval\/rvv008","volume":"24","author":"A Ancaiani","year":"2015","unstructured":"Ancaiani, A., Anfossi, A. F., Barbara, A., Benedetto, S., Blasi, B., Carletti, V., et al. (2015). Evaluating scientific research in Italy: The 2004\u201310 research evaluation exercise. Research Evaluation, 24(3), 242\u2013255.","journal-title":"Research Evaluation"},{"key":"3184_CR15","unstructured":"ANVUR. (2013). Valutazione della qualit\u00e0 della ricerca 2004\u20132010. Rapporto finale. http:\/\/www.anvur.it\/rapporto\/ . Last Accessed 12 June 2019."},{"key":"3184_CR16","unstructured":"Baccini, A., Barabesi, L., & De Nicolao, G. (2018). The Holy Grail and the bad sampling: a test for the homogeneity of missing proportions for evaluating the agreement between peer review and bibliometrics in the Italian research assessment exercises. arXiv:1810.12430v1 ."},{"issue":"3","key":"3184_CR17","doi-asserted-by":"publisher","first-page":"1651","DOI":"10.1007\/s11192-016-1929-y","volume":"108","author":"A Baccini","year":"2016","unstructured":"Baccini, A., & De Nicolao, G. (2016). Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise. Scientometrics, 108(3), 1651\u20131671.","journal-title":"Scientometrics"},{"issue":"2","key":"3184_CR18","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1016\/j.respol.2014.08.004","volume":"44","author":"G Bertocchi","year":"2015","unstructured":"Bertocchi, G., Gambardella, A., Jappelli, T., Nappi, C. A., & Peracchi, F. (2015). Bibliometric evaluation versus informed peer review: Evidence from Italy. Research Policy, 44(2), 451\u2013466.","journal-title":"Research Policy"},{"key":"3184_CR19","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1002\/aris.2011.1440450112","volume":"45","author":"L Bornmann","year":"2011","unstructured":"Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199\u2013245.","journal-title":"Annual Review of Information Science and Technology"},{"issue":"3","key":"3184_CR20","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1007\/s11192-005-0281-4","volume":"65","author":"L Bornmann","year":"2005","unstructured":"Bornmann, L., & Daniel, H.-D. (2005). Does the h-index for ranking of scientists really work? Scientometrics, 65(3), 391\u2013392.","journal-title":"Scientometrics"},{"issue":"1","key":"3184_CR21","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1108\/00220410810844150","volume":"64","author":"L Bornmann","year":"2008","unstructured":"Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45\u201380.","journal-title":"Journal of Documentation"},{"issue":"2","key":"3184_CR22","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1016\/j.joi.2012.12.003","volume":"7","author":"L Bornmann","year":"2013","unstructured":"Bornmann, L., & Leydesdorff, L. (2013). The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000. Journal of Informetrics, 7(2), 286\u2013291.","journal-title":"Journal of Informetrics"},{"issue":"6","key":"3184_CR23","doi-asserted-by":"publisher","first-page":"e68258","DOI":"10.1371\/journal.pone.0068258","volume":"8","author":"\u00c1 Cabezas-Clavijo","year":"2013","unstructured":"Cabezas-Clavijo, \u00c1., Robinson-Garc\u00eda, N., Escabias, M., & Jim\u00e9nez-Contreras, E. (2013). Reviewers\u2019 ratings and bibliometric indicators: Hand in hand when assessing over research proposals? PLoS ONE, 8(6), e68258.","journal-title":"PLoS ONE"},{"key":"3184_CR24","volume-title":"The manufacture of knowledge: An essay on the constructivist and contextual nature of science","author":"KK Cetina","year":"1981","unstructured":"Cetina, K. K. (1981). The manufacture of knowledge: An essay on the constructivist and contextual nature of science. New York: Pergamon Press."},{"issue":"1","key":"3184_CR25","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37\u201346.","journal-title":"Educational and Psychological Measurement"},{"issue":"4","key":"3184_CR26","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1037\/h0026256","volume":"70","author":"J Cohen","year":"1968","unstructured":"Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213\u2013220.","journal-title":"Psychological Bulletin"},{"key":"3184_CR27","volume-title":"Social stratification in science","author":"JR Cole","year":"1973","unstructured":"Cole, J. R., & Cole, S. (1973). Social stratification in science. Chicago: University of Chicago Press."},{"issue":"4","key":"3184_CR28","first-page":"5","volume":"3","author":"F Cuccurullo","year":"2006","unstructured":"Cuccurullo, F. (2006). La valutazione triennale della ricerca\u2013VTR del CIVR. Analysis, 3(4), 5\u20137.","journal-title":"Analysis"},{"key":"3184_CR29","doi-asserted-by":"publisher","DOI":"10.1002\/0471445428","volume-title":"Statistical methods for rates and proportions","author":"JL Fleiss","year":"2003","unstructured":"Fleiss, J. L., Levin, B., & Myunghee, C. P. (2003). Statistical methods for rates and proportions. Hoboken, NJ: Wiley."},{"issue":"2","key":"3184_CR30","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1016\/j.joi.2010.12.002","volume":"5","author":"M Franceschet","year":"2011","unstructured":"Franceschet, M., & Costantini, A. (2011). The first Italian research assessment exercise: A bibliometric perspective. Journal of Informetrics, 5(2), 275\u2013291.","journal-title":"Journal of Informetrics"},{"key":"3184_CR31","volume-title":"Citation indexing-its theory and application in science, technology, and humanities","author":"E Garfield","year":"1979","unstructured":"Garfield, E. (1979). Citation indexing-its theory and application in science, technology, and humanities. New York, NY: Wiley."},{"key":"3184_CR32","first-page":"5","volume":"21","author":"E Garfield","year":"1980","unstructured":"Garfield, E. (1980). Premature discovery or delayed recognition: Why? Current Contents, 21, 5\u201310.","journal-title":"Current Contents"},{"key":"3184_CR33","volume-title":"Proceedings of WIS fourth international conference on webometrics, informetrics and scientometrics & ninth COLLNET meeting","author":"W Gl\u00e4nzel","year":"2008","unstructured":"Gl\u00e4nzel, W. (2008). Seven myths in bibliometrics. About facts and fiction in quantitative science studies. In H. Kretschmer & F. Havemann (Eds.), Proceedings of WIS fourth international conference on webometrics, informetrics and scientometrics & ninth COLLNET meeting. Berlin: Institute for Library and Information Science."},{"issue":"1","key":"3184_CR34","doi-asserted-by":"publisher","first-page":"103","DOI":"10.3354\/esep00088","volume":"8","author":"S Harnad","year":"2008","unstructured":"Harnad, S. (2008). Validating research performance metrics against peer rankings. Ethics in Science and Environmental Politics, 8(1), 103\u2013107.","journal-title":"Ethics in Science and Environmental Politics"},{"issue":"1","key":"3184_CR35","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1007\/s11192-018-2669-y","volume":"115","author":"D Herrmannova","year":"2018","unstructured":"Herrmannova, D., Patton, R., Knoth, P., & Stahl, C. (2018). Do citations and readership identify seminal publications? Scientometrics, 115(1), 239\u2013262.","journal-title":"Scientometrics"},{"issue":"10","key":"3184_CR36","doi-asserted-by":"publisher","first-page":"1438","DOI":"10.1001\/jama.1990.03440100162024","volume":"263","author":"DF Horrobin","year":"1990","unstructured":"Horrobin, D. F. (1990). The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association, 263(10), 1438\u20131441.","journal-title":"Journal of the American Medical Association"},{"issue":"24","key":"3184_CR37","doi-asserted-by":"publisher","first-page":"7426","DOI":"10.1073\/pnas.1424329112","volume":"112","author":"Q Ke","year":"2015","unstructured":"Ke, Q., Ferrara, E., Radicchi, F., & Flammini, A. (2015). Defining and identifying sleeping beauties in science. Proceedings of the National Academy of Sciences, 112(24), 7426\u20137431.","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"48","key":"3184_CR38","first-page":"11","volume":"5","author":"G Kreiman","year":"2011","unstructured":"Kreiman, G., & Maunsell, J. H. R. (2011). Nine criteria for a measure of scientific output. Frontiers in Computational Neuroscience, 5(48), 11.","journal-title":"Frontiers in Computational Neuroscience"},{"issue":"1","key":"3184_CR39","doi-asserted-by":"publisher","first-page":"282","DOI":"10.1016\/j.joi.2017.01.001","volume":"11","author":"E Kulczycki","year":"2017","unstructured":"Kulczycki, E., Korze\u0144, M., & Korytkowski, P. (2017). Toward an excellence-based research funding system: Evidence from Poland. Journal of Informetrics, 11(1), 282\u2013298.","journal-title":"Journal of Informetrics"},{"key":"3184_CR40","volume-title":"Science in action: How to follow scientists and engineers through society","author":"B Latour","year":"1987","unstructured":"Latour, B. (1987). Science in action: How to follow scientists and engineers through society. Cambridge, MA: Harvard University Press."},{"issue":"1","key":"3184_CR41","first-page":"1","volume":"1","author":"L Leydesdorff","year":"2016","unstructured":"Leydesdorff, L., Bornmann, L., Comins, J. A., & Milojevi\u0107, S. (2016). Citations: Indicators of quality? The impact fallacy. Frontiers in Research Metrics and Analytics, 1(1), 1\u201315.","journal-title":"Frontiers in Research Metrics and Analytics"},{"issue":"1","key":"3184_CR42","doi-asserted-by":"publisher","first-page":"255","DOI":"10.2307\/2532051","volume":"45","author":"LI-K Lin","year":"1989","unstructured":"Lin, L. I.-K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255\u2013268.","journal-title":"Biometrics"},{"issue":"1","key":"3184_CR43","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1111\/j.0006-341X.2000.00324.x","volume":"56","author":"LI-K Lin","year":"2000","unstructured":"Lin, L. I.-K. (2000). Erratum: A note on the concordance correlation coefficient (biometrics (1989) (214)). Biometrics, 56(1), 324\u2013325.","journal-title":"Biometrics"},{"key":"3184_CR44","doi-asserted-by":"publisher","unstructured":"Mahdi, S., D\u2019Este, P., & Neely, A. (2008). Citation counts: are they good predictors of RAE scores? Technical Report February. Advanced Institute of Management Research. https:\/\/doi.org\/10.2139\/ssrn.1154053 .","DOI":"10.2139\/ssrn.1154053"},{"issue":"2","key":"3184_CR45","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"BR Martin","year":"1983","unstructured":"Martin, B. R., & Irvine, J. (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy. Research Policy, 12(2), 61\u201390.","journal-title":"Research Policy"},{"key":"3184_CR46","unstructured":"McBride, G. B. (2005). A proposal for strength-of-agreement criteria for lins concordance correlation coefficient. NIWA Client Report, HAM2005-062."},{"issue":"2","key":"3184_CR47","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1002\/(SICI)1097-4571(2000)51:2<123::AID-ASI4>3.0.CO;2-N","volume":"51","author":"LI Meho","year":"2000","unstructured":"Meho, L. I., & Sonnenwald, D. H. (2000). Citation ranking versus peer evaluation of senior faculty research performance: a case study of Kurdish Scholarship. Journal of the American Society for Information Science, 51(2), 123\u2013138.","journal-title":"Journal of the American Society for Information Science"},{"key":"3184_CR48","first-page":"286","volume-title":"The sociology of science: Theoretical and empirical investigations","author":"RK Merton","year":"1973","unstructured":"Merton, R. K. (1973). Priorities in scientific discovery. In R. K. Merton (Ed.), The sociology of science: Theoretical and empirical investigations (pp. 286\u2013324). Chicago: University of Chicago Press."},{"issue":"1","key":"3184_CR49","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ejor.2015.04.002","volume":"246","author":"J Mingers","year":"2015","unstructured":"Mingers, J., & Leydesdorff, L. (2015). A review of theory and practice in scientometrics. European Journal of Operational Research, 246(1), 1\u201319.","journal-title":"European Journal of Operational Research"},{"issue":"1","key":"3184_CR50","first-page":"7","volume":"5","author":"H Moxam","year":"1992","unstructured":"Moxam, H., & Anderson, J. (1992a). Peer review. A view from the inside. Science and Technology Policy, 5(1), 7\u201315.","journal-title":"Science and Technology Policy"},{"issue":"1","key":"3184_CR51","first-page":"7","volume":"5","author":"H Moxam","year":"1992","unstructured":"Moxam, H., & Anderson, J. (1992b). Peer review. A view from the inside. Science and Technology Policy, 5(1), 7\u201315.","journal-title":"Science and Technology Policy"},{"issue":"3","key":"3184_CR52","doi-asserted-by":"publisher","first-page":"1013","DOI":"10.1007\/s11192-015-1567-9","volume":"104","author":"O Mryglod","year":"2015","unstructured":"Mryglod, O., Kenna, R., Holovatch, Y., & Berche, B. (2015). Predicting results of the research excellence framework using departmental h-index: revisited. Scientometrics, 104(3), 1013\u20131017.","journal-title":"Scientometrics"},{"issue":"5","key":"3184_CR53","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1108\/EUM0000000007207","volume":"53","author":"C Oppenheim","year":"1997","unstructured":"Oppenheim, C. (1997). The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology. Journal of Documentation, 53(5), 477\u2013487.","journal-title":"Journal of Documentation"},{"issue":"6","key":"3184_CR54","first-page":"709","volume":"56","author":"C Oppenheim","year":"2003","unstructured":"Oppenheim, C., & Norris, M. (2003). Citation counts and the research assessment exercise V: Archaeology and the 2001 RAE. Journal of Documentation, 56(6), 709\u2013730.","journal-title":"Journal of Documentation"},{"issue":"1","key":"3184_CR55","first-page":"1","volume":"57","author":"DA Pendlebury","year":"2009","unstructured":"Pendlebury, D. A. (2009). The use and misuse of journal metrics and other citation indicators. Scientometrics, 57(1), 1\u201311.","journal-title":"Scientometrics"},{"issue":"2","key":"3184_CR56","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1023\/A:1016070029935","volume":"54","author":"P Pichappan","year":"2002","unstructured":"Pichappan, P., & Sarasvady, S. (2002). The other side of the coin: The intricacies of author self-citations. Scientometrics, 54(2), 285\u2013290.","journal-title":"Scientometrics"},{"key":"3184_CR57","doi-asserted-by":"publisher","unstructured":"Pride, D., & Knoth, P. (2018). Peer review and citation data in predicting university rankings, a large-scale analysis. In International conference on theory and practice of digital libraries, TPDL 2018: Digital libraries for open knowledge, 195\u2013207. https:\/\/doi.org\/10.1007\/978-3-030-00066-0_17 . Last Accessed 12 June 2019.","DOI":"10.1007\/978-3-030-00066-0_17"},{"issue":"3","key":"3184_CR58","doi-asserted-by":"publisher","first-page":"216","DOI":"10.3152\/095820207X227501","volume":"16","author":"E Reale","year":"2007","unstructured":"Reale, E., Barbara, A., & Costantini, A. (2007). Peer review for the evaluation of academic research: Lessons from the Italian experience. Research Evaluation, 16(3), 216\u2013228.","journal-title":"Research Evaluation"},{"issue":"3","key":"3184_CR59","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1093\/reseval\/rvx019","volume":"26","author":"E Reale","year":"2017","unstructured":"Reale, E., & Zinilli, A. (2017). Evaluation for the allocation of university research project funding: Can rules improve the peer review? Research Evaluation, 26(3), 190\u2013198.","journal-title":"Research Evaluation"},{"issue":"1","key":"3184_CR60","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1016\/S0048-7333(98)00026-2","volume":"27","author":"EJ Rinia","year":"1998","unstructured":"Rinia, E. J., van Leeuwen, T., van Vuren, H. G., & van Raan, A. F. J. (1998). Comparative analysis of a set of bibliometric indicators and central peer-review criteria: Evaluation of condensed matter physics in the Netherlands. Research Policy, 27(1), 95\u2013107.","journal-title":"Research Policy"},{"key":"3184_CR61","doi-asserted-by":"publisher","DOI":"10.1201\/9781420036268","volume-title":"Handbook of parametric and nonparametric statistical procedures","author":"DJ Sheskin","year":"2003","unstructured":"Sheskin, D. J. (2003). Handbook of parametric and nonparametric statistical procedures. London: Chapman & Hall."},{"key":"3184_CR62","doi-asserted-by":"crossref","DOI":"10.1093\/wentk\/9780190640118.001.0001","volume-title":"Measuring research","author":"CR Sugimoto","year":"2018","unstructured":"Sugimoto, C. R., & Larivi\u00e8re, V. (2018). Measuring research. Oxford: Oxford University Press."},{"issue":"2","key":"3184_CR63","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1111\/j.1467-8551.2010.00722.x","volume":"22","author":"J Taylor","year":"2011","unstructured":"Taylor, J. (2011a). The assessment of research quality in UK universities: Peer review or metrics? British Journal of Management, 22(2), 202\u2013217.","journal-title":"British Journal of Management"},{"issue":"2","key":"3184_CR64","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1111\/j.1467-8551.2010.00722.x","volume":"22","author":"J Taylor","year":"2011","unstructured":"Taylor, J. (2011b). The assessment of research quality in UK universities: Peer review or metrics? British Journal of Management, 22(2), 202\u2013217.","journal-title":"British Journal of Management"},{"issue":"3","key":"3184_CR65","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1007\/BF02459050","volume":"41","author":"PR Thomas","year":"1998","unstructured":"Thomas, P. R., & Watkins, D. S. (1998). Institutional research rankings via bibliometric analysis and direct peer-review: A comparative case study with policy implications. Scientometrics, 41(3), 335\u2013355.","journal-title":"Scientometrics"},{"key":"3184_CR66","doi-asserted-by":"publisher","DOI":"10.1057\/s41599-019-0233-x","volume-title":"Systematic analysis of agreement between metrics and peer review in the UK REF","author":"VA Traag","year":"2019","unstructured":"Traag, V. A., & Waltman, L. (2019). Systematic analysis of agreement between metrics and peer review in the UK REF. London: Palgrave Communications."},{"issue":"3","key":"3184_CR67","first-page":"461","volume":"59","author":"AFJ van Raan","year":"2004","unstructured":"van Raan, A. F. J. (2004). Sleeping beauties in science. Scientometrics, 59(3), 461\u2013466.","journal-title":"Scientometrics"},{"issue":"3","key":"3184_CR68","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1556\/Scient.67.2006.3.10","volume":"67","author":"AFJ van Raan","year":"2006","unstructured":"van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491\u2013502.","journal-title":"Scientometrics"},{"issue":"3","key":"3184_CR69","doi-asserted-by":"publisher","first-page":"560","DOI":"10.1002\/asi.22981","volume":"65","author":"ES Vieira","year":"2014","unstructured":"Vieira, E. S., Cabral, J. A. S., & Gomes, J. A. N. F. (2014a). Definition of a model based on bibliometric indicators for assessing applicants to academic positions. Journal of the Association for Information Science and Technology, 65(3), 560\u2013577.","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"2","key":"3184_CR70","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1016\/j.joi.2014.01.012","volume":"8","author":"ES Vieira","year":"2014","unstructured":"Vieira, E. S., Cabral, J. A. S., & Gomes, J. A. N. F. (2014b). How good is a model based on bibliometric indicators in predicting the final decisions made by peers? Journal of Informetrics, 8(2), 390\u2013405.","journal-title":"Journal of Informetrics"},{"issue":"3","key":"3184_CR71","doi-asserted-by":"publisher","first-page":"246","DOI":"10.1093\/reseval\/rvy009","volume":"27","author":"ES Vieira","year":"2018","unstructured":"Vieira, E. S., & Gomes, J. A. N. F. (2018). The peer-review process: The most valued dimensions according to the researcher\u2019s scientific career. Research Evaluation, 27(3), 246\u2013261.","journal-title":"Research Evaluation"},{"key":"3184_CR72","doi-asserted-by":"publisher","DOI":"10.4135\/9781473978782","volume-title":"The Metric Tide: Report of the independent review of the role of metrics in research assessment and management","author":"J Wilsdon","year":"2015","unstructured":"Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., et al. (2015). The Metric Tide: Report of the independent review of the role of metrics in research assessment and management. Bristol: HEFCE."}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-019-03184-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11192-019-03184-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-019-03184-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,11]],"date-time":"2021-01-11T19:52:07Z","timestamp":1610394727000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11192-019-03184-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,11]]},"references-count":72,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,10]]}},"alternative-id":["3184"],"URL":"https:\/\/doi.org\/10.1007\/s11192-019-03184-y","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,11]]},"assertion":[{"value":"29 April 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 July 2019","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}