{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,26]],"date-time":"2026-04-26T06:57:03Z","timestamp":1777186623824,"version":"3.51.4"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,2,3]],"date-time":"2020-02-03T00:00:00Z","timestamp":1580688000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,2,3]],"date-time":"2020-02-03T00:00:00Z","timestamp":1580688000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000921","name":"European Cooperation in Science and Technology","doi-asserted-by":"publisher","award":["Cost Action TD1306"],"award-info":[{"award-number":["Cost Action TD1306"]}],"id":[{"id":"10.13039\/501100000921","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2020,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The purpose of this paper is to analyze the causes and effects of arbitrariness in the peer review process. This paper focuses on two main reasons for the arbitrariness in peer review. The first is that referees are not homogenous and display <jats:italic>homophily<\/jats:italic> in their taste and perception of innovative ideas. The second element is that reviewers are different in the time they allocate for peer review. Our model replicates the NIPS experiment of 2014, showing that the ratings of peer review are not robust, and that altering reviewers leads to a dramatic impact on the ranking of the papers. This paper also shows that innovative works are not highly ranked in the existing peer review process, and in consequence are often rejected.<\/jats:p>","DOI":"10.1007\/s11192-020-03348-1","type":"journal-article","created":{"date-parts":[[2020,2,3]],"date-time":"2020-02-03T17:04:25Z","timestamp":1580749465000},"page":"393-411","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":65,"title":["Arbitrariness in the peer review process"],"prefix":"10.1007","volume":"123","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7954-8110","authenticated-orcid":false,"given":"Elise S.","family":"Brezis","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4925-9131","authenticated-orcid":false,"given":"Aliaksandr","family":"Birukou","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,2,3]]},"reference":[{"key":"3348_CR1","doi-asserted-by":"publisher","unstructured":"Avin, S. (2015). Breaking the grant cycle: On the rational allocation of public resources to scientific research projects. Ph.D. Thesis, University of Cambridge. https:\/\/doi.org\/10.17863\/CAM.16172.","DOI":"10.17863\/CAM.16172"},{"issue":"1","key":"3348_CR2","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1007\/s11192-016-2176-y","volume":"110","author":"C Bartneck","year":"2017","unstructured":"Bartneck, C. (2017). Reviewers\u2019 scores do not predict impact: bibliometric analysis of the proceedings of the human\u2013robot interaction conference. Scientometrics,110(1), 179\u2013194. https:\/\/doi.org\/10.1007\/s11192-016-2176-y.","journal-title":"Scientometrics"},{"key":"3348_CR3","series-title":"Advances in group decision and negotiation","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-007-5574-1_1","volume-title":"Models for intercultural collaboration and negotiation","author":"A Birukou","year":"2013","unstructured":"Birukou, A., Blanzieri, E., Giorgini, P., & Giunchiglia, F. (2013). A formal definition of culture. In K. Sycara, M. Gelfand, & A. Abbe (Eds.), Models for intercultural collaboration and negotiation (Vol. 6)., Advances in group decision and negotiation Dordrecht: Springer."},{"key":"3348_CR4","doi-asserted-by":"publisher","first-page":"56","DOI":"10.3389\/fncom.2011.00056","volume":"5","author":"A Birukou","year":"2011","unstructured":"Birukou, A., Wakeling, J., Bartolini, C., Casati, F., Marchese, M., Mirylenka, K., et al. (2011). Alternatives to peer review: Novel approaches for research evaluation. Frontiers in Computational Neuroscience,5, 56.","journal-title":"Frontiers in Computational Neuroscience"},{"issue":"10","key":"3348_CR5","doi-asserted-by":"publisher","first-page":"2765","DOI":"10.1287\/mnsc.2015.2285","volume":"62","author":"K Boudreau","year":"2016","unstructured":"Boudreau, K., Eva, J., Guinan, C., Lakhani, K. R., & Riedl, C. (2016). Looking across and looking beyond the knowledge frontier: Intellectual distance, novelty, and resource allocation in science. Management Science,62(10), 2765\u20132783. https:\/\/doi.org\/10.1287\/mnsc.2015.2285.","journal-title":"Management Science"},{"issue":"9","key":"3348_CR6","doi-asserted-by":"publisher","first-page":"691","DOI":"10.3152\/030234207X265394","volume":"34","author":"ES Brezis","year":"2007","unstructured":"Brezis, E. S. (2007). Focal randomization: An optimal mechanism for the evaluation of R&D projects. Science and Public Policy,34(9), 691\u2013698.","journal-title":"Science and Public Policy"},{"key":"3348_CR7","unstructured":"Francois, O. (2015). Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment."},{"key":"3348_CR8","first-page":"3","volume":"9","author":"E Garfield","year":"1986","unstructured":"Garfield, E. (1986). Refereeing and peer review: Opinion and conjecture on the effectiveness of refereeing. Essays of an Information Scientists,9, 3\u201311.","journal-title":"Essays of an Information Scientists"},{"key":"3348_CR9","first-page":"21","volume":"10","author":"E Garfield","year":"1987","unstructured":"Garfield, E. (1987). Refereeing and peer review: How the peer review of research grant proposals works and what scientists say about it. Essays of an Information Scientists,10, 21\u201326.","journal-title":"Essays of an Information Scientists"},{"key":"3348_CR10","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3000065","author":"K Gross","year":"2019","unstructured":"Gross, K., & Bergstrom, C. T. (2019). Contest models highlight inherent inefficiencies of scientific funding competitions. PLoS Biology. https:\/\/doi.org\/10.1371\/journal.pbio.3000065.","journal-title":"PLoS Biology"},{"issue":"2","key":"3348_CR11","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1038\/scientificamerican0275-88","volume":"232","author":"A Hallam","year":"1975","unstructured":"Hallam, A. (1975). Alfred Wegener and the hypothesis of continental drift. Scientific American,232(2), 88\u201397.","journal-title":"Scientific American"},{"key":"3348_CR12","doi-asserted-by":"publisher","first-page":"609","DOI":"10.1038\/d41586-018-07245-9","volume":"563","author":"D Heaven","year":"2018","unstructured":"Heaven, D. (2018). The age of AI peer reviews. Nature,563, 609\u2013610. https:\/\/doi.org\/10.1038\/d41586-018-07245-9.","journal-title":"Nature"},{"key":"3348_CR13","doi-asserted-by":"crossref","unstructured":"Hirshman, B. R., Birukou, A., Martin, M. A., Bigrigg, M. W., & Carley, K. M. (2008). The impact of educational interventions on real and stylized cities. Technical Report CMU-ISR-08-114, Carnegie Mellon University.","DOI":"10.2139\/ssrn.2728450"},{"issue":"1","key":"3348_CR14","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1007\/s11192-017-2375-1","volume":"113","author":"M Kovanis","year":"2017","unstructured":"Kovanis, M., Trinquart, L., Ravaud, P., & Porcher, R. (2017). Evaluating alternative systems of peer review: A large-scale agent-based modelling approach to scientific publication. Scientometrics,113(1), 651\u2013671.","journal-title":"Scientometrics"},{"key":"3348_CR15","doi-asserted-by":"publisher","first-page":"1936","DOI":"10.1016\/j.respol.2016.07.004","volume":"45","author":"Jonathan Linton","year":"2016","unstructured":"Linton, Jonathan. (2016). Improving the peer review process: Capturing more information and enabling high-risk high-return research. Research Policy,45, 1936\u20131938.","journal-title":"Research Policy"},{"issue":"2","key":"3348_CR16","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1093\/reseval\/rvs001","volume":"21","author":"Tertu Luukkonen","year":"2012","unstructured":"Luukkonen, Tertu. (2012). Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation,21(2), 48\u201360. https:\/\/doi.org\/10.1093\/reseval\/rvs001.","journal-title":"Research Evaluation"},{"key":"3348_CR17","unstructured":"Mali\u010dki, M., Mihajlov, M., Birukou, A., & Bryl, V. (2017). Peer review in computer science conferences. In Eighth international congress on peer review and scientific publication (PRC8), Chicago, IL."},{"key":"3348_CR18","first-page":"69","volume":"5","author":"B Martin","year":"1997","unstructured":"Martin, B. (1997). Peer review as scholarly conformity. Suppression Stories,5, 69\u201383.","journal-title":"Suppression Stories"},{"issue":"1","key":"3348_CR19","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1146\/annurev.soc.27.1.415","volume":"27","author":"M McPherson","year":"2001","unstructured":"McPherson, M., Lovin, L. S., & Cook, J. M. (2001). Birds of a feather: Homophily in social networks. Annual Review of Sociology,27(1), 415\u2013444.","journal-title":"Annual Review of Sociology"},{"issue":"12","key":"3348_CR20","doi-asserted-by":"publisher","first-page":"2952","DOI":"10.1073\/pnas.1714379115","volume":"115","author":"E Pier","year":"2018","unstructured":"Pier, E., Brauer, L. M., Filut, A., Kaatz, A., Raclaw, J., Nathan, Mitchell J., et al. (2018). Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences of the United States of America,115(12), 2952\u20132957.","journal-title":"Proceedings of the National Academy of Sciences of the United States of America"},{"issue":"2","key":"3348_CR21","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/s11192-013-1002-z","volume":"97","author":"A Ragone","year":"2013","unstructured":"Ragone, A., Mirylenka, K., Casati, F., & Marchese, M. (2013). On peer review in computer science: Analysis of its effectiveness and suggestions for improvement. Scientometrics,97(2), 317\u2013356.","journal-title":"Scientometrics"},{"issue":"6","key":"3348_CR22","doi-asserted-by":"publisher","first-page":"994","DOI":"10.1177\/0162243918822744","volume":"44","author":"L Roumbanis","year":"2019","unstructured":"Roumbanis, L. (2019). Peer review or lottery? A critical analysis of two different forms of decision-making mechanisms for allocation of research grants. Science, Technology and Human Values,44(6), 994\u20131019. https:\/\/doi.org\/10.1177\/0162243918822744.","journal-title":"Science, Technology and Human Values"},{"key":"3348_CR23","series-title":"Lecture notes in computer science","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30796-7_31","volume-title":"The semantic web\u2014ISWC 2019","author":"AA Salatino","year":"2019","unstructured":"Salatino, A. A., Osborne, F., Birukou, A., & Motta, E. (2019). Improving editorial workflow and metadata quality at springer nature. In C. Ghidini, et al. (Eds.), The semantic web\u2014ISWC 2019 (Vol. 11779)., Lecture notes in computer science Cham: Springer. https:\/\/doi.org\/10.1007\/978-3-030-30796-7_31."},{"key":"3348_CR24","first-page":"1","volume":"19","author":"NB Shah","year":"2018","unstructured":"Shah, N. B., Tabibian, B., Muandet, K., Guyon, I., & von Luxburg, U. (2018). Design and analysis of the NIPS 2016 review process. Journal of Machine Learning Research,19, 1\u201334.","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"3348_CR25","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1007\/s11192-017-2518-4","volume":"113","author":"F Squazzoni","year":"2017","unstructured":"Squazzoni, F., Brezis, E., & Marusic, A. (2017). Scientometrics of peer review. Scientometrics,113(1), 501\u2013502.","journal-title":"Scientometrics"},{"issue":"3","key":"3348_CR26","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1177\/016224399101600303","volume":"16","author":"GDL Travis","year":"1991","unstructured":"Travis, G. D. L., & Collins, H. M. (1991). New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology and Human Values,16(3), 322\u2013341. https:\/\/doi.org\/10.1177\/016224399101600303.","journal-title":"Science, Technology and Human Values"},{"key":"3348_CR27","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1038\/s41586-019-0941-9","volume":"566","author":"L Wu","year":"2019","unstructured":"Wu, L., Wang, D., & Evans, J. A. (2019). Large teams develop and small teams disrupt science and technology. Nature,566, 378\u2013382. https:\/\/doi.org\/10.1038\/s41586-019-0941-9.","journal-title":"Nature"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03348-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11192-020-03348-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03348-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,2,2]],"date-time":"2021-02-02T02:34:33Z","timestamp":1612233273000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11192-020-03348-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,3]]},"references-count":27,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,4]]}},"alternative-id":["3348"],"URL":"https:\/\/doi.org\/10.1007\/s11192-020-03348-1","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,3]]},"assertion":[{"value":"3 October 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 February 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}