{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T14:24:15Z","timestamp":1778250255022,"version":"3.51.4"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2021,4,13]],"date-time":"2021-04-13T00:00:00Z","timestamp":1618272000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["CIF: 1942124, CIF: 1763734"],"award-info":[{"award-number":["CIF: 1942124, CIF: 1763734"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,4,13]]},"abstract":"<jats:p>Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent reviewers is growing at a much slower rate. To curb this trend and reduce the burden on reviewers, several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers. Such initiatives have been met with skepticism among authors, who raise the concern about a potential bias in reviewers' recommendations induced by this information. In this work, we investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue, focusing on a population of novice reviewers who constitute a large fraction of the reviewer pool in leading machine learning and computer science conferences. We design and conduct a randomized controlled trial closely replicating the relevant components of the peer-review pipeline with $133$ reviewers (master's, junior PhD students, and recent graduates of top US universities) writing reviews for $19$ papers. The analysis reveals that reviewers indeed become negatively biased when they receive a signal about paper being a resubmission, giving almost 1 point lower overall score on a 10-point Likert item (\u0394 = -0.78, 95% CI = [-1.30, -0.24]) than reviewers who do not receive such a signal. Looking at specific criteria scores (originality, quality, clarity and significance), we observe that novice reviewers tend to underrate quality the most.<\/jats:p>","DOI":"10.1145\/3449149","type":"journal-article","created":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T17:51:09Z","timestamp":1619113869000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Prior and Prejudice"],"prefix":"10.1145","volume":"5","author":[{"given":"Ivan","family":"Stelmakh","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nihar B.","family":"Shah","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aarti","family":"Singh","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"suffix":"III","given":"Hal","family":"Daum\u00e9","sequence":"additional","affiliation":[{"name":"University of Maryland &amp; Microsoft Research, College Park, MD, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,4,22]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0065-2601(08)60236-1"},{"key":"e_1_2_2_2_1","volume-title":"Groups, leadership and men","author":"Asch S. E.","unstructured":"S. E. Asch. 1951. Effects of group pressure upon the modification and distortion of judgments. In Groups, leadership and men; research in human relations, H. Guetzkow (Ed.). Carnegie Press, 177--190."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2019.06.004"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1348\/000712608X377117"},{"key":"e_1_2_2_5_1","volume-title":"The manuscript review and decision-making process. American Sociological Review","author":"Bakanic Von","year":"1987","unstructured":"Von Bakanic, Clark McPhail, and Rita J Simon. 1987. The manuscript review and decision-making process. American Sociological Review (1987), 631--642."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.54.4.569"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.71.5.915"},{"key":"e_1_2_2_8_1","first-page":"5","article-title":"The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review","volume":"81","author":"Blank Rebecca M","year":"1991","unstructured":"Rebecca M Blank. 1991. The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review. American Economic Review 81, 5 (December 1991), 1041--1067. https:\/\/ideas.repec.org\/a\/ aea\/aecrev\/v81y1991i5p1041-67.html","journal-title":"American Economic Review"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.280.3.254"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1370\/afm.1448"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1559-1816.1983.tb01741.x"},{"key":"e_1_2_2_12_1","unstructured":"L. Charlin and R. S. Zemel. 2013. The Toronto Paper Matching System: An automated paper-reviewer assignment system. http:\/\/www.cs.toronto.edu\/~zemel\/documents\/tpms.pdf"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.55.090902.142015"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.112.1.155"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1001\/archinternmed.2010.406"},{"key":"e_1_2_2_16_1","first-page":"178","article-title":"Reviewer bias: a blinded experimental study","volume":"124","author":"Ernst Edzard","year":"1994","unstructured":"Edzard Ernst and Karl-Ludwig Resch. 1994. Reviewer bias: a blinded experimental study. The Journal of laboratory and clinical medicine 124, 2 (1994), 178--182.","journal-title":"The Journal of laboratory and clinical medicine"},{"key":"e_1_2_2_17_1","volume-title":"Conference on Uncertainty in Artificial Intelligence.","author":"Fiez T","year":"2020","unstructured":"T Fiez, N Shah, and L Ratliff. 2020. A SUPER* Algorithm to Optimize Paper Bidding in Peer Review. In Conference on Uncertainty in Artificial Intelligence."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/0030-5073(75)90002-1"},{"key":"e_1_2_2_19_1","volume-title":"The design of experiments. Oliver & Boyd","author":"Fisher Ronald A.","unstructured":"Ronald A. Fisher. 1935. The design of experiments. Oliver & Boyd, Oxford, England."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/1387629.1387636"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1257\/jep.8.1.165"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-020-03357-0"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","unstructured":"N. Garg T. Kavitha A. Kumar K. Mehlhorn and J. Mestre. 2010. Assigning Papers to Referees. Algorithmica 58 1 (01 Sep 2010) 119--136. https:\/\/doi.org\/10.1007\/s00453-009-9386-0","DOI":"10.1007\/s00453-009-9386-0"},{"key":"e_1_2_2_24_1","unstructured":"Hong Ge Max Welling and Zoubin Ghahramani. 2013. A Bayesian model for calibrating conference review scores. http:\/\/mlg.eng.cam.ac.uk\/hong\/nipsrevcal.pdf"},{"key":"e_1_2_2_25_1","doi-asserted-by":"crossref","unstructured":"Thomas Gilovich Daniel Kahneman and Thomas Gilovich. 2002. Heuristics and Biases: The Psychology of Intuitive Judgment. (2002) 857.","DOI":"10.1017\/CBO9780511808098"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.1099464"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.107.3.311"},{"key":"e_1_2_2_28_1","unstructured":"Steven Jecmen Hanrui Zhang Ryan Liu Nihar B. Shah Vincent Conitzer and Fei Fang. 2020. Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments. In NeurIPS."},{"key":"e_1_2_2_29_1","volume-title":"fast and slow. Farrar, Straus and Giroux","author":"Kahneman Daniel","unstructured":"Daniel Kahneman. 2011. Thinking, fast and slow. Farrar, Straus and Giroux, New York."},{"key":"e_1_2_2_30_1","volume-title":"Representativeness revisited: Attribute substitution in intuitive judgment. Heuristics and biases: The psychology of intuitive judgment 49 (01","author":"Kahneman Daniel","year":"2002","unstructured":"Daniel Kahneman and Shane Frederick. 2002. Representativeness revisited: Attribute substitution in intuitive judgment. Heuristics and biases: The psychology of intuitive judgment 49 (01 2002), 49--81. https:\/\/doi.org\/10.1017\/ CBO9780511808098.004"},{"key":"e_1_2_2_31_1","doi-asserted-by":"crossref","first-page":"132","DOI":"10.2307\/255467","article-title":"Manuscript characteristics which influence acceptance for management and social science journals","volume":"20","author":"Kerr Steven","year":"1977","unstructured":"Steven Kerr, James Tolliver, and Doretta Petree. 1977. Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal 20, 1 (1977), 132--141.","journal-title":"Academy of Management Journal"},{"key":"e_1_2_2_32_1","volume-title":"Paper Matching with Local Fairness Constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.","author":"Kobren Ari","year":"2019","unstructured":"Ari Kobren, Barna Saha, and Andrew McCallum. 2019. Paper Matching with Local Fairness Constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining."},{"key":"e_1_2_2_33_1","volume-title":"How professors think","author":"Lamont Mich\u00e8le","unstructured":"Mich\u00e8le Lamont. 2009. How professors think. Harvard University Press."},{"key":"e_1_2_2_34_1","unstructured":"John Langford. 2012. ICML acceptance statistics. http:\/\/hunch.net\/?p=2517 [Accessed: 05\/30\/2020]."},{"key":"e_1_2_2_35_1","unstructured":"Neil Lawrence and Corinna Cortes. 2014. The NIPS experiment. http:\/\/inverseprobability.com\/2014\/12\/16\/the-nips-experiment. [Accessed: 05\/30\/2020]."},{"key":"e_1_2_2_36_1","volume-title":"Getting Started with NeurIPS","author":"Lin Hsuan-Tien","year":"2020","unstructured":"Hsuan-Tien Lin, Maria-Florina Balcan, Raia Hadsell, and Marc'Aurelio Ranzato. 2020. Getting Started with NeurIPS 2020. https:\/\/medium.com\/@NeurIPSConf\/getting-started-with-neurips-2020-e350f9b39c28. Accessed: 05\/30\/2020."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.37.11.2098"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.F.01152"},{"key":"e_1_2_2_39_1","volume-title":"Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research 1, 2","author":"Mahoney Michael J","year":"1977","unstructured":"Michael J Mahoney. 1977. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research 1, 2 (1977), 161--175."},{"key":"e_1_2_2_40_1","volume-title":"INFORMS Workshop on Data Science.","author":"Manzoor Emaad","year":"2020","unstructured":"Emaad Manzoor and Nihar B Shah. 2020. Uncovering Latent Biases in Text: Method and Application to Peer Review. In INFORMS Workshop on Data Science."},{"key":"e_1_2_2_41_1","first-page":"31","article-title":"An Experimental Investigation of the Outcome Bias In Salesperson Performance Evaluations","volume":"13","author":"Marshall Greg W.","year":"1993","unstructured":"Greg W. Marshall and John C. Mowen. 1993. An Experimental Investigation of the Outcome Bias In Salesperson Performance Evaluations. Journal of Personal Selling & Sales Management 13, 3 (1993), 31--47. https:\/\/doi.org\/10.1080\/ 08853134.1993.10753956","journal-title":"Journal of Personal Selling & Sales Management"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.159.3810.56"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1553-2712.1994.tb02443.x"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1521\/soco.19.2.145.20705"},{"key":"e_1_2_2_45_1","volume-title":"ICML workshop on Incentives in Machine Learning.","author":"Noothigattu Ritesh","year":"2020","unstructured":"Ritesh Noothigattu, Nihar B Shah, and Ariel D Procaccia. 2020. Loss Functions, Axioms, and Peer Review. In ICML workshop on Incentives in Machine Learning."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.11014"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1714379115"},{"key":"e_1_2_2_48_1","unstructured":"Eric Price. 2014. The NeurIPS experiment. http:\/\/blog.mrtz.org\/2014\/12\/15\/the-nips-experiment.html. [Accessed: 05\/30\/2020]."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1162\/003355399555945"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-36822-7_9"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612454303"},{"key":"e_1_2_2_52_1","volume-title":"AAAI Conference on Artificial Intelligence.","author":"Roos Magnus","year":"2011","unstructured":"Magnus Roos, J\u00f6rg Rothe, and Bj\u00f6rn Scheuermann. 2011. How to Calibrate the Scores of Biased Reviewers by Quadratic Programming. In AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.86.3.638"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022--3514.32.5.880"},{"key":"e_1_2_2_55_1","volume-title":"Wiltschko","author":"Sculley D.","year":"2019","unstructured":"D. Sculley, Jasper Snoek, and Alexander B. Wiltschko. 2019. Avoiding a Tragedy of the Commons in the Peer Review Process. CoRR abs\/1901.06246 (2019). arXiv:1901.06246 http:\/\/arxiv.org\/abs\/1901.06246"},{"key":"e_1_2_2_56_1","volume-title":"Retrieved","author":"Shah Nihar B.","year":"2019","unstructured":"Nihar B. Shah. 2019. Principled Methods to Improve Peer Review. Retrieved May 27, 2020 from http:\/\/www.cs.cmu.edu\/ ~nihars\/publications\/survey_peerreview_niharshah.pdf"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291125.3309611"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2011.12.005"},{"key":"e_1_2_2_59_1","volume-title":"Who Is Rational? Studies of Individual Differences in Reasoning","author":"Stanovich Keith","unstructured":"Keith Stanovich. 1999. Who Is Rational? Studies of Individual Differences in Reasoning. Mahwah, NJ: Erlbaum."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022--3514.94.4.672"},{"key":"e_1_2_2_61_1","volume-title":"PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review. arXiv preprint arxiv:1806.06237","author":"Stelmakh Ivan","year":"2018","unstructured":"Ivan Stelmakh, Nihar Shah, and Aarti Singh. 2018. PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review. arXiv preprint arxiv:1806.06237 (2018)."},{"key":"e_1_2_2_62_1","unstructured":"Ivan Stelmakh Nihar Shah and Aarti Singh. 2019. On Testing for Biases in Peer Review. In NeurIPS."},{"key":"e_1_2_2_63_1","volume-title":"ICML Workshop on Incentives in Machine Learning.","author":"Stelmakh I.","unstructured":"I. Stelmakh, N. B. Shah, and A. Singh. 2020. Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment. In ICML Workshop on Incentives in Machine Learning."},{"key":"e_1_2_2_64_1","doi-asserted-by":"crossref","unstructured":"Ivan Stelmakh Nihar B. Shah Aarti Singh and Hal Daum\u00e9 III. 2020. A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences. (2020). Preprint http:\/\/www.cs.cmu.edu\/afs\/cs.cmu.edu\/user\/istelmak\/www\/icml\/novice.pdf.","DOI":"10.1609\/aaai.v35i6.16610"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.73.3.437"},{"key":"e_1_2_2_66_1","volume-title":"Lakhani","author":"Teplitskiy Misha","year":"2019","unstructured":"Misha Teplitskiy, Hardeep Ranub, Gary S. Grayb, Michael Meniettid, Eva C. Guinan, and Karim R. Lakhani. 2019. Social Influence among Experts: Field Experimental Evidence from Peer Review."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1140\/epjb\/e2011-20545-7"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1707323114arXiv:https:"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1177\/016224399101600303"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.185.4157.1124"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.3102\/10769986025002101"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23349"},{"key":"e_1_2_2_73_1","unstructured":"Jingyan Wang and Nihar B Shah. 2019. Your 2 is My 1 Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings.In AAMAS."},{"key":"e_1_2_2_74_1","volume-title":"On Strategyproof Conference Review. In IJCAI.","author":"Xu Yichong","year":"2019","unstructured":"Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar Shah. 2019. On Strategyproof Conference Review. In IJCAI."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449149","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449149","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449149","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:17Z","timestamp":1750191497000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449149"}},"subtitle":["The Novice Reviewers' Bias against Resubmissions in Conference Peer Review"],"short-title":[],"issued":{"date-parts":[[2021,4,13]]},"references-count":74,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2021,4,13]]}},"alternative-id":["10.1145\/3449149"],"URL":"https:\/\/doi.org\/10.1145\/3449149","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,13]]},"assertion":[{"value":"2021-04-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}