{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T11:31:25Z","timestamp":1750764685711,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":50,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T00:00:00Z","timestamp":1615161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NSF","award":["CAREER"],"award-info":[{"award-number":["CAREER"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,8]]},"DOI":"10.1145\/3437963.3441660","type":"proceedings-article","created":{"date-parts":[[2021,3,6]],"date-time":"2021-03-06T04:34:28Z","timestamp":1615005268000},"page":"1131-1133","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["WSDM 2021 Tutorial on Systematic Challenges and Computational Solutions on Bias and Unfairness in Peer Review"],"prefix":"10.1145","author":[{"given":"Nihar B.","family":"Shah","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,3,8]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2000378.2000390"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2254756.2254799"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Haris Aziz Omer Lev Nicholas Mattei Jeffrey S Rosenschein and Toby Walsh. 2016. Strategyproof Peer Selection: Mechanisms Analyses and Experiments.. In AAAI. 397--403.  Haris Aziz Omer Lev Nicholas Mattei Jeffrey S Rosenschein and Toby Walsh. 2016. Strategyproof Peer Selection: Mechanisms Analyses and Experiments.. In AAAI. 397--403.","DOI":"10.1609\/aaai.v30i1.10038"},{"key":"e_1_3_2_1_4_1","volume-title":"Statistical Quality Estimation for General Crowdsourcing Tasks. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.","author":"Baba Yukino","year":"2013","unstructured":"Yukino Baba and Hisashi Kashima . 2013 . Statistical Quality Estimation for General Crowdsourcing Tasks. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Yukino Baba and Hisashi Kashima. 2013. Statistical Quality Estimation for General Crowdsourcing Tasks. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining."},{"key":"e_1_3_2_1_5_1","volume-title":"The manuscript review and decision-making process. American Sociological Review","author":"Bakanic Von","year":"1987","unstructured":"Von Bakanic , Clark McPhail , and Rita J Simon . 1987. The manuscript review and decision-making process. American Sociological Review ( 1987 ), 631--642. Von Bakanic, Clark McPhail, and Rita J Simon. 1987. The manuscript review and decision-making process. American Sociological Review (1987), 631--642."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1603723113"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2005.02.002"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tree.2007.07.008"},{"volume-title":"ICML Workshop on Peer Reviewing and Publishing Models.","author":"Charlin L.","key":"e_1_3_2_1_9_1","unstructured":"L. Charlin and R. S. Zemel . 2013. The Toronto Paper Matching System: An automated paper-reviewer assignment system . In ICML Workshop on Peer Reviewing and Publishing Models. L. Charlin and R. S. Zemel. 2013. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In ICML Workshop on Peer Reviewing and Publishing Models."},{"key":"e_1_3_2_1_10_1","unstructured":"L. Charlin R. S. Zemel and C. Boutilier. 2012. A Framework for Optimizing Paper Matching. CoRR Vol. abs\/1202.3706 (2012). arxiv: 1202.3706 http:\/\/arxiv.org\/abs\/1202.3706  L. Charlin R. S. Zemel and C. Boutilier. 2012. A Framework for Optimizing Paper Matching. CoRR Vol. abs\/1202.3706 (2012). arxiv: 1202.3706 http:\/\/arxiv.org\/abs\/1202.3706"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1162\/089120105775299131"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jet.2007.06.005"},{"key":"e_1_3_2_1_13_1","unstructured":"Wenxin Ding Nihar B. and Weina Wang. 2020. On the Privacy-Utility Tradeoff in Peer-Review Data Analysis. In AAAI Privacy-Preserving Artificial Intelligence (PPAI-21) workshop.  Wenxin Ding Nihar B. and Weina Wang. 2020. On the Privacy-Utility Tradeoff in Peer-Review Data Analysis. In AAAI Privacy-Preserving Artificial Intelligence (PPAI-21) workshop."},{"key":"e_1_3_2_1_14_1","volume-title":"Conference on Uncertainty in Artificial Intelligence (UAI).","author":"Fiez T","year":"2020","unstructured":"T Fiez , N Shah , and L Ratliff . 2020 . A SUPER* Algorithm to Optimize Paper Bidding in Peer Review . In Conference on Uncertainty in Artificial Intelligence (UAI). T Fiez, N Shah, and L Ratliff. 2020. A SUPER* Algorithm to Optimize Paper Bidding in Peer Review. In Conference on Uncertainty in Artificial Intelligence (UAI)."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1137\/140995775"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1809400.1809413"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/945365.964285"},{"key":"e_1_3_2_1_18_1","first-page":"9386","volume-title":"Algorithmica","volume":"58","author":"Garg N.","year":"2010","unstructured":"N. Garg , T. Kavitha , A. Kumar , K. Mehlhorn , and J. Mestre . 2010. Assigning Papers to Referees . Algorithmica , Vol. 58 , 1 (01 Sep 2010 ), 119--136. https:\/\/doi.org\/10.1007\/s00453-009-- 9386 - 9380 N. Garg, T. Kavitha, A. Kumar, K. Mehlhorn, and J. Mestre. 2010. Assigning Papers to Referees. Algorithmica, Vol. 58, 1 (01 Sep 2010), 119--136. https:\/\/doi.org\/10.1007\/s00453-009--9386-0"},{"key":"e_1_3_2_1_19_1","unstructured":"Hong Ge Max Welling and Zoubin Ghahramani. 2013. A Bayesian model for calibrating conference review scores. http:\/\/mlg.eng.cam.ac.uk\/hong\/nipsrevcal.pdf  Hong Ge Max Welling and Zoubin Ghahramani. 2013. A Bayesian model for calibrating conference review scores. http:\/\/mlg.eng.cam.ac.uk\/hong\/nipsrevcal.pdf"},{"key":"e_1_3_2_1_20_1","volume-title":"The AI conference paper assignment problem.","volume":"10","author":"Goldsmith Judy","year":"2007","unstructured":"Judy Goldsmith and Robert H. Sloan . 2007 . The AI conference paper assignment problem. , Vol. WS-07-- 10 (12 2007 ), 53--57. Judy Goldsmith and Robert H. Sloan. 2007. The AI conference paper assignment problem., Vol. WS-07--10 (12 2007), 53--57."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ibusrev.2009.03.001"},{"key":"e_1_3_2_1_22_1","volume-title":"Provost","author":"Hill Shawndra","year":"2003","unstructured":"Shawndra Hill and Foster J . Provost . 2003 . The myth of the double-blind review? Author identification using only citations. SIGKDD Explorations , Vol. 5 (01 2003), 179--184. Shawndra Hill and Foster J. Provost. 2003. The myth of the double-blind review? Author identification using only citations. SIGKDD Explorations, Vol. 5 (01 2003), 179--184."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022670432373"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.3982\/ECTA10523"},{"key":"e_1_3_2_1_25_1","unstructured":"Steven Jecmen Hanrui Zhang Ryan Liu Nihar B. Shah Vincent Conitzer and Fei Fang. 2020. Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments. In NeurIPS.  Steven Jecmen Hanrui Zhang Ryan Liu Nihar B. Shah Vincent Conitzer and Fei Fang. 2020. Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments. In NeurIPS."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"crossref","first-page":"132","DOI":"10.2307\/255467","article-title":"Manuscript characteristics which influence acceptance for management and social science journals","volume":"20","author":"Kerr Steven","year":"1977","unstructured":"Steven Kerr , James Tolliver , and Doretta Petree . 1977 . Manuscript characteristics which influence acceptance for management and social science journals . Academy of Management Journal , Vol. 20 , 1 (1977), 132 -- 141 . Steven Kerr, James Tolliver, and Doretta Petree. 1977. Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal, Vol. 20, 1 (1977), 132--141.","journal-title":"Academy of Management Journal"},{"key":"e_1_3_2_1_28_1","volume-title":"Paper Matching with Local Fairness Constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.","author":"Kobren Ari","year":"2019","unstructured":"Ari Kobren , Barna Saha , and Andrew McCallum . 2019 . Paper Matching with Local Fairness Constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Ari Kobren, Barna Saha, and Andrew McCallum. 2019. Paper Matching with Local Fairness Constraints. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining."},{"key":"e_1_3_2_1_29_1","unstructured":"David Kurokawa Omer Lev Jamie Morgenstern and Ariel D Procaccia. 2015. Impartial Peer Review. In IJCAI. 582--588.  David Kurokawa Omer Lev Jamie Morgenstern and Ariel D Procaccia. 2015. Impartial Peer Review. In IJCAI. 582--588."},{"volume-title":"How professors think","author":"Lamont Mich\u00e8le","key":"e_1_3_2_1_30_1","unstructured":"Mich\u00e8le Lamont . 2009. How professors think . Harvard University Press . Mich\u00e8le Lamont. 2009. How professors think .Harvard University Press."},{"key":"e_1_3_2_1_31_1","unstructured":"N. Lawrence and C. Cortes. 2014. The NIPS Experiment. http:\/\/inverseprobability.com\/2014\/12\/16\/the-nips-experiment. [Online; accessed 11-June-2018].  N. Lawrence and C. Cortes. 2014. The NIPS Experiment. http:\/\/inverseprobability.com\/2014\/12\/16\/the-nips-experiment. [Online; accessed 11-June-2018]."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1086\/683652"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2013.13"},{"key":"e_1_3_2_1_34_1","volume-title":"Royal Society Open Science","volume":"4","author":"MacKay R. S.","year":"2017","unstructured":"R. S. MacKay , R. Kenna , R. J. Low , and S. Parker . 2017. Calibration with confidence: a principled method for panel assessment . Royal Society Open Science , Vol. 4 , 2 ( 2017 ). https:\/\/doi.org\/10.1098\/rsos.160760 R. S. MacKay, R. Kenna, R. J. Low, and S. Parker. 2017. Calibration with confidence: a principled method for panel assessment. Royal Society Open Science, Vol. 4, 2 (2017). https:\/\/doi.org\/10.1098\/rsos.160760"},{"key":"e_1_3_2_1_35_1","volume-title":"Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research","author":"Mahoney Michael J","year":"1977","unstructured":"Michael J Mahoney . 1977. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research , Vol. 1 , 2 ( 1977 ), 161--175. Michael J Mahoney. 1977. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research, Vol. 1, 2 (1977), 161--175."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Emaad Manzoor and Nihar B Shah. 2021. Uncovering Latent Biases in Text: Method and Application to Peer Review. In AAAI.  Emaad Manzoor and Nihar B Shah. 2021. Uncovering Latent Biases in Text: Method and Application to Peer Review. In AAAI.","DOI":"10.1609\/aaai.v35i6.16608"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.159.3810.56"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/Allerton.2011.6120296"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22798"},{"key":"e_1_3_2_1_40_1","unstructured":"Sahand Negahban Sewoong Oh and Devavrat Shah. 2012. Iterative ranking from pair-wise comparisons. In Advances in Neural Information Processing Systems.  Sahand Negahban Sewoong Oh and Devavrat Shah. 2012. Iterative ranking from pair-wise comparisons. In Advances in Neural Information Processing Systems."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1087\/20150104"},{"key":"e_1_3_2_1_42_1","volume-title":"ICML Workshop on Incentives in Machine Learning.","author":"Noothigattu Ritesh","year":"2020","unstructured":"Ritesh Noothigattu , Nihar Shah , and Ariel Procaccia . 2020 . Loss Functions, Axioms, and Peer Review . In ICML Workshop on Incentives in Machine Learning. Ritesh Noothigattu, Nihar Shah, and Ariel Procaccia. 2020. Loss Functions, Axioms, and Peer Review. In ICML Workshop on Incentives in Machine Learning."},{"key":"e_1_3_2_1_43_1","volume-title":"JAMA","volume":"316","author":"Okike Kanu","year":"2016","unstructured":"Kanu Okike , Kevin T. Hug , Mininder S. Kocher , and Seth S. Leopold . 2016. Single-blind vs Double-blind Peer Review in the Setting of Author Prestige . JAMA , Vol. 316 , 12 (09 2016 ), 1315--1316. https:\/\/doi.org\/10.1001\/jama.2016.11014 https:\/\/doi.org\/10.1073\/pnas.1707323114 showeprinthttps:\/\/www.pnas.org\/content\/114\/48\/12708.full.pdf Kanu Okike, Kevin T. Hug, Mininder S. Kocher, and Seth S. Leopold. 2016. Single-blind vs Double-blind Peer Review in the Setting of Author Prestige. JAMA, Vol. 316, 12 (09 2016), 1315--1316. https:\/\/doi.org\/10.1001\/jama.2016.11014 https:\/\/doi.org\/10.1073\/pnas.1707323114 showeprinthttps:\/\/www.pnas.org\/content\/114\/48\/12708.full.pdf"},{"key":"e_1_3_2_1_44_1","unstructured":"Chris R Triggle and David J Triggle. 2007. What is the future of peer review? Why is there fraud in science? Is plagiarism out of control? Why do scientists do bad things? Is it all a case of: \"All that is necessary for the triumph of evil is that good men do nothing?\". Vascular health and risk management Vol. 3 1 (2007) 39.  Chris R Triggle and David J Triggle. 2007. What is the future of peer review? Why is there fraud in science? Is plagiarism out of control? Why do scientists do bad things? Is it all a case of: \"All that is necessary for the triumph of evil is that good men do nothing?\". Vascular health and risk management Vol. 3 1 (2007) 39."},{"key":"e_1_3_2_1_45_1","unstructured":"Jingyan Wang and Nihar Shah. 2018. There?s Lots in a Name (Whereas There Shouldn't Be). Research on Research blog. https:\/\/researchonresearch.blog\/2018\/11\/28\/theres-lots-in-a-name\/.  Jingyan Wang and Nihar Shah. 2018. There?s Lots in a Name (Whereas There Shouldn't Be). Research on Research blog. https:\/\/researchonresearch.blog\/2018\/11\/28\/theres-lots-in-a-name\/."},{"key":"e_1_3_2_1_46_1","unstructured":"Jingyan Wang and Nihar Shah. 2019 a. Gender Distributions of Paper Awards. Research on Research blog. https:\/\/researchonresearch.blog\/2019\/06\/18\/gender-distributions-of-paper-awards\/.  Jingyan Wang and Nihar Shah. 2019 a. Gender Distributions of Paper Awards. Research on Research blog. https:\/\/researchonresearch.blog\/2019\/06\/18\/gender-distributions-of-paper-awards\/."},{"key":"e_1_3_2_1_47_1","unstructured":"Jingyan Wang and Nihar B Shah. 2019 b. Your 2 is My 1 Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings. In AAMAS.  Jingyan Wang and Nihar B Shah. 2019 b. Your 2 is My 1 Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings. In AAMAS."},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"crossref","unstructured":"Jingyan Wang Ivan Stelmakh Yuting Wei and Nihar Shah. 2021. Debiasing Evaluations that are Biased by Evaluations. In AAAI.  Jingyan Wang Ivan Stelmakh Yuting Wei and Nihar Shah. 2021. Debiasing Evaluations that are Biased by Evaluations. In AAAI.","DOI":"10.1609\/aaai.v35i11.17214"},{"key":"e_1_3_2_1_49_1","first-page":"1","article-title":"Peer review: benefits, perceptions and alternatives","volume":"4","author":"Ware Mark","year":"2008","unstructured":"Mark Ware . 2008 . Peer review: benefits, perceptions and alternatives . Publishing Research Consortium , Vol. 4 (2008), 1 -- 20 . Mark Ware. 2008. Peer review: benefits, perceptions and alternatives. Publishing Research Consortium, Vol. 4 (2008), 1--20.","journal-title":"Publishing Research Consortium"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tree.2008.03.003"},{"key":"e_1_3_2_1_51_1","volume-title":"On Strategyproof Conference Review. In IJCAI.","author":"Xu Yichong","year":"2019","unstructured":"Yichong Xu , Han Zhao , Xiaofei Shi , and Nihar Shah . 2019 . On Strategyproof Conference Review. In IJCAI. Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar Shah. 2019. On Strategyproof Conference Review. In IJCAI."}],"event":{"name":"WSDM '21: The Fourteenth ACM International Conference on Web Search and Data Mining","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Virtual Event Israel","acronym":"WSDM '21"},"container-title":["Proceedings of the 14th ACM International Conference on Web Search and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441660","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3437963.3441660","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3437963.3441660","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:27Z","timestamp":1750195467000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441660"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,8]]},"references-count":50,"alternative-id":["10.1145\/3437963.3441660","10.1145\/3437963"],"URL":"https:\/\/doi.org\/10.1145\/3437963.3441660","relation":{},"subject":[],"published":{"date-parts":[[2021,3,8]]},"assertion":[{"value":"2021-03-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}