{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T07:43:14Z","timestamp":1772005394370,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":32,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T00:00:00Z","timestamp":1657584000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000183","name":"Army Research Office","doi-asserted-by":"publisher","award":["W911NF1810208"],"award-info":[{"award-number":["W911NF1810208"]}],"id":[{"id":"10.13039\/100000183","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2007256, IIS-200788"],"award-info":[{"award-number":["2007256, IIS-200788"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,12]]},"DOI":"10.1145\/3490486.3538235","type":"proceedings-article","created":{"date-parts":[[2022,7,13]],"date-time":"2022-07-13T13:29:36Z","timestamp":1657718976000},"page":"1041-1080","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["A System-Level Analysis of Conference Peer Review"],"prefix":"10.1145","author":[{"given":"Yichi","family":"Zhang","sequence":"first","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fang-Yi","family":"Yu","sequence":"additional","affiliation":[{"name":"Harvard University, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Grant","family":"Schoenebeck","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Kempe","sequence":"additional","affiliation":[{"name":"University of South California, Los Angeles, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"ICLR 2020 review data. https:\/\/github.com\/shaohua0116\/ICLR2020-OpenReviewData.","unstructured":"2020. ICLR 2020 review data. https:\/\/github.com\/shaohua0116\/ICLR2020-OpenReviewData. 2020. ICLR 2020 review data. https:\/\/github.com\/shaohua0116\/ICLR2020-OpenReviewData."},{"key":"e_1_3_2_1_2_1","volume-title":"ICLR 2021 review data. https:\/\/github.com\/evanzd\/ICLR2021-OpenReviewData.","unstructured":"2021. ICLR 2021 review data. https:\/\/github.com\/evanzd\/ICLR2021-OpenReviewData. 2021. ICLR 2021 review data. https:\/\/github.com\/evanzd\/ICLR2021-OpenReviewData."},{"key":"e_1_3_2_1_3_1","volume-title":"Modeling peer review: An agent-based approach. Ideas in Ecology and Evolution 5, 2","author":"Allesina Stefano","year":"2012","unstructured":"Stefano Allesina . 2012. Modeling peer review: An agent-based approach. Ideas in Ecology and Evolution 5, 2 ( 2012 ). Stefano Allesina. 2012. Modeling peer review: An agent-based approach. Ideas in Ecology and Evolution 5, 2 (2012)."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2318857.2254799"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-018-2825-4"},{"key":"e_1_3_2_1_6_1","volume-title":"The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and brain sciences 14, 1","author":"Cicchetti Domenic V.","year":"1991","unstructured":"Domenic V. Cicchetti . 1991. The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and brain sciences 14, 1 ( 1991 ), 119--135. Domenic V. Cicchetti. 1991. The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and brain sciences 14, 1 (1991), 119--135."},{"key":"e_1_3_2_1_7_1","volume-title":"Simon","author":"Cole Stephen","year":"1981","unstructured":"Stephen Cole , Jonathan R. Cole , and Gary A . Simon . 1981 . Chance and consensus in peer review. Science 214, 4523 (1981), 881--886. Stephen Cole, Jonathan R. Cole, and Gary A. Simon. 1981. Chance and consensus in peer review. Science 214, 4523 (1981), 881--886."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2005.05.030"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"crossref","first-page":"e0186111","DOI":"10.1371\/journal.pone.0186111","article-title":"Can editors save peer review from peer reviewers","volume":"12","author":"D'Andrea Rafael","year":"2017","unstructured":"Rafael D'Andrea and James P. O'Dwyer . 2017 . Can editors save peer review from peer reviewers ? PloS One 12 , 10 (2017), e0186111 . Rafael D'Andrea and James P. O'Dwyer. 2017. Can editors save peer review from peer reviewers? PloS One 12, 10 (2017), e0186111.","journal-title":"PloS One"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02288803"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-019-03205-w"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jet.2012.01.019"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00464-012-2171-1"},{"key":"e_1_3_2_1_15_1","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020","author":"Jecmen Steven","year":"2020","unstructured":"Steven Jecmen , Hanrui Zhang , Ryan Liu , Nihar B. Shah , Vincent Conitzer , and Fei Fang . 2020 . Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments . In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 , NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/93fb39474c51b8a82a68413e2a5ae17a-Abstract.html Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. 2020. Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/93fb39474c51b8a82a68413e2a5ae17a-Abstract.html"},{"key":"e_1_3_2_1_16_1","unstructured":"Sampath Kannan Mingzi Niu Aaron Roth and Rakesh Vohra. 2021. Best vs. All: Equity and Accuracy of Standardized Test Score Reporting. (2021). arXiv preprint arXiv:2102.07809.  Sampath Kannan Mingzi Niu Aaron Roth and Rakesh Vohra. 2021. Best vs. All: Equity and Accuracy of Standardized Test Score Reporting. (2021). arXiv preprint arXiv:2102.07809."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-015-1800-6"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-017-2375-1"},{"key":"e_1_3_2_1_19_1","volume-title":"A model of forum shopping. American economic review 96, 4","author":"Lerner Josh","year":"2006","unstructured":"Josh Lerner and Jean Tirole . 2006. A model of forum shopping. American economic review 96, 4 ( 2006 ), 1091--1113. Josh Lerner and Jean Tirole. 2006. A model of forum shopping. American economic review 96, 4 (2006), 1091--1113."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1641\/0006-3568(2006)56[333:IPRAGO]2.0.CO;2"},{"key":"e_1_3_2_1_21_1","unstructured":"Ritesh Noothigattu Nihar B. Shah and Ariel Procaccia. 2018. Choosing how to choose papers. (2018). arXiv preprint arxiv:1808.09057.  Ritesh Noothigattu Nihar B. Shah and Ariel Procaccia. 2018. Choosing how to choose papers. (2018). arXiv preprint arxiv:1808.09057."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0018680"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.112"},{"key":"e_1_3_2_1_24_1","first-page":"1","article-title":"Design and analysis of the NIPS 2016 review process","volume":"19","author":"Shah Nihar B.","year":"2018","unstructured":"Nihar B. Shah , Behzad Tabibian , Krikamol Muandet , Isabelle Guyon , and Ulrike Von Luxburg . 2018 . Design and analysis of the NIPS 2016 review process . Journal of Machine Learning Research 19 (2018), 1 -- 49 . Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. 2018. Design and analysis of the NIPS 2016 review process. Journal of Machine Learning Research 19 (2018), 1--49.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_25_1","unstructured":"Lones Smith and Andrea Wilson. 2021. Accept this Paper. (2021). Manuscript in preparation. Slide deck available at https:\/\/econ.la.psu.edu\/events\/seminar-documents\/accept2021.pdf.  Lones Smith and Andrea Wilson. 2021. Accept this Paper. (2021). Manuscript in preparation. Slide deck available at https:\/\/econ.la.psu.edu\/events\/seminar-documents\/accept2021.pdf."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2011.12.005"},{"key":"e_1_3_2_1_27_1","unstructured":"Siddarth Srinivasan and Jamie Morgenstern. 2021. Auctions and Prediction Markets for Scientific Peer Review. (2021). arXiv preprint arXiv:2109.00923.  Siddarth Srinivasan and Jamie Morgenstern. 2021. Auctions and Prediction Markets for Scientific Peer Review. (2021). arXiv preprint arXiv:2109.00923."},{"key":"e_1_3_2_1_28_1","unstructured":"Weijie J. Su. 2021. You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism. arXiv:cs.LG\/2110.14802  Weijie J. Su. 2021. You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism. arXiv:cs.LG\/2110.14802"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1140\/epjb\/e2011-20545-7"},{"key":"e_1_3_2_1_30_1","volume-title":"An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process. (2020","author":"Tran David","year":"2020","unstructured":"David Tran , Alex Valtchanov , Keshav Ganapathy , Raymond Feng , Eric Slud , Micah Goldblum , and Tom Goldstein . 2020 . An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process. (2020 ). arXiv preprint arXiv:2010.05137. David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, and Tom Goldstein. 2020. An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process. (2020). arXiv preprint arXiv:2010.05137."},{"key":"e_1_3_2_1_31_1","volume-title":"Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13--17","author":"Wang Jingyan","year":"2019","unstructured":"Jingyan Wang and Nihar B. Shah . 2019. Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings . In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13--17 , 2019 , Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor (Eds.). International Foundation for Autonomous Agents and Multiagent Systems, 864--872. http:\/\/dl.acm.org\/citation.cfm?id=3331778 Jingyan Wang and Nihar B. Shah. 2019. Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13--17, 2019, Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor (Eds.). International Foundation for Autonomous Agents and Multiagent Systems, 864--872. http:\/\/dl.acm.org\/citation.cfm?id=3331778"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i6.16727"}],"event":{"name":"EC '22: The 23rd ACM Conference on Economics and Computation","location":"Boulder CO USA","acronym":"EC '22","sponsor":["SIGecom Special Interest Group on Economics and Computation"]},"container-title":["Proceedings of the 23rd ACM Conference on Economics and Computation"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490486.3538235","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3490486.3538235","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3490486.3538235","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:03Z","timestamp":1750188663000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490486.3538235"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,12]]},"references-count":32,"alternative-id":["10.1145\/3490486.3538235","10.1145\/3490486"],"URL":"https:\/\/doi.org\/10.1145\/3490486.3538235","relation":{},"subject":[],"published":{"date-parts":[[2022,7,12]]},"assertion":[{"value":"2022-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}