{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:34:19Z","timestamp":1750307659635,"version":"3.41.0"},"reference-count":24,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2009,4,21]],"date-time":"2009-04-21T00:00:00Z","timestamp":1240272000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2009,4,21]]},"abstract":"<jats:p>This paper develops a model of computer systems research to help prospective authors understand the often obscure workings of conference program committees. We present data to show that the variability between reviewers is often the dominant factor as to whether a paper is accepted. We argue that paper merit is likely to be zipf distributed, making it inherently difficult for program committees to distinguish between most papers. We use game theory to show that with noisy reviews and zipf merit, authors have an incentive to submit papers too early and too often. These factors make conference reviewing, and systems research as a whole, less efficient and less effective. We describe some recent changes in conference design to address these issues, and we suggest some further potential improvements.<\/jats:p>","DOI":"10.1145\/1531793.1531815","type":"journal-article","created":{"date-parts":[[2009,4,28]],"date-time":"2009-04-28T14:58:07Z","timestamp":1240930687000},"page":"108-116","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Conference reviewing considered harmful"],"prefix":"10.1145","volume":"43","author":[{"given":"Thomas","family":"Anderson","sequence":"first","affiliation":[{"name":"University of Washington"}]}],"member":"320","published-online":{"date-parts":[[2009,4,21]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1095810.1095816"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1355734.1355741"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-003-0128-3"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1506409.1506421"},{"key":"e_1_2_1_5_1","volume-title":"Second Workshop on the Economics of Peer-to-Peer Systems","author":"Buchegger S.","year":"2004","unstructured":"S. Buchegger and J.-Y. L. Boudec . A robust reputation system for P2P and mobile ad-hoc networks . In Second Workshop on the Economics of Peer-to-Peer Systems , 2004 . S. Buchegger and J.-Y. L. Boudec. A robust reputation system for P2P and mobile ad-hoc networks. In Second Workshop on the Economics of Peer-to-Peer Systems, 2004."},{"key":"e_1_2_1_6_1","volume-title":"Workshop on Economics of Peer-to-Peer Systems","author":"Cohen B.","year":"2003","unstructured":"B. Cohen . Incentives build robustness in BitTorrent . In Workshop on Economics of Peer-to-Peer Systems , 2003 . B. Cohen. Incentives build robustness in BitTorrent. In Workshop on Economics of Peer-to-Peer Systems, 2003."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1435417.1435430"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.149.3683.510"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531793.1531816"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1086\/341871"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1070873.1070889"},{"key":"e_1_2_1_12_1","volume-title":"A mind of its own: How your brain distorts and deceives","author":"Fine C.","year":"2006","unstructured":"C. Fine . A mind of its own: How your brain distorts and deceives . Icon Books , 2006 . C. Fine. A mind of its own: How your brain distorts and deceives. Icon Books, 2006."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/52324.52356"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.769447"},{"issue":"3","key":"e_1_2_1_15_1","first-page":"35","article-title":"An evaluation of the ninth SOSP submissions; or how (and how not) to write a good systems paper","volume":"17","author":"Levin R.","year":"1983","unstructured":"R. Levin and D. Redell . An evaluation of the ninth SOSP submissions; or how (and how not) to write a good systems paper . ACM SIGOPS Operating Systems Review , 17 ( 3 ): 35 -- 40 , 1983 . R. Levin and D. Redell. An evaluation of the ninth SOSP submissions; or how (and how not) to write a good systems paper. ACM SIGOPS Operating Systems Review, 17(3):35--40, 1983.","journal-title":"ACM SIGOPS Operating Systems Review"},{"key":"e_1_2_1_16_1","volume-title":"NSDI","author":"Mahajan R.","year":"2007","unstructured":"R. Mahajan , D. Wetherall , and T. Anderson . Mutually controlled routing with independent ISPs . In NSDI , 2007 . R. Mahajan, D. Wetherall, and T. Anderson. Mutually controlled routing with independent ISPs. In NSDI, 2007."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/11551362_39"},{"key":"e_1_2_1_18_1","volume-title":"NSDI","author":"Piatek M.","year":"2007","unstructured":"M. Piatek , T. Isdal , A. Krishnamurthy , and T. Anderson . Do incentives build robustness in BitTorrent ? In NSDI , 2007 . M. Piatek, T. Isdal, A. Krishnamurthy, and T. Anderson. Do incentives build robustness in BitTorrent? In NSDI, 2007."},{"key":"e_1_2_1_19_1","volume-title":"NSDI","author":"Piatek M.","year":"2008","unstructured":"M. Piatek , T. Isdal , A. Krishnamurthy , and T. Anderson . One hop reputations for peer to peer file sharing workloads . In NSDI , 2008 . M. Piatek, T. Isdal, A. Krishnamurthy, and T. Anderson. One hop reputations for peer to peer file sharing workloads. In NSDI, 2008."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s100510050359"},{"key":"e_1_2_1_21_1","unstructured":"S. Shenker J. Kurose and T. Anderson. Improving SIGCOMM: A few straw proposals. In www.sigcomm.org\/admin\/July2001RepFinal.pdf.  S. Shenker J. Kurose and T. Anderson. Improving SIGCOMM: A few straw proposals. In www.sigcomm.org\/admin\/July2001RepFinal.pdf."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/383059.383071"},{"key":"e_1_2_1_23_1","volume-title":"Workshop on the Economics of Peer-to-Peer Systems","author":"Vishnumurthy V.","year":"2003","unstructured":"V. Vishnumurthy , S. Chandrakumar , and E. Sirer . KARMA: A secure economic framework for peer-to-peer resource sharing . In Workshop on the Economics of Peer-to-Peer Systems , 2003 . V. Vishnumurthy, S. Chandrakumar, and E. Sirer. KARMA: A secure economic framework for peer-to-peer resource sharing. In Workshop on the Economics of Peer-to-Peer Systems, 2003."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1015467.1015509"}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1531793.1531815","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1531793.1531815","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T13:29:28Z","timestamp":1750253368000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1531793.1531815"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,4,21]]},"references-count":24,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2009,4,21]]}},"alternative-id":["10.1145\/1531793.1531815"],"URL":"https:\/\/doi.org\/10.1145\/1531793.1531815","relation":{},"ISSN":["0163-5980"],"issn-type":[{"type":"print","value":"0163-5980"}],"subject":[],"published":{"date-parts":[[2009,4,21]]},"assertion":[{"value":"2009-04-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}