{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T16:01:46Z","timestamp":1773763306886,"version":"3.50.1"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,10,18]],"date-time":"2022-10-18T00:00:00Z","timestamp":1666051200000},"content-version":"vor","delay-in-days":370,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1552663"],"award-info":[{"award-number":["IIS-1552663"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,13]]},"abstract":"<jats:p>Peer review has been used in both online and offline classrooms to inspire creativity, gather feedback, and lessen instructor grading loads, especially for design-based tasks without definitive rubrics. To explore the nuances and quality of peer feedback, we developed UX Factor, a peer grading platform that aims to characterize the behavior of peer reviews and the consistency of the ranking models used to aggregate these reviews. This system harnesses the power of pairwise comparisons to minimize bias and encourage context-driven analysis. We adopted UX Factor in a user interface course of 133 students and teaching assistants (TAs) across 3 different individual design projects over a semester and found that the system was effective in eliciting high-quality feedback. We saw that raters have higher agreement than random preferences, and with at least 15 ratings per submission, a simple average of ratings produced rankings that were consistent to both the raw ratings and other more complex models. These rankings were robust to disagreeable raters and changing class sizes, demonstrating the potential of comparative peer review to match the quality of expert feedback at scale.<\/jats:p>","DOI":"10.1145\/3479863","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T02:30:11Z","timestamp":1634610611000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["The UX Factor: Using Comparative Peer Review to Evaluate Designs through User Preferences"],"prefix":"10.1145","volume":"5","author":[{"given":"Sarah","family":"Bawabe","sequence":"first","affiliation":[{"name":"Brown University, Providence, RI, USA"}]},{"given":"Laura","family":"Wilson","sequence":"additional","affiliation":[{"name":"Brown University, Providence, RI, USA"}]},{"given":"Tongyu","family":"Zhou","sequence":"additional","affiliation":[{"name":"Brown University, Providence, RI, USA"}]},{"given":"Ezra","family":"Marks","sequence":"additional","affiliation":[{"name":"Brown University, Providence, RI, USA"}]},{"given":"Jeff","family":"Huang","sequence":"additional","affiliation":[{"name":"Brown University, Providence, RI, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1257\/jep.23.3.77"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1080\/00461520.2013.775712"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.aar8260"},{"key":"e_1_2_1_4_1","first-page":"40","article-title":"Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review?","volume":"8","author":"Balfour Stephen P","year":"2013","unstructured":"Stephen P Balfour. 2013. Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review?. Research & Practice in Assessment, Vol. 8 (2013), 40--48. https:\/\/eric.ed.gov\/?id=EJ1062843","journal-title":"Research & Practice in Assessment"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.2307\/2334029"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173868"},{"key":"e_1_2_1_7_1","unstructured":"Orville L Chapman. 1999. Calibrated Peer Review\u00ae. http:\/\/cpr.molsci.ucla.edu\/Home"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433420"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2015.1126551"},{"key":"e_1_2_1_10_1","first-page":"50","article-title":"Student Peer Assessment Using Adaptive Comparative Judgment: Grading Accuracy versus Quality of Feedback","volume":"12","author":"Demonacos Constantinos","year":"2019","unstructured":"Constantinos Demonacos, Steven Ellis, and Jill Barber. 2019. Student Peer Assessment Using Adaptive Comparative Judgment: Grading Accuracy versus Quality of Feedback. Practitioner Research in Higher Education, Vol. 12, 1 (2019), 50--59. https:\/\/eric.ed.gov\/?id=EJ1212981","journal-title":"Practitioner Research in Higher Education"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.3102\/00346543070003287"},{"key":"e_1_2_1_12_1","unstructured":"Leslie Feigenbaum and Nancy Holland. 1997. Using Peer Evaluations to Assign Grades on Group Projects. http:\/\/ascpro0.ascweb.org\/archives\/1997\/feigenbaum97.htm"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218127407018439"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0305-0483(00)00021--9"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858195"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1086\/209556"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3051459"},{"key":"e_1_2_1_18_1","volume-title":"Digital Humanities 2014 : Book of Abstracts. Alliance of Digital Humanities Organizations","author":"Kaplan Fr\u00e9d\u00e9ric","year":"2009","unstructured":"Fr\u00e9d\u00e9ric Kaplan and Cyril Bornet. 2014. A Preparatory Analysis of Peer-Grading for a Digital Humanities MOOC. In Digital Humanities 2014 : Book of Abstracts. Alliance of Digital Humanities Organizations, Lausanne, Switzerland, 227--229. https:\/\/infoscience.epfl.ch\/record\/200911"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11251-010--9133--6"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2876034.2893424"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the CHI 2014 Learning Innovation at Scale workshop. ACM, NY, NY, USA, 6 pages.","author":"Komarov Steven","year":"2014","unstructured":"Steven Komarov and Krzysztof Z Gajos. 2014. Organic Peer Assessment. In Proceedings of the CHI 2014 Learning Innovation at Scale workshop. ACM, NY, NY, USA, 6 pages."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3054004"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505057"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951718756684"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033--295x.110.3.472"},{"key":"e_1_2_1_26_1","volume-title":"Rank Centrality: Ranking from Pair-wise Comparisons. arxiv: 1209.1688 [cs.LG]","author":"Negahban Sahand","year":"2015","unstructured":"Sahand Negahban, Sewoong Oh, and Devavrat Shah. 2015. Rank Centrality: Ranking from Pair-wise Comparisons. arxiv: 1209.1688 [cs.LG]"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386527.3406734"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2013.795518"},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Lawrence Page Sergey Brin Rajeev Motwani and Terry Winograd. 1999. The PageRank Citation Ranking: Bringing Order to the Web. Technical Report 1999--66. Stanford InfoLab. http:\/\/ilpubs.stanford.edu:8090\/422\/","DOI":"10.1016\/B978-088415228-6\/50166-0"},{"key":"e_1_2_1_30_1","unstructured":"Chris Piech Jonathan Huang Zhenghao Chen Chuong Do Andrew Ng and Daphne Koller. 2013. Tuned Models of Peer Assessment in MOOCs. arxiv: 1307.2579 [cs.LG]"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1080\/0969594X.2012.665354"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.20343\/teachlearninqu.5.2.8"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623654"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797615623770"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15326977ea1101_1"},{"key":"e_1_2_1_36_1","volume-title":"NIPS Workshop on Data Driven Education. Curran Associates","author":"Shah Nihar B","year":"2013","unstructured":"Nihar B Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. 2013. A Case for Ordinal Peer-evaluation in MOOCs. In NIPS Workshop on Data Driven Education. Curran Associates, Lake Tahoe, Nevada, USA, 8 pages."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2876034.2876043"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124960"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0087275"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v048.i09"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2017.1405238"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07458-0_24"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the Eighth International Conference on Mobile, Hybrid, and On-line Learning. International Academy, Research, and Industry Association","author":"Wahid Usman","year":"2016","unstructured":"Usman Wahid, Mohamed Amine Chatti, and Ulrik Schroeder. 2016. A Systematic Analysis of Peer Assessment in the MOOC Era and Future Perspectives. In Proceedings of the Eighth International Conference on Mobile, Hybrid, and On-line Learning. International Academy, Research, and Industry Association, Venice, Italy, 64--69."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/0735633115571303"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479863","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479863","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479863","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:54:53Z","timestamp":1752468893000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479863"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":44,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2021,10,13]]}},"alternative-id":["10.1145\/3479863"],"URL":"https:\/\/doi.org\/10.1145\/3479863","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2021-10-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}