{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T22:51:41Z","timestamp":1780527101358,"version":"3.54.1"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2023,4,14]],"date-time":"2023-04-14T00:00:00Z","timestamp":1681430400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2016908"],"award-info":[{"award-number":["2016908"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,4,14]]},"abstract":"<jats:p>Peer evaluations are a well-established tool for evaluating individual and team performance in collaborative contexts, but are susceptible to social and cognitive biases. Current peer evaluation tools have also yet to address the unique opportunities that online collaborative technologies provide for addressing these biases. In this work, we explore the potential of one such opportunity for peer evaluations: data traces automatically generated by collaborative tools, which we refer to as \"activity traces\". We conduct a between-subjects experiment with 101 students and MTurk workers, investigating the effects of reviewing activity traces on peer evaluations of team members in an online collaborative task. Our findings show that the usage of activity traces led participants to make more and greater revisions to their evaluations compared to a control condition. These revisions also increased the consistency and participants' perceived accuracy of the evaluations that they received. Our findings demonstrate the value of activity traces as an approach for performing more reliable and objective peer evaluations of teamwork. Based on our findings as well as qualitative analysis of free-form responses in our study, we also identify and discuss key considerations and design recommendations for incorporating activity traces into real-world peer evaluation systems.<\/jats:p>","DOI":"10.1145\/3579627","type":"journal-article","created":{"date-parts":[[2023,4,16]],"date-time":"2023-04-16T17:23:07Z","timestamp":1681665787000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["The Value of Activity Traces in Peer Evaluations: An Experimental Study"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4217-803X","authenticated-orcid":false,"given":"Wenxuan Wendy","family":"Shi","sequence":"first","affiliation":[{"name":"University of Illinois, Urbana-Champaign, Champaign, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7211-5665","authenticated-orcid":false,"given":"Sneha R.","family":"Krishna Kumaran","sequence":"additional","affiliation":[{"name":"University of Illinois, Urbana Champaign, Urbana, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3315-6055","authenticated-orcid":false,"given":"Hari","family":"Sundaram","sequence":"additional","affiliation":[{"name":"University of Illinois, Urbana-Champaign, Urbana, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1859-986X","authenticated-orcid":false,"given":"Brian P.","family":"Bailey","sequence":"additional","affiliation":[{"name":"University of Illinois, Urbana-Champaign, Urbana, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,4,16]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2021. Are peer reviews the future of performance evaluations? https:\/\/hbr.org\/2021\/01\/are-peer-reviews-the-future-of-performance-evaluations"},{"key":"e_1_2_1_2_1","unstructured":"2022. The rise of performative work. https:\/\/www.economist.com\/business\/2022\/01\/07\/the-rise-of-performative-work"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/1052562907310489"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021--9010.85.2.232"},{"key":"e_1_2_1_5_1","article-title":"Pyro: Deep Universal Probabilistic Programming","volume":"20","author":"Bingham Eli","year":"2019","unstructured":"Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. J. Mach. Learn. Res. 20 (2019), 28:1--28:6. http:\/\/jmlr.org\/papers\/v20\/18--403.html","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145396"},{"key":"e_1_2_1_7_1","volume-title":"A Cognitive Approach to Performance Appraisal","author":"DeNisi Angelo S.","unstructured":"Angelo S. DeNisi. 1996. A Cognitive Approach to Performance Appraisal. Routledge, London."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/0030--5073(84)90029--1"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1037\/apl0000085"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5465\/ambpp.2015.21"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/143457.143468"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1177\/1052562906286713"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1177\/1046496419878269"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467--8535.00291"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174074"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.4135\/9781849208574"},{"key":"e_1_2_1_17_1","first-page":"460","article-title":"Managerial Discretion in the Use of Self-ratings in an Appraisal System: The Antecedents and Consequences","volume":"16","author":"Inderrieden Edward J.","year":"2004","unstructured":"Edward J. Inderrieden, Robert E. Allen, and Timothy J. Keaveny. 2004. Managerial Discretion in the Use of Self-ratings in an Appraisal System: The Antecedents and Consequences. Journal of Managerial Issues 16, 4 (2004), 460--482. http:\/\/www.jstor.org\/stable\/40604464","journal-title":"Journal of Managerial Issues"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.2001.0517"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033--2909.85.3.555"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858465"},{"key":"e_1_2_1_21_1","unstructured":"J.F. Kihlstrom Eric Eich D. Sandbrand and B.A. Tobias. 2000. Emotion and memory: Implications for self-report (with a critique of retrospective analyses). The science of self-report: Implications for research and practice (01 2000) 81--99."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1992.10485228"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-016-0727-z"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467--8535.2008.00922.x"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1177\/0273475313499023"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998322"},{"key":"e_1_2_1_27_1","volume-title":"Impression Formation in Online Peer Production: Activity Traces and Personal Profiles in Github","author":"Marlow Jennifer","unstructured":"Jennifer Marlow, Laura Dabbish, and Jim Herbsleb. 2013. Impression Formation in Online Peer Production: Activity Traces and Personal Profiles in Github. Association for Computing Machinery, New York, NY, USA, 117--128. https:\/\/doi-org.proxy2.library.illinois.edu\/10.1145\/2441776.2441792"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675250"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1201\/9780429029608"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1177\/0149206306297633"},{"key":"e_1_2_1_31_1","volume-title":"Cleveland","author":"Murphy Kevin R.","year":"1991","unstructured":"Kevin R. Murphy and Jeanette N. Cleveland. 1991. Performance appraisal: An organizational perspective. Allyn & Bacon, Needham Heights, MA, US. xiv, 349--xiv, 349 pages."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021--9010.67.5.562"},{"key":"e_1_2_1_33_1","first-page":"n2","article-title":"A Preliminary Evaluation of Using WebPA for Online Peer Assessment of Collaborative Performance by Groups of Online Distance Learners","volume":"30","author":"Murray Jo-Anne","year":"2015","unstructured":"Jo-Anne Murray and Sharon Boyd. 2015. A Preliminary Evaluation of Using WebPA for Online Peer Assessment of Collaborative Performance by Groups of Online Distance Learners. International Journal of E-Learning & Distance Education 30, 2 (2015), n2.","journal-title":"International Journal of E-Learning & Distance Education"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5465\/amle.2010.0177"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1177\/0963721414531598"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1930297500002205"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","unstructured":"Christina Peterson and N. Peterson. 2011. Impact of Peer Evaluation Confidentiality on Student Marks. International Journal for the Scholarship of Teaching and Learning 5 (07 2011) 1--13. https:\/\/doi.org\/10.20429\/ijsotl.2011.050213","DOI":"10.20429\/ijsotl.2011.050213"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1111\/jasp.12449"},{"key":"e_1_2_1_39_1","volume-title":"Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro. arXiv preprint arXiv:1912.11554","author":"Phan Du","year":"2019","unstructured":"Du Phan, Neeraj Pradhan, and Martin Jankowiak. 2019. Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro. arXiv preprint arXiv:1912.11554 (2019)."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021--9010.78.3.450"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998300"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518713"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479576"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441791"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0047060"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-012--9164--4"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145275"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021--9010.73.3.497"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1177\/1046496405284382"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1559--1816.2009.00521.x"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1025003106150"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702517"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274465"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1080\/08957340902984075"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579627","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3579627","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3579627","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:43Z","timestamp":1750182523000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579627"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,14]]},"references-count":55,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2023,4,14]]}},"alternative-id":["10.1145\/3579627"],"URL":"https:\/\/doi.org\/10.1145\/3579627","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,14]]},"assertion":[{"value":"2023-04-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}