{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,2]],"date-time":"2025-12-02T03:18:30Z","timestamp":1764645510684,"version":"3.41.0"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2020,5,28]],"date-time":"2020-05-28T00:00:00Z","timestamp":1590624000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["1816620"],"award-info":[{"award-number":["1816620"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,5,28]]},"abstract":"<jats:p>Review processes involve complex and often subjective decision-making tasks in which individual reviewers must read and rate submissions, such as a college application, along many relevant dimensions and typically with a rubric in mind. A common part of the work is committee review, where individual reviewers meet to discuss the merits of a particular submission in order to recommend an accept or reject decision. Prior work indicates that visualization and sensemaking support may be beneficial in such processes where reviewers must present the \"story\" of the applicant under question. We conducted a series of participatory design workshops with reviewers in the domain of holistic college admissions to better understand the challenges and opportunities regarding storytelling. Based on these workshops, we contribute a characterization for how reviewers in this domain construct visual stories, we provide guidance for designing for evidence capture and storytelling, and we draw parallels and distinctions between this domain and other reviewing domains.<\/jats:p>","DOI":"10.1145\/3392870","type":"journal-article","created":{"date-parts":[[2020,5,29]],"date-time":"2020-05-29T16:01:06Z","timestamp":1590768066000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Supporting Storytelling With Evidence in Holistic Review Processes: A Participatory Design Approach"],"prefix":"10.1145","volume":"4","author":[{"given":"Ronald A.","family":"Metoyer","sequence":"first","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tee","family":"Chuanromanee","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gina M.","family":"Girgis","sequence":"additional","affiliation":[{"name":"Troy University, Troy, AL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiyu","family":"Zhi","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eleanor C.","family":"Kinyon","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,5,29]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_2_2_1_1","DOI":"10.1007\/s10606-013-9192-8"},{"doi-asserted-by":"publisher","key":"e_1_2_2_2_1","DOI":"10.1007\/978-3-540-73400-0_10"},{"doi-asserted-by":"publisher","key":"e_1_2_2_3_1","DOI":"10.17705\/1jais.00204"},{"doi-asserted-by":"publisher","key":"e_1_2_2_4_1","DOI":"10.3102\/0162373716642407"},{"doi-asserted-by":"publisher","key":"e_1_2_2_5_1","DOI":"10.1145\/2531602.2531621"},{"volume-title":"The shape of the river: Long-term consequences of considering race in college and university admissions","author":"Bowen William G","unstructured":"William G Bowen and Derek Bok . 1999. The shape of the river: Long-term consequences of considering race in college and university admissions . Princeton University Press Princeton , NJ. William G Bowen and Derek Bok. 1999. The shape of the river: Long-term consequences of considering race in college and university admissions .Princeton University Press Princeton, NJ.","key":"e_1_2_2_6_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_7_1","DOI":"10.1145\/2940299.2940316"},{"key":"e_1_2_2_8_1","volume-title":"Group Differences in Standardized Testing and Social Stratification. Report No. 99--5","author":"Camara Wayne J","year":"1999","unstructured":"Wayne J Camara and Amy Elizabeth Schmidt . 1999. Group Differences in Standardized Testing and Social Stratification. Report No. 99--5 . College Entrance Examination Board ( 1999 ). Wayne J Camara and Amy Elizabeth Schmidt. 1999. Group Differences in Standardized Testing and Social Stratification. Report No. 99--5. College Entrance Examination Board (1999)."},{"doi-asserted-by":"publisher","key":"e_1_2_2_9_1","DOI":"10.1518\/001872008X354183"},{"key":"e_1_2_2_10_1","volume-title":"Visual storytelling: A beneficial but challenging method for health research with young people. Qualitative health research","author":"Drew Sarah E","year":"2010","unstructured":"Sarah E Drew , Rony E Duncan , and Susan M Sawyer . 2010. Visual storytelling: A beneficial but challenging method for health research with young people. Qualitative health research , Vol. 20 , 12 ( 2010 ), 1677--1688. Sarah E Drew, Rony E Duncan, and Susan M Sawyer. 2010. Visual storytelling: A beneficial but challenging method for health research with young people. Qualitative health research, Vol. 20, 12 (2010), 1677--1688."},{"volume-title":"The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges-and Who Gets Left Outside the Gates","author":"Golden Daniel","unstructured":"Daniel Golden . 2007. The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges-and Who Gets Left Outside the Gates . Broadway Books . Daniel Golden. 2007. The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges-and Who Gets Left Outside the Gates. Broadway Books.","key":"e_1_2_2_11_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_12_1","DOI":"10.1145\/353485.353487"},{"volume-title":"Contextual design: defining customer-centered systems","author":"Holtzblatt Karen","unstructured":"Karen Holtzblatt and Hugh Beyer . 1997. Contextual design: defining customer-centered systems . Elsevier . Karen Holtzblatt and Hugh Beyer. 1997. Contextual design: defining customer-centered systems .Elsevier.","key":"e_1_2_2_13_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_14_1","DOI":"10.1145\/2339530.2339742"},{"doi-asserted-by":"publisher","key":"e_1_2_2_15_1","DOI":"10.1145\/1531674.1531702"},{"doi-asserted-by":"publisher","key":"e_1_2_2_16_1","DOI":"10.1109\/ICSMC.2009.5346207"},{"volume-title":"fast and slow","author":"Kahneman Daniel","unstructured":"Daniel Kahneman . 2011. Thinking , fast and slow . Macmillan . Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.","key":"e_1_2_2_17_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_18_1","DOI":"10.1057\/palgrave.ivs.9500142"},{"doi-asserted-by":"publisher","key":"e_1_2_2_19_1","DOI":"10.1007\/s10606-014-9205-2"},{"doi-asserted-by":"crossref","unstructured":"Narges Mahyar Weichen Liu Sijia Xiao Jacob Browne Ming Yang and Steven P Dow. 2017. ConsesnsUs: Visualizing Points of Disagreement for Multi-Criteria Collaborative Decision Making. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM 17--20.  Narges Mahyar Weichen Liu Sijia Xiao Jacob Browne Ming Yang and Steven P Dow. 2017. ConsesnsUs: Visualizing Points of Disagreement for Multi-Criteria Collaborative Decision Making. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM 17--20.","key":"e_1_2_2_20_1","DOI":"10.1145\/3022198.3023269"},{"doi-asserted-by":"publisher","key":"e_1_2_2_21_1","DOI":"10.1109\/TVCG.2014.2346573"},{"doi-asserted-by":"publisher","key":"e_1_2_2_22_1","DOI":"10.1145\/1518701.1518904"},{"volume-title":"Human-computer interaction","author":"Muller Michael J","unstructured":"Michael J Muller . 2009. Participatory design: the third space in HCI . In Human-computer interaction . CRC press , 181--202. Michael J Muller. 2009. Participatory design: the third space in HCI. In Human-computer interaction. CRC press, 181--202.","key":"e_1_2_2_23_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_24_1","DOI":"10.1016\/j.procs.2014.05.193"},{"doi-asserted-by":"publisher","key":"e_1_2_2_25_1","DOI":"10.1016\/j.knosys.2013.07.003"},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of international conference on intelligence analysis","volume":"5","author":"Pirolli Peter","year":"2005","unstructured":"Peter Pirolli and Stuart Card . 2005 . The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis . In Proceedings of international conference on intelligence analysis , Vol. 5 . McLean, VA, USA, 2--4. Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2--4."},{"unstructured":"Katie Reilly. 2018 (accessed August 12 2019). A Lawsuit by Asian-American Students Against Harvard Could End Affirmative Action as We Know It. https:\/\/time.com\/5425147\/harvard-affirmative-action-trial-asian-american-students\/  Katie Reilly. 2018 (accessed August 12 2019). A Lawsuit by Asian-American Students Against Harvard Could End Affirmative Action as We Know It. https:\/\/time.com\/5425147\/harvard-affirmative-action-trial-asian-american-students\/","key":"e_1_2_2_27_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_28_1","DOI":"10.1037\/0022-3514.37.3.322"},{"doi-asserted-by":"publisher","key":"e_1_2_2_29_1","DOI":"10.1080\/07370024.2017.1300063"},{"doi-asserted-by":"publisher","key":"e_1_2_2_30_1","DOI":"10.1080\/07370024.2017.1300063"},{"key":"e_1_2_2_31_1","volume-title":"Narrative visualization: Telling stories with data","author":"Segel Edward","year":"2010","unstructured":"Edward Segel and Jeffrey Heer . 2010. Narrative visualization: Telling stories with data . IEEE transactions on visualization and computer graphics, Vol. 16 , 6 ( 2010 ), 1139--1148. Edward Segel and Jeffrey Heer. 2010. Narrative visualization: Telling stories with data. IEEE transactions on visualization and computer graphics, Vol. 16, 6 (2010), 1139--1148."},{"doi-asserted-by":"publisher","key":"e_1_2_2_32_1","DOI":"10.1109\/VL.1996.545307"},{"volume-title":"The gatekeepers: Inside the admissions process of a premier college","author":"Steinberg Jacques","unstructured":"Jacques Steinberg . 2003. The gatekeepers: Inside the admissions process of a premier college . Penguin . Jacques Steinberg. 2003. The gatekeepers: Inside the admissions process of a premier college .Penguin.","key":"e_1_2_2_33_1"},{"volume-title":"Qualitative analysis for social scientists","author":"Strauss Anselm L","unstructured":"Anselm L Strauss . 1987. Qualitative analysis for social scientists . Cambridge university press . Anselm L Strauss. 1987. Qualitative analysis for social scientists .Cambridge university press.","key":"e_1_2_2_34_1"},{"volume-title":"Cognitive Biases in Visualizations","author":"Sukumar Poorna Talkad","unstructured":"Poorna Talkad Sukumar and Ronald Metoyer . 2018. A Visualization Approach to Addressing Reviewer Bias in Holistic College Admissions . In Cognitive Biases in Visualizations . Springer , 161--175. Poorna Talkad Sukumar and Ronald Metoyer. 2018. A Visualization Approach to Addressing Reviewer Bias in Holistic College Admissions. In Cognitive Biases in Visualizations. Springer, 161--175.","key":"e_1_2_2_35_1"},{"key":"e_1_2_2_36_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"2","author":"Sukumar Poorna Talkad","year":"2018","unstructured":"Poorna Talkad Sukumar , Ronald Metoyer , and Shuai He . 2018 . Making a Pecan Pie: Understanding and Supporting The Holistic Review Process in Admissions . Proceedings of the ACM on Human-Computer Interaction , Vol. 2 , CSCW (2018), 169. Poorna Talkad Sukumar, Ronald Metoyer, and Shuai He. 2018. Making a Pecan Pie: Understanding and Supporting The Holistic Review Process in Admissions. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 169."},{"key":"e_1_2_2_37_1","volume-title":"Henny PA Boshuizen, and Paul A Kirschner","author":"Van Bruggen Jan M","year":"2003","unstructured":"Jan M Van Bruggen , Henny PA Boshuizen, and Paul A Kirschner . 2003 . A cognitive framework for cooperative problem solving with argument visualization. In Visualizing argumentation. Springer , 25--47. Jan M Van Bruggen, Henny PA Boshuizen, and Paul A Kirschner. 2003. A cognitive framework for cooperative problem solving with argument visualization. In Visualizing argumentation. Springer, 25--47."},{"key":"e_1_2_2_38_1","volume-title":"Workshop notes of the ECAI-06 workshop on computational models of natural argument. 67--75","author":"van den Braak Susan W","year":"2008","unstructured":"Susan W van den Braak , H van Oostendorp , Henry Prakken , and Gerard AW Vreeswijk . 2008 . A critical review of argument visualization tools: Do users become better reasoners? . In Workshop notes of the ECAI-06 workshop on computational models of natural argument. 67--75 . Susan W van den Braak, H van Oostendorp, Henry Prakken, and Gerard AW Vreeswijk. 2008. A critical review of argument visualization tools: Do users become better reasoners?. In Workshop notes of the ECAI-06 workshop on computational models of natural argument. 67--75."},{"key":"e_1_2_2_39_1","volume-title":"Proc. of the 2nd International Conference on Evidence Law and Forensic Science. 32--49","author":"Walton Douglas","year":"2009","unstructured":"Douglas Walton . 2009 . Argument visualization tools for corroborative evidence . In Proc. of the 2nd International Conference on Evidence Law and Forensic Science. 32--49 . Douglas Walton. 2009. Argument visualization tools for corroborative evidence. In Proc. of the 2nd International Conference on Evidence Law and Forensic Science. 32--49."},{"doi-asserted-by":"publisher","key":"e_1_2_2_40_1","DOI":"10.1002\/pra2.2015.145052010063"},{"doi-asserted-by":"publisher","key":"e_1_2_2_42_1","DOI":"10.1145\/3290605.3300499"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392870","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3392870","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:16Z","timestamp":1750200076000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392870"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,28]]},"references-count":41,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2020,5,28]]}},"alternative-id":["10.1145\/3392870"],"URL":"https:\/\/doi.org\/10.1145\/3392870","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2020,5,28]]},"assertion":[{"value":"2020-05-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}