{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T16:18:19Z","timestamp":1776097099109,"version":"3.50.1"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1816620"],"award-info":[{"award-number":["1816620"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Holistic reviews are a common practice employed by universities in the USA to make admissions decisions. It is an individualized review process where reviewers assess an applicant's potential by considering various criteria including academic metrics, adversities faced, and personal attributes. While the factors considered in such reviews are broadly known, a detailed walk-through of the process is absent in existing literature. This is important to understand what is done in practice and to identify opportunities for technological interventions to support the complex and changing process. We employed cognitive task analysis and a socio-organizational approach to understand the holistic review process at a highly-selective, private university. We found the process to be nuanced and complex owing its complexity both to the numerous variables involved and the reviewers' thought processes. We present a rigorous, structured characterization of the review process and suggest possible leverage points for applying visualization decision-support tools.<\/jats:p>","DOI":"10.1145\/3274438","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Making a Pecan Pie"],"prefix":"10.1145","volume":"2","author":[{"given":"Poorna","family":"Talkad Sukumar","sequence":"first","affiliation":[{"name":"University of Notre Dame, South Bend, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ronald","family":"Metoyer","sequence":"additional","affiliation":[{"name":"University of Notre Dame, South Bend, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shuai","family":"He","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Best Practices in Admissions Decisions","year":"2002","unstructured":"2002. Best Practices in Admissions Decisions. College Entrance Examination Board (2002)."},{"key":"e_1_2_1_2_1","unstructured":"Accessed: 09--12--2017. Apply to College with Common App | The Common Application. http:\/\/www.commonapp.org\/"},{"key":"e_1_2_1_3_1","volume-title":"09--12--2017","author":"Accessed","unstructured":"Accessed: 09--12--2017. College Admissions Advice - The Choice Blog - The New York Times. https:\/\/thechoice.blogs. nytimes.com\/?mcubz=3"},{"key":"e_1_2_1_4_1","unstructured":"Accessed: 09--12--2017. The Vandy Admissions Blog | Vanderbilt University. https:\/\/admissions.vanderbilt.edu\/ vandybloggers\/"},{"key":"e_1_2_1_5_1","unstructured":"Accessed: 09--19--2017. Coalition for Access & Affordability. http:\/\/www.coalitionforcollegeaccess.org\/"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753336"},{"key":"e_1_2_1_7_1","volume-title":"The theory of cognitive dissonance: A current perspective. Advances in experimental social psychology 4","author":"Aronson Elliot","year":"1969","unstructured":"Elliot Aronson. 1969. The theory of cognitive dissonance: A current perspective. Advances in experimental social psychology 4 (1969), 1--34."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.3102\/0162373716642407"},{"key":"e_1_2_1_9_1","volume-title":"The shape of the river: Long-term consequences of considering race in college and university admissions","author":"Bowen William G","unstructured":"William G Bowen and Derek Bok. 1999. The shape of the river: Long-term consequences of considering race in college and university admissions. Princeton University Press Princeton, NJ."},{"key":"e_1_2_1_10_1","volume-title":"Group Differences in Standardized Testing and Social Stratification. Report No. 99--5","author":"Camara Wayne J","year":"1999","unstructured":"Wayne J Camara and Amy Elizabeth Schmidt. 1999. Group Differences in Standardized Testing and Social Stratification. Report No. 99--5. College Entrance Examination Board (1999)."},{"key":"e_1_2_1_11_1","volume-title":"DOJ Looks Into Whether Harvard Discriminates Against Asian-Americans. (August","author":"Carapezza Kirk","year":"2017","unstructured":"Kirk Carapezza. August, 2017. DOJ Looks Into Whether Harvard Discriminates Against Asian-Americans. (August, 2017). Retrieved from http:\/\/www.npr.org\/2017\/08\/03\/541430130\/ trump-admin-looking-into-whether-harvard-discriminates-against-asian-americans."},{"key":"e_1_2_1_12_1","volume-title":"Cognitive task analysis. Spector","author":"Clark Richard E","year":"2006","unstructured":"Richard E Clark, D Feldon, J Van Merrienboer, Kenneth Yates, and Sean Early. 2006. Cognitive task analysis. Spector, JM, Merrill, MD, van Merri\u00ebnboer, JJ G., & Driscoll, MP (Eds.) Handbook of research on educational communications and technology 3 (2006)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.1994.1083"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1203012"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124855"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/3159007"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/143457.143468"},{"key":"e_1_2_1_18_1","volume-title":"Graphesis: Visual forms of knowledge production","author":"Drucker Johanna","year":"2014","unstructured":"Johanna Drucker. 2014. Graphesis: Visual forms of knowledge production. Harvard University Press."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-70956-5_1"},{"key":"e_1_2_1_20_1","volume-title":"The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges-and Who Gets Left Outside the Gates","author":"Golden Daniel","unstructured":"Daniel Golden. 2007. The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges-and Who Gets Left Outside the Gates. Broadway Books."},{"key":"e_1_2_1_21_1","volume-title":"Contextual inquiry: A participatory technique for system design. Participatory design: Principles and practices","author":"Holtzblatt Karen","year":"1993","unstructured":"Karen Holtzblatt and Sandra Jones. 1993. Contextual inquiry: A participatory technique for system design. Participatory design: Principles and practices (1993), 177--210."},{"key":"e_1_2_1_22_1","volume-title":"fast and slow","author":"Kahneman Daniel","unstructured":"Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan."},{"key":"e_1_2_1_23_1","volume-title":"Conditions for intuitive expertise: a failure to disagree. American psychologist 64, 6","author":"Kahneman Daniel","year":"2009","unstructured":"Daniel Kahneman and Gary Klein. 2009. Conditions for intuitive expertise: a failure to disagree. American psychologist 64, 6 (2009), 515."},{"key":"e_1_2_1_24_1","unstructured":"Anne Kim. 2016. How the Internet Wrecked College Admissions. (2016). Retrieved from https:\/\/www.theatlantic.com\/ education\/archive\/2016\/10\/how-the-internet-wrecked-college-admissions\/504062\/."},{"key":"e_1_2_1_25_1","volume-title":"Storytelling with data: A data visualization guide for business professionals","author":"Knaflic Cole Nussbaumer","unstructured":"Cole Nussbaumer Knaflic. 2015. Storytelling with data: A data visualization guide for business professionals. John Wiley & Sons."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2013.36"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/765891.765961"},{"key":"e_1_2_1_28_1","volume-title":"How Admission Decisions Get Made. Handbook of strategic enrollment management","author":"Lucido Jerome A","year":"2014","unstructured":"Jerome A Lucido. 2014. How Admission Decisions Get Made. Handbook of strategic enrollment management (2014), 147--173."},{"key":"e_1_2_1_29_1","unstructured":"Alan M MacEachren. 2015. Visual analytics and uncertainty: Its not about the data. (2015)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.199"},{"key":"e_1_2_1_31_1","volume-title":"Visualization analysis and design","author":"Munzner Tamara","unstructured":"Tamara Munzner. 2014. Visualization analysis and design. CRC press."},{"key":"e_1_2_1_32_1","volume-title":"Confirmation bias: A ubiquitous phenomenon in many guises. Review of general psychology 2, 2","author":"Nickerson Raymond S","year":"1998","unstructured":"Raymond S Nickerson. 1998. Confirmation bias: A ubiquitous phenomenon in many guises. Review of general psychology 2, 2 (1998), 175."},{"key":"e_1_2_1_33_1","volume-title":"Mario P\u00e9rez-Montoro Guti\u00e9rrez, and Mitchell Whitelan","author":"Nualart Jaume","year":"2014","unstructured":"Jaume Nualart, Mario P\u00e9rez-Montoro Guti\u00e9rrez, and Mitchell Whitelan. 2014. How we draw texts: a review of approaches to text visualization and exploration. El Profesional de la Informaci\u00f3n, 2014, vol. 23, num. 3, p. 221--235 (2014)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1057\/palgrave.ejis.3000639"},{"key":"e_1_2_1_35_1","volume-title":"Paul KJ Han, and Remco Chang","author":"Ottley Alvitta","year":"2016","unstructured":"Alvitta Ottley, Evan M Peck, Lane T Harrison, Daniel Afergan, Caroline Ziemkiewicz, Holly A Taylor, Paul KJ Han, and Remco Chang. 2016. Improving Bayesian reasoning: The effects of phrasing, visualization, and spatial ability. IEEE transactions on visualization and computer graphics 22, 1 (2016), 529--538."},{"key":"e_1_2_1_36_1","volume-title":"Toward a Taxonomy of the Admissions Decision-Making Process: A Public Document Based on the First and Second College Board Conferences on Admissions Models","author":"Perfetto Greg","year":"1999","unstructured":"Greg Perfetto, M\u00e9rida Escand\u00f3n, Steve Graff, Gretchen Rigol, and Amy Schmidt. 1999. Toward a Taxonomy of the Admissions Decision-Making Process: A Public Document Based on the First and Second College Board Conferences on Admissions Models. College Entrance Examination Board (1999)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2010.216"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of international conference on intelligence analysis","volume":"5","author":"Pirolli Peter","year":"2005","unstructured":"Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. 2--4."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00752449"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2010.179"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357247"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1466620.1466622"},{"key":"e_1_2_1_43_1","volume-title":"The Gatekeepers: Inside the Admissions Process of a Premier College","author":"Steinberg Jacques","year":"2003","unstructured":"Jacques Steinberg. 2003. The Gatekeepers: Inside the Admissions Process of a Premier College. Penguin."},{"key":"e_1_2_1_44_1","volume-title":"Cognitive Biases in Visualizations","author":"Sukumar Poorna Talkad","unstructured":"Poorna Talkad Sukumar and Ronald Metoyer. 2018. A Visualization Approach to Addressing Reviewer Bias in Holistic College Admissions. In Cognitive Biases in Visualizations, Geoffrey Ellis (Ed.). Springer International Publishing, Chapter 12."},{"key":"e_1_2_1_45_1","volume-title":"IEEE VIS","author":"Sukumar Poorna Talkad","year":"2017","unstructured":"Poorna Talkad Sukumar, Ronald Metoyer, and Shuai He. 2017. Holistic Reviews in Admissions: Reviewer Biases and Visualization Strategies to Mitigate Them. In IEEE VIS 2017."},{"key":"e_1_2_1_46_1","unstructured":"B Alden Thresher. 1966. College Admissions and the Public Interest. (1966)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1037\/e578902012-079"},{"key":"e_1_2_1_48_1","volume-title":"Utility, probability, and human decision making","author":"Tversky Amos","unstructured":"Amos Tversky and Daniel Kahneman. 1975. Judgment under uncertainty: Heuristics and biases. In Utility, probability, and human decision making. Springer, 141--162."},{"key":"e_1_2_1_49_1","volume-title":"IEEE VIS","author":"Valdez Andr\u00e9 Caldero","year":"2017","unstructured":"Andr\u00e9 Caldero Valdez, Martina Ziefle, and Michael Sedlmair. 2017. A Framework for Studying Biases in Visualization Research. In IEEE VIS 2017."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/IV.2007.108"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124890"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274438","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274438","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274438","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:36Z","timestamp":1750207476000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274438"}},"subtitle":["Understanding and Supporting The Holistic Review Process in Admissions"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":51,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274438"],"URL":"https:\/\/doi.org\/10.1145\/3274438","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}