{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:31:39Z","timestamp":1760661099740,"version":"build-2065373602"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"name":"Australian Research Council Discovery Grant","award":["DP190103414"],"award-info":[{"award-number":["DP190103414"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>High-stakes algorithmic decisions should be contestable, yet there is limited guidance on how to design for contestability, resulting in inadequate contestation processes. Using an algorithmic university admissions decision scenario, we investigate what type of review process people prefer when contesting algorithmic decisions. Through a choice-based conjoint experiment, we explore review process preferences, asking participants to choose their preferred review process from a choice of two that differ across five design attributes: who the reviewer is, how independent the review is, the ability to communicate with the reviewer, the style of the review, and how much the review costs. Expanding on existing studies that focus on the perspective of the decision subject, we introduce eight different scenario perspectives to explore the extent to which particular design attributes matter to different stakeholders.<\/jats:p>\n          <jats:p>People consistently prefer human reviewers who can make fresh decisions and communicate directly, while also prioritising affordable review processes. These findings highlight the importance of human involvement and cost-effectiveness in review processes. While the impact of perspective was relatively small, our qualitative analysis reveals important underlying tensions between efficiency, fairness, and individual needs. These subtle variations underscore the complexity of designing universally acceptable review processes. We propose design considerations that can help decision-makers to design review processes that are tailored to the specific decision-making context.<\/jats:p>","DOI":"10.1145\/3757468","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:32:00Z","timestamp":1760635920000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Diametrically Opposed Attributes? Review Process Preferences and Priorities When Contesting Algorithmic Decisions"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6570-4565","authenticated-orcid":false,"given":"Henrietta","family":"Lyons","sequence":"first","affiliation":[{"name":"School of Computing and Information Systems, University of Melbourne, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4908-6063","authenticated-orcid":false,"given":"Tim","family":"Miller","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science, Universtity of Queensland, Brisbane, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4414-2249","authenticated-orcid":false,"given":"Eduardo","family":"Velloso","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Sydney, Sydney, Australia"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"[n. d.]. Houston Federation of Teachers Local 2415 et al v Houston Independent School District 251 F.Supp.3d 116."},{"key":"e_1_2_1_2_1","volume-title":"Ofqual 'blindsided' government by revoking A-level appeals process. Retrieved","author":"Adams Richard","year":"2021","unstructured":"Richard Adams and Heather Stewart. 2020. Ofqual 'blindsided' government by revoking A-level appeals process. Retrieved May 4, 2021 from https:\/\/www.theguardian.com\/education\/2020\/aug\/16\/ofqual-blindsided-government-by-revoking-a-level-appeals-process."},{"key":"e_1_2_1_3_1","volume-title":"Contestable AI by Design: Towards a Framework. Minds and Machines","author":"Alfrink Kars","year":"2022","unstructured":"Kars Alfrink, Ianus Keller, Gerd Kortuem, and Neelke Doorn. 2022. Contestable AI by Design: Towards a Framework. Minds and Machines (2022), 1-27."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3582515.3609518"},{"key":"e_1_2_1_5_1","first-page":"2","volume-title":"Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law (ICAIL '19)","author":"Almada Marco","year":"2019","unstructured":"Marco Almada. 2019. Human Intervention in Automated Decision-making: Toward the Construction of Contestable Systems. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law (ICAIL '19). 2-11."},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Theo Araujo Natali Helberger Sanne Kruikemeier and Claes H De Vreese. 2020. In AI we trust? Perceptions about automated decision-making by artificial intelligence. AI & Society (2020) 1-13.","DOI":"10.1007\/s00146-019-00931-w"},{"key":"e_1_2_1_7_1","volume-title":"Australia's Artificial Intelligence Ethics Framework. Retrieved on","author":"Science Australian Government Department of Industry and Resources. 2024.","year":"2024","unstructured":"Science Australian Government Department of Industry and Resources. 2024. Australia's Artificial Intelligence Ethics Framework. Retrieved on 22 April 2024 from https:\/\/www.industry.gov.au\/publications\/australias-artificial-intelligence-ethics-framework."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1017\/pan.2022.16"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13347-017-0263-5"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173951"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_2_1_12_1","article-title":"Artificial Intelligence and Role-Reversible Judgment","volume":"109","author":"Brennan-Marquez Kiel","year":"2019","unstructured":"Kiel Brennan-Marquez and Stephen Henderson. 2019. Artificial Intelligence and Role-Reversible Judgment. Journal of Criminal Law and Criminology 109, 1, Article 2 (2019), 27 pages.","journal-title":"Journal of Criminal Law and Criminology"},{"key":"e_1_2_1_13_1","unstructured":"Lilah Burke. 2023. Should colleges use AI in admissions? https:\/\/www.highereddive.com\/news\/ai-admissions-ethics-college-university-higher-ed\/686588\/"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302304"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.86.3.425"},{"key":"e_1_2_1_16_1","unstructured":"Jason A Colquitt and Jessica B Rodell. 2015. Measuring justice and fairness. (2015)."},{"key":"e_1_2_1_17_1","volume-title":"Carolina Del Pino, Edson Pontes Pinto, Camila Barbosa, Diogo Massmann, Rodrigo Mambrini, Luiza Galv\u00e3o, Edmund Terem, et al.","author":"Corr\u00eaa Nicholas Kluge","year":"2023","unstructured":"Nicholas Kluge Corr\u00eaa, Camila Galv\u00e3o, James William Santos, Carolina Del Pino, Edson Pontes Pinto, Camila Barbosa, Diogo Massmann, Rodrigo Mambrini, Luiza Galv\u00e3o, Edmund Terem, et al. 2023. Worldwide AI ethics: A review of 200 guidelines and recommendations for AI governance. Patterns 4, 10 (2023)."},{"key":"e_1_2_1_18_1","volume-title":"The right to appeal. Retrieved on","author":"Courts and Tribunals Judiciary (United Kingdom). 2020.","year":"2020","unstructured":"Courts and Tribunals Judiciary (United Kingdom). 2020. The right to appeal. Retrieved on 31 March 2020 from https:\/\/www.judiciary.uk\/about-the-judiciary\/the-judiciary-the-government-and-the-constitution\/jud-acc-ind\/right-2-appeal\/."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1037\/xge0000033"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 24th International Conference on Intelligent User Interfaces (2019","author":"Dodge Jonathan","year":"2019","unstructured":"Jonathan Dodge, Q. Vera Liao, Yunfeng Zhang, Rachel K. E. Bellamy, and Casey Dugan. 2019. Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment. Proceedings of the 24th International Conference on Intelligent User Interfaces (2019). http:\/\/arxiv.org\/abs\/1901.07694"},{"key":"e_1_2_1_21_1","volume-title":"OECD Principles on Artificial Intelligence. Retrieved","author":"Development Organisation","year":"2020","unstructured":"Organisation for Economic Co-operation and Development. 2019. OECD Principles on Artificial Intelligence. Retrieved 24 February 2020 from https:\/\/www.oecd.org\/going-digital\/ai\/principles\/."},{"key":"e_1_2_1_22_1","volume-title":"Basic group processes","author":"Greenberg Jerald","unstructured":"Jerald Greenberg and Robert Folger. 1983. Procedural justice, participation, and the fair process effect in groups and organizations. In Basic group processes. Springer, 235-256."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1416587112"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1093\/pan\/mpt024"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","first-page":"107446","DOI":"10.1016\/j.chb.2022.107446","article-title":"Let the user have a say-voice in automated decision-making","volume":"138","author":"Hellwig Paul","year":"2023","unstructured":"Paul Hellwig, Victoria Buchholz, Stefan Kopp, and G\u00fcnter W Maier. 2023. Let the user have a say-voice in automated decision-making. Computers in Human Behavior 138 (2023), 107446.","journal-title":"Computers in Human Behavior"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 2017 Conference on Designing Interactive Systems. 95-99","author":"Hirsch Tad","unstructured":"Tad Hirsch, Kritzia Merced, Shrikanth Narayanan, Zac E. Imel, and David C. Atkins. 2017. Designing Contestability: Interaction Design, Machine Learning, and Mental Health. In Proceedings of the 2017 Conference on Designing Interactive Systems. 95-99."},{"key":"e_1_2_1_27_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1017\/S1930297500008275","article-title":"Veil-of-ignorance reasoning mitigates self-serving bias in resource allocation during the COVID-19 crisis","volume":"16","author":"Huang Karen","year":"2021","unstructured":"Karen Huang, Regan M Bernhard, Netta Barak-Corren, Max H Bazerman, and Joshua D Greene. 2021. Veil-of-ignorance reasoning mitigates self-serving bias in resource allocation during the COVID-19 crisis. Judgment and Decision Making 16, 1 (2021), 1-19.","journal-title":"Judgment and Decision Making"},{"key":"e_1_2_1_28_1","unstructured":"Scott Jaschik. 2023. Admissions Offices Cautiously Start Using AI. https:\/\/www.insidehighered.com\/news\/admissions\/2023\/05\/15\/admissions-offices-cautiously-start-using-ai"},{"key":"e_1_2_1_29_1","volume-title":"Twenty-Eighth European Conference on Information Systems (ECIS2020)","author":"Jussupow Ekaterina","year":"2020","unstructured":"Ekaterina Jussupow, Izak Benbasat, and Armin Heinzl. 2020. Why are we averse towards Algorithms? A comprehensive literature Review on Algorithm aversion.. In Twenty-Eighth European Conference on Information Systems (ECIS2020)."},{"key":"e_1_2_1_30_1","first-page":"1529","article-title":"Binary Governance: Lessons from the GDPR's Approach to Algorithmic Accountability","volume":"92","author":"Kaminski Margot","year":"2019","unstructured":"Margot Kaminski. 2019. Binary Governance: Lessons from the GDPR's Approach to Algorithmic Accountability. Southern California Law Review 92, 6 (2019), 1529-1616.","journal-title":"Southern California Law Review"},{"key":"e_1_2_1_31_1","first-page":"1957","article-title":"The right to contest AI","volume":"121","author":"Kaminski Margot E","year":"2021","unstructured":"Margot E Kaminski and Jennifer M Urban. 2021. The right to contest AI. Columbia Law Review 121, 7 (2021), 1957-2048.","journal-title":"Columbia Law Review"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1002\/berj.3705"},{"key":"e_1_2_1_33_1","article-title":"Automated decision support technologies and the legal profession","volume":"34","author":"Kluttz Daniel","year":"2020","unstructured":"Daniel Kluttz and Deirdre K Mulligan. 2020. Automated decision support technologies and the legal profession. Berkeley Technology Law Journal 34, 3 (2020).","journal-title":"Berkeley Technology Law Journal"},{"key":"e_1_2_1_34_1","volume-title":"What the world can learn from the UK's A-level grading fiasco. Retrieved","author":"Kolkman Daan","year":"2023","unstructured":"Daan Kolkman. 2020. ''F**k the algorithm''?: What the world can learn from the UK's A-level grading fiasco. Retrieved September 4, 2023 from https:\/\/blogs.lse.ac.uk\/impactofsocialsciences\/2020\/08\/26\/fk-the-algorithm-what-the-world-can-learn-from-the-uks-a-level-grading-fiasco\/."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278752"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1024959820557"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951718756684"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359284"},{"key":"e_1_2_1_39_1","unstructured":"Thomas J Leeper Matthew Barnfield and Maintainer Thomas J Leeper. 2018. Package ''cregg''."},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1017\/pan.2019.30","article-title":"Measuring subgroup preferences in conjoint experiments","volume":"28","author":"Leeper Thomas J","year":"2020","unstructured":"Thomas J Leeper, Sara B Hobolt, and James Tilley. 2020. Measuring subgroup preferences in conjoint experiments. Political Analysis 28, 2 (2020), 207-221.","journal-title":"Political Analysis"},{"key":"e_1_2_1_41_1","volume-title":"Willis R.H Gergen K.J.","author":"Leventhal Gerald","unstructured":"Gerald Leventhal. 1980. What Should Be Done With Equity Theory? In Social Exchange, Willis R.H Gergen K.J., Greenberg M.S. (Ed.). Plenum Press."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.59.5.952"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2018.12.005"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594041"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449180"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517606"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2021.121390"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372867"},{"key":"e_1_2_1_49_1","doi-asserted-by":"crossref","first-page":"66","DOI":"10.3390\/bs11050066","article-title":"Moral decision making: From bentham to veil of ignorance via perspective taking accessibility","volume":"11","author":"Martin Rose","year":"2021","unstructured":"Rose Martin, Petko Kusev, Joseph Teal, Victoria Baranova, and Bruce Rigal. 2021. Moral decision making: From bentham to veil of ignorance via perspective taking accessibility. Behavioral Sciences 11, 5 (2021), 66.","journal-title":"Behavioral Sciences"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2021.104666"},{"key":"e_1_2_1_51_1","volume-title":"Bygrave","author":"Mendoza Isak","year":"2017","unstructured":"Isak Mendoza and Lee A. Bygrave. 2017. The Right not to be Subject to Automated Decisions based on Profiling. In EU Internet Law: Regulation and Enforcement, Tatiani Synodinou, Philippe Jougleux, Christiana Markou, and Thalia Prastitou (Eds.). Springer."},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","first-page":"4366","DOI":"10.1177\/1461444818773059","article-title":"Censored, suspended, shadowbanned: User interpretations of content moderation on social media platforms","volume":"2","author":"West Sarah Myers","year":"2018","unstructured":"Sarah Myers West. 2018. Censored, suspended, shadowbanned: User interpretations of content moderation on social media platforms. New Media & Society 2, 11 (2018), 4366-4383.","journal-title":"New Media & Society"},{"key":"e_1_2_1_53_1","volume-title":"Ethics Guidelines for Trustworthy AI. Retrieved","author":"Independent High-Level Expert Group on Artificial Intelligence. 2019.","year":"2020","unstructured":"Independent High-Level Expert Group on Artificial Intelligence. 2019. Ethics Guidelines for Trustworthy AI. Retrieved 24 February 2020 from https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/ethics-guidelines-trustworthy-ai."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2018.07.022"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbef.2017.12.004"},{"key":"e_1_2_1_56_1","volume-title":"A Theory of Justice","author":"Rawls John","unstructured":"John Rawls. 1971. A Theory of Justice. Harvard University Press, Cambridge, MA."},{"key":"e_1_2_1_57_1","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1177\/0022022113505356","article-title":"Voice or consistency? What you perceive as procedurally fair depends on your level of power distance","volume":"45","author":"Summereder Susanne","year":"2014","unstructured":"Susanne Summereder, Bernhard Streicher, and Bernad Batinic. 2014. Voice or consistency? What you perceive as procedurally fair depends on your level of power distance. Journal of Cross-Cultural Psychology 45, 2 (2014), 192-212.","journal-title":"Journal of Cross-Cultural Psychology"},{"key":"e_1_2_1_58_1","volume-title":"Procedural Justice: A Psychological Analysis","author":"Thibaut John","year":"1975","unstructured":"John Thibaut and Laurens Walker. 1975. Procedural Justice: A Psychological Analysis. Lawrence Erlbaum Associates, Hillsdale, NJ."},{"key":"e_1_2_1_59_1","volume-title":"Advances in Experimental Social Psychology.","author":"Tyler Tom R","unstructured":"Tom R Tyler and E Allan Lind. 1992. A relational model of authority in groups. In Advances in Experimental Social Psychology. Vol. 25. Elsevier, 115-191."},{"key":"e_1_2_1_60_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 4, CSCW2","author":"Vaccaro Kristen","year":"2020","unstructured":"Kristen Vaccaro, Christian Sandvig, and Karrie Karahalios. 2020. ''At the End of the Day Facebook Does What It Wants'' How Users Experience Contesting Algorithmic Content Moderation. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1-22."},{"key":"e_1_2_1_61_1","first-page":"841","article-title":"Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR","volume":"31","author":"Wachter Sandra","year":"2018","unstructured":"Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law and Technology 31, 2 (2018), 841-887.","journal-title":"Harvard Journal of Law and Technology"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376813"},{"key":"e_1_2_1_63_1","volume-title":"Emre Bayamlioglu","author":"Yeung Karen","unstructured":"Karen Yeung and Adrian Weller. 2018. How is 'Transparency' Understood by Legal Scholars and the Machine Learning Community? In Being Profiled, Emre Bayamlioglu, Irina Baraliuc, Lissa Janssens, and Mireille Hildebrandt (Eds.). Amsterdam University Press."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581161"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757468","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:38:03Z","timestamp":1760636283000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757468"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":64,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757468"],"URL":"https:\/\/doi.org\/10.1145\/3757468","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}