{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T10:53:45Z","timestamp":1779101625431,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":37,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,10,14]],"date-time":"2023-10-14T00:00:00Z","timestamp":1697241600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF (National Science Foundation)","doi-asserted-by":"publisher","award":["2040942"],"award-info":[{"award-number":["2040942"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,10,14]]},"DOI":"10.1145\/3584931.3611279","type":"proceedings-article","created":{"date-parts":[[2023,10,13]],"date-time":"2023-10-13T22:23:10Z","timestamp":1697235790000},"page":"556-559","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Supporting User Engagement in Testing, Auditing, and Contesting AI"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3375-5285","authenticated-orcid":false,"given":"Wesley Hanwen","family":"Deng","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3448-5961","authenticated-orcid":false,"given":"Michelle S.","family":"Lam","sequence":"additional","affiliation":[{"name":"Stanford University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0348-3362","authenticated-orcid":false,"given":"\u00c1ngel Alexander","family":"Cabrera","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9359-6090","authenticated-orcid":false,"given":"Dana\u00eb","family":"Metaxa","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1499-3045","authenticated-orcid":false,"given":"Motahhare","family":"Eslami","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6730-922X","authenticated-orcid":false,"given":"Kenneth","family":"Holstein","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,10,14]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Open AI. 2022. ChatGPT Feedback Contest: Official Rules. https:\/\/cdn.openai.com\/chatgpt\/ChatGPT_Feedback_Contest_Rules.pdf  Open AI. 2022. ChatGPT Feedback Contest: Official Rules. https:\/\/cdn.openai.com\/chatgpt\/ChatGPT_Feedback_Contest_Rules.pdf"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580984"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2700832"},{"key":"e_1_3_2_1_4_1","volume-title":"Conference on fairness, accountability and transparency. PMLR, 77\u201391","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on fairness, accountability and transparency. PMLR, 77\u201391 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77\u201391."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479569"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359249"},{"key":"e_1_3_2_1_7_1","volume-title":"Introducing Twitter\u2019s first algorithmic bias bounty challenge. URl: https:\/\/blog. twitter. com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bountychallenge","author":"Chowdhury Rumman","year":"2021","unstructured":"Rumman Chowdhury and Jutta Williams . 2021. Introducing Twitter\u2019s first algorithmic bias bounty challenge. URl: https:\/\/blog. twitter. com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bountychallenge ( 2021 ). Rumman Chowdhury and Jutta Williams. 2021. Introducing Twitter\u2019s first algorithmic bias bounty challenge. URl: https:\/\/blog. twitter. com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bountychallenge (2021)."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278156"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581026"},{"key":"e_1_3_2_1_10_1","volume-title":"Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, Seoul Republic of Korea, 473\u2013484","author":"Deng Wesley\u00a0Hanwen","year":"2022","unstructured":"Wesley\u00a0Hanwen Deng , Manish Nagireddy , Michelle Seng\u00a0Ah Lee , Jatinder Singh , Zhiwei\u00a0Steven Wu , Kenneth Holstein , and Haiyi Zhu . 2022 . Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, Seoul Republic of Korea, 473\u2013484 . https:\/\/doi.org\/10.1145\/3531146.3533113 10.1145\/3531146.3533113 Wesley\u00a0Hanwen Deng, Manish Nagireddy, Michelle Seng\u00a0Ah Lee, Jatinder Singh, Zhiwei\u00a0Steven Wu, Kenneth Holstein, and Haiyi Zhu. 2022. Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, Seoul Republic of Korea, 473\u2013484. https:\/\/doi.org\/10.1145\/3531146.3533113"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594037"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517441"},{"key":"e_1_3_2_1_13_1","unstructured":"White House. 2022. Blueprint for an AI Bill of Rights. https:\/\/www.whitehouse.gov\/ostp\/ai-bill-of-rights\/  White House. 2022. Blueprint for an AI Bill of Rights. https:\/\/www.whitehouse.gov\/ostp\/ai-bill-of-rights\/"},{"key":"e_1_3_2_1_14_1","unstructured":"Nicolas Kaufmann Thimo Schulze and Daniel Veit. 2011. More than fun and money. worker motivation in crowdsourcing\u2013a study on mechanical turk. (2011).  Nicolas Kaufmann Thimo Schulze and Daniel Veit. 2011. More than fun and money. worker motivation in crowdsourcing\u2013a study on mechanical turk. (2011)."},{"key":"e_1_3_2_1_15_1","volume-title":"Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337","author":"Kiela Douwe","year":"2021","unstructured":"Douwe Kiela , Max Bartolo , Yixin Nie , Divyansh Kaushik , Atticus Geiger , Zhengxuan Wu , Bertie Vidgen , Grusha Prasad , Amanpreet Singh , Pratik Ringshia , 2021 . Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337 (2021). Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, 2021. Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337 (2021)."},{"key":"e_1_3_2_1_16_1","volume-title":"Understanding Frontline Workers","author":"Kuo Tzu-Sheng","year":"2023","unstructured":"Tzu-Sheng Kuo , Hong Shen , Jisoo Geum , Nev Jones , Jason\u00a0 I Hong , Haiyi Zhu , and Kenneth Holstein . 2023. Understanding Frontline Workers \u2019 and Unhoused Individuals\u2019 Perspectives on AI Used in Homeless Services. arXiv preprint arXiv:2303.09743 ( 2023 ). Tzu-Sheng Kuo, Hong Shen, Jisoo Geum, Nev Jones, Jason\u00a0I Hong, Haiyi Zhu, and Kenneth Holstein. 2023. Understanding Frontline Workers\u2019 and Unhoused Individuals\u2019 Perspectives on AI Used in Homeless Services. arXiv preprint arXiv:2303.09743 (2023)."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555625"},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW","author":"Noothigattu Ritesh","year":"2019","unstructured":"Min\u00a0Kyung Lee, Daniel Kusbit, Anson Kahng, Ji\u00a0Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu , Siheon Lee , Alexandros Psomas , 2019 . WeBuildAI: Participatory framework for algorithmic governance . Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1\u201335. Min\u00a0Kyung Lee, Daniel Kusbit, Anson Kahng, Ji\u00a0Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1\u201335."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3582074"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376727"},{"key":"e_1_3_2_1_21_1","volume-title":"Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends\u00ae in Human\u2013Computer Interaction 14, 4","author":"Metaxa Dana\u00eb","year":"2021","unstructured":"Dana\u00eb Metaxa , Joon\u00a0Sung Park , Ronald\u00a0 E Robertson , Karrie Karahalios , Christo Wilson , Jeff Hancock , Christian Sandvig , 2021. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends\u00ae in Human\u2013Computer Interaction 14, 4 ( 2021 ), 272\u2013344. Dana\u00eb Metaxa, Joon\u00a0Sung Park, Ronald\u00a0E Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, Christian Sandvig, 2021. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends\u00ae in Human\u2013Computer Interaction 14, 4 (2021), 272\u2013344."},{"key":"e_1_3_2_1_22_1","unstructured":"NIST. 2023. Artificial Intelligence Risk Management Framework. https:\/\/www.nist.gov\/itl\/ai-risk-management-framework  NIST. 2023. Artificial Intelligence Risk Management Framework. https:\/\/www.nist.gov\/itl\/ai-risk-management-framework"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13337"},{"key":"e_1_3_2_1_24_1","volume-title":"Search Atlas: Visualizing Divergent Search Results Across Geopolitical Borders. In Designing Interactive Systems Conference 2021. 1970","author":"Ochigame Rodrigo","year":"2021","unstructured":"Rodrigo Ochigame and Katherine Ye . 2021 . Search Atlas: Visualizing Divergent Search Results Across Geopolitical Borders. In Designing Interactive Systems Conference 2021. 1970 \u20131983. Rodrigo Ochigame and Katherine Ye. 2021. Search Atlas: Visualizing Divergent Search Results Across Geopolitical Borders. In Designing Interactive Systems Conference 2021. 1970\u20131983."},{"key":"e_1_3_2_1_25_1","unstructured":"Giada Pistilli. 2022. HuggingFace announcedthe new feature to flag any Model Dataset or Space on the Hub. https:\/\/twitter.com\/GiadaPistilli\/status\/1571865167092396033?s=20&t=LRhhEu63s6ftPmtZdfz8Cw  Giada Pistilli. 2022. HuggingFace announcedthe new feature to flag any Model Dataset or Space on the Hub. https:\/\/twitter.com\/GiadaPistilli\/status\/1571865167092396033?s=20&t=LRhhEu63s6ftPmtZdfz8Cw"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_3_2_1_28_1","volume-title":"Supporting Human-AI Collaboration in Auditing LLMs with LLMs. arXiv preprint arXiv:2304.09991","author":"Rastogi Charvi","year":"2023","unstructured":"Charvi Rastogi , Marco\u00a0Tulio Ribeiro , Nicholas King , and Saleema Amershi . 2023. Supporting Human-AI Collaboration in Auditing LLMs with LLMs. arXiv preprint arXiv:2304.09991 ( 2023 ). Charvi Rastogi, Marco\u00a0Tulio Ribeiro, Nicholas King, and Saleema Amershi. 2023. Supporting Human-AI Collaboration in Auditing LLMs with LLMs. arXiv preprint arXiv:2304.09991 (2023)."},{"key":"e_1_3_2_1_29_1","volume-title":"Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig , Kevin Hamilton , Karrie Karahalios , and Cedric Langbort . 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry ( 2014 ). Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry (2014)."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479577"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533110"},{"key":"e_1_3_2_1_32_1","unstructured":"Rumman\u00a0Chowdhury Sven\u00a0Cattell and Austin Carson. 2023. AI Village at DEF CON announces largest-ever public Generative AI Red Team. https:\/\/aivillage.org\/  Rumman\u00a0Chowdhury Sven\u00a0Cattell and Austin Carson. 2023. AI Village at DEF CON announces largest-ever public Generative AI Red Team. https:\/\/aivillage.org\/"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460276.2460278"},{"key":"e_1_3_2_1_34_1","volume-title":"Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges.","author":"Wang Qiaosi","year":"2023","unstructured":"Qiaosi Wang , Michael\u00a0Adam Madaio , Shivani Kapania , Shaun Kane , Michael Terry , Lauren Wilcox , 2023 . Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges. (2023). Qiaosi Wang, Michael\u00a0Adam Madaio, Shivani Kapania, Shaun Kane, Michael Terry, Lauren Wilcox, 2023. Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges. (2023)."},{"key":"e_1_3_2_1_35_1","unstructured":"Tris Warkentin and Josh Woodward. 2022. AI Test Kitchen. https:\/\/blog.google\/technology\/ai\/join-us-in-the-ai-test-kitchen\/  Tris Warkentin and Josh Woodward. 2022. AI Test Kitchen. https:\/\/blog.google\/technology\/ai\/join-us-in-the-ai-test-kitchen\/"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580900"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"}],"event":{"name":"CSCW '23: Computer Supported Cooperative Work and Social Computing","location":"Minneapolis MN USA","acronym":"CSCW '23","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Computer Supported Cooperative Work and Social Computing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3584931.3611279","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3584931.3611279","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:07Z","timestamp":1750178227000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3584931.3611279"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,14]]},"references-count":37,"alternative-id":["10.1145\/3584931.3611279","10.1145\/3584931"],"URL":"https:\/\/doi.org\/10.1145\/3584931.3611279","relation":{},"subject":[],"published":{"date-parts":[[2023,10,14]]},"assertion":[{"value":"2023-10-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}