{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T23:12:34Z","timestamp":1776121954500,"version":"3.50.1"},"reference-count":133,"publisher":"Association for Computing Machinery (ACM)","issue":"7","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>There has been growing interest from both practitioners and researchers in engaging end users in AI auditing, to draw upon users' unique knowledge and lived experiences. However, we know little about how to effectively scaffold end users in auditing in ways that can generate actionable insights for AI practitioners. Through formative studies with both users and AI practitioners, we first identified a set of design goals to support user-engaged AI auditing. We then developed WeAudit, a workflow and system that supports end users in auditing AI both individually and collectively. We evaluated WeAudit through a three-week user study with user auditors and interviews with industry Generative AI practitioners. Our findings offer insights into how WeAudit supports users in noticing and reflecting upon potential AI harms and in articulating their findings in ways that industry practitioners can act upon. Based on our observations and feedback from both users and practitioners, we identify several opportunities to better support user engagement in AI auditing processes. We discuss implications for future research to support effective and responsible user engagement in AI auditing.<\/jats:p>","DOI":"10.1145\/3757702","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:32:00Z","timestamp":1760635920000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3375-5285","authenticated-orcid":false,"given":"Wesley Hanwen","family":"Deng","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-3562-055X","authenticated-orcid":false,"given":"Wang","family":"Claire","sequence":"additional","affiliation":[{"name":"University of California, Berkeley, Berkeley, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-5556-7297","authenticated-orcid":false,"given":"Howard Ziyu","family":"Han","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9856-9654","authenticated-orcid":false,"given":"Jason I.","family":"Hong","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6730-922X","authenticated-orcid":false,"given":"Kenneth","family":"Holstein","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1499-3045","authenticated-orcid":false,"given":"Motahhare","family":"Eslami","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Open AI. 2022. ChatGPT Feedback Contest: Official Rules. https:\/\/cdn.openai.com\/chatgpt\/ChatGPT_Feedback_Contest_Rules.pdf"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357311"},{"key":"e_1_2_1_3_1","unstructured":"Anthropic. 2023. Model card and evaluations for Claude Models. https:\/\/www-cdn.anthropic.com\/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226\/Model-Card-Claude-2.pdf"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v14i1.7276"},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2700832","article-title":"Beat the machine: Challenging humans to find a predictive model's ''unknown unknowns","volume":"6","author":"Attenberg Joshua","year":"2015","unstructured":"Joshua Attenberg, Panos Ipeirotis, and Foster Provost. 2015. Beat the machine: Challenging humans to find a predictive model's ''unknown unknowns''. Journal of Data and Information Quality (JDIQ), Vol. 6, 1 (2015), 1-17.","journal-title":"Journal of Data and Information Quality (JDIQ)"},{"key":"e_1_2_1_6_1","first-page":"1","volume-title":"Proceedings of the acm on human-computer interaction","volume":"5","author":"Bandy Jack","year":"2021","unstructured":"Jack Bandy. 2021. Problematic machine behavior: A systematic literature review of algorithm audits. Proceedings of the acm on human-computer interaction, Vol. 5, CSCW1 (2021), 1-34."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594095"},{"key":"e_1_2_1_9_1","volume-title":"Human-computer interaction and collective intelligence. Handbook of collective intelligence","author":"Bigham Jeffrey P","year":"2015","unstructured":"Jeffrey P Bigham, Michael S Bernstein, and Eytan Adar. 2015. Human-computer interaction and collective intelligence. Handbook of collective intelligence, Vol. 57 (2015)."},{"key":"e_1_2_1_10_1","volume-title":"2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 612-643","author":"Birhane Abeba","year":"2024","unstructured":"Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, and Inioluwa Deborah Raji. 2024. AI auditing: The broken bus on the road to AI accountability. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 612-643."},{"key":"e_1_2_1_11_1","unstructured":"Miranda Bogen and Amy A Winecoff. [n.d.]. https:\/\/cdt.org\/insights\/applying-sociotechnical-approaches-to-ai-governance-in-practice\/"},{"key":"e_1_2_1_12_1","volume-title":"Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health","author":"Braun Virginia","year":"2019","unstructured":"Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health, Vol. 11, 4 (2019), 589-597."},{"key":"e_1_2_1_13_1","volume-title":"Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, and Saleema Amershi.","author":"Bu\u00e7inca Zana","year":"2023","unstructured":"Zana Bu\u00e7inca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, and Saleema Amershi. 2023. AHA!: Facilitating AI Impact Assessment by Generating Examples of Harms. arXiv preprint arXiv:2306.03280 (2023)."},{"key":"e_1_2_1_14_1","volume-title":"Conference on fairness, accountability and transparency. PMLR, 77-91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77-91."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979023"},{"key":"e_1_2_1_16_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"5","author":"Cabrera \u00c1ngel Alexander","year":"2021","unstructured":"\u00c1ngel Alexander Cabrera, Abraham J Druck, Jason I Hong, and Adam Perer. 2021. Discovering and validating ai errors with crowdsourced failure reports. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1-22."},{"key":"e_1_2_1_17_1","volume-title":"Bongshin Lee, Rob DeLine, Adam Perer, and Steven M Drucker.","author":"Cabrera \u00c1ngel Alexander","year":"2022","unstructured":"\u00c1ngel Alexander Cabrera, Marco Tulio Ribeiro, Bongshin Lee, Rob DeLine, Adam Perer, and Steven M Drucker. 2022. What Did My AI Learn? How Data Scientists Make Sense of Model Behavior. ACM Transactions on Computer-Human Interaction (2022)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/1744987120927206"},{"key":"e_1_2_1_19_1","volume-title":"Tom Hope, Dafna Shahaf, and Aniket Kittur.","author":"Chan Joel","year":"2018","unstructured":"Joel Chan, Joseph Chee Chang, Tom Hope, Dafna Shahaf, and Aniket Kittur. 2018. Solvent: A mixed initiative system for finding analogies between research papers. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 1-21."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 23rd International Conference on Intelligent User Interfaces. 269-280","author":"Chen Nan-Chen","year":"2018","unstructured":"Nan-Chen Chen, Jina Suh, Johan Verwey, Gonzalo Ramos, Steven Drucker, and Patrice Simard. 2018. AnchorViz: Facilitating classifier error discovery through interactive semantic data exploration. In Proceedings of the 23rd International Conference on Intelligent User Interfaces. 269-280."},{"key":"e_1_2_1_21_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"7","author":"Chen Quan Ze","year":"2023","unstructured":"Quan Ze Chen and Amy X Zhang. 2023. Judgment Sieve: Reducing uncertainty in group judgments through interventions targeting ambiguity versus disagreement. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW2 (2023), 1-26."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_1_23_1","volume-title":"Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi.","author":"Chiu Yu Ying","year":"2024","unstructured":"Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024. CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs'(Lack of) Multicultural Knowledge. arXiv preprint arXiv:2404.06664 (2024)."},{"key":"e_1_2_1_24_1","volume-title":"Introducing Twitter's first algorithmic bias bounty challenge. URl: https:\/\/blog. twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bountychallenge","author":"Chowdhury Rumman","year":"2021","unstructured":"Rumman Chowdhury and Jutta Williams. 2021. Introducing Twitter's first algorithmic bias bounty challenge. URl: https:\/\/blog. twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bountychallenge (2021)."},{"key":"e_1_2_1_25_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"3","author":"Young Chung John Joon","year":"2019","unstructured":"John Joon Young Chung, Jean Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter S Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction, Vol. 3, CSCW (2019), 1-25."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2023301118"},{"key":"e_1_2_1_27_1","volume-title":"Work in Progress of the AAAI Conference on Human Computation and Crowdsourcing","author":"Claire Wang","year":"2024","unstructured":"Wang Claire, Wesley Hanwen Deng, Jason Hong, Ken Holstein, and Motahhare Eslami. 2024. Designing a Crowdsourcing Pipeline to Verify Reports from User AI Audits. Work in Progress of the AAAI Conference on Human Computation and Crowdsourcing (2024)."},{"key":"e_1_2_1_28_1","unstructured":"DEF CON. 2024. Red Team Village. https:\/\/redteamvillage.io\/"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1216295.1216309"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278156"},{"key":"e_1_2_1_31_1","volume-title":"Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work. arXiv preprint arXiv:2408.01057","author":"Deng Wesley Hanwen","year":"2024","unstructured":"Wesley Hanwen Deng, Solon Barocas, and Jennifer Wortman Vaughan. 2024a. Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work. arXiv preprint arXiv:2408.01057 (2024)."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581026"},{"key":"e_1_2_1_33_1","volume-title":"Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing. 556-559","author":"Deng Wesley Hanwen","year":"2023","unstructured":"Wesley Hanwen Deng, Michelle S Lam, \u00c1ngel Alexander Cabrera, Dana\u00eb Metaxa, Motahhare Eslami, and Kenneth Holstein. 2023b. Supporting user engagement in testing, auditing, and contesting AI. In Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing. 556-559."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533113"},{"key":"e_1_2_1_35_1","first-page":"148","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"12","author":"Deng Wesley Hanwen","year":"2024","unstructured":"Wesley Hanwen Deng, Mireia Yurrita, Mark D\u00edaz, Jina Suh, Nick Judd, Lara Groves, Hong Shen, Motahhare Eslami, and Kenneth Holstein. 2024b. Responsible Crowdsourcing for Responsible Generative AI: Engaging Crowds in AI Auditing and Evaluation. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 12. 148-150."},{"key":"e_1_2_1_36_1","volume-title":"Building stereotype repositories with llms and community engagement for scale and depth. Cross-Cultural Considerations in NLP@ EACL","author":"Dev Sunipa","year":"2023","unstructured":"Sunipa Dev, Akshita Jha, Jaya Goyal, Dinesh Tewari, Shachi Dave, and Vinodkumar Prabhakaran. 2023. Building stereotype repositories with llms and community engagement for scale and depth. Cross-Cultural Considerations in NLP@ EACL, Vol. 84 (2023)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517441"},{"key":"e_1_2_1_38_1","volume-title":"The 2024 ACM Conference on Fairness, Accountability, and Transparency. 1093-1106","author":"DeVrio Alicia","year":"2024","unstructured":"Alicia DeVrio, Motahhare Eslami, and Kenneth Holstein. 2024. Building, Shifting, & Employing Power: A Taxonomy of Responses From Below to Algorithmic Harm. In The 2024 ACM Conference on Fairness, Accountability, and Transparency. 1093-1106."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300372"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376545"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_42_1","volume-title":"Be careful","author":"Eslami Motahhare","year":"2017","unstructured":"Motahhare Eslami, Kristen Vaccaro, Karrie Karahalios, and Kevin Hamilton. 2017. Be careful; Things can be worse than they appear - Understanding biased algorithms and users' behavior around them in rating platforms''. (2017), 62-71. Funding Information: This work was funded by NSF grant CHS-1564041. Publisher Copyright: textcopyright Copyright 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.; 11th International Conference on Web and Social Media, ICWSM 2017 ; Conference date: 15-05-2017 Through 18-05-2017."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300724"},{"key":"e_1_2_1_44_1","first-page":"421","volume-title":"Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society","volume":"7","author":"Feffer Michael","year":"2024","unstructured":"Michael Feffer, Anusha Sinha, Wesley H Deng, Zachary C Lipton, and Hoda Heidari. 2024. Red-teaming for generative ai: Silver bullet or security theater?. In Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, Vol. 7. 421-437."},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the 12th international conference on Intelligent user interfaces. 52-61","author":"Freyne Jill","year":"2007","unstructured":"Jill Freyne, Rosta Farzan, Peter Brusilovsky, Barry Smyth, and Maurice Coyle. 2007. Collecting community wisdom: integrating social search & social navigation. In Proceedings of the 12th international conference on Intelligent user interfaces. 52-61."},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-21","author":"Gero Katy Ilonka","year":"2024","unstructured":"Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K Kummerfeld, and Elena L Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-21."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_2_1_48_1","volume-title":"Ghost work: How to stop Silicon Valley from building a new global underclass","author":"Gray Mary L","unstructured":"Mary L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594071"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663716.2663744"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174023"},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"Kenneth Holstein Bruce M McLaren and Vincent Aleven. 2019. Designing for complementarity: Teacher and student needs for orchestration support in AI-enhanced classrooms. (2019) 157-171.","DOI":"10.1007\/978-3-030-23204-7_14"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3706599.3719757"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3586183.3606737"},{"key":"e_1_2_1_56_1","unstructured":"Nicolas Kaufmann Thimo Schulze and Daniel Veit. 2011. More than fun and money. worker motivation in crowdsourcing-a study on mechanical turk. (2011)."},{"key":"e_1_2_1_57_1","volume-title":"Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337","author":"Kiela Douwe","year":"2021","unstructured":"Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al., 2021. Dynabench: Rethinking benchmarking in NLP. arXiv preprint arXiv:2104.14337 (2021)."},{"key":"e_1_2_1_58_1","volume-title":"Anticipating impacts: using large-scale scenario-writing to explore diverse implications of generative AI in the news environment. AI and Ethics","author":"Kieslich Kimon","year":"2024","unstructured":"Kimon Kieslich, Nicholas Diakopoulos, and Natali Helberger. 2024. Anticipating impacts: using large-scale scenario-writing to explore diverse implications of generative AI in the news environment. AI and Ethics (2024), 1-23."},{"key":"e_1_2_1_59_1","first-page":"75","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"12","author":"Kingsley Sara","year":"2024","unstructured":"Sara Kingsley, Jiayin Zhi, Wesley Hanwen Deng, Jaimie Lee, Sizhe Zhang, Motahhare Eslami, Kenneth Holstein, Jason I Hong, Tianshi Li, and Hong Shen. 2024. Investigating What Factors Influence Users' Rating of Harmful Algorithmic Bias and Discrimination. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 12. 75-85."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531644"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1002\/acp.1699"},{"key":"e_1_2_1_63_1","volume-title":"Encouraging contribution to online communities. Building successful online communities: Evidence-based social design","author":"Kraut Robert E","year":"2011","unstructured":"Robert E Kraut and Paul Resnick. 2011. Encouraging contribution to online communities. Building successful online communities: Evidence-based social design (2011), 21-76."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145249"},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-24","author":"Kuo Tzu-Sheng","year":"2024","unstructured":"Tzu-Sheng Kuo, Aaron Lee Halfaker, Zirui Cheng, Jiwoo Kim, Meng-Hsin Wu, Tongshuang Wu, Kenneth Holstein, and Haiyi Zhu. 2024. Wikibench: Community-driven data curation for AI evaluation on Wikipedia. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-24."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555625"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753616"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3582074"},{"key":"e_1_2_1_69_1","unstructured":"Mun Ling Lo. 2012. Variation theory and the improvement of teaching and learning. G\u00f6teborg: Acta Universitatis Gothoburgensis."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642149"},{"key":"e_1_2_1_71_1","unstructured":"Shayne Longpre Sayash Kapoor Kevin Klyman Ashwin Ramaswami Rishi Bommasani Borhane Blili-Hamelin Yangsibo Huang Aviya Skowron Zheng-Xin Yong Suhas Kotha et al. 2024. A safe harbor for ai evaluation and red teaming. arXiv preprint arXiv:2403.04893 (2024)."},{"key":"e_1_2_1_72_1","volume-title":"Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408","author":"Luccioni Alexandra Sasha","year":"2023","unstructured":"Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2023. Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408 (2023)."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675283"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-23","author":"Mack Kelly Avery","year":"2024","unstructured":"Kelly Avery Mack, Rida Qadri, Remi Denton, Shaun K Kane, and Cynthia L Bennett. 2024. ''They only care to show us the wheelchair'': disability representation in text-to-image AI models. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-23."},{"key":"e_1_2_1_75_1","volume-title":"Jennifer Wortman Vaughan, and Hanna Wallach","author":"Madaio Michael","year":"2021","unstructured":"Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, and Hanna Wallach. 2021. Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support. arXiv preprint arXiv:2112.05675 (2021)."},{"key":"e_1_2_1_76_1","volume-title":"Tailor","author":"Madaio Michael A","year":"2024","unstructured":"Michael A Madaio, Jingya Chen, Hanna Wallach, and Jennifer Wortman Vaughan. 2024. Tinker, Tailor, Configure, Customize: The Articulation Work of Contextualizing an AI Fairness Checklist. Proceedings of the ACM on Human-Computer Interaction, Vol. 8, CSCW1 (2024), 1-20."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376445"},{"key":"e_1_2_1_78_1","volume-title":"MIRAGE: Multi-model Interface for Reviewing and Auditing Generative Text-to-Image AI. Demo of the AAAI Conference on Human Computation and Crowdsourcing","author":"Maldaner Matheus Kunzler","year":"2024","unstructured":"Matheus Kunzler Maldaner, Wesley Hanwen Deng, Jason Hong, Ken Holstein, and Motahhare Eslami. 2024. MIRAGE: Multi-model Interface for Reviewing and Auditing Generative Text-to-Image AI. Demo of the AAAI Conference on Human Computation and Crowdsourcing (2024)."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1561\/9781680839173"},{"key":"e_1_2_1_80_1","volume-title":"Wisdom for the crowd: discoursive power in annotation instructions for computer vision. arXiv preprint arXiv:2105.10990","author":"Miceli Milagros","year":"2021","unstructured":"Milagros Miceli and Julian Posada. 2021. Wisdom for the crowd: discoursive power in annotation instructions for computer vision. arXiv preprint arXiv:2105.10990 (2021)."},{"key":"e_1_2_1_81_1","unstructured":"Microsoft. 2023. Planning red teaming for large language models (LLMs) and their applications - Azure OpenAI Service. https:\/\/learn.microsoft.com\/en-us\/azure\/ai-services\/openai\/concepts\/red-teaming"},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-18","author":"Mim Nusrat Jahan","year":"2024","unstructured":"Nusrat Jahan Mim, Dipannita Nandi, Sadaf Sumyia Khan, Arundhuti Dey, and Syed Ishtiaque Ahmed. 2024. In-Between Visuals and Visible: The Impacts of Text-to-Image Generative AI Tools on Digital Image-making Practices in the Global South. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-18."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/1294211.1294215"},{"key":"e_1_2_1_84_1","first-page":"997","volume-title":"Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society","volume":"7","author":"Mun Jimin","year":"2024","unstructured":"Jimin Mun, Liwei Jiang, Jenny Liang, Inyoung Cheong, Nicole DeCairo, Yejin Choi, Tadayoshi Kohno, and Maarten Sap. 2024. Particip-ai: A democratic surveying framework for anticipating future ai use cases, harms and benefits. In Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, Vol. 7. 997-1010."},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600211.3604711"},{"key":"e_1_2_1_86_1","unstructured":"NIST. 2023. Artificial Intelligence Risk Management Framework. https:\/\/www.nist.gov\/itl\/ai-risk-management-framework"},{"key":"e_1_2_1_87_1","volume-title":"Algorithms of oppression: How search engines reinforce racism","author":"Noble Safiya Umoja","unstructured":"Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. NYU Press."},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13337"},{"key":"e_1_2_1_89_1","volume-title":"Search Atlas: Visualizing Divergent Search Results Across Geopolitical Borders. In Designing Interactive Systems Conference 2021","author":"Ochigame Rodrigo","year":"2021","unstructured":"Rodrigo Ochigame and Katherine Ye. 2021. Search Atlas: Visualizing Divergent Search Results Across Geopolitical Borders. In Designing Interactive Systems Conference 2021. 1970-1983."},{"key":"e_1_2_1_90_1","volume-title":"Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling. arXiv preprint arXiv:2402.17861","author":"Ojewale Victor","year":"2024","unstructured":"Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane, and Inioluwa Deborah Raji. 2024. Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling. arXiv preprint arXiv:2402.17861 (2024)."},{"key":"e_1_2_1_91_1","unstructured":"OpenAI. 2022a. Open AI ChatGPT. https:\/\/openai.com\/blog\/chatgpt\/"},{"key":"e_1_2_1_92_1","unstructured":"OpenAI. 2022b. OpenAI: Our approach to alignment research. https:\/\/openai.com\/blog\/our-approach-to-alignment-research\/"},{"key":"e_1_2_1_93_1","unstructured":"OpenAI. 2023. https:\/\/cdn.openai.com\/papers\/gpt-4-system-card.pdf"},{"key":"e_1_2_1_94_1","volume-title":"Anticipating Unintended Consequences of Technology Using Insights from Creativity Support Tools. arXiv preprint arXiv:2304.05687","author":"Pang Rock Yuren","year":"2023","unstructured":"Rock Yuren Pang and Katharina Reinecke. 2023. Anticipating Unintended Consequences of Technology Using Insights from Creativity Support Tools. arXiv preprint arXiv:2304.05687 (2023)."},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642054"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/3526113.3545616"},{"key":"e_1_2_1_97_1","unstructured":"Tina Park. [n.d.]. Stakeholder Engagement for Responsible AI: Introducing PAI's Guidelines for Participatory and Inclusive AI. https:\/\/partnershiponai.org\/stakeholder-engagement-for-responsible-ai-introducing-pais-guidelines-for-participatory-and-inclusive-ai\/"},{"key":"e_1_2_1_98_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"2","author":"Passi Samir","year":"2018","unstructured":"Samir Passi and Steven J Jackson. 2018. Trust in data science: Collaboration, translation, and accountability in corporate data science projects. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 1-28."},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445103"},{"key":"e_1_2_1_100_1","volume-title":"Information foraging. Psychological review","author":"Pirolli Peter","year":"1999","unstructured":"Peter Pirolli and Stuart Card. 1999. Information foraging. Psychological review, Vol. 106, 4 (1999), 643."},{"key":"e_1_2_1_101_1","unstructured":"Giada Pistilli. 2022. HuggingFace announcedthe new feature to flag any Model Dataset or Space on the Hub."},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-019-04144-6"},{"key":"e_1_2_1_103_1","volume-title":"Aart: Ai-assisted red-teaming with diverse data generation for new llm-powered applications. arXiv preprint arXiv:2311.08592","author":"Radharapu Bhaktipriya","year":"2023","unstructured":"Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, and Preethi Lahoti. 2023. Aart: Ai-assisted red-teaming with diverse data generation for new llm-powered applications. arXiv preprint arXiv:2311.08592 (2023)."},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_2_1_105_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"5","author":"Rakova Bogdana","year":"2021","unstructured":"Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2021. Where responsible AI meets reality: Practitioner perspectives on enablers for shifting organizational practices. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW1 (2021), 1-23."},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600211.3604712"},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/3487607"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702508"},{"key":"e_1_2_1_109_1","volume-title":"Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry, Vol. 22, 2014 (2014), 4349-4357."},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600211.3604673"},{"key":"e_1_2_1_111_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17","author":"Shelby Renee","year":"2024","unstructured":"Renee Shelby, Shalaleh Rismani, and Negar Rostamzadeh. 2024. Generative AI in Creative Practice: ML-Artist Folk Theories of T2I Use, Harm, and Harm-Reduction. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-17."},{"key":"e_1_2_1_112_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445971"},{"key":"e_1_2_1_113_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"5","author":"Shen Hong","year":"2021","unstructured":"Hong Shen, Alicia DeVos, Motahhare Eslami, and Kenneth Holstein. 2021b. Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1-29."},{"key":"e_1_2_1_114_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533110"},{"key":"e_1_2_1_115_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675239"},{"key":"e_1_2_1_116_1","doi-asserted-by":"crossref","first-page":"1565","DOI":"10.2146\/ajhp100029","article-title":"Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business","volume":"67","author":"Sindlinger Ted S.","year":"2010","unstructured":"Ted S. Sindlinger. 2010. Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business. American Journal of Health-system Pharmacy, Vol. 67 (2010), 1565-1566.","journal-title":"American Journal of Health-system Pharmacy"},{"key":"e_1_2_1_117_1","volume-title":"Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. 2098-2111","author":"Solyst Jaemarie","year":"2025","unstructured":"Jaemarie Solyst, Cindy Peng, Wesley Hanwen Deng, Praneetha Pratapa, Amy Ogan, Jessica Hammer, Jason Hong, and Motahhare Eslami. 2025. Investigating Youth AI Auditing. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. 2098-2111."},{"key":"e_1_2_1_118_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"7","author":"Solyst Jaemarie","year":"2023","unstructured":"Jaemarie Solyst, Ellia Yang, Shixian Xie, Amy Ogan, Jessica Hammer, and Motahhare Eslami. 2023. The Potential of Diverse Youth as Stakeholders in Identifying and Mitigating Algorithmic Bias for a Future of Fairer AI. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW2 (2023), 1-27."},{"key":"e_1_2_1_119_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445092"},{"key":"e_1_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460276.2460278"},{"key":"e_1_2_1_121_1","unstructured":"The White House. 2023. Executive Order on the Safe Secure and Trustworthy Development and Use of Artificial Intelligence. https:\/\/www.whitehouse.gov\/briefing-room\/presidential-actions\/2023\/10\/30\/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence\/"},{"key":"e_1_2_1_122_1","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3242050"},{"key":"e_1_2_1_123_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445885"},{"key":"e_1_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581278"},{"key":"e_1_2_1_125_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-40","author":"Wang Zijie J.","year":"2024","unstructured":"Zijie J. Wang, Chinmay Kulkarni, Lauren Wilcox, Michael Terry, and Michael Madaio. 2024. Farsight: Fostering Responsible AI Awareness During AI Application Prototyping. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-40."},{"key":"e_1_2_1_126_1","unstructured":"Tris Warkentin and Josh Woodward. 2022. AI Test Kitchen. https:\/\/blog.google\/technology\/ai\/join-us-in-the-ai-test-kitchen\/"},{"key":"e_1_2_1_127_1","volume-title":"Nahema Marchal, Ravin Kumar, Kristian Lum, Canfer Akbulut, Mark Diaz, Stevie Bergman, Mikel Rodriguez, et al.","author":"Weidinger Laura","year":"2024","unstructured":"Laura Weidinger, John Mellor, Bernat Guillen Pegueroles, Nahema Marchal, Ravin Kumar, Kristian Lum, Canfer Akbulut, Mark Diaz, Stevie Bergman, Mikel Rodriguez, et al., 2024. STAR: SocioTechnical Approach to Red Teaming Language Models. arXiv preprint arXiv:2406.11757 (2024)."},{"key":"e_1_2_1_128_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533088"},{"key":"e_1_2_1_129_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"8","author":"Widder David Gray","year":"2024","unstructured":"David Gray Widder, Laura Dabbish, James D Herbsleb, and Nikolas Martelaro. 2024. Power and Play: Investigating'' License to Critique'' in Teams' AI Ethics Discussions. Proceedings of the ACM on Human-Computer Interaction, Vol. 8, CSCW2 (2024), 1-23."},{"key":"e_1_2_1_130_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1073"},{"key":"e_1_2_1_131_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-019-09497-z"},{"key":"e_1_2_1_132_1","volume-title":"EMPOWERING RED TEAMS WITH GENERATIVE AI: TRANSFORMING PENETRATION TESTING THROUGH ADAPTIVE INTELLIGENCE. EDPACS","author":"Zaydi Mounia","year":"2024","unstructured":"Mounia Zaydi and Yassine Maleh. 2024. EMPOWERING RED TEAMS WITH GENERATIVE AI: TRANSFORMING PENETRATION TESTING THROUGH ADAPTIVE INTELLIGENCE. EDPACS (2024), 1-26."},{"key":"e_1_2_1_133_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-25","author":"Zhang Lili","year":"2024","unstructured":"Lili Zhang, Xi Liao, Zaijia Yang, Baihang Gao, Chunjie Wang, Qiuling Yang, and Deshun Li. 2024. Partiality and Misconception: Investigating Cultural Representativeness in Text-to-Image Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1-25."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757702","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:49:24Z","timestamp":1760636964000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757702"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":133,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757702"],"URL":"https:\/\/doi.org\/10.1145\/3757702","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}