{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T19:45:28Z","timestamp":1784835928060,"version":"3.55.0"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>Because algorithm audits are conducted by technical experts, audits are necessarily limited to the hypotheses that experts think to test. End users hold the promise to expand this purview, as they inhabit spaces and witness algorithmic impacts that auditors do not. In pursuit of this goal, we propose end-user audits-system-scale audits led by non-technical users-and present an approach that scaffolds end users in hypothesis generation, evidence identification, and results communication. Today, performing a system-scale audit requires substantial user effort to label thousands of system outputs, so we introduce a collaborative filtering technique that leverages the algorithmic system's own disaggregated training data to project from a small number of end user labels onto the full test set. Our end-user auditing tool, IndieLabel, employs these predicted labels so that users can rapidly explore where their opinions diverge from the algorithmic system's outputs. By highlighting topic areas where the system is under-performing for the user and surfacing sets of likely error cases, the tool guides the user in authoring an audit report. In an evaluation of end-user audits on a popular comment toxicity model with 17 non-technical participants, participants both replicated issues that formal audits had previously identified and also raised previously underreported issues such as under-flagging on veiled forms of hate that perpetuate stigma and over-flagging of slurs that have been reclaimed by marginalized communities.<\/jats:p>","DOI":"10.1145\/3555625","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T22:58:54Z","timestamp":1668207534000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":82,"title":["End-User Audits: A System Empowering Communities to Lead Large-Scale Investigations of Harmful Algorithmic Behavior"],"prefix":"10.1145","volume":"6","author":[{"given":"Michelle S.","family":"Lam","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mitchell L.","family":"Gordon","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dana\u00eb","family":"Metaxa","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey T.","family":"Hancock","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James A.","family":"Landay","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael S.","family":"Bernstein","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v14i1.7276"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2700832"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449148"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v14i1.7277"},{"key":"e_1_2_2_6_1","volume-title":"Who Moderates the Social Media Giants? A Call to End Outsourcing","author":"Barrett Paul M","year":"2020","unstructured":"Paul M Barrett . 2020. Who Moderates the Social Media Giants? A Call to End Outsourcing . NYU Stern Center for Business and Human Rights ( 2020 ). Paul M Barrett. 2020. Who Moderates the Social Media Giants? A Call to End Outsourcing. NYU Stern Center for Business and Human Rights (2020)."},{"key":"e_1_2_2_7_1","volume-title":"Race After Technology: Abolitionist Tools for the New Jim Code","author":"Benjamin Ruha","unstructured":"Ruha Benjamin . 2019. Race After Technology: Abolitionist Tools for the New Jim Code . Polity . Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Polity."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720983865"},{"key":"e_1_2_2_9_1","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research","volume":"91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification . In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research , Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77-- 91 . https:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77--91. https:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479569"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/2056305120936636"},{"key":"e_1_2_2_12_1","volume-title":"Constructing grounded theory: A practical guide through qualitative analysis","author":"Charmaz Kathy","unstructured":"Kathy Charmaz . 2006. Constructing grounded theory: A practical guide through qualitative analysis . Sage . Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through qualitative analysis. Sage."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174225"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815675.2815681"},{"key":"e_1_2_2_15_1","volume-title":"Introducing Twitter's first algorithmic bias bounty challenge. Twitter Engineering Blog","author":"Chowdhury Rumman","year":"2021","unstructured":"Rumman Chowdhury and Jutta Williams . 2021. Introducing Twitter's first algorithmic bias bounty challenge. Twitter Engineering Blog ( 2021 ). https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bountychallenge Rumman Chowdhury and Jutta Williams. 2021. Introducing Twitter's first algorithmic bias bounty challenge. Twitter Engineering Blog (2021). https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bountychallenge"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533213"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tele.2018.07.004"},{"key":"e_1_2_2_18_1","volume-title":"Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics","author":"Crenshaw Kimberl\u00e9","unstructured":"Kimberl\u00e9 Crenshaw . 1989. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics . University of Chicago Legal Forum (1989) , 139--168. Kimberl\u00e9 Crenshaw. 1989. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics. University of Chicago Legal Forum (1989), 139--168."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517441"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1257\/app.20160213"},{"key":"e_1_2_2_22_1","volume-title":"An Algorithmic Framework for Bias Bounties. In 2022 ACM Conference on Fairness, Accountability, and Transparency","author":"Globus-Harris Ira","year":"2022","unstructured":"Ira Globus-Harris , Michael Kearns , and Aaron Roth . 2022 . An Algorithmic Framework for Bias Bounties. In 2022 ACM Conference on Fairness, Accountability, and Transparency ( Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 1106--1124. https:\/\/doi.org\/10.1145\/3531146.3533172 10.1145\/3531146.3533172 Ira Globus-Harris, Michael Kearns, and Aaron Roth. 2022. An Algorithmic Framework for Bias Bounties. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 1106--1124. https:\/\/doi.org\/10.1145\/3531146.3533172"},{"key":"e_1_2_2_23_1","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems","author":"Gordon Mitchell L.","year":"2004","unstructured":"Mitchell L. Gordon , Michelle S. Lam , Joon Sung Park , Kayur Patel , Jeff Hancock , Tatsunori Hashimoto , and Michael S. Bernstein . 2022. Jury Learning: Integrating Dissenting Voices into Machine Learning Models . In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems ( New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 115, 19 pages. https:\/\/doi.org\/10.1145\/3491102.350 2004 10.1145\/3491102.3502004 Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury Learning: Integrating Dissenting Voices into Machine Learning Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 115, 19 pages. https:\/\/doi.org\/10.1145\/3491102.3502004"},{"key":"e_1_2_2_24_1","volume-title":"Bernstein","author":"Gordon Mitchell L.","year":"2021","unstructured":"Mitchell L. Gordon , Kaitlyn Zhou , Kayur Patel , Tatsunori Hashimoto , and Michael S . Bernstein . 2021 . The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. Association for Computing Machinery , New York, NY, USA. https:\/\/doi.org\/10.1145\/3411764.3445423 10.1145\/3411764.3445423 Mitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S. Bernstein. 2021. The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3411764.3445423"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951719897945"},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security","author":"Gr\u00f6ndahl Tommi","unstructured":"Tommi Gr\u00f6ndahl , Luca Pajola , Mika Juuti , Mauro Conti , and N. Asokan . 2018. All You Need is \"Love\": Evading Hate Speech Detection . In Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security ( Toronto, Canada) (AISec '18). Association for Computing Machinery, New York, NY, USA, 2--12. https:\/\/doi.org\/10.1145\/3270101.3270103 10.1145\/3270101.3270103 Tommi Gr\u00f6ndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N. Asokan. 2018. All You Need is \"Love\": Evading Hate Speech Detection. In Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security (Toronto, Canada) (AISec '18). Association for Computing Machinery, New York, NY, USA, 2--12. https:\/\/doi.org\/10.1145\/3270101.3270103"},{"key":"#cr-split#-e_1_2_2_27_1.1","unstructured":"Maarten Grootendorst. 2020. BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics. https:\/\/doi.org\/10.5281\/zenodo.4381785 10.5281\/zenodo.4381785"},{"key":"#cr-split#-e_1_2_2_27_1.2","unstructured":"Maarten Grootendorst. 2020. BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics. https:\/\/doi.org\/10.5281\/zenodo.4381785"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479610"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372826"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663716.2663744"},{"key":"e_1_2_2_31_1","volume-title":"Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency","author":"Harris Camille","year":"2022","unstructured":"Camille Harris , Matan Halevy , Ayanna Howard , Amy Bruckman , and Diyi Yang . 2022 . Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency ( Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 789--798. https:\/\/doi.org\/10.1145\/3531146.3533144 10.1145\/3531146.3533144 Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the Role of Grammar and Word Choice in Bias Toward African American English (AAE) in Hate Speech Classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT '22). Association for Computing Machinery, New York, NY, USA, 789--798. https:\/\/doi.org\/10.1145\/3531146.3533144"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2019.1573912"},{"key":"e_1_2_2_33_1","volume-title":"Hal Daum\u00e9, Miro Dudik, and Hanna Wallach.","author":"Holstein Kenneth","year":"2019","unstructured":"Kenneth Holstein , Jennifer Wortman Vaughan , Hal Daum\u00e9, Miro Dudik, and Hanna Wallach. 2019 . Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? Association for Computing Machinery , New York, NY, USA, 1--16. https:\/\/doi.org\/10.1145\/3290605.3300830 10.1145\/3290605.3300830 Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daum\u00e9, Miro Dudik, and Hanna Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? Association for Computing Machinery, New York, NY, USA, 1--16. https:\/\/doi.org\/10.1145\/3290605.3300830"},{"key":"e_1_2_2_34_1","unstructured":"Hossein Hosseini Sreeram Kannan Baosen Zhang and Radha Poovendran. 2017. Deceiving Google's Perspective API Built for Detecting Toxic Comments. arXiv:1702.08138 [cs.LG]  Hossein Hosseini Sreeram Kannan Baosen Zhang and Radha Poovendran. 2017. Deceiving Google's Perspective API Built for Detecting Toxic Comments. arXiv:1702.08138 [cs.LG]"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3422841.3423534"},{"key":"e_1_2_2_36_1","volume-title":"Casey Fiesler, and Jed R. Brubaker.","author":"Jiang Jialun Aaron","year":"2021","unstructured":"Jialun Aaron Jiang , Morgan Klaus Scheuerman , Casey Fiesler, and Jed R. Brubaker. 2021 . Understanding international perceptions of the severity of harmful content online. PLOS ONE 16, 8 (08 2021), 1--22. https:\/\/doi.org\/10.1371\/journal.pone.0256762 10.1371\/journal.pone.0256762 Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R. Brubaker. 2021. Understanding international perceptions of the severity of harmful content online. PLOS ONE 16, 8 (08 2021), 1--22. https:\/\/doi.org\/10.1371\/journal.pone.0256762"},{"key":"e_1_2_2_37_1","unstructured":"Jigsaw. 2019. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-biasin-toxicity-classification\/overview  Jigsaw. 2019. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-biasin-toxicity-classification\/overview"},{"key":"e_1_2_2_38_1","unstructured":"Jigsaw. 2022. Perspective API. Retrieved January 2 2022 from https:\/\/www.perspectiveapi.com\/  Jigsaw. 2022. Perspective API. Retrieved January 2 2022 from https:\/\/www.perspectiveapi.com\/"},{"key":"e_1_2_2_39_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"2572","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns , Seth Neel , Aaron Roth , and Zhiwei Steven Wu . 2018 . Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness . In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research , Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 2564-- 2572 . https:\/\/proceedings.mlr.press\/v80\/kearns18a.html Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 2564--2572. https:\/\/proceedings.mlr.press\/v80\/kearns18a.html"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1915768117"},{"key":"e_1_2_2_41_1","volume-title":"Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021","author":"Kumar Deepak","year":"2021","unstructured":"Deepak Kumar , Patrick Gage Kelley , Sunny Consolvo , Joshua Mason , Elie Bursztein , Zakir Durumeric , Kurt Thomas , and Michael Bailey . 2021 . Designing Toxic Content Classification for a Diversity of Perspectives . In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021 ). USENIX Association, 299--318. https:\/\/www.usenix.org\/conference\/soups 2021\/presentation\/kumar Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing Toxic Content Classification for a Diversity of Perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021). USENIX Association, 299--318. https:\/\/www.usenix.org\/conference\/soups2021\/presentation\/kumar"},{"key":"e_1_2_2_42_1","volume-title":"You Can't See What You Can't See: Experimental Evidence for How Much Relevant Information May Be Missed Due to Google's Web Search Personalisation","author":"Lai Cameron","unstructured":"Cameron Lai and Markus Luczak-Roesch . 2019. You Can't See What You Can't See: Experimental Evidence for How Much Relevant Information May Be Missed Due to Google's Web Search Personalisation . In Social Informatics, Ingmar Weber, Kareem M. Darwish, Claudia Wagner, Emilio Zagheni, Laura Nelson, Samin Aref, and Fabian Fl\u00f6ck (Eds.). Springer International Publishing , Cham , 253--266. Cameron Lai and Markus Luczak-Roesch. 2019. You Can't See What You Can't See: Experimental Evidence for How Much Relevant Information May Be Missed Due to Google's Web Search Personalisation. In Social Informatics, Ingmar Weber, Kareem M. Darwish, Claudia Wagner, Emilio Zagheni, Laura Nelson, Samin Aref, and Fabian Fl\u00f6ck (Eds.). Springer International Publishing, Cham, 253--266."},{"key":"e_1_2_2_43_1","doi-asserted-by":"crossref","unstructured":"Michael Madaio Lisa Egede Hariharan Subramonyam JenniferWortman Vaughan and HannaWallach. 2021. Assessing the Fairness of AI Systems: AI Practitioners' Processes Challenges and Needs for Support. arXiv:2112.05675 [cs.AI]  Michael Madaio Lisa Egede Hariharan Subramonyam JenniferWortman Vaughan and HannaWallach. 2021. Assessing the Fairness of AI Systems: AI Practitioners' Processes Challenges and Needs for Support. arXiv:2112.05675 [cs.AI]","DOI":"10.1145\/3512899"},{"key":"e_1_2_2_45_1","volume-title":"Austin Hounsel, and Nick Feamster","author":"Matias J. Nathan","year":"2021","unstructured":"J. Nathan Matias , Austin Hounsel, and Nick Feamster . 2021 . Software-Supported Audits of Decision-Making Systems: Testing Google and Facebook's Political Advertising Policies . arXiv:2103.00064 [cs.HC] J. Nathan Matias, Austin Hounsel, and Nick Feamster. 2021. Software-Supported Audits of Decision-Making Systems: Testing Google and Facebook's Political Advertising Policies. arXiv:2103.00064 [cs.HC]"},{"key":"e_1_2_2_46_1","volume-title":"Coronavirus disrupts social media's first line of defense. WIRED Magazine 18","author":"Matsakis Louise","year":"2020","unstructured":"Louise Matsakis and Paris Martineau . 2020. Coronavirus disrupts social media's first line of defense. WIRED Magazine 18 ( 2020 ). Louise Matsakis and Paris Martineau. 2020. Coronavirus disrupts social media's first line of defense. WIRED Magazine 18 (2020)."},{"key":"e_1_2_2_47_1","volume-title":"YouTube Regrets: A crowdsourced investigation into YouTube's recommendation algorithm. Retrieved","author":"McCrosky Jesse","year":"2022","unstructured":"Jesse McCrosky and Brandi Geurkink . 2021. YouTube Regrets: A crowdsourced investigation into YouTube's recommendation algorithm. Retrieved January 2, 2022 from https:\/\/foundation.mozilla.org\/en\/youtube\/findings\/ Jesse McCrosky and Brandi Geurkink. 2021. YouTube Regrets: A crowdsourced investigation into YouTube's recommendation algorithm. Retrieved January 2, 2022 from https:\/\/foundation.mozilla.org\/en\/youtube\/findings\/"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359174"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359231"},{"key":"e_1_2_2_50_1","volume-title":"Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig.","author":"Metaxa Dana\u00eb","year":"2021","unstructured":"Dana\u00eb Metaxa , Joon Sung Park , Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig. 2021 . Auditing Algorithms : Understanding Algorithmic Systems from the Outside In. Foundations and Trends\u00ae in Human--Computer Interaction 14, 4 (2021), 272--344. https:\/\/doi.org\/10.1561\/1100000083 10.1561\/1100000083 Dana\u00eb Metaxa, Joon Sung Park, Ronald E. Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, and Christian Sandvig. 2021. Auditing Algorithms: Understanding Algorithmic Systems from the Outside In. Foundations and Trends\u00ae in Human--Computer Interaction 14, 4 (2021), 272--344. https:\/\/doi.org\/10.1561\/1100000083"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2535372.2535415"},{"key":"e_1_2_2_52_1","volume-title":"Article arXiv:2110.05719 (Oct.","author":"Davani Aida Mostafazadeh","year":"2021","unstructured":"Aida Mostafazadeh Davani , Mark D\u00edaz , and Vinodkumar Prabhakaran . 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arXiv e-prints , Article arXiv:2110.05719 (Oct. 2021 ), arXiv:2110.05719 pages. arXiv:2110.05719 [cs.CL] Aida Mostafazadeh Davani, Mark D\u00edaz, and Vinodkumar Prabhakaran. 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arXiv e-prints, Article arXiv:2110.05719 (Oct. 2021), arXiv:2110.05719 pages. arXiv:2110.05719 [cs.CL]"},{"key":"e_1_2_2_53_1","doi-asserted-by":"crossref","unstructured":"Dat Quoc Nguyen Thanh Vu and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arXiv:2005.10200 [cs.CL]  Dat Quoc Nguyen Thanh Vu and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arXiv:2005.10200 [cs.CL]","DOI":"10.18653\/v1\/2020.emnlp-demos.2"},{"key":"e_1_2_2_54_1","volume-title":"Algorithms of oppression","author":"Noble Safiya Umoja","unstructured":"Safiya Umoja Noble . 2018. Algorithms of oppression . New York University Press . Safiya Umoja Noble. 2018. Algorithms of oppression. New York University Press."},{"key":"e_1_2_2_55_1","volume-title":"Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464","author":"Obermeyer Ziad","year":"2019","unstructured":"Ziad Obermeyer , Brian Powers , Christine Vogeli , and Sendhil Mullainathan . 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 ( 2019 ), 447--453. https:\/\/doi.org\/10.1126\/science.aax2342 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/science.aax2342 10.1126\/science.aax2342 Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447--453. https:\/\/doi.org\/10.1126\/science.aax2342 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/science.aax2342"},{"key":"e_1_2_2_56_1","volume-title":"Critical Race Theory for HCI","author":"Ogbonnaya-Ogburu Ihudiya Finda","unstructured":"Ihudiya Finda Ogbonnaya-Ogburu , Angela D.R. Smith , Alexandra To , and Kentaro Toyama . 2020. Critical Race Theory for HCI . Association for Computing Machinery , New York, NY, USA , 1--16. https:\/\/doi.org\/10.1145\/3313831.3376392 10.1145\/3313831.3376392 Ihudiya Finda Ogbonnaya-Ogburu, Angela D.R. Smith, Alexandra To, and Kentaro Toyama. 2020. Critical Race Theory for HCI. Association for Computing Machinery, New York, NY, USA, 1--16. https:\/\/doi.org\/10.1145\/3313831.3376392"},{"key":"e_1_2_2_57_1","volume-title":"Aida Mostafazadeh Davani, and Mark D\u00edaz","author":"Prabhakaran Vinodkumar","year":"2021","unstructured":"Vinodkumar Prabhakaran , Aida Mostafazadeh Davani, and Mark D\u00edaz . 2021 . On Releasing Annotator-Level Labels and Information in Datasets . arXiv:2110.05699 [cs.CL] Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark D\u00edaz. 2021. On Releasing Annotator-Level Labels and Information in Datasets. arXiv:2110.05699 [cs.CL]"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1410"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274417"},{"key":"e_1_2_2_62_1","volume-title":"Roweis (Eds.)","volume":"20","author":"Salakhutdinov Russ R","year":"2008","unstructured":"Russ R Salakhutdinov and Andriy Mnih . 2008 . Probabilistic Matrix Factorization. In Advances in Neural Information Processing Systems, J. Platt, D. Koller, Y. Singer, and S . Roweis (Eds.) , Vol. 20 . Curran Associates, Inc. https:\/\/proceedings. neurips.cc\/paper\/ 2007\/file\/d7322ed717dedf1eb4e6e52a37ea7bcd-Paper.pdf Russ R Salakhutdinov and Andriy Mnih. 2008. Probabilistic Matrix Factorization. In Advances in Neural Information Processing Systems, J. Platt, D. Koller, Y. Singer, and S. Roweis (Eds.), Vol. 20. Curran Associates, Inc. https:\/\/proceedings. neurips.cc\/paper\/2007\/file\/d7322ed717dedf1eb4e6e52a37ea7bcd-Paper.pdf"},{"key":"e_1_2_2_63_1","volume-title":"Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig , Kevin Hamilton , Karrie Karahalios , and Cedric Langbort . 2014 . Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry (2014), 1--23. Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry (2014), 1--23."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1163"},{"key":"e_1_2_2_65_1","volume-title":"Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems","author":"Schlesinger Ari","unstructured":"Ari Schlesinger , W. Keith Edwards , and Rebecca E. Grinter . 2017. Intersectional HCI: Engaging Identity through Gender, Race, and Class . In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems ( Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 5412--5427. https:\/\/doi.org\/10.1145\/3025453.3025766 10.1145\/3025453.3025766 Ari Schlesinger, W. Keith Edwards, and Rebecca E. Grinter. 2017. Intersectional HCI: Engaging Identity through Gender, Race, and Class. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 5412--5427. https:\/\/doi.org\/10.1145\/3025453.3025766"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479577"},{"key":"e_1_2_2_67_1","volume-title":"Athanasios Andreou, Pedro Olmo Vaz de Melo, Oana Goga, and Fabricio Benevenuto.","author":"Silva M\u00e1rcio","year":"2020","unstructured":"M\u00e1rcio Silva , Lucas Santos de Oliveira , Athanasios Andreou, Pedro Olmo Vaz de Melo, Oana Goga, and Fabricio Benevenuto. 2020 . Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook. Association for Computing Machinery , New York, NY, USA, 224--234. https:\/\/doi.org\/10.1145\/3366423.3380109 10.1145\/3366423.3380109 M\u00e1rcio Silva, Lucas Santos de Oliveira, Athanasios Andreou, Pedro Olmo Vaz de Melo, Oana Goga, and Fabricio Benevenuto. 2020. Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook. Association for Computing Machinery, New York, NY, USA, 224--234. https:\/\/doi.org\/10.1145\/3366423.3380109"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376783"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-019-05855-6"},{"key":"e_1_2_2_70_1","volume-title":"The State of Online Harassment","author":"Vogels Emily A","year":"2021","unstructured":"Emily A Vogels . 2021. The State of Online Harassment . Pew Research Center 13 ( 2021 ). Emily A Vogels. 2021. The State of Online Harassment. Pew Research Center 13 (2021)."},{"key":"e_1_2_2_71_1","volume-title":"Proceedings of the Web Conference","author":"Shivanna Rakesh","year":"2021","unstructured":"RuoxiWang, Rakesh Shivanna , Derek Cheng , Sagar Jain , Dong Lin , Lichan Hong , and Ed Chi . 2021 . DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems . In Proceedings of the Web Conference 2021. 1785--1797. RuoxiWang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. In Proceedings of the Web Conference 2021. 1785--1797."},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445928"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449280"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-019-09497-z"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555625","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555625","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:18Z","timestamp":1750182558000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555625"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":74,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555625"],"URL":"https:\/\/doi.org\/10.1145\/3555625","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}