{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T00:55:19Z","timestamp":1776128119379,"version":"3.50.1"},"reference-count":162,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2021,4,13]],"date-time":"2021-04-13T00:00:00Z","timestamp":1618272000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,4,13]]},"abstract":"<jats:p>While algorithm audits are growing rapidly in commonality and public importance, relatively little scholarly work has gone toward synthesizing prior work and strategizing future research in the area. This systematic literature review aims to do just that, following PRISMA guidelines in a review of over 500 English articles that yielded 62 algorithm audit studies. The studies are synthesized and organized primarily by behavior (discrimination, distortion, exploitation, and misjudgement), with codes also provided for domain (e.g. search, vision, advertising, etc.), organization (e.g. Google, Facebook, Amazon, etc.), and audit method (e.g. sock puppet, direct scrape, crowdsourcing, etc.). The review shows how previous audit studies have exposed public-facing algorithms exhibiting problematic behavior, such as search algorithms culpable of distortion and advertising algorithms culpable of discrimination. Based on the studies reviewed, it also suggests some behaviors (e.g. discrimination on the basis of intersectional identities), domains (e.g. advertising algorithms), methods (e.g. code auditing), and organizations (e.g. Twitter, TikTok, LinkedIn) that call for future audit attention. The paper concludes by offering the common ingredients of successful audits, and discussing algorithm auditing in the context of broader research working toward algorithmic justice.<\/jats:p>","DOI":"10.1145\/3449148","type":"journal-article","created":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T17:51:09Z","timestamp":1619113869000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":153,"title":["Problematic Machine Behavior"],"prefix":"10.1145","volume":"5","author":[{"given":"Jack","family":"Bandy","sequence":"first","affiliation":[{"name":"Northwestern University, Evanston, IL, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,4,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174156"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372871"},{"key":"e_1_2_1_3_1","volume-title":"The Expanding News Desert","author":"Abernathy Penelope Muse","year":"2018","unstructured":"Penelope Muse Abernathy. 2018. The Expanding News Desert. University of North Carolina Press Chapel Hill. https:\/\/www.cislm.org\/wp-content\/uploads\/2018\/10\/The-Expanding-News-Desert-10_14-Web.pdf"},{"key":"e_1_2_1_4_1","volume-title":"The new Jim Crow: Mass incarceration in the age of colorblindness","author":"Alexander Michelle","unstructured":"Michelle Alexander. 2010. The new Jim Crow: Mass incarceration in the age of colorblindness. The New Press."},{"key":"e_1_2_1_5_1","unstructured":"Muhammad Ali Piotr Sapiezynski Aleksandra Korolova Alan Mislove and Aaron Rieke. 2019. Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging. (2019). http:\/\/arxiv.org\/abs\/1912.04255"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2018.23191"},{"key":"e_1_2_1_7_1","unstructured":"Michael Armbrust Ion Stoica Matei Zaharia Armando Fox Rean Griffith Anthony D. Joseph Randy Katz Andy Konwinski Gunho Lee David Patterson and Ariel Rabkin. 2018. It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process. https:\/\/acm-fca.org\/2018\/03\/29\/negativeimpacts\/"},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Joshua Asplund Motahare Eslami Hari Sundaram Christian Sandvig and Karrie Karahalios. 2020. Auditing Race and Gender Discrimination in Online Housing Markets. In ICWSM. https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/7276","DOI":"10.1609\/icwsm.v14i1.7276"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5220\/0006938702820289"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaa1160"},{"key":"e_1_2_1_11_1","volume-title":"Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News. International conference on web and social media (ICWSM)","author":"Bandy Jack","year":"2020","unstructured":"Jack Bandy and Nicholas Diakopoulos. 2020. Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News. International conference on web and social media (ICWSM) (2020)."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 583--591","author":"Barlas Pinar","year":"2019","unstructured":"Pinar Barlas, Kyriakos Kyriakou, Styliani Kleanthous, and Jahna Otterbacher. 2019. Social B(eye)as: Human and machine descriptions of people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 583--591. https:\/\/www.aaai.org\/ojs\/index.php\/ICWSM\/article\/view\/3255\/3123"},{"key":"e_1_2_1_13_1","volume-title":"9th Annual Conference of the Special Interest Group for Computing, Information and Society.","author":"Barocas Solon","year":"2017","unstructured":"Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual Conference of the Special Interest Group for Computing, Information and Society."},{"key":"e_1_2_1_14_1","unstructured":"Alistair Barr. 2015. Google Mistakenly Tags Black People as \"Gorillas ' Showing Limits of Algorithms. https:\/\/www.wsj.com\/articles\/BL-DGB-42522"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23392"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2018.1510741"},{"key":"e_1_2_1_17_1","volume-title":"Race After Technology","author":"Benjamin Ruha","unstructured":"Ruha Benjamin. 2019. Race After Technology. Polity Press."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-67256-4_32"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-015-9380-y"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_2_1_21_1","first-page":"1","article-title":"Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification","volume":"81","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of Machine Learning Research, Vol. 81. 1--15. http:\/\/proceedings.mlr.press\/v81\/buolamwini18a\/buolamwini18a.pdf","journal-title":"Proceedings of Machine Learning Research"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025903"},{"key":"e_1_2_1_23_1","volume-title":"Unveiling and Quantifying Facebook Exploitation of Sensitive Personal Data for Advertising Purposes. In USENIX Security Symposium. 479--495","author":"Caba\u00f1as Jos\u00e9 Gonz\u00e1lez","year":"2018","unstructured":"Jos\u00e9 Gonz\u00e1lez Caba\u00f1as, \u00c1ngel Cuevas, and Rub\u00e9n Cuevas. 2018. Unveiling and Quantifying Facebook Exploitation of Sensitive Personal Data for Advertising Purposes. In USENIX Security Symposium. 479--495. https:\/\/www.usenix.org\/conference\/usenixsecurity18\/presentation\/cabanas"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.3145\/epi.2019.mar.13"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASONAM.2018.8508812"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174225"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883089"},{"key":"e_1_2_1_28_1","unstructured":"Ed H. Chi. 2011. On the Importance of Replication in HCI and Social Computing Research. https:\/\/cacm.acm.org\/blogs\/blog-cacm\/109916-on-the-importance-of-replication-in-hci-and-social-computing-research\/fulltext"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0047"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.21606\/drs.2018.679"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1177\/1527476418796632"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tele.2018.07.004"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781315582924--10"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1093\/joc"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency","volume":"81","author":"Datta Amit","year":"2018","unstructured":"Amit Datta, Anupam Datta, Jael Makagon, Deirdre K Mulligan, and Michael Carl Tschantz. 2018. Discrimination in Online Advertising: A Multidisciplinary Inquiry. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, Vol. 81 (2018), 20--34. http:\/\/proceedings.mlr.press\/v81\/datta18a\/datta18a.pdf"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1515\/popets-2015-0007"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287572"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557214"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858081"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2016.1178592"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52---59","author":"DeVries Terrance","year":"1906","unstructured":"Terrance DeVries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 52---59. arxiv: 1906.02659"},{"key":"e_1_2_1_42_1","volume-title":"Amazon, Facebook, and Apple","author":"Diakopoulos Nicholas","year":"2018","unstructured":"Nicholas Diakopoulos, Daniel Trielli, Jennifer Stark, and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. Digital Dominance: The Power of Google, Amazon, Facebook, and Apple (2018)."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359228"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173986"},{"key":"e_1_2_1_45_1","volume-title":"Data Feminism","author":"D'Ignazio Catherine","unstructured":"Catherine D'Ignazio and Lauren F Klein. 2020. Data Feminism. MIT Press."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134673"},{"key":"e_1_2_1_47_1","unstructured":"Emily Dreyfuss. 2019. What the Google-Genius Copyright Dispute Is Really About. https:\/\/www.wired.com\/story\/what-the-google-genius-copyright-dispute-is-really-about\/"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1177\/0887403417718608"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1177\/0887403415604899"},{"key":"e_1_2_1_50_1","unstructured":"Editors. 2020. Price Gouging Laws by State. https:\/\/consumer.findlaw.com\/consumer-transactions\/price-gouging-laws-by-state.html"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302262"},{"key":"e_1_2_1_52_1","unstructured":"Elsevier. 2016. Scopus | The largest database of peer-reviewed literature."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134677"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.25595\/1449"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017. 62--71","author":"Eslami Motahhare","year":"2017","unstructured":"Motahhare Eslami, Kristen Vaccaro, Karrie Karahalios, and Kevin Hamilton. 2017. \"Be careful; Things can be worse than they appear\" - Understanding biased algorithms and users' behavior around them in rating platforms. In Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017. 62--71. https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/14898"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1080\/10999922.2018.1511671"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1096\/fj.07-9492LSF"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-020-00954-0"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2018.1502045"},{"key":"e_1_2_1_61_1","volume-title":"A Rejoinder to \"Machine Bias\". HeinOnline","author":"Flores Anthony","year":"2016","unstructured":"Anthony Flores, Christopher T. Lowenkamp, and Kristin Bechtel. 2016. False Positives, False Negatives, and False Analyses: A Rejoinder to \"Machine Bias\". HeinOnline (2016)."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3022198.3022201"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753629"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330691"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2016.1199721"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1093\/bjsw"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2843948"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009079"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2017.1338145"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372826"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998327"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488435"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663716.2663744"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.4135\/9781483346229.n142"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359250"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.1002106"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","unstructured":"Felicitas Hesselmann Verena Graf Marion Schmidt and Martin Reinhart. 2017. The visibility of scientific misconduct: A review of the literature on retracted journal articles. 814--845 pages. https:\/\/doi.org\/10.1177\/0011392116663807","DOI":"10.1177\/0011392116663807"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2019.1573912"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313654"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3176258.3176338"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392854"},{"key":"e_1_2_1_83_1","unstructured":"Andrew Hutchinson. 2020. Facebook Removes Over 1 000 Ad Targeting Options Due to Low Usage. https:\/\/www.socialmediatoday.com\/news\/facebook-removes-over-1000-ad-targeting-options-due-to-low-usage\/583406\/"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1080\/01972240050133634"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287565"},{"key":"e_1_2_1_86_1","volume-title":"Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019","volume":"289","author":"Jiang Shan","year":"2019","unstructured":"Shan Jiang, Ronald E. Robertson, and Christo Wilson. 2019 b. Bias misperceived: The role of partisanship and misinformation in YouTube comment moderation. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019, Vol. 13(01). 278--289. https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/3229"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.5441\/002"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702520"},{"key":"e_1_2_1_89_1","volume-title":"International Conference on Machine Learning. http:\/\/proceedings.mlr.press\/v80\/kearns18a.html","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning. http:\/\/proceedings.mlr.press\/v80\/kearns18a.html"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3310433"},{"key":"e_1_2_1_91_1","unstructured":"Lauren Kirchner Surya Mattu Jeff Larson and Julia Angwin. 2016. Machine Bias - ProPublica. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2016.1154087"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815675.2815714"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998321"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-018-9341-2"},{"key":"e_1_2_1_96_1","volume-title":"Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 313--322","author":"Kyriakou Kyriakos","year":"2019","unstructured":"Kyriakos Kyriakou, Pinar Barlas, Styliani Kleanthous, and Jahna Otterbacher. 2019. Fairness in proprietary image tagging algorithms: A cross-platform audit on people images. In Proceedings of the 13th International Conference on Web and Social Media, ICWSM 2019. 313--322. https:\/\/www.aaai.org\/ojs\/index.php\/ICWSM\/article\/view\/3232\/3100"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-34971-4_17"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2018.3093"},{"key":"e_1_2_1_99_1","unstructured":"Nadia Latif. 2017. It's lit! How film finally learned to light black skin. https:\/\/www.theguardian.com\/film\/2017\/sep\/21\/its-lit-how-film-finally-learned-how-to-light-black-skin"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702548"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432953"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998281"},{"key":"e_1_2_1_103_1","volume-title":"Proceedings of the International Florida Artificial Intelligence Research Society Conference. https:\/\/par.nsf.gov\/biblio\/10101277","author":"Lurie Emma","year":"2019","unstructured":"Emma Lurie and Eni Mustafaraj. 2019. Opening Up the Black Box: Auditing Google's Top Stories Algorithm. In Proceedings of the International Florida Artificial Intelligence Research Society Conference. https:\/\/par.nsf.gov\/biblio\/10101277"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.3384\/cu.2000.1525.1792212"},{"key":"e_1_2_1_105_1","volume-title":"Coronavirus Disrupts Social Media's First Line of Defense","author":"Matsakis Louise","year":"2020","unstructured":"Louise Matsakis and Paris Martineau. [n.d.]. Coronavirus Disrupts Social Media's First Line of Defense., 2020 pages. https:\/\/www.wired.com\/story\/coronavirus-social-media-automated-content-moderation\/"},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314279"},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09685-x"},{"key":"e_1_2_1_108_1","volume-title":"Proceedings of the 11th International AAAI Conference on Web and Social Media. https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/14883","author":"Mcmahon Connor","year":"2017","unstructured":"Connor Mcmahon, Isaac Johnson, and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. In Proceedings of the 11th International AAAI Conference on Web and Social Media. https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/14883"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359231"},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/2390231.2390245"},{"key":"e_1_2_1_112_1","doi-asserted-by":"publisher","unstructured":"Jakub Mikians L\u00e1szl\u00f3 Gyarmati Vijay Erramilli and Nikolaos Laoutaris. 2013a. Crowd-assisted Search for Price Discrimination in E-Commerce: First results. (2013). https:\/\/doi.org\/10.1145\/2535372.2535415","DOI":"10.1145\/2535372.2535415"},{"key":"e_1_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.1145\/2535372.2535415"},{"key":"e_1_2_1_114_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951716679679"},{"key":"e_1_2_1_115_1","doi-asserted-by":"publisher","DOI":"10.1177\/2056305118817038"},{"key":"e_1_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13347-020-00405-8"},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","unstructured":"David Moher Alessandro Liberati Jennifer Tetzlaff Douglas G. Altman Doug Altman Gerd Antes David Atkins Virginia Barbour Nick Barrowman Jesse A. Berlin Jocalyn Clark Mike Clarke Deborah Cook Roberto D'Amico Jonathan J. Deeks P. J. Devereaux Kay Dickersin Matthias Egger Edzard Ernst Peter C. G\u00f8tzsche Jeremy Grimshaw Gordon Guyatt Julian Higgins John P.A. Ioannidis Jos Kleijnen Tom Lang Nicola Magrini David McNamee Lorenzo Moja Cynthia Mulrow Maryann Napoli Andy Oxman Ba' Pham Drummond Rennie Margaret Sampson Kenneth F. Schulz Paul G. Shekelle David Tovey and Peter Tugwell. 2009. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. https:\/\/doi.org\/10.7326\/0003-4819-151-4-200908180-00135","DOI":"10.7326\/0003-4819-151-4-200908180-00135"},{"key":"e_1_2_1_118_1","doi-asserted-by":"publisher","unstructured":"David Moher Lesley Stewart and Paul Shekelle. 2015. All in the Family: Systematic reviews rapid reviews scoping reviews realist reviews and more. 183 pages. https:\/\/doi.org\/10.1186\/s13643-015-0163--7","DOI":"10.1186\/s13643-015-0163--7"},{"key":"e_1_2_1_119_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2018.1444076"},{"key":"e_1_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372835"},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2018.07.043"},{"key":"e_1_2_1_122_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.910905"},{"key":"e_1_2_1_123_1","volume-title":"Google search: Hyper-visibility as a means of rendering black women and girls invisible. InVisible Culture","author":"Noble Safiya","year":"2013","unstructured":"Safiya Noble. 2013. Google search: Hyper-visibility as a means of rendering black women and girls invisible. InVisible Culture (2013)."},{"key":"e_1_2_1_124_1","volume-title":"Algorithms of Oppression: How Search Engines Reinforce Racism","author":"Noble Safiya Umoja","unstructured":"Safiya Umoja Noble. 2018. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press."},{"key":"e_1_2_1_125_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376392"},{"key":"e_1_2_1_126_1","volume-title":"Weapons of math destruction: how big data increases inequality and threatens democracy","author":"O'Neil Cathy","unstructured":"Cathy O'Neil. 2017. Weapons of math destruction: how big data increases inequality and threatens democracy. Broadway Books."},{"key":"e_1_2_1_127_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2019.1593484"},{"key":"e_1_2_1_129_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2018.1539626"},{"key":"e_1_2_1_130_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_2_1_131_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1138-y"},{"key":"e_1_2_1_132_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_2_1_133_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_2_1_134_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372879"},{"key":"e_1_2_1_135_1","doi-asserted-by":"publisher","DOI":"10.1177\/1354856517736982"},{"key":"e_1_2_1_136_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274417"},{"key":"e_1_2_1_137_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292522.3326047"},{"key":"e_1_2_1_138_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292522.3326047"},{"key":"e_1_2_1_139_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-019-09502-5"},{"key":"e_1_2_1_140_1","volume-title":"Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Data and discrimination: converting critical concerns into productive inquiry.","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In Data and discrimination: converting critical concerns into productive inquiry."},{"key":"e_1_2_1_141_1","volume-title":"ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. 1668--1678","author":"Sap Maarten","unstructured":"Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2020. The risk of racial bias in hate speech detection. In ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference. 1668--1678."},{"key":"e_1_2_1_142_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380109"},{"key":"e_1_2_1_143_1","doi-asserted-by":"publisher","DOI":"10.3384\/cu.2000.1525.1792184"},{"key":"e_1_2_1_144_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883016"},{"key":"e_1_2_1_145_1","unstructured":"Leo Sun. 2020. TikTok Is Still Growing Faster than Facebook and Snapchat. https:\/\/www.nasdaq.com\/articles\/tiktok-is-still-growing-faster-than-facebook-and-snapchat-2020-09-14"},{"key":"e_1_2_1_146_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.2208240"},{"key":"e_1_2_1_147_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2900022"},{"key":"e_1_2_1_148_1","doi-asserted-by":"publisher","DOI":"10.1145\/3322640.3326705"},{"key":"e_1_2_1_149_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300683"},{"key":"e_1_2_1_150_1","doi-asserted-by":"publisher","unstructured":"Michael Carl Tschantz Serge Egelman Jaeyoung Choi Nicholas Weaver and Gerald Friedland. 2018. The Accuracy of the Demographic Inferences Shown on Google's Ad Settings. In WPES. 33--41. https:\/\/doi.org\/10.1145\/3267323.3268962","DOI":"10.1145\/3267323.3268962"},{"key":"e_1_2_1_151_1","doi-asserted-by":"publisher","unstructured":"Andrew Tutt. 2017. An FDA for algorithms. 83--123 pages. https:\/\/doi.org\/10.2139\/ssrn.2747994","DOI":"10.2139\/ssrn.2747994"},{"key":"e_1_2_1_152_1","doi-asserted-by":"publisher","DOI":"10.1145\/2335356.2335362"},{"key":"e_1_2_1_153_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2017.03.010"},{"key":"e_1_2_1_154_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313666"},{"key":"e_1_2_1_155_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313742"},{"key":"e_1_2_1_156_1","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media (ICWSM). https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/3248","author":"Vincent Nicholas","year":"2019","unstructured":"Nicholas Vincent, Isaac Johnson, Patrick Sheehan, and Brent Hecht. 2019 b. Measuring the Importance of User-Generated Content to Search Engines. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM). https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/3248"},{"key":"e_1_2_1_157_1","volume-title":"Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech Sexism","author":"Wachter-Boettcher Sara","year":"2017","unstructured":"Sara Wachter-Boettcher. 2017. Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech Sexism. WW Norton & Company."},{"key":"e_1_2_1_158_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2017.1366865"},{"key":"e_1_2_1_159_1","doi-asserted-by":"publisher","DOI":"10.1177\/0007650317718185"},{"key":"e_1_2_1_160_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372833"},{"key":"e_1_2_1_161_1","unstructured":"Max Willens. 2019. Publishers see mobile traffic spikes from Google Discover. https:\/\/digiday.com\/media\/publishers-see-mobile-traffic-spikes-google-discover\/"},{"key":"e_1_2_1_162_1","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2479636"},{"key":"e_1_2_1_163_1","doi-asserted-by":"publisher","DOI":"10.1145\/2907069"},{"key":"e_1_2_1_164_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531643"},{"key":"e_1_2_1_165_1","doi-asserted-by":"publisher","DOI":"10.1057\/jit.2015.5"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449148","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449148","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:17Z","timestamp":1750191497000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449148"}},"subtitle":["A Systematic Literature Review of Algorithm Audits"],"short-title":[],"issued":{"date-parts":[[2021,4,13]]},"references-count":162,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2021,4,13]]}},"alternative-id":["10.1145\/3449148"],"URL":"https:\/\/doi.org\/10.1145\/3449148","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,13]]},"assertion":[{"value":"2021-04-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}