{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T08:18:52Z","timestamp":1786090732044,"version":"3.56.0"},"reference-count":137,"publisher":"Emerald","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,11,25]]},"abstract":"<jats:p>Algorithms are ubiquitous and critical sources of information online, increasingly acting as gatekeepers for users accessing or sharing information about virtually any topic, including their personal lives and those of friends and family, news and politics, entertainment, and even information about health and well-being. As a result, algorithmically-curated content is drawing increased attention and scrutiny from users, the media, and lawmakers alike. However, studying such content poses considerable challenges, as it is both dynamic and ephemeral: these algorithms are constantly changing, and frequently changing silently, with no record of the content to which users have been exposed over time. One strategy that has proven effective is the algorithm audit: a method of repeatedly querying an algorithm and observing its output in order to draw conclusions about the algorithm\u2019s opaque inner workings and possible external impact. In this work, we present an overview of the algorithm audit methodology, including the history of audit studies in the social sciences from which this method is derived; a summary of key algorithm audits over the last two decades in a variety of domains, including health, politics, discrimination, and others; and a set of best practices for conducting algorithm audits today, contextualizing these practices using search engine audits as a case study. Finally, we conclude by discussing the social, ethical, and political dimensions of auditing algorithms, and propose normative standards for the use of this method.<\/jats:p>","DOI":"10.1561\/1100000083","type":"journal-article","created":{"date-parts":[[2021,11,25]],"date-time":"2021-11-25T03:58:36Z","timestamp":1637812716000},"page":"272-344","source":"Crossref","is-referenced-by-count":116,"title":["Auditing Algorithms: Understanding Algorithmic Systems from the Outside In"],"prefix":"10.1108","volume":"14","author":[{"given":"Dana\u00eb","family":"Metaxa","sequence":"first","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joon Sung","family":"Park","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ronald E.","family":"Robertson","sequence":"additional","affiliation":[{"name":"Northeastern University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karrie","family":"Karahalios","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christo","family":"Wilson","sequence":"additional","affiliation":[{"name":"Northeastern University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeff","family":"Hancock","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian","family":"Sandvig","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","published-online":{"date-parts":[[2021,11,25]]},"reference":[{"key":"2026040213480640900_ref001","doi-asserted-by":"crossref","unstructured":"Abebe\n              R.\n            \n            \n              Hill\n              S.\n            \n            \n              Vaughan\n              J. W.\n            \n            \n              Small\n              P. M.\n            \n            \n              Schwartz\n              H. A.\n            \n          \n          2019\n          \u201cUsing Search Queries to Understand Health Information Needs in Africa\u201d\n          Proceedings of the International AAAI Conference on Web and Social Media\n          13\n          01\n          3\u201314https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/3360","DOI":"10.1609\/icwsm.v13i01.3360"},{"issue":"6","key":"2026040213480640900_ref002","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1038\/s42256-021-00359-2","article-title":"\u201cLarge language models associate Muslims with\nviolence\u201d","volume":"3","author":"Abid","year":"(2021)","journal-title":"Nature Machine Intelligence"},{"key":"2026040213480640900_ref003","unstructured":"ACLU\n          \n          (2020)\n          \u201cFederal Court Rules \u2018Big Data\u2019 Discrimination\nStudies Do Not Violate Federal Anti-Hacking Law\u201d\n          Mar.\n          https:\/\/www.aclu.org\/press-releases\/federal-court-rules-big-data-discrimination-studies-do-not-violate-federal-anti\n          (accessed on 08\/26\/2020)"},{"issue":"4","key":"2026040213480640900_ref004","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2642","article-title":"\u201cThe Impact of Search Engine Selection and\nSorting Criteria on Vaccination Beliefs and Attitudes:Two Experiments Manipulating Google Output\u201d","volume":"16","author":"Allam","year":"(2014)","journal-title":"Journal of Medical Internet Research"},{"key":"2026040213480640900_ref005","unstructured":"Angwin\n              J.\n            \n            \n              Larson\n              J.\n            \n            \n              Mattu\n              S.\n            \n            \n              Kirchner\n              L.\n            \n          \n          (2016)\n          \u201cMachine Bias\u201d\n          ProPublica\n          May\n          https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing\n          (accessed on 01\/05\/2021)"},{"key":"2026040213480640900_ref006","unstructured":"Angwin\n              J.\n            \n            \n              Mattu\n              S.\n            \n          \n          2016\n          \u201cAmazon Says It Puts Customers First. But Its\nPricing Algorithm Doesn\u2019t\u201d\n          ProPublica\n          Sept.\n          https:\/\/www.propublica.org\/article\/amazon-says-it-puts-customers-first-but-its-pricing-algorithm-doesnt\n          (accessed on 01\/05\/2021)"},{"key":"2026040213480640900_ref007","first-page":"10","article-title":"\u201cAuditing Race and Gender Discrimination in\nOnline Housing Markets\u201d","author":"Asplund","year":"2020","journal-title":"Proceedings of the Eleventh International AAAI Conference on Web and Social Media (ICWSM 2020)"},{"key":"2026040213480640900_ref008","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1609\/icwsm.v14i1.7276","article-title":"\u201cAuditing race and gender discrimination in\nonline housing markets\u201d In:","volume":"14","author":"Asplund","year":"2020","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"issue":"7","key":"2026040213480640900_ref009","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v20i7.5597","article-title":"\u201cGoogle Chemtrails: A Methodology to Analyze\nTopic Representation in Search Engine Results\u201d","volume":"20","author":"Ballatore","year":"(2015)","journal-title":"First Monday"},{"issue":"5","key":"2026040213480640900_ref010","doi-asserted-by":"publisher","first-page":"1194","DOI":"10.1080\/24694452.2017.1308240","article-title":"\u201cDigital Hegemonies: The Localness of Search\nEngine \u201d","volume":"107","author":"Ballatore","year":"(2017)","journal-title":"Annals of the American Association of Geographers"},{"key":"2026040213480640900_ref011","doi-asserted-by":"crossref","unstructured":"Bandy\n              J.\n            \n          \n          (2021)\n          \u201cProblematic Machine Behavior: A Systematic\nLiterature Review of Algorithm Audits\u201d\n          arXiv:2102.04256 [cs.CY]","DOI":"10.1145\/3449148"},{"issue":"1","key":"2026040213480640900_ref012","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1609\/icwsm.v14i1.7277","article-title":"\u201cAuditing News Curation Systems: A Case Study\nExamining Algorithmic and Editorial Logic in Apple News\u201d","volume":"14","author":"Bandy","year":"2020","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"key":"2026040213480640900_ref013","first-page":"1","article-title":"\u201cFairness in machine \u201d","author":"Barocas","year":"2017","journal-title":"NIPS Tutorial"},{"issue":"8","key":"2026040213480640900_ref014","doi-asserted-by":"publisher","first-page":"990","DOI":"10.1080\/21670811.2018.1510741","article-title":"\u201cAre We Exposed to the Same \u201cNews\u201d in the News\nFeed?\u201d","volume":"6","author":"Bechmann","year":"2018","journal-title":"Digital Journalism"},{"key":"2026040213480640900_ref015","article-title":"\u201cESR: Ethics and Society Review of Artificial\nIntelligence Research\u201d","author":"Bernstein","year":"2021"},{"issue":"4","key":"2026040213480640900_ref016","doi-asserted-by":"crossref","first-page":"991","DOI":"10.1257\/0002828042002561","article-title":"\u201cAre Emily and Greg more employable than\nLakisha and Jamal? A field experiment on labor market discrimination\u201d","volume":"94","author":"Bertrand","year":"2004","journal-title":"American Economic Review"},{"key":"2026040213480640900_ref017","article-title":"\u201cFacebook Disables Access for NYU Research Into\nPolitical-Ad Targeting\u201d","author":"Bobrowsky","year":"2021","journal-title":"The Wall Street Journal"},{"issue":"1","key":"2026040213480640900_ref018","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720983865","article-title":"\u201cThe algorithm audit: Scoring the algorithms\nthat score us\u201d","volume":"8","author":"Brown","year":"2021","journal-title":"Big Data & Society"},{"key":"2026040213480640900_ref019","unstructured":"Buolamwini\n              J.\n            \n          \n          2019\n          \u201cHearing on Facial Recognition Technology (Part\n1): Its Impact on our Civil Rights and Liberties\u201d\n          https:\/\/www.congress.gov\/116\/meeting\/house\/109521\/witnesses\/HHRG-116GO00-Wstate-BuolamwiniJ-20190522.pdf"},{"key":"2026040213480640900_ref020","first-page":"77","article-title":"\u201cGender shades: Intersectional accuracy\ndisparities in commercial gender classification\u201d In:","author":"Buolamwini","year":"2018","journal-title":"Conference on Fairness, Accountability and Transparency"},{"key":"2026040213480640900_ref021","first-page":"65","article-title":"\u201cIntentions and Attention in Exploratory Health\nSearch\u201d In:","author":"Cartright","year":"2011","journal-title":"Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval"},{"key":"2026040213480640900_ref022","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3173574.3174225","article-title":"\u201cInvestigating the Impact of Gender on Rank in\nResume Search Engines\u201d In:","author":"Chen","year":"2018","journal-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)"},{"key":"2026040213480640900_ref023","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1145\/2815675.2815681","article-title":"\u201cPeeking beneath the hood of Uber\u201d In:","author":"Chen","year":"2015","journal-title":"Proceedings of the 2015 Internet Measurement Conference"},{"key":"2026040213480640900_ref024","unstructured":"Chowdhury\n              R.\n            \n            \n              Williams\n              J.\n            \n          \n          2021\n          \u201cIntroducing Twitter\u2019s first algorithmic bias\nbounty challenge\u201d\n          https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bounty-challenge"},{"key":"2026040213480640900_ref025","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/9780262015424.001.0001","volume-title":"Programmed Visions: Software and Memory","author":"Chun","year":"2011"},{"key":"2026040213480640900_ref026","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1145\/3442188.3445921","article-title":"\u201cReviewable Automated Decision-Making: A\nFramework for Accountable Algorithmic Systems\u201d","author":"Cobbe","year":"2021","journal-title":"Proceedings of the 2021 ACM Conference on Fairness,\nAccountability, and Transparency (FAccT '21)"},{"key":"2026040213480640900_ref027","unstructured":"Coders' Rights Project\n          \n          2020\n          https:\/\/www.eff.org\/issues\/coders\n          (accessed on 10\/27\/2020)"},{"key":"2026040213480640900_ref028","unstructured":"Association for Computing Machinery\n          \n          1992\n          \u201cACM Code of Ethics and Professional\nConduct\u201d\n          Code of Ethics"},{"issue":"5","key":"2026040213480640900_ref029","doi-asserted-by":"crossref","first-page":"1297","DOI":"10.1086\/511799","article-title":"\u201cGetting a job: Is there a motherhood\npenalty?\u201d","volume":"112","author":"Correll","year":"2007","journal-title":"American Journal of Sociology"},{"key":"2026040213480640900_ref030","doi-asserted-by":"crossref","first-page":"1241","DOI":"10.2307\/1229039","article-title":"\u201cMapping the margins: Intersectionality,\nidentity politics, and violence against women of color\u201d","volume":"43","author":"Crenshaw","year":"1990","journal-title":"Stanford Law Review"},{"key":"2026040213480640900_ref031","volume":"1084","author":"Daniel","year":"1968","journal-title":"\u201cRacial discrimination in England: based on the PEP\nreport\u201d"},{"key":"2026040213480640900_ref032","unstructured":"Dave\n              K.\n            \n          \n          2019\n          \u201cSystemic Algorithmic Harms\u201d\n          https:\/\/points.datasociety.net\/systemic-algorithmic-harms-e00f99e72c42\n          (accessed on 08\/01\/2021)"},{"key":"2026040213480640900_ref033","first-page":"1365","article-title":"\u201cSeeking and Sharing Health Information Online:\nComparing Search Engines and Social Media\u201d","author":"DeChoudhury","year":"2014","journal-title":"Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems"},{"key":"2026040213480640900_ref034","first-page":"22","article-title":"\u201cI vote for\u2014how search informs our choice of\ncandidate\u201d","author":"Diakopoulos","year":"2018","journal-title":"Digital Dominance: The Power of Google, Amazon, Facebook, and Apple"},{"key":"2026040213480640900_ref035","first-page":"1","article-title":"\u201cAddressing age-related bias in sentiment\nanalysis\u201dIn:","author":"D\u00edaz","year":"2018","journal-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems"},{"key":"2026040213480640900_ref036","unstructured":"DuckDuckGo\n          \n          2018\n          \u201cMeasuring the \u201cFilter Bubble\u201d: How Google Is\nInfluencing What You \u201d\n          https:\/\/spreadprivacy.com\/google-filter-bubble-study\/"},{"issue":"2","key":"2026040213480640900_ref037","first-page":"1","article-title":"\u201cRacial discrimination in the sharing economy:\nEvidence from a field \u201d","volume":"9","author":"Edelman","year":"2017","journal-title":"American Economic Journal: Applied Economics"},{"key":"2026040213480640900_ref038","doi-asserted-by":"crossref","unstructured":"Edelman\n              B. G.\n            \n            \n              Luca\n              M.\n            \n          \n          2014\n          \u201cDigital discrimination: The case of\nAirbnb.com\u201d\n          Harvard Business School NOM Unit Working Paper\n          14\n          54","DOI":"10.2139\/ssrn.2377353"},{"key":"2026040213480640900_ref039","unstructured":"Engler\n              A. C.\n            \n          \n          2021\n          \u201cIndependent auditors are struggling to hold AI\ncompanies accountable\u201d\n          Fast Company\n          Jan\n          https:\/\/www.fastcompany.com\/90597594\/ai-algorithm-auditing-hirevue\n          (accessed on 01\/05\/2021)"},{"issue":"33","key":"2026040213480640900_ref040","doi-asserted-by":"publisher","first-page":"E4512","DOI":"10.1073\/pnas.1419828112","article-title":"\u201cThe Search Engine Manipulation Effect (SEME)\nand Its Possible Impact on the Outcomes of Elections\u201d","volume":"112","author":"Epstein","year":"2015","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"CSCW","key":"2026040213480640900_ref041","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3134677","article-title":"\u201cSuppressing the Search Engine Manipulation\nEffect (SEME)\u201d","volume":"1","author":"Epstein","year":"2017","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"issue":"2","key":"2026040213480640900_ref042","doi-asserted-by":"crossref","first-page":"163","DOI":"10.3384\/cu.2000.1525.1792163","article-title":"\u201cTracking gendered streams\u201d","volume":"9","author":"Eriksson","year":"2017","journal-title":"Culture Unbound"},{"issue":"34","key":"2026040213480640900_ref043","doi-asserted-by":"publisher","first-page":"12684","DOI":"10.1073\/pnas.0605525103","article-title":"\u201cTopical Interests and the Mitigation of Search\nEngine Bias\u201d","volume":"103","author":"Fortunato","year":"2006","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2026040213480640900_ref044","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-71153-9","volume-title":"Audit studies: Behind the scenes with theory,\nmethod, and nuance","author":"Gaddis","year":"2018"},{"key":"2026040213480640900_ref045","unstructured":"U.S. Government Accountability Office (GAO)\n          \n          2021\n          \u201cArtificial Intelligence: An Accountability\nFramework for Federal Agencies and Other Entities\u201d\n          https:\/\/www.gao.gov\/products\/gao-21-519sp"},{"issue":"7232","key":"2026040213480640900_ref046","doi-asserted-by":"crossref","first-page":"1012","DOI":"10.1038\/nature07634","article-title":"\u201cDetecting Influenza Epidemics Using Search\nEngine Query dataprivacylab\u201d","volume":"457","author":"Ginsberg","year":"2009","journal-title":"Nature"},{"key":"2026040213480640900_ref047","article-title":"\u201cthe Are the disabled discriminated against in\nproduct markets? Evidence from field experiments\u201d","author":"Gneezy","year":"2004","journal-title":"American Economic Association Annual Meeting"},{"key":"2026040213480640900_ref048","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/978-3-540-75829-7_8","volume-title":"Web Search","author":"Goldman","year":"2008"},{"key":"2026040213480640900_ref049","unstructured":"Google Terms of Service\n          \n          2017\n          https:\/\/policies.google.com\/terms?hl=en-US\n          (accessed on 08\/22\/2019)"},{"key":"2026040213480640900_ref050","unstructured":"Google China\n          \n          2021\n          https:\/\/en.wikipedia.org\/wiki\/Google_China\n          (accessed on 08\/01\/2021)"},{"issue":"5","key":"2026040213480640900_ref051","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1080\/01972243.2010.511560","article-title":"The\u201c Politics of Search: A Decade\nRetrospective\u201d","volume":"26","author":"Granka","year":"2010","journal-title":"The Information Society"},{"key":"2026040213480640900_ref052","unstructured":"Gross\n              T.\n            \n          \n          2017\n          \u201cA \u2018Forgotten History\u2019 Of How The U.S.\nGovernment Segregated America\u201d\n          NPR\n          May\n          https:\/\/www.npr.org\/2017\/05\/03\/526655831\/a-forgotten-history-of-how-the-u-s-government-segregated-america\n          (accessed on 01\/05\/2021)"},{"key":"2026040213480640900_ref053","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1145\/2488388.2488435","article-title":"\u201cMeasuring Personalization of Web Search\u201d","author":"Hannak","year":"2013","journal-title":"Proceedings of the 22nd International Conference on World\nWide Web (WWW '13)"},{"key":"2026040213480640900_ref054","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1145\/2663716.2663744","article-title":"\u201cMeasuring price discrimination and steering on\ne-commerce web sites\u201d","author":"Hannak","year":"2014","journal-title":"Proceedings of the 2014 Internet Measurement Conference"},{"key":"2026040213480640900_ref055","doi-asserted-by":"crossref","DOI":"10.1145\/2998181.2998327","article-title":"\u201cBias in Online Freelance Marketplaces:\nEvidence from TaskRabbit and Fiverr\u201d","author":"Hannak","year":"2017","journal-title":"20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017)"},{"key":"2026040213480640900_ref056","doi-asserted-by":"publisher","first-page":"693","DOI":"10.1145\/3308558.3313654","article-title":"\u201cAuditing the Partisanship of Google Search Snippets\u201d","author":"Hu","year":"2019","journal-title":"The Web Conference (WWW \u201919)"},{"issue":"CSCW1","key":"2026040213480640900_ref057","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3392854","article-title":"\u201cMeasuring Misinformation in Video Search\nPlatforms: An Audit Study on YouTube\u201d","volume":"4","author":"Hussein","year":"2020","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"2026040213480640900_ref058","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/3287560.3287600","article-title":"\u201c50 Years of Test (Un)Fairness: Lessons for\nMachine Learning\u201d","author":"Hutchinson","year":"2019","journal-title":"Proceedings of the Conference on Fairness, Accountability,\nand Transparency (FAT* '19)"},{"issue":"3","key":"2026040213480640900_ref059","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1080\/01972240050133634","article-title":"\u201cShaping the Web: Why the Politics of Search\nEngines Matters\u201d","volume":"16","author":"Introna","year":"2000","journal-title":"The Information Society"},{"issue":"2","key":"2026040213480640900_ref060","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1145\/1229179.1229181","article-title":"\u201cEvaluating the Accuracy of Implicit Feedback\nfrom Clicks and Query Reformulations in Web Search\u201d","volume":"25","author":"Joachims","year":"2007","journal-title":"ACM Transactions on Information Systems"},{"key":"2026040213480640900_ref061","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445250","article-title":"Auditing\u201c E-Commerce Platforms for\nAlgorithmically Curated Vaccine Misinformation\u201d","author":"Juneja","year":"2021","journal-title":"Proceedings of the 2021 CHI Conference on Human Factors in\nComputing Systems (CHI '21)"},{"key":"2026040213480640900_ref062","unstructured":"Karahalios\n              K.\n            \n          \n          \u201cDiscrimination Audits: Challenges to\nDiscrimination Studies\u201d\n          http:\/\/social.cs.illinois.edu\/presentations\/lawdata-summit\/karahalios.pdf"},{"key":"2026040213480640900_ref063","doi-asserted-by":"crossref","first-page":"868","DOI":"10.1609\/icwsm.v14i1.7352","article-title":"\u201cThe Media Coverage of the 2020 US Presidential\nElection Candidates through the Lens of Google\u2019s Top Stories\u201d","volume":"14","author":"Kawakami","year":"2020","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"key":"2026040213480640900_ref064","doi-asserted-by":"publisher","first-page":"3819","DOI":"10.1145\/2702123.2702520","article-title":"\u201cUnequal Representation and Gender Stereotypes\nin Image Search Results for Occupations\u201d","author":"Kay","year":"2015","journal-title":"Proceedings of the 33rd Annual ACM Conference on Human\nFactors in Computing Systems (CHI '15)"},{"key":"2026040213480640900_ref065","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3310433","article-title":"\u201cA Mulching Proposal: Analysing and Improving\nan Algorithmic System for Turning the Elderly into High-Nutrient Slurry\u201d","author":"Keyes","year":"2019","journal-title":"Extended Abstracts of the 2019 CHI Conference on Human\nFactors in Computing Systems (CHI EA '19)"},{"key":"2026040213480640900_ref066","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1145\/2815675.2815714","article-title":"\u201cLocation, Location, Location: The Impact of\nGeolocation on Web Search Personalization\u201d","author":"Kliman-Silver","year":"2015","journal-title":"Proceedings of the 2015 ACM Internet Measurement Conference\n(IMC '15)"},{"key":"2026040213480640900_ref067","unstructured":"Koshiyama\n              A.\n            \n            \n              Kazim\n              E.\n            \n            \n              Treleaven\n              P.\n            \n            \n              Rai\n              P.\n            \n            \n              Szpruch\n              L.\n            \n            \n              Pavey\n              G.\n            \n            \n              Ahamat\n              G.\n            \n            \n              Leutner\n              F.\n            \n            \n              Goebel\n              R.\n            \n            \n              Knight\n              A.\n            \n            \n          \n          \u201cTowards Algorithm Auditing: A Survey on\nManaging Legal, Ethical and Technological Risks of AI, ML and Associated Algorithms\u201d"},{"issue":"2","key":"2026040213480640900_ref068","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1177\/0022146516647098","article-title":"\u201c\u201cSorry, I'm Not Accepting New\nPatients\u201d: An Audit Study of Access to Mental Health Care\u201d","volume":"57","author":"Kugelmass","year":"2016","journal-title":"Journal of Health and Social Behavior"},{"issue":"1-2","key":"2026040213480640900_ref069","doi-asserted-by":"publisher","first-page":"188","DOI":"10.1007\/s10791-018-9341-2","article-title":"Search Bias Quantification: Investigating Political Bias in Social Media and Web Search","volume":"22","author":"Kulshrestha","year":"2019","journal-title":"Information Retrieval Journal"},{"key":"2026040213480640900_ref070","doi-asserted-by":"publisher","DOI":"10.1038\/srep12760","article-title":"Advances\u201c in Nowcasting Influenza-like Illness\nRates Using Search Query Logs\u201d","volume":"5","author":"Lampos","year":"2015","journal-title":"Scientific Reports"},{"issue":"6740","key":"2026040213480640900_ref071","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1038\/21987","article-title":"\u201cAccessibility of Information on the Web\u201d","volume":"400","author":"Lawrence","year":"1999","journal-title":"Nature"},{"issue":"6176","key":"2026040213480640900_ref072","doi-asserted-by":"crossref","first-page":"1203","DOI":"10.1126\/science.1248506","article-title":"\u201cThe Parable of Google Flu: Traps in Big Data\nAnalysis\u201d","volume":"343","author":"Lazer","year":"2014","journal-title":"Science"},{"key":"2026040213480640900_ref073","doi-asserted-by":"publisher","first-page":"2957","DOI":"10.1145\/3308558.3313682","article-title":"\u201cMeasuring Political Personalization of Google\nNews Search\u201d","author":"Le","year":"2019","journal-title":"The World Wide Web Conference (WWW \u201919)"},{"key":"2026040213480640900_ref074","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1145\/3201064.3201095","article-title":"\u201c\nInvestigating the Effects of Google\u2019s Search Engine Result Page in\nEvaluating the Credibility of Online News Sources\u201d","author":"Lurie","year":"2018","journal-title":"Proceedings of the 10th ACM Conference on Web Science (WebSci\n'18)"},{"key":"2026040213480640900_ref075","doi-asserted-by":"publisher","DOI":"10.37016\/mr-2020-017","article-title":"\u2013How Search Engines Disseminate Information\nabout COVID-19 and Why They Should Do Better\u2013","author":"Makhortykh","year":"2020","journal-title":"Harvard Kennedy School Misinformation Review"},{"key":"2026040213480640900_ref076","unstructured":"Matias\n              J. N.\n            \n          \n          2020\n          \u2013Why We Need Industry-Independent Research on Tech & Society\n          https:\/\/citizensandtech.org\/2020\/01\/industry-independent-research\/"},{"key":"2026040213480640900_ref077","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1145\/1255175.1255237","article-title":"\u2013Agreeing to Disagree: Search Engines and Their Public Interfaces\u2013","author":"McCown","year":"2007","journal-title":"Proceedings of the 2007 Conference on Digital Libraries (JCDL'07)"},{"key":"2026040213480640900_ref078","unstructured":"McCullagh\n              D.\n            \n          \n          2006\n          \u2013AOL's Disturbing Glimpse into Users' Lives\u2013\n          CNET\n          https:\/\/www.cnet.com\/news\/aols-disturbing-glimpse-into-users-lives\/"},{"key":"2026040213480640900_ref079","first-page":"10","article-title":"\u2013The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship between Peer Production Communities and Information Technologies\u2013","author":"McMahon","year":"2017","journal-title":"Proceedings of the Eleventh International AAAI Conference on Web and Social Media (ICWSM 2018)"},{"key":"2026040213480640900_ref080","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1145\/3041021.3054197","article-title":"\u2013Auditing Search Engines for Differential Satisfaction Across Demographics\u2013","author":"Mehrotra","year":"2017","journal-title":"Proceedings of the 26th International Conference on World Wide Web Companion (WWW '17 Companion)"},{"key":"2026040213480640900_ref081","article-title":"\u2013Search Media and Elections: A Longitudinal Investigation of Political Search Results in the 2018 U.S. Elections\u2013","author":"Metaxa","year":"2019","journal-title":"Proceedings of the 22nd ACM Conference on Computer-Supported Cooperative Work and Social Computing"},{"key":"2026040213480640900_ref082","article-title":"\u2013The Battle for the 2008 US Congressional Elections on the Web\u2013","author":"Metaxas","year":"2009","journal-title":"Proceedings of the 2009 WebScience: Society On-Line Conference"},{"key":"2026040213480640900_ref083","first-page":"6","article-title":"\u2013Manipulation of Search Engine Results during the 2016 US Congressional Elections\u2013","author":"Metaxas","year":"2017","journal-title":"Proceedings of the ICIW 2017"},{"key":"2026040213480640900_ref084","first-page":"79","article-title":"\u2013Detecting price and search discrimination on the internet\u2013","author":"Mikians","year":"2012","journal-title":"Proceedings of the 11th ACM Workshop on Hot Topics in Networks"},{"issue":"2","key":"2026040213480640900_ref085","doi-asserted-by":"crossref","first-page":"57","DOI":"10.14361\/dcs-2016-0205","article-title":"\"The alternative epistemologies of data activism\"","volume":"2","author":"Milan","year":"2016","journal-title":"Digital Culture & Society"},{"issue":"7","key":"2026040213480640900_ref086","doi-asserted-by":"publisher","first-page":"710","DOI":"10.1177\/0956797611434539","article-title":"\"Temporal Distance and Discrimination: An Audit Study in Academia\"","volume":"23","author":"Milkman","year":"2012","journal-title":"Psychological Science"},{"issue":"1","key":"2026040213480640900_ref087","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1016\/S0306-4573(01)00020-6","article-title":"\"Assessing Bias in Search Engines","volume":"38","author":"Mowshowitz","year":"2002","journal-title":"Information Processing & Management\""},{"issue":"9","key":"2026040213480640900_ref088","doi-asserted-by":"publisher","DOI":"10.1145\/567498.567527","article-title":"\"Bias on the Web\"","volume":"45","author":"Mowshowitz","year":"2002","journal-title":"Communications of the ACM"},{"issue":"5","key":"2026040213480640900_ref089","doi-asserted-by":"publisher","first-page":"1193","DOI":"10.1016\/j.ipm.2004.05.005","article-title":"\"Measuring Search Engine Bias\"","volume":"41","author":"Mowshowitz","year":"2005","journal-title":"Information Processing & Management"},{"key":"2026040213480640900_ref090","doi-asserted-by":"crossref","unstructured":"Mullainathan\n              S.\n            \n            \n              Noeth\n              M.\n            \n            \n              Schoar\n              A.\n            \n          \n          2012\n          \"The market for financial advice: An audit study\"\n          National Bureau of Economic Research\n          Tech. Report","DOI":"10.3386\/w17929"},{"key":"2026040213480640900_ref091","first-page":"11","article-title":"\"The Case for Voter-Centered Audits of Search Engines During Political Elections\"","author":"Mustafaraj","year":"2020","journal-title":"Conference on Fairness, Accountability, and Transparency (FAT* \u201920)"},{"issue":"19","key":"2026040213480640900_ref092","article-title":"\"Google search: Hyper-visibility as a means of rendering black women and girls invisible\"","author":"Noble","year":"2013","journal-title":"InVisible Culture"},{"key":"2026040213480640900_ref093","author":"Noble","year":"2018"},{"key":"2026040213480640900_ref094","article-title":"\"Google Has a Striking History of Bias Against Black Girls\"","author":"Noble","year":"2018","journal-title":"Time Magazine"},{"issue":"3","key":"2026040213480640900_ref095","doi-asserted-by":"crossref","first-page":"e3000691","DOI":"10.1371\/journal.pbio.3000691","article-title":"\"What is replication?\"","volume":"18","author":"Nosek","year":"2020","journal-title":"PLOS Biology"},{"issue":"6464","key":"2026040213480640900_ref096","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"\"Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations\"","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"issue":"4","key":"2026040213480640900_ref097","first-page":"148","article-title":"\"Why do skilled immigrants struggle in the labor market? A field experiment with thirteen thousand resumes\"","volume":"3","author":"Oreopoulos","year":"2011","journal-title":"American Economic Journal: Economic Policy"},{"issue":"3","key":"2026040213480640900_ref098","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1111\/j.1083-6101.2007.00351.x","article-title":"\"In Google We Trust: Users\u2019 Decisions on Rank, Position, and Relevance\"","volume":"12","author":"Pan","year":"2007","journal-title":"Journal of Computer-Mediated Communication"},{"issue":"4","key":"2026040213480640900_ref099","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1145\/3186549.3186553","article-title":"\"On Measuring Bias in Online Information\"","volume":"46","author":"Pitoura","year":"2018","journal-title":"ACM SIGMOD Record"},{"key":"2026040213480640900_ref100","unstructured":"President\n              E. O. of the\n            \n          \n          2016\n          \"Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights\"\n          https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/microsites\/ostp\/2016_0504_data_discrimination.pdf"},{"issue":"6","key":"2026040213480640900_ref101","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1038\/s42256-019-0059-7","article-title":"\"Is it time to rethink institutional review boards for the era of big data?\"","volume":"1","author":"Prosperi","year":"2019","journal-title":"Nature Machine Intelligence"},{"key":"2026040213480640900_ref102","unstructured":"Purcell\n              K.\n            \n            \n              Brenner\n              J.\n            \n          \n          2012\n          \"Search Engine Use 2012\"\n          http:\/\/www.pewinternet.org\/2012\/03\/09\/search-engine-use-2012\/"},{"key":"2026040213480640900_ref103","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1145\/3306618.3314244","article-title":"\"Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products\"","volume-title":"Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society","author":"Raji","year":"2019"},{"key":"2026040213480640900_ref104","unstructured":"Robertson\n              A.\n            \n          \n          2019\n          \"Scraping public data from a website probably isn\u2019t hacking, says court\"\n          The Verge\n          https:\/\/www.theverge.com\/2019\/9\/10\/20859399\/linkedin-hiq-data-scraping-cfaa-lawsuitninth-circuit-ruling"},{"issue":"CSCW","key":"2026040213480640900_ref105","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3274417","article-title":"\"Auditing Partisan Audience Bias within Google Search\"","volume":"2","author":"Robertson","year":"2018","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"2026040213480640900_ref106","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1145\/3292522.3326047","article-title":"\"Auditing Autocomplete: Suggestion Networks and Recursive Algorithm Interrogation\"","author":"Robertson","year":"2019","journal-title":"Proceedings of the 10th ACM Conference on Web Science (WebSci '19)"},{"key":"2026040213480640900_ref107","doi-asserted-by":"publisher","first-page":"955","DOI":"10.1145\/3178876.3186143","article-title":"\"Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages\"","volume-title":"Proceedings of the 2018 World Wide Web Conference (WWW '18)","author":"Robertson","year":"2018"},{"key":"2026040213480640900_ref108","article-title":"\"How the Interplay of Google and Wikipedia Affects Perceptions of Online News Sources\"","volume-title":"Computation + Journalism Symposium","author":"Rothshild","year":"2019"},{"key":"2026040213480640900_ref109","unstructured":"Sandvig\n              C.\n            \n            \n              Hamilton\n              K.\n            \n            \n              Karahalios\n              K.\n            \n            \n              Langbort\n              C.\n            \n          \n          2014\n          \"Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms\"\n          Data and Discrimination: Converting Critical Concerns into Productive Inquiry\n          22"},{"key":"2026040213480640900_ref110","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1145\/3308560.3317595","article-title":"\"Quantifying the Impact of User Attention on Fair Group Representation in Ranked Lists\"","volume-title":"Companion Proceedings of the 2019 World Wide Web Conference (WWW '19)","author":"Sapiezynski","year":"2019"},{"key":"2026040213480640900_ref111","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1016\/j.econlet.2015.11.005","article-title":"\"Insurance, race\/ethnicity, and sex in the search for a new physician\"","volume":"137","author":"Sharma","year":"2015","journal-title":"Economics Letters"},{"key":"2026040213480640900_ref112","doi-asserted-by":"crossref","unstructured":"Shen\n              H.\n            \n            \n              DeVos\n              A.\n            \n            \n              Eslami\n              M.\n            \n            \n              Holstein\n              K.\n            \n          \n          2021\n          \"Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors\"\n          arXiv preprint\n          arXiv:2105.02980","DOI":"10.1145\/3479577"},{"key":"2026040213480640900_ref113","unstructured":"Siegelman\n              P.\n            \n            \n              Heckman\n              J.\n            \n          \n          1993\n          \"The Urban Institute Audit Studies: Their Methods and Findings\"\n          Clear and Convincing Evidence: Measurement of Discrimination in America\n          187\u2013258Washington"},{"key":"2026040213480640900_ref114","unstructured":"Sloane\n              M.\n            \n          \n          2021\n          \"The Algorithmic Auditing Trap\"\n          https:\/\/onezero.medium.com\/the-algorithmic-auditing-trap-9a6f2d4d461d"},{"key":"2026040213480640900_ref115","article-title":"\"Potential for discrimination in online targeted advertising\"","author":"Speicher","year":"2018","journal-title":"Conference on Fairness, Accountability, and Transparency (FAT '20)"},{"key":"2026040213480640900_ref116","unstructured":"\"Search Engine Market Share China\"\n          \n          2019\n          https:\/\/gs.statcounter.com\/search-engine-market-share\/all\/china\n          Accessed 08\/22\/2019"},{"key":"2026040213480640900_ref117","unstructured":"\"Search Engine Market Share Worldwide\"\n          \n          2019\n          https:\/\/gs.statcounter.com\/search-engine-market-share\n          Accessed 08\/22\/2019"},{"key":"2026040213480640900_ref118","doi-asserted-by":"crossref","unstructured":"Sweeney\n              L.\n            \n          \n          2013\n          \"Discrimination in Online Ad Delivery\"\n          https:\/\/dataprivacylab.org\/projects\/onlineads\/index.html\n          Accessed 10\/26\/2020","DOI":"10.2139\/ssrn.2208240"},{"issue":"3","key":"2026040213480640900_ref119","doi-asserted-by":"publisher","first-page":"10:10","DOI":"10.1145\/2460276.2460278","article-title":"\"Discrimination in Online Ad Delivery\"","volume":"11","author":"Sweeney","year":"2013","journal-title":"Queue"},{"key":"2026040213480640900_ref120","first-page":"81","author":"\"Google\u2019s Algorithm \u2013 Seek and You Shall Find\"","year":"2019","journal-title":"The Economist"},{"key":"2026040213480640900_ref121","unstructured":"Thomas\n              L.\n            \n          \n          2021\n          \"Supreme Court ruling that limits hacking law supports U M researcher\""},{"key":"2026040213480640900_ref122","unstructured":"Tober\n              M.\n            \n            \n              Grundmann\n              J.\n            \n            \n              Thakur\n              A.\n            \n          \n          2016\n          \"Universal & Extended Search 2016: Facts, Trends and Optimization Tips\"\n          searchmetrics"},{"key":"2026040213480640900_ref123","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3290605.3300683","article-title":"\"Search as News Curator: The Role of Google in Shaping Attention to News Information\"","author":"Trielli","year":"2019","journal-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19)"},{"key":"2026040213480640900_ref124","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/1369118X.2020.1764605","article-title":"\"Partisan Search Behavior and Google Results in the 2018 U.S. Midterm Elections\"","author":"Trielli","year":"2020","journal-title":"Information, Communication & Society"},{"key":"2026040213480640900_ref125","unstructured":"Tripodi\n              F.\n            \n          \n          2018\n          \"Searching for Alternative Facts\"\n          Data & Society\n          64"},{"key":"2026040213480640900_ref126","unstructured":"\"U.S. Government Accountability Office (U.S. GAO)\"\n          \n          https:\/\/www.gao.gov\n          Accessed 08\/01\/2021"},{"key":"2026040213480640900_ref127","unstructured":"Union\n              A. C. L.\n            \n          \n          2019\n          \"Sandvig v. Barr\u2013 Challenge to CFAA Prohibition on Uncovering Racial Discrimination Online\"\n          https:\/\/www.aclu.org\/cases\/sandvig-v-barr-challenge-cfaa-prohibition-uncovering-racial-discrimination-online\n          Accessed 01\/05\/2021"},{"key":"2026040213480640900_ref128","doi-asserted-by":"crossref","DOI":"10.1525\/9780520948693","volume-title":"The Googlization of Everything","author":"Vaidhyanathan","year":"2011"},{"issue":"CSCW1","key":"2026040213480640900_ref129","doi-asserted-by":"publisher","DOI":"10.1145\/3449078","article-title":"\"A Deeper Investigation of the Importance of Wikipedia Links to Search Engine Results\"","volume":"5","author":"Vincent","year":"2021","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"2026040213480640900_ref130","doi-asserted-by":"crossref","DOI":"10.1609\/icwsm.v13i01.3248","article-title":"\"Measuring the Importance of User-Generated Content to Search Engines\"","volume":"13","author":"Vincent","year":"2019","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"issue":"4","key":"2026040213480640900_ref131","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1145\/1629096.1629101","article-title":"\"Cyberchondria: Studies of the Escalation of Medical Concerns in Web Search\"","volume":"27","author":"White","year":"2009","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"2026040213480640900_ref132","first-page":"12","article-title":"\"Building and Auditing Fair Algorithms: A Case Study in Candidate Screening\"","author":"Wilson","year":"2021","journal-title":"Conference on Fairness, Accountability, and Transparency (FAccT '21)"},{"key":"2026040213480640900_ref133","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/978-3-319-04918-2_13","article-title":"\"Exposing Inconsistent Web Search Results with Bobble\"","author":"Xing","year":"2014","journal-title":"Passive and Active Measurement"},{"issue":"2","key":"2026040213480640900_ref134","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1257\/jep.12.2.23","article-title":"\"Evidence on discrimination in consumer markets\"","volume":"12","author":"Yinger","year":"1998","journal-title":"Journal of Economic Perspectives"},{"key":"2026040213480640900_ref135","unstructured":"Zetter\n              K.\n            \n          \n          2010\n          \"Wiseguys Plead Guilty in Ticketmaster Captcha Case\"\n          https:\/\/www.wired.com\/2010\/11\/wiseguysplead-guilty\/"},{"key":"2026040213480640900_ref136","unstructured":"Zetter\n              K.\n            \n          \n          2015\n          \"Experts Say Myspace Suicide Indictment Sets 'Scary' Legal Precedent\"\n          https:\/\/www.wired.com\/2008\/05\/myspace-indictm\/"},{"key":"2026040213480640900_ref137","unstructured":"Zetter\n              K.\n            \n          \n          2016\n          \"Researchers Sue the Government Over Computer Hacking Law\"\n          Wired\n          https:\/\/www.wired.com\/2016\/06\/researchers-sue-government-computer-hacking-law\/"}],"container-title":["Foundations and Trends\u00ae in Human\u2013Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/fthci\/article-pdf\/14\/4\/272\/10902946\/1100000083en.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.emerald.com\/fthci\/article-pdf\/14\/4\/272\/10902946\/1100000083en.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T14:26:29Z","timestamp":1777472789000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/fthci\/article\/14\/4\/272\/1320887\/Auditing-Algorithms-Understanding-Algorithmic"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,25]]},"references-count":137,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,11,25]]}},"URL":"https:\/\/doi.org\/10.1561\/1100000083","relation":{},"ISSN":["1551-3955","1551-3963"],"issn-type":[{"value":"1551-3955","type":"print"},{"value":"1551-3963","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,25]]}}}