{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T05:41:46Z","timestamp":1782279706559,"version":"3.54.5"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,9,28]]},"abstract":"<jats:p>Algorithm audits are powerful tools for studying black-box systems without direct knowledge of their inner workings. While very effective in examining technical components, the method stops short of a sociotechnical frame, which would also consider users themselves as an integral and dynamic part of the system. Addressing this limitation, we propose the concept of sociotechnical auditing: auditing methods that evaluate algorithmic systems at the sociotechnical level, focusing on the interplay between algorithms and users as each impacts the other. Just as algorithm audits probe an algorithm with varied inputs and observe outputs, a sociotechnical audit (STA) additionally probes users, exposing them to different algorithmic behavior and measuring their resulting attitudes and behaviors. As an example of this method, we develop Intervenr, a platform for conducting browser-based, longitudinal sociotechnical audits with consenting, compensated participants. Intervenr investigates the algorithmic content users encounter online, and also coordinates systematic client-side interventions to understand how users change in response. As a case study, we deploy Intervenr in a two-week sociotechnical audit of online advertising (N = 244) to investigate the central premise that personalized ad targeting is more effective on users. In the first week, we observe and collect all browser ads delivered to users, and in the second, we deploy an ablation-style intervention that disrupts normal targeting by randomly pairing participants and swapping all their ads. We collect user-oriented metrics (self-reported ad interest and feeling of representation) and advertiser-oriented metrics (ad views, clicks, and recognition) throughout, along with a total of over 500,000 ads. Our STA finds that targeted ads indeed perform better with users, but also that users begin to acclimate to different ads in only a week, casting doubt on the primacy of personalized ad targeting given the impact of repeated exposure. In comparison with other evaluation methods that only study technical components, or only experiment on users, sociotechnical audits evaluate sociotechnical systems through the interplay of their technical and human components.<\/jats:p>","DOI":"10.1145\/3610209","type":"journal-article","created":{"date-parts":[[2023,10,4]],"date-time":"2023-10-04T15:54:10Z","timestamp":1696434850000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":35,"title":["Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3448-5961","authenticated-orcid":false,"given":"Michelle S.","family":"Lam","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2630-8356","authenticated-orcid":false,"given":"Ayush","family":"Pandit","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-6353-8492","authenticated-orcid":false,"given":"Colin H.","family":"Kalicki","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9705-9057","authenticated-orcid":false,"given":"Rachit","family":"Gupta","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, Atlanta, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9121-5378","authenticated-orcid":false,"given":"Poonam","family":"Sahoo","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9359-6090","authenticated-orcid":false,"given":"Dana\u00eb","family":"Metaxa","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,10,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359301"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441801"},{"key":"e_1_2_1_3_1","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica May 23."},{"key":"e_1_2_1_4_1","volume-title":"Facebook Lets Advertisers Exclude Users by Race. ProPublica","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin and Terry Parris Jr. 2016. Facebook Lets Advertisers Exclude Users by Race. ProPublica (2016). https:\/\/www.propublica.org\/article\/facebook-lets-advertisers-exclude-users-by-race"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1177\/0022243719879711"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v14i1.7276"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2229012.2229027"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449148"},{"key":"e_1_2_1_9_1","unstructured":"Brakkton Booker. 2019. Housing Department Slaps Facebook With Discrimination Charge. https:\/\/www.npr.org\/2019\/03\/28\/707614254\/hud-slaps-facebook-with-housing-discrimination-charge"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research","volume":"91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77--91. https:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html"},{"key":"e_1_2_1_11_1","volume-title":"Exploring Trends in New York City Ads for College Scholarships. arXiv preprint arXiv:2109.15294","author":"Herbert Chang Ho-Chun","year":"2021","unstructured":"Ho-Chun Herbert Chang, Matt Bui, and Charlton McIlwain. 2021. Targeted Ads and\/as Racial Discrimination: Exploring Trends in New York City Ads for College Scholarships. arXiv preprint arXiv:2109.15294 (2021)."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174225"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815675.2815681"},{"key":"e_1_2_1_14_1","volume-title":"The principles of sociotechnical design. Human relations 29, 8","author":"Cherns Albert","year":"1976","unstructured":"Albert Cherns. 1976. The principles of sociotechnical design. Human relations 29, 8 (1976), 783--792."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533213"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858332"},{"key":"e_1_2_1_17_1","volume-title":"Michael Carl Tschantz, and Anupam Datta","author":"Datta Amit","year":"2014","unstructured":"Amit Datta, Michael Carl Tschantz, and Anupam Datta. 2014. Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. arXiv preprint arXiv:1408.6491 (2014)."},{"key":"e_1_2_1_18_1","volume-title":"Facebook Disables Ad Observatory","author":"DeLong Lois Anne","year":"2021","unstructured":"Lois Anne DeLong. 2021. Facebook Disables Ad Observatory; Academicians and Journalists Fire Back. https:\/\/cyber.nyu.edu\/2021\/08\/21\/facebook-disables-ad-observatory-academicians-and-journalists-fire-back\/"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1257\/app.20160213"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174006"},{"key":"e_1_2_1_21_1","volume-title":"Facebook's advertising platform: New attack vectors and the need for interventions. arXiv preprint arXiv:1803.10099","author":"Faizullabhoy Irfan","year":"2018","unstructured":"Irfan Faizullabhoy and Aleksandra Korolova. 2018. Facebook's advertising platform: New attack vectors and the need for interventions. arXiv preprint arXiv:1803.10099 (2018)."},{"key":"e_1_2_1_22_1","volume-title":"Philosophy and principles of auditing: an introduction","author":"Flint David","unstructured":"David Flint. 1988. Philosophy and principles of auditing: an introduction. Macmillan Education."},{"key":"e_1_2_1_23_1","unstructured":"E. Goffman. 1979. Gender Advertisements. Harper & Row. https:\/\/books.google.com\/books?id=HXxHAAAAMAAJ"},{"key":"e_1_2_1_24_1","volume-title":"Digital ad revenue in the US surpassed $100 billion for the first time","author":"Graham Megan","year":"2018","unstructured":"Megan Graham. 2019. Digital ad revenue in the US surpassed $100 billion for the first time in 2018. CNBC (2019). https:\/\/www.cnbc.com\/2019\/05\/07\/digital-ad-revenue-in-the-us-topped-100-billion-for-the-first-time.html"},{"key":"e_1_2_1_25_1","unstructured":"Tim Hwang. 2020. Subprime attention crisis: advertising and the time bomb at the heart of the Internet. FSG originals."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450077"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3097992"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170427.3188533"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3517745.3561450"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702520"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13063-020-4084-y"},{"key":"e_1_2_1_32_1","volume-title":"You Can't See What You Can't See: Experimental Evidence for How Much Relevant Information May Be Missed Due to Google's Web Search Personalisation","author":"Lai Cameron","unstructured":"Cameron Lai and Markus Luczak-Roesch. 2019. You Can't See What You Can't See: Experimental Evidence for How Much Relevant Information May Be Missed Due to Google's Web Search Personalisation. In Social Informatics, Ingmar Weber, Kareem M. Darwish, Claudia Wagner, Emilio Zagheni, Laura Nelson, Samin Aref, and Fabian Fl\u00f6ck (Eds.). Springer International Publishing, Cham, 253--266."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555625"},{"key":"e_1_2_1_34_1","volume-title":"Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management science 65, 7","author":"Lambrecht Anja","year":"2019","unstructured":"Anja Lambrecht and Catherine Tucker. 2019. Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads. Management science 65, 7 (2019), 2966--2981."},{"key":"e_1_2_1_35_1","volume-title":"Congress. arXiv:2109","author":"Lurie Emma","unstructured":"Emma Lurie and Deirdre K. Mulligan. 2021. Searching for Representation: A sociotechnical audit of googling for members of U.S. Congress. arXiv:2109.07012 [cs.CY]"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376672"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274388"},{"key":"e_1_2_1_38_1","volume-title":"Characterizing the Use of Browser-Based Blocking Extensions To Prevent Online Tracking. In Fourteenth symposium on usable privacy and security (SOUPS","author":"Mathur Arunesh","year":"2018","unstructured":"Arunesh Mathur, Jessica Vitak, Arvind Narayanan, and Marshini Chetty. 2018. Characterizing the Use of Browser-Based Blocking Extensions To Prevent Online Tracking. In Fourteenth symposium on usable privacy and security (SOUPS 2018). 103--116."},{"key":"e_1_2_1_39_1","volume-title":"Humans and algorithms work together-so study them together. Nature 617, 7960","author":"Matias J. Nathan","year":"2023","unstructured":"J. Nathan Matias. 2023. Humans and algorithms work together-so study them together. Nature 617, 7960 (2023), 248--251."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-38277-5"},{"key":"e_1_2_1_41_1","volume-title":"Algorithmic Discrimination: A Framework and Approach to Auditing & Measuring the Impact of Race-Targeted Digital Advertising. PolicyLink (May","author":"McIlwain Charlton","year":"2023","unstructured":"Charlton McIlwain. 2023. Algorithmic Discrimination: A Framework and Approach to Auditing & Measuring the Impact of Race-Targeted Digital Advertising. PolicyLink (May 2023)."},{"key":"e_1_2_1_42_1","volume-title":"Accelerated Hierarchical Density Based Clustering. In Data Mining Workshops (ICDMW), 2017 IEEE International Conference on. IEEE, IEEE","author":"McInnes Leland","year":"2017","unstructured":"Leland McInnes and John Healy. 2017. Accelerated Hierarchical Density Based Clustering. In Data Mining Workshops (ICDMW), 2017 IEEE International Conference on. IEEE, IEEE, New York, NY, 33--42."},{"key":"e_1_2_1_43_1","volume-title":"UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. ArXiv e-prints (Feb","author":"McInnes L.","year":"2018","unstructured":"L. McInnes, J. Healy, and J. Melville. 2018. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. ArXiv e-prints (Feb 2018), 63 pages. arXiv:1802.03426 [stat.ML]"},{"key":"e_1_2_1_44_1","volume-title":"Bias and Discrimination in Digital Advertising. PolicyLink (May","author":"Metaxa Dana\u00eb","year":"2023","unstructured":"Dana\u00eb Metaxa. 2023. Bias and Discrimination in Digital Advertising. PolicyLink (May 2023)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449100"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359231"},{"key":"e_1_2_1_47_1","volume-title":"Ronald E Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, Christian Sandvig, et al .","author":"Metaxa Dana\u00eb","year":"2021","unstructured":"Dana\u00eb Metaxa, Joon Sung Park, Ronald E Robertson, Karrie Karahalios, Christo Wilson, Jeff Hancock, Christian Sandvig, et al . 2021. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends\u00ae in Human--Computer Interaction 14, 4 (2021), 272--344."},{"key":"e_1_2_1_48_1","volume-title":"Constantin Waubert de Puiseau, and Tobias Meisen","author":"Meyes Richard","year":"2019","unstructured":"Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. 2019. Ablation studies in artificial neural networks. arXiv preprint arXiv:1901.08644 (2019)."},{"key":"e_1_2_1_49_1","unstructured":"Mozilla. 2021. Take back control from the algorithm. https:\/\/foundation.mozilla.org\/en\/youtube\/regretsreporter\/."},{"key":"e_1_2_1_50_1","unstructured":"Mozilla. 2021. Take control over your data with Rally a novel privacy-first data sharing platform. https:\/\/blog.mozilla.org\/en\/mozilla\/take-control-over-your-data-with-rally-a-novel-privacy-first-data-sharing-platform\/."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372835"},{"key":"e_1_2_1_52_1","volume-title":"TrackMeNot: Resisting Surveillance in Web Search. Lessons from the Identity Trail: Anonymity, Privacy, and Identity in a Networked Society 1, 1","author":"Nissenbaum Helen","year":"2009","unstructured":"Helen Nissenbaum and Howe Daniel. 2009. TrackMeNot: Resisting Surveillance in Web Search. Lessons from the Identity Trail: Anonymity, Privacy, and Identity in a Networked Society 1, 1 (2009), 23 pages."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aax2342"},{"key":"e_1_2_1_54_1","volume-title":"Mobile Technology and Home Broadband","author":"Perrin Andrew","year":"2021","unstructured":"Andrew Perrin. 2021. Mobile Technology and Home Broadband 2021. https:\/\/www.pewresearch.org\/internet\/2021\/06\/03\/mobile-technology-and-home-broadband-2021\/"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1002\/ev.1041"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2103.00020"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370024.2019.1578652"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2013.847762"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274417"},{"key":"e_1_2_1_62_1","unstructured":"Lydia Saad. 2015. Most Smartphone Users Still Rely on Computer for Web Purchases. https:\/\/news.gallup.com\/poll\/184037\/smartphone-users-rely-computer-web-purchases.aspx"},{"key":"e_1_2_1_63_1","volume-title":"Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and discrimination: converting critical concerns into productive inquiry 22","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014), 23 pages."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514094.3534135"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4398428"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479577"},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research","volume":"19","author":"Speicher Till","year":"2018","unstructured":"Till Speicher, Muhammad Ali, Giridhari Venkatadri, Filipe Ribeiro, George Arvanitakis, Fabr\u00edcio Benevenuto, Krishna Gummadi, Patrick Loiseau, and Alan Mislove. 2018. Potential for discrimination in online targeted advertising. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 5--19. https:\/\/proceedings.mlr.press\/v81\/speicher18a.html"},{"key":"e_1_2_1_68_1","volume-title":"Comparing to perfection: How cultural norms for appearance affect social comparisons and self-image. Body image 3, 3","author":"Strahan Erin J","year":"2006","unstructured":"Erin J Strahan, Anne E Wilson, Kate E Cressman, and Vanessa M Buote. 2006. Comparing to perfection: How cultural norms for appearance affect social comparisons and self-image. Body image 3, 3 (2006), 211--227."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460276.2460278"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1509\/jmr.10.0355"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419394.3423641"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445928"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445459"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3517745.3561460"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610209","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610209","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T04:23:43Z","timestamp":1755750223000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610209"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"references-count":74,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2023,9,28]]}},"alternative-id":["10.1145\/3610209"],"URL":"https:\/\/doi.org\/10.1145\/3610209","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,28]]},"assertion":[{"value":"2023-10-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}