{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T10:25:28Z","timestamp":1790245528654,"version":"4.1.0"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2022,3,30]],"date-time":"2022-03-30T00:00:00Z","timestamp":1648598400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["1953513"],"award-info":[{"award-number":["1953513"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,3,30]]},"abstract":"<jats:p>How can society understand and hold accountable complex human and algorithmic decision-making systems whose systematic errors are opaque to the public? These systems routinely make decisions on individual rights and well-being, and on protecting society and the democratic process. Practical and statistical constraints on external audits--such as dimensional complexity--can lead researchers and regulators to miss important sources of error in these complex decision-making systems. In this paper, we design and implement a software-supported approach to audit studies that auto-generates audit materials and coordinates volunteer activity. We implemented this software in the case of political advertising policies enacted by Facebook and Google during the 2018 U.S. election. Guided by this software, a team of volunteers posted 477 auto-generated ads and analyzed the companies' actions, finding systematic errors in how companies enforced policies. We find that software can overcome some common constraints of audit studies, within limitations related to sample size and volunteer capacity.<\/jats:p>","DOI":"10.1145\/3512965","type":"journal-article","created":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T16:54:59Z","timestamp":1649350499000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Software-Supported Audits of Decision-Making Systems: Testing Google and Facebook's Political Advertising Policies"],"prefix":"10.1145","volume":"6","author":[{"given":"J. Nathan","family":"Matias","sequence":"first","affiliation":[{"name":"Cornell University, Ithaca, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Austin","family":"Hounsel","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, NJ, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nick","family":"Feamster","sequence":"additional","affiliation":[{"name":"University of Chicago, Chicago, IL, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,4,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-017-1116-3"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359301"},{"key":"e_1_2_1_4_1","volume-title":"Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging. arXiv preprint arXiv:1912.04255","author":"Ali Muhammad","year":"2019","unstructured":"Muhammad Ali , Piotr Sapiezynski , Aleksandra Korolova , Alan Mislove , and Aaron Rieke . 2019. Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging. arXiv preprint arXiv:1912.04255 ( 2019 ). Muhammad Ali, Piotr Sapiezynski, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2019. Ad Delivery Algorithms: The Hidden Arbiters of Political Messaging. arXiv preprint arXiv:1912.04255 (2019)."},{"key":"e_1_2_1_5_1","volume-title":"ProPublica","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016 . Machine bias . ProPublica , May 23 (2016), 2016. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, May 23 (2016), 2016."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1257\/jep.25.3.63"},{"key":"e_1_2_1_7_1","volume-title":"Engineering the American dream: a study of bias and perceptions of merit in the high-tech labor market. Master's thesis","author":"Marie Barabas Chelsea Chelsea","unstructured":"Chelsea Chelsea Marie Barabas . 2015. Engineering the American dream: a study of bias and perceptions of merit in the high-tech labor market. Master's thesis . Massachusetts Institute of Technology . Chelsea Chelsea Marie Barabas. 2015. Engineering the American dream: a study of bias and perceptions of merit in the high-tech labor market. Master's thesis. Massachusetts Institute of Technology."},{"key":"e_1_2_1_8_1","volume-title":"Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American economic review 94, 4","author":"Bertrand Marianne","year":"2004","unstructured":"Marianne Bertrand and Sendhil Mullainathan . 2004. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American economic review 94, 4 ( 2004 ), 991--1013. Marianne Bertrand and Sendhil Mullainathan. 2004. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American economic review 94, 4 (2004), 991--1013."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0003055419000194"},{"key":"e_1_2_1_10_1","volume-title":"Facebook and Twitter Plan New Ways to Regulate Political Ads - The New York Times. New York Times (May","author":"Bowles Nellie","year":"2018","unstructured":"Nellie Bowles and Sheera Frenkel . 2018. Facebook and Twitter Plan New Ways to Regulate Political Ads - The New York Times. New York Times (May 2018 ). https:\/\/www.nytimes.com\/2018\/05\/24\/technology\/twitter-political-ad-restrictions.html Nellie Bowles and Sheera Frenkel. 2018. Facebook and Twitter Plan New Ways to Regulate Political Ads - The New York Times. New York Times (May 2018). https:\/\/www.nytimes.com\/2018\/05\/24\/technology\/twitter-political-ad-restrictions.html"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1009213286"},{"key":"e_1_2_1_12_1","volume-title":"Encore: Lightweight measurement of web censorship with cross-origin requests. In ACM SIGCOMM Computer Communication Review","author":"Burnett Sam","year":"2015","unstructured":"Sam Burnett and Nick Feamster . 2015 . Encore: Lightweight measurement of web censorship with cross-origin requests. In ACM SIGCOMM Computer Communication Review , Vol. 45 . ACM , 653--667. Sam Burnett and Nick Feamster. 2015. Encore: Lightweight measurement of web censorship with cross-origin requests. In ACM SIGCOMM Computer Communication Review, Vol. 45. ACM, 653--667."},{"key":"e_1_2_1_13_1","unstructured":"C-Span. 2018. Senator Al Franken Questions Facebook V.P. About Political Ads Purchased with Foreign Currency. http:\/\/cs.pn\/2FxrMSf. Accessed 12 March 2018.  C-Span. 2018. Senator Al Franken Questions Facebook V.P. About Political Ads Purchased with Foreign Currency. http:\/\/cs.pn\/2FxrMSf. Accessed 12 March 2018."},{"key":"e_1_2_1_14_1","unstructured":"Catherine Card. 2018. How Facebook AI Helps Suicide Prevention | Facebook Newsroom. https:\/\/newsroom.fb.com\/news\/2018\/09\/inside-feed-suicide-prevention-and-ai\/  Catherine Card. 2018. How Facebook AI Helps Suicide Prevention | Facebook Newsroom. https:\/\/newsroom.fb.com\/news\/2018\/09\/inside-feed-suicide-prevention-and-ai\/"},{"key":"e_1_2_1_15_1","volume-title":"Can AI Help Reduce Disparities in General Medical and Mental Health Care? AMA journal of ethics 21, 2","author":"Chen Irene Y.","year":"2019","unstructured":"Irene Y. Chen , Peter Szolovits , and Marzyeh Ghassemi . 2019. Can AI Help Reduce Disparities in General Medical and Mental Health Care? AMA journal of ethics 21, 2 ( 2019 ), 167--179. Irene Y. Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019. Can AI Help Reduce Disparities in General Medical and Mental Health Care? AMA journal of ethics 21, 2 (2019), 167--179."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3055589.3055594"},{"key":"e_1_2_1_17_1","volume-title":"Conference on Fairness, Accountability and Transparency. 134--148","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova , Diana Benavides-Prado , Oleksandr Fialko , and Rhema Vaithianathan . 2018 . A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions . In Conference on Fairness, Accountability and Transparency. 134--148 . Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148."},{"key":"e_1_2_1_18_1","volume-title":"Full-cycle social psychology. Applied social psychology annual","author":"Cialdini Robert B.","year":"1980","unstructured":"Robert B. Cialdini . 1980. Full-cycle social psychology. Applied social psychology annual ( 1980 ). Robert B. Cialdini. 1980. Full-cycle social psychology. Applied social psychology annual (1980)."},{"key":"e_1_2_1_19_1","volume-title":"Facebook will hire 1,000 and make ads visible to fight election interference. TechCrunch","author":"Constine John","year":"2017","unstructured":"John Constine . 2017. Facebook will hire 1,000 and make ads visible to fight election interference. TechCrunch ( 2017 ). https:\/\/techcrunch.com\/2017\/10\/02\/facebook-will-hire-1000-and-make-ads-visible-to-fight-election-interference\/ John Constine. 2017. Facebook will hire 1,000 and make ads visible to fight election interference. TechCrunch (2017). https:\/\/techcrunch.com\/2017\/10\/02\/facebook-will-hire-1000-and-make-ads-visible-to-fight-election-interference\/"},{"key":"e_1_2_1_20_1","volume-title":"The small effects of political advertising are small regardless of context, message, sender, or receiver: Evidence from 59 real-time randomized experiments. Science Advances 6, 36 (Sept","author":"Coppock Alexander","year":"2020","unstructured":"Alexander Coppock , Seth J. Hill , and Lynn Vavreck . 2020. The small effects of political advertising are small regardless of context, message, sender, or receiver: Evidence from 59 real-time randomized experiments. Science Advances 6, 36 (Sept . 2020 ), eabc4046. https:\/\/doi.org\/10.1126\/sciadv.abc4046 Publisher : American Association for the Advancement of Science Section: Research Article. 10.1126\/sciadv.abc4046 Alexander Coppock, Seth J. Hill, and Lynn Vavreck. 2020. The small effects of political advertising are small regardless of context, message, sender, or receiver: Evidence from 59 real-time randomized experiments. Science Advances 6, 36 (Sept. 2020), eabc4046. https:\/\/doi.org\/10.1126\/sciadv.abc4046 Publisher: American Association for the Advancement of Science Section: Research Article."},{"key":"e_1_2_1_21_1","volume-title":"Ethical Challenges and Some Solutions for Field Experiments","author":"Desposato Scott","unstructured":"Scott Desposato . 2014. Ethical Challenges and Some Solutions for Field Experiments . San Diego : University of California . Available at www. desposato. org\/ethicsfieldexperiments. pdf (2014). Scott Desposato. 2014. Ethical Challenges and Some Solutions for Field Experiments. San Diego: University of California. Available at www. desposato. org\/ethicsfieldexperiments. pdf (2014)."},{"key":"e_1_2_1_22_1","unstructured":"Nicholas Diakopoulos. 2014. Algorithmic accountability reporting: On the investigation of black boxes. (2014).  Nicholas Diakopoulos. 2014. Algorithmic accountability reporting: On the investigation of black boxes. (2014)."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP40000.2020.00084"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978313"},{"key":"e_1_2_1_25_1","volume-title":"Hard Questions: Russian Ads Delivered to Congress","year":"2018","unstructured":"Facebook. 2018 . Hard Questions: Russian Ads Delivered to Congress . http:\/\/bit.ly\/2xbGOnD. Accessed 05 March 2018. Facebook. 2018. Hard Questions: Russian Ads Delivered to Congress. http:\/\/bit.ly\/2xbGOnD. Accessed 05 March 2018."},{"key":"e_1_2_1_26_1","volume-title":"OONI: Open Observatory of Network Interference.. In FOCI.","author":"Filasto Arturo","year":"2012","unstructured":"Arturo Filasto and Jacob Appelbaum . 2012 . OONI: Open Observatory of Network Interference.. In FOCI. Arturo Filasto and Jacob Appelbaum. 2012. OONI: Open Observatory of Network Interference.. In FOCI."},{"key":"e_1_2_1_27_1","volume-title":"Facebook's Problem With Veterans. Wall Street Journal (Oct","author":"Gale David","year":"2018","unstructured":"David Gale . 2018. Facebook's Problem With Veterans. Wall Street Journal (Oct . 2018 ). https:\/\/www.wsj.com\/articles\/facebooks-problem-with-veterans-1533682511?mod=searchresults&page=3&pos=3&ns=prod\/accounts-wsj David Gale. 2018. Facebook's Problem With Veterans. Wall Street Journal (Oct. 2018). https:\/\/www.wsj.com\/articles\/facebooks-problem-with-veterans-1533682511?mod=searchresults&page=3&pos=3&ns=prod\/accounts-wsj"},{"key":"e_1_2_1_28_1","volume-title":"Trace ethnography: Following coordination through documentary practices. In 2011 44th hawaii international conference on system sciences","author":"Stuart Geiger R.","unstructured":"R. Stuart Geiger and David Ribes . 2011. Trace ethnography: Following coordination through documentary practices. In 2011 44th hawaii international conference on system sciences . IEEE , 1--10. R. Stuart Geiger and David Ribes. 2011. Trace ethnography: Following coordination through documentary practices. In 2011 44th hawaii international conference on system sciences. IEEE, 1--10."},{"key":"e_1_2_1_29_1","volume-title":"Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media","author":"Gillespie Tarleton","unstructured":"Tarleton Gillespie . 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media . Yale University Press . Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press."},{"key":"e_1_2_1_30_1","volume-title":"Google's Tool to Tame Election Influence Has Flaws. Wall Street Journal","author":"Glazer Emily","year":"2019","unstructured":"Emily Glazer and Patience Haggin . 2019. Google's Tool to Tame Election Influence Has Flaws. Wall Street Journal ( 2019 ). https:\/\/www.wsj.com\/articles\/google-archive-of-political-ads-is-fraught-with-missing-content-delays-11563355800 Emily Glazer and Patience Haggin. 2019. Google's Tool to Tame Election Influence Has Flaws. Wall Street Journal (2019). https:\/\/www.wsj.com\/articles\/google-archive-of-political-ads-is-fraught-with-missing-content-delays-11563355800"},{"key":"e_1_2_1_31_1","unstructured":"Jon Hakken. 1979. Discrimination against Chicanos in the Dallas rental housing market: An experimental extension of the housing market practices survey. Division of Evaluation US Dept. of Housing and Urban Development Office of Policy Development.  Jon Hakken. 1979. Discrimination against Chicanos in the Dallas rental housing market: An experimental extension of the housing market practices survey. Division of Evaluation US Dept. of Housing and Urban Development Office of Policy Development."},{"key":"e_1_2_1_32_1","first-page":"63","article-title":"When Does an Advertisement about Issues become an Issues Ad","volume":"49","author":"Hayward Allison R.","year":"1999","unstructured":"Allison R. Hayward . 1999 . When Does an Advertisement about Issues become an Issues Ad . Cath. UL Rev. 49 (1999), 63 . Allison R. Hayward. 1999. When Does an Advertisement about Issues become an Issues Ad. Cath. UL Rev. 49 (1999), 63.","journal-title":"Cath. UL Rev."},{"key":"e_1_2_1_33_1","volume-title":"Franken Blasts Facebook For Accepting Rubles For U.S. Election Ads. https:\/\/bit.ly\/2znoeyb. Accessed","author":"Hill The","year":"2018","unstructured":"The Hill . 2017. Franken Blasts Facebook For Accepting Rubles For U.S. Election Ads. https:\/\/bit.ly\/2znoeyb. Accessed 30 April 2018 . The Hill. 2017. Franken Blasts Facebook For Accepting Rubles For U.S. Election Ads. https:\/\/bit.ly\/2znoeyb. Accessed 30 April 2018."},{"key":"e_1_2_1_34_1","volume-title":"Russian Influence Reached 126 Million Through Facebook Alone. The New York Times (Oct","author":"Isaac Mike","year":"2017","unstructured":"Mike Isaac and Daisuke Wakabayashi . 2017. Russian Influence Reached 126 Million Through Facebook Alone. The New York Times (Oct . 2017 ). https:\/\/www.nytimes.com\/2017\/10\/30\/technology\/facebook-google-russia.html Mike Isaac and Daisuke Wakabayashi. 2017. Russian Influence Reached 126 Million Through Facebook Alone. The New York Times (Oct. 2017). https:\/\/www.nytimes.com\/2017\/10\/30\/technology\/facebook-google-russia.html"},{"key":"e_1_2_1_35_1","volume-title":"Subjective probability: A judgment of representativeness. Cognitive psychology 3, 3","author":"Kahneman Daniel","year":"1972","unstructured":"Daniel Kahneman and Amos Tversky . 1972. Subjective probability: A judgment of representativeness. Cognitive psychology 3, 3 ( 1972 ), 430--454. Daniel Kahneman and Amos Tversky. 1972. Subjective probability: A judgment of representativeness. Cognitive psychology 3, 3 (1972), 430--454."},{"key":"e_1_2_1_36_1","volume-title":"International Conference on Machine Learning. 2564--2572","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns , Seth Neel , Aaron Roth , and Zhiwei Steven Wu . 2018 . Preventing fairness gerrymandering: Auditing and learning for subgroup fairness . In International Conference on Machine Learning. 2564--2572 . Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning. 2564--2572."},{"key":"e_1_2_1_37_1","volume-title":"Roberts","author":"King Gary","year":"2014","unstructured":"Gary King , Jennifer Pan , and Margaret E . Roberts . 2014 . Reverse-engineering censorship in China : Randomized experimentation and participant observation. Science 345, 6199 (2014), 1251722. Gary King, Jennifer Pan, and Margaret E. Roberts. 2014. Reverse-engineering censorship in China: Randomized experimentation and participant observation. Science 345, 6199 (2014), 1251722."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2016.1154087"},{"key":"e_1_2_1_39_1","unstructured":"Rob Leathern. 2018. Shining a Light on Ads With Political Content | Facebook Newsroom. https:\/\/newsroom.fb.com\/news\/2018\/05\/ads-with-political-content\/  Rob Leathern. 2018. Shining a Light on Ads With Political Content | Facebook Newsroom. https:\/\/newsroom.fb.com\/news\/2018\/05\/ads-with-political-content\/"},{"key":"e_1_2_1_40_1","volume-title":"Consent of the networked: The worldwide struggle for Internet freedom","author":"MacKinnon Rebecca","unstructured":"Rebecca MacKinnon . 2012. Consent of the networked: The worldwide struggle for Internet freedom . Vol. 50 . Basic Books . Rebecca MacKinnon. 2012. Consent of the networked: The worldwide struggle for Internet freedom. Vol. 50. Basic Books."},{"key":"e_1_2_1_41_1","volume-title":"Protecting Privacy and Expression Online. Access Controlled: The Shaping of Power, Rights, and Rules in Cyberspace","author":"Maclay Colin","year":"2010","unstructured":"Colin Maclay . 2010. Protecting Privacy and Expression Online. Access Controlled: The Shaping of Power, Rights, and Rules in Cyberspace ( 2010 ), 87--108. Colin Maclay. 2010. Protecting Privacy and Expression Online. Access Controlled: The Shaping of Power, Rights, and Rules in Cyberspace (2010), 87--108."},{"key":"e_1_2_1_42_1","volume-title":"Will Facebook's New Ad Transparency Protect Democracy? The Atlantic (May","author":"Madrigal Alexis","year":"2018","unstructured":"Alexis Madrigal . 2018. Will Facebook's New Ad Transparency Protect Democracy? The Atlantic (May 2018 ). https:\/\/www.theatlantic.com\/technology\/archive\/2018\/05\/facebook-ad-transparency-democracy\/559853\/ Alexis Madrigal. 2018. Will Facebook's New Ad Transparency Protect Democracy? The Atlantic (May 2018). https:\/\/www.theatlantic.com\/technology\/archive\/2018\/05\/facebook-ad-transparency-democracy\/559853\/"},{"key":"e_1_2_1_43_1","volume-title":"Facebook Thought an Ad From Bush's Baked Beans Was \"Political\" and Removed It. Slate Magazine (May","author":"Mak Aaron","year":"2018","unstructured":"Aaron Mak . 2018. Facebook Thought an Ad From Bush's Baked Beans Was \"Political\" and Removed It. Slate Magazine (May 2018 ). https:\/\/slate.com\/technology\/2018\/05\/bushs-baked-beans-fell-victim-to-facebooks-political-ads-system.html Aaron Mak. 2018. Facebook Thought an Ad From Bush's Baked Beans Was \"Political\" and Removed It. Slate Magazine (May 2018). https:\/\/slate.com\/technology\/2018\/05\/bushs-baked-beans-fell-victim-to-facebooks-political-ads-system.html"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173583"},{"key":"e_1_2_1_45_1","unstructured":"J. Nathan Matias Megan Steiner and Aimee Rickman. 2017. Who Gets to Use Facebook's Rainbow 'Pride' Reaction? https:\/\/www.theatlantic.com\/technology\/archive\/2017\/06\/facebook-pride-reaction\/531633\/  J. Nathan Matias Megan Steiner and Aimee Rickman. 2017. Who Gets to Use Facebook's Rainbow 'Pride' Reaction? https:\/\/www.theatlantic.com\/technology\/archive\/2017\/06\/facebook-pride-reaction\/531633\/"},{"key":"e_1_2_1_46_1","volume-title":"Merrill and Ariana Tobin","author":"Jeremy","year":"2018","unstructured":"Jeremy B. Merrill and Ariana Tobin . 2018 . Facebook's Screening for Political Ads Nabs News Sites Instead of Politicians. ProPublica ( 2018). https:\/\/www.propublica.org\/article\/facebook-new-screening-system-flags-the-wrong-ads-as-political Jeremy B. Merrill and Ariana Tobin. 2018. Facebook's Screening for Political Ads Nabs News Sites Instead of Politicians. ProPublica (2018). https:\/\/www.propublica.org\/article\/facebook-new-screening-system-flags-the-wrong-ads-as-political"},{"key":"e_1_2_1_47_1","volume-title":"Facebook's experiment causes a lot of fuss for little result","author":"Morin Rich","year":"2014","unstructured":"Rich Morin . 2014. Facebook's experiment causes a lot of fuss for little result . Pew Research Center 2 ( 2014 ). Rich Morin. 2014. Facebook's experiment causes a lot of fuss for little result. Pew Research Center 2 (2014)."},{"key":"e_1_2_1_48_1","volume-title":"Biased algorithms are easier to fix than biased people. The New York Times","author":"Mullainathan Sendhil","year":"2019","unstructured":"Sendhil Mullainathan . 2019. Biased algorithms are easier to fix than biased people. The New York Times ( 2019 ). Sendhil Mullainathan. 2019. Biased algorithms are easier to fix than biased people. The New York Times (2019)."},{"key":"e_1_2_1_49_1","volume-title":"Transparent data mining for big and small data","author":"Narayanan Arvind","unstructured":"Arvind Narayanan and Dillon Reisman . 2017. The princeton web transparency and accountability project . In Transparent data mining for big and small data . Springer , 45--67. Arvind Narayanan and Dillon Reisman. 2017. The princeton web transparency and accountability project. In Transparent data mining for big and small data. Springer, 45--67."},{"key":"e_1_2_1_50_1","unstructured":"Barbara Ortutay and Ap Technology Writer. 2019. Facebook tightens political ad rules but leaves loopholes. https:\/\/www.chron.com\/business\/technology\/article\/Facebook-tightens-political-ad-rules-but-leaves-14384058.php  Barbara Ortutay and Ap Technology Writer. 2019. Facebook tightens political ad rules but leaves loopholes. https:\/\/www.chron.com\/business\/technology\/article\/Facebook-tightens-political-ad-rules-but-leaves-14384058.php"},{"key":"e_1_2_1_51_1","volume-title":"The mark of a criminal record. American journal of sociology 108, 5","author":"Pager Devah","year":"2003","unstructured":"Devah Pager . 2003. The mark of a criminal record. American journal of sociology 108, 5 ( 2003 ), 937--975. Devah Pager. 2003. The mark of a criminal record. American journal of sociology 108, 5 (2003), 937--975."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1177\/0002716206294796"},{"key":"e_1_2_1_53_1","unstructured":"Politico. 2018. Full Mueller indictment on Russian election case. http:\/\/politi.co\/2F7GknX. Accessed 05 March 2018.  Politico. 2018. Full Mueller indictment on Russian election case. http:\/\/politi.co\/2F7GknX. Accessed 05 March 2018."},{"key":"e_1_2_1_54_1","volume-title":"The open society and its enemies","author":"Popper Karl","unstructured":"Karl Popper . 1947. The open society and its enemies . Routledge . Karl Popper. 1947. The open society and its enemies. Routledge."},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","unstructured":"Robert D. Putnam. 2000. Bowling alone: The collapse and revival of American community. Simon and schuster.  Robert D. Putnam. 2000. Bowling alone: The collapse and revival of American community. Simon and schuster.","DOI":"10.1145\/358916.361990"},{"key":"e_1_2_1_56_1","volume-title":"Midtb\u00f8en","author":"Quillian Lincoln","year":"2017","unstructured":"Lincoln Quillian , Devah Pager , Ole Hexel , and Arnfinn H . Midtb\u00f8en . 2017 . Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences 114, 41 (Oct. 2017), 10870--10875. https:\/\/doi.org\/10.1073\/pnas.1706255114 Publisher : National Academy of Sciences Section : Social Sciences. 10.1073\/pnas.1706255114 Lincoln Quillian, Devah Pager, Ole Hexel, and Arnfinn H. Midtb\u00f8en. 2017. Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences 114, 41 (Oct. 2017), 10870--10875. https:\/\/doi.org\/10.1073\/pnas.1706255114 Publisher: National Academy of Sciences Section: Social Sciences."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1138-y"},{"key":"e_1_2_1_58_1","volume-title":"Politicians who want to run ads on Google will have new rules to follow. CNET (May","year":"2018","unstructured":"Reardon. 2018. Politicians who want to run ads on Google will have new rules to follow. CNET (May 2018 ). https:\/\/www.cnet.com\/news\/google-is-rolling-out-new-rules-for-us-political-ads\/ Reardon. 2018. Politicians who want to run ads on Google will have new rules to follow. CNET (May 2018). https:\/\/www.cnet.com\/news\/google-is-rolling-out-new-rules-for-us-political-ads\/"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287580"},{"key":"e_1_2_1_61_1","volume-title":"Facebook blocked many gay-themed ads as part of its new advertising policy, angering LGBT groups. Washington Post (Oct","author":"Rosenberg Eli","year":"2018","unstructured":"Eli Rosenberg . 2018. Facebook blocked many gay-themed ads as part of its new advertising policy, angering LGBT groups. Washington Post (Oct . 2018 ). https:\/\/www.washingtonpost.com\/technology\/2018\/10\/03\/facebook-blocked-many-gay-themed-ads-part-its-new-advertising-policy-angering-lgbt-groups\/ Eli Rosenberg. 2018. Facebook blocked many gay-themed ads as part of its new advertising policy, angering LGBT groups. Washington Post (Oct. 2018). https:\/\/www.washingtonpost.com\/technology\/2018\/10\/03\/facebook-blocked-many-gay-themed-ads-part-its-new-advertising-policy-angering-lgbt-groups\/"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1715243114"},{"key":"e_1_2_1_63_1","volume-title":"Watts","author":"Salganik Matthew J.","year":"2009","unstructured":"Matthew J. Salganik and Duncan J . Watts . 2009 . Web-based experiments for the study of collective social dynamics in cultural markets. Topics in cognitive science 1, 3 (2009), 439--468. Matthew J. Salganik and Duncan J. Watts. 2009. Web-based experiments for the study of collective social dynamics in cultural markets. Topics in cognitive science 1, 3 (2009), 439--468."},{"key":"e_1_2_1_64_1","unstructured":"Sheryl Sandberg. 2019. A Second Update on Our Civil Rights Audit | Facebook Newsroom. https:\/\/newsroom.fb.com\/news\/2019\/06\/second-update-civil-rights-audit\/  Sheryl Sandberg. 2019. A Second Update on Our Civil Rights Audit | Facebook Newsroom. https:\/\/newsroom.fb.com\/news\/2019\/06\/second-update-civil-rights-audit\/"},{"key":"e_1_2_1_65_1","volume-title":"Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig , Kevin Hamilton , Karrie Karahalios , and Cedric Langbort . 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 ( 2014 ). Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22 (2014)."},{"key":"e_1_2_1_66_1","first-page":"171","article-title":"Free speech unmoored in copyright's safe harbor: Chilling effects of the DMCA on the first amendment","volume":"24","author":"Seltzer Wendy","year":"2010","unstructured":"Wendy Seltzer . 2010 . Free speech unmoored in copyright's safe harbor: Chilling effects of the DMCA on the first amendment . Harv. JL & Tech. 24 (2010), 171 . Publisher: HeinOnline. Wendy Seltzer. 2010. Free speech unmoored in copyright's safe harbor: Chilling effects of the DMCA on the first amendment. Harv. JL & Tech. 24 (2010), 171. Publisher: HeinOnline.","journal-title":"Harv. JL & Tech."},{"key":"e_1_2_1_67_1","volume-title":"Race as a bundle of sticks: Designs that estimate effects of seemingly immutable characteristics. Annual Review of Political Science 19","author":"Sen Maya","year":"2016","unstructured":"Maya Sen and Omar Wasow . 2016. Race as a bundle of sticks: Designs that estimate effects of seemingly immutable characteristics. Annual Review of Political Science 19 ( 2016 ). Maya Sen and Omar Wasow. 2016. Race as a bundle of sticks: Designs that estimate effects of seemingly immutable characteristics. Annual Review of Political Science 19 (2016)."},{"key":"e_1_2_1_68_1","volume-title":"Campbell","author":"Shadish William R.","year":"2002","unstructured":"William R. Shadish , Thomas D. Cook , and Donald T . Campbell . 2002 . Experimental and quasi-experimental designs for generalized causal inference. (2002). William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. (2002)."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380109"},{"key":"e_1_2_1_70_1","volume-title":"Introducing a new transparency report for political ads. Google","author":"Smith Michee","year":"2018","unstructured":"Michee Smith . 2018. Introducing a new transparency report for political ads. Google ( 2018 ). https:\/\/www.blog.google\/technology\/ads\/introducing-new-transparency-report-political-ads\/ Michee Smith. 2018. Introducing a new transparency report for political ads. Google (2018). https:\/\/www.blog.google\/technology\/ads\/introducing-new-transparency-report-political-ads\/"},{"key":"e_1_2_1_71_1","volume-title":"Facebook denies Houston's ads promoting fair housing over race, religion references. Houston Chronicle (May","author":"Smith Sarah","year":"2019","unstructured":"Sarah Smith . 2019. Facebook denies Houston's ads promoting fair housing over race, religion references. Houston Chronicle (May 2019 ). https:\/\/www.chron.com\/news\/houston-texas\/houston\/article\/Facebook-denies-Houston-s-ads-promoting-fair-13847765.php Sarah Smith. 2019. Facebook denies Houston's ads promoting fair housing over race, religion references. Houston Chronicle (May 2019). https:\/\/www.chron.com\/news\/houston-texas\/houston\/article\/Facebook-denies-Houston-s-ads-promoting-fair-13847765.php"},{"key":"e_1_2_1_72_1","volume-title":"2015 IEEE 28th Computer Security Foundations Symposium. IEEE, 554--568","author":"Tschantz Michael Carl","unstructured":"Michael Carl Tschantz , Amit Datta , Anupam Datta , and Jeannette M. Wing . 2015. A methodology for information flow experiments . In 2015 IEEE 28th Computer Security Foundations Symposium. IEEE, 554--568 . Michael Carl Tschantz, Amit Datta, Anupam Datta, and Jeannette M. Wing. 2015. A methodology for information flow experiments. In 2015 IEEE 28th Computer Security Foundations Symposium. IEEE, 554--568."},{"key":"e_1_2_1_73_1","volume-title":"I Approved This Facebook Message - But You Don't Know That. ProPublica (Feb","author":"Valentino-DeVries Jennifer","year":"2018","unstructured":"Jennifer Valentino-DeVries . 2018. I Approved This Facebook Message - But You Don't Know That. ProPublica (Feb . 2018 ). https:\/\/www.propublica.org\/article\/i-approved-this-facebook-message-but-you-dont-know-that Jennifer Valentino-DeVries. 2018. I Approved This Facebook Message - But You Don't Know That. ProPublica (Feb. 2018). https:\/\/www.propublica.org\/article\/i-approved-this-facebook-message-but-you-dont-know-that"},{"key":"e_1_2_1_74_1","volume-title":"Google Will Ask Buyers of U.S. Election Ads to Prove Identities - The New York Times. The New York Times (May","author":"Wakabayashi Daisuke","year":"2018","unstructured":"Daisuke Wakabayashi . 2018. Google Will Ask Buyers of U.S. Election Ads to Prove Identities - The New York Times. The New York Times (May 2018 ). https:\/\/www.nytimes.com\/2018\/05\/04\/technology\/google-election-ad-rules.html Daisuke Wakabayashi. 2018. Google Will Ask Buyers of U.S. Election Ads to Prove Identities - The New York Times. The New York Times (May 2018). https:\/\/www.nytimes.com\/2018\/05\/04\/technology\/google-election-ad-rules.html"},{"key":"e_1_2_1_75_1","volume-title":"has denied entry to some because of others' social media. TechCrunch (Aug","author":"Whittaker Zach","year":"2019","unstructured":"Zach Whittaker . 2019. U. S. has denied entry to some because of others' social media. TechCrunch (Aug . 2019 ). http:\/\/social.techcrunch.com\/2019\/08\/27\/border-deny-entry-united-states-social-media\/ Zach Whittaker. 2019. U.S. has denied entry to some because of others' social media. TechCrunch (Aug. 2019). http:\/\/social.techcrunch.com\/2019\/08\/27\/border-deny-entry-united-states-social-media\/"},{"key":"e_1_2_1_76_1","unstructured":"Ronald E. Wienk. 1979. Measuring Racial Discrimination in American Housing Markets: The Housing Market Practices Survey. (1979).  Ronald E. Wienk. 1979. Measuring Racial Discrimination in American Housing Markets: The Housing Market Practices Survey. (1979)."},{"key":"e_1_2_1_77_1","volume-title":"Matt Davis, Richard T Guy, Steven HD Haddock, Kathryn D Huff, Ian M Mitchell, Mark D Plumbley, et al.","author":"Wilson Greg","year":"2014","unstructured":"Greg Wilson , Dhavide A Aruliah , C Titus Brown , Neil P Chue Hong , Matt Davis, Richard T Guy, Steven HD Haddock, Kathryn D Huff, Ian M Mitchell, Mark D Plumbley, et al. 2014 . Best practices for scientific computing. PLoS biology 12, 1 (2014), e1001745. Greg Wilson, Dhavide A Aruliah, C Titus Brown, Neil P Chue Hong, Matt Davis, Richard T Guy, Steven HD Haddock, Kathryn D Huff, Ian M Mitchell, Mark D Plumbley, et al. 2014. Best practices for scientific computing. PLoS biology 12, 1 (2014), e1001745."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512965","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3512965","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3512965","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:19Z","timestamp":1750188679000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512965"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,30]]},"references-count":75,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2022,3,30]]}},"alternative-id":["10.1145\/3512965"],"URL":"https:\/\/doi.org\/10.1145\/3512965","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,30]]},"assertion":[{"value":"2022-04-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}