{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T00:28:59Z","timestamp":1777854539919,"version":"3.51.4"},"reference-count":58,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2022,5,2]],"date-time":"2022-05-02T00:00:00Z","timestamp":1651449600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["100001CL_182630\/1"],"award-info":[{"award-number":["100001CL_182630\/1"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Friends of the Institute of Communication and Media Science at the University of Bern","award":["Algorithmic curation of (political) information an"],"award-info":[{"award-number":["Algorithmic curation of (political) information an"]}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["MA 2244\/9-1"],"award-info":[{"award-number":["MA 2244\/9-1"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Information Science"],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:p>Algorithm audits have increased in recent years due to a growing need to independently assess the performance of automatically curated services that process, filter and rank the large and dynamic amount of information available on the Internet. Among several methodologies to perform such audits, virtual agents stand out because they offer the ability to perform systematic experiments, simulating human behaviour without the associated costs of recruiting participants. Motivated by the importance of research transparency and replicability of results, this article focuses on the challenges of such an approach. It provides methodological details, recommendations, lessons learned and limitations based on our experience of setting up experiments for eight search engines (including main, news, image and video sections) with hundreds of virtual agents placed in different regions. We demonstrate the successful performance of our research infrastructure across multiple data collections, with diverse experimental designs, and point to different changes and strategies that improve the quality of the method. We conclude that virtual agents are a promising venue for monitoring the performance of algorithms across long periods of time, and we hope that this article can serve as a basis for further research in this area.<\/jats:p>","DOI":"10.1177\/01655515221093029","type":"journal-article","created":{"date-parts":[[2022,5,2]],"date-time":"2022-05-02T04:24:22Z","timestamp":1651465462000},"page":"404-419","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":18,"title":["Scaling up search engine audits: Practical insights for algorithm auditing"],"prefix":"10.1177","volume":"50","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9870-5505","authenticated-orcid":false,"given":"Roberto","family":"Ulloa","sequence":"first","affiliation":[{"name":"Computational Social Science, GESIS \u2013 Leibniz-Institute for the Social Sciences, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mykola","family":"Makhortykh","sequence":"additional","affiliation":[{"name":"Institute of Communication and Media Studies, University of Bern, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aleksandra","family":"Urman","sequence":"additional","affiliation":[{"name":"Social Computing Group, University of Zurich, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2022,5,2]]},"reference":[{"key":"bibr1-01655515221093029","first-page":"167","volume":"167","author":"Gillespie T","year":"2014","journal-title":"Media Technol Essays Commun Mater Soc"},{"key":"bibr2-01655515221093029","doi-asserted-by":"publisher","DOI":"10.18574\/nyu\/9781479833641.001.0001"},{"key":"bibr3-01655515221093029","volume-title":"Weapons of math destruction: how big data increases inequality and threatens democracy","author":"O\u2019Neil C","year":"2016"},{"key":"bibr4-01655515221093029","first-page":"12","volume":"10","author":"Mittelstadt B","year":"2016","journal-title":"Int J Commun"},{"key":"bibr5-01655515221093029","unstructured":"Bandy J. Problematic machine behavior: a systematic literature review of algorithm audits. ArXiv210204256 Cs, http:\/\/arxiv.org\/abs\/2102.04256 (2021, accessed 23 April 2021)."},{"key":"bibr6-01655515221093029","first-page":"22","volume-title":"Digital dominance: the power of Google, Amazon, Facebook, and Apple","author":"Diakopoulos N","year":"2018"},{"key":"bibr7-01655515221093029","first-page":"693","volume-title":"The World Wide Web conference","author":"Hu D"},{"key":"bibr8-01655515221093029","first-page":"417","volume-title":"Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing","author":"Kulshrestha J"},{"key":"bibr9-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1145\/3359231"},{"key":"bibr10-01655515221093029","first-page":"1","volume-title":"Proceedings of the 2019 CHI conference on human factors in computing systems","author":"Trielli D"},{"key":"bibr11-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1177\/08944393211006863"},{"key":"bibr12-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1016\/j.tele.2018.07.004"},{"key":"bibr13-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46963-8_8"},{"key":"bibr14-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2017.1338145"},{"key":"bibr15-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2018.1539626"},{"key":"bibr16-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1145\/3274417"},{"key":"bibr17-01655515221093029","first-page":"955","volume-title":"Proceedings of the 2018 World Wide Web conference","author":"Robertson RE"},{"key":"bibr18-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488435"},{"key":"bibr19-01655515221093029","first-page":"121","volume-title":"Proceedings of the 2015 Internet measurement conference","author":"Kliman-Silver C"},{"key":"bibr20-01655515221093029","first-page":"6620","volume-title":"Proceedings of the 2017 CHI conference on human factors in computing systems","author":"Otterbacher J"},{"key":"bibr21-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1002\/asi.24335"},{"key":"bibr22-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-78818-6_5"},{"key":"bibr23-01655515221093029","first-page":"e280212","volume":"28","author":"Cano-Or\u00f3n L.","year":"2019","journal-title":"Prof Inf"},{"key":"bibr24-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1080\/10410236.2015.1113484"},{"key":"bibr25-01655515221093029","first-page":"1","volume":"1","author":"Makhortykh M","year":"2020","journal-title":"Harv Kennedy Sch Misinformation Rev"},{"key":"bibr26-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-020-00954-0"},{"key":"bibr27-01655515221093029","unstructured":"Lurie E, Mustafaraj E. Opening up the black box: auditing Google\u2019s top stories algorithm. Proc Int Fla Artif Intell Res Soc Conf 2019; 32: 376\u2013382, https:\/\/par.nsf.gov\/biblio\/10101277-opening-up-black-box-auditing-googles-top-stories-algorithm (accessed 7 May 2021)."},{"key":"bibr28-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2018.07.043"},{"key":"bibr29-01655515221093029","first-page":"232","volume-title":"Companion proceedings of the web conference","author":"Urman A"},{"key":"bibr30-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1145\/3392854"},{"key":"bibr31-01655515221093029","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v26i10.11562"},{"key":"bibr32-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1177\/0163443718764565"},{"key":"bibr33-01655515221093029","doi-asserted-by":"crossref","unstructured":"McMahon C, Johnson I, Hecht B. The substantial interdependence of Wikipedia and Google: a case study on the relationship between peer production communities and information technologies. Proc Int AAAI Conf Web Soc Media 2017; 11, https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/14883 (accessed 7 May 2021).","DOI":"10.1609\/icwsm.v11i1.14883"},{"key":"bibr34-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v13i01.3248"},{"key":"bibr35-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1080\/1461670X.2019.1702892"},{"key":"bibr36-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1515\/popets-2015-0007"},{"key":"bibr37-01655515221093029","first-page":"309","volume-title":"Proceedings of the 7th ACM\/IEEE-CS joint conference on digital libraries","author":"McCown F"},{"key":"bibr38-01655515221093029","first-page":"1105","volume-title":"The 41st International ACM SIGIR conference on research & development in information retrieval","author":"Jimmy"},{"key":"bibr39-01655515221093029","unstructured":"Bodo B, Helberger N, Irion K et al. Tackling the algorithmic control crisis \u2013 the technical, legal, and ethical challenges of research into algorithmic agents. Yale J Law Technol 2018; 19: 133\u2013180, https:\/\/digitalcommons.law.yale.edu\/yjolt\/vol19\/iss1\/3"},{"key":"bibr40-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1177\/0894439319828012"},{"key":"bibr41-01655515221093029","unstructured":"Mattu S, Yin L, Waller A et al. How we built a Facebook inspector. The Markup, 5 January 2021, https:\/\/themarkup.org\/citizen-browser\/2021\/01\/05\/how-we-built-a-facebook-inspector (accessed 6 May 2021)."},{"key":"bibr42-01655515221093029","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v16i2.3344"},{"key":"bibr43-01655515221093029","first-page":"79","volume-title":"Proceedings of the 11th ACM workshop on hot topics in networks","author":"Mikians J"},{"key":"bibr44-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1177\/1461444818801010"},{"key":"bibr45-01655515221093029","unstructured":"Urman A, Makhortykh M, Ulloa R. Visual representation of migrants in web search results, https:\/\/boris.unibe.ch\/156714\/"},{"key":"bibr46-01655515221093029","unstructured":"Meyers PJ. YouTube dominates Google video in 2020. MOZ, 14 October 2020, https:\/\/moz.com\/blog\/youtube-dominates-google-video-results-in-2020 (accessed 6 May 2021)."},{"key":"bibr47-01655515221093029","unstructured":"Schechner S, Grind K, West J. Searching for video? Google pushes YouTube over rivals. The Wall Street Journal, 14 July 2020, https:\/\/www.wsj.com\/articles\/google-steers-users-to-youtube-over-rivals-11594745232 (accessed 6 May 2021)."},{"key":"bibr48-01655515221093029","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v14i1.7276"},{"key":"bibr49-01655515221093029","first-page":"305","volume-title":"Proceedings of the 2014 conference on internet measurement conference","author":"Hannak A"},{"key":"bibr50-01655515221093029","first-page":"76","volume-title":"Proceedings of the eighth ACM conference on data and application security and privacy","author":"Hupperich T"},{"key":"bibr51-01655515221093029","doi-asserted-by":"publisher","DOI":"10.3384\/cu.2000.1525.1792163"},{"key":"bibr52-01655515221093029","doi-asserted-by":"publisher","DOI":"10.3384\/cu.2000.1525.1792184"},{"key":"bibr53-01655515221093029","first-page":"540","volume-title":"2018 IEEE\/ACM international conference on advances in social networks analysis and mining (ASONAM)","author":"Chakraborty A"},{"key":"bibr54-01655515221093029","unstructured":"WebBot Ulloa R. (3.2) [Computer software]. GESIS \u2013 Leibniz Institute for the Social Sciences, 2021, https:\/\/github.com\/gesiscss\/WebBot."},{"key":"bibr55-01655515221093029","unstructured":"Aigenseer V, Urman A, Christner C et al. Webtrack \u2013 desktop extension for tracking users\u2019 browsing behaviour using screen-scraping, https:\/\/boris.unibe.ch\/139219\/"},{"key":"bibr56-01655515221093029","unstructured":"Chrome Developers. chrome.BrowsingData, https:\/\/developer.chrome.com\/docs\/extensions\/reference\/browsingData\/ (2021, accessed 4 June 2021)."},{"key":"bibr57-01655515221093029","unstructured":"MDN Web Docs. browsingData.DataTypeSet. MDN Web Docs, 27 October 2021, https:\/\/developer.mozilla.org\/en-US\/docs\/Mozilla\/Add-ons\/WebExtensions\/API\/browsingData\/DataTypeSet (accessed 4 June 2021)."},{"key":"bibr58-01655515221093029","unstructured":"Search blocking and captcha \u2013 captcha. Feedback, https:\/\/yandex.com\/support\/captcha\/ (accessed 20 April 2021)."}],"container-title":["Journal of Information Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01655515221093029","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/01655515221093029","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01655515221093029","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T23:09:39Z","timestamp":1777504179000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/01655515221093029"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,2]]},"references-count":58,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["10.1177\/01655515221093029"],"URL":"https:\/\/doi.org\/10.1177\/01655515221093029","relation":{},"ISSN":["0165-5515","1741-6485"],"issn-type":[{"value":"0165-5515","type":"print"},{"value":"1741-6485","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,2]]}}}