{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T06:57:51Z","timestamp":1775199471585,"version":"3.50.1"},"publisher-location":"New York, New York, USA","reference-count":45,"publisher":"ACM Press","license":[{"start":{"date-parts":[[2017,1,1]],"date-time":"2017-01-01T00:00:00Z","timestamp":1483228800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017]]},"DOI":"10.1145\/3041021.3054197","type":"proceedings-article","created":{"date-parts":[[2018,1,11]],"date-time":"2018-01-11T18:39:25Z","timestamp":1515695965000},"page":"626-633","source":"Crossref","is-referenced-by-count":54,"title":["Auditing Search Engines for Differential Satisfaction Across Demographics"],"prefix":"10.1145","author":[{"given":"Rishabh","family":"Mehrotra","sequence":"first","affiliation":[{"name":"University College London, London, United Kingdom"}]},{"given":"Ashton","family":"Anderson","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York, NY, USA"}]},{"given":"Fernando","family":"Diaz","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}]},{"given":"Amit","family":"Sharma","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York, NY, USA"}]},{"given":"Hanna","family":"Wallach","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York, NY, USA"}]},{"given":"Emine","family":"Yilmaz","sequence":"additional","affiliation":[{"name":"University College London, London, United Kingdom"}]}],"member":"320","reference":[{"key":"key-10.1145\/3041021.3054197-1","unstructured":"P. Adler, C. Falk, S. A. Friedler, G. Rybeck, C. Scheidegger, B. Smith, and S. Venkatasubramanian. Auditing black-box models for indirect influence. arXiv:1602.07043."},{"key":"key-10.1145\/3041021.3054197-2","doi-asserted-by":"crossref","unstructured":"H. Ajutsu, G. E. Legge, J. A. Ross, and K. J. Scheubel. Psychophysics of reading--X. Effects of age-related changes in vision. Journal of Gerontology, 46(6):325--331, 1991.","DOI":"10.1093\/geronj\/46.6.P325"},{"key":"key-10.1145\/3041021.3054197-3","unstructured":"J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine bias. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing, 2016."},{"key":"key-10.1145\/3041021.3054197-4","doi-asserted-by":"crossref","unstructured":"S. Barocas and A. D. Selbst. Big data's disparate impact. California Law Review, 104:671--732, 2016.","DOI":"10.2139\/ssrn.2477899"},{"key":"key-10.1145\/3041021.3054197-5","doi-asserted-by":"crossref","unstructured":"P. N. Bennett, K. Svore, and S. T. Dumais. Classification-enhanced ranking. In WWW, 2010.","DOI":"10.1145\/1772690.1772703"},{"key":"key-10.1145\/3041021.3054197-6","doi-asserted-by":"crossref","unstructured":"B. Bin, M. Shokouhi, M. Kosinski, and T. Graepel. Inferring the demographics of search users: Social data meets search queries. In WWW, 2013.","DOI":"10.1145\/2488388.2488401"},{"key":"key-10.1145\/3041021.3054197-7","unstructured":"T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama, and A. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In NIPS, 2016."},{"key":"key-10.1145\/3041021.3054197-8","doi-asserted-by":"crossref","unstructured":"A. Borisov, I. Markov, M. de Rijke, and P. Serdyukov. A context-aware time model for web search. In SIGIR, 2016.","DOI":"10.1145\/2911451.2911504"},{"key":"key-10.1145\/3041021.3054197-9","doi-asserted-by":"crossref","unstructured":"G. Buscher, L. van Elst, and A. Dengel. Segment-level time as implicit feedback: A comparison to eye tracking. In SIGIR, 2009.","DOI":"10.1145\/1571941.1571955"},{"key":"key-10.1145\/3041021.3054197-10","doi-asserted-by":"crossref","unstructured":"B. Carterette, E. Kanoulas, and E. Yilmaz. Incorporating variability in user behavior into systems based evaluation. In CIKM, 2012.","DOI":"10.1145\/2396761.2396782"},{"key":"key-10.1145\/3041021.3054197-11","doi-asserted-by":"crossref","unstructured":"S. L. Connelly, L. Hasher, and R. T. Zacks. Age and reading: The impact of distraction. Psychology and Aging, 6(4):533--541, 1991.","DOI":"10.1037\/0882-7974.6.4.533"},{"key":"key-10.1145\/3041021.3054197-12","doi-asserted-by":"crossref","unstructured":"A. Datta, S. Sen, and Y. Zick. Algorithmic transparency via quantitative input influence. In Proceedings of the Thirty-Seventh IEEE Symposium on Security and Privacy, 2016.","DOI":"10.1109\/SP.2016.42"},{"key":"key-10.1145\/3041021.3054197-13","doi-asserted-by":"crossref","unstructured":"N. Diakopoulos. Algorithmic accountability. Digital Journalism, 3(3):398--415, 2015.","DOI":"10.1080\/21670811.2014.976411"},{"key":"key-10.1145\/3041021.3054197-14","doi-asserted-by":"crossref","unstructured":"D. Downey, S. Dumais, and E. Horvitz. Heads and tails: Studies of web search with common and rare queries. In SIGIR, 2007.","DOI":"10.1145\/1277741.1277939"},{"key":"key-10.1145\/3041021.3054197-15","doi-asserted-by":"crossref","unstructured":"R. Epstein and R. E. Robertson. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. PNAS, 2015.","DOI":"10.1073\/pnas.1419828112"},{"key":"key-10.1145\/3041021.3054197-16","doi-asserted-by":"crossref","unstructured":"H. A. Feild, J. Allan, and R. Jones. Predicting searcher frustration. In SIGIR, 2010.","DOI":"10.1145\/1835449.1835458"},{"key":"key-10.1145\/3041021.3054197-17","doi-asserted-by":"crossref","unstructured":"M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian. Certifying and removing disparate impact. In KDD, 2015.","DOI":"10.1145\/2783258.2783311"},{"key":"key-10.1145\/3041021.3054197-18","doi-asserted-by":"crossref","unstructured":"N. Ford, D. Miller, and N. Moss. Web search strategies and human individual differences: Cognitive and demographic factors, internet attitudes, and approaches. JASIST, 56(7):741--756, 2005.","DOI":"10.1002\/asi.20168"},{"key":"key-10.1145\/3041021.3054197-19","unstructured":"G. M. Fulgoni. The \"professional respondent\" problem in online survey panels today. Presented at the Marketing Research Association Annual Conference, 2005."},{"key":"key-10.1145\/3041021.3054197-20","unstructured":"L. Garcia-Febo, A. Hustad, H. Rösch, P. Sturges, and A. Vallotton. IFLA code of ethics for librarians and other information workers. International Federation of Library Associations and Institutions, 2012."},{"key":"key-10.1145\/3041021.3054197-21","doi-asserted-by":"crossref","unstructured":"A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. Bayesian Data Analysis. CRC Press, 2014.","DOI":"10.1201\/b16018"},{"key":"key-10.1145\/3041021.3054197-22","doi-asserted-by":"crossref","unstructured":"A. Gelman and J. Hill. Data Analysis Using Regression and Multilevel\/Hierarchical Models. Cambridge University Press, 2006.","DOI":"10.1017\/CBO9780511790942"},{"key":"key-10.1145\/3041021.3054197-23","doi-asserted-by":"crossref","unstructured":"A. Hannak, P. Sapiezynski, A. M. Kakhki, B. Krishnamurthy, D. Lazer, A. Mislove, and C. Wilson. Measuring personalization of web search. In WWW, 2013.","DOI":"10.1145\/2488388.2488435"},{"key":"key-10.1145\/3041021.3054197-24","unstructured":"M. Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. In NIPS, 2016."},{"key":"key-10.1145\/3041021.3054197-25","doi-asserted-by":"crossref","unstructured":"A. Hassan, X. Shi, N. Craswell, and B. Ramsey. Beyond clicks: Query reformulation as a predictor of search satisfaction. In CIKM, 2013.","DOI":"10.1145\/2505515.2505682"},{"key":"key-10.1145\/3041021.3054197-26","doi-asserted-by":"crossref","unstructured":"A. Hassan and R. W. White. Personalized models of search satisfaction. In CKIM, 2013.","DOI":"10.1145\/2505515.2505681"},{"key":"key-10.1145\/3041021.3054197-27","doi-asserted-by":"crossref","unstructured":"Y. Hu, Y. Koren, and C. Volinsky. Collaborative filtering for implicit feedback datasets. In ICDM, 2008.","DOI":"10.1109\/ICDM.2008.22"},{"key":"key-10.1145\/3041021.3054197-28","doi-asserted-by":"crossref","unstructured":"J. Jiang, A. Hassan, Z. Shi, and R. W. White. Understanding and predicting graded search satisfaction. In WSDM, 2015.","DOI":"10.1145\/2684822.2685319"},{"key":"key-10.1145\/3041021.3054197-29","doi-asserted-by":"crossref","unstructured":"Y. Kim, A. Hassan, R. W. White, and I. Zitouni. Modeling dwell time to predict click-level satisfaction. In WSDM, 2014.","DOI":"10.1145\/2556195.2556220"},{"key":"key-10.1145\/3041021.3054197-30","doi-asserted-by":"crossref","unstructured":"A. Large, J. Beheshti, and T. Rahman. Gender differences in collaborative web searching behavior: An elementary school study. Information Processing and Management, 2002.","DOI":"10.1016\/S0306-4573(01)00034-6"},{"key":"key-10.1145\/3041021.3054197-31","doi-asserted-by":"crossref","unstructured":"M. Lecuyer, R. Spahn, Y. Spiliopoulos, A. Chaintreau, R. Geambasu, and D. Hsu. Sunlight: Fine-grained targeting detection at scale with statistical confidence. In CCS, 2015.","DOI":"10.1145\/2810103.2813614"},{"key":"key-10.1145\/3041021.3054197-32","doi-asserted-by":"crossref","unstructured":"L. Lorigo, B. Pan, H. Hembrooke, T. Joachims, L. Granka, and G. Gay. The influence of task and gender on search and evaluation behavior using Google. Information Processing and Management, 2006.","DOI":"10.1016\/j.ipm.2005.10.001"},{"key":"key-10.1145\/3041021.3054197-33","doi-asserted-by":"crossref","unstructured":"E. R. Núnez-Valdéz, J. M. C. Lovelle, O. S. Martínez, V. García-Díaz, P. O. de Pablos, and C. Marín. Implicit feedback techniques on recommender systems applied to electronic books. Computers in Human Behavior, 28(4):1186--1193, 2012.","DOI":"10.1016\/j.chb.2012.02.001"},{"key":"key-10.1145\/3041021.3054197-34","doi-asserted-by":"crossref","unstructured":"F. Radlinski, M. Szummer, and N. Craswell. Inferring query intent from reformulations and clicks. In WWW, 2010.","DOI":"10.1145\/1772690.1772859"},{"key":"key-10.1145\/3041021.3054197-35","doi-asserted-by":"crossref","unstructured":"M. Roy and M. T. H. Chi. Gender differences in patterns of searching the web. Journal of Educational Computing Research, 29(3), 2003.","DOI":"10.2190\/7BR8-VXA0-07A7-8AVN"},{"key":"key-10.1145\/3041021.3054197-36","doi-asserted-by":"crossref","unstructured":"D. B. Rubin. Matched Sampling for Causal Effects. Cambridge University Press, 2006.","DOI":"10.1017\/CBO9780511810725"},{"key":"key-10.1145\/3041021.3054197-37","unstructured":"C. Sandvig, K. Hamilton, K. Karahalios, and C. Langbort. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Presented at Data and Discrimination: Converting Critical Concerns into Productive Inquiry: A Preconference to the Sixty-Fourth Annual Meeting of the International Communication Association, 2014."},{"key":"key-10.1145\/3041021.3054197-38","unstructured":"M. Smith, D. Patil, and C. Mu\\ noz. Big data: A report on algorithmic systems, opportunity, and civil rights. Technical report, Executive Office of the President of the United States, 2016."},{"key":"key-10.1145\/3041021.3054197-39","unstructured":"Y. Wang and E. Agichtein. Query ambiguity revisited: Clickthrough measures for distinguishing informational and ambiguous queries. In NAACL, 2010."},{"key":"key-10.1145\/3041021.3054197-40","doi-asserted-by":"crossref","unstructured":"I. Weber and C. Castillo. The demographics of web search. In SIGIR, 2010.","DOI":"10.1145\/1835449.1835537"},{"key":"key-10.1145\/3041021.3054197-41","doi-asserted-by":"crossref","unstructured":"I. Weber and A. Jaimes. Who uses web search for what: and how. In WSDM, 2011.","DOI":"10.1145\/1935826.1935839"},{"key":"key-10.1145\/3041021.3054197-42","doi-asserted-by":"crossref","unstructured":"R. W. White. Interactions with Search Systems. Cambridge University Press, 2016.","DOI":"10.1017\/CBO9781139525305"},{"key":"key-10.1145\/3041021.3054197-43","doi-asserted-by":"crossref","unstructured":"R. W. White and S. Dumais. Characterizing and predicting search engine switching behavior. In CIKM, 2009.","DOI":"10.1145\/1645953.1645967"},{"key":"key-10.1145\/3041021.3054197-44","doi-asserted-by":"crossref","unstructured":"P. Yin, P. Luo, W.-C. Lee, and M. Wang. Silence is also evidence: Interpreting dwell time for recommendation from psychological perspective. In KDD, 2013.","DOI":"10.1145\/2487575.2487663"},{"key":"key-10.1145\/3041021.3054197-45","doi-asserted-by":"crossref","unstructured":"M. Zhou. Gender difference in web search perceptions and behavior: Does it vary by task performance? Computers and Education, 78:174--184, 2014.","DOI":"10.1016\/j.compedu.2014.06.005"}],"event":{"name":"the 26th International Conference","location":"Perth, Australia","acronym":"WWW '17 Companion","number":"26","sponsor":["SIGWEB, ACM Special Interest Group on Hypertext, Hypermedia, and Web","IW3C2, International World Wide Web Conference Committee"],"start":{"date-parts":[[2017,4,3]]},"end":{"date-parts":[[2017,4,7]]}},"container-title":["Proceedings of the 26th International Conference on World Wide Web Companion - WWW '17 Companion"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3041021.3054197","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/dl.acm.org\/ft_gateway.cfm?id=3054197&ftid=1865141&dwn=1","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:03:19Z","timestamp":1750215799000},"score":1,"resource":{"primary":{"URL":"http:\/\/dl.acm.org\/citation.cfm?doid=3041021.3054197"}},"subtitle":[],"proceedings-subject":"World Wide Web Companion","short-title":[],"issued":{"date-parts":[[2017]]},"references-count":45,"URL":"https:\/\/doi.org\/10.1145\/3041021.3054197","relation":{},"subject":[],"published":{"date-parts":[[2017]]}}}