{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T23:25:37Z","timestamp":1777937137573,"version":"3.51.4"},"reference-count":53,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T00:00:00Z","timestamp":1643241600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"Cornell Center for the Social Sciences","award":["\u201cAlgorithms, Big Data & Inequality\u201d Collaborative"],"award-info":[{"award-number":["\u201cAlgorithms, Big Data & Inequality\u201d Collaborative"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["New Media &amp; Society"],"published-print":{"date-parts":[[2024,2]]},"abstract":"<jats:p>\n                    While search engines are discursively framed as automated, self-governing machines, they remain dependent on human laborers. This behind-the-screen workforce includes not only programmers and engineers but also categories of workers that have drawn far less attention. The latter include search quality raters, contingent workers tasked with supplying a \u201chuman check\u201d on search algorithms. To explore the human\u2013machine entanglements in search engine evaluation, we draw upon 21 in-depth interviews with raters located across the globe; we supplement our interview data with an analysis of worker training documents. Our findings reveal that despite Google\u2019s stated efforts to make algorithmic systems more sensitive to human subtlety, raters experience their work in ways that are\u2014paradoxically\u2014algorithmic. Indeed, workers framed their experiences through regimes of\n                    <jats:italic toggle=\"yes\">standardization<\/jats:italic>\n                    ,\n                    <jats:italic toggle=\"yes\">discipline<\/jats:italic>\n                    , and\n                    <jats:italic toggle=\"yes\">invisibility<\/jats:italic>\n                    . We conclude by discussing the broader implications of our analysis, including worker dehumanization and the exacerbation of power hierarchies in the global, on-demand economy.\n                  <\/jats:p>","DOI":"10.1177\/14614448211063860","type":"journal-article","created":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T05:50:30Z","timestamp":1643262630000},"page":"1018-1033","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":20,"title":["The labor of search engine evaluation: Making algorithms more human or humans more algorithmic?"],"prefix":"10.1177","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7398-2269","authenticated-orcid":false,"given":"Colten","family":"Meisner","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6314-8027","authenticated-orcid":false,"given":"Brooke Erin","family":"Duffy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6643-195X","authenticated-orcid":false,"given":"Malte","family":"Ziewitz","sequence":"additional","affiliation":[{"name":"Cornell University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2022,1,27]]},"reference":[{"key":"e_1_3_3_2_1","doi-asserted-by":"publisher","DOI":"10.4324\/9780203321454"},{"key":"e_1_3_3_3_1","first-page":"1","volume-title":"Big Data & Society","author":"Bili\u0107 P","year":"2016","unstructured":"Bili\u0107 P (2016) Search algorithms, hidden labour and information control. Big Data & Society 3(1): 1\u20139."},{"key":"e_1_3_3_4_1","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780190493028.001.0001"},{"issue":"21","key":"e_1_3_3_5_1","first-page":"3934","article-title":"Digital labor studies go global: toward a digital decolonial turn","volume":"11","author":"Casilli AA","year":"2017","unstructured":"Casilli AA (2017) Digital labor studies go global: toward a digital decolonial turn. International Journal of Communication 11(21): 3934\u20133954.","journal-title":"International Journal of Communication"},{"key":"e_1_3_3_6_1","doi-asserted-by":"publisher","DOI":"10.1177\/1461444818815684"},{"key":"e_1_3_3_7_1","doi-asserted-by":"publisher","DOI":"10.1525\/9780520961630"},{"key":"e_1_3_3_8_1","volume-title":"Objectivity","author":"Daston L","year":"2007","unstructured":"Daston L, Galison P (2007) Objectivity. New York: Zone Books."},{"key":"e_1_3_3_9_1","doi-asserted-by":"publisher","DOI":"10.18574\/nyu\/9781479890118.003.0011"},{"key":"e_1_3_3_10_1","doi-asserted-by":"crossref","unstructured":"Ekbia HR Nardi BA (2014) Heteromation and its (dis)contents: the invisible division of labor between humans and machines. First Monday 19(6).","DOI":"10.5210\/fm.v19i6.5331"},{"key":"e_1_3_3_11_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262050937.001.0001"},{"key":"e_1_3_3_12_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9042.003.0013"},{"key":"e_1_3_3_13_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2016.1199721"},{"key":"e_1_3_3_14_1","volume-title":"Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media","author":"Gillespie T","year":"2018","unstructured":"Gillespie T (2018) Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media. London: Yale University Press."},{"key":"e_1_3_3_15_1","unstructured":"Google (2021a) How Google\u2019s Algorithm Is Focused on Its Users. Google. Available at: https:\/\/www.google.com\/search\/howsearchworks\/mission\/users\/"},{"key":"e_1_3_3_16_1","unstructured":"Google (2021b) How Our Quality Raters Make Search Better. Google. Available at: https:\/\/support.google.com\/websearch\/answer\/9281931?hl=en"},{"key":"e_1_3_3_17_1","unstructured":"Google (2021c) Webmaster Guidelines. Google Search Central. Available at: https:\/\/developers.google.com\/search\/docs\/advanced\/guidelines\/webmaster-guidelines"},{"key":"e_1_3_3_18_1","volume-title":"Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass","author":"Gray ML","year":"2019","unstructured":"Gray ML, Suri S (2019) Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Boston, MA: Houghton Mifflin Harcourt."},{"key":"e_1_3_3_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0160-9327(00)01338-7"},{"key":"e_1_3_3_20_1","volume-title":"When Computers Were Human","author":"Grier DA","year":"2005","unstructured":"Grier DA (2005) When Computers Were Human. Princeton, NJ: Princeton University Press."},{"key":"e_1_3_3_21_1","volume-title":"Search Engine Society","author":"Halavais A","year":"2018","unstructured":"Halavais A (2018) Search Engine Society. Malden, MA: Polity."},{"key":"e_1_3_3_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/0950017016674894"},{"key":"e_1_3_3_23_1","volume-title":"Programmed Inequality: How Britain Discarded Women Technologists and Lost Its Edge in Computing","author":"Hicks M","year":"2017","unstructured":"Hicks M (2017) Programmed Inequality: How Britain Discarded Women Technologists and Lost Its Edge in Computing. Cambridge, MA: MIT Press."},{"key":"e_1_3_3_24_1","doi-asserted-by":"publisher","DOI":"10.1080\/01972240050133634"},{"key":"e_1_3_3_25_1","doi-asserted-by":"publisher","DOI":"10.1177\/1461444813511926"},{"key":"e_1_3_3_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_3_3_27_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781315720111"},{"key":"e_1_3_3_28_1","unstructured":"Lavietes M McCoy M (2020) Waiting for Work: Pandemic Leaves U.S. Gig Workers Clamoring for Jobs. Reuters. Available at: https:\/\/www.reuters.com\/article\/us-biggerpicture-health-coronavirus-gigw\/waiting-for-work-pandemic-leaves-u-s-gig-workers-clamoring-for-jobs-idUSKBN2741DM"},{"key":"e_1_3_3_29_1","doi-asserted-by":"publisher","DOI":"10.1353\/tech.1999.0128"},{"key":"e_1_3_3_30_1","unstructured":"Lohr S (2013) Big Data Trying to Build Better Workers. Available at: https:\/\/www.nytimes.com\/2013\/04\/21\/technology\/big-data-trying-to-build-better-workers.html"},{"key":"e_1_3_3_31_1","volume-title":"The Communist Manifesto","author":"Marx K","year":"2011","unstructured":"Marx K, Engels F (2011) The Communist Manifesto. London: Penguin Books."},{"key":"e_1_3_3_32_1","volume-title":"Qualitative Research Design: An Interactive Approach","author":"Maxwell JA","year":"2013","unstructured":"Maxwell JA (2013) Qualitative Research Design: An Interactive Approach. 3rd ed. London: SAGE.","edition":"3"},{"key":"e_1_3_3_33_1","volume-title":"Audience Economics: Media Institutions and the Audience Marketplace","author":"Napoli P","year":"2003","unstructured":"Napoli P (2003) Audience Economics: Media Institutions and the Audience Marketplace. New York: Columbia University Press."},{"key":"e_1_3_3_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/0263276414530477"},{"key":"e_1_3_3_35_1","doi-asserted-by":"publisher","DOI":"10.18574\/nyu\/9781479833641.001.0001"},{"key":"e_1_3_3_36_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720939605"},{"key":"e_1_3_3_37_1","doi-asserted-by":"publisher","DOI":"10.1177\/2056305119879995"},{"key":"e_1_3_3_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_3_3_39_1","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctvhrcz0v"},{"key":"e_1_3_3_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300512"},{"key":"e_1_3_3_41_1","doi-asserted-by":"publisher","DOI":"10.4324\/9780203145791"},{"key":"e_1_3_3_42_1","unstructured":"Search Engine Journal (2021) History of Google Algorithm Updates. Available at: https:\/\/www.searchenginejournal.com\/google-algorithm-history\/"},{"key":"e_1_3_3_43_1","doi-asserted-by":"publisher","DOI":"10.14506\/ca33.3.04"},{"key":"e_1_3_3_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/1461444818773727"},{"key":"e_1_3_3_45_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720919776."},{"key":"e_1_3_3_46_1","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226817514.001.0001"},{"key":"e_1_3_3_47_1","doi-asserted-by":"publisher","DOI":"10.1525\/9780520948693"},{"key":"e_1_3_3_48_1","first-page":"866","volume-title":"Market","author":"van Couvering E","year":"2007","unstructured":"van Couvering E (2007) Is relevance relevant? Market, science, and war: discourses of search engine quality. Journal of Computer-Mediated Communication 12(3): 866\u2013887."},{"key":"e_1_3_3_49_1","author":"van Couvering E","year":"2010","unstructured":"van Couvering E (2010) Search engine bias: the structuration of traffic on the World-Wide Web. PhD Thesis, London School of Economics and Political Sciences, London.","journal-title":"Search engine bias: the structuration of traffic on the World-Wide Web"},{"key":"e_1_3_3_50_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2017.1294194"},{"key":"e_1_3_3_51_1","unstructured":"Wakabayashi D (2019) Google\u2019s shadow work force: temps who outnumber full-time employees. The New York Times 28 May. Available at: https:\/\/www.nytimes.com\/2019\/05\/28\/technology\/google-temp-workers.html"},{"key":"e_1_3_3_52_1","doi-asserted-by":"publisher","DOI":"10.1177\/0162243915608948"},{"key":"e_1_3_3_53_1","doi-asserted-by":"publisher","DOI":"10.1177\/0306312719865607"},{"key":"e_1_3_3_54_1","volume-title":"The Age of Surveillance Capitalism: The Fight For a Human Future at the New Frontier of Power","author":"Zuboff S","year":"2019","unstructured":"Zuboff S (2019) The Age of Surveillance Capitalism: The Fight For a Human Future at the New Frontier of Power. New York: PublicAffairs."}],"container-title":["New Media &amp; Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14614448211063860","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/14614448211063860","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14614448211063860","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T03:53:32Z","timestamp":1777694012000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/14614448211063860"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,27]]},"references-count":53,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2]]}},"alternative-id":["10.1177\/14614448211063860"],"URL":"https:\/\/doi.org\/10.1177\/14614448211063860","relation":{},"ISSN":["1461-4448","1461-7315"],"issn-type":[{"value":"1461-4448","type":"print"},{"value":"1461-7315","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,27]]}}}