{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T17:43:31Z","timestamp":1781113411147,"version":"3.54.1"},"reference-count":48,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2021,7,1]],"date-time":"2021-07-01T00:00:00Z","timestamp":1625097600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["Project 262513311 - SFB 1187 Media of Cooperation"],"award-info":[{"award-number":["Project 262513311 - SFB 1187 Media of Cooperation"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Big Data &amp; Society"],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:p>Over recent years, the stakes and complexity of online content moderation have been steadily raised, swelling from concerns about personal conflict in smaller communities to worries about effects on public life and democracy. Because of the massive growth in online expressions, automated tools based on machine learning are increasingly used to moderate speech. While \u2018design-based governance\u2019 through complex algorithmic techniques has come under intense scrutiny, critical research covering algorithmic content moderation is still rare. To add to our understanding of concrete instances of machine moderation, this article examines Perspective API, a system for the automated detection of \u2018toxicity\u2019 developed and run by the Google unit Jigsaw that can be used by websites to help moderate their forums and comment sections. The article proceeds in four steps. First, we present our methodological strategy and the empirical materials we were able to draw on, including interviews, documentation, and GitHub repositories. We then summarize our findings along five axes to identify the various threads Perspective API brings together to deliver a working product. The third section discusses two conflicting organizational logics within the project, paying attention to both critique and what can be learned from the specific case at hand. We conclude by arguing that the opposition between \u2018human\u2019 and \u2018machine\u2019 in speech moderation obscures the many ways these two come together in concrete systems, and suggest that the way forward requires proactive engagement with the design of technologies as well as the institutions they are embedded in.<\/jats:p>","DOI":"10.1177\/20539517211046181","type":"journal-article","created":{"date-parts":[[2021,10,16]],"date-time":"2021-10-16T08:12:56Z","timestamp":1634371976000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":61,"title":["The fabrics of machine moderation: Studying the technical, normative,  and organizational structure  of Perspective API"],"prefix":"10.1177","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2404-9277","authenticated-orcid":false,"given":"Bernhard","family":"Rieder","sequence":"first","affiliation":[{"name":"Mediastudies, University of Amsterdam, Amsterdam, Noord-Holland, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yarden","family":"Skop","sequence":"additional","affiliation":[{"name":"University of Siegen, Siegen, Nordrhein-Westfalen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2021,10,16]]},"reference":[{"key":"bibr1-20539517211046181","first-page":"131","volume-title":"Social Science, Technical Systems, and Cooperative Work: Beyond the Great Divide.","author":"Agre PE","year":"1997"},{"key":"bibr2-20539517211046181","unstructured":"Ahmed N, Wahed M (2020) The de-democratization of AI: Deep learning and the compute divide in artificial intelligence research. https:\/\/arxiv.org\/abs\/2010.15581."},{"key":"bibr3-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/1461444816676645"},{"key":"bibr4-20539517211046181","unstructured":"Anti-Defamation League (2019) Online hate and harassment: The American experience. Report. https:\/\/www.adl.org\/onlineharassment."},{"key":"bibr5-20539517211046181","volume-title":"Politics and the Other Scene","author":"Balibar \u00c9","year":"2002"},{"key":"bibr6-20539517211046181","doi-asserted-by":"crossref","unstructured":"Belle V, Papantonis I (2020) Principles and practice of explainable machine learning. https:\/\/arxiv.org\/abs\/2009.11698.","DOI":"10.3389\/fdata.2021.688969"},{"key":"bibr7-20539517211046181","unstructured":"Blue V (2017) Google\u2019s comment-ranking system will be a hit with the alt-right. Engadget, 1 September. https:\/\/www.engadget.com\/2017-09-01-google-perspective-comment-ranking-system.html."},{"key":"bibr8-20539517211046181","doi-asserted-by":"crossref","unstructured":"Borkan D, Dixon L, Sorensen J, et al. (2019) Nuanced metrics for measuring unintended bias with real data for text classification. https:\/\/arxiv.org\/abs\/1903.04561.","DOI":"10.1145\/3308560.3317593"},{"key":"bibr9-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780190493028.001.0001"},{"key":"bibr10-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/2053951715622512"},{"key":"bibr11-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1016\/0277-9390(92)90001-R"},{"key":"bibr12-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-954X.1990.tb03351.x"},{"issue":"1","key":"bibr13-20539517211046181","volume":"89","author":"Citron DK","year":"2014","journal-title":"Washington Law Review"},{"key":"bibr14-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/1461444814543163"},{"key":"bibr15-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"key":"bibr16-20539517211046181","unstructured":"Etim B (2017) The times sharply increases articles open for comments, using Google\u2019s technology.\n                      The New York Times\n                      , 13 June. https:\/\/www.nytimes.com\/2017\/06\/13\/insider\/have-a-comment-leave-a-comment.html"},{"key":"bibr17-20539517211046181","first-page":"101","volume-title":"Habermas and the Public Sphere","author":"Fraser N","year":"1992"},{"key":"bibr18-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287589"},{"key":"bibr19-20539517211046181","volume-title":"Custodians of the Internet","author":"Gillespie T","year":"2018"},{"key":"bibr20-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720943234"},{"key":"bibr21-20539517211046181","doi-asserted-by":"publisher","DOI":"10.14763\/2020.4.1512"},{"key":"bibr22-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/2053951719897945"},{"key":"bibr23-20539517211046181","unstructured":"Greenberg A (2016) Inside Google\u2019s internet justice league and its AI-powered war on trolls.\n                      Wired\n                      , 19 September. https:\/\/www.wired.com\/2016\/09\/inside-googles-internet-justice-league-ai-powered-war-trolls\/"},{"issue":"1","key":"bibr24-20539517211046181","volume":"17","author":"Grimmelmann J","year":"2017","journal-title":"Yale Journal of Law & Technology"},{"key":"bibr25-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1111\/rego.12367"},{"key":"bibr26-20539517211046181","doi-asserted-by":"crossref","unstructured":"Gr\u00f6ndahl T, Pajola L, Juuti M, et al. (2018) All you need is \u2018love\u2019: Evading hate-speech detection. https:\/\/arxiv.org\/abs\/1808.09115.","DOI":"10.1145\/3270101.3270103"},{"key":"bibr27-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1080\/01972243.2017.1391913"},{"key":"bibr28-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/2056305115603080"},{"key":"bibr29-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1080\/1461670X.2016.1209977"},{"key":"bibr30-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1007\/BF01059830"},{"key":"bibr31-20539517211046181","first-page":"155","volume-title":"The Offensive Internet: Speech, Privacy, and Reputation","author":"Leiter B","year":"2010"},{"key":"bibr32-20539517211046181","unstructured":"Long K (2017) Keeping the times civil, 16 million comments and counting.\n                      The New York Times\n                      , 1 July. https:\/\/www.nytimes.com\/2017\/07\/01\/insider\/times-comments.html."},{"key":"bibr33-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/1461444815608807"},{"key":"bibr34-20539517211046181","unstructured":"Mehrabi N, Morstatter F, Saxena N, et al. (2019) A survey on bias and fairness in machine learning. https:\/\/arxiv.org\/abs\/1908.09635."},{"key":"bibr35-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"bibr36-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/1461444818773059"},{"key":"bibr37-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/1464884917725163"},{"key":"bibr38-20539517211046181","unstructured":"Noever D (2018) Machine learning suites for online toxicity detection. https:\/\/arxiv.org\/abs\/1810.01869."},{"key":"bibr39-20539517211046181","doi-asserted-by":"crossref","unstructured":"Pennington J, Socher R, Manning C (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on Empirical methods in natural language processing, Doha, Qatar, 2014, pp. 1532\u20131543. Association for Computational Linguistics.","DOI":"10.3115\/v1\/D14-1162"},{"key":"bibr40-20539517211046181","doi-asserted-by":"crossref","unstructured":"Price I, Gifford-Moore J, Flemming J, et al. (2020) Six attributes of unhealthy conversation. https:\/\/arxiv.org\/abs\/2010.07410.","DOI":"10.18653\/v1\/2020.alw-1.15"},{"key":"bibr41-20539517211046181","doi-asserted-by":"publisher","DOI":"10.14763\/2020.4.1535"},{"key":"bibr42-20539517211046181","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctvhrcz0v"},{"key":"bibr43-20539517211046181","doi-asserted-by":"crossref","unstructured":"Schmidt A, Wiegand M (2017) A survey on hate speech detection using natural language processing. In: Proceedings of the fifth international workshop on Natural language processing for social media, Valencia, Spain, 2017, pp. 1\u201310. Association for Computational Linguistics.","DOI":"10.18653\/v1\/W17-1101"},{"key":"bibr44-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/2053951717738104"},{"key":"bibr45-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0084-6"},{"key":"bibr46-20539517211046181","doi-asserted-by":"publisher","DOI":"10.1177\/1463499618780870"},{"key":"bibr47-20539517211046181","doi-asserted-by":"crossref","unstructured":"Waseem Z, Davidson T, Warmsley D, et al. (2017) Understanding abuse: A typology of abusive language detection subtasks. https:\/\/arxiv.org\/abs\/1705.09899.","DOI":"10.18653\/v1\/W17-3012"},{"key":"bibr48-20539517211046181","doi-asserted-by":"crossref","unstructured":"Wulczyn E, Thain N, Dixon L (2017) Ex machina: Personal attacks seen at scale. https:\/\/arxiv.org\/abs\/1610.08914.","DOI":"10.1145\/3038912.3052591"}],"container-title":["Big Data &amp; Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20539517211046181","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/20539517211046181","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/20539517211046181","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T12:58:21Z","timestamp":1777381101000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/20539517211046181"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7]]},"references-count":48,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["10.1177\/20539517211046181"],"URL":"https:\/\/doi.org\/10.1177\/20539517211046181","relation":{},"ISSN":["2053-9517","2053-9517"],"issn-type":[{"value":"2053-9517","type":"print"},{"value":"2053-9517","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7]]},"article-number":"20539517211046181"}}