{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T13:20:16Z","timestamp":1783948816018,"version":"3.55.0"},"reference-count":86,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,3,21]],"date-time":"2023-03-21T00:00:00Z","timestamp":1679356800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,3,22]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Social stereotypes negatively impact individuals\u2019 judgments about different groups and may have a critical role in understanding language directed toward marginalized groups. Here, we assess the role of social stereotypes in the automated detection of hate speech in the English language by examining the impact of social stereotypes on annotation behaviors, annotated datasets, and hate speech classifiers. Specifically, we first investigate the impact of novice annotators\u2019 stereotypes on their hate-speech-annotation behavior. Then, we examine the effect of normative stereotypes in language on the aggregated annotators\u2019 judgments in a large annotated corpus. Finally, we demonstrate how normative stereotypes embedded in language resources are associated with systematic prediction errors in a hate-speech classifier. The results demonstrate that hate-speech classifiers reflect social stereotypes against marginalized groups, which can perpetuate social inequalities when propagated at scale. This framework, combining social-psychological and computational-linguistic methods, provides insights into sources of bias in hate-speech moderation, informing ongoing debates regarding machine learning fairness.<\/jats:p>","DOI":"10.1162\/tacl_a_00550","type":"journal-article","created":{"date-parts":[[2023,3,21]],"date-time":"2023-03-21T18:36:30Z","timestamp":1679423790000},"page":"300-319","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":31,"title":["Hate Speech Classifiers Learn Normative Social Stereotypes"],"prefix":"10.1162","volume":"11","author":[{"given":"Aida Mostafazadeh","family":"Davani","sequence":"first","affiliation":[{"name":"University of Southern California, USA. mostafaz@usc.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad","family":"Atari","sequence":"additional","affiliation":[{"name":"University of Southern California, USA. atari@usc.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brendan","family":"Kennedy","sequence":"additional","affiliation":[{"name":"University of Southern California, USA. btkenned@usc.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Morteza","family":"Dehghani","sequence":"additional","affiliation":[{"name":"University of Southern California, USA. mdehghan@usc.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,3,22]]},"reference":[{"issue":"1","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1140\/epjds\/s13688-022-00319-9","article-title":"Tackling racial bias in automated online hate detection: Towards fair and accurate detection of hateful users with geometric deep learning","volume":"11","author":"Zo","year":"2022","journal-title":"EPJ Data Science"},{"key":"2023032118361820000_","article-title":"Whose opinions matter? Perspective-aware models to identify opinions of hate speech victims in abusive language detection","author":"Akhtar","year":"2021","journal-title":"arXiv preprint arXiv:2106.15896"},{"key":"2023032118361820000_","article-title":"Ground-truth, whose truth?\u2013examining the challenges with annotating toxic text datasets","author":"Arhin","year":"2021","journal-title":"arXiv preprint arXiv:2112.03529"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1100","DOI":"10.1145\/3308560.3317083","article-title":"Crowdsourcing subjective tasks: The case study of understanding toxicity in online discussions","volume-title":"Companion Proceedings of The 2019 World Wide Web Conference","author":"Aroyo","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1145\/3308558.3313504","article-title":"Stereotypical bias removal for hate speech detection task using knowledge-based generalizations","volume-title":"The World Wide Web Conference","author":"Badjatiya","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1145\/3442188.3445922","article-title":"On the dangers of stochastic parrots: Can language models be too big?","volume-title":"Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency","author":"Bender","year":"2021"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"5454","DOI":"10.18653\/v1\/2020.acl-main.485","article-title":"Language (technology) is power: A critical survey of \u201cbias\u201d in NLP","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Lin Blodgett","year":"2020"},{"key":"2023032118361820000_","article-title":"Racial disparity in natural language processing: A case study of social media African American English","volume-title":"Fairness, Accountability, and Transparency in Machine Learning (FAT\/ML) Workshop, KDD","author":"Lin Blodgett","year":"2017"},{"key":"2023032118361820000_","first-page":"4349","article-title":"Man is to computer programmer as woman is to homemaker? Debiasing word embeddings","volume-title":"Advances in Neural Information Processing Systems","author":"Bolukbasi","year":"2016"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1145\/3308560.3317593","article-title":"Nuanced metrics for measuring unintended bias with real data for text classification","volume-title":"Companion Proceedings of the 2019 World Wide Web Conference","author":"Borkan","year":"2019"},{"issue":"6334","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1126\/science.aal4230","article-title":"Semantics derived automatically from language corpora contain human-like biases","volume":"356","author":"Caliskan","year":"2017","journal-title":"Science"},{"issue":"6","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1111\/spc3.12181","article-title":"Group-based differences in perceptions of racism: What counts, to whom, and why?","volume":"9","author":"Carter","year":"2015","journal-title":"Social and Personality Psychology Compass"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1177\/0956797620963619","article-title":"Gender stereotypes in natural language: Word embeddings show robust consistency across child and adult language corpora of more than 65 million words","volume":"32","author":"Charlesworth","year":"2021","journal-title":"Psychological Science"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"114","DOI":"10.18653\/v1\/2021.woah-1.12","article-title":"Mitigating biases in toxic language detection through invariant rationalization","volume-title":"Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)","author":"Chuang","year":"2021"},{"issue":"3","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-vector networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Machine Learning"},{"issue":"2","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1111\/1540-4560.00259","article-title":"Hate speech and constitutional protection: Priming values of equality and freedom","volume":"58","author":"Cowan","year":"2002","journal-title":"Journal of Social Issues"},{"key":"2023032118361820000_","article-title":"The trouble with bias","volume-title":"Conference on Neural Information Processing Systems, invited speaker","author":"Crawford","year":"2017"},{"issue":"4","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"631","DOI":"10.1037\/0022-3514.92.4.631","article-title":"The bias map: Behaviors from intergroup affect and stereotypes.","volume":"92","author":"Cuddy","year":"2007","journal-title":"Journal of Personality and Social Psychology"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/S0065-2601(07)00002-0","article-title":"Warmth and competence as universal dimensions of social perception: The stereotype content model and the bias map","volume":"40","author":"Cuddy","year":"2008","journal-title":"Advances in Experimental Social Psychology"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1249","DOI":"10.1162\/tacl_a_00425","article-title":"Quantifying social biases in NLP: A generalization and empirical comparison of extrinsic fairness metrics","volume":"9","author":"Czarnowska","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1162\/tacl_a_00449","article-title":"Dealing with disagreements: Looking beyond the majority vote in subjective annotations","volume":"10","author":"Davani","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"25","DOI":"10.18653\/v1\/W19-3504","article-title":"Racial bias in hate speech and abusive language detection datasets","volume-title":"Proceedings of the Third Workshop on Abusive Language Online","author":"Davidson","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","DOI":"10.1609\/icwsm.v11i1.14955","article-title":"Automated hate speech detection and the problem of offensive language","volume-title":"Eleventh international aaai conference on web and social media","author":"Davidson","year":"2017"},{"key":"2023032118361820000_","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"NAACL-HLT","author":"Devlin","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"2342","DOI":"10.1145\/3531146.3534647","article-title":"Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation","volume-title":"2022 ACM Conference on Fairness, Accountability, and Transparency","author":"D\u00edaz","year":"2022"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1145\/3278721.3278729","article-title":"Measuring and mitigating unintended bias in text classification","volume-title":"Proceedings of the 2018 AAAI\/ACM Conference on AI, Ethics, and Society","author":"Dixon","year":"2018"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1145\/2783258.2783311","article-title":"Certifying and removing disparate impact","volume-title":"proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining","author":"Feldman","year":"2015"},{"issue":"6","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"878","DOI":"10.1037\/0022-3514.82.6.878","article-title":"A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition.","volume":"82","author":"Fiske","year":"2002","journal-title":"Journal of Personality and Social Psychology"},{"issue":"5","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters.","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychological bulletin"},{"key":"2023032118361820000_","article-title":"Pushshift gab corpus","author":"Gaffney","year":"2018"},{"issue":"16","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"E3635","DOI":"10.1073\/pnas.1720347115","article-title":"Word embeddings quantify 100 years of gender and ethnic stereotypes","volume":"115","author":"Garg","year":"2018","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1145\/3306618.3317950","article-title":"Counterfactual fairness in text classification through robustness","volume-title":"Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society","author":"Garg","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.18653\/v1\/D19-1107","article-title":"Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Geva","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"937","DOI":"10.1145\/3038912.3052693","article-title":"Clustered model adaption for personalized sentiment analysis","volume-title":"Proceedings of the 26th International Conference on World Wide Web","author":"Gong","year":"2017"},{"key":"2023032118361820000_","first-page":"2474","article-title":"Humor in word embeddings: Cockamamie gobbledegook for nincompoops","volume-title":"International Conference on Machine Learning","author":"Gultchin","year":"2019"},{"key":"2023032118361820000_","first-page":"3315","article-title":"Equality of opportunity in supervised learning","volume-title":"Advances in Neural Information Processing Systems","author":"Hardt","year":"2016"},{"issue":"10","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1369","DOI":"10.1177\/0146167205275613","article-title":"A meta-analysis on the correlation between the implicit association test and explicit self-report measures","volume":"31","author":"Hofmann","year":"2005","journal-title":"Personality and Social Psychology Bulletin"},{"key":"2023032118361820000_","first-page":"1120","article-title":"Learning whom to trust with mace","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Hovy","year":"2013"},{"issue":"8","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"e12432","DOI":"10.1111\/lnc3.12432","article-title":"Five sources of bias in natural language processing","volume":"15","author":"Hovy","year":"2021","journal-title":"Language and Linguistics Compass"},{"issue":"3","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"556","DOI":"10.1111\/j.1532-7795.2012.00785.x","article-title":"Foreign wars and domestic prejudice: How media exposure to the Israeli-Palestinian conflict predicts ethnic stereotyping by Jewish and Arab American adolescents","volume":"22","author":"Rowell Huesmann","year":"2012","journal-title":"Journal of Research on Adolescence"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"5491","DOI":"10.18653\/v1\/2020.acl-main.487","article-title":"Social biases in NLP models as barriers for persons with disabilities","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Hutchinson","year":"2020"},{"issue":"8","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"e0256762","DOI":"10.1371\/journal.pone.0256762","article-title":"Understanding international perceptions of the severity of harmful content online","volume":"16","author":"Jiang","year":"2021","journal-title":"PloS One"},{"issue":"1","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1007\/s10579-021-09569-x","article-title":"Introducing the gab hate corpus: Defining and applying hate-based rhetoric to social media posts at scale","volume":"56","author":"Kennedy","year":"2022","journal-title":"Language Resources and Evaluation"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"5435","DOI":"10.18653\/v1\/2020.acl-main.483","article-title":"Contextualizing hate speech classifiers with post-hoc explanation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Kennedy","year":"2020"},{"key":"2023032118361820000_","article-title":"A typology and coding manual for the study of hate-based rhetoric","volume":"18","author":"Kennedy","year":"2018","journal-title":"PsyArXiv. July"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1613\/jair.1.12590","article-title":"Confronting abusive language online: A survey from the ethical and human rights perspective","volume":"71","author":"Kiritchenko","year":"2021","journal-title":"Journal of Artificial Intelligence Research"},{"issue":"5","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"675","DOI":"10.1037\/pspa0000046","article-title":"The abc of stereotypes about groups: Agency\/socioeconomic success, conservative\u2013progressive beliefs, and communion.","volume":"110","author":"Koch","year":"2016","journal-title":"Journal of Personality and Social Psychology"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"1168","DOI":"10.1109\/ICDM51629.2021.00140","article-title":"Learning personal human biases and representations for subjective tasks in natural language processing","volume-title":"2021 IEEE International Conference on Data Mining (ICDM)","author":"Koco\u0144","year":"2021"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v27i1.8539","article-title":"Locate the hate: Detecting tweets against blacks","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Kwok","year":"2013"},{"key":"2023032118361820000_","first-page":"648","article-title":"Building an evaluation scale using item response theory","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing","author":"Lalor","year":"2016"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"615","DOI":"10.18653\/v1\/N19-1062","article-title":"Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Manzini","year":"2019"},{"issue":"12","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"2024","DOI":"10.1093\/jamia\/ocaa085","article-title":"Patient safety and quality improvement: Ethical principles for a regulatory approach to bias in healthcare machine learning","volume":"27","author":"McCradden","year":"2020","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"6","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi","year":"2021","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"92","DOI":"10.18653\/v1\/2021.woah-1.10","article-title":"Improving counterfactual generation for fair hate speech detection","volume-title":"Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)","author":"Davani","year":"2021"},{"issue":"8","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"e0237861","DOI":"10.1371\/journal.pone.0237861","article-title":"Hate speech detection and racial bias mitigation in social media based on bert model","volume":"15","author":"Mozafari","year":"2020","journal-title":"PloS ONE"},{"issue":"3","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1177\/1745691611406922","article-title":"Whites see racism as a zero-sum game that they are now losing","volume":"6","author":"Norton","year":"2011","journal-title":"Perspectives on Psychological Science"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1145\/3350546.3352512","article-title":"Unintended bias in misogyny detection","volume-title":"IEEE\/WIC\/ACM International Conference on Web Intelligence","author":"Nozza","year":"2019"},{"issue":"6464","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Obermeyer","year":"2019","journal-title":"Science"},{"key":"2023032118361820000_","article-title":"Reducing gender bias in abusive language detection","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Ji","year":"2018"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2019.260","article-title":"Annotating social media data from vulnerable populations: Evaluating disagreement between domain experts and graduate student annotators","volume-title":"Proceedings of the 52nd Hawaii International Conference on System Sciences","author":"Patton","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1162\/tacl_a_00293","article-title":"Inherent disagreements in human textual inferences","volume":"7","author":"Pavlick","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023032118361820000_","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: Global vectors for word representation","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Pennington","year":"2014"},{"issue":"5","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1002\/ejsp.2561","article-title":"The big two dictionaries: Capturing agency and communion in natural language","volume":"49","author":"Pietraszkiewicz","year":"2019","journal-title":"European Journal of Social Psychology"},{"key":"2023032118361820000_","article-title":"Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics","volume-title":"Eleventh International AAAI Conference on Web and Social Media","author":"Posch","year":"2018"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"5740","DOI":"10.18653\/v1\/D19-1578","article-title":"Perturbation sensitivity analysis to detect unintended model biases","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Prabhakaran","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"133","DOI":"10.18653\/v1\/2021.law-1.14","article-title":"On releasing annotator-level labels and information in datasets","volume-title":"Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop","author":"Prabhakaran","year":"2021"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1145\/2684822.2685316","article-title":"Sarcasm detection on twitter: A behavioral modeling approach","volume-title":"Proceedings of the eighth ACM international conference on web search and data mining","author":"Rajadesingan","year":"2015"},{"key":"2023032118361820000_","volume-title":"Probabilistic Models for Some Intelligence and Attainment Tests","author":"Rasch","year":"1993"},{"key":"2023032118361820000_","article-title":"Measuring the reliability of hate speech annotations: The case of the european refugee crisis","volume-title":"Proceedings of the Workshop on Natural Language Processing for Computer Mediated Communication","author":"Ross","year":"2017"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"175","DOI":"10.18653\/v1\/2022.naacl-main.13","article-title":"Two contrasting data annotation paradigms for subjective NLP tasks","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Rottger","year":"2022"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"1668","DOI":"10.18653\/v1\/P19-1163","article-title":"The risk of racial bias in hate speech detection","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Sap","year":"2019"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2022.naacl-main.431","article-title":"Annotators with attitudes: How annotator beliefs and identities bias toxic language detection","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Sap","year":"2022"},{"key":"2023032118361820000_","first-page":"11","article-title":"Rasch measurement v. item response theory: Knowing when to cross the line.","volume":"26","author":"Stemler","year":"2021","journal-title":"Practical Assessment, Research & Evaluation"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1145\/3306618.3314270","article-title":"What are the biases in my word embedding?","volume-title":"Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society","author":"Swinger","year":"2019"},{"key":"2023032118361820000_","first-page":"138","article-title":"Are you a racist or am I seeing things? Annotator influence on hate speech detection on twitter","volume-title":"Proceedings of the first workshop on NLP and computational social science","author":"Talat","year":"2016"},{"key":"2023032118361820000_","article-title":"Disembodied machine learning: On the illusion of objectivity in NLP","author":"Talat","year":"2021"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1385","DOI":"10.1613\/jair.1.12752","article-title":"Learning from disagreement: A survey","volume":"72","author":"Uma","year":"2021","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2023032118361820000_","first-page":"683","article-title":"Empirical analysis of multi-task learning for reducing identity bias in toxic comment detection","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media","author":"Vaidya","year":"2020"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"1667","DOI":"10.18653\/v1\/2021.acl-long.132","article-title":"Learning from the worst: Dynamically generated datasets to improve online hate detection","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Vidgen","year":"2021"},{"issue":"7866","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1038\/s41586-021-03666-1","article-title":"Measuring algorithmically infused societies","volume":"595","author":"Wagner","year":"2021","journal-title":"Nature"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"191","DOI":"10.18653\/v1\/2020.alw-1.22","article-title":"Investigating annotator bias with a graph-based approach","volume-title":"Proceedings of the fourth workshop on online abuse and harms","author":"Wich","year":"2020"},{"key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2023032118361820000_","doi-asserted-by":"crossref","first-page":"7","DOI":"10.18653\/v1\/2020.socialnlp-1.2","article-title":"Demoting racial bias in hate speech detection","volume-title":"Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media","author":"Xia","year":"2020"},{"key":"2023032118361820000_","first-page":"1218","article-title":"A robustly optimized BERT pre-training approach with post-training","volume-title":"Proceedings of the 20th Chinese National Conference on Computational Linguistics","author":"Zhuang","year":"2021"},{"issue":"5","key":"2023032118361820000_","doi-asserted-by":"publisher","first-page":"696","DOI":"10.1037\/pspa0000080","article-title":"Two axes of subordination: A new model of racial position.","volume":"112","author":"Zou","year":"2017","journal-title":"Journal of Personality and Social Psychology"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00550\/2075730\/tacl_a_00550.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00550\/2075730\/tacl_a_00550.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,9]],"date-time":"2023-12-09T08:52:08Z","timestamp":1702111928000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00550\/115347\/Hate-Speech-Classifiers-Learn-Normative-Social"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,22]]},"references-count":86,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00550","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,22]]}}}