{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T19:58:00Z","timestamp":1784404680578,"version":"3.55.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>Machine learning models are commonly used to detect toxicity in online conversations. These models are trained on datasets annotated by human raters. We explore how raters' self-described identities impact how they annotate toxicity in online comments. We first define the concept of Specialized Rater Pools: rater pools formed based on raters' self-described identities, rather than at random. We formed three such rater pools for this study - specialized rater pools of raters from the U.S. who identify as African American, LGBTQ, and those who identify as neither. Each of these rater pools annotated the same set of comments, which contains many references to these identity groups. We found that rater identity is a statistically significant factor in how raters will annotate toxicity for identity-related annotations. Using preliminary content analysis, we examined the comments with the most disagreement between rater pools and found nuanced differences in the toxicity annotations. Next, we trained models on the annotations from each of the different rater pools, and compared the scores of these models on comments from several test sets. Finally, we discuss how using raters that self-identify with the subjects of comments can create more inclusive machine learning models, and provide more nuanced ratings than those by random raters.<\/jats:p>","DOI":"10.1145\/3555088","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T22:58:54Z","timestamp":1668207534000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":63,"title":["Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation"],"prefix":"10.1145","volume":"6","author":[{"given":"Nitesh","family":"Goyal","sequence":"first","affiliation":[{"name":"Google, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ian D.","family":"Kivlichan","sequence":"additional","affiliation":[{"name":"Jigsaw (Google), New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rachel","family":"Rosen","sequence":"additional","affiliation":[{"name":"Jigsaw (Google), New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucy","family":"Vasserman","sequence":"additional","affiliation":[{"name":"Jigsaw (Google), New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_1_1_1","series-title":"Wiley Series in Probability and Statistics","volume-title":"Categorical Data Analysis","author":"Agresti Alan","year":"2012","unstructured":"Alan Agresti . 2002. Categorical Data Analysis . Wiley Series in Probability and Statistics , Vol. 482 . Wiley , Hoboken, New Jersey. 2012 009792 https:\/\/doi.org\/10.1002\/0471249688 10.1002\/0471249688 Alan Agresti. 2002. Categorical Data Analysis. Wiley Series in Probability and Statistics, Vol. 482. Wiley, Hoboken, New Jersey. 2012009792 https:\/\/doi.org\/10.1002\/0471249688"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v8i1.7473"},{"key":"e_1_2_1_3_1","volume-title":"Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arXiv preprint arXiv:2106.15896","author":"Akhtar Sohail","year":"2021","unstructured":"Sohail Akhtar , Valerio Basile , and Viviana Patti . 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arXiv preprint arXiv:2106.15896 ( 2021 ). Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arXiv preprint arXiv:2106.15896 (2021)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.alw-1.21"},{"key":"e_1_2_1_5_1","volume-title":"Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465","author":"Aluru Sai Saketh","year":"2020","unstructured":"Sai Saketh Aluru , Binny Mathew , Punyajoy Saha , and Animesh Mukherjee . 2020. Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465 ( 2020 ). Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465 (2020)."},{"key":"e_1_2_1_6_1","volume-title":"AIxIA 2020 -- Advances in Artificial Intelligence","author":"Basile Valerio","unstructured":"Valerio Basile . 2021. It's the End of the Gold Standard as We Know It . In AIxIA 2020 -- Advances in Artificial Intelligence , Matteo Baldoni and Stefania Bandini (Eds.). Springer International Publishing , Cham , 441--453. Valerio Basile. 2021. It's the End of the Gold Standard as We Know It. In AIxIA 2020 -- Advances in Artificial Intelligence, Matteo Baldoni and Stefania Bandini (Eds.). Springer International Publishing, Cham, 441--453."},{"key":"e_1_2_1_7_1","volume-title":"Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. arXiv preprint arXiv:2109.04270","author":"Basile Valerio","year":"2021","unstructured":"Valerio Basile , Federico Cabitza , Andrea Campagner , and Michael Fell . 2021. Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. arXiv preprint arXiv:2109.04270 ( 2021 ). Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021. Toward a Perspectivist Turn in Ground Truthing for Predictive Computing. arXiv preprint arXiv:2109.04270 (2021)."},{"key":"e_1_2_1_8_1","first-page":"543","article-title":"Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers","volume":"37","author":"Berg Janine","year":"2015","unstructured":"Janine Berg . 2015 . Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers . Comp. Lab. L. & Pol'y J. , Vol. 37 (2015), 543 . Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comp. Lab. L. & Pol'y J., Vol. 37 (2015), 543.","journal-title":"Comp. Lab. L. & Pol'y J."},{"key":"e_1_2_1_9_1","volume-title":"Max Van Kleek, and Nigel Shadbolt","author":"Binns Reuben","year":"2017","unstructured":"Reuben Binns , Michael Veale , Max Van Kleek, and Nigel Shadbolt . 2017 . Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation. In Social Informatics, Giovanni Luca Ciampaglia, Afra Mashhadi, and Taha Yasseri (Eds.). Springer International Publishing , Cham, 405--415. Reuben Binns, Michael Veale, Max Van Kleek, and Nigel Shadbolt. 2017. Like Trainer, Like Bot? Inheritance of Bias in Algorithmic Content Moderation. In Social Informatics, Giovanni Luca Ciampaglia, Afra Mashhadi, and Taha Yasseri (Eds.). Springer International Publishing, Cham, 405--415."},{"key":"e_1_2_1_10_1","volume-title":"Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. In Companion Proceedings of The 2019 World Wide Web Conference","author":"Borkan Daniel","year":"2019","unstructured":"Daniel Borkan , Lucas Dixon , Jeffrey Sorensen , Nithum Thain , and Lucy Vasserman . 2019 . Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. In Companion Proceedings of The 2019 World Wide Web Conference ( San Francisco, USA) (WWW '19). Association for Computing Machinery, New York, NY, USA, 491--500. https:\/\/doi.org\/10.1145\/3308560.3317593 10.1145\/3308560.3317593 Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. In Companion Proceedings of The 2019 World Wide Web Conference (San Francisco, USA) (WWW '19). Association for Computing Machinery, New York, NY, USA, 491--500. https:\/\/doi.org\/10.1145\/3308560.3317593"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-3504"},{"key":"e_1_2_1_12_1","volume-title":"Perspective: Model Cards","author":"Developers The Perspective","year":"2020","unstructured":"The Perspective Developers . 2020 . Perspective: Model Cards . http:\/\/developers.perspectiveapi.com\/s\/about-the-api-model-cards. Accessed: 2021-07--13. The Perspective Developers. 2020. Perspective: Model Cards. http:\/\/developers.perspectiveapi.com\/s\/about-the-api-model-cards. Accessed: 2021-07--13."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","volume":"1","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171--4186. https:\/\/doi.org\/10. 18653\/v1\/N19--1423 10.18653\/v1 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171--4186. https:\/\/doi.org\/10.18653\/v1\/N19--1423"},{"key":"e_1_2_1_14_1","unstructured":"Lucas Dixon. 2018. Annotation instructions for Toxicity with sub-attributes. https:\/\/github.com\/conversationai\/conversationai.github.io\/blob\/master\/crowdsourcing_annotation_schemes\/toxicity_with_subattributes.md. Accessed: 2020--10--12.  Lucas Dixon. 2018. Annotation instructions for Toxicity with sub-attributes. https:\/\/github.com\/conversationai\/conversationai.github.io\/blob\/master\/crowdsourcing_annotation_schemes\/toxicity_with_subattributes.md. Accessed: 2020--10--12."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s43545-021-00234-2"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2020.3032461"},{"key":"e_1_2_1_18_1","unstructured":"Alessandra Gomes Dennys Antonialli and Thiago Dias Oliva. 2019. Drag queens and Artificial Intelligence: should computers decide what is ?toxic' on the internet? https:\/\/www.internetlab.org.br\/en\/freedom-of-expression\/drag-queens-and-artificial-intelligence-should-computers-decide-what-is-toxic-on-the-internet. Accessed: 2020-09-01.  Alessandra Gomes Dennys Antonialli and Thiago Dias Oliva. 2019. Drag queens and Artificial Intelligence: should computers decide what is ?toxic' on the internet? https:\/\/www.internetlab.org.br\/en\/freedom-of-expression\/drag-queens-and-artificial-intelligence-should-computers-decide-what-is-toxic-on-the-internet. Accessed: 2020-09-01."},{"key":"e_1_2_1_19_1","volume-title":"Equity and Access in Algorithms, Mechanisms, and Optimization (New York","author":"Halevy Matan","unstructured":"Matan Halevy , Camille Harris , Amy Bruckman , Diyi Yang , and Ayanna Howard . 2021. Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Framework . In Equity and Access in Algorithms, Mechanisms, and Optimization (New York , NY , USA) (EAAMO '21). Association for Computing Machinery , New York, NY, USA, Article 7, 11 pages. https:\/\/doi.org\/10.1145\/3465416.3483299 10.1145\/3465416.3483299 Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 2021. Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Framework. In Equity and Access in Algorithms, Mechanisms, and Optimization (New York, NY, USA) (EAAMO '21). Association for Computing Machinery, New York, NY, USA, Article 7, 11 pages. https:\/\/doi.org\/10.1145\/3465416.3483299"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association","author":"Huang Xiaolei","year":"2020","unstructured":"Xiaolei Huang , Linzi Xing , Franck Dernoncourt , and Michael J. Paul . 2020. Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition . In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association , Marseille, France, 1440--1448. https:\/\/aclanthology.org\/ 2020 .lrec-1.180 Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J. Paul. 2020. Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 1440--1448. https:\/\/aclanthology.org\/2020.lrec-1.180"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375627.3375809"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102643"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.alw-1.17"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.socialnlp-1.7"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.7"},{"key":"e_1_2_1_26_1","volume-title":"Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis. arXiv preprint arXiv:2109.08256","author":"Mohammad Saif M","year":"2021","unstructured":"Saif M Mohammad . 2021. Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis. arXiv preprint arXiv:2109.08256 ( 2021 ). Saif M Mohammad. 2021. Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysis. arXiv preprint arXiv:2109.08256 (2021)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.socialnlp-1.8"},{"key":"e_1_2_1_28_1","unstructured":"Frank Newport. 2018. In US estimate of LGBT population rises to 4.5%. Gallup News (2018).  Frank Newport. 2018. In US estimate of LGBT population rises to 4.5%. Gallup News (2018)."},{"key":"e_1_2_1_29_1","volume-title":"Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. arXiv preprint arXiv:1812.05948","author":"Posch Lisa","year":"2018","unstructured":"Lisa Posch , Arnim Bleier , Fabian Fl\u00f6ck , and Markus Strohmaier . 2018. Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. arXiv preprint arXiv:1812.05948 ( 2018 ). Lisa Posch, Arnim Bleier, Fabian Fl\u00f6ck, and Markus Strohmaier. 2018. Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. arXiv preprint arXiv:1812.05948 (2018)."},{"key":"e_1_2_1_30_1","volume-title":"Logistic regression diagnostics. The annals of statistics","author":"Pregibon Daryl","year":"1981","unstructured":"Daryl Pregibon . 1981. Logistic regression diagnostics. The annals of statistics , Vol. 9 , 4 ( 1981 ), 705--724. Daryl Pregibon. 1981. Logistic regression diagnostics. The annals of statistics, Vol. 9, 4 (1981), 705--724."},{"key":"e_1_2_1_31_1","volume-title":"Black vs. African-American: The complex conversation Black Americans are having about identity #ForTheCulture. WUSA","author":"Quander Michael","year":"2019","unstructured":"Michael Quander and Lauryn Froneberger . 2019. Black vs. African-American: The complex conversation Black Americans are having about identity #ForTheCulture. WUSA ( 2019 ). http:\/\/www.wusa9.com\/article\/news\/local\/black-history\/black-vs-african-american-the-complex-conversation-black-americans-are-having-about-identity-fortheculture\/65--80dde243--23be-4cfb-9b0f-bf5898bcf069 Michael Quander and Lauryn Froneberger. 2019. Black vs. African-American: The complex conversation Black Americans are having about identity #ForTheCulture. WUSA (2019). http:\/\/www.wusa9.com\/article\/news\/local\/black-history\/black-vs-african-american-the-complex-conversation-black-americans-are-having-about-identity-fortheculture\/65--80dde243--23be-4cfb-9b0f-bf5898bcf069"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"R\u00f6ttger Paul","year":"1865","unstructured":"Paul R\u00f6ttger , Bertie Vidgen , Dong Nguyen , Zeerak Waseem , Helen Margetts , and Janet Pierrehumbert . 2021. HateCheck: Functional Tests for Hate Speech Detection Models . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics , Online , 41--58. https:\/\/doi.org\/10. 1865 3\/v1\/2021.acl-long.4 10.18653\/v1 Paul R\u00f6ttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional Tests for Hate Speech Detection Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 41--58. https:\/\/doi.org\/10.18653\/v1\/2021.acl-long.4"},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics","author":"Sap Maarten","year":"1865","unstructured":"Maarten Sap , Dallas Card , Saadia Gabriel , Yejin Choi , and Noah A. Smith . 2019. The Risk of Racial Bias in Hate Speech Detection . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics , Florence, Italy, 1668--1678. https:\/\/doi.org\/10. 1865 3\/v1\/P19--1163 10.18653\/v1 Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The Risk of Racial Bias in Hate Speech Detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 1668--1678. https:\/\/doi.org\/10.18653\/v1\/P19--1163"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.193"},{"key":"e_1_2_1_35_1","unstructured":"U.S. Census Bureau. 2019. QuickFacts: United States. https:\/\/www.census.gov\/quickfacts\/fact\/table\/US\/RHI125219. Accessed: 2021-07--13.  U.S. Census Bureau. 2019. QuickFacts: United States. https:\/\/www.census.gov\/quickfacts\/fact\/table\/US\/RHI125219. Accessed: 2021-07--13."},{"key":"e_1_2_1_36_1","volume-title":"Assessing Multilingual Fairness in Pre-trained Multimodal Representations. arXiv preprint arXiv:2106.06683","author":"Wang Jialu","year":"2021","unstructured":"Jialu Wang , Yang Liu , and Xin Eric Wang . 2021. Assessing Multilingual Fairness in Pre-trained Multimodal Representations. arXiv preprint arXiv:2106.06683 ( 2021 ). Jialu Wang, Yang Liu, and Xin Eric Wang. 2021. Assessing Multilingual Fairness in Pre-trained Multimodal Representations. arXiv preprint arXiv:2106.06683 (2021)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-5618"},{"key":"e_1_2_1_38_1","volume-title":"Basic content analysis 2 ed.)","author":"Weber Robert Philip","unstructured":"Robert Philip Weber . 1990. Basic content analysis 2 ed.) . Sage, Thousand Oaks , California . https:\/\/doi.org\/10.4135\/9781412983488 10.4135\/9781412983488 Robert Philip Weber. 1990. Basic content analysis 2 ed.). Sage, Thousand Oaks, California. https:\/\/doi.org\/10.4135\/9781412983488"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.alw-1.22"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.alw-1.7"},{"key":"e_1_2_1_41_1","volume-title":"Burak Enes Cakici, and Georg Groh","author":"Wich Maximilian","year":"2021","unstructured":"Maximilian Wich , Adrian Gorniak , Tobias Eder , Daniel Bartmann , Burak Enes Cakici, and Georg Groh . 2021 . Introducing an Abusive Language Classification Framework for Telegram to Investigate the German Hater Community . arXiv preprint arXiv:2109.07346 (2021). Maximilian Wich, Adrian Gorniak, Tobias Eder, Daniel Bartmann, Burak Enes Cakici, and Georg Groh. 2021. Introducing an Abusive Language Classification Framework for Telegram to Investigate the German Hater Community. arXiv preprint arXiv:2109.07346 (2021)."},{"key":"#cr-split#-e_1_2_1_42_1.1","doi-asserted-by":"crossref","unstructured":"Ellery Wulczyn Nithum Thain and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web (Perth Australia) (WWW '17). International World Wide Web Conferences Steering Committee Republic and Canton of Geneva CHE 1391--1399. https:\/\/doi.org\/10.1145\/3038912.3052591 10.1145\/3038912.3052591","DOI":"10.1145\/3038912.3052591"},{"key":"#cr-split#-e_1_2_1_42_1.2","doi-asserted-by":"crossref","unstructured":"Ellery Wulczyn Nithum Thain and Lucas Dixon. 2017. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web (Perth Australia) (WWW '17). International World Wide Web Conferences Steering Committee Republic and Canton of Geneva CHE 1391--1399. https:\/\/doi.org\/10.1145\/3038912.3052591","DOI":"10.1145\/3038912.3052591"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Zeinert Philine","year":"1865","unstructured":"Philine Zeinert , Nanna Inie , and Leon Derczynski . 2021. Annotating Online Misogyny . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics , Online , 3181--3197. https:\/\/doi.org\/10. 1865 3\/v1\/2021.acl-long.247 10.18653\/v1 Philine Zeinert, Nanna Inie, and Leon Derczynski. 2021. Annotating Online Misogyny. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 3181--3197. https:\/\/doi.org\/10.18653\/v1\/2021.acl-long.247"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555088","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555088","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:46:51Z","timestamp":1750178811000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555088"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":44,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555088"],"URL":"https:\/\/doi.org\/10.1145\/3555088","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}