{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T16:37:31Z","timestamp":1781109451693,"version":"3.54.1"},"reference-count":79,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T00:00:00Z","timestamp":1637280000000},"content-version":"vor","delay-in-days":322,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,11,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Measuring bias is key for better understanding and addressing unfairness in NLP\/ML models. This is often done via fairness metrics, which quantify the differences in a model\u2019s behaviour across a range of demographic groups. In this work, we shed more light on the differences and similarities between the fairness metrics used in NLP. First, we unify a broad range of existing metrics under three generalized fairness metrics, revealing the connections between them. Next, we carry out an extensive empirical comparison of existing metrics and demonstrate that the observed differences in bias measurement can be systematically explained via differences in parameter choices for our generalized metrics.<\/jats:p>","DOI":"10.1162\/tacl_a_00425","type":"journal-article","created":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T17:02:40Z","timestamp":1637341360000},"page":"1249-1267","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":46,"title":["Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics"],"prefix":"10.1162","volume":"9","author":[{"given":"Paula","family":"Czarnowska","sequence":"first","affiliation":[{"name":"University of Cambridge, UK. pjc211@cam.ac.uk"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yogarshi","family":"Vyas","sequence":"additional","affiliation":[{"name":"Amazon AI, USA. yogarshi@amazon.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kashif","family":"Shah","sequence":"additional","affiliation":[{"name":"Amazon AI, USA. shahkas@amazon.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,11,5]]},"reference":[{"key":"2021111917023313100_bib1","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/3308558.3313504","article-title":"Stereotypical bias removal for hate speech detection task using knowledge- based generalizations","volume-title":"The World Wide Web Conference","author":"Badjatiya","year":"2019"},{"key":"2021111917023313100_bib2","first-page":"15479","article-title":"Differential privacy has disparate impact on model accuracy","volume-title":"Advances in Neural Information Processing Systems","author":"Bagdasaryan","year":"2019"},{"key":"2021111917023313100_bib3","first-page":"2138","article-title":"An annotated dataset of literary entities","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Bamman","year":"2019"},{"key":"2021111917023313100_bib4","article-title":"Data decisions and theoretical implications when adversarially learning fair representations","author":"Beutel","year":"2017","journal-title":"Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT\/ML 2017)"},{"key":"2021111917023313100_bib5","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1145\/3306618.3314234","article-title":"Putting fairness principles into practice: Challenges, metrics, and improvements","volume-title":"Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society","author":"Beutel","year":"2019"},{"key":"2021111917023313100_bib6","doi-asserted-by":"publisher","first-page":"62","DOI":"10.18653\/v1\/W19-3809","article-title":"Good secretaries, bad truck drivers? Occupational gender stereotypes in sentiment analysis","volume-title":"Proceedings of the First Workshop on Gender Bias in Natural Language Processing","author":"Bhaskaran","year":"2019"},{"key":"2021111917023313100_bib7","doi-asserted-by":"publisher","first-page":"5454","DOI":"10.18653\/v1\/2020.acl-main.485","article-title":"Language (technology) is power: A critical survey of \u201cbias\u201d in NLP","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Lin Blodgett","year":"2020"},{"key":"2021111917023313100_bib8","article-title":"Racial Disparity in Natural Language Processing: A case study of social media African- American English","author":"Lin Blodgett","year":"2017","journal-title":"Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT\/ML 2017)"},{"key":"2021111917023313100_bib9","doi-asserted-by":"publisher","first-page":"1415","DOI":"10.18653\/v1\/P18-1131","article-title":"Twitter universal dependency parsing for African-American and mainstream American English","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Lin Blodgett","year":"2018"},{"key":"2021111917023313100_bib10","first-page":"4356","article-title":"Man is to computer programmer as woman is to homemaker? Debiasing word embeddings","volume-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems","author":"Bolukbasi","year":"2016"},{"key":"2021111917023313100_bib11","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1145\/3308560.3317593","article-title":"Nuanced metrics for measuring unintended bias with real data for text classification","volume-title":"Companion Proceedings of The 2019 World Wide Web Conference","author":"Borkan","year":"2019"},{"key":"2021111917023313100_bib12","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1126\/science.aal4230","article-title":"Semantics derived automatically from language corpora contain human- like biases","volume":"356","author":"Caliskan","year":"2017","journal-title":"Science"},{"key":"2021111917023313100_bib13","doi-asserted-by":"crossref","first-page":"4568","DOI":"10.18653\/v1\/2020.acl-main.418","article-title":"Toward gender-inclusive coreference resolution","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Cao","year":"2020"},{"key":"2021111917023313100_bib14","doi-asserted-by":"publisher","first-page":"25","DOI":"10.18653\/v1\/W19-3804","article-title":"Measuring gender bias in word embeddings across domains and discovering new gender bias word categories","volume-title":"Proceedings of the First Workshop on Gender Bias in Natural Language Processing","author":"Chaloner","year":"2019"},{"issue":"5","key":"2021111917023313100_bib15","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1145\/3376898","article-title":"A snapshot of the frontiers of fairness in machine learning","volume":"63","author":"Chouldechova","year":"2020","journal-title":"Communications of the ACM"},{"key":"2021111917023313100_bib16","article-title":"ELECTRA: Pre-training text encoders as discriminators rather than generators","author":"Clark","year":"2020","journal-title":"The International Conference on Learning Representations (ICLR)"},{"key":"2021111917023313100_bib17","doi-asserted-by":"publisher","first-page":"25","DOI":"10.18653\/v1\/W19-3504","article-title":"Racial bias in hate speech and abusive language detection datasets","volume-title":"Proceedings of the Third Workshop on Abusive Language Online","author":"Davidson","year":"2019"},{"key":"2021111917023313100_bib18","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1145\/3287560.3287572","article-title":"Bias in bios: A case study of semantic representation bias in a high-stakes setting","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency","author":"De-Arteaga","year":"2019"},{"key":"2021111917023313100_bib19","first-page":"879","article-title":"Attenuating bias in word vectors","volume-title":"Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics","author":"Dev","year":"2019"},{"key":"2021111917023313100_bib20","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1145\/3278721.3278729","article-title":"Measuring and mitigating unintended bias in text classification","volume-title":"Proceedings of the 2018 AAAI\/ ACM Conference on AI, Ethics, and Society","author":"Dixon","year":"2018"},{"key":"2021111917023313100_bib21","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1145\/2090236.2090255","article-title":"Fairness through awareness","volume-title":"Proceedings of the 3rd Innovations in Theoretical Computer Science Conference","author":"Dwork","year":"2012"},{"issue":"2","key":"2021111917023313100_bib22","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/0002-9343(92)90100-P","article-title":"Absolutely relative: How research results are summarized can affect treatment decisions","volume":"92","author":"Forrow","year":"1992","journal-title":"The American Journal of Medicine"},{"key":"2021111917023313100_bib23","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1145\/3306618.3317950","article-title":"Counterfactual fairness in text classification through robustness","volume-title":"Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society","author":"Garg","year":"2019"},{"key":"2021111917023313100_bib24","doi-asserted-by":"publisher","first-page":"3493","DOI":"10.18653\/v1\/P19-1339","article-title":"Women\u2019s syntactic resilience and men\u2019s grammatical luck: Gender-bias in part-of-speech tagging and dependency parsing","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Garimella","year":"2019"},{"key":"2021111917023313100_bib25","doi-asserted-by":"publisher","first-page":"2943","DOI":"10.18653\/v1\/2020.acl-main.265","article-title":"Towards understanding gender bias in relation extraction","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Gaut","year":"2020"},{"issue":"01","key":"2021111917023313100_bib26","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1109\/MIC.2020.3032461","article-title":"Cyberbullying detection with fairness constraints","volume":"25","author":"Gencoglu","year":"2021","journal-title":"IEEE Internet Computing"},{"key":"2021111917023313100_bib27","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.150","article-title":"Intrinsic bias metrics do not correlate with application bias","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Goldfarb-Tarrant","year":"2021"},{"key":"2021111917023313100_bib28","first-page":"609","article-title":"Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Gonen","year":"2019"},{"key":"2021111917023313100_bib29","doi-asserted-by":"publisher","first-page":"1991","DOI":"10.18653\/v1\/2020.findings-emnlp.180","article-title":"Automatically identifying gender issues in machine translation using perturbations","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Gonen","year":"2020"},{"key":"2021111917023313100_bib30","doi-asserted-by":"publisher","first-page":"5267","DOI":"10.18653\/v1\/D19-1530","article-title":"It\u2019s all in the name: Mitigating gender bias with name- based counterfactual data substitution","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Maudslay","year":"2019"},{"key":"2021111917023313100_bib31","first-page":"3323","article-title":"Equality of opportunity in supervised learning","volume-title":"Proceedings of the 30th International Conference on Neural Information Processing Systems","author":"Hardt","year":"2016"},{"key":"2021111917023313100_bib32","first-page":"483","article-title":"Tagging performance correlates with author age","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","author":"Hovy","year":"2015"},{"key":"2021111917023313100_bib33","doi-asserted-by":"publisher","first-page":"65","DOI":"10.18653\/v1\/2020.findings-emnlp.7","article-title":"Reducing sentiment bias in language models via counterfactual evaluation","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Huang","year":"2020"},{"key":"2021111917023313100_bib34","first-page":"1440","article-title":"Multilingual twitter corpus and baselines for evaluating demographic bias in hate speech recognition","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Huang","year":"2020"},{"key":"2021111917023313100_bib35","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/3287560.3287600","article-title":"50 Years of test (un)fairness: Lessons for machine learning","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency","author":"Hutchinson","year":"2019"},{"key":"2021111917023313100_bib36","doi-asserted-by":"publisher","first-page":"5491","DOI":"10.18653\/v1\/2020.acl-main.487","article-title":"Social biases in NLP models as barriers for persons with disabilities","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Hutchinson","year":"2020"},{"key":"2021111917023313100_bib37","doi-asserted-by":"publisher","first-page":"706","DOI":"10.1145\/3351095.3375671","article-title":"The meaning and measurement of bias: Lessons from natural language processing","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency","author":"Jacobs","year":"2020"},{"key":"2021111917023313100_bib38","first-page":"862","article-title":"Wasserstein fair classification","volume-title":"Proceedings of The 35th Uncertainty in Artificial Intelligence Conference","author":"Jiang","year":"2020"},{"key":"2021111917023313100_bib39","doi-asserted-by":"publisher","first-page":"43","DOI":"10.18653\/v1\/S18-2005","article-title":"Examining gender and race bias in two hundred sentiment analysis systems","volume-title":"Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics","author":"Kiritchenko","year":"2018"},{"key":"2021111917023313100_bib40","doi-asserted-by":"publisher","first-page":"166","DOI":"10.18653\/v1\/W19-3823","article-title":"Measuring bias in contextualized word representations","volume-title":"Proceedings of the First Workshop on Gender Bias in Natural Language Processing","author":"Kurita","year":"2019"},{"key":"2021111917023313100_bib41","first-page":"4066","article-title":"Counterfactual fairness","volume-title":"Advances in Neural Information Processing Systems","author":"Kusner","year":"2017"},{"key":"2021111917023313100_bib42","first-page":"282","article-title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","volume-title":"Proceedings of the Eighteenth International Conference on Machine Learning","author":"Lafferty","year":"2001"},{"key":"2021111917023313100_bib43","doi-asserted-by":"publisher","first-page":"5502","DOI":"10.18653\/v1\/2020.acl-main.488","article-title":"Towards debiasing sentence representations","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Liang","year":"2020"},{"key":"2021111917023313100_bib44","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"CoRR"},{"issue":"1","key":"2021111917023313100_bib45","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1214\/aoms\/1177730491","article-title":"On a test of whether one of two random variables is stochastically larger than the other","volume":"18","author":"Mann","year":"1947","journal-title":"Ann. Math. Statist."},{"key":"2021111917023313100_bib46","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1145\/3372923.3404804","article-title":"Man is to person as woman is to location: Measuring gender bias in named entity recognition","volume-title":"Proceedings of the 31st ACM Conference on Hypertext and Social Media","author":"Mehrabi","year":"2020"},{"key":"2021111917023313100_bib47","article-title":"A survey on bias and fairness in machine learning","volume":"abs\/1908.09635. Version 2","author":"Mehrabi","year":"2019","journal-title":"CoRR"},{"key":"2021111917023313100_bib48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/S18-1001","article-title":"SemEval-2018 Task 1: Affect in Tweets","volume-title":"Proceedings of The 12th International Workshop on Semantic Evaluation","author":"Mohammad","year":"2018"},{"key":"2021111917023313100_bib49","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.416","article-title":"StereoSet: Measuring stereotypical bias in pretrained language models","author":"Nadeem","year":"2020","journal-title":"CoRR"},{"key":"2021111917023313100_bib50","doi-asserted-by":"publisher","first-page":"1953","DOI":"10.18653\/v1\/2020.emnlp-main.154","article-title":"CrowS-Pairs: A challenge dataset for measuring social biases in masked language models","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Nangia","year":"2020"},{"key":"2021111917023313100_bib51","doi-asserted-by":"publisher","first-page":"ii13","DOI":"10.1093\/ndt\/gfw465","article-title":"Relative risk versus absolute risk: One cannot be interpreted without the other: Clinical epidemiology in nephrology","volume":"32","author":"Noordzij","year":"2017","journal-title":"Nephrology Dialysis Transplantation"},{"key":"2021111917023313100_bib52","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1145\/3091478.3098871","article-title":"The limits of abstract evaluation metrics: The case of hate speech detection","volume-title":"Proceedings of the 2017 ACM on Web Science Conference","author":"Olteanu","year":"2017"},{"key":"2021111917023313100_bib53","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1145\/3351095.3372843","article-title":"Bias in word embeddings","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency","author":"Papakyriakopoulos","year":"2020"},{"key":"2021111917023313100_bib54","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1007\/978-3-030-59491-6_8","article-title":"Joint multiclass debiasing of word embeddings","volume-title":"Foundations of Intelligent Systems","author":"Popovi\u0107","year":"2020"},{"key":"2021111917023313100_bib55","doi-asserted-by":"publisher","first-page":"5740","DOI":"10.18653\/v1\/D19-1578","article-title":"Perturbation sensitivity analysis to detect unintended model biases","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Prabhakaran","year":"2019"},{"key":"2021111917023313100_bib56","doi-asserted-by":"publisher","first-page":"69","DOI":"10.18653\/v1\/W19-3810","article-title":"Debiasing embeddings for reduced gender bias in text classification","volume-title":"Proceedings of the First Workshop on Gender Bias in Natural Language Processing","author":"Prost","year":"2019"},{"key":"2021111917023313100_bib57","doi-asserted-by":"publisher","first-page":"147","DOI":"10.3115\/1596374.1596399","article-title":"Design challenges and misconceptions in named entity recognition","volume-title":"Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)","author":"Ratinov","year":"2009"},{"key":"2021111917023313100_bib58","doi-asserted-by":"publisher","first-page":"4902","DOI":"10.18653\/v1\/2020.acl-main.442","article-title":"Beyond accuracy: Behavioral testing of NLP models with Checklist","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Ribeiro","year":"2020"},{"key":"2021111917023313100_bib59","doi-asserted-by":"publisher","first-page":"881","DOI":"10.1609\/aaai.v34i01.5434","article-title":"FuzzE: Fuzzy fairness evaluation of offensive language classifiers on African-American English","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Rios","year":"2020"},{"key":"2021111917023313100_bib60","doi-asserted-by":"publisher","first-page":"3371","DOI":"10.18653\/v1\/2020.coling-main.299","article-title":"An empirical study of the downstream reliability of pre-trained word embeddings","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Rios","year":"2020"},{"key":"2021111917023313100_bib61","doi-asserted-by":"publisher","first-page":"8","DOI":"10.18653\/v1\/N18-2002","article-title":"Gender bias in coreference resolution","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Rudinger","year":"2018"},{"key":"2021111917023313100_bib62","doi-asserted-by":"crossref","first-page":"1668","DOI":"10.18653\/v1\/P19-1163","article-title":"The risk of racial bias in hate speech detection","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Sap","year":"2019"},{"key":"2021111917023313100_bib63","doi-asserted-by":"publisher","first-page":"7724","DOI":"10.18653\/v1\/2020.acl-main.690","article-title":"Reducing gender bias in neural machine translation as a domain adaptation problem","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Saunders","year":"2020"},{"key":"2021111917023313100_bib64","doi-asserted-by":"crossref","DOI":"10.1162\/tacl_a_00401","article-title":"Gender bias in machine translation","author":"Savoldi","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021111917023313100_bib65","doi-asserted-by":"publisher","first-page":"55","DOI":"10.18653\/v1\/W19-3808","article-title":"The role of protected class word lists in bias identification of contextualized word representations","volume-title":"Proceedings of the First Workshop on Gender Bias in Natural Language Processing","author":"Sedoc","year":"2019"},{"key":"2021111917023313100_bib66","doi-asserted-by":"crossref","first-page":"5248","DOI":"10.18653\/v1\/2020.acl-main.468","article-title":"Predictive biases in natural language processing models: A conceptual framework and overview","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Shah","year":"2020"},{"key":"2021111917023313100_bib67","doi-asserted-by":"publisher","first-page":"3239","DOI":"10.18653\/v1\/2020.findings-emnlp.291","article-title":"Towards controllable biases in language generation","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Sheng","year":"2020"},{"key":"2021111917023313100_bib68","article-title":"Societal biases in language generation: Progress and challenges","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Sheng","year":"2021"},{"key":"2021111917023313100_bib69","doi-asserted-by":"publisher","first-page":"3126","DOI":"10.18653\/v1\/2020.findings-emnlp.280","article-title":"Neutralizing gender bias in word embeddings with latent disentanglement and counterfactual generation","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Shin","year":"2020"},{"key":"2021111917023313100_bib70","first-page":"1631","article-title":"Recursive deep models for semantic compositionality over a sentiment treebank","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Socher","year":"2013"},{"key":"2021111917023313100_bib71","first-page":"629","article-title":"Mitigating gender bias in machine translation with target gender annotations","volume-title":"Proceedings of the 5th Conference on Machine Translation (WMT)","author":"Stafanovi\u010ds","year":"2020"},{"key":"2021111917023313100_bib72","doi-asserted-by":"publisher","first-page":"1679","DOI":"10.18653\/v1\/P19-1164","article-title":"Evaluating Gender Bias in Machine Translation","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Stanovsky","year":"2019"},{"key":"2021111917023313100_bib73","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1016\/j.shpsc.2015.06.003","article-title":"Measuring effectiveness","volume":"54","author":"Stegenga","year":"2015","journal-title":"Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences"},{"key":"2021111917023313100_bib74","doi-asserted-by":"publisher","first-page":"1630","DOI":"10.18653\/v1\/P19-1159","article-title":"Mitigating gender bias in natural language processing: Literature review","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Sun","year":"2019"},{"key":"2021111917023313100_bib75","doi-asserted-by":"crossref","first-page":"2920","DOI":"10.18653\/v1\/2020.acl-main.263","article-title":"It\u2019s morphin\u2019 time! Combating linguistic discrimination with inflectional perturbations","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Tan","year":"2020"},{"key":"2021111917023313100_bib76","doi-asserted-by":"publisher","first-page":"142","DOI":"10.3115\/1119176.1119195","article-title":"Introduction to the CoNLL-2003 Shared Task: Language-independent named entity recognition","volume-title":"Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003","author":"Sang","year":"2003"},{"key":"2021111917023313100_bib77","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1162\/tacl_a_00240","article-title":"Mind the GAP: A balanced corpus of gendered ambiguous pronouns","volume":"6","author":"Webster","year":"2018","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021111917023313100_bib78","doi-asserted-by":"publisher","first-page":"15","DOI":"10.18653\/v1\/N18-2003","article-title":"Gender bias in coreference resolution: Evaluation and debiasing methods","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Zhao","year":"2018"},{"key":"2021111917023313100_bib79","article-title":"Mitigation of unintended biases against non-native english texts in sentiment analysis","volume-title":"Proceedings for the 27th AIAI Irish Conference on Artificial Intelligence and Cognitive Science","author":"Zhiltsova","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00425\/1972677\/tacl_a_00425.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00425\/1972677\/tacl_a_00425.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,19]],"date-time":"2021-11-19T17:03:06Z","timestamp":1637341386000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00425\/108201\/Quantifying-Social-Biases-in-NLP-A-Generalization"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":79,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00425","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}