{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T15:12:58Z","timestamp":1788448378129,"version":"build-2803163510"},"publisher-location":"New York, NY, USA","reference-count":19,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T00:00:00Z","timestamp":1557705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,13]]},"DOI":"10.1145\/3308560.3317593","type":"proceedings-article","created":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T08:17:59Z","timestamp":1557735479000},"page":"491-500","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":156,"title":["Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification"],"prefix":"10.1145","author":[{"given":"Daniel","family":"Borkan","sequence":"first","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucas","family":"Dixon","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey","family":"Sorensen","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nithum","family":"Thain","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucy","family":"Vasserman","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,5,13]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Alex Beutel Jilin Chen Zhe Zhao and Ed\u00a0H. Chi. 2017. Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations. CoRR abs\/1707.00075(2017). http:\/\/arxiv.org\/abs\/1707.00075  Alex Beutel Jilin Chen Zhe Zhao and Ed\u00a0H. Chi. 2017. Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations. CoRR abs\/1707.00075(2017). http:\/\/arxiv.org\/abs\/1707.00075"},{"key":"e_1_3_2_1_2_1","volume-title":"Limitations of Pinned AUC for Measuring Unintended Bias. CoRR","author":"Borkan Daniel","year":"2019","unstructured":"Daniel Borkan , Lucas Dixon , John Li , Jeffrey Sorensen , Nithum Thain , and Lucy Vasserman . 2019. Limitations of Pinned AUC for Measuring Unintended Bias. CoRR ( 2019 ). Daniel Borkan, Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Limitations of Pinned AUC for Measuring Unintended Bias. CoRR (2019)."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/0093854808326545"},{"key":"e_1_3_2_1_4_1","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle\u00a0A. Friedler and Christo Wilson (Eds.), Vol.\u00a081","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification . In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle\u00a0A. Friedler and Christo Wilson (Eds.), Vol.\u00a081 . PMLR, New York, NY, USA, 77\u201391. http:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle\u00a0A. Friedler and Christo Wilson (Eds.), Vol.\u00a081. PMLR, New York, NY, USA, 77\u201391. http:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1002"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_2_1_8_1","unstructured":"Sorelle\u00a0A. Friedler Carlos Scheidegger and Suresh Venkatasubramanian. 2016. On the (im)possibility of fairness. CoRR abs\/1609.07236(2016). http:\/\/arxiv.org\/abs\/1609.07236  Sorelle\u00a0A. Friedler Carlos Scheidegger and Suresh Venkatasubramanian. 2016. On the (im)possibility of fairness. CoRR abs\/1609.07236(2016). http:\/\/arxiv.org\/abs\/1609.07236"},{"key":"e_1_3_2_1_9_1","unstructured":"Sahaj Garg Vincent Perot Nicole Limtiaco Ankur Taly Ed\u00a0H. Chi and Alex Beutel. 2018. Counterfactual Fairness in Text Classification through Robustness. CoRR abs\/1809.10610(2018). arxiv:1809.10610http:\/\/arxiv.org\/abs\/1809.10610  Sahaj Garg Vincent Perot Nicole Limtiaco Ankur Taly Ed\u00a0H. Chi and Alex Beutel. 2018. Counterfactual Fairness in Text Classification through Robustness. CoRR abs\/1809.10610(2018). arxiv:1809.10610http:\/\/arxiv.org\/abs\/1809.10610"},{"key":"e_1_3_2_1_10_1","unstructured":"Moritz Hardt Eric Price and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. CoRR abs\/1610.02413(2016). http:\/\/arxiv.org\/abs\/1610.02413  Moritz Hardt Eric Price and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. CoRR abs\/1610.02413(2016). http:\/\/arxiv.org\/abs\/1610.02413"},{"key":"e_1_3_2_1_11_1","unstructured":"Jigsaw. 2017. Perspective API. https:\/\/www.perspectiveapi.com\/  Jigsaw. 2017. Perspective API. https:\/\/www.perspectiveapi.com\/"},{"key":"e_1_3_2_1_12_1","unstructured":"Jon\u00a0M. Kleinberg Sendhil Mullainathan and Manish Raghavan. 2016. Inherent Trade-Offs in the Fair Determination of Risk Scores. CoRR abs\/1609.05807(2016). http:\/\/arxiv.org\/abs\/1609.05807  Jon\u00a0M. Kleinberg Sendhil Mullainathan and Manish Raghavan. 2016. Inherent Trade-Offs in the Fair Determination of Risk Scores. CoRR abs\/1609.05807(2016). http:\/\/arxiv.org\/abs\/1609.05807"},{"key":"e_1_3_2_1_13_1","unstructured":"Lucy Vasserman John Li CJ Adams Lucas Dixon. 2018. Unintended bias and names of frequently targeted groups. https:\/\/medium.com\/the-false-positive\/unintended-bias-and-names-of-frequently-targeted-groups-8e0b81f80a23  Lucy Vasserman John Li CJ Adams Lucas Dixon. 2018. Unintended bias and names of frequently targeted groups. https:\/\/medium.com\/the-false-positive\/unintended-bias-and-names-of-frequently-targeted-groups-8e0b81f80a23"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"crossref","unstructured":"S.\u00a0J. Mason and N.\u00a0E. Graham. {n. d.}. Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society 128 584 ({n. d.}) 2145\u20132166. arXiv:https:\/\/rmets.onlinelibrary.wiley.com\/doi\/pdf\/10.1256\/003590002320603584  S.\u00a0J. Mason and N.\u00a0E. Graham. {n. d.}. Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society 128 584 ({n. d.}) 2145\u20132166. arXiv:https:\/\/rmets.onlinelibrary.wiley.com\/doi\/pdf\/10.1256\/003590002320603584","DOI":"10.1256\/003590002320603584"},{"key":"e_1_3_2_1_15_1","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle\u00a0A. Friedler and Christo Wilson (Eds.), Vol.\u00a081","author":"Menon Aditya\u00a0Krishna","year":"2018","unstructured":"Aditya\u00a0Krishna Menon and Robert\u00a0 C Williamson . 2018 . The cost of fairness in binary classification . In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle\u00a0A. Friedler and Christo Wilson (Eds.), Vol.\u00a081 . PMLR, New York, NY, USA, 107\u2013118. http:\/\/proceedings.mlr.press\/v81\/menon18a.html Aditya\u00a0Krishna Menon and Robert\u00a0C Williamson. 2018. The cost of fairness in binary classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research), Sorelle\u00a0A. Friedler and Christo Wilson (Eds.), Vol.\u00a081. PMLR, New York, NY, USA, 107\u2013118. http:\/\/proceedings.mlr.press\/v81\/menon18a.html"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1302"},{"key":"e_1_3_2_1_18_1","volume-title":"Wikipedia Talk Labels: Toxicity. (2","author":"Thain Nithum","year":"2017","unstructured":"Nithum Thain , Lucas Dixon , and Ellery Wulczyn . 2017. Wikipedia Talk Labels: Toxicity. (2 2017 ). Nithum Thain, Lucas Dixon, and Ellery Wulczyn. 2017. Wikipedia Talk Labels: Toxicity. (2 2017)."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052591"}],"event":{"name":"WWW '19: The Web Conference","location":"San Francisco USA","acronym":"WWW '19","sponsor":["IW3C2 International World Wide Web Conference Committee"]},"container-title":["Companion Proceedings of The 2019 World Wide Web Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308560.3317593","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3308560.3317593","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:53:35Z","timestamp":1750190015000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308560.3317593"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,13]]},"references-count":19,"alternative-id":["10.1145\/3308560.3317593","10.1145\/3308560"],"URL":"https:\/\/doi.org\/10.1145\/3308560.3317593","relation":{},"subject":[],"published":{"date-parts":[[2019,5,13]]},"assertion":[{"value":"2019-05-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}