{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,13]],"date-time":"2025-09-13T15:45:52Z","timestamp":1757778352913,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":29,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T00:00:00Z","timestamp":1626825600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,21]]},"DOI":"10.1145\/3461702.3462557","type":"proceedings-article","created":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T01:21:38Z","timestamp":1627694498000},"page":"327-335","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Measuring Model Biases in the Absence of Ground Truth"],"prefix":"10.1145","author":[{"given":"Osman","family":"Aka","sequence":"first","affiliation":[{"name":"Google, Mountain View, CA, USA"}]},{"given":"Ken","family":"Burke","sequence":"additional","affiliation":[{"name":"Google, Mountain View, CA, USA"}]},{"given":"Alex","family":"Bauerle","sequence":"additional","affiliation":[{"name":"Ulm University, Ulm, Germany"}]},{"given":"Christina","family":"Greer","sequence":"additional","affiliation":[{"name":"Google, Mountain View, CA, USA"}]},{"given":"Margaret","family":"Mitchell","sequence":"additional","affiliation":[{"name":"Ethical AI LLC, Seattle, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,7,30]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013)","author":"Aletras Nikolaos","year":"2013","unstructured":"Nikolaos Aletras and Mark Stevenson . 2013 . Evaluating Topic Coherence Using Distributional Semantics . In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) -- Long Papers. Association for Computational Linguistics, Potsdam, Germany, 13--22. https:\/\/www.aclweb.org\/ anthology\/W13-0102 Nikolaos Aletras and Mark Stevenson. 2013. Evaluating Topic Coherence Using Distributional Semantics. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) -- Long Papers. Association for Computational Linguistics, Potsdam, Germany, 13--22. https:\/\/www.aclweb.org\/ anthology\/W13-0102"},{"key":"e_1_3_2_2_2_1","volume-title":"Selbst","author":"Barocas Solon","year":"2014","unstructured":"Solon Barocas and Andrew D . Selbst . 2014 . Big Data's Disparate Impact. SSRN eLibrary (2014). Solon Barocas and Andrew D. Selbst. 2014. Big Data's Disparate Impact. SSRN eLibrary (2014)."},{"key":"e_1_3_2_2_3_1","first-page":"6","article-title":"A primer on fairness in criminal justice risk assessments","volume":"41","author":"Berk Richard","year":"2016","unstructured":"Richard Berk . 2016 . A primer on fairness in criminal justice risk assessments . The Criminologist 41 , 6 (2016), 6 -- 9 . Richard Berk. 2016. A primer on fairness in criminal justice risk assessments. The Criminologist 41, 6 (2016), 6--9.","journal-title":"The Criminologist"},{"key":"e_1_3_2_2_4_1","unstructured":"G. Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction.  G. Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction."},{"key":"e_1_3_2_2_5_1","volume-title":"Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (Proceedings of Machine Learning Research","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (Proceedings of Machine Learning Research , Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 77-- 91 . http:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 77--91. http:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html"},{"key":"e_1_3_2_2_6_1","volume-title":"Trevor Darrell, and Anna Rohrbach.","author":"Burns Kaylee","year":"2018","unstructured":"Kaylee Burns , Lisa Anne Hendricks , Trevor Darrell, and Anna Rohrbach. 2018 . Women also Snowboard : Overcoming Bias in Captioning Models. CoRR abs\/1803.09797 (2018). arXiv:1803.09797 http:\/\/arxiv.org\/abs\/1803.09797 Kaylee Burns, Lisa Anne Hendricks, Trevor Darrell, and Anna Rohrbach. 2018. Women also Snowboard: Overcoming Bias in Captioning Models. CoRR abs\/1803.09797 (2018). arXiv:1803.09797 http:\/\/arxiv.org\/abs\/1803.09797"},{"key":"e_1_3_2_2_7_1","volume-title":"Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334","author":"Caliskan Aylin","year":"2017","unstructured":"Aylin Caliskan , Joanna J. Bryson , and Arvind Narayanan . 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 ( 2017 ), 183--186. https:\/\/doi.org\/10.1126\/science.aal4230 arXiv:https:\/\/science.sciencemag.org\/content\/356\/6334\/183.full.pdf Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186. https:\/\/doi.org\/10.1126\/science.aal4230 arXiv:https:\/\/science.sciencemag.org\/content\/356\/6334\/183.full.pdf"},{"key":"e_1_3_2_2_8_1","unstructured":"Alexandra Chouldechova. 2016. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. arXiv:1610.07524 [stat.AP]  Alexandra Chouldechova. 2016. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. arXiv:1610.07524 [stat.AP]"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.5555\/89086.89095"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"crossref","unstructured":"J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.  J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_2_11_1","volume-title":"Zemel","author":"Dwork Cynthia","year":"2011","unstructured":"Cynthia Dwork , Moritz Hardt , Toniann Pitassi , Omer Reingold , and Richard S . Zemel . 2011 . Fairness Through Awareness. CoRR abs\/1104.3913 (2011). arXiv:1104.3913 http:\/\/arxiv.org\/abs\/1104.3913 Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. 2011. Fairness Through Awareness. CoRR abs\/1104.3913 (2011). arXiv:1104.3913 http:\/\/arxiv.org\/abs\/1104.3913"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-014-0733-5"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1119\/1.1937609"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"crossref","unstructured":"A. G. Greenwald D. E. McGhee and J. L. Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology 74 (1998). Issue 6.  A. G. Greenwald D. E. McGhee and J. L. Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology 74 (1998). Issue 6.","DOI":"10.1037\/0022-3514.74.6.1464"},{"key":"e_1_3_2_2_15_1","unstructured":"Moritz Hardt Eric Price and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. arXiv:1610.02413 [cs.LG]  Moritz Hardt Eric Price and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. arXiv:1610.02413 [cs.LG]"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1080\/00437956.1954.11659520"},{"key":"e_1_3_2_2_17_1","volume-title":"Martin","author":"Jurafsky Dan","year":"2009","unstructured":"Dan Jurafsky and James H . Martin . 2009 . Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition. Pearson Prentice Hall , Upper Saddle River, N.J. http:\/\/www.amazon.com\/Speech-Language-Processing-2nd-Edition\/dp\/ 0131873210\/ref=pd_bxgy_b_img_y Dan Jurafsky and James H. Martin. 2009. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition. Pearson Prentice Hall, Upper Saddle River, N.J. http:\/\/www.amazon.com\/Speech-Language-Processing-2nd-Edition\/dp\/ 0131873210\/ref=pd_bxgy_b_img_y"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_2_2_19_1","volume-title":"The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. CoRR abs\/1811.00982","author":"Kuznetsova Alina","year":"2018","unstructured":"Alina Kuznetsova , Hassan Rom , Neil Alldrin , Jasper R. R. Uijlings , Ivan Krasin , Jordi Pont-Tuset , Shahab Kamali , Stefan Popov , Matteo Malloci , Tom Duerig , and Vittorio Ferrari . 2018. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. CoRR abs\/1811.00982 ( 2018 ). arXiv:1811.00982 http:\/\/arxiv.org\/abs\/1811.00982 Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. 2018. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. CoRR abs\/1811.00982 (2018). arXiv:1811.00982 http:\/\/arxiv.org\/abs\/1811.00982"},{"key":"e_1_3_2_2_20_1","volume-title":"Microsoft COCO: Common Objects in Context. CoRR abs\/1405.0312","author":"Lin Tsung-Yi","year":"2014","unstructured":"Tsung-Yi Lin , Michael Maire , Serge J. Belongie , Lubomir D. Bourdev , Ross B. Girshick , James Hays , Pietro Perona , Deva Ramanan , Piotr Doll\u00e1r , and C. Lawrence Zitnick . 2014. Microsoft COCO: Common Objects in Context. CoRR abs\/1405.0312 ( 2014 ). arXiv:1405.0312 http:\/\/arxiv.org\/abs\/1405.0312 Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. CoRR abs\/1405.0312 (2014). arXiv:1405.0312 http:\/\/arxiv.org\/abs\/1405.0312"},{"key":"e_1_3_2_2_21_1","volume-title":"Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - KDIR, (IC3K","author":"Role Fran\u00c7ois","year":"2011","unstructured":"Fran\u00c7ois Role and Mohamed Nadif . 2011 . Handling the Impact of Low Frequency Events on Co-Occurrence Based Measures of Word Similarity - A Case Study of Pointwise Mutual Information . In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - KDIR, (IC3K 2011). SciTePress, 218--223. Fran\u00c7ois Role and Mohamed Nadif. 2011. Handling the Impact of Low Frequency Events on Co-Occurrence Based Measures of Word Similarity - A Case Study of Pointwise Mutual Information. In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - KDIR, (IC3K 2011). SciTePress, 218--223."},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb01338.x"},{"key":"e_1_3_2_2_24_1","unstructured":"Jacob Snow. 2018. Amazon's Face Recognition Falsely Matched 28 Members of Congress With Mugshots. (2018).  Jacob Snow. 2018. Amazon's Face Recognition Falsely Matched 28 Members of Congress With Mugshots. (2018)."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.2307\/1412159"},{"key":"e_1_3_2_2_26_1","volume-title":"http:\/\/image-net.org\/explore","author":"Lab Stanford Vision","year":"2020","unstructured":"Stanford Vision Lab . 2020. ImageNet. http:\/\/image-net.org\/explore ( 2020 ). accessed 6.Oct.2020. Stanford Vision Lab. 2020. ImageNet. http:\/\/image-net.org\/explore (2020). accessed 6.Oct.2020."},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01231-1_31"},{"key":"e_1_3_2_2_28_1","unstructured":"M. P. Toglia and W. F. Battig. 1978. Handbook of semantic word norms. Lawrence Erlbaum.  M. P. Toglia and W. F. Battig. 1978. Handbook of semantic word norms. Lawrence Erlbaum."},{"key":"e_1_3_2_2_29_1","volume-title":"Predictive Inequity in Object Detection. CoRR abs\/1902.11097","author":"Wilson Benjamin","year":"2019","unstructured":"Benjamin Wilson , Judy Hoffman , and Jamie Morgenstern . 2019. Predictive Inequity in Object Detection. CoRR abs\/1902.11097 ( 2019 ). arXiv:1902.11097 http:\/\/arxiv.org\/abs\/1902.11097 Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. 2019. Predictive Inequity in Object Detection. CoRR abs\/1902.11097 (2019). arXiv:1902.11097 http:\/\/arxiv.org\/abs\/1902.11097"}],"event":{"name":"AIES '21: AAAI\/ACM Conference on AI, Ethics, and Society","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","AAAI"],"location":"Virtual Event USA","acronym":"AIES '21"},"container-title":["Proceedings of the 2021 AAAI\/ACM Conference on AI, Ethics, and Society"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462557","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3461702.3462557","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:49:06Z","timestamp":1750193346000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462557"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,21]]},"references-count":29,"alternative-id":["10.1145\/3461702.3462557","10.1145\/3461702"],"URL":"https:\/\/doi.org\/10.1145\/3461702.3462557","relation":{},"subject":[],"published":{"date-parts":[[2021,7,21]]},"assertion":[{"value":"2021-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}