{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T17:49:27Z","timestamp":1777657767277,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":31,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T00:00:00Z","timestamp":1626825600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,21]]},"DOI":"10.1145\/3461702.3462594","type":"proceedings-article","created":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T01:21:32Z","timestamp":1627694492000},"page":"916-925","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["A Step Toward More Inclusive People Annotations for Fairness"],"prefix":"10.1145","author":[{"given":"Candice","family":"Schumann","sequence":"first","affiliation":[{"name":"Google, Pittsburgh, PA, USA"}]},{"given":"Susanna","family":"Ricco","sequence":"additional","affiliation":[{"name":"Google, Mountain View, CA, USA"}]},{"given":"Utsav","family":"Prabhu","sequence":"additional","affiliation":[{"name":"Google, Pittsburgh, PA, USA"}]},{"given":"Vittorio","family":"Ferrari","sequence":"additional","affiliation":[{"name":"Google, Zurich, Switzerland"}]},{"given":"Caroline","family":"Pantofaru","sequence":"additional","affiliation":[{"name":"Google, Mountain View, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,7,30]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"ECCV Workshop on Bias Estimation in Face Analytics .","author":"Alvi M.","unstructured":"M. Alvi , A. Zisserman , and C. Nellaker . 2018. Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings . In ECCV Workshop on Bias Estimation in Face Analytics . M. Alvi, A. Zisserman, and C. Nellaker. 2018. Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings. In ECCV Workshop on Bias Estimation in Face Analytics ."},{"key":"e_1_3_2_1_2_1","volume-title":"The NeurIPS '18 Competition","author":"Atwood James","unstructured":"James Atwood , Yoni Halpern , Pallavi Baljekar , Eric Breck , D. Sculley , Pavel Ostyakov , Sergey I. Nikolenko , Igor Ivanov , Roman Solovyev , Weimin Wang , and Miha Skalic . 2020. The Inclusive Images Competition . In The NeurIPS '18 Competition , Sergio Escalera and Ralf Herbrich (Eds.). Springer International Publishing , Cham , 155--186. James Atwood, Yoni Halpern, Pallavi Baljekar, Eric Breck, D. Sculley, Pavel Ostyakov, Sergey I. Nikolenko, Igor Ivanov, Roman Solovyev, Weimin Wang, and Miha Skalic. 2020. The Inclusive Images Competition. In The NeurIPS '18 Competition, Sergio Escalera and Ralf Herbrich (Eds.). Springer International Publishing, Cham, 155--186."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314234"},{"key":"e_1_3_2_1_4_1","volume-title":"Proceedings of GSCL","author":"Bouma Gerlof","year":"2009","unstructured":"Gerlof Bouma . 2009 . Normalized (pointwise) mutual information in collocation extraction . Proceedings of GSCL (2009), 31--40. Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL (2009), 31--40."},{"key":"e_1_3_2_1_5_1","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Fairness Accountability and Transparency (FAT*). PMLR 77--91.  Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Fairness Accountability and Transparency (FAT*). PMLR 77--91."},{"key":"e_1_3_2_1_6_1","volume-title":"CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV) .","author":"de Vries Terrance","unstructured":"Terrance de Vries , Ishan Misra , Changhan Wang , and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone? . In CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV) . Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019. Does Object Recognition Work for Everyone?. In CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV) ."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_1_9_1","volume-title":"CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV) .","author":"Dulhanty Chris","year":"2019","unstructured":"Chris Dulhanty and Alexander Wong . 2019 . Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets . In CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV) . Chris Dulhanty and Alexander Wong. 2019. Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets. In CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV) ."},{"key":"e_1_3_2_1_10_1","volume-title":"Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford.","author":"Gebru Timnit","year":"2018","unstructured":"Timnit Gebru , Jamie Morgenstern , Briana Vecchione , Jennifer Wortman Vaughan , Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018 . Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018). Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018)."},{"key":"e_1_3_2_1_11_1","volume-title":"Ongoing Face Recognition Vendor Test (FRVT) Part 1: Verification","author":"Grother Patrick","year":"2018","unstructured":"Patrick Grother , Mei Ngan , and Kayee Hanaoka . 2018. Ongoing Face Recognition Vendor Test (FRVT) Part 1: Verification . National Institute of Standards and Technology ( 2018 ). Patrick Grother, Mei Ngan, and Kayee Hanaoka. 2018. Ongoing Face Recognition Vendor Test (FRVT) Part 1: Verification. National Institute of Standards and Technology (2018)."},{"key":"e_1_3_2_1_12_1","volume-title":"The Social Implications of Embedded Gender Recognition Systems. In CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada). 1--13","author":"Hamidi Foad","unstructured":"Foad Hamidi , Morgan Klaus Scheuerman , and Stacy M. Branham . 2018. Gender Recognition or Gender Reductionism? The Social Implications of Embedded Gender Recognition Systems. In CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada). 1--13 . https:\/\/doi.org\/10.1145\/3173574.3173582 Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M. Branham. 2018. Gender Recognition or Gender Reductionism? The Social Implications of Embedded Gender Recognition Systems. In CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada). 1--13. https:\/\/doi.org\/10.1145\/3173574.3173582"},{"key":"e_1_3_2_1_13_1","volume-title":"Garnett (Eds.)","volume":"29","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt , Eric Price , Eric Price , and Nati Srebro . 2016 . Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R . Garnett (Eds.) , Vol. 29 . Curran Associates, Inc., 3315--3323. https:\/\/proceedings.neurips.cc\/paper\/ 2016\/file\/9d2682367c3935defcb1f9e247a97c0d-Paper.pdf Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. 2016. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc., 3315--3323. https:\/\/proceedings.neurips.cc\/paper\/2016\/file\/9d2682367c3935defcb1f9e247a97c0d-Paper.pdf"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00159"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274357"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Alina Kuznetsova Hassan Rom Neil Alldrin Jasper Uijlings Ivan Krasin Jordi Pont-Tuset Shahab Kamali Stefan Popov Matteo Malloci Alexander Kolesnikov etal 2020. The open images dataset v4: Unified image classification object detection and visual relationship detection at scale. International Journal of Computer Vision (IJCV) (2020) 1--26.  Alina Kuznetsova Hassan Rom Neil Alldrin Jasper Uijlings Ivan Krasin Jordi Pont-Tuset Shahab Kamali Stefan Popov Matteo Malloci Alexander Kolesnikov et al. 2020. The open images dataset v4: Unified image classification object detection and visual relationship detection at scale. International Journal of Computer Vision (IJCV) (2020) 1--26.","DOI":"10.1007\/s11263-020-01316-z"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_1_18_1","volume-title":"Deep Learning Face Attributes in the Wild. In International Conference on Computer Vision (ICCV).","author":"Liu Ziwei","year":"2015","unstructured":"Ziwei Liu , Ping Luo , Xiaogang Wang , and Xiaoou Tang . 2015 . Deep Learning Face Attributes in the Wild. In International Conference on Computer Vision (ICCV). Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep Learning Face Attributes in the Wild. In International Conference on Computer Vision (ICCV)."},{"key":"e_1_3_2_1_19_1","volume-title":"International Conference on Machine Learning. PMLR, 3384--3393","author":"Madras David","year":"2018","unstructured":"David Madras , Elliot Creager , Toniann Pitassi , and Richard Zemel . 2018 . Learning adversarially fair and transferable representations . In International Conference on Machine Learning. PMLR, 3384--3393 . David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. 2018. Learning adversarially fair and transferable representations. In International Conference on Machine Learning. PMLR, 3384--3393."},{"key":"e_1_3_2_1_20_1","volume-title":"Diversity in faces. arXiv preprint arXiv:1901.10436","author":"Merler Michele","year":"2019","unstructured":"Michele Merler , Nalini Ratha , Rogerio S Feris , and John R Smith . 2019. Diversity in faces. arXiv preprint arXiv:1901.10436 ( 2019 ). Michele Merler, Nalini Ratha, Rogerio S Feris, and John R Smith. 2019. Diversity in faces. arXiv preprint arXiv:1901.10436 (2019)."},{"key":"e_1_3_2_1_21_1","volume-title":"Inioluwa Deborah Raji, and Timnit Gebru","author":"Mitchell Margaret","year":"2019","unstructured":"Margaret Mitchell , Simone Wu , Andrew Zaldivar , Parker Barnes , Lucy Vasserman , Ben Hutchinson , Elena Spitzer , Inioluwa Deborah Raji, and Timnit Gebru . 2019 . Model Cards for Model Reporting. In Fairness, Accountability, and Transparency (FAT *). 220--229. https:\/\/doi.org\/10.1145\/3287560.3287596 Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Fairness, Accountability, and Transparency (FAT*). 220--229. https:\/\/doi.org\/10.1145\/3287560.3287596"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_3_2_1_23_1","volume-title":"Inclusivefacenet: Improving face attribute detection with race and gender diversity. arXiv preprint arXiv:1712.00193","author":"Ryu Hee Jung","year":"2017","unstructured":"Hee Jung Ryu , Hartwig Adam , and Margaret Mitchell . 2017 . Inclusivefacenet: Improving face attribute detection with race and gender diversity. arXiv preprint arXiv:1712.00193 (2017). Hee Jung Ryu, Hartwig Adam, and Margaret Mitchell. 2017. Inclusivefacenet: Improving face attribute detection with race and gender diversity. arXiv preprint arXiv:1712.00193 (2017)."},{"key":"e_1_3_2_1_24_1","volume-title":"Branham","author":"Scheuerman Morgan Klaus","year":"2020","unstructured":"Morgan Klaus Scheuerman , Katta Speil , Oliver L. Haimson , Foad Hamidi , and Stacy M . Branham . 2020 . HCI Guidelines for Gender Equity and Inclusivity. http:\/\/morgan-klaus.com\/gender-guidelines.html. Version 1.1. Morgan Klaus Scheuerman, Katta Speil, Oliver L. Haimson, Foad Hamidi, and Stacy M. Branham. 2020. HCI Guidelines for Gender Equity and Inclusivity. http:\/\/morgan-klaus.com\/gender-guidelines.html. Version 1.1."},{"key":"e_1_3_2_1_25_1","volume-title":"NeurIPS Workshop: Machine Learning for the Developing World.","author":"Shankar Shreya","unstructured":"Shreya Shankar , Yoni Halpern , Eric Breck , James Atwood , Jimbo Wilson , and D. Sculley . 2017. No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World . In NeurIPS Workshop: Machine Learning for the Developing World. Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. 2017. No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World. In NeurIPS Workshop: Machine Learning for the Developing World."},{"key":"e_1_3_2_1_26_1","unstructured":"Craig I Watson. 2016. NIST Special Database 18. NIST Mugshot Identification Database (MID).  Craig I Watson. 2016. NIST Special Database 18. NIST Mugshot Identification Database (MID)."},{"key":"e_1_3_2_1_27_1","volume-title":"The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=Boy&oldid=992668770. [Online","author":"Wikipedia Wikipedia","year":"2021","unstructured":"Wikipedia contributors. 2020. Boy -- Wikipedia , The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=Boy&oldid=992668770. [Online ; accessed 29- January - 2021 ]. Wikipedia contributors. 2020. Boy -- Wikipedia, The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=Boy&oldid=992668770. [Online; accessed 29-January-2021]."},{"key":"e_1_3_2_1_28_1","volume-title":"The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=Girl&oldid=1000129400. [Online","author":"Wikipedia Wikipedia","year":"2021","unstructured":"Wikipedia contributors. 2021. Girl -- Wikipedia , The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=Girl&oldid=1000129400. [Online ; accessed 29- January - 2021 ]. Wikipedia contributors. 2021. Girl -- Wikipedia, The Free Encyclopedia. https:\/\/en.wikipedia.org\/w\/index.php?title=Girl&oldid=1000129400. [Online; accessed 29-January-2021]."},{"key":"e_1_3_2_1_29_1","volume-title":"Predictive Inequity in Object Detection. In CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV).","author":"Wilson Benjamin","year":"2019","unstructured":"Benjamin Wilson , Judy Hoffman , and Jamie Morgenstern . 2019 . Predictive Inequity in Object Detection. In CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV). Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. 2019. Predictive Inequity in Object Detection. In CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision (FATE CV)."},{"key":"e_1_3_2_1_30_1","volume-title":"Soft sampling for robust object detection. arXiv preprint arXiv:1806.06986","author":"Wu Zhe","year":"2018","unstructured":"Zhe Wu , Navaneeth Bodla , Bharat Singh , Mahyar Najibi , Rama Chellappa , and Larry S Davis . 2018. Soft sampling for robust object detection. arXiv preprint arXiv:1806.06986 ( 2018 ). Zhe Wu, Navaneeth Bodla, Bharat Singh, Mahyar Najibi, Rama Chellappa, and Larry S Davis. 2018. Soft sampling for robust object detection. arXiv preprint arXiv:1806.06986 (2018)."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Kaiyu Yang Klint Qinami Li Fei-Fei Jia Deng and Olga Russakovsky. 2020. Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy. In Fairness Accountability and Transparency (FAT*).  Kaiyu Yang Klint Qinami Li Fei-Fei Jia Deng and Olga Russakovsky. 2020. Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy. In Fairness Accountability and Transparency (FAT*).","DOI":"10.1145\/3351095.3375709"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.463"}],"event":{"name":"AIES '21: AAAI\/ACM Conference on AI, Ethics, and Society","location":"Virtual Event USA","acronym":"AIES '21","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","AAAI"]},"container-title":["Proceedings of the 2021 AAAI\/ACM Conference on AI, Ethics, and Society"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462594","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3461702.3462594","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:06Z","timestamp":1750191426000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462594"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,21]]},"references-count":31,"alternative-id":["10.1145\/3461702.3462594","10.1145\/3461702"],"URL":"https:\/\/doi.org\/10.1145\/3461702.3462594","relation":{},"subject":[],"published":{"date-parts":[[2021,7,21]]},"assertion":[{"value":"2021-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}