{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T21:48:55Z","timestamp":1776116935888,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":155,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,6,12]],"date-time":"2023-06-12T00:00:00Z","timestamp":1686528000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,6,12]]},"DOI":"10.1145\/3593013.3594019","type":"proceedings-article","created":{"date-parts":[[2023,6,12]],"date-time":"2023-06-12T14:40:46Z","timestamp":1686580846000},"page":"519-533","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Representation in AI Evaluations"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4331-1357","authenticated-orcid":false,"given":"A. Stevie","family":"Bergman","sequence":"first","affiliation":[{"name":"DeepMind, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9340-5143","authenticated-orcid":false,"given":"Lisa Anne","family":"Hendricks","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-9036-4230","authenticated-orcid":false,"given":"Maribeth","family":"Rauh","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0641-6065","authenticated-orcid":false,"given":"Boxi","family":"Wu","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1362-554X","authenticated-orcid":false,"given":"William","family":"Agnew","sequence":"additional","affiliation":[{"name":"University of Washington, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3818-7897","authenticated-orcid":false,"given":"Markus","family":"Kunesch","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9766-0886","authenticated-orcid":false,"given":"Isabella","family":"Duan","sequence":"additional","affiliation":[{"name":"University of Chicago, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7552-4576","authenticated-orcid":false,"given":"Iason","family":"Gabriel","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1297-5409","authenticated-orcid":false,"given":"William","family":"Isaac","sequence":"additional","affiliation":[{"name":"DeepMind, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2023,6,12]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Thirty Sixth Conference on Neural Information Processing Systems (NeurIPS 2022","year":"2022","unstructured":"NeurIPS 2022 . 2022 . Thirty Sixth Conference on Neural Information Processing Systems (NeurIPS 2022 ). Retrieved January 25, 2023 from https:\/\/neurips.cc\/ NeurIPS 2022. 2022. Thirty Sixth Conference on Neural Information Processing Systems (NeurIPS 2022). Retrieved January 25, 2023 from https:\/\/neurips.cc\/"},{"key":"e_1_3_2_2_2_1","volume-title":"Proceedings of the 2019 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 801\u2013809","author":"Abbasi Mohsen","year":"2019","unstructured":"Mohsen Abbasi , Sorelle A Friedler , Carlos Scheidegger , and Suresh Venkatasubramanian . 2019 . Fairness in representation: quantifying stereotyping as a representational harm . In Proceedings of the 2019 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 801\u2013809 . Mohsen Abbasi, Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2019. Fairness in representation: quantifying stereotyping as a representational harm. In Proceedings of the 2019 SIAM International Conference on Data Mining (SDM). Society for Industrial and Applied Mathematics, 801\u2013809."},{"key":"e_1_3_2_2_3_1","unstructured":"[\n  3\n  ]  Parity AI. 2023. Retrieved January 16 2023 from https:\/\/www.getparity.ai\/  [3] Parity AI. 2023. Retrieved January 16 2023 from https:\/\/www.getparity.ai\/"},{"key":"e_1_3_2_2_4_1","volume-title":"Deep Face Matchers Say No. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE Computer Society","author":"Albiero V.","year":"2020","unstructured":"V. Albiero , K. W. Bowyer , K. Vangara , and M. C. King . 2020. Does Face Recognition Accuracy Get Better With Age? Deep Face Matchers Say No. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE Computer Society , Los Alamitos, CA, USA, 250\u2013258. https:\/\/doi.org\/10.1109\/WACV45572. 2020 .9093357 10.1109\/WACV45572.2020.9093357 V. Albiero, K. W. Bowyer, K. Vangara, and M. C. King. 2020. Does Face Recognition Accuracy Get Better With Age? Deep Face Matchers Say No. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE Computer Society, Los Alamitos, CA, USA, 250\u2013258. https:\/\/doi.org\/10.1109\/WACV45572.2020.9093357"},{"key":"e_1_3_2_2_5_1","volume-title":"Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Albrecht Joshua","year":"2022","unstructured":"Joshua Albrecht , Abraham J Fetterman , Bryden Fogelman , Ellie Kitanidis , Bartosz Wr\u00f3blewski , Nicole Seo , Michael Rosenthal , Maksis Knutins , Zachary Polizzi , James B Simon , 2022 . Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Joshua Albrecht, Abraham J Fetterman, Bryden Fogelman, Ellie Kitanidis, Bartosz Wr\u00f3blewski, Nicole Seo, Michael Rosenthal, Maksis Knutins, Zachary Polizzi, James B Simon, 2022. Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_6_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Alfassy Amit","year":"2022","unstructured":"Amit Alfassy , Assaf Arbelle , Oshri Halimi , Sivan Harary , Roei Herzig , Eli Schwartz , Rameswar Panda , Michele Dolfi , Christoph Auer , Kate Saenko , 2022 . FETA: Towards Specializing Foundation Models for Expert Task Applications . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Amit Alfassy, Assaf Arbelle, Oshri Halimi, Sivan Harary, Roei Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer, Kate Saenko, 2022. FETA: Towards Specializing Foundation Models for Expert Task Applications. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_7_1","volume-title":"A Ladder of Citizen Participation. 35 (July","author":"Arnstein Sherry R.","year":"1969","unstructured":"Sherry R. Arnstein . 1969. A Ladder of Citizen Participation. 35 (July 1969 ), 216\u2013224. Issue 4. Sherry R. Arnstein. 1969. A Ladder of Citizen Participation. 35 (July 1969), 216\u2013224. Issue 4."},{"key":"e_1_3_2_2_8_1","unstructured":"Solon Barocas Moritz Hardt and Arvind Narayanan. 2019. Fairness and Machine Learning. fairmlbook.org. http:\/\/www.fairmlbook.org.  Solon Barocas Moritz Hardt and Arvind Narayanan. 2019. Fairness and Machine Learning. fairmlbook.org. http:\/\/www.fairmlbook.org."},{"key":"e_1_3_2_2_10_1","volume-title":"Big data and the underground railroad. Slate","author":"Bedoya Alvaro","year":"2014","unstructured":"Alvaro Bedoya . 2014. Big data and the underground railroad. Slate ( 2014 ). http:\/\/www.slate.com\/articles\/technology\/future_tense\/2014\/11\/big_ data_underground_railroad_history_says_unfettered_collection_of_data.html. Alvaro Bedoya. 2014. Big data and the underground railroad. Slate (2014). http:\/\/www.slate.com\/articles\/technology\/future_tense\/2014\/11\/big_ data_underground_railroad_history_says_unfettered_collection_of_data.html."},{"key":"e_1_3_2_2_11_1","volume-title":"The #BenderRule: On Naming the Languages We Study and Why It Matters. The Gradient","author":"Bender Emily","year":"2019","unstructured":"Emily Bender . 2019. The #BenderRule: On Naming the Languages We Study and Why It Matters. The Gradient ( 2019 ). https:\/\/thegradient.pub\/the-benderrule-on-naming-the-languages-we-study-and-why-it-matters\/. Emily Bender. 2019. The #BenderRule: On Naming the Languages We Study and Why It Matters. The Gradient (2019). https:\/\/thegradient.pub\/the-benderrule-on-naming-the-languages-we-study-and-why-it-matters\/."},{"key":"e_1_3_2_2_12_1","volume-title":"On Achieving and Evaluating Language-Independence in NLP. LiLT 6 (Oct","author":"Bender Emily M","year":"2011","unstructured":"Emily M Bender . 2011. On Achieving and Evaluating Language-Independence in NLP. LiLT 6 (Oct . 2011 ). Emily M Bender. 2011. On Achieving and Evaluating Language-Independence in NLP. LiLT 6 (Oct. 2011)."},{"key":"e_1_3_2_2_13_1","volume-title":"Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives. CoRR abs\/1206.5538","author":"Bengio Yoshua","year":"2012","unstructured":"Yoshua Bengio , Aaron C. Courville , and Pascal Vincent . 2012. Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives. CoRR abs\/1206.5538 ( 2012 ). arXiv:1206.5538http:\/\/arxiv.org\/abs\/1206.5538 Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2012. Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives. CoRR abs\/1206.5538 (2012). arXiv:1206.5538http:\/\/arxiv.org\/abs\/1206.5538"},{"key":"e_1_3_2_2_14_1","volume-title":"Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics","author":"Bergman A. Stevie","year":"2022","unstructured":"A. Stevie Bergman , Gavin Abercrombie , Shannon Spruit , Dirk Hovy , Emily Dinan , Y- Lan Boureau , and Verena Rieser . 2022 . Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design . In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics , Edinburgh, UK, 39\u201352. https:\/\/aclanthology.org\/ 2022.sigdial-1.4 A. Stevie Bergman, Gavin Abercrombie, Shannon Spruit, Dirk Hovy, Emily Dinan, Y-Lan Boureau, and Verena Rieser. 2022. Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, Edinburgh, UK, 39\u201352. https:\/\/aclanthology.org\/2022.sigdial-1.4"},{"key":"e_1_3_2_2_15_1","volume-title":"Towards Responsible Natural Language Annotation for the Varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022","author":"Stevie Bergman A.","year":"2022","unstructured":"A. Stevie Bergman and Mona Diab . 2022 . Towards Responsible Natural Language Annotation for the Varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022 . Association for Computational Linguistics, Dublin, Ireland, 364\u2013371. https:\/\/doi.org\/10. 18653\/v1\/2022.findings-acl.31 10.18653\/v1 A. Stevie Bergman and Mona Diab. 2022. Towards Responsible Natural Language Annotation for the Varieties of Arabic. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, Dublin, Ireland, 364\u2013371. https:\/\/doi.org\/10.18653\/v1\/2022.findings-acl.31"},{"key":"e_1_3_2_2_16_1","volume-title":"Iason Gabriel","author":"Birhane Abeba","year":"2022","unstructured":"Abeba Birhane , William Isaac , Vinodkumar Prabhakaran , Mark D\u00edaz , Madeleine Clare Elish , Iason Gabriel , and Shakir Mohamed. 2022 . Power to the People? Opportunities and Challenges for Participatory AI. In Equity and Access in Algorithms, Mechanisms, and Optimization (Arlington, VA, USA) (EAAMO \u201922). Association for Computing Machinery , New York, NY, USA, Article 6, 8 pages. https:\/\/doi.org\/10.1145\/3551624.3555290 10.1145\/3551624.3555290 Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark D\u00edaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022. Power to the People? Opportunities and Challenges for Participatory AI. In Equity and Access in Algorithms, Mechanisms, and Optimization (Arlington, VA, USA) (EAAMO \u201922). Association for Computing Machinery, New York, NY, USA, Article 6, 8 pages. https:\/\/doi.org\/10.1145\/3551624.3555290"},{"key":"e_1_3_2_2_17_1","volume-title":"The Values Encoded in Machine Learning Research. In 2022 ACM Conference on Fairness, Accountability, and Transparency","author":"Birhane Abeba","year":"2022","unstructured":"Abeba Birhane , Pratyusha Kalluri , Dallas Card , William Agnew , Ravit Dotan , and Michelle Bao . 2022 . The Values Encoded in Machine Learning Research. In 2022 ACM Conference on Fairness, Accountability, and Transparency ( Seoul, Republic of Korea) (FAccT \u201922). Association for Computing Machinery, New York, NY, USA, 173\u2013184. https:\/\/doi.org\/10.1145\/3531146.3533083 10.1145\/3531146.3533083 Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022. The Values Encoded in Machine Learning Research. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT \u201922). Association for Computing Machinery, New York, NY, USA, 173\u2013184. https:\/\/doi.org\/10.1145\/3531146.3533083"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.485"},{"key":"e_1_3_2_2_19_1","volume-title":"Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets","author":"Blodgett Su Lin","unstructured":"Su Lin Blodgett , Gilsinia Lopez , Alexandra Olteanu , Robert Sim , and Hanna Wallach . 2021. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets . Association for Computational Linguistics . Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. Association for Computational Linguistics."},{"key":"e_1_3_2_2_20_1","unstructured":"Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson S. Buch Dallas Card Rodrigo Castellon Niladri S. Chatterji Annie S. Chen Kathleen A. Creel Jared Davis Dora Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren E. Gillespie Karan Goel Noah D. Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas F. Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani O. Khattab Pang Wei Koh Mark S. Krass Ranjay Krishna Rohith Kuditipudi Ananya Kumar Faisal Ladhak Mina Lee Tony Lee Jure Leskovec Isabelle Levent Xiang Lisa Li Xuechen Li Tengyu Ma Ali Malik Christopher D. Manning Suvir P. Mirchandani Eric Mitchell Zanele Munyikwa Suraj Nair Avanika Narayan Deepak Narayanan Benjamin Newman Allen Nie Juan Carlos Niebles Hamed Nilforoshan J. F. Nyarko Giray Ogut Laurel Orr Isabel Papadimitriou Joon Sung Park Chris Piech Eva Portelance Christopher Potts Aditi Raghunathan Robert Reich Hongyu Ren Frieda Rong Yusuf H. Roohani Camilo Ruiz Jack Ryan Christopher R\u2019e Dorsa Sadigh Shiori Sagawa Keshav Santhanam Andy Shih Krishna Parasuram Srinivasan Alex Tamkin Rohan Taori Armin W. Thomas Florian Tram\u00e8r Rose E. Wang William Wang Bohan Wu Jiajun Wu Yuhuai Wu Sang Michael Xie Michihiro Yasunaga Jiaxuan You Matei A. Zaharia Michael Zhang Tianyi Zhang Xikun Zhang Yuhui Zhang Lucia Zheng Kaitlyn Zhou and Percy Liang. 2021. On the Opportunities and Risks of Foundation Models. ArXiv (2021). https:\/\/crfm.stanford.edu\/assets\/report.pdf  Rishi Bommasani Drew A. Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S. Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson S. Buch Dallas Card Rodrigo Castellon Niladri S. Chatterji Annie S. Chen Kathleen A. Creel Jared Davis Dora Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren E. Gillespie Karan Goel Noah D. Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas F. Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani O. Khattab Pang Wei Koh Mark S. Krass Ranjay Krishna Rohith Kuditipudi Ananya Kumar Faisal Ladhak Mina Lee Tony Lee Jure Leskovec Isabelle Levent Xiang Lisa Li Xuechen Li Tengyu Ma Ali Malik Christopher D. Manning Suvir P. Mirchandani Eric Mitchell Zanele Munyikwa Suraj Nair Avanika Narayan Deepak Narayanan Benjamin Newman Allen Nie Juan Carlos Niebles Hamed Nilforoshan J. F. Nyarko Giray Ogut Laurel Orr Isabel Papadimitriou Joon Sung Park Chris Piech Eva Portelance Christopher Potts Aditi Raghunathan Robert Reich Hongyu Ren Frieda Rong Yusuf H. Roohani Camilo Ruiz Jack Ryan Christopher R\u2019e Dorsa Sadigh Shiori Sagawa Keshav Santhanam Andy Shih Krishna Parasuram Srinivasan Alex Tamkin Rohan Taori Armin W. Thomas Florian Tram\u00e8r Rose E. Wang William Wang Bohan Wu Jiajun Wu Yuhuai Wu Sang Michael Xie Michihiro Yasunaga Jiaxuan You Matei A. Zaharia Michael Zhang Tianyi Zhang Xikun Zhang Yuhui Zhang Lucia Zheng Kaitlyn Zhou and Percy Liang. 2021. On the Opportunities and Risks of Foundation Models. ArXiv (2021). https:\/\/crfm.stanford.edu\/assets\/report.pdf"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462612"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/2838885"},{"key":"e_1_3_2_2_23_1","volume-title":"Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research","volume":"91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification . In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research , Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77\u2013 91 . https:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77\u201391. https:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html"},{"key":"e_1_3_2_2_24_1","first-page":"4","article-title":"Statistics versus machine learning","volume":"15","author":"Bzdok Danilo","year":"2018","unstructured":"Danilo Bzdok , Naomi Altman , and Martin Krzywinski . 2018 . Statistics versus machine learning . Nat. Methods 15 , 4 (April 2018), 233\u2013234. Danilo Bzdok, Naomi Altman, and Martin Krzywinski. 2018. Statistics versus machine learning. Nat. Methods 15, 4 (April 2018), 233\u2013234.","journal-title":"Nat. Methods"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533203"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445872"},{"key":"e_1_3_2_2_27_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Chen Daoyuan","year":"2022","unstructured":"Daoyuan Chen , Dawei Gao , Weirui Kuang , Yaliang Li , and Bolin Ding . 2022 . pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding. 2022. pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/3327144.3327272"},{"key":"e_1_3_2_2_29_1","first-page":"1","article-title":"Exploring racial and gender disparities in voice biometrics. Sci","volume":"12","author":"Chen Xingyu","year":"2022","unstructured":"Xingyu Chen , Zhengxiong Li , Srirangaraj Setlur , and Wenyao Xu . 2022 . Exploring racial and gender disparities in voice biometrics. Sci . Rep. 12 , 1 (March 2022), 3723. Xingyu Chen, Zhengxiong Li, Srirangaraj Setlur, and Wenyao Xu. 2022. Exploring racial and gender disparities in voice biometrics. Sci. Rep. 12, 1 (March 2022), 3723.","journal-title":"Rep."},{"key":"e_1_3_2_2_30_1","volume-title":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Marta R","year":"2022","unstructured":"Marta R Costa-juss\u00e0, Christine Basta , Oriol Domingo , and Andr\u00e9 Niyongabo Rubungo . 2022 . Occgen: Selection of real-world multilingual parallel data balanced in gender within occupations . In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Marta R Costa-juss\u00e0, Christine Basta, Oriol Domingo, and Andr\u00e9 Niyongabo Rubungo. 2022. Occgen: Selection of real-world multilingual parallel data balanced in gender within occupations. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_31_1","volume-title":"Design Justice: Towards an Intersectional Feminist Framework for Design Theory and Practice. (June","author":"Costanza-Chock Sasha","year":"2018","unstructured":"Sasha Costanza-Chock . 2018 . Design Justice: Towards an Intersectional Feminist Framework for Design Theory and Practice. (June 2018). Sasha Costanza-Chock. 2018. Design Justice: Towards an Intersectional Feminist Framework for Design Theory and Practice. (June 2018)."},{"key":"e_1_3_2_2_32_1","volume-title":"Idea density\u2014A potentially informative characteristic of retrieved documents","author":"Covington Michael A","year":"2009","unstructured":"Michael A Covington . 2009. Idea density\u2014A potentially informative characteristic of retrieved documents . In IEEE Southeastcon 2009 . IEEE , 201\u2013203. Michael A Covington. 2009. Idea density\u2014A potentially informative characteristic of retrieved documents. In IEEE Southeastcon 2009. IEEE, 201\u2013203."},{"key":"e_1_3_2_2_33_1","unstructured":"Kate Crawford. 2017. The trouble with bias. (2017). https:\/\/www.youtube.com\/watch?v=fMym_BKWQzk NeurIPS.  Kate Crawford. 2017. The trouble with bias. (2017). https:\/\/www.youtube.com\/watch?v=fMym_BKWQzk NeurIPS."},{"key":"e_1_3_2_2_34_1","volume-title":"Excavating AI: the politics of images in machine learning training sets. AI & SOCIETY (06","author":"Crawford Kate","year":"2021","unstructured":"Kate Crawford and Trevor Paglen . 2021. Excavating AI: the politics of images in machine learning training sets. AI & SOCIETY (06 2021 ). https:\/\/doi.org\/10.1007\/s00146-021-01162-8 10.1007\/s00146-021-01162-8 Kate Crawford and Trevor Paglen. 2021. Excavating AI: the politics of images in machine learning training sets. AI & SOCIETY (06 2021). https:\/\/doi.org\/10.1007\/s00146-021-01162-8"},{"key":"e_1_3_2_2_35_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Cui Ganqu","year":"2022","unstructured":"Ganqu Cui , Lifan Yuan , Bingxiang He , Yangyi Chen , Zhiyuan Liu , and Maosong Sun . 2022 . A unified evaluation of textual backdoor learning: Frameworks and benchmarks . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. 2022. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_36_1","volume-title":"Amazon scraps secret AI recruiting tool that showed bias against women","author":"Dastin Jeffrey","year":"2018","unstructured":"Jeffrey Dastin . 2018. Amazon scraps secret AI recruiting tool that showed bias against women . Reuters ( 2018 ). https:\/\/www.reuters.com\/article\/us-amazon-com-jobs-automation-insight\/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G. Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters (2018). https:\/\/www.reuters.com\/article\/us-amazon-com-jobs-automation-insight\/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G."},{"key":"e_1_3_2_2_37_1","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops. 52\u201359","author":"Vries Terrance De","year":"2019","unstructured":"Terrance De Vries , Ishan Misra , Changhan Wang , and Laurens Van der Maaten . 2019 . Does object recognition work for everyone? . In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops. 52\u201359 . Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten. 2019. Does object recognition work for everyone?. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops. 52\u201359."},{"key":"e_1_3_2_2_38_1","unstructured":"Joseph DelPreto Chao Liu Yiyue Luo Michael Foshey Yunzhu Li Antonio Torralba Wojciech Matusik and Daniela Rus. 2022. ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment. In Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks. https:\/\/action-sense.csail.mit.edu  Joseph DelPreto Chao Liu Yiyue Luo Michael Foshey Yunzhu Li Antonio Torralba Wojciech Matusik and Daniela Rus. 2022. ActionSense: A Multimodal Dataset and Recording Framework for Human Activities Using Wearable Sensors in a Kitchen Environment. In Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks. https:\/\/action-sense.csail.mit.edu"},{"key":"e_1_3_2_2_39_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Deng Chengyuan","year":"2022","unstructured":"Chengyuan Deng , Shihang Feng , Hanchen Wang , Xitong Zhang , Peng Jin , Yinan Feng , Qili Zeng , Yinpeng Chen , and Youzuo Lin . 2022 . OpenFWI: Large-Scale Multi-Structural Benchmark Datasets for Seismic Full Waveform Inversion . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Chengyuan Deng, Shihang Feng, Hanchen Wang, Xitong Zhang, Peng Jin, Yinan Feng, Qili Zeng, Yinpeng Chen, and Youzuo Lin. 2022. OpenFWI: Large-Scale Multi-Structural Benchmark Datasets for Seismic Full Waveform Inversion. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1177\/20539517211035955"},{"key":"e_1_3_2_2_42_1","volume-title":"Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH). Association for Computational Linguistics","author":"D\u00edaz Mark","year":"2022","unstructured":"Mark D\u00edaz , Razvan Amironesei , Laura Weidinger , and Iason Gabriel . 2022 . Accounting for Offensive Speech as a Practice of Resistance . In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH). Association for Computational Linguistics , Seattle, Washington (Hybrid), 192\u2013202. Mark D\u00edaz, Razvan Amironesei, Laura Weidinger, and Iason Gabriel. 2022. Accounting for Offensive Speech as a Practice of Resistance. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH). Association for Computational Linguistics, Seattle, Washington (Hybrid), 192\u2013202."},{"key":"e_1_3_2_2_43_1","volume-title":"CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation. In 2022 ACM Conference on Fairness, Accountability, and Transparency","author":"D\u00edaz Mark","year":"2022","unstructured":"Mark D\u00edaz , Ian Kivlichan , Rachel Rosen , Dylan Baker , Razvan Amironesei , Vinodkumar Prabhakaran , and Emily Denton . 2022 . CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation. In 2022 ACM Conference on Fairness, Accountability, and Transparency ( Seoul, Republic of Korea) (FAccT \u201922). Association for Computing Machinery, New York, NY, USA, 2342\u20132351. Mark D\u00edaz, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vinodkumar Prabhakaran, and Emily Denton. 2022. CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT \u201922). Association for Computing Machinery, New York, NY, USA, 2342\u20132351."},{"key":"e_1_3_2_2_44_1","volume-title":"Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging STS 5 (March","author":"Elish Madeleine Clare","year":"2019","unstructured":"Madeleine Clare Elish . 2019. Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging STS 5 (March 2019 ), 40\u201360. Madeleine Clare Elish. 2019. Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction. Engaging STS 5 (March 2019), 40\u201360."},{"key":"e_1_3_2_2_45_1","volume-title":"Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor","year":"2018","unstructured":"Eubanks, Virginia. 2018 . Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor . St. Martin\u2019s Press, Inc. , USA. Eubanks, Virginia. 2018. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin\u2019s Press, Inc., USA."},{"key":"e_1_3_2_2_46_1","volume-title":"Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence","author":"European Commission","unstructured":"European Commission . 2021. Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence ( Artificial Intelligence Act) and Amending Cerntain Union Legislative Acts . https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELLAR:e0649735-a372-11eb-9585-01aa75ed71a1. European Commission. 2021. Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Cerntain Union Legislative Acts. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELLAR:e0649735-a372-11eb-9585-01aa75ed71a1."},{"key":"e_1_3_2_2_47_1","first-page":"1","article-title":"Diversity in sociotechnical machine learning systems","volume":"9","author":"Fazelpour Sina","year":"2022","unstructured":"Sina Fazelpour and Maria De-Arteaga . 2022 . Diversity in sociotechnical machine learning systems . Big Data & Society 9 , 1 (Jan. 2022), 20539517221082027. Sina Fazelpour and Maria De-Arteaga. 2022. Diversity in sociotechnical machine learning systems. Big Data & Society 9, 1 (Jan. 2022), 20539517221082027.","journal-title":"Big Data & Society"},{"key":"e_1_3_2_2_48_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Feng Shangbin","year":"2022","unstructured":"Shangbin Feng , Zhaoxuan Tan , Herun Wan , Ningnan Wang , Zilong Chen , Binchi Zhang , Qinghua Zheng , Wenqian Zhang , Zhenyu Lei , Shujie Yang , 2022 . TwiBot-22: Towards graph-based Twitter bot detection . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Shangbin Feng, Zhaoxuan Tan, Herun Wan, Ningnan Wang, Zilong Chen, Binchi Zhang, Qinghua Zheng, Wenqian Zhang, Zhenyu Lei, Shujie Yang, 2022. TwiBot-22: Towards graph-based Twitter bot detection. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_49_1","first-page":"4","article-title":"Making the future together: Shaping autism research through meaningful participation","volume":"23","author":"Fletcher-Watson Sue","year":"2019","unstructured":"Sue Fletcher-Watson , Jon Adams , Kabie Brook , Tony Charman , Laura Crane , James Cusack , Susan Leekam , Damian Milton , Jeremy R Parr , and Elizabeth Pellicano . 2019 . Making the future together: Shaping autism research through meaningful participation . Autism 23 , 4 (May 2019), 943\u2013953. Sue Fletcher-Watson, Jon Adams, Kabie Brook, Tony Charman, Laura Crane, James Cusack, Susan Leekam, Damian Milton, Jeremy R Parr, and Elizabeth Pellicano. 2019. Making the future together: Shaping autism research through meaningful participation. Autism 23, 4 (May 2019), 943\u2013953.","journal-title":"Autism"},{"key":"e_1_3_2_2_50_1","unstructured":"[\n  50\n  ]  Sortition Foundation. 2023. Retrieved January 16 2023 from https:\/\/www.sortitionfoundation.org\/  [50] Sortition Foundation. 2023. Retrieved January 16 2023 from https:\/\/www.sortitionfoundation.org\/"},{"key":"e_1_3_2_2_51_1","volume-title":"The Vendi Score: A Diversity Evaluation Metric for Machine Learning. arXiv preprint arXiv:2210.02410","author":"Friedman Dan","year":"2022","unstructured":"Dan Friedman and Adji Bousso Dieng . 2022. The Vendi Score: A Diversity Evaluation Metric for Machine Learning. arXiv preprint arXiv:2210.02410 ( 2022 ). Dan Friedman and Adji Bousso Dieng. 2022. The Vendi Score: A Diversity Evaluation Metric for Machine Learning. arXiv preprint arXiv:2210.02410 (2022)."},{"key":"e_1_3_2_2_52_1","volume-title":"How an Attempt at Correcting Bias in Tech Goes Wrong. The Atlantic","author":"Fussell Sidney","year":"2019","unstructured":"Sidney Fussell . 2019. How an Attempt at Correcting Bias in Tech Goes Wrong. The Atlantic ( 2019 ). https:\/\/www.theatlantic.com\/technology\/archive\/2019\/10\/google-allegedly-used-homeless-train-pixel-phone\/599668\/. Sidney Fussell. 2019. How an Attempt at Correcting Bias in Tech Goes Wrong. The Atlantic (2019). https:\/\/www.theatlantic.com\/technology\/archive\/2019\/10\/google-allegedly-used-homeless-train-pixel-phone\/599668\/."},{"key":"e_1_3_2_2_53_1","volume-title":"(Aug.","author":"Ganguli Deep","year":"2022","unstructured":"Deep Ganguli , Liane Lovitt , Jackson Kernion , Amanda Askell , Yuntao Bai , Saurav Kadavath , Ben Mann , Ethan Perez , Nicholas Schiefer , Kamal Ndousse , Andy Jones , Sam Bowman , Anna Chen , Tom Conerly , Nova DasSarma , Dawn Drain , Nelson Elhage , Sheer El-Showk , Stanislav Fort , Zac Hatfield-Dodds , Tom Henighan , Danny Hernandez , Tristan Hume , Josh Jacobson , Scott Johnston , Shauna Kravec , Catherine Olsson , Sam Ringer , Eli Tran-Johnson , Dario Amodei , Tom Brown , Nicholas Joseph , Sam McCandlish , Chris Olah , Jared Kaplan , and Jack Clark . 2022. Red Teaming Language Models to Reduce Harms: Methods , Scaling Behaviors, and Lessons Learned. (Aug. 2022 ). arxiv:2209.07858 [cs.CL] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. (Aug. 2022). arxiv:2209.07858 [cs.CL]"},{"key":"e_1_3_2_2_54_1","volume-title":"Retrieved","year":"2020","unstructured":"GapMinder.org. 2020 . Dollar Street . Retrieved January 24, 2023 from https:\/\/www.gapminder.org\/dollar-street GapMinder.org. 2020. Dollar Street. Retrieved January 24, 2023 from https:\/\/www.gapminder.org\/dollar-street"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_3_2_2_56_1","unstructured":"Amelia Glaese Nat McAleese Maja Tr\u0119bacz John Aslanides Vlad Firoiu Timo Ewalds Maribeth Rauh Laura Weidinger Martin Chadwick Phoebe Thacker Lucy Campbell-Gillingham Jonathan Uesato Po-Sen Huang Ramona Comanescu Fan Yang Abigail See Sumanth Dathathri Rory Greig Charlie Chen Doug Fritz Jaume Sanchez Elias Richard Green So\u0148a Mokr\u00e1 Nicholas Fernando Boxi Wu Rachel Foley Susannah Young Iason Gabriel William Isaac John Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements. (Sept. 2022). arxiv:2209.14375 [cs.LG]  Amelia Glaese Nat McAleese Maja Tr\u0119bacz John Aslanides Vlad Firoiu Timo Ewalds Maribeth Rauh Laura Weidinger Martin Chadwick Phoebe Thacker Lucy Campbell-Gillingham Jonathan Uesato Po-Sen Huang Ramona Comanescu Fan Yang Abigail See Sumanth Dathathri Rory Greig Charlie Chen Doug Fritz Jaume Sanchez Elias Richard Green So\u0148a Mokr\u00e1 Nicholas Fernando Boxi Wu Rachel Foley Susannah Young Iason Gabriel William Isaac John Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements. (Sept. 2022). arxiv:2209.14375 [cs.LG]"},{"key":"e_1_3_2_2_57_1","volume-title":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Gokul NC","year":"2022","unstructured":"NC Gokul , Manideep Ladi , Sumit Negi , Prem Selvaraj , Pratyush Kumar , and Mitesh M Khapra . 2022 . Addressing Resource Scarcity across Sign Languages with Multilingual Pretraining and Unified-Vocabulary Datasets . In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. NC Gokul, Manideep Ladi, Sumit Negi, Prem Selvaraj, Pratyush Kumar, and Mitesh M Khapra. 2022. Addressing Resource Scarcity across Sign Languages with Multilingual Pretraining and Unified-Vocabulary Datasets. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_58_1","volume-title":"Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt. https:\/\/books.google.com\/books?id=u10-uQEACAAJ","author":"Gray M.L.","year":"2019","unstructured":"M.L. Gray and S. Suri . 2019 . Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt. https:\/\/books.google.com\/books?id=u10-uQEACAAJ M.L. Gray and S. Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt. https:\/\/books.google.com\/books?id=u10-uQEACAAJ"},{"key":"e_1_3_2_2_59_1","volume-title":"Google engineer apologizes after Photos app tags two black people as gorillas. The Verge","author":"Grush Loren","year":"2015","unstructured":"Loren Grush . 2015. Google engineer apologizes after Photos app tags two black people as gorillas. The Verge ( 2015 ). https:\/\/www.theverge.com\/2015\/7\/1\/8880363\/google-apologizes-photos-app-tags-two-black-people-gorillas. Loren Grush. 2015. Google engineer apologizes after Photos app tags two black people as gorillas. The Verge (2015). https:\/\/www.theverge.com\/2015\/7\/1\/8880363\/google-apologizes-photos-app-tags-two-black-people-gorillas."},{"key":"e_1_3_2_2_60_1","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2562\u20132580","author":"Gururangan Suchin","year":"2022","unstructured":"Suchin Gururangan , Dallas Card , Sarah Dreier , Emily Gade , Leroy Wang , Zeyu Wang , Luke Zettlemoyer , and Noah A. Smith . 2022. Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2562\u20132580 . https:\/\/aclanthology.org\/ 2022 .emnlp-main.165 Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. 2022. Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2562\u20132580. https:\/\/aclanthology.org\/2022.emnlp-main.165"},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445929"},{"key":"e_1_3_2_2_62_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Han Songqiao","year":"2022","unstructured":"Songqiao Han , Xiyang Hu , Hailiang Huang , Mingqi Jiang , and Yue Zhao . 2022 . Adbench: Anomaly detection benchmark . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Songqiao Han, Xiyang Hu, Hailiang Huang, Mingqi Jiang, and Yue Zhao. 2022. Adbench: Anomaly detection benchmark. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372826"},{"key":"e_1_3_2_2_64_1","unstructured":"Nabil Hassein. 2017. Against black inclusion in facial recognition. (2017). https:\/\/digitaltalkingdrum.com\/2017\/08\/15\/against-black-inclusion-in-facial-recognition\/.  Nabil Hassein. 2017. Against black inclusion in facial recognition. (2017). https:\/\/digitaltalkingdrum.com\/2017\/08\/15\/against-black-inclusion-in-facial-recognition\/."},{"key":"e_1_3_2_2_65_1","volume-title":"Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness. (Nov","author":"Hazirbas Caner","year":"2022","unstructured":"Caner Hazirbas , Yejin Bang , Tiezheng Yu , Parisa Assar , Bilal Porgali , V\u00edtor Albiero , Stefan Hermanek , Jacqueline Pan , Emily McReynolds , Miranda Bogen , Pascale Fung , and Cristian Canton Ferrer . 2022. Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness. (Nov . 2022 ). arxiv:2211.05809 [cs.CV] Caner Hazirbas, Yejin Bang, Tiezheng Yu, Parisa Assar, Bilal Porgali, V\u00edtor Albiero, Stefan Hermanek, Jacqueline Pan, Emily McReynolds, Miranda Bogen, Pascale Fung, and Cristian Canton Ferrer. 2022. Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness. (Nov. 2022). arxiv:2211.05809 [cs.CV]"},{"key":"e_1_3_2_2_66_1","volume-title":"Towards measuring fairness in AI: The casual conversations dataset","author":"Hazirbas Caner","year":"2021","unstructured":"Caner Hazirbas , Joanna Bitton , Brian Dolhansky , Jacqueline Pan , Albert Gordo , and Cristian Canton Ferrer . 2021. Towards measuring fairness in AI: The casual conversations dataset . IEEE trans. biom. behav. identity sci. ( 2021 ), 1\u20131. Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo, and Cristian Canton Ferrer. 2021. Towards measuring fairness in AI: The casual conversations dataset. IEEE trans. biom. behav. identity sci. (2021), 1\u20131."},{"key":"e_1_3_2_2_67_1","volume-title":"The Oxford Handbook of Value Theory","author":"Hirose Iwao","year":"1999","unstructured":"Iwao Hirose and Jonas Olson . 2015. The Oxford Handbook of Value Theory . Oxford University Press . https:\/\/doi.org\/10.1093\/oxfordhb\/9780 1999 59303.001.0001 10.1093\/oxfordhb Iwao Hirose and Jonas Olson. 2015. The Oxford Handbook of Value Theory. Oxford University Press. https:\/\/doi.org\/10.1093\/oxfordhb\/9780199959303.001.0001"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1177\/1461444820958725"},{"key":"e_1_3_2_2_69_1","volume-title":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Hormazabal Rodrigo","year":"2022","unstructured":"Rodrigo Hormazabal , Changyoung Park , Soonyoung Lee , Sehui Han , Yeonsik Jo , Jaewan Lee , Ahra Jo , Seung Hwan Kim , Jaegul Choo , Moontae Lee , 2022 . CEDe: A collection of expert-curated datasets with atom-level entity annotations for Optical Chemical Structure Recognition . In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Rodrigo Hormazabal, Changyoung Park, Soonyoung Lee, Sehui Han, Yeonsik Jo, Jaewan Lee, Ahra Jo, Seung Hwan Kim, Jaegul Choo, Moontae Lee, 2022. CEDe: A collection of expert-curated datasets with atom-level entity annotations for Optical Chemical Structure Recognition. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445918"},{"key":"e_1_3_2_2_71_1","volume-title":"Don\u2019t ask if AI is good or fair, ask how it shifts power. Nature","author":"Kalluri Pratyusha","year":"2020","unstructured":"Pratyusha Kalluri . 2020. Don\u2019t ask if AI is good or fair, ask how it shifts power. Nature ( 2020 ). Pratyusha Kalluri. 2020. Don\u2019t ask if AI is good or fair, ask how it shifts power. Nature (2020)."},{"key":"e_1_3_2_2_72_1","volume-title":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems","author":"Kay Matthew","unstructured":"Matthew Kay , Cynthia Matuszek , and Sean A. Munson . 2015. Unequal Representation and Gender Stereotypes in Image Search Results for Occupations . In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems ( Seoul, Republic of Korea) (CHI \u201915). Association for Computing Machinery, New York, NY, USA, 3819\u20133828. https:\/\/doi.org\/10.1145\/2702123.2702520 10.1145\/2702123.2702520 Matthew Kay, Cynthia Matuszek, and Sean A. Munson. 2015. Unequal Representation and Gender Stereotypes in Image Search Results for Occupations. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI \u201915). Association for Computing Machinery, New York, NY, USA, 3819\u20133828. https:\/\/doi.org\/10.1145\/2702123.2702520"},{"key":"e_1_3_2_2_73_1","unstructured":"Os Keyes. 2019. Counting the Countless: Why data science is a profound threat for queer people. https:\/\/reallifemag.com\/counting-the-countless\/  Os Keyes. 2019. Counting the Countless: Why data science is a profound threat for queer people. https:\/\/reallifemag.com\/counting-the-countless\/"},{"key":"e_1_3_2_2_74_1","volume-title":"How Transferable are Video Representations Based on Synthetic Data?Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Mishra Samarth","year":"2022","unstructured":"Yo-whan Kim, Samarth Mishra , SouYoung Jin , Rameswar Panda , Hilde Kuehne , Leonid Karlinsky , Venkatesh Saligrama , Kate Saenko , Aude Oliva , and Rogerio Feris . 2022. How Transferable are Video Representations Based on Synthetic Data?Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks ( 2022 ). Yo-whan Kim, Samarth Mishra, SouYoung Jin, Rameswar Panda, Hilde Kuehne, Leonid Karlinsky, Venkatesh Saligrama, Kate Saenko, Aude Oliva, and Rogerio Feris. 2022. How Transferable are Video Representations Based on Synthetic Data?Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_75_1","volume-title":"Retrieved","author":"Kirka Danica","year":"2020","unstructured":"Danica Kirka . 2020 . UK court says face recognition violates human rights . Retrieved January 24, 2023 from https:\/\/techxplore.com\/news\/2020-08-uk-court-recognition-violates-human.html Danica Kirka. 2020. UK court says face recognition violates human rights. Retrieved January 24, 2023 from https:\/\/techxplore.com\/news\/2020-08-uk-court-recognition-violates-human.html"},{"key":"e_1_3_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533114"},{"key":"e_1_3_2_2_77_1","volume-title":"Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Korotin Alexander","year":"2022","unstructured":"Alexander Korotin , Alexander Kolesov , and Evgeny Burnaev . 2022. Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks ( 2022 ). Alexander Korotin, Alexander Kolesov, and Evgeny Burnaev. 2022. Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/TTS.2020.2974996"},{"key":"e_1_3_2_2_79_1","volume-title":"Proceedings of the Twelfth Language Resources and Evaluation Conference. European Language Resources Association","author":"Lai Yi-An","year":"2020","unstructured":"Yi-An Lai , Xuan Zhu , Yi Zhang , and Mona Diab . 2020 . Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections . In Proceedings of the Twelfth Language Resources and Evaluation Conference. European Language Resources Association , Marseille, France, 1739\u20131746. https:\/\/aclanthology.org\/ 2020.lrec-1.215 Yi-An Lai, Xuan Zhu, Yi Zhang, and Mona Diab. 2020. Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections. In Proceedings of the Twelfth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 1739\u20131746. https:\/\/aclanthology.org\/2020.lrec-1.215"},{"key":"e_1_3_2_2_80_1","volume-title":"AI-Driven Dermatology Could Leave Dark-Skinned Patients Behind. The Atlantic","author":"Lashbrook Angela","year":"2018","unstructured":"Angela Lashbrook . 2018. AI-Driven Dermatology Could Leave Dark-Skinned Patients Behind. The Atlantic ( 2018 ). https:\/\/www.theatlantic.com\/health\/archive\/2018\/08\/machine-learning-dermatology-skin-color\/567619\/. Angela Lashbrook. 2018. AI-Driven Dermatology Could Leave Dark-Skinned Patients Behind. The Atlantic (2018). https:\/\/www.theatlantic.com\/health\/archive\/2018\/08\/machine-learning-dermatology-skin-color\/567619\/."},{"key":"e_1_3_2_2_81_1","unstructured":"Hugo Lauren\u00e7on Lucile Saulnier Thomas Wang Christopher Akiki Albert Villanova del Moral Teven Le Scao Leandro Von Werra Chenghao Mou Eduardo Gonz\u00e1lez Ponferrada Huu Nguyen J\u00f6rg Frohberg Mario \u0160a\u0161ko Quentin Lhoest Angelina McMillan-Major G\u00e9rard Dupont Stella Biderman Anna Rogers Loubna Ben allal Francesco De Toni Giada Pistilli Olivier Nguyen Somaieh Nikpoor Maraim Masoud Pierre Colombo Javier de la Rosa Paulo Villegas Tristan Thrush Shayne Longpre Sebastian Nagel Leon Weber Manuel Romero Mu\u00f1oz Jian Zhu Daniel Van Strien Zaid Alyafeai Khalid Almubarak Vu Minh Chien Itziar Gonzalez-Dios Aitor Soroa Kyle Lo Manan Dey Pedro Ortiz Suarez Aaron Gokaslan Shamik Bose David Ifeoluwa Adelani Long Phan Hieu Tran Ian Yu Suhas Pai Jenny Chim Violette Lepercq Suzana Ilic Margaret Mitchell Sasha Luccioni and Yacine Jernite. 2022. The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https:\/\/openreview.net\/forum?id=UoEw6KigkUn  Hugo Lauren\u00e7on Lucile Saulnier Thomas Wang Christopher Akiki Albert Villanova del Moral Teven Le Scao Leandro Von Werra Chenghao Mou Eduardo Gonz\u00e1lez Ponferrada Huu Nguyen J\u00f6rg Frohberg Mario \u0160a\u0161ko Quentin Lhoest Angelina McMillan-Major G\u00e9rard Dupont Stella Biderman Anna Rogers Loubna Ben allal Francesco De Toni Giada Pistilli Olivier Nguyen Somaieh Nikpoor Maraim Masoud Pierre Colombo Javier de la Rosa Paulo Villegas Tristan Thrush Shayne Longpre Sebastian Nagel Leon Weber Manuel Romero Mu\u00f1oz Jian Zhu Daniel Van Strien Zaid Alyafeai Khalid Almubarak Vu Minh Chien Itziar Gonzalez-Dios Aitor Soroa Kyle Lo Manan Dey Pedro Ortiz Suarez Aaron Gokaslan Shamik Bose David Ifeoluwa Adelani Long Phan Hieu Tran Ian Yu Suhas Pai Jenny Chim Violette Lepercq Suzana Ilic Margaret Mitchell Sasha Luccioni and Yacine Jernite. 2022. The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https:\/\/openreview.net\/forum?id=UoEw6KigkUn"},{"key":"e_1_3_2_2_82_1","first-page":"29348","article-title":"Mind the gap: Assessing temporal generalization in neural language models","volume":"34","author":"Lazaridou Angeliki","year":"2021","unstructured":"Angeliki Lazaridou , Adhi Kuncoro , Elena Gribovskaya , Devang Agrawal , Adam Liska , Tayfun Terzi , Mai Gimenez , Cyprien de Masson d\u2019 Autume , Tomas Kocisky , Sebastian Ruder , 2021 . Mind the gap: Assessing temporal generalization in neural language models . Advances in Neural Information Processing Systems 34 (2021), 29348 \u2013 29363 . Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d\u2019Autume, Tomas Kocisky, Sebastian Ruder, 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems 34 (2021), 29348\u201329363.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.818"},{"key":"e_1_3_2_2_84_1","volume-title":"Fluctuating Identities: Re-Learning Human Categorization. In 2022 ACM Conference on Fairness, Accountability, and Transparency","author":"Lu Christina","year":"2022","unstructured":"Christina Lu , Jackie Kay , and Kevin McKee . 2022 . Subverting Machines , Fluctuating Identities: Re-Learning Human Categorization. In 2022 ACM Conference on Fairness, Accountability, and Transparency ( Seoul, Republic of Korea) (FAccT \u201922). Association for Computing Machinery, New York, NY, USA, 1005\u20131015. https:\/\/doi.org\/10.1145\/3531146.3533161 10.1145\/3531146.3533161 Christina Lu, Jackie Kay, and Kevin McKee. 2022. Subverting Machines, Fluctuating Identities: Re-Learning Human Categorization. In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, Republic of Korea) (FAccT \u201922). Association for Computing Machinery, New York, NY, USA, 1005\u20131015. https:\/\/doi.org\/10.1145\/3531146.3533161"},{"key":"e_1_3_2_2_85_1","volume-title":"To predict and serve?Signif. (Oxf.) 13, 5 (Oct","author":"Lum Kristian","year":"2016","unstructured":"Kristian Lum and William Isaac . 2016. To predict and serve?Signif. (Oxf.) 13, 5 (Oct . 2016 ), 14\u201319. Kristian Lum and William Isaac. 2016. To predict and serve?Signif. (Oxf.) 13, 5 (Oct. 2016), 14\u201319."},{"key":"e_1_3_2_2_86_1","volume-title":"MOMA-LRG: Language-Refined Graphs for Multi-Object Multi-Actor Activity Parsing. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Luo Zelun","year":"2022","unstructured":"Zelun Luo , Zane Durante , Linden Li , Wanze Xie , Ruochen Liu , Emily Jin , Zhuoyi Huang , Lun Yu Li , Jiajun Wu , Juan Carlos Niebles , 2022 . MOMA-LRG: Language-Refined Graphs for Multi-Object Multi-Actor Activity Parsing. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Zelun Luo, Zane Durante, Linden Li, Wanze Xie, Ruochen Liu, Emily Jin, Zhuoyi Huang, Lun Yu Li, Jiajun Wu, Juan Carlos Niebles, 2022. MOMA-LRG: Language-Refined Graphs for Multi-Object Multi-Actor Activity Parsing. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_87_1","volume-title":"Facebook says its AI mislabeling a video of Black men as \"primates\" was \"unacceptable\". The Verge","author":"Lyons Kim","year":"2021","unstructured":"Kim Lyons . 2021. Facebook says its AI mislabeling a video of Black men as \"primates\" was \"unacceptable\". The Verge ( 2021 ). https:\/\/www.theverge.com\/2021\/9\/4\/22657026\/facebook-mislabeling-video-black-men-primates-algorithm. Kim Lyons. 2021. Facebook says its AI mislabeling a video of Black men as \"primates\" was \"unacceptable\". The Verge (2021). https:\/\/www.theverge.com\/2021\/9\/4\/22657026\/facebook-mislabeling-video-black-men-primates-algorithm."},{"key":"e_1_3_2_2_88_1","volume-title":"Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics. ICLR Machine Learning In Real Life (ML-IRL) Workshop, Addis Ababa, Ethiopia.","author":"Martin Donald","year":"2020","unstructured":"Donald Martin , Jr, Vinodkumar Prabhakaran , Jill Kuhlberg , Andrew Smart , and William S Isaac . 2020 . Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics. ICLR Machine Learning In Real Life (ML-IRL) Workshop, Addis Ababa, Ethiopia. Donald Martin, Jr, Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S Isaac. 2020. Participatory Problem Formulation for Fairer Machine Learning Through Community Based System Dynamics. ICLR Machine Learning In Real Life (ML-IRL) Workshop, Addis Ababa, Ethiopia."},{"key":"e_1_3_2_2_89_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Mazeika Mantas","year":"2022","unstructured":"Mantas Mazeika , Eric Tang , Andy Zou , Steven Basart , Jun Shern Chan , Dawn Song , David Forsyth , Jacob Steinhardt , and Dan Hendrycks . 2022 . How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Mantas Mazeika, Eric Tang, Andy Zou, Steven Basart, Jun Shern Chan, Dawn Song, David Forsyth, Jacob Steinhardt, and Dan Hendrycks. 2022. How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_90_1","volume-title":"Retrieved","year":"2019","unstructured":"MediaWell. 2019 . \u201c Please do not include us\": Workshop on AI Ethics and Inclusion . Retrieved January 19, 2023 from https:\/\/mediawell.ssrc.org\/event\/please-do-not-include-us-workshop-on-ai-ethics-and-inclusion\/ MediaWell. 2019. \u201cPlease do not include us\": Workshop on AI Ethics and Inclusion. Retrieved January 19, 2023 from https:\/\/mediawell.ssrc.org\/event\/please-do-not-include-us-workshop-on-ai-ethics-and-inclusion\/"},{"key":"e_1_3_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555623"},{"key":"e_1_3_2_2_93_1","volume-title":"The Emotion Machine","author":"Minsky Marvin","unstructured":"Marvin Minsky . 2006. The Emotion Machine . Simon & Schuster , New York . Marvin Minsky. 2006. The Emotion Machine. Simon & Schuster, New York."},{"key":"e_1_3_2_2_94_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Mishra Shubhanshu","year":"2022","unstructured":"Shubhanshu Mishra , Aman Saini , Raheleh Makki , Sneha Mehta , Aria Haghighi , and Ali Mollahosseini . 2022 . TweetNERD\u2013End to End Entity Linking Benchmark for Tweets . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Shubhanshu Mishra, Aman Saini, Raheleh Makki, Sneha Mehta, Aria Haghighi, and Ali Mollahosseini. 2022. TweetNERD\u2013End to End Entity Linking Benchmark for Tweets. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375627.3375832"},{"key":"e_1_3_2_2_96_1","volume-title":"Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela.","author":"Mitchell Margaret","year":"2022","unstructured":"Margaret Mitchell , Alexandra Sasha Luccioni , Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela. 2022 . Measuring Data . (Dec. 2022). arxiv:2212.05129 [cs.AI] Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela. 2022. Measuring Data. (Dec. 2022). arxiv:2212.05129 [cs.AI]"},{"key":"e_1_3_2_2_97_1","volume-title":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Moayeri Mazda","year":"2022","unstructured":"Mazda Moayeri , Sahil Singla , and Soheil Feizi . 2022 . Hard imagenet: Segmentations for objects with strong spurious cues . In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Mazda Moayeri, Sahil Singla, and Soheil Feizi. 2022. Hard imagenet: Segmentations for objects with strong spurious cues. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_98_1","volume-title":"2nd Draft. https:\/\/www.nist.gov\/itl\/ai-risk-management-framework.","author":"National Institute of Standards and Technology (NIST). 2022.","unstructured":"National Institute of Standards and Technology (NIST). 2022. AI Risk Management Framework , 2nd Draft. https:\/\/www.nist.gov\/itl\/ai-risk-management-framework. National Institute of Standards and Technology (NIST). 2022. AI Risk Management Framework, 2nd Draft. https:\/\/www.nist.gov\/itl\/ai-risk-management-framework."},{"key":"e_1_3_2_2_99_1","volume-title":"Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. arXiv preprint arXiv:1907.11274","author":"Ovadya Aviv","year":"2019","unstructured":"Aviv Ovadya and Jess Whittlestone . 2019. Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. arXiv preprint arXiv:1907.11274 ( 2019 ). Aviv Ovadya and Jess Whittlestone. 2019. Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning. arXiv preprint arXiv:1907.11274 (2019)."},{"key":"e_1_3_2_2_100_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2005.69"},{"key":"e_1_3_2_2_101_1","unstructured":"Partnership on AI. 2020. Publication Norms for Responsible AI: Ongoing Initiative.  Partnership on AI. 2020. Publication Norms for Responsible AI: Ongoing Initiative."},{"key":"e_1_3_2_2_102_1","unstructured":"Partnership on AI. 2021. Managing the Risks of AI Research: Six Recommendations for Responsible Publication.  Partnership on AI. 2021. Managing the Risks of AI Research: Six Recommendations for Responsible Publication."},{"key":"e_1_3_2_2_103_1","unstructured":"Partnership on AI. 2021. Responsible Sourcing of Data Enrichment Services.  Partnership on AI. 2021. Responsible Sourcing of Data Enrichment Services."},{"key":"e_1_3_2_2_104_1","volume-title":"You\u2019re Not Wrong. The New York Times","author":"Pasquale Frank","year":"2021","unstructured":"Frank Pasquale and Gianclaudio Malgieri . 2021. If You Don\u2019t Trust A.I. Yet , You\u2019re Not Wrong. The New York Times ( 2021 ). https:\/\/www.nytimes.com\/2021\/07\/30\/opinion\/artificial-intelligence-european-union.html. Frank Pasquale and Gianclaudio Malgieri. 2021. If You Don\u2019t Trust A.I. Yet, You\u2019re Not Wrong. The New York Times (2021). https:\/\/www.nytimes.com\/2021\/07\/30\/opinion\/artificial-intelligence-european-union.html."},{"key":"e_1_3_2_2_105_1","volume-title":"Emily M Bender, Emily Denton, and Alex Hanna.","author":"Paullada Amandalynne","year":"2020","unstructured":"Amandalynne Paullada , Inioluwa Deborah Raji , Emily M Bender, Emily Denton, and Alex Hanna. 2020 . Data and its (dis)contents: A survey of dataset development and use in machine learning research. (Dec. 2020). arxiv:2012.05345 [cs.LG] Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2020. Data and its (dis)contents: A survey of dataset development and use in machine learning research. (Dec. 2020). arxiv:2012.05345 [cs.LG]"},{"key":"e_1_3_2_2_106_1","volume-title":"Varieties of public representation","author":"Pettit Philip","year":"1813","unstructured":"Philip Pettit . 2010. Varieties of public representation . Cambridge University Press , 61\u201389. https:\/\/doi.org\/10.1017\/CBO978051 1813 146.005 10.1017\/CBO9780511813146.005 Philip Pettit. 2010. Varieties of public representation. Cambridge University Press, 61\u201389. https:\/\/doi.org\/10.1017\/CBO9780511813146.005"},{"key":"e_1_3_2_2_107_1","volume-title":"The Concept of Representation","author":"Pitkin Hanna Fenichel","year":"2034","unstructured":"Hanna Fenichel Pitkin . 1967. The Concept of Representation . University of California Press , Berkeley. https:\/\/doi.org\/doi:10.1525\/97805 2034 0503 10.1525\/9780520340503 Hanna Fenichel Pitkin. 1967. The Concept of Representation. University of California Press, Berkeley. https:\/\/doi.org\/doi:10.1525\/9780520340503"},{"key":"e_1_3_2_2_108_1","unstructured":"Julian Posada. 2021. Family Units. https:\/\/logicmag.io\/beacons\/family-units\/. Accessed: 2022-11-28.  Julian Posada. 2021. Family Units. https:\/\/logicmag.io\/beacons\/family-units\/. Accessed: 2022-11-28."},{"key":"e_1_3_2_2_109_1","first-page":"2","article-title":"Participatory Machine Learning Using Community-Based System Dynamics","volume":"22","author":"Prabhakaran Vinodkumar","year":"2020","unstructured":"Vinodkumar Prabhakaran and Donald Martin , Jr. 2020 . Participatory Machine Learning Using Community-Based System Dynamics . Health Hum. Rights 22 , 2 (Dec. 2020), 71\u201374. Vinodkumar Prabhakaran and Donald Martin, Jr. 2020. Participatory Machine Learning Using Community-Based System Dynamics. Health Hum. Rights 22, 2 (Dec. 2020), 71\u201374.","journal-title":"Health Hum. Rights"},{"key":"e_1_3_2_2_110_1","volume-title":"Institutionalizing ethics in AI through broader impact requirements. Nature Machine Intelligence","author":"Prunkl Carina","year":"2021","unstructured":"Carina Prunkl , Carolyn Ashurst , Markus Anderljung , Helena Webb , Jan Leike , and Allan Dafoe . 2021. Institutionalizing ethics in AI through broader impact requirements. Nature Machine Intelligence ( 2021 ). Carina Prunkl, Carolyn Ashurst, Markus Anderljung, Helena Webb, Jan Leike, and Allan Dafoe. 2021. Institutionalizing ethics in AI through broader impact requirements. Nature Machine Intelligence (2021)."},{"key":"e_1_3_2_2_111_1","volume-title":"Feminist Theory Reader","author":"Puar Jasbir K","unstructured":"Jasbir K Puar . 2020. \u201c I would rather be a cyborg than a goddess\u201d: Becoming-intersectional in assemblage theory . In Feminist Theory Reader . Routledge , 405\u2013415. Jasbir K Puar. 2020. \u201cI would rather be a cyborg than a goddess\u201d: Becoming-intersectional in assemblage theory. In Feminist Theory Reader. Routledge, 405\u2013415."},{"key":"e_1_3_2_2_112_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Qin Rongjun","year":"2022","unstructured":"Rongjun Qin , Songyi Gao , Xingyuan Zhang , Zhen Xu , Shengkai Huang , Zewen Li , Weinan Zhang , and Yang Yu . 2022 . NeoRL: A near real-world benchmark for offline reinforcement learning . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu. 2022. NeoRL: A near real-world benchmark for offline reinforcement learning. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_113_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Qin Yijian","year":"2022","unstructured":"Yijian Qin , Ziwei Zhang , Xin Wang , Zeyang Zhang , and Wenwu Zhu . 2022 . NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Yijian Qin, Ziwei Zhang, Xin Wang, Zeyang Zhang, and Wenwu Zhu. 2022. NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_114_1","unstructured":"Stephanie Carroll Rainie. 2019. Indigenous Data Sovereignty. In The State of Open Data: Histories and Horizons. Cape Town and Ottawa: African Minds and International Development Research Centre. https:\/\/www.stateofopendata.od4d.net\/chapters\/issues\/indigenous-data.html  Stephanie Carroll Rainie. 2019. Indigenous Data Sovereignty. In The State of Open Data: Histories and Horizons. Cape Town and Ottawa: African Minds and International Development Research Centre. https:\/\/www.stateofopendata.od4d.net\/chapters\/issues\/indigenous-data.html"},{"key":"e_1_3_2_2_115_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.).","volume":"1","author":"Raji Deborah","year":"2021","unstructured":"Deborah Raji , Emily Denton , Emily M. Bender , Alex Hanna , and Amandalynne Paullada . 2021 . AI and the Everything in the Whole Wide World Benchmark . In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.). Vol. 1 . https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/file\/084b6fbb10729ed4da8c3d3f5a3ae7c9-Paper-round2.pdf Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. AI and the Everything in the Whole Wide World Benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.). Vol. 1. https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/file\/084b6fbb10729ed4da8c3d3f5a3ae7c9-Paper-round2.pdf"},{"key":"e_1_3_2_2_116_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_3_2_2_117_1","volume-title":"Dora Zhao, Aaron B Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky.","author":"Ramaswamy Vikram V","year":"2023","unstructured":"Vikram V Ramaswamy , Sing Yu Lin , Dora Zhao, Aaron B Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. 2023 . Beyond web-scraping: Crowd-sourcing a geographically diverse image dataset. arXiv preprint arXiv:2301.02560 (2023). Vikram V Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron B Adcock, Laurens van der Maaten, Deepti Ghadiyaram, and Olga Russakovsky. 2023. Beyond web-scraping: Crowd-sourcing a geographically diverse image dataset. arXiv preprint arXiv:2301.02560 (2023)."},{"key":"e_1_3_2_2_118_1","volume-title":"Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Rauh Maribeth","year":"2022","unstructured":"Maribeth Rauh , John Mellor , Jonathan Uesato , Po-Sen Huang , Johannes Welbl , Laura Weidinger , Sumanth Dathathri , Amelia Glaese , Geoffrey Irving , Iason Gabriel , William Isaac , and Lisa Anne Hendricks . 2022 . Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Maribeth Rauh, John Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, Iason Gabriel, William Isaac, and Lisa Anne Hendricks. 2022. Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_119_1","volume-title":"Retrieved","author":"Reference The Oxford","year":"2023","unstructured":"The Oxford Reference . 2023 . Overview: representation . Retrieved January 23, 2023 from https:\/\/www.oxfordreference.com\/display\/10.1093\/oi\/authority.20111014165925770;jsessionid=8033B4F2345255BA0941E2E13889BD54 The Oxford Reference. 2023. Overview: representation. Retrieved January 23, 2023 from https:\/\/www.oxfordreference.com\/display\/10.1093\/oi\/authority.20111014165925770;jsessionid=8033B4F2345255BA0941E2E13889BD54"},{"key":"e_1_3_2_2_120_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"key":"e_1_3_2_2_121_1","volume-title":"David Kanter, Vijay Janapa Reddi, and Cody Coleman.","author":"Gaviria Rojas William A","year":"2022","unstructured":"William A Gaviria Rojas , Sudnya Diamos , Keertan Ranjan Kini , David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022 . The Dollar Street Dataset: Images Representing the Geographic and Socioeconomic Diversity of the World. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track . 201\u2013203. William A Gaviria Rojas, Sudnya Diamos, Keertan Ranjan Kini, David Kanter, Vijay Janapa Reddi, and Cody Coleman. 2022. The Dollar Street Dataset: Images Representing the Geographic and Socioeconomic Diversity of the World. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 201\u2013203."},{"key":"e_1_3_2_2_122_1","volume-title":"Proceedings of the 38th International Conference on Machine Learning, ICML 2021","volume":"9051","author":"Rolf Esther","year":"2021","unstructured":"Esther Rolf , Theodora T. Worledge , Benjamin Recht , and Michael I. Jordan . 2021. Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data . In Proceedings of the 38th International Conference on Machine Learning, ICML 2021 , 18-24 July 2021 , Virtual Event(Proceedings of Machine Learning Research , Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 9040\u2013 9051 . Esther Rolf, Theodora T. Worledge, Benjamin Recht, and Michael I. Jordan. 2021. Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 9040\u20139051."},{"key":"e_1_3_2_2_123_1","first-page":"4","article-title":"Seeing like a dataset from the global south","volume":"28","author":"Sambasivan Nithya","year":"2021","unstructured":"Nithya Sambasivan . 2021 . Seeing like a dataset from the global south . Interactions 28 , 4 (July 2021), 76\u201378. Nithya Sambasivan. 2021. Seeing like a dataset from the global south. Interactions 28, 4 (July 2021), 76\u201378.","journal-title":"Interactions"},{"key":"e_1_3_2_2_124_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445896"},{"key":"e_1_3_2_2_125_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445518"},{"key":"e_1_3_2_2_126_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Sanders Kate","year":"2022","unstructured":"Kate Sanders , Reno Kriz , Anqi Liu , and Benjamin Van Durme . 2022 . Ambiguous Images With Human Judgments for Robust Visual Event Classification . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Kate Sanders, Reno Kriz, Anqi Liu, and Benjamin Van Durme. 2022. Ambiguous Images With Human Judgments for Robust Visual Event Classification. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_127_1","volume-title":"Proc. ACM Hum. Comput. Interact. 5, CSCW2","author":"Scheuerman Morgan Klaus","year":"2021","unstructured":"Morgan Klaus Scheuerman , Alex Hanna , and Emily Denton . 2021 . Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development . Proc. ACM Hum. Comput. Interact. 5, CSCW2 (2021), 317:1\u2013317:37. https:\/\/doi.org\/10.1145\/3476058 10.1145\/3476058 Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021. Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development. Proc. ACM Hum. Comput. Interact. 5, CSCW2 (2021), 317:1\u2013317:37. https:\/\/doi.org\/10.1145\/3476058"},{"key":"e_1_3_2_2_128_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Schmarje Lars","year":"2022","unstructured":"Lars Schmarje , Vasco Grossmann , Claudius Zelenka , Sabine Dippel , Rainer Kiko , Mariusz Oszust , Matti Pastell , Jenny Stracke , Anna Valros , Nina Volkmann , 2022 . Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, Rainer Kiko, Mariusz Oszust, Matti Pastell, Jenny Stracke, Anna Valros, Nina Volkmann, 2022. Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_129_1","unstructured":"Jessica Schrouff Natalie Harris Oluwasanmi O Koyejo Ibrahim Alabdulmohsin Eva Schnider Krista Opsahl-Ong Alexander Brown Subhrajit Roy Diana Mincu Chrsitina Chen Awa Dieng Yuan Liu Vivek Natarajan Alan Karthikesalingam Katherine A Heller Silvia Chiappa and Alexander D\u2019Amour. 2022. Diagnosing failures of fairness transfer across distribution shift in real-world medical settings. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.).  Jessica Schrouff Natalie Harris Oluwasanmi O Koyejo Ibrahim Alabdulmohsin Eva Schnider Krista Opsahl-Ong Alexander Brown Subhrajit Roy Diana Mincu Chrsitina Chen Awa Dieng Yuan Liu Vivek Natarajan Alan Karthikesalingam Katherine A Heller Silvia Chiappa and Alexander D\u2019Amour. 2022. Diagnosing failures of fairness transfer across distribution shift in real-world medical settings. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.)."},{"key":"e_1_3_2_2_130_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Schuhmann Christoph","year":"2022","unstructured":"Christoph Schuhmann , Romain Beaumont , Richard Vencu , Cade Gordon , Ross Wightman , Mehdi Cherti , Theo Coombes , Aarush Katta , Clayton Mullis , Mitchell Wortsman , 2022 . Laion-5b: An open large-scale dataset for training next generation image-text models . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_131_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_3_2_2_132_1","volume-title":"NIPS 2017: Workshop on Machine Learning for the Developing World.","author":"Shankar Shreya","year":"2017","unstructured":"Shreya Shankar , Yoni Halpern , Eric Breck , James Atwood , Jimbo Wilson , and D Sculley . 2017 . No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World . NIPS 2017: Workshop on Machine Learning for the Developing World. Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. 2017. No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World. NIPS 2017: Workshop on Machine Learning for the Developing World."},{"key":"e_1_3_2_2_133_1","volume-title":"Emilio Garcia, and Gurleen Virk.","author":"Shelby Renee","year":"2022","unstructured":"Renee Shelby , Shalaleh Rismani , Kathryn Henne , Ajung Moon , Negar Rostamzadeh , Paul Nicholas , N\u2019mah Yilla , Jess Gallegos , Andrew Smart , Emilio Garcia, and Gurleen Virk. 2022 . Sociotechnical Harms : Scoping a Taxonomy for Harm Reduction . (Oct. 2022). arxiv:2210.05791 [cs.HC] Renee Shelby, Shalaleh Rismani, Kathryn Henne, Ajung Moon, Negar Rostamzadeh, Paul Nicholas, N\u2019mah Yilla, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. 2022. Sociotechnical Harms: Scoping a Taxonomy for Harm Reduction. (Oct. 2022). arxiv:2210.05791 [cs.HC]"},{"key":"e_1_3_2_2_134_1","volume-title":"Equity and Access in Algorithms, Mechanisms, and Optimization (\u2013","author":"Suresh Harini","unstructured":"Harini Suresh and John Guttag . 2021. A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle . In Equity and Access in Algorithms, Mechanisms, and Optimization (\u2013 , NY , USA) (EAAMO \u201921). Association for Computing Machinery , New York, NY, USA, Article 17, 9 pages. https:\/\/doi.org\/10.1145\/3465416.3483305 10.1145\/3465416.3483305 Harini Suresh and John Guttag. 2021. A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. In Equity and Access in Algorithms, Mechanisms, and Optimization (\u2013, NY, USA) (EAAMO \u201921). Association for Computing Machinery, New York, NY, USA, Article 17, 9 pages. https:\/\/doi.org\/10.1145\/3465416.3483305"},{"key":"e_1_3_2_2_135_1","volume-title":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Tamkin Alex","year":"2022","unstructured":"Alex Tamkin , Gaurab Banerjee , Mohamed Owda , Vincent Liu , Shashank Rammoorthy , and Noah Goodman . 2022 . DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision . In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Alex Tamkin, Gaurab Banerjee, Mohamed Owda, Vincent Liu, Shashank Rammoorthy, and Noah Goodman. 2022. DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_136_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.15"},{"key":"e_1_3_2_2_137_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462540"},{"key":"e_1_3_2_2_138_1","unstructured":"Shari Trewin. 2018. AI Fairness for People With Disabilities: Point of View.  Shari Trewin. 2018. AI Fairness for People With Disabilities: Point of View."},{"key":"e_1_3_2_2_139_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Tu Renbo","year":"2022","unstructured":"Renbo Tu , Nicholas Roberts , Mikhail Khodak , Junhong Shen , Frederic Sala , and Ameet Talwalkar . 2022 . NAS-bench-360: Benchmarking neural architecture search on diverse tasks . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar. 2022. NAS-bench-360: Benchmarking neural architecture search on diverse tasks. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_140_1","volume-title":"Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Ullah Ihsan","year":"2022","unstructured":"Ihsan Ullah , Dustin Carri\u00f3n-Ojeda , Sergio Escalera , Isabelle M Guyon , Mike Huisman , Felix Mohr , Jan N van Rijn , Haozhe Sun , Joaquin Vanschoren , and Phan Anh Vu . 2022 . Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Ihsan Ullah, Dustin Carri\u00f3n-Ojeda, Sergio Escalera, Isabelle M Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. 2022. Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_141_1","volume-title":"J\u00falia Corbera Serraj\u00f2rdia, and Aneta Swianiewicz","author":"Valdivia Ana","year":"2022","unstructured":"Ana Valdivia , J\u00falia Corbera Serraj\u00f2rdia, and Aneta Swianiewicz . 2022 . There is an elephant in the room: towards a critique on the use of fairness in biometrics. AI and Ethics (Dec . 2022). Ana Valdivia, J\u00falia Corbera Serraj\u00f2rdia, and Aneta Swianiewicz. 2022. There is an elephant in the room: towards a critique on the use of fairness in biometrics. AI and Ethics (Dec. 2022)."},{"key":"e_1_3_2_2_142_1","volume-title":"REVISE: A tool for measuring and mitigating bias in visual datasets. International Journal of Computer Vision","author":"Wang Angelina","year":"2022","unstructured":"Angelina Wang , Alexander Liu , Ryan Zhang , Anat Kleiman , Leslie Kim , Dora Zhao , Iroha Shirai , Arvind Narayanan , and Olga Russakovsky . 2022 . REVISE: A tool for measuring and mitigating bias in visual datasets. International Journal of Computer Vision (2022), 1\u201321. Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky. 2022. REVISE: A tool for measuring and mitigating bias in visual datasets. International Journal of Computer Vision (2022), 1\u201321."},{"key":"e_1_3_2_2_143_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533101"},{"key":"e_1_3_2_2_144_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00541"},{"key":"e_1_3_2_2_145_1","volume-title":"Finding Naturally Occurring Physical Backdoors in Image Datasets. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Wenger Emily","year":"2022","unstructured":"Emily Wenger , Roma Bhattacharjee , Arjun Nitin Bhagoji , Josephine Passananti , Emilio Andere , Haitao Zheng , and Ben Zhao . 2022 . Finding Naturally Occurring Physical Backdoors in Image Datasets. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Emily Wenger, Roma Bhattacharjee, Arjun Nitin Bhagoji, Josephine Passananti, Emilio Andere, Haitao Zheng, and Ben Zhao. 2022. Finding Naturally Occurring Physical Backdoors in Image Datasets. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_146_1","volume-title":"Comparison of MUTCD-influenced traffic signs. Retrieved","year":"2023","unstructured":"Wikipedia. 2023. Comparison of MUTCD-influenced traffic signs. Retrieved February 6, 2023 from https:\/\/en.wikipedia.org\/wiki\/Comparison_of_MUTCD-influenced_traffic_signs Wikipedia. 2023. Comparison of MUTCD-influenced traffic signs. Retrieved February 6, 2023 from https:\/\/en.wikipedia.org\/wiki\/Comparison_of_MUTCD-influenced_traffic_signs"},{"key":"e_1_3_2_2_147_1","volume-title":"Retrieved","year":"2023","unstructured":"Wiktionary. 2023 . Represent . Retrieved January 23, 2023 from https:\/\/en.wiktionary.org\/wiki\/represent Wiktionary. 2023. Represent. Retrieved January 23, 2023 from https:\/\/en.wiktionary.org\/wiki\/represent"},{"key":"e_1_3_2_2_148_1","volume-title":"The exploited labor behind artificial intelligence. (Oct","author":"Williams Adrienne","year":"2022","unstructured":"Adrienne Williams . 2022. The exploited labor behind artificial intelligence. (Oct . 2022 ). Adrienne Williams. 2022. The exploited labor behind artificial intelligence. (Oct. 2022)."},{"key":"e_1_3_2_2_149_1","first-page":"3","article-title":"Fairness, respect and the egalitarian ethos revisited","volume":"14","author":"Wolff Jonathan","year":"2010","unstructured":"Jonathan Wolff . 2010 . Fairness, respect and the egalitarian ethos revisited . J. Ethics 14 , 3 - 4 (Dec. 2010), 335\u2013350. Jonathan Wolff. 2010. Fairness, respect and the egalitarian ethos revisited. J. Ethics 14, 3-4 (Dec. 2010), 335\u2013350.","journal-title":"J. Ethics"},{"key":"e_1_3_2_2_150_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Xu Minghao","year":"2022","unstructured":"Minghao Xu , Zuobai Zhang , Jiarui Lu , Zhaocheng Zhu , Yangtian Zhang , Chang Ma , Runcheng Liu , and Jian Tang . 2022 . Peer: a comprehensive and multi-task benchmark for protein sequence understanding . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Chang Ma, Runcheng Liu, and Jian Tang. 2022. Peer: a comprehensive and multi-task benchmark for protein sequence understanding. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_151_1","volume-title":"Pang Wei Koh, and Chelsea Finn","author":"Yao Huaxiu","year":"2022","unstructured":"Huaxiu Yao , Caroline Choi , Bochuan Cao , Yoonho Lee , Pang Wei Koh, and Chelsea Finn . 2022 . Wild-time : A benchmark of in-the-wild distribution shift over time. arXiv preprint arXiv:2211.14238 (2022). Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. 2022. Wild-time: A benchmark of in-the-wild distribution shift over time. arXiv preprint arXiv:2211.14238 (2022)."},{"key":"#cr-split#-e_1_3_2_2_152_1.1","doi-asserted-by":"crossref","unstructured":"Iris Marion Young. 2002. Inclusion and Democracy. Oxford University Press. https:\/\/doi.org\/10.1093\/0198297556.003.0001 10.1093\/0198297556.003.0001","DOI":"10.1093\/0198297556.003.0001"},{"key":"#cr-split#-e_1_3_2_2_152_1.2","doi-asserted-by":"crossref","unstructured":"Iris Marion Young. 2002. Inclusion and Democracy. Oxford University Press. https:\/\/doi.org\/10.1093\/0198297556.003.0001","DOI":"10.1093\/0198297556.003.0001"},{"key":"e_1_3_2_2_153_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Yu Chao","year":"2022","unstructured":"Chao Yu , Akash Velu , Eugene Vinitsky , Yu Wang , Alexandre Bayen , and Yi Wu . 2022 . The surprising effectiveness of ppo in cooperative, multi-agent games . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative, multi-agent games. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."},{"key":"e_1_3_2_2_154_1","volume-title":"Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Zaporojets Klim","year":"2022","unstructured":"Klim Zaporojets , Lucie-Aim\u00e9e Kaffee , Thomas Demeester , Chris Develder , and I Augenstein . 2022 . TempEL: Linking dynamically evolving and newly emerging entities . In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Klim Zaporojets, Lucie-Aim\u00e9e Kaffee, Thomas Demeester, Chris Develder, and I Augenstein. 2022. TempEL: Linking dynamically evolving and newly emerging entities. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_2_155_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","author":"Zheng Kaizhi","year":"2022","unstructured":"Kaizhi Zheng , Xiaotong Chen , Odest Chadwicke Jenkins , and Xin Eric Wang . 2022 . Vlmbench: A compositional benchmark for vision-and-language manipulation . Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022). Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Eric Wang. 2022. Vlmbench: A compositional benchmark for vision-and-language manipulation. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (2022)."}],"event":{"name":"FAccT '23: the 2023 ACM Conference on Fairness, Accountability, and Transparency","location":"Chicago IL USA","acronym":"FAccT '23"},"container-title":["2023 ACM Conference on Fairness, Accountability, and Transparency"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3593013.3594019","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3593013.3594019","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:48:03Z","timestamp":1750178883000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3593013.3594019"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,12]]},"references-count":155,"alternative-id":["10.1145\/3593013.3594019","10.1145\/3593013"],"URL":"https:\/\/doi.org\/10.1145\/3593013.3594019","relation":{},"subject":[],"published":{"date-parts":[[2023,6,12]]},"assertion":[{"value":"2023-06-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}