{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:25:50Z","timestamp":1776111950633,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":63,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,8,8]],"date-time":"2023-08-08T00:00:00Z","timestamp":1691452800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,8,8]]},"DOI":"10.1145\/3600211.3604682","type":"proceedings-article","created":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T18:41:37Z","timestamp":1693334497000},"page":"375-386","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7540-1402","authenticated-orcid":false,"given":"Nathan","family":"Dennler","sequence":"first","affiliation":[{"name":"University of Southern California, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0531-7520","authenticated-orcid":false,"given":"Anaelia","family":"Ovalle","sequence":"additional","affiliation":[{"name":"Computer Science, University of California, Los Angeles, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9418-0430","authenticated-orcid":false,"given":"Ashwin","family":"Singh","sequence":"additional","affiliation":[{"name":"QueerInAI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6998-9863","authenticated-orcid":false,"given":"Luca","family":"Soldaini","sequence":"additional","affiliation":[{"name":"QueerInAI and Allen Institute for AI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0415-3800","authenticated-orcid":false,"given":"Arjun","family":"Subramonian","sequence":"additional","affiliation":[{"name":"University of California, Los Angeles, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7843-252X","authenticated-orcid":false,"given":"Huy","family":"Tu","sequence":"additional","affiliation":[{"name":"QueerInAI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1362-554X","authenticated-orcid":false,"given":"William","family":"Agnew","sequence":"additional","affiliation":[{"name":"Computer Science and Engineering, QueerInAI and University of Washington, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8540-3698","authenticated-orcid":false,"given":"Avijit","family":"Ghosh","sequence":"additional","affiliation":[{"name":"Khoury College of Computer Sciences, Queer in AI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6856-752X","authenticated-orcid":false,"given":"Kyra","family":"Yee","sequence":"additional","affiliation":[{"name":"Queer in AI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1571-0828","authenticated-orcid":false,"given":"Irene Font","family":"Peradejordi","sequence":"additional","affiliation":[{"name":"Queer in AI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5503-867X","authenticated-orcid":false,"given":"Zeerak","family":"Talat","sequence":"additional","affiliation":[{"name":"Queer in AI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7080-6331","authenticated-orcid":false,"given":"Mayra","family":"Russo","sequence":"additional","affiliation":[{"name":"Scientific Data Management, Queer in AI, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8457-1756","authenticated-orcid":false,"given":"Jess De Jesus De Pinho","family":"Pinhal","sequence":"additional","affiliation":[{"name":"Queer in AI, France"}]}],"member":"320","published-online":{"date-parts":[[2023,8,29]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"Blaise Ag\u00fcera\u00a0y Arcas Margaret Mitchell and Alexander Todorov. 2017. Physiognomy\u2019s New Clothes. https:\/\/medium.com\/@blaisea\/physiognomys-new-clothes-f2d4b59fdd6a"},{"key":"e_1_3_2_2_2_1","unstructured":"Amit\u00a0Elazari Bar\u00a0On. 2018. We need bug bounties for bad algorithms. https:\/\/www.vice.com\/en\/article\/8xkyj3\/we-need-bug-bounties-for-bad-algorithms"},{"key":"e_1_3_2_2_3_1","unstructured":"Rachel K.\u00a0E. Bellamy Kuntal Dey Michael Hind Samuel\u00a0C. Hoffman Stephanie Houde Kalapriya Kannan Pranay Lohia Jacquelyn Martino Sameep Mehta Aleksandra Mojsilovic Seema Nagar Karthikeyan\u00a0Natesan Ramamurthy John Richards Diptikalyan Saha Prasanna Sattigeri Moninder Singh Kush\u00a0R. Varshney and Yunfeng Zhang. 2018. AI Fairness 360: An Extensible Toolkit for Detecting Understanding and Mitigating Unwanted Algorithmic Bias. https:\/\/arxiv.org\/abs\/1810.01943"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00041"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551624.3555290"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_3_2_2_7_1","unstructured":"Michelle Boorstein and Heather Kelly. 2022. Catholic group spent millions on app data that tracked gay priests. https:\/\/www.washingtonpost.com\/dc-md-va\/2023\/03\/09\/catholics-gay-priests-grindr-data-bishops\/"},{"key":"e_1_3_2_2_8_1","unstructured":"Bias Buccaneers. 2023. Who are the Bias Buccaneers?https:\/\/www.biasbuccaneers.org\/"},{"key":"e_1_3_2_2_9_1","unstructured":"Madeleine Carlisle. 2021. How the Alleged Outing of a Catholic Priest Shows the Sorry State of Data Privacy in America. https:\/\/time.com\/6083323\/bishop-pillar-grindr-data\/"},{"key":"e_1_3_2_2_10_1","unstructured":"Rumman Chowdhury and Jutta Williams. 2021. Introducing Twitter\u2019s first algorithmic bias bounty challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bounty-challenge"},{"key":"e_1_3_2_2_11_1","volume-title":"Thematic analysis. Qualitative psychology: A practical guide to research methods 3","author":"Clarke Victoria","year":"2015","unstructured":"Victoria Clarke, Virginia Braun, and Nikki Hayfield. 2015. Thematic analysis. Qualitative psychology: A practical guide to research methods 3 (2015), 222\u2013248."},{"key":"e_1_3_2_2_12_1","unstructured":"Bill Cooke and Uma Kothari. 2001. Participation. Zed Books London England."},{"key":"e_1_3_2_2_13_1","volume-title":"No Access: LGBTIQ Website Censorship in Six Countries. https:\/\/citizenlab.ca\/2021\/08\/no-access-lgbtiq-website-censorship-in-six-countries\/","author":"Dalek Jakub","year":"2021","unstructured":"Jakub Dalek, Nica Dumlao, Miles Kenyon, Irene Poetranto, Adam Senft, Caroline Wesley, Arturo Filast\u00f2, Maria Xynou, and Amie Bishop. 2021. No Access: LGBTIQ Website Censorship in Six Countries. https:\/\/citizenlab.ca\/2021\/08\/no-access-lgbtiq-website-censorship-in-six-countries\/"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.150"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517441"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12119-020-09790-w"},{"key":"e_1_3_2_2_17_1","volume-title":"Data feminism","author":"D\u2019Ignazio Catherine","unstructured":"Catherine D\u2019Ignazio and Lauren\u00a0F Klein. 2023. Data feminism. MIT Press, London, England."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.98"},{"key":"e_1_3_2_2_19_1","unstructured":"AI dungeon. 2019. AI dungeon. https:\/\/play.aidungeon.io\/main\/home"},{"key":"e_1_3_2_2_20_1","unstructured":"Tom Embury-Dennis. 2020. Bullied and blackmailed: Gay men in Morocco falling victims to outing campaign sparked by Instagram model. https:\/\/www.independent.co.uk\/news\/world\/africa\/gay-men-morocco-dating-apps-grindr-instagram-sofia-taloni-a9486386.html"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.507"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/1476750306066801"},{"key":"e_1_3_2_2_23_1","volume-title":"Egypt police \u2018using dating apps","author":"Fox Gemma","unstructured":"Gemma Fox. 2020. Egypt police \u2018using dating apps\u2019 to find and imprison LGBT+ people. https:\/\/www.independent.co.uk\/news\/world\/middle-east\/egypt-lgbt-gay-facebook-grindr-jail-torture-police-hrw-b742231.html"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-2501"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1080\/0951839890020105"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533172"},{"key":"e_1_3_2_2_28_1","volume-title":"Ghost work","author":"Gray L","unstructured":"Mary\u00a0L Gray and Siddharth Suri. 2019. Ghost work. Houghton Mifflin Harcourt Publishing Company, Boston, MA."},{"key":"e_1_3_2_2_29_1","volume-title":"Community-based participatory research for health, M.\u00a0Minkler and N.\u00a0Wallerstein (Eds.). Jossey-Bass","author":"Green LW","unstructured":"LW Green, MA George, 2003. Appendix C: Guidelines for participatory research in health promotion. In Community-based participatory research for health, M.\u00a0Minkler and N.\u00a0Wallerstein (Eds.). Jossey-Bass, San Francisco, CA."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594071"},{"key":"e_1_3_2_2_31_1","unstructured":"Karen Hacker and J.\u00a0Glover Taylor. 2011. Community-Engaged Research 101. https:\/\/catalyst.harvard.edu\/publications-documents\/community-engaged-research-101-2\/"},{"key":"e_1_3_2_2_32_1","unstructured":"Foad Hamidi Morgan Scheuerman and Stacy Branham. 2020. Gender is personal\u2013not computational."},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445778"},{"key":"e_1_3_2_2_34_1","unstructured":"Nicolas Kayser-Bril. 2021. Twitter\u2019s algorithmic bias bug bounty could be the way forward if regulators step in. https:\/\/algorithmwatch.org\/en\/twitters-algorithmic-bias-bug-bounty\/"},{"key":"e_1_3_2_2_35_1","unstructured":"Josh Kenway Camille Fran\u00e7ois Sasha Costanza-Chock Inioluwa\u00a0Deborah Raji and Joy Buolamwini. 2022. Bug bounties for algorithmic harms?https:\/\/www.ajl.org\/bugs"},{"key":"e_1_3_2_2_36_1","unstructured":"Os Keyes. 2019. Counting the Countless: Why data science is a profound threat for queer people."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1177\/20539517221113772"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-022-02137-y"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3582074"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533161"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445935"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_2_43_1","volume-title":"The digital closet","author":"Monea Alexander","unstructured":"Alexander Monea. 2022. The digital closet. MIT Press, Cambridge, MA."},{"key":"e_1_3_2_2_44_1","volume-title":"Glide: Towards photorealistic image generation and editing with text-guided diffusion models.","author":"Nichol Alex","year":"2021","unstructured":"Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: Towards photorealistic image generation and editing with text-guided diffusion models."},{"key":"e_1_3_2_2_45_1","unstructured":"Mimi Onuoha and Nucera Nucera. 2018. The People\u2019s Guide to Artificial Intelligence. Allied Media Projects virtual."},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594078"},{"key":"e_1_3_2_2_47_1","volume-title":"Digital harassment and abuse: Experiences of sexuality and gender minority adults. European journal of criminology 17, 2","author":"Powell Anastasia","year":"2020","unstructured":"Anastasia Powell, Adrian\u00a0J Scott, and Nicola Henry. 2020. Digital harassment and abuse: Experiences of sexuality and gender minority adults. European journal of criminology 17, 2 (2020), 199\u2013223."},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594134"},{"key":"e_1_3_2_2_49_1","volume-title":"Rebuilding Trust: Queer in AI Approach to Artificial Intelligence Risk Management. queerinai.org\/risk-management","author":"Organizers\u00a0of","year":"2021","unstructured":"Organizers\u00a0of QueerInAI, Ashwin S, William Agnew, Hetvi Jethwani, and Arjun Subramonian. 2021. Rebuilding Trust: Queer in AI Approach to Artificial Intelligence Risk Management. queerinai.org\/risk-management"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514094.3534181"},{"key":"e_1_3_2_2_52_1","volume-title":"Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a0139)","author":"Ramesh Aditya","year":"2021","unstructured":"Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol.\u00a0139), Marina Meila and Tong Zhang (Eds.). PMLR, virtual, 8821\u20138831. https:\/\/proceedings.mlr.press\/v139\/ramesh21a.html"},{"key":"e_1_3_2_2_53_1","unstructured":"Javier Rando Daniel Paleka David Lindner Lennard Heim and Florian Tram\u00e8r. 2022. Red-Teaming the Stable Diffusion Safety Filter."},{"key":"e_1_3_2_2_54_1","volume-title":"Aequitas: A Bias and Fairness Audit Toolkit.","author":"Saleiro Pedro","year":"2018","unstructured":"Pedro Saleiro, Benedict Kuester, Abby Stevens, Ari Anisfeld, Loren Hinkson, Jesse London, and Rayid Ghani. 2018. Aequitas: A Bias and Fairness Audit Toolkit."},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479577"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551624.3555285"},{"key":"e_1_3_2_2_57_1","unstructured":"Shakira Smith Oliver\u00a0L Haimson Claire Fitzsimmons and Nikki\u00a0Echarte Brown. 2021. Censorship of Marginalized Communities on Instagram. https:\/\/saltyworld.net\/exclusive-report-censorship-of-marginalized-communities-on-instagram-2021-pdf-download\/"},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1215\/01642472-7971139"},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445107"},{"key":"e_1_3_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533132"},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462540"},{"key":"e_1_3_2_2_62_1","volume-title":"On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective. https:\/\/openreview.net\/forum?id=uw6HSkgoM29","author":"Wang Jindong","unstructured":"Jindong Wang, Xixu HU, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Wei Ye, Haojun Huang, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xing Xie. 2023. On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective. https:\/\/openreview.net\/forum?id=uw6HSkgoM29"},{"key":"e_1_3_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3488666"}],"event":{"name":"AIES '23: AAAI\/ACM Conference on AI, Ethics, and Society","location":"Montr\u00e9al QC Canada","acronym":"AIES '23","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence"]},"container-title":["Proceedings of the 2023 AAAI\/ACM Conference on AI, Ethics, and Society"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3600211.3604682","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3600211.3604682","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:39Z","timestamp":1750178259000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3600211.3604682"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,8]]},"references-count":63,"alternative-id":["10.1145\/3600211.3604682","10.1145\/3600211"],"URL":"https:\/\/doi.org\/10.1145\/3600211.3604682","relation":{},"subject":[],"published":{"date-parts":[[2023,8,8]]},"assertion":[{"value":"2023-08-29","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}