{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:53:58Z","timestamp":1776110038360,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":91,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,4,27]],"date-time":"2022-04-27T00:00:00Z","timestamp":1651017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,27]]},"DOI":"10.1145\/3491102.3502004","type":"proceedings-article","created":{"date-parts":[[2022,4,28]],"date-time":"2022-04-28T16:33:27Z","timestamp":1651163607000},"page":"1-19","source":"Crossref","is-referenced-by-count":85,"title":["Jury Learning: Integrating Dissenting Voices into Machine Learning Models"],"prefix":"10.1145","author":[{"given":"Mitchell L.","family":"Gordon","sequence":"first","affiliation":[{"name":"Computer Science Department, Stanford University, United States"}]},{"given":"Michelle S.","family":"Lam","sequence":"additional","affiliation":[{"name":"Stanford University, United States"}]},{"given":"Joon Sung","family":"Park","sequence":"additional","affiliation":[{"name":"Computer Science Department, Stanford University, United States"}]},{"given":"Kayur","family":"Patel","sequence":"additional","affiliation":[{"name":"Apple Inc, United States"}]},{"given":"Jeff","family":"Hancock","sequence":"additional","affiliation":[{"name":"Department of Communication, Stanford University, United States"}]},{"given":"Tatsunori","family":"Hashimoto","sequence":"additional","affiliation":[{"name":"Computer Science Department, Stanford University, United States"}]},{"given":"Michael S.","family":"Bernstein","sequence":"additional","affiliation":[{"name":"Computer Science, Stanford University, United States"}]}],"member":"320","published-online":{"date-parts":[[2022,4,28]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978318"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372871"},{"key":"e_1_3_2_2_3_1","volume-title":"International Conference on Machine Learning. PMLR, 60\u201369","author":"Agarwal Alekh","year":"2018","unstructured":"Alekh Agarwal , Alina Beygelzimer , Miroslav Dud\u00edk , John Langford , and Hanna Wallach . 2018 . A reductions approach to fair classification . In International Conference on Machine Learning. PMLR, 60\u201369 . Alekh Agarwal, Alina Beygelzimer, Miroslav Dud\u00edk, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In International Conference on Machine Learning. PMLR, 60\u201369."},{"key":"e_1_3_2_2_4_1","unstructured":"Sohail Akhtar Valerio Basile and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arxiv:2106.15896\u00a0[cs.CL]  Sohail Akhtar Valerio Basile and Viviana Patti. 2021. Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection. arxiv:2106.15896\u00a0[cs.CL]"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"crossref","unstructured":"Jennifer N\u00a0L Allen Antonio\u00a0A Arechar Gordon Pennycook and David\u00a0G Rand. 2020. Scaling Up Fact-Checking Using the Wisdom of Crowds. https:\/\/doi.org\/10.31234\/osf.io\/9qdza  Jennifer N\u00a0L Allen Antonio\u00a0A Arechar Gordon Pennycook and David\u00a0G Rand. 2020. Scaling Up Fact-Checking Using the Wisdom of Crowds. https:\/\/doi.org\/10.31234\/osf.io\/9qdza","DOI":"10.31234\/osf.io\/9qdza"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300773"},{"key":"e_1_3_2_2_10_1","first-page":"671","article-title":"Big data\u2019s disparate impact","volume":"104","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew\u00a0 D Selbst . 2016 . Big data\u2019s disparate impact . Calif. L. Rev. 104 (2016), 671 . Solon Barocas and Andrew\u00a0D Selbst. 2016. Big data\u2019s disparate impact. Calif. L. Rev. 104(2016), 671.","journal-title":"Calif. L. Rev."},{"key":"e_1_3_2_2_11_1","volume-title":"Who Moderates the Social Media Giants?Center for Business","author":"Barrett M","year":"2020","unstructured":"Paul\u00a0 M Barrett . 2020. Who Moderates the Social Media Giants?Center for Business ( 2020 ). Paul\u00a0M Barrett. 2020. Who Moderates the Social Media Giants?Center for Business (2020)."},{"key":"e_1_3_2_2_12_1","volume-title":"ESR: Ethics and Society Review of Artificial Intelligence Research. arXiv preprint arXiv:2106.11521(2021).","author":"Bernstein S","year":"2021","unstructured":"Michael\u00a0 S Bernstein , Margaret Levi , David Magnus , Betsy Rajala , Debra Satz , and Charla Waeiss . 2021 . ESR: Ethics and Society Review of Artificial Intelligence Research. arXiv preprint arXiv:2106.11521(2021). Michael\u00a0S Bernstein, Margaret Levi, David Magnus, Betsy Rajala, Debra Satz, and Charla Waeiss. 2021. ESR: Ethics and Society Review of Artificial Intelligence Research. arXiv preprint arXiv:2106.11521(2021)."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242598"},{"key":"e_1_3_2_2_14_1","volume-title":"Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2","author":"Caplan Robyn","year":"2020","unstructured":"Robyn Caplan and Tarleton Gillespie . 2020. Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2 ( 2020 ), 2056305120936636. Robyn Caplan and Tarleton Gillespie. 2020. Tiered governance and demonetization: The shifting terms of labor and compensation in the platform economy. Social Media+ Society 6, 2 (2020), 2056305120936636."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_3_2_2_16_1","unstructured":"Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through qualitative research.(2006).  Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through qualitative research.(2006)."},{"key":"e_1_3_2_2_17_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019)","author":"Joon\u00a0Young Chung John","year":"2019","unstructured":"John Joon\u00a0Young Chung , Jean\u00a0 Y Song , Sindhu Kutty , Sungsoo Hong , Juho Kim , and Walter\u00a0 S Lasecki . 2019 . Efficient elicitation approaches to estimate collective crowd answers . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1\u201325. John Joon\u00a0Young Chung, Jean\u00a0Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter\u00a0S Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1\u201325."},{"key":"e_1_3_2_2_18_1","volume-title":"Design justice: Community-led practices to build the worlds we need","author":"Costanza-Chock Sasha","unstructured":"Sasha Costanza-Chock . 2020. Design justice: Community-led practices to build the worlds we need . The MIT Press . Sasha Costanza-Chock. 2020. Design justice: Community-led practices to build the worlds we need. The MIT Press."},{"key":"e_1_3_2_2_19_1","unstructured":"Aida\u00a0Mostafazadeh Davani Mark D\u00edaz and Vinodkumar Prabhakaran. 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arxiv:2110.05719\u00a0[cs.CL]  Aida\u00a0Mostafazadeh Davani Mark D\u00edaz and Vinodkumar Prabhakaran. 2021. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. arxiv:2110.05719\u00a0[cs.CL]"},{"key":"e_1_3_2_2_20_1","volume-title":"Democracy in America","author":"De\u00a0Toqueville Alexis","year":"1835","unstructured":"Alexis De\u00a0Toqueville . 1835. Democracy in America . New York : A Mentor Book from New American Library ( 1835 ). Alexis De\u00a0Toqueville. 1835. Democracy in America. New York: A Mentor Book from New American Library (1835)."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"crossref","unstructured":"J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.  J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_2_22_1","volume-title":"Jury decision making: 45 years of empirical research on deliberating groups.Psychology, public policy, and law 7, 3","author":"Devine J","year":"2001","unstructured":"Dennis\u00a0 J Devine , Laura\u00a0 D Clayton , Benjamin\u00a0 B Dunford , Rasmy Seying , and Jennifer Pryce . 2001. Jury decision making: 45 years of empirical research on deliberating groups.Psychology, public policy, and law 7, 3 ( 2001 ), 622. Dennis\u00a0J Devine, Laura\u00a0D Clayton, Benjamin\u00a0B Dunford, Rasmy Seying, and Jennifer Pryce. 2001. Jury decision making: 45 years of empirical research on deliberating groups.Psychology, public policy, and law 7, 3 (2001), 622."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173694"},{"key":"e_1_3_2_2_24_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018)."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/pra2.2018.14505501013"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18818-8_43"},{"key":"e_1_3_2_2_27_1","unstructured":"Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. arXiv preprint arXiv:1701.02185(2017).  Anca Dumitrache Lora Aroyo and Chris Welty. 2017. Crowdsourcing ground truth for medical relation extraction. arXiv preprint arXiv:1701.02185(2017)."},{"key":"e_1_3_2_2_28_1","unstructured":"Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. arXiv preprint arXiv:1805.00270(2018).  Anca Dumitrache Lora Aroyo and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. arXiv preprint arXiv:1805.00270(2018)."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702556"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604056"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376293"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3409559"},{"key":"e_1_3_2_2_34_1","unstructured":"Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer\u00a0Wortman Vaughan Hanna Wallach Hal Daum\u00e9\u00a0III and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010(2018).  Timnit Gebru Jamie Morgenstern Briana Vecchione Jennifer\u00a0Wortman Vaughan Hanna Wallach Hal Daum\u00e9\u00a0III and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010(2018)."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3200947.3208069"},{"key":"e_1_3_2_2_36_1","volume-title":"Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media","author":"Gillespie Tarleton","unstructured":"Tarleton Gillespie . 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media . Yale University Press . Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_3_2_2_38_1","volume-title":"Ghost work: How to stop Silicon Valley from building a new global underclass","author":"Gray L","unstructured":"Mary\u00a0 L Gray and Siddharth Suri . 2019. Ghost work: How to stop Silicon Valley from building a new global underclass . Eamon Dolan Books . Mary\u00a0L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books."},{"key":"e_1_3_2_2_39_1","volume-title":"Equality of opportunity in supervised learning. Advances in neural information processing systems 29","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt , Eric Price , and Nati Srebro . 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 ( 2016 ), 3315\u20133323. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016), 3315\u20133323."},{"key":"e_1_3_2_2_40_1","volume-title":"Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction. Applied Sciences 7, 12","author":"Haruna Khalid","year":"2017","unstructured":"Khalid Haruna , Maizatul Akmar\u00a0Ismail , Suhendroyono Suhendroyono , Damiasih Damiasih , Adi\u00a0Cilik Pierewan , Haruna Chiroma , and Tutut Herawan . 2017. Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction. Applied Sciences 7, 12 ( 2017 ). https:\/\/doi.org\/10.3390\/app7121211 Khalid Haruna, Maizatul Akmar\u00a0Ismail, Suhendroyono Suhendroyono, Damiasih Damiasih, Adi\u00a0Cilik Pierewan, Haruna Chiroma, and Tutut Herawan. 2017. Context-Aware Recommender System: A Review of Recent Developmental Process and Future Research Direction. Applied Sciences 7, 12 (2017). https:\/\/doi.org\/10.3390\/app7121211"},{"key":"e_1_3_2_2_41_1","volume-title":"International Conference on Machine Learning. PMLR","author":"Hashimoto Tatsunori","year":"2018","unstructured":"Tatsunori Hashimoto , Megha Srivastava , Hongseok Namkoong , and Percy Liang . 2018 . Fairness without demographics in repeated loss minimization . In International Conference on Machine Learning. PMLR , 1929\u20131938. Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning. PMLR, 1929\u20131938."},{"key":"e_1_3_2_2_42_1","volume-title":"Inside the jury","author":"Hastie Reid","unstructured":"Reid Hastie , Steven\u00a0 D Penrod , and Nancy Pennington . 2013. Inside the jury . Harvard University Press . Reid Hastie, Steven\u00a0D Penrod, and Nancy Pennington. 2013. Inside the jury. Harvard University Press."},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0256762"},{"key":"e_1_3_2_2_44_1","unstructured":"Jigsaw. 2019. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-bias-in-toxicity-classification\/overview  Jigsaw. 2019. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-bias-in-toxicity-classification\/overview"},{"key":"e_1_3_2_2_45_1","unstructured":"Jeremy Kahn. 2020. Can Facebook\u2019s new A.I. banish Pepe the Frog?https:\/\/fortune.com\/2020\/05\/12\/facebook-a-i-hate-speech-covid-19-misinformation\/  Jeremy Kahn. 2020. Can Facebook\u2019s new A.I. banish Pepe the Frog?https:\/\/fortune.com\/2020\/05\/12\/facebook-a-i-hate-speech-covid-19-misinformation\/"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_3_2_2_47_1","first-page":"2418","article-title":"The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression","volume":"129","author":"Klonick Kate","year":"2019","unstructured":"Kate Klonick . 2019 . The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression . Yale LJ 129 (2019), 2418 . Kate Klonick. 2019. The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression. Yale LJ 129(2019), 2418.","journal-title":"Yale LJ"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134697"},{"key":"e_1_3_2_2_49_1","unstructured":"Deepak Kumar Patrick\u00a0Gage Kelley Sunny Consolvo Joshua Mason Elie Bursztein Zakir Durumeric Kurt Thomas and Michael Bailey. 2021. Designing Toxic Content Classification for a Diversity of Perspectives. arXiv preprint arXiv:2106.04511(2021).  Deepak Kumar Patrick\u00a0Gage Kelley Sunny Consolvo Joshua Mason Elie Bursztein Zakir Durumeric Kurt Thomas and Michael Bailey. 2021. Designing Toxic Content Classification for a Diversity of Perspectives. arXiv preprint arXiv:2106.04511(2021)."},{"key":"e_1_3_2_2_50_1","volume-title":"Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.","author":"Lease Matthew","year":"2011","unstructured":"Matthew Lease . 2011 . On quality control and machine learning in crowdsourcing . In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. Matthew Lease. 2011. On quality control and machine learning in crowdsourcing. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1214\/19-AOS1828"},{"key":"e_1_3_2_2_52_1","volume-title":"Second AAAI Conference on Human Computation and Crowdsourcing.","author":"Lee David\u00a0Timothy","year":"2014","unstructured":"David\u00a0Timothy Lee , Ashish Goel , Tanja Aitamurto , and Helene Landemore . 2014 . Crowdsourcing for participatory democracies: Efficient elicitation of social choice functions . In Second AAAI Conference on Human Computation and Crowdsourcing. David\u00a0Timothy Lee, Ashish Goel, Tanja Aitamurto, and Helene Landemore. 2014. Crowdsourcing for participatory democracies: Efficient elicitation of social choice functions. In Second AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_3_2_2_53_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019)","author":"Noothigattu Ritesh","year":"2019","unstructured":"Min\u00a0Kyung Lee, Daniel Kusbit, Anson Kahng, Ji\u00a0Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu , Siheon Lee , Alexandros Psomas , 2019 . WeBuildAI: Participatory framework for algorithmic governance . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1\u201335. Min\u00a0Kyung Lee, Daniel Kusbit, Anson Kahng, Ji\u00a0Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, 2019. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1\u201335."},{"key":"e_1_3_2_2_54_1","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1\u201315","author":"Jia-Jun Li Toby","year":"2021","unstructured":"Toby Jia-Jun Li , Lindsay Popowski , Tom Mitchell , and Brad\u00a0 A Myers . 2021 . Screen2Vec: Semantic Embedding of GUI Screens and GUI Components . In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1\u201315 . Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad\u00a0A Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1\u201315."},{"key":"e_1_3_2_2_55_1","volume-title":"Patterns of democracy","author":"Lijphart Arend","unstructured":"Arend Lijphart . 1999. Patterns of democracy . Yale university press . Arend Lijphart. 1999. Patterns of democracy. Yale university press."},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317082"},{"key":"e_1_3_2_2_57_1","volume-title":"Sixth AAAI Conference on Human Computation and Crowdsourcing.","author":"Manam K\u00a0Chaithanya","year":"2018","unstructured":"V K\u00a0Chaithanya Manam and Alexander\u00a0 J Quinn . 2018 . Wingit: Efficient refinement of unclear task instructions . In Sixth AAAI Conference on Human Computation and Crowdsourcing. VK\u00a0Chaithanya Manam and Alexander\u00a0J Quinn. 2018. Wingit: Efficient refinement of unclear task instructions. In Sixth AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_3_2_2_58_1","unstructured":"Brandeis Marshall. 2021. Algorithmic misogynoir in content moderation practice. Heinrich-B\u00f6ll-Stiftung(2021).  Brandeis Marshall. 2021. Algorithmic misogynoir in content moderation practice. Heinrich-B\u00f6ll-Stiftung(2021)."},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_2_60_1","volume-title":"Innovations in Big Data Mining and Embedded Knowledge","author":"Moreau Erwan","unstructured":"Erwan Moreau , Carl Vogel , and Marguerite Barry . 2019. A paradigm for democratizing artificial intelligence research . In Innovations in Big Data Mining and Embedded Knowledge . Springer , 137\u2013166. Erwan Moreau, Carl Vogel, and Marguerite Barry. 2019. A paradigm for democratizing artificial intelligence research. In Innovations in Big Data Mining and Embedded Knowledge. Springer, 137\u2013166."},{"key":"e_1_3_2_2_61_1","volume-title":"Designing Ground Truth and the Social Life of Labels","author":"Muller Michael","unstructured":"Michael Muller , Christine\u00a0 T. Wolf , Josh Andres , Michael Desmond , Narendra\u00a0Nath Joshi , Zahra Ashktorab , Aabhas Sharma , Kristina Brimijoin , Qian Pan , Evelyn Duesterwald , and Casey Dugan . 2021. Designing Ground Truth and the Social Life of Labels . Association for Computing Machinery , New York, NY, USA . https:\/\/doi.org\/10.1145\/3411764.3445402 Michael Muller, Christine\u00a0T. Wolf, Josh Andres, Michael Desmond, Narendra\u00a0Nath Joshi, Zahra Ashktorab, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Evelyn Duesterwald, and Casey Dugan. 2021. Designing Ground Truth and the Social Life of Labels. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3411764.3445402"},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"crossref","unstructured":"Dat\u00a0Quoc Nguyen Thanh Vu and Anh\u00a0Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arxiv:2005.10200\u00a0[cs.CL]  Dat\u00a0Quoc Nguyen Thanh Vu and Anh\u00a0Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. arxiv:2005.10200\u00a0[cs.CL]","DOI":"10.18653\/v1\/2020.emnlp-demos.2"},{"key":"e_1_3_2_2_63_1","unstructured":"Kayur Patel James Fogarty James\u00a0A Landay and Beverly\u00a0L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563\u20131566.  Kayur Patel James Fogarty James\u00a0A Landay and Beverly\u00a0L Harrison. 2008. Examining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning.. In AAAI. 1563\u20131566."},{"key":"e_1_3_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00293"},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00971"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.law-1.14"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-017-9430-8"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1177\/019251219201300107"},{"key":"e_1_3_2_2_70_1","unstructured":"Anna Rogers. 2021. Changing the World by Changing the Data. arxiv:2105.13947\u00a0[cs.CL]  Anna Rogers. 2021. Changing the World by Changing the Data. arxiv:2105.13947\u00a0[cs.CL]"},{"key":"e_1_3_2_2_71_1","unstructured":"Christian Sandvig Kevin Hamilton Karrie Karahalios and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22(2014).  Christian Sandvig Kevin Hamilton Karrie Karahalios and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and discrimination: converting critical concerns into productive inquiry 22(2014)."},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359178"},{"key":"e_1_3_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376506"},{"key":"e_1_3_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_3_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476058"},{"key":"e_1_3_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445742"},{"key":"e_1_3_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_3_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376783"},{"key":"e_1_3_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.199668"},{"key":"e_1_3_2_2_80_1","unstructured":"Harini Suresh and John\u00a0V Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002(2019).  Harini Suresh and John\u00a0V Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002(2019)."},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.5555\/3298023.3298188"},{"key":"e_1_3_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.6762"},{"key":"e_1_3_2_2_83_1","doi-asserted-by":"crossref","unstructured":"Betty van Aken Julian Risch Ralf Krestel and Alexander L\u00f6ser. 2018. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572(2018).  Betty van Aken Julian Risch Ralf Krestel and Alexander L\u00f6ser. 2018. Challenges for toxic comment classification: An in-depth error analysis. arXiv preprint arXiv:1809.07572(2018).","DOI":"10.18653\/v1\/W18-5105"},{"key":"e_1_3_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13739-6_25"},{"key":"e_1_3_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.5555\/3367471.3367557"},{"key":"e_1_3_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450078"},{"key":"e_1_3_2_2_87_1","unstructured":"Tian Wang Yuri\u00a0M Brovman and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. arXiv preprint arXiv:2102.06156(2021).  Tian Wang Yuri\u00a0M Brovman and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. arXiv preprint arXiv:2102.06156(2021)."},{"key":"e_1_3_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357236.3395528"},{"key":"e_1_3_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415858"},{"key":"e_1_3_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3136792"},{"key":"e_1_3_2_2_91_1","unstructured":"Xinyi Zhou and Reza Zafarani. 2018. Fake news: A survey of research detection methods and opportunities. arXiv preprint arXiv:1812.00315(2018).  Xinyi Zhou and Reza Zafarani. 2018. Fake news: A survey of research detection methods and opportunities. arXiv preprint arXiv:1812.00315(2018)."},{"key":"e_1_3_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"}],"event":{"name":"CHI '22: CHI Conference on Human Factors in Computing Systems","location":"New Orleans LA USA","acronym":"CHI '22","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491102.3502004","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3491102.3502004","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:06Z","timestamp":1750188666000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491102.3502004"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,27]]},"references-count":91,"alternative-id":["10.1145\/3491102.3502004","10.1145\/3491102"],"URL":"https:\/\/doi.org\/10.1145\/3491102.3502004","relation":{},"subject":[],"published":{"date-parts":[[2022,4,27]]}}}