{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T13:24:18Z","timestamp":1767705858121,"version":"3.41.0"},"reference-count":89,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2017,12,6]],"date-time":"2017-12-06T00:00:00Z","timestamp":1512518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["0964695, 1218826, 1111201, 1017697, IIS-1707296, IIS-1707319"],"award-info":[{"award-number":["0964695, 1218826, 1111201, 1017697, IIS-1707296, IIS-1707319"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2017,12,6]]},"abstract":"<jats:p>As the gig economy continues to grow and freelance work moves online, five-star reputation systems are becoming more and more common. At the same time, there are increasing accounts of race and gender bias in evaluations of gig workers, with negative impacts for those workers. We report on a series of four Mechanical Turk-based studies in which participants who rated simulated gig work did not show race- or gender bias, while manipulation checks showed they reliably distinguished between low- and high-quality work. Given prior research, this was a striking result. To explore further, we used a Bayesian approach to verify absence of ratings bias (as opposed to merely not detecting bias). This Bayesian test let us identify an upper- bound: if any bias did exist in our studies, it was below an average of 0.2 stars on a five-star scale. We discuss possible interpretations of our results and outline future work to better understand the results.<\/jats:p>","DOI":"10.1145\/3134736","type":"journal-article","created":{"date-parts":[[2017,12,6]],"date-time":"2017-12-06T21:23:15Z","timestamp":1512595395000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Simulation Experiments on (the Absence of) Ratings Bias in Reputation Systems"],"prefix":"10.1145","volume":"1","author":[{"given":"Jacob","family":"Thebault-Spieker","sequence":"first","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"given":"Daniel","family":"Kluver","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"given":"Maximilian A.","family":"Klein","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"given":"Aaron","family":"Halfaker","sequence":"additional","affiliation":[{"name":"Wikimedia, NEED, USA"}]},{"given":"Brent","family":"Hecht","sequence":"additional","affiliation":[{"name":"Northwestern University, Evanston, IL, USA"}]},{"given":"Loren","family":"Terveen","sequence":"additional","affiliation":[{"name":"The University of Minnesota, Minneapolis, MN, USA"}]},{"given":"Joseph A.","family":"Konstan","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]}],"member":"320","published-online":{"date-parts":[[2017,12,6]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02247-0_24"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979041"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.74.2.170"},{"key":"e_1_2_1_4_1","volume-title":"Retrieved","author":"Benson Alan","year":"2015","unstructured":"Alan Benson , Aaron J. Sojourner , and Akhmed Umyarov . 2015 . Can Reputation Discipline the Gig Economy? Experimental Evidence from an Online Labor Market . Retrieved February 22, 2016 from http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2696299 Alan Benson, Aaron J. Sojourner, and Akhmed Umyarov. 2015. Can Reputation Discipline the Gig Economy? Experimental Evidence from an Online Labor Market. Retrieved February 22, 2016 from http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2696299"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1093\/pan\/mpr057"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.2307\/2095230"},{"volume-title":"Freelancing in America: A national survey of the new workforce","author":"Berland Edelman","key":"e_1_2_1_7_1","unstructured":"Edelman Berland . 2014. Freelancing in America: A national survey of the new workforce . Freelancers Union and Elance-oDesk . Edelman Berland. 2014. Freelancing in America: A national survey of the new workforce. Freelancers Union and Elance-oDesk."},{"key":"e_1_2_1_8_1","volume-title":"Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination","author":"Bertrand Marianne","year":"2015","unstructured":"Marianne Bertrand and Sendhil Mullainathan . 2003. Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination . National Bureau of Economic Research . Retrieved December 14, 2015 from http:\/\/www.nber.org\/papers\/w9873 Marianne Bertrand and Sendhil Mullainathan. 2003. Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination. National Bureau of Economic Research. Retrieved December 14, 2015 from http:\/\/www.nber.org\/papers\/w9873"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.2307\/2654937"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.61.1.80"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11199-011-0012-7"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2013.05.009"},{"key":"e_1_2_1_13_1","unstructured":"R. H. B. Christensen. 2015. ordinal--Regression Models for Ordinal Data.  R. H. B. Christensen. 2015. ordinal--Regression Models for Ordinal Data."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025495"},{"key":"e_1_2_1_15_1","first-page":"2015","volume-title":"Business Insider. Retrieved","author":"Cook James","year":"2015","unstructured":"James Cook . 2015 . Uber's internal charts show how its driver-rating system actually works . Business Insider. Retrieved May 24, 2016 from http:\/\/www.businessinsider.com\/leaked-charts-show-how-ubers-driver-rating-system-works- 2015 - 2012 James Cook. 2015. Uber's internal charts show how its driver-rating system actually works. Business Insider. Retrieved May 24, 2016 from http:\/\/www.businessinsider.com\/leaked-charts-show-how-ubers-driver-rating-system-works-2015-2"},{"key":"e_1_2_1_16_1","unstructured":"Sara Cook. 2013. Providing Feedback on Student Writing. Retrieved from http:\/\/www.sjsu.edu\/aanapisi\/docs\/ProvidingFeedbackonStudentWritingbySaraCook.pdf  Sara Cook. 2013. Providing Feedback on Student Writing. Retrieved from http:\/\/www.sjsu.edu\/aanapisi\/docs\/ProvidingFeedbackonStudentWritingbySaraCook.pdf"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642713"},{"key":"e_1_2_1_18_1","volume-title":"Vox. Retrieved","author":"Crockett Emily","year":"2016","unstructured":"Emily Crockett . 2016 . Airbnb and the \"sharing economy\" still suffer from old-fashioned racism . Vox. Retrieved May 22, 2016 from http:\/\/www.vox.com\/2015\/12\/14\/10113432\/airbnb-sharing-economy-racism Emily Crockett. 2016. Airbnb and the \"sharing economy\" still suffer from old-fashioned racism. Vox. Retrieved May 22, 2016 from http:\/\/www.vox.com\/2015\/12\/14\/10113432\/airbnb-sharing-economy-racism"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.2466\/pr0.1973.32.1.261"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702189"},{"key":"e_1_2_1_21_1","volume-title":"The Verge. Retrieved","author":"Dzieza Josh","year":"2015","unstructured":"Josh Dzieza . 2015 . The rating game: how Uber and its peers turned us into horrible bosses . The Verge. Retrieved May 22, 2016 from http:\/\/www.theverge.com\/2015\/10\/28\/9625968\/rating-system-on-demand-economy-uber-olive-garden Josh Dzieza. 2015. The rating game: how Uber and its peers turned us into horrible bosses. The Verge. Retrieved May 22, 2016 from http:\/\/www.theverge.com\/2015\/10\/28\/9625968\/rating-system-on-demand-economy-uber-olive-garden"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.111.1.3"},{"key":"e_1_2_1_23_1","first-page":"14","article-title":"Digital Discrimination: The Case of Airbnb. com","author":"Edelman Benjamin G.","year":"2014","unstructured":"Benjamin G. Edelman and Michael Luca . 2014 . Digital Discrimination: The Case of Airbnb. com . Harvard Business School NOM Unit Working Paper , 14 - 054 . Retrieved March 21, 2015 from http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2377353 Benjamin G. Edelman and Michael Luca. 2014. Digital Discrimination: The Case of Airbnb. com. Harvard Business School NOM Unit Working Paper, 14-054. Retrieved March 21, 2015 from http:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=2377353","journal-title":"Harvard Business School NOM Unit Working Paper"},{"key":"e_1_2_1_24_1","first-page":"16","article-title":"Racial Discrimination in the Sharing Economy: Evidence from a Field Experiment","author":"Edelman Benjamin","year":"2015","unstructured":"Benjamin Edelman , Michael Luca , and Dan Svirsky . 2015 . Racial Discrimination in the Sharing Economy: Evidence from a Field Experiment . Harvard Business School NOM Unit Working Paper , 16 - 069 . Retrieved January 13, 2016 from http:\/\/www.benedelman.org\/publications\/airbnb-guest-discrimination-2015-12-09.pdf Benjamin Edelman, Michael Luca, and Dan Svirsky. 2015. Racial Discrimination in the Sharing Economy: Evidence from a Field Experiment. Harvard Business School NOM Unit Working Paper, 16-069. Retrieved January 13, 2016 from http:\/\/www.benedelman.org\/publications\/airbnb-guest-discrimination-2015-12-09.pdf","journal-title":"Harvard Business School NOM Unit Working Paper"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10869-015-9412-y"},{"key":"e_1_2_1_26_1","volume-title":"The New Yorker. Retrieved","author":"Fan Jiayang","year":"2015","unstructured":"Jiayang Fan . 2015 . How China Wants to Rate Its Citizens . The New Yorker. Retrieved May 22, 2016 from http:\/\/www.newyorker.com\/news\/daily-comment\/how-china-wants-to-rate-its-citizens Jiayang Fan. 2015. How China Wants to Rate Its Citizens. The New Yorker. Retrieved May 22, 2016 from http:\/\/www.newyorker.com\/news\/daily-comment\/how-china-wants-to-rate-its-citizens"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bodyim.2012.06.006"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3386\/w22776"},{"volume-title":"Bayesian data analysis","author":"Gelman Andrew","key":"e_1_2_1_29_1","unstructured":"Andrew Gelman , John B Carlin , Hal S Stern , David Dunson , Aki Vehtari , and Donald B Rubin . 2014. Bayesian data analysis . Taylor & Francis . Andrew Gelman, John B Carlin, Hal S Stern, David Dunson, Aki Vehtari, and Donald B Rubin. 2014. Bayesian data analysis. Taylor & Francis."},{"key":"e_1_2_1_30_1","volume-title":"Orchestrating Impartiality: The Impact of","author":"Goldin Claudia","year":"1997","unstructured":"Claudia Goldin and Cecilia Rouse . 1997 . Orchestrating Impartiality: The Impact of . National Bureau of Economic Research . Claudia Goldin and Cecilia Rouse. 1997. Orchestrating Impartiality: The Impact of. National Bureau of Economic Research."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1006\/obhd.1993.1034"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0037503"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998327"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1075-2935(96)90004-5"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753370"},{"key":"e_1_2_1_36_1","volume-title":"NYU. Retrieved","author":"Horton John","year":"2015","unstructured":"John Horton and Joseph Golden . 2015 . Reputation Inflation: Evidence from an Online Labor Market. Work. Pap ., NYU. Retrieved February 22, 2016 from http:\/\/squash.tamu.edu\/common\/files\/workshops\/Theory and Experimental Economics\/ 2015_3_5_John_Horton.pdf John Horton and Joseph Golden. 2015. Reputation Inflation: Evidence from an Online Labor Market. Work. Pap., NYU. Retrieved February 22, 2016 from http:\/\/squash.tamu.edu\/common\/files\/workshops\/Theory and Experimental Economics\/2015_3_5_John_Horton.pdf"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10683-011-9273-9"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.83.2.179"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858123"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026015"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365974"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691611406925"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038558.2038560"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038558.2038560"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2007.308"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702548"},{"key":"e_1_2_1_49_1","volume-title":"Salon. Retrieved","author":"Leong Nancy","year":"2014","unstructured":"Nancy Leong . 2014 . The sharing economy has a race problem . Salon. Retrieved May 22, 2016 from http:\/\/www.salon.com\/2014\/11\/02\/the_sharing_economy_has_a_race_problem\/ Nancy Leong. 2014. The sharing economy has a race problem. Salon. Retrieved May 22, 2016 from http:\/\/www.salon.com\/2014\/11\/02\/the_sharing_economy_has_a_race_problem\/"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-014-0532-5"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1111\/1475-6773.11194"},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"Connor McMahon Isaac Johnson and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies.  Connor McMahon Isaac Johnson and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies.","DOI":"10.1609\/icwsm.v11i1.14883"},{"key":"e_1_2_1_53_1","unstructured":"Meta. 2016. Objective Revision Evaluation Service -- Meta discussion about Wikimedia projects. Retrieved from https:\/\/meta.wikimedia.org\/w\/index.php?title=Objective_Revision_Evaluation_Service&oldid=15597359  Meta. 2016. Objective Revision Evaluation Service -- Meta discussion about Wikimedia projects. Retrieved from https:\/\/meta.wikimedia.org\/w\/index.php?title=Objective_Revision_Evaluation_Service&oldid=15597359"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1037\/sgd0000050"},{"key":"e_1_2_1_55_1","volume-title":"Motherboard. Retrieved","author":"Neal Meghan","year":"2014","unstructured":"Meghan Neal . 2014 . The Sharing Economy Gets Accused of Racism . Motherboard. Retrieved May 22, 2016 from http:\/\/motherboard.vice.com\/blog\/are-airbnb-users-really-being-racist Meghan Neal. 2014. The Sharing Economy Gets Accused of Racism. Motherboard. Retrieved May 22, 2016 from http:\/\/motherboard.vice.com\/blog\/are-airbnb-users-really-being-racist"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507188"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.5465\/amr.1980.4288749"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1037\/1089-2699.6.1.101"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.83.1.44"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.79.6.909"},{"key":"e_1_2_1_61_1","volume-title":"Retrieved","author":"Quattrone Giovanni","year":"2016","unstructured":"Giovanni Quattrone , Davide Proserpio , Daniele Quercia , Licia Capra , and Mirco Musolesi . 2016 . Who Benefits from the \"Sharing\" Economy of Airbnb? arXiv:1602.02238 {physics} . Retrieved February 25, 2016 from http:\/\/arxiv.org\/abs\/1602.02238 Giovanni Quattrone, Davide Proserpio, Daniele Quercia, Licia Capra, and Mirco Musolesi. 2016. Who Benefits from the \"Sharing\" Economy of Airbnb? arXiv:1602.02238 {physics}. Retrieved February 25, 2016 from http:\/\/arxiv.org\/abs\/1602.02238"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820026"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10683-006-4309-2"},{"key":"e_1_2_1_64_1","volume-title":"Retrieved","author":"Robinson Andrew","year":"2016","unstructured":"Andrew Robinson . 2016 . equivalence: Provides Tests and Graphics for Assessing Tests of Equivalence . Retrieved July 31, 2016 from https:\/\/cran.r-project.org\/web\/packages\/equivalence\/index.html Andrew Robinson. 2016. equivalence: Provides Tests and Graphics for Assessing Tests of Equivalence. Retrieved July 31, 2016 from https:\/\/cran.r-project.org\/web\/packages\/equivalence\/index.html"},{"key":"e_1_2_1_65_1","volume-title":"Uber's Drivers: Information Asymmetries and Control in Dynamic Work. Social Science Research Network","author":"Rosenblat Alex","year":"2015","unstructured":"Alex Rosenblat and Luke Stark . 2015. Uber's Drivers: Information Asymmetries and Control in Dynamic Work. Social Science Research Network , Rochester, NY . Retrieved November 10, 2015 from http:\/\/papers.ssrn.com\/abstract=2686227 Alex Rosenblat and Luke Stark. 2015. Uber's Drivers: Information Asymmetries and Control in Dynamic Work. Social Science Research Network, Rochester, NY. Retrieved November 10, 2015 from http:\/\/papers.ssrn.com\/abstract=2686227"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvb.2008.09.008"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31454-4_20"},{"key":"e_1_2_1_68_1","volume-title":"Retrieved","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig , Kevin Hamilton , Karrie Karahalios , and Cedric Langbort . 2014 . Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry . Retrieved August 24, 2015 from http:\/\/www-personal.umich.edu\/~csandvig\/research\/Auditing%20Algorithms%20-%20Sandvig%20-%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry. Retrieved August 24, 2015 from http:\/\/www-personal.umich.edu\/~csandvig\/research\/Auditing%20Algorithms%20-%20Sandvig%20-%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf"},{"key":"e_1_2_1_69_1","unstructured":"Christian Sandvig Kyrasto Karahalios Alan Mislove and First Look Media Works Inc. undecided. Sandvig v. Lynch. Retrieved from https:\/\/www.aclu.org\/sites\/default\/files\/field_document\/cfaa_complaint.pdf  Christian Sandvig Kyrasto Karahalios Alan Mislove and First Look Media Works Inc. undecided. Sandvig v. Lynch. Retrieved from https:\/\/www.aclu.org\/sites\/default\/files\/field_document\/cfaa_complaint.pdf"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675285"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1177\/014920630202800306"},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '08)","author":"Snow Rion","year":"2016","unstructured":"Rion Snow , Brendan O'Connor , Daniel Jurafsky , and Andrew Y. Ng . 2008. Cheap and Fast--but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks . In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '08) , 254--263. Retrieved August 1, 2016 from http:\/\/dl.acm.org\/citation.cfm?id=1613715.1613751 Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and Fast--but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '08), 254--263. Retrieved August 1, 2016 from http:\/\/dl.acm.org\/citation.cfm?id=1613715.1613751"},{"key":"e_1_2_1_73_1","volume-title":"The Washington Post. Retrieved","author":"Stark Jennifer","year":"2016","unstructured":"Jennifer Stark and Nicholas Diakopoulos . 2016 . Uber seems to offer better service in areas with more white people. That raises some tough questions . The Washington Post. Retrieved March 10, 2016 from https:\/\/www.washingtonpost.com\/news\/wonk\/wp\/2016\/03\/10\/uber-seems-to-offer-better-service-in-areas-with-more-white-people-that-raises-some-tough-questions\/ Jennifer Stark and Nicholas Diakopoulos. 2016. Uber seems to offer better service in areas with more white people. That raises some tough questions. The Washington Post. Retrieved March 10, 2016 from https:\/\/www.washingtonpost.com\/news\/wonk\/wp\/2016\/03\/10\/uber-seems-to-offer-better-service-in-areas-with-more-white-people-that-raises-some-tough-questions\/"},{"key":"e_1_2_1_74_1","volume-title":"Power Imbalance and Rating Systems. In Tenth International AAAI Conference on Web and Social Media.","author":"State Bogdan","year":"2016","unstructured":"Bogdan State , Bruno Abrahao , and Karen Cook . 2016 . Power Imbalance and Rating Systems. In Tenth International AAAI Conference on Web and Social Media. Bogdan State, Bruno Abrahao, and Karen Cook. 2016. Power Imbalance and Rating Systems. In Tenth International AAAI Conference on Web and Social Media."},{"key":"e_1_2_1_75_1","doi-asserted-by":"crossref","unstructured":"Toshiro Tango. 1998. Equivalence test and confidence interval for the difference in proportions for the paired-sample design. Statistics in medicine 17 8: 891--908.  Toshiro Tango. 1998. Equivalence test and confidence interval for the difference in proportions for the paired-sample design. Statistics in medicine 17 8: 891--908.","DOI":"10.1002\/(SICI)1097-0258(19980430)17:8<891::AID-SIM780>3.0.CO;2-B"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675278"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3058499"},{"key":"e_1_2_1_78_1","volume-title":"States Congress.","author":"United","year":"1986","unstructured":"United States Congress. 1986 . Computer Fraud and Abuse Act. Retrieved from https:\/\/www.law.cornell.edu\/uscode\/text\/18\/1030 United States Congress. 1986. Computer Fraud and Abuse Act. Retrieved from https:\/\/www.law.cornell.edu\/uscode\/text\/18\/1030"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.76.6.897"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/61.1.165"},{"key":"e_1_2_1_81_1","volume-title":"Olson","author":"Zanna Mark P.","year":"2013","unstructured":"Mark P. Zanna and James M . Olson . 2013 . Advances in Experimental Social Psychology, Volume 48 . Academic Press , San Diego. Mark P. Zanna and James M. Olson. 2013. Advances in Experimental Social Psychology, Volume 48. Academic Press, San Diego."},{"key":"e_1_2_1_82_1","volume-title":"Where Every Stay is Above Average. Social Science Research Network","author":"Zervas Georgios","year":"2016","unstructured":"Georgios Zervas , Davide Proserpio , and John Byers . 2015. A First Look at Online Reputation on Airbnb , Where Every Stay is Above Average. Social Science Research Network , Rochester, NY . Retrieved May 27, 2016 from http:\/\/papers.ssrn.com\/abstract=2554500 Georgios Zervas, Davide Proserpio, and John Byers. 2015. A First Look at Online Reputation on Airbnb, Where Every Stay is Above Average. Social Science Research Network, Rochester, NY. Retrieved May 27, 2016 from http:\/\/papers.ssrn.com\/abstract=2554500"},{"volume-title":"SAT Suite of Assessments. Retrieved","year":"2016","key":"e_1_2_1_83_1","unstructured":"2015. Essay Sample 1 Bogard . SAT Suite of Assessments. Retrieved May 27, 2016 from https:\/\/collegereadiness.collegeboard.org\/sample-questions\/essay\/1 2015. Essay Sample 1 Bogard. SAT Suite of Assessments. Retrieved May 27, 2016 from https:\/\/collegereadiness.collegeboard.org\/sample-questions\/essay\/1"},{"volume-title":"SAT Suite of Assessments. Retrieved","year":"2016","key":"e_1_2_1_84_1","unstructured":"2015. Essay Sample 2 Gioia . SAT Suite of Assessments. Retrieved May 27, 2016 from https:\/\/collegereadiness.collegeboard.org\/sample-questions\/essay\/2 2015. Essay Sample 2 Gioia. SAT Suite of Assessments. Retrieved May 27, 2016 from https:\/\/collegereadiness.collegeboard.org\/sample-questions\/essay\/2"},{"key":"e_1_2_1_85_1","volume-title":"Retrieved","author":"Version","year":"2016","unstructured":"2015. Wikipedia: Version 1. 0 Editorial Team\/Assessment. Wikipedia, the free encyclopedia . Retrieved May 27, 2016 from https:\/\/en.wikipedia.org\/w\/index.php?title=Wikipedia:Version_1.0_Editorial_Team\/Assessment&oldid=690783073 2015. Wikipedia:Version 1.0 Editorial Team\/Assessment. Wikipedia, the free encyclopedia. Retrieved May 27, 2016 from https:\/\/en.wikipedia.org\/w\/index.php?title=Wikipedia:Version_1.0_Editorial_Team\/Assessment&oldid=690783073"},{"key":"e_1_2_1_86_1","unstructured":"2016. Wikipedia:WikiProject Wikipedia\/Assessment. Wikipedia. Retrieved April 27 2017 from https:\/\/en.wikipedia.org\/w\/index.php?title=Wikipedia:WikiProject_Wikipedia\/Assessment&oldid=754983998  2016. Wikipedia:WikiProject Wikipedia\/Assessment. Wikipedia. Retrieved April 27 2017 from https:\/\/en.wikipedia.org\/w\/index.php?title=Wikipedia:WikiProject_Wikipedia\/Assessment&oldid=754983998"},{"key":"e_1_2_1_87_1","unstructured":"What is a Mechanical Turk Master? FAQ > Worker Web Site FAQs. Retrieved May 27 2016 from https:\/\/www.mturk.com\/mturk\/help?helpPage=worker#what_is_master_worker  What is a Mechanical Turk Master? FAQ > Worker Web Site FAQs. Retrieved May 27 2016 from https:\/\/www.mturk.com\/mturk\/help?helpPage=worker#what_is_master_worker"},{"key":"e_1_2_1_88_1","unstructured":"West Virginia Department of Education Teach 21 Writing Rubrics. Retrieved May 27 2016 from https:\/\/wvde.state.wv.us\/teach21\/writingrubrics\/  West Virginia Department of Education Teach 21 Writing Rubrics. Retrieved May 27 2016 from https:\/\/wvde.state.wv.us\/teach21\/writingrubrics\/"},{"key":"e_1_2_1_89_1","volume-title":"Retrieved","author":"Most Popular Baby Names by Sex and Mother's Ethnic Group","year":"2016","unstructured":"Most Popular Baby Names by Sex and Mother's Ethnic Group , New York City . NYC Open Data . Retrieved May 27, 2016 from https:\/\/data.cityofnewyork.us\/Health\/Most-Popular-Baby-Names-by-Sex-and-Mother-s-Ethnic\/25th-nujf Most Popular Baby Names by Sex and Mother's Ethnic Group, New York City. NYC Open Data. Retrieved May 27, 2016 from https:\/\/data.cityofnewyork.us\/Health\/Most-Popular-Baby-Names-by-Sex-and-Mother-s-Ethnic\/25th-nujf"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3134736","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134736","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134736","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:11:24Z","timestamp":1750212684000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3134736"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,12,6]]},"references-count":89,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2017,12,6]]}},"alternative-id":["10.1145\/3134736"],"URL":"https:\/\/doi.org\/10.1145\/3134736","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2017,12,6]]},"assertion":[{"value":"2017-12-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}