{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T21:49:24Z","timestamp":1771710564281,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":55,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,2]],"date-time":"2019-05-02T00:00:00Z","timestamp":1556755200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"DESIR","award":["731081"],"award-info":[{"award-number":["731081"]}]},{"name":"ERC Advanced Grant ALEXANDRIA","award":["339233"],"award-info":[{"award-number":["339233"]}]},{"name":"AFEL","award":["687916"],"award-info":[{"award-number":["687916"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,2]]},"DOI":"10.1145\/3290605.3300637","type":"proceedings-article","created":{"date-parts":[[2019,4,29]],"date-time":"2019-04-29T17:36:58Z","timestamp":1556559418000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":82,"title":["Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments"],"prefix":"10.1145","author":[{"given":"Christoph","family":"Hube","sequence":"first","affiliation":[{"name":"Leibniz Universt\u00e4t Hannover, Hannover, Germany"}]},{"given":"Besnik","family":"Fetahu","sequence":"additional","affiliation":[{"name":"Leibniz Universt\u00e4t Hannover, Hannover, Germany"}]},{"given":"Ujwal","family":"Gadiraju","sequence":"additional","affiliation":[{"name":"Leibniz Universt\u00e4t Hannover, Hannover, Lower Saxony, Germany"}]}],"member":"320","published-online":{"date-parts":[[2019,5,2]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209542.3209558"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209581"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226345055.001.0001"},{"key":"e_1_3_2_1_4_1","volume-title":"Balance as bias: global warming and the US prestige press. Global environmental change 14, 2","author":"Boykoff Maxwell T","year":"2004","unstructured":"Maxwell T Boykoff and Jules M Boykoff . 2004. Balance as bias: global warming and the US prestige press. Global environmental change 14, 2 ( 2004 ), 125--136. Maxwell T Boykoff and Jules M Boykoff. 2004. Balance as bias: global warming and the US prestige press. Global environmental change 14, 2 (2004), 125--136."},{"key":"e_1_3_2_1_5_1","volume-title":"The Pronouns of the Power and Solidarity","author":"Brown R\u00f3ger","year":"1960","unstructured":"R\u00f3ger Brown . 1960. Gilman. The Pronouns of the Power and Solidarity ( 1960 ). R\u00f3ger Brown. 1960. Gilman. The Pronouns of the Power and Solidarity (1960)."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858237"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837889"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159654"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v2i1.13145"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1020313710388"},{"key":"e_1_3_2_1_13_1","volume-title":"Language in the News: Discourse and Ideology in the Press","author":"Fowler Roger","unstructured":"Roger Fowler . 2013. Language in the News: Discourse and Ideology in the Press . Routledge . Roger Fowler. 2013. Language in the News: Discourse and Ideology in the Press. Routledge."},{"key":"e_1_3_2_1_14_1","volume-title":"Tiebreaker: Using language models to quantify gender bias in sports journalism. arXiv preprint arXiv:1607.03895","author":"Fu Liye","year":"2016","unstructured":"Liye Fu , Cristian Danescu-Niculescu-Mizil , and Lillian Lee . 2016 . Tiebreaker: Using language models to quantify gender bias in sports journalism. arXiv preprint arXiv:1607.03895 (2016). Liye Fu, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Tiebreaker: Using language models to quantify gender bias in sports journalism. arXiv preprint arXiv:1607.03895 (2016)."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130914"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3119930"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_3_2_1_20_1","volume-title":"Proceedings of human language technologies: The 2009 annual conference of the north american","author":"Greene Stephan","unstructured":"Stephan Greene and Philip Resnik . 2009. More than words: Syntactic packaging and implicit sentiment . In Proceedings of human language technologies: The 2009 annual conference of the north american chapter of the association for computational linguistics. Association for Computational Linguistics , 503--511. Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of human language technologies: The 2009 annual conference of the north american chapter of the association for computational linguistics. Association for Computational Linguistics, 503--511."},{"key":"e_1_3_2_1_21_1","volume-title":"A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics","author":"Holm Sture","year":"1979","unstructured":"Sture Holm . 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics ( 1979 ), 65--70. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65--70."},{"key":"e_1_3_2_1_22_1","volume-title":"Dimensions of projection. Psychological bulletin 69, 4","author":"Holmes David S","year":"1968","unstructured":"David S Holmes . 1968. Dimensions of projection. Psychological bulletin 69, 4 ( 1968 ), 248. David S Holmes. 1968. Dimensions of projection. Psychological bulletin 69, 4 (1968), 248."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.85.4.677"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2013.2291663"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3191640"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291018"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41154-0_1"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1105"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/3067421.3067431"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2015.98"},{"key":"e_1_3_2_1_32_1","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing.","author":"Kamar Ece","year":"2015","unstructured":"Ece Kamar , Ashish Kapoor , and Eric Horvitz . 2015 . Identifying and accounting for task-dependent bias in crowdsourcing . In Third AAAI Conference on Human Computation and Crowdsourcing. Ece Kamar, Ashish Kapoor, and Eric Horvitz. 2015. Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_3_2_1_33_1","volume-title":"Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems","author":"Karger David R.","year":"2011","unstructured":"David R. Karger , Sewoong Oh , and Devavrat Shah . 2011. Iterative Learning for Reliable Crowdsourcing Systems . In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011 . Proceedings of a meeting held 1214 December 2011, Granada, Spain . 1953--1961. http:\/\/papers.nips.cc\/ paper\/4396-iterative-learning-for-reliable-crowdsourcing-systems David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative Learning for Reliable Crowdsourcing Systems. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 1214 December 2011, Granada, Spain. 1953--1961. http:\/\/papers.nips.cc\/ paper\/4396-iterative-learning-for-reliable-crowdsourcing-systems"},{"key":"e_1_3_2_1_34_1","first-page":"1","article-title":"More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk","volume":"11","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk .. In AMCIS , Vol. 11. 1 -- 11 . Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk.. In AMCIS, Vol. 11. 1--11.","journal-title":"AMCIS"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_3_2_1_36_1","unstructured":"Klaus Krippendorff. 2011. Computing Krippendorff's alpha-reliability. (2011). Klaus Krippendorff. 2011. Computing Krippendorff's alpha-reliability. (2011)."},{"key":"e_1_3_2_1_37_1","volume-title":"Ihler","author":"Liu Qiang","year":"2012","unstructured":"Qiang Liu , Jian Peng , and Alexander T . Ihler . 2012 . Variational Inference for Crowdsourcing. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3--6, 2012, Lake Tahoe, Nevada, United States . 701--709. http:\/\/papers.nips.cc\/paper\/ 4627-variational-inference-for-crowdsourcing Qiang Liu, Jian Peng, and Alexander T. Ihler. 2012. Variational Inference for Crowdsourcing. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3--6, 2012, Lake Tahoe, Nevada, United States. 701--709. http:\/\/papers.nips.cc\/paper\/ 4627-variational-inference-for-crowdsourcing"},{"key":"e_1_3_2_1_38_1","volume-title":"Volume.","author":"Lyons John","year":"1970","unstructured":"John Lyons . 1970. New horizons in linguistics , Volume. ( 1970 ). John Lyons. 1970. New horizons in linguistics, Volume. (1970)."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2464464.2464485"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1050.0379"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858490"},{"key":"e_1_3_2_1_42_1","volume-title":"A Bayesian truth serum for subjective data. science 306, 5695","author":"Prelec Drazen","year":"2004","unstructured":"Drazen Prelec . 2004. A Bayesian truth serum for subjective data. science 306, 5695 ( 2004 ), 462--466. Drazen Prelec. 2004. A Bayesian truth serum for subjective data. science 306, 5695 (2004), 462--466."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553488"},{"key":"e_1_3_2_1_44_1","unstructured":"Marta Recasens Cristian Danescu-Niculescu-Mizil and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language.. In ACL (1). 1650--1659. Marta Recasens Cristian Danescu-Niculescu-Mizil and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language.. In ACL (1). 1650--1659."},{"key":"e_1_3_2_1_45_1","first-page":"17","article-title":"An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets","volume":"11","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius , Vassilis Kostakos , Aniket Kittur , Boris Smus , Jim Laredo , and Maja Vukovic . 2011 . An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets . ICWSM 11 (2011), 17 -- 21 . Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. ICWSM 11 (2011), 17--21.","journal-title":"ICWSM"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Suzanne Romaine et al. 2000. Language in society: An introduction to sociolinguistics. Oxford University Press. Suzanne Romaine et al. 2000. Language in society: An introduction to sociolinguistics. Oxford University Press.","DOI":"10.1093\/oso\/9780198731924.001.0001"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1460-2466.1999.tb02784.x"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.54.4.558"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_3_2_1_50_1","volume-title":"Qualitative analysis for social scientists","author":"Strauss Anselm L","unstructured":"Anselm L Strauss . 1987. Qualitative analysis for social scientists . Cambridge University Press . Anselm L Strauss. 1987. Qualitative analysis for social scientists. Cambridge University Press."},{"key":"e_1_3_2_1_51_1","unstructured":"James Surowiecki. 2005. The wisdom of crowds. Anchor. James Surowiecki. 2005. The wisdom of crowds. Anchor."},{"key":"e_1_3_2_1_52_1","volume-title":"Jordan","author":"Wauthier Fabian L.","year":"2011","unstructured":"Fabian L. Wauthier and Michael I . Jordan . 2011 . Bayesian Bias Mitigation for Crowdsourcing. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12--14 December 2011, Granada, Spain . 1800--1808. http:\/\/papers.nips.cc\/paper\/ 4311-bayesian-bias-mitigation-for-crowdsourcing Fabian L. Wauthier and Michael I. Jordan. 2011. Bayesian Bias Mitigation for Crowdsourcing. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12--14 December 2011, Granada, Spain. 1800--1808. http:\/\/papers.nips.cc\/paper\/ 4311-bayesian-bias-mitigation-for-crowdsourcing"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1162\/0891201041850885"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866719"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684822.2685301"}],"event":{"name":"CHI '19: CHI Conference on Human Factors in Computing Systems","location":"Glasgow Scotland Uk","acronym":"CHI '19","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3290605.3300637","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3290605.3300637","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:07Z","timestamp":1750204387000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3290605.3300637"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,2]]},"references-count":55,"alternative-id":["10.1145\/3290605.3300637","10.1145\/3290605"],"URL":"https:\/\/doi.org\/10.1145\/3290605.3300637","relation":{},"subject":[],"published":{"date-parts":[[2019,5,2]]},"assertion":[{"value":"2019-05-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}