{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T23:35:36Z","timestamp":1757547336102,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":29,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T00:00:00Z","timestamp":1557705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,13]]},"DOI":"10.1145\/3308558.3313459","type":"proceedings-article","created":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T12:17:59Z","timestamp":1557749879000},"page":"1028-1038","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotations"],"prefix":"10.1145","author":[{"given":"Yuan","family":"Li","sequence":"first","affiliation":[{"name":"The University of Melbourne, Australia"}]},{"given":"Benjamin","family":"I. P. Rubinstein","sequence":"additional","affiliation":[{"name":"The University of Melbourne, Australia"}]},{"given":"Trevor","family":"Cohn","sequence":"additional","affiliation":[{"name":"The University of Melbourne, Australia"}]}],"member":"320","published-online":{"date-parts":[[2019,5,13]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas.","author":"Aydin Bahadir Ismail","year":"2014","unstructured":"Bahadir Ismail Aydin , Yavuz Selim Yilmaz , Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. 2014 . Crowdsourcing for Multiple-Choice Question Answering . Bahadir Ismail Aydin, Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. 2014. Crowdsourcing for Multiple-Choice Question Answering."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866697"},{"key":"e_1_3_2_1_3_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics","author":"Dawid Alexander Philip","year":"1979","unstructured":"Alexander Philip Dawid and Allan M Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics ( 1979 ), 20-28. Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics (1979), 20-28."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_3_2_1_5_1","series-title":"Series B (methodological)","volume-title":"Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society","author":"Dempster Arthur P","year":"1977","unstructured":"Arthur P Dempster , Nan M Laird , and Donald B Rubin . 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society . Series B (methodological) ( 1977 ), 1-38. Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society. Series B (methodological) (1977), 1-38."},{"key":"e_1_3_2_1_6_1","first-page":"26","volume-title":"Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourcing Platforms. CrowdSearch 2012 workshop at WWW(2012)","author":"Difallah Djellel E.","year":"2012","unstructured":"Djellel E. Difallah , Gianluca Demartini , and Philippe Cudre\u00b4-Mauroux . 2012 . Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourcing Platforms. CrowdSearch 2012 workshop at WWW(2012) , 26 - 30 . Djellel E. Difallah, Gianluca Demartini, and Philippe Cudre\u00b4-Mauroux. 2012. Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowdsourcing Platforms. CrowdSearch 2012 workshop at WWW(2012), 26-30."},{"key":"e_1_3_2_1_7_1","volume-title":"Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (1 ed.)","author":"Howe Jeff","year":"2008","unstructured":"Jeff Howe . 2008 . Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (1 ed.) . Crown Publishing Group , New York, NY, USA . Jeff Howe. 2008. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (1 ed.). Crown Publishing Group, New York, NY, USA."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007665907178"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i2.2537"},{"key":"e_1_3_2_1_10_1","first-page":"619","article-title":"Bayesian classifier combination","author":"Kim Hyun-Chul","year":"2012","unstructured":"Hyun-Chul Kim and Zoubin Ghahramani . 2012 . Bayesian classifier combination . In Artificial Intelligence and Statistics. 619 - 627 . Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Artificial Intelligence and Statistics. 619-627.","journal-title":"Artificial Intelligence and Statistics."},{"key":"e_1_3_2_1_11_1","unstructured":"Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097-1105.   Alex Krizhevsky Ilya Sutskever and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097-1105."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735496.2735505"},{"key":"e_1_3_2_1_13_1","volume-title":"Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods 44, 1","author":"Mason Winter","year":"2012","unstructured":"Winter Mason and Siddharth Suri . 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods 44, 1 ( 2012 ), 1-23. Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods 44, 1 (2012), 1-23."},{"key":"e_1_3_2_1_14_1","volume-title":"Yee Whye Teh, and Fernando Perez-Cruz","author":"Moreno Pablo G","year":"2015","unstructured":"Pablo G Moreno , Antonio Arte\u00b4s-Rodr\u00edguez , Yee Whye Teh, and Fernando Perez-Cruz . 2015 . Bayesian nonparametric crowdsourcing. Journal of Machine Learning Research( 2015). Pablo G Moreno, Antonio Arte\u00b4s-Rodr\u00edguez, Yee Whye Teh, and Fernando Perez-Cruz. 2015. Bayesian nonparametric crowdsourcing. Journal of Machine Learning Research(2015)."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/2735471.2735474"},{"key":"e_1_3_2_1_16_1","volume-title":"Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 207-215","author":"Novotney Scott","year":"2010","unstructured":"Scott Novotney and Chris Callison-Burch . 2010 . Cheap, fast and good enough: Automatic speech recognition with non-expert transcription . In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 207-215 . Scott Novotney and Chris Callison-Burch. 2010. Cheap, fast and good enough: Automatic speech recognition with non-expert transcription. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 207-215."},{"key":"e_1_3_2_1_17_1","first-page":"2383","article-title":"SQuAD: 100, 000+ Questions for Machine Comprehension of Text","author":"Rajpurkar Pranav","year":"2016","unstructured":"Pranav Rajpurkar , Jian Zhang , Konstantin Lopyrev , and Percy Liang . 2016 . SQuAD: 100, 000+ Questions for Machine Comprehension of Text . In EMNLP. The Association for Computational Linguistics , 2383 - 2392 . Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. In EMNLP. The Association for Computational Linguistics, 2383-2392.","journal-title":"EMNLP. The Association for Computational Linguistics"},{"key":"e_1_3_2_1_18_1","first-page":"1297","article-title":"Learning from crowds","author":"Raykar Vikas C","year":"2010","unstructured":"Vikas C Raykar , Shipeng Yu , Linda H Zhao , Gerardo Hermosillo Valadez , Charles Florin , Luca Bogoni , and Linda Moy . 2010 . Learning from crowds . Journal of Machine Learning Research 11 , Apr (2010), 1297 - 1322 . Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research 11, Apr (2010), 1297-1322.","journal-title":"Journal of Machine Learning Research 11"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2013.05.012"},{"volume-title":"Decision making and imperfection","author":"Simpson Edwin","key":"e_1_3_2_1_20_1","unstructured":"Edwin Simpson , Stephen Roberts , Ioannis Psorakis , and Arfon Smith . 2013. Dynamic bayesian combination of multiple imperfect classifiers . In Decision making and imperfection . Springer , 1-35. Edwin Simpson, Stephen Roberts, Ioannis Psorakis, and Arfon Smith. 2013. Dynamic bayesian combination of multiple imperfect classifiers. In Decision making and imperfection. Springer, 1-35."},{"key":"e_1_3_2_1_21_1","first-page":"254","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing(EMNLP '08)","author":"Snow Rion","unstructured":"Rion Snow , Brendan O'Connor , Daniel Jurafsky , and Andrew Y. Ng . 2008. Cheap and Fast-but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks . In Proceedings of the Conference on Empirical Methods in Natural Language Processing(EMNLP '08) . Association for Computational Linguistics, Stroudsburg, PA, USA , 254 - 263 . Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and Fast-but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing(EMNLP '08). Association for Computational Linguistics, Stroudsburg, PA, USA, 254-263."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13256"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350263"},{"key":"e_1_3_2_1_25_1","unstructured":"Peter Welinder Steve Branson Pietro Perona and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424-2432.   Peter Welinder Steve Branson Pietro Perona and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In Advances in neural information processing systems. 2424-2432."},{"key":"e_1_3_2_1_26_1","unstructured":"Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier R Movellan and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035-2043.   Jacob Whitehill Ting-fan Wu Jacob Bergsma Javier R Movellan and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems. 2035-2043."},{"key":"e_1_3_2_1_27_1","unstructured":"Yuchen Zhang Xi Chen Dengyong Zhou and Michael I Jordan. 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In Advances in neural information processing systems. 1260-1268.   Yuchen Zhang Xi Chen Dengyong Zhou and Michael I Jordan. 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In Advances in neural information processing systems. 1260-1268."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"e_1_3_2_1_29_1","unstructured":"Denny Zhou Sumit Basu Yi Mao and John C Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In Advances in neural information processing systems. 2195-2203.   Denny Zhou Sumit Basu Yi Mao and John C Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In Advances in neural information processing systems. 2195-2203."}],"event":{"name":"WWW '19: The Web Conference","sponsor":["IW3C2 International World Wide Web Conference Committee"],"location":"San Francisco CA USA","acronym":"WWW '19"},"container-title":["The World Wide Web Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308558.3313459","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3308558.3313459","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:02:17Z","timestamp":1750208537000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308558.3313459"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,13]]},"references-count":29,"alternative-id":["10.1145\/3308558.3313459","10.1145\/3308558"],"URL":"https:\/\/doi.org\/10.1145\/3308558.3313459","relation":{},"subject":[],"published":{"date-parts":[[2019,5,13]]},"assertion":[{"value":"2019-05-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}