{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:03:35Z","timestamp":1776107015820,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":87,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1651575, 1464376"],"award-info":[{"award-number":["1651575, 1464376"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445399","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T05:58:10Z","timestamp":1620453490000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Comparing Generic and Community-Situated Crowdsourcing for Data Validation in the Context of Recovery from Substance Use Disorders"],"prefix":"10.1145","author":[{"given":"Sabirat","family":"Rubya","sequence":"first","affiliation":[{"name":"GroupLens Research University of Minnesota, United States"}]},{"given":"Joseph","family":"Numainville","sequence":"additional","affiliation":[{"name":"University of Minnesota, United States"}]},{"given":"Svetlana","family":"Yarosh","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering University of Minnesota, United States"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819943event-place:"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3151759.3151844"},{"key":"e_1_3_2_2_3_1","volume-title":"ACIS 2012 : Proceedings of the 23rd Australasian Conference on Information Systems. Australasian Conference on Information Systems. https:\/\/openresearch-repository.anu.edu.au\/handle\/1885\/154077","author":"Alam Sultana\u00a0Lubna","year":"2012","unstructured":"Sultana\u00a0Lubna Alam and John Campbell . 2012 . Crowdsourcing Motivations in a not-for-profit GLAM context: The Australian Newspapers Digitisation Program . In ACIS 2012 : Proceedings of the 23rd Australasian Conference on Information Systems. Australasian Conference on Information Systems. https:\/\/openresearch-repository.anu.edu.au\/handle\/1885\/154077 Sultana\u00a0Lubna Alam and John Campbell. 2012. Crowdsourcing Motivations in a not-for-profit GLAM context: The Australian Newspapers Digitisation Program. In ACIS 2012 : Proceedings of the 23rd Australasian Conference on Information Systems. Australasian Conference on Information Systems. https:\/\/openresearch-repository.anu.edu.au\/handle\/1885\/154077"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935831event-place:"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3083671.3083689"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.2196\/resprot.6904"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1746259.1746260"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2938503.2938511"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13673-016-0068-z"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2017.10.102"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2809915.2809918"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441915event-place:"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.metip.2020.100022"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145356"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702338event-place:"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538900event-place:"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3154862.3154875event-place:"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-385522-0.00005-6"},{"key":"e_1_3_2_2_21_1","volume-title":"Hanging with the right crowd: Matching crowdsourcing need to crowd characteristics.AMCIS 2012 Proceedings (29 7","author":"Erickson Lee","year":"2012","unstructured":"Lee Erickson , Irene Petrick , and Eileen Trauth . 2012. Hanging with the right crowd: Matching crowdsourcing need to crowd characteristics.AMCIS 2012 Proceedings (29 7 2012 ). https:\/\/aisel.aisnet.org\/amcis2012\/proceedings\/VirtualCommunities\/3 Lee Erickson, Irene Petrick, and Eileen Trauth. 2012. Hanging with the right crowd: Matching crowdsourcing need to crowd characteristics.AMCIS 2012 Proceedings (29 7 2012). https:\/\/aisel.aisnet.org\/amcis2012\/proceedings\/VirtualCommunities\/3"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD005032.pub2PMID:"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078852"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.1450400137"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2874356"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2493432.2493481event-place:"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2800835.2800974event-place:"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2598510.2598514event-place:"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1093\/iwc"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"crossref","unstructured":"Daniel Gonz\u00e1lez Regina Motz and Libertad Tansini. 2013. Recommendations Given from Socially-Connected People Yan\u00a0Tang Demey and Herv\u00e9 Panetto (Eds.). On the Move to Meaningful Internet Systems: OTM 2013 Workshops 649\u2013655.  Daniel Gonz\u00e1lez Regina Motz and Libertad Tansini. 2013. Recommendations Given from Socially-Connected People Yan\u00a0Tang Demey and Herv\u00e9 Panetto (Eds.). On the Move to Meaningful Internet Systems: OTM 2013 Workshops 649\u2013655.","DOI":"10.1007\/978-3-642-41033-8_82"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3011141.3011173"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702402[Online;"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647362event-place:"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819936"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026032"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858327"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567988"},{"key":"e_1_3_2_2_39_1","unstructured":"Mark Irvine. [n.d.]. Facebook Ad Benchmarks for Industry Data. https:\/\/www.wordstream.com\/blog\/ws\/2017\/02\/28\/facebook-advertising-benchmarks Library Catalog: www.wordstream.com.  Mark Irvine. [n.d.]. Facebook Ad Benchmarks for Industry Data. https:\/\/www.wordstream.com\/blog\/ws\/2017\/02\/28\/facebook-advertising-benchmarks Library Catalog: www.wordstream.com."},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3281449"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3281449"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274895.3274902event-place:"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134691"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2596695.2596713"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300541"},{"key":"e_1_3_2_2_46_1","unstructured":"Nicolas Kaufmann and Thimo Schulze. 0. Worker Motivation in Crowdsourcing and Human Computation. http:\/\/www.humancomputation.com\/Program.html [Online; accessed 2019-05-13].  Nicolas Kaufmann and Thimo Schulze. 0. Worker Motivation in Crowdsourcing and Human Computation. http:\/\/www.humancomputation.com\/Program.html [Online; accessed 2019-05-13]."},{"key":"e_1_3_2_2_47_1","volume-title":"AMCIS 2011 Proceedings - All Submissions (6 8 2011","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker Motivation in Crowdsourcing \u2013 A Study on Mechanical Turk . AMCIS 2011 Proceedings - All Submissions (6 8 2011 ). https:\/\/aisel.aisnet.org\/amcis2011_submissions\/340 Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing \u2013 A Study on Mechanical Turk. AMCIS 2011 Proceedings - All Submissions (6 8 2011). https:\/\/aisel.aisnet.org\/amcis2011_submissions\/340"},{"key":"e_1_3_2_2_48_1","volume-title":"Proc. CHI 2008","author":"Kittur Aniket","year":"2008","unstructured":"Aniket Kittur , E Chi , and Bongwon Suh . 2008 . Crowdsourcing for usability: Using micro-task markets for rapid, remote, and low-cost user measurements . Proc. CHI 2008 (2008). Aniket Kittur, E Chi, and Bongwon Suh. 2008. Crowdsourcing for usability: Using micro-task markets for rapid, remote, and low-cost user measurements. Proc. CHI 2008 (2008)."},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675255"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1142\/S1363919614500315"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858115"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_3_2_2_53_1","volume-title":"Proceedings of the 1st Biannual Conference on the Design of Experimental Search & Information REtrieval Systems (DESIRES)","author":"Kutlu Mucahid","year":"2018","unstructured":"Mucahid Kutlu , Tyler McDonnell , Aashish Sheshadri , Tamer Elsayed , and Matthew Lease . 2018 . Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably . In Proceedings of the 1st Biannual Conference on the Design of Experimental Search & Information REtrieval Systems (DESIRES) (2018), 42\u201346. Mucahid Kutlu, Tyler McDonnell, Aashish Sheshadri, Tamer Elsayed, and Matthew Lease. 2018. Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably. In Proceedings of the 1st Biannual Conference on the Design of Experimental Search & Information REtrieval Systems (DESIRES) (2018), 42\u201346."},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568033event-place:"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835797"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2016.09.008"},{"key":"e_1_3_2_2_57_1","volume-title":"Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing (2013","author":"Mao Andrew","year":"2020","unstructured":"Andrew Mao , Ece Kamar , Yiling Chen , Eric Horvitz , Megan\u00a0 E. Schwamb , Chris\u00a0 J. Lintott , and Arfon\u00a0 M. Smith . [n.d.]. Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing (2013 ). https:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP13\/paper\/view\/7497 Retrieved January 9, 2020 from. Andrew Mao, Ece Kamar, Yiling Chen, Eric Horvitz, Megan\u00a0E. Schwamb, Chris\u00a0J. Lintott, and Arfon\u00a0M. Smith. [n.d.]. Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing (2013). https:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP13\/paper\/view\/7497 Retrieved January 9, 2020 from."},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2596695.2596716event-place:"},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_3_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866714"},{"key":"e_1_3_2_2_61_1","unstructured":"AA membership. 2019. Worldwide A.A. Individual and Group Membership. https:\/\/www.aa.org\/assets\/en_US\/aa-literature\/smf-132-estimates-worldwide-aa-individual-and-group-membership. Accessed: 2020-09-30.  AA membership. 2019. Worldwide A.A. Individual and Group Membership. https:\/\/www.aa.org\/assets\/en_US\/aa-literature\/smf-132-estimates-worldwide-aa-individual-and-group-membership. Accessed: 2020-09-30."},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531707event-place:"},{"key":"e_1_3_2_2_63_1","volume-title":"Fourth International AAAI Conference on Weblogs and Social Media. https:\/\/www.aaai.org\/ocs\/index.php\/ICWSM\/ICWSM10\/paper\/view\/1518 [Online; accessed 2019-05-13].","author":"Morris Meredith\u00a0Ringel","year":"2010","unstructured":"Meredith\u00a0Ringel Morris , Jaime Teevan , and Katrina Panovich . 2010 . A Comparison of Information Seeking Using Search Engines and Social Networks . Fourth International AAAI Conference on Weblogs and Social Media. https:\/\/www.aaai.org\/ocs\/index.php\/ICWSM\/ICWSM10\/paper\/view\/1518 [Online; accessed 2019-05-13]. Meredith\u00a0Ringel Morris, Jaime Teevan, and Katrina Panovich. 2010. A Comparison of Information Seeking Using Search Engines and Social Networks. Fourth International AAAI Conference on Weblogs and Social Media. https:\/\/www.aaai.org\/ocs\/index.php\/ICWSM\/ICWSM10\/paper\/view\/1518 [Online; accessed 2019-05-13]."},{"key":"e_1_3_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.4167PMID:"},{"key":"e_1_3_2_2_65_1","unstructured":"Swaprava Nath Pankaj Dayama Dinesh Garg Y. Narahari and James Zou. 2012. Threats and Trade-Offs in Resource Critical Crowdsourcing Tasks Over Networks. In Twenty-Sixth AAAI Conference on Artificial Intelligence. https:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI12\/paper\/view\/4807  Swaprava Nath Pankaj Dayama Dinesh Garg Y. Narahari and James Zou. 2012. Threats and Trade-Offs in Resource Critical Crowdsourcing Tasks Over Networks. In Twenty-Sixth AAAI Conference on Artificial Intelligence. https:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI12\/paper\/view\/4807"},{"key":"e_1_3_2_2_66_1","unstructured":"1615 L.\u00a0St NW Suite 800Washington and DC\u00a020036USA202-419-4300 | Main202-857-8562 | Fax202-419-4372 |\u00a0Media Inquiries. [n.d.]. How Mechanical Turk Works. https:\/\/www.pewresearch.org\/internet\/2016\/07\/11\/research-in-the-crowdsourcing-age-a-case-study\/pi_2016-07-11_mechanical-turk_0-01\/  1615 L.\u00a0St NW Suite 800Washington and DC\u00a020036USA202-419-4300 | Main202-857-8562 | Fax202-419-4372 |\u00a0Media Inquiries. [n.d.]. How Mechanical Turk Works. https:\/\/www.pewresearch.org\/internet\/2016\/07\/11\/research-in-the-crowdsourcing-age-a-case-study\/pi_2016-07-11_mechanical-turk_0-01\/"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3335081"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660118"},{"key":"e_1_3_2_2_69_1","volume-title":"An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. In Fifth International AAAI Conference on Weblogs and Social Media (2011","author":"Rogstadius Jakob","year":"2019","unstructured":"Jakob Rogstadius , Vassilis Kostakos , Aniket Kittur , Boris Smus , Jim Laredo , and Maja Vukovic . [n.d.]. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. In Fifth International AAAI Conference on Weblogs and Social Media (2011 ). https:\/\/www.aaai.org\/ocs\/index.php\/ICWSM\/ICWSM11\/paper\/view\/2778 Retrieved April 11, 2019 from. Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. [n.d.]. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. In Fifth International AAAI Conference on Weblogs and Social Media (2011). https:\/\/www.aaai.org\/ocs\/index.php\/ICWSM\/ICWSM11\/paper\/view\/2778 Retrieved April 11, 2019 from."},{"key":"e_1_3_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145238"},{"key":"e_1_3_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14505001119"},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3048431"},{"key":"e_1_3_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298933"},{"key":"e_1_3_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134726"},{"key":"e_1_3_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557181event-place:"},{"key":"e_1_3_2_2_76_1","article-title":"Towards a characterization of crowdsourcing practices","volume":"7","author":"Schenk Eric","year":"2011","unstructured":"Eric Schenk and Claude Guittard . 2011 . Towards a characterization of crowdsourcing practices . Journal of Innovation Economics Management n\u00b0 7 , 1 (6 4 2011), 93\u2013107. Eric Schenk and Claude Guittard. 2011. Towards a characterization of crowdsourcing practices. Journal of Innovation Economics Management n\u00b07, 1 (6 4 2011), 93\u2013107.","journal-title":"Journal of Innovation Economics Management n\u00b0"},{"key":"e_1_3_2_2_77_1","volume-title":"Cognitive Surplus: How Technology Makes Consumers into Collaborators","author":"Shirky Clay","year":"2010","unstructured":"Clay Shirky . 2010 . Cognitive Surplus: How Technology Makes Consumers into Collaborators . Penguin. Google-Books-ID : PjeTO822t_4C. Clay Shirky. 2010. Cognitive Surplus: How Technology Makes Consumers into Collaborators. Penguin. Google-Books-ID: PjeTO822t_4C."},{"key":"e_1_3_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1083-6101.2005.tb00270.x"},{"key":"e_1_3_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1038\/nmeth.2855"},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025477event-place:"},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.7189\/jogh.08.010502"},{"key":"e_1_3_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820063"},{"key":"e_1_3_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466468[Online;"},{"key":"e_1_3_2_2_84_1","volume-title":"The Effects of Performance-Contingent Financial Incentives in Online Labor Markets. In Twenty-Seventh AAAI Conference on Artificial Intelligence (2013","author":"Yin Ming","year":"2020","unstructured":"Ming Yin , Yiling Chen , and Yu-An Sun . [n.d.]. The Effects of Performance-Contingent Financial Incentives in Online Labor Markets. In Twenty-Seventh AAAI Conference on Artificial Intelligence (2013 ). https:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI13\/paper\/view\/6301 Retrieved January 9, 2020 from. Ming Yin, Yiling Chen, and Yu-An Sun. [n.d.]. The Effects of Performance-Contingent Financial Incentives in Online Labor Markets. In Twenty-Seventh AAAI Conference on Artificial Intelligence (2013). https:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI13\/paper\/view\/6301 Retrieved January 9, 2020 from."},{"key":"e_1_3_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSCC.2012.20"},{"key":"e_1_3_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819953"},{"key":"e_1_3_2_2_87_1","volume-title":"SIGIR 2010 workshop on crowdsourcing for search evaluation. 17\u201320","author":"Zhu Dongqing","year":"2010","unstructured":"Dongqing Zhu and Ben Carterette . 2010 . An analysis of assessor behavior in crowdsourced preference judgments . In SIGIR 2010 workshop on crowdsourcing for search evaluation. 17\u201320 . Dongqing Zhu and Ben Carterette. 2010. An analysis of assessor behavior in crowdsourced preference judgments. In SIGIR 2010 workshop on crowdsourcing for search evaluation. 17\u201320."}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445399","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445399","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445399","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:34Z","timestamp":1750195714000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445399"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":87,"alternative-id":["10.1145\/3411764.3445399","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445399","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}