{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:05:02Z","timestamp":1755839102992,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":53,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T00:00:00Z","timestamp":1615161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"RSF","award":["19-18-00282"],"award-info":[{"award-number":["19-18-00282"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,8]]},"DOI":"10.1145\/3437963.3441831","type":"proceedings-article","created":{"date-parts":[[2021,3,6]],"date-time":"2021-03-06T04:34:28Z","timestamp":1615005268000},"page":"67-75","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["On the Impact of Predicate Complexity in Crowdsourced Classification Tasks"],"prefix":"10.1145","author":[{"given":"Jorge","family":"Ram\u00edrez","sequence":"first","affiliation":[{"name":"University of Trento, Trento, Italy"}]},{"given":"Marcos","family":"Baez","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Claude Bernard Lyon 1, Lyon, France"}]},{"given":"Fabio","family":"Casati","sequence":"additional","affiliation":[{"name":"University of Trento & ServiceNow, Trento, Italy"}]},{"given":"Luca","family":"Cernuzzi","sequence":"additional","affiliation":[{"name":"Catholic University Nuestra Se\u00f1ora de la Asunci\u00f3n, Asunci\u00f3n, Paraguay"}]},{"given":"Boualem","family":"Benatallah","sequence":"additional","affiliation":[{"name":"University of New South Wales, Sydney, Australia"}]},{"given":"Ekaterina A.","family":"Taran","sequence":"additional","affiliation":[{"name":"Tomsk Polytechnic University, Tomsk, Russian Fed."}]},{"given":"Veronika A.","family":"Malanina","sequence":"additional","affiliation":[{"name":"Tomsk Polytechnic University, Tomsk, Russian Fed."}]}],"member":"320","published-online":{"date-parts":[[2021,3,8]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jebo.2013.03.003"},{"volume-title":"Flock: Hybrid Crowd-Machine Learning Classifiers. In CSCW 2015 .","author":"Cheng Justin","key":"e_1_3_2_1_2_1","unstructured":"Justin Cheng and Michael S. Bernstein . 2015 . Flock: Hybrid Crowd-Machine Learning Classifiers. In CSCW 2015 . Justin Cheng and Michael S. Bernstein. 2015. Flock: Hybrid Crowd-Machine Learning Classifiers. In CSCW 2015 ."},{"key":"e_1_3_2_1_3_1","volume-title":"Devlin","author":"Jacob","year":"2018","unstructured":"Jacob et al. Devlin . 2018 . BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding . arXiv preprint arXiv:1810.04805 (2018). Jacob et al. Devlin. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1964.10490181"},{"volume-title":"SIGMOD 2012 .","author":"Aditya","key":"e_1_3_2_1_5_1","unstructured":"Aditya G. Parameswaran et al. 2012a. CrowdScreen: algorithms for filtering data with humans . In SIGMOD 2012 . Aditya G. Parameswaran et al. 2012a. CrowdScreen: algorithms for filtering data with humans. In SIGMOD 2012 ."},{"key":"e_1_3_2_1_6_1","volume-title":"PVLDB","volume":"10","author":"Ayush","year":"2017","unstructured":"Ayush Jain et al. 2017a. Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace . PVLDB , Vol. 10 , 7 ( 2017 ). Ayush Jain et al. 2017a. Understanding Workers, Developing Effective Tasks, and Enhancing Marketplace Dynamics: A Study of a Large Crowdsourcing Marketplace. PVLDB , Vol. 10, 7 (2017)."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"crossref","unstructured":"Angli Liu et al. 2016. Effective Crowd Annotation for Relation Extraction. In NAACL HLT 2016 .  Angli Liu et al. 2016. Effective Crowd Annotation for Relation Extraction. In NAACL HLT 2016 .","DOI":"10.18653\/v1\/N16-1104"},{"key":"e_1_3_2_1_8_1","unstructured":"Chien-Ju Ho etal 2015a. Incentivizing High Quality Crowdwork. In WWW 2015 .  Chien-Ju Ho et al. 2015a. Incentivizing High Quality Crowdwork. In WWW 2015 ."},{"volume-title":"Demographics and Dynamics of Mechanical Turk Workers. In WSDM 2018 . 135--143","author":"Eddine Djellel","key":"e_1_3_2_1_9_1","unstructured":"Djellel Eddine Difallah et al. 2018a . Demographics and Dynamics of Mechanical Turk Workers. In WSDM 2018 . 135--143 . Djellel Eddine Difallah et al. 2018a. Demographics and Dynamics of Mechanical Turk Workers. In WSDM 2018 . 135--143."},{"volume-title":"Dynamic Filter: Adaptive Query Processing with the Crowd. In HCOMP 2017 .","author":"Doren","key":"e_1_3_2_1_10_1","unstructured":"Doren Lan et al. 2017b . Dynamic Filter: Adaptive Query Processing with the Crowd. In HCOMP 2017 . Doren Lan et al. 2017b. Dynamic Filter: Adaptive Query Processing with the Crowd. In HCOMP 2017 ."},{"volume-title":"Crowdsourcing Paper Screening in Systematic Literature Reviews. In HCOMP 2017 .","author":"Evgeny","key":"e_1_3_2_1_11_1","unstructured":"Evgeny Krivosheev et al. 2017c . Crowdsourcing Paper Screening in Systematic Literature Reviews. In HCOMP 2017 . Evgeny Krivosheev et al. 2017c. Crowdsourcing Paper Screening in Systematic Literature Reviews. In HCOMP 2017 ."},{"key":"e_1_3_2_1_12_1","volume-title":"Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. ACM Comput. Surv.","author":"Florian Daniel","year":"2018","unstructured":"Florian Daniel et al. 2018 b. Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. ACM Comput. Surv. , Vol. 51 , 1 (2018), 7:1--7:40. https:\/\/doi.org\/10.1145\/3148148 10.1145\/3148148 Florian Daniel et al. 2018b. Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. ACM Comput. Surv. , Vol. 51, 1 (2018), 7:1--7:40. https:\/\/doi.org\/10.1145\/3148148"},{"volume-title":"UIST 2010 .","author":"Greg","key":"e_1_3_2_1_13_1","unstructured":"Greg Little et al. 2010. TurKit: human computation algorithms on mechanical turk . In UIST 2010 . Greg Little et al. 2010. TurKit: human computation algorithms on mechanical turk. In UIST 2010 ."},{"volume-title":"Break It Down: A Comparison of Macro- and Microtasks. In CHI 2015 . 4061--4064","author":"Justin","key":"e_1_3_2_1_14_1","unstructured":"Justin Cheng et al. 2015b . Break It Down: A Comparison of Macro- and Microtasks. In CHI 2015 . 4061--4064 . https:\/\/doi.org\/10.1145\/2702123.2702146 10.1145\/2702123.2702146 Justin Cheng et al. 2015b. Break It Down: A Comparison of Macro- and Microtasks. In CHI 2015 . 4061--4064. https:\/\/doi.org\/10.1145\/2702123.2702146"},{"volume-title":"All Those Wasted Hours: On Task Abandonment in Crowdsourcing. In WSDM 2019 . https:\/\/doi.org\/10","author":"Lei","key":"e_1_3_2_1_15_1","unstructured":"Lei Han et al. 2019 . All Those Wasted Hours: On Task Abandonment in Crowdsourcing. In WSDM 2019 . https:\/\/doi.org\/10 .1145\/3289600.3291035 10.1145\/3289600.3291035 Lei Han et al. 2019. All Those Wasted Hours: On Task Abandonment in Crowdsourcing. In WSDM 2019 . https:\/\/doi.org\/10.1145\/3289600.3291035"},{"volume-title":"SIGMOD 2011 .","author":"Michael","key":"e_1_3_2_1_16_1","unstructured":"Michael J. Franklin et al. 2011. CrowdDB: answering queries with crowdsourcing . In SIGMOD 2011 . Michael J. Franklin et al. 2011. CrowdDB: answering queries with crowdsourcing. In SIGMOD 2011 ."},{"volume-title":"LREC 2014 . 859--866","author":"Marta","key":"e_1_3_2_1_17_1","unstructured":"Marta Sabou et al. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines . In LREC 2014 . 859--866 . Marta Sabou et al. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. In LREC 2014 . 859--866."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_2_1_19_1","volume-title":"CSCW","author":"Steven","year":"2012","unstructured":"Steven Dow et al. 2012b. Shepherding the crowd yields better work . In CSCW 2012 . 1013--1022. https:\/\/doi.org\/10.1145\/2145204.2145355 10.1145\/2145204.2145355 Steven Dow et al. 2012b. Shepherding the crowd yields better work. In CSCW 2012. 1013--1022. https:\/\/doi.org\/10.1145\/2145204.2145355"},{"key":"e_1_3_2_1_20_1","volume-title":"MechanicalHeart: A Human-Machine Framework for the Classification of Phonocardiograms. PACMHCI","volume":"2","author":"William","year":"2018","unstructured":"William Callaghan et al. 2018c . MechanicalHeart: A Human-Machine Framework for the Classification of Phonocardiograms. PACMHCI , Vol. 2 , CSCW ( 2018 ). William Callaghan et al. 2018c. MechanicalHeart: A Human-Machine Framework for the Classification of Phonocardiograms. PACMHCI , Vol. 2, CSCW (2018)."},{"key":"e_1_3_2_1_21_1","volume-title":"HCOMP 2014 .","author":"Faltings Boi","year":"2014","unstructured":"Boi Faltings , Radu Jurca , Pearl Pu , and Bao Duy Tran . 2014 . Incentives to Counter Bias in Human Computation . In HCOMP 2014 . Boi Faltings, Radu Jurca, Pearl Pu, and Bao Duy Tran. 2014. Incentives to Counter Bias in Human Computation. In HCOMP 2014 ."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-60131-1_11"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_3_2_1_25_1","first-page":"1642","article-title":"Argonaut","volume":"8","author":"Haas Daniel","year":"2015","unstructured":"Daniel Haas , Jason Ansel , Lydia Gu , and Adam Marcus . 2015 . Argonaut : Macrotask Crowdsourcing for Complex Data Processing. PVLDB , Vol. 8 , 12 (2015), 1642 -- 1653 . Daniel Haas, Jason Ansel, Lydia Gu, and Adam Marcus. 2015. Argonaut: Macrotask Crowdsourcing for Complex Data Processing. PVLDB , Vol. 8, 12 (2015), 1642--1653.","journal-title":"Macrotask Crowdsourcing for Complex Data Processing. PVLDB"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_2_1_27_1","volume-title":"CSCW 2013 . https:\/\/doi.org\/10","author":"Kittur Aniket","year":"1923","unstructured":"Aniket Kittur , Jeffrey V. Nickerson , and Michael S . Bernstein et al. 2013. The future of crowd work . In CSCW 2013 . https:\/\/doi.org\/10 .1145\/2441776.244 1923 10.1145\/2441776.2441923 Aniket Kittur, Jeffrey V. Nickerson, and Michael S. Bernstein et al. 2013. The future of crowd work. In CSCW 2013 . https:\/\/doi.org\/10.1145\/2441776.2441923"},{"key":"e_1_3_2_1_28_1","volume-title":"HCOMP","author":"Krause Markus","year":"2015","unstructured":"Markus Krause and Ren\u00e9 F. Kizilcec . 2015. To Play or Not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing . In HCOMP 2015 . 102--109. Markus Krause and Ren\u00e9 F. Kizilcec. 2015. To Play or Not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing. In HCOMP 2015. 102--109."},{"key":"e_1_3_2_1_29_1","volume-title":"Combining Crowd and Machines for Multi-predicate Item Screening. PACMHCI","volume":"2","author":"Evgeny","year":"2018","unstructured":"Evgeny Krivosheev et al. 2018 . Combining Crowd and Machines for Multi-predicate Item Screening. PACMHCI , Vol. 2 , CSCW, Article 97 ( Nov. 2018 ). Evgeny Krivosheev et al. 2018. Combining Crowd and Machines for Multi-predicate Item Screening. PACMHCI , Vol. 2, CSCW, Article 97 (Nov. 2018)."},{"key":"e_1_3_2_1_30_1","volume-title":"Decoupled Weight Decay Regularization. In ICLR 2019 . https:\/\/openreview.net\/forum?id=Bkg6RiCqY7","author":"Loshchilov Ilya","year":"2019","unstructured":"Ilya Loshchilov and Frank Hutter . 2019 . Decoupled Weight Decay Regularization. In ICLR 2019 . https:\/\/openreview.net\/forum?id=Bkg6RiCqY7 Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In ICLR 2019 . https:\/\/openreview.net\/forum?id=Bkg6RiCqY7"},{"key":"e_1_3_2_1_31_1","volume-title":"Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. In CHI","author":"Mitra Tanushree","year":"2015","unstructured":"Tanushree Mitra , Clayton J. Hutto , and Eric Gilbert . 2015 . Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. In CHI 2015. 1345--1354. https:\/\/doi.org\/10.1145\/2702123.2702553 10.1145\/2702123.2702553 Tanushree Mitra, Clayton J. Hutto, and Eric Gilbert. 2015. Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. In CHI 2015. 1345--1354. https:\/\/doi.org\/10.1145\/2702123.2702553"},{"key":"e_1_3_2_1_32_1","volume-title":"Mortensen et al","author":"Michael","year":"2016","unstructured":"Michael L. Mortensen et al . 2016 . An exploration of crowdsourcing citation screening for systematic reviews. Research Synthesis Methods ( 2016). Michael L. Mortensen et al. 2016. An exploration of crowdsourcing citation screening for systematic reviews. Research Synthesis Methods (2016)."},{"key":"e_1_3_2_1_33_1","volume-title":"Combining Crowd and Expert Labels Using Decision Theoretic Active Learning. In HCOMP 2015 . 120--129","author":"Nguyen An Thanh","year":"2015","unstructured":"An Thanh Nguyen , Byron C. Wallace , and Matthew Lease . 2015 . Combining Crowd and Expert Labels Using Decision Theoretic Active Learning. In HCOMP 2015 . 120--129 . http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP15\/paper\/view\/11567 An Thanh Nguyen, Byron C. Wallace, and Matthew Lease. 2015. Combining Crowd and Expert Labels Using Decision Theoretic Active Learning. In HCOMP 2015 . 120--129. http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP15\/paper\/view\/11567"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536206.2536207"},{"key":"e_1_3_2_1_35_1","volume-title":"PLOS ONE","volume":"15","author":"Porter Nathaniel D.","year":"2020","unstructured":"Nathaniel D. Porter , Ashton M. Verdery , and S. Michael Gaddis . 2020. Enhancing big data in the social sciences with crowdsourcing: Data augmentation practices, techniques, and opportunities . PLOS ONE , Vol. 15 , 6 (06 2020 ), 1--21. https:\/\/doi.org\/10.1371\/journal.pone.0233154 10.1371\/journal.pone.0233154 Nathaniel D. Porter, Ashton M. Verdery, and S. Michael Gaddis. 2020. Enhancing big data in the social sciences with crowdsourcing: Data augmentation practices, techniques, and opportunities. PLOS ONE , Vol. 15, 6 (06 2020), 1--21. https:\/\/doi.org\/10.1371\/journal.pone.0233154"},{"key":"e_1_3_2_1_36_1","volume-title":"Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks. In HCOMP 2019 .","author":"Qarout Rehab Kamal","year":"2019","unstructured":"Rehab Kamal Qarout , Alessandro Checco , Gianluca Demartini , and Kalina Bontcheva . 2019 . Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks. In HCOMP 2019 . Rehab Kamal Qarout, Alessandro Checco, Gianluca Demartini, and Kalina Bontcheva. 2019. Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks. In HCOMP 2019 ."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13104-019-4858-z"},{"key":"e_1_3_2_1_38_1","volume-title":"Understanding the Impact of Text Highlighting in Crowdsourcing Tasks. In HCOMP 2019","volume":"7","author":"Ram\u00edrez Jorge","year":"2019","unstructured":"Jorge Ram\u00edrez , Marcos Baez , Fabio Casati , and Boualem Benatallah . 2019 . Understanding the Impact of Text Highlighting in Crowdsourcing Tasks. In HCOMP 2019 , Vol. 7 . AAAI, 144--152. Jorge Ram\u00edrez, Marcos Baez, Fabio Casati, and Boualem Benatallah. 2019. Understanding the Impact of Text Highlighting in Crowdsourcing Tasks. In HCOMP 2019 , Vol. 7. AAAI, 144--152."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406865.3418318"},{"key":"e_1_3_2_1_40_1","volume-title":"Fabio Casati, and Boualem Benatallah.","author":"Jorge Ram'i","year":"2019","unstructured":"Jorge Ram'i rez, Simone Degiacomi , Davide Zanella , Marcos B\u00e1 ez , Fabio Casati, and Boualem Benatallah. 2019 . CrowdHub : Extending crowdsourcing platforms for the controlled evaluation of tasks designs. arXiv preprint arXiv:1909.02800 (2019). arxiv: 1909.02800 http:\/\/arxiv.org\/abs\/1909.02800 Jorge Ram'i rez, Simone Degiacomi, Davide Zanella, Marcos B\u00e1 ez, Fabio Casati, and Boualem Benatallah. 2019. CrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designs. arXiv preprint arXiv:1909.02800 (2019). arxiv: 1909.02800 http:\/\/arxiv.org\/abs\/1909.02800"},{"key":"e_1_3_2_1_41_1","volume-title":"CI 2018 .","author":"Jorge Ram\u00ed","year":"2018","unstructured":"Jorge Ram\u00ed rez, Evgeny Krivosheev , Marcos B\u00e1 ez, Fabio Casati , and Boualem Benatallah . 2018 . CrowdRev: A platform for Crowd-based Screening of Literature Reviews. In Collective Intelligence , CI 2018 . Jorge Ram\u00ed rez, Evgeny Krivosheev, Marcos B\u00e1 ez, Fabio Casati, and Boualem Benatallah. 2018. CrowdRev: A platform for Crowd-based Screening of Literature Reviews. In Collective Intelligence, CI 2018 ."},{"volume-title":"CRUX: Adaptive Querying for Efficient Crowdsourced Data Extraction. In CIKM 2019 . 841--850","author":"Rekatsinas Theodoros","key":"e_1_3_2_1_42_1","unstructured":"Theodoros Rekatsinas , Amol Deshpande , and Aditya G. Parameswaran . 2019 . CRUX: Adaptive Querying for Efficient Crowdsourced Data Extraction. In CIKM 2019 . 841--850 . https:\/\/doi.org\/10.1145\/3357384.3357976 10.1145\/3357384.3357976 Theodoros Rekatsinas, Amol Deshpande, and Aditya G. Parameswaran. 2019. CRUX: Adaptive Querying for Efficient Crowdsourced Data Extraction. In CIKM 2019 . 841--850. https:\/\/doi.org\/10.1145\/3357384.3357976"},{"key":"e_1_3_2_1_43_1","volume-title":"Valentine","author":"Retelny Daniela","year":"2017","unstructured":"Daniela Retelny , Michael S. Bernstein , and Melissa A . Valentine . 2017 . No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work. PACMHCI , Vol. 1 , CSCW ( 2017), 89:1--89:23. https:\/\/doi.org\/10.1145\/3134724 10.1145\/3134724 Daniela Retelny, Michael S. Bernstein, and Melissa A. Valentine. 2017. No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work. PACMHCI , Vol. 1, CSCW (2017), 89:1--89:23. https:\/\/doi.org\/10.1145\/3134724"},{"key":"e_1_3_2_1_44_1","volume-title":"a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108","author":"Sanh Victor","year":"2019","unstructured":"Victor Sanh , Lysandre Debut , Julien Chaumond , and Thomas Wolf . 2019. DistilBERT , a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 ( 2019 ). http:\/\/arxiv.org\/abs\/1910.01108 Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). http:\/\/arxiv.org\/abs\/1910.01108"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"crossref","unstructured":"Wilson Shomir and etal 2016. Crowdsourcing Annotations for Websites' Privacy Policies: Can It Really Work?. In WWW 2018 .  Wilson Shomir and et al. 2016. Crowdsourcing Annotations for Websites' Privacy Policies: Can It Really Work?. In WWW 2018 .","DOI":"10.1145\/2872427.2883035"},{"key":"e_1_3_2_1_46_1","volume-title":"Iain James Marshall, and Byron C. Wallace","author":"Sun Yalin","year":"2016","unstructured":"Yalin Sun , Pengxiang Cheng , Shengwei Wang , Hao Lyu , Matthew Lease , Iain James Marshall, and Byron C. Wallace . 2016 . Crowdsourcing Information Extraction for Biomedical Systematic Reviews . arXiv preprint arXiv:1609.01017 (2016). Yalin Sun, Pengxiang Cheng, Shengwei Wang, Hao Lyu, Matthew Lease, Iain James Marshall, and Byron C. Wallace. 2016. Crowdsourcing Information Extraction for Biomedical Systematic Reviews. arXiv preprint arXiv:1609.01017 (2016)."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocx053"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.22215\/timreview\/963"},{"key":"e_1_3_2_1_49_1","volume-title":"Crowdsourced Selection on Multi-Attribute Data. In CIKM","author":"Weng Xueping","year":"2017","unstructured":"Xueping Weng , Guoliang Li , Huiqi Hu , and Jianhua Feng . 2017 . Crowdsourced Selection on Multi-Attribute Data. In CIKM 2017. 307--316. https:\/\/doi.org\/10.1145\/3132847.3132891 10.1145\/3132847.3132891 Xueping Weng, Guoliang Li, Huiqi Hu, and Jianhua Feng. 2017. Crowdsourced Selection on Multi-Attribute Data. In CIKM 2017. 307--316. https:\/\/doi.org\/10.1145\/3132847.3132891"},{"volume-title":"Fair Work: Crowd Work Minimum Wage with One Line of Code. In HCOMP 2019 .","author":"Whiting Mark E.","key":"e_1_3_2_1_50_1","unstructured":"Mark E. Whiting , Grant Hugh , and Michael S. Bernstein . 2019 . Fair Work: Crowd Work Minimum Wage with One Line of Code. In HCOMP 2019 . Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019. Fair Work: Crowd Work Minimum Wage with One Line of Code. In HCOMP 2019 ."},{"volume-title":"Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk. In HCOMP 2017 . https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP17\/paper\/view\/15943","author":"Wu Meng-Han","key":"e_1_3_2_1_51_1","unstructured":"Meng-Han Wu and Alexander J. Quinn . 2017 . Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk. In HCOMP 2017 . https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP17\/paper\/view\/15943 Meng-Han Wu and Alexander J. Quinn. 2017. Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk. In HCOMP 2017 . https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP17\/paper\/view\/15943"},{"key":"e_1_3_2_1_52_1","volume-title":"Wikipedia Talk Labels: Personal Attacks . (2","author":"Wulczyn Ellery","year":"2017","unstructured":"Ellery Wulczyn , Nithum Thain , and Lucas Dixon . 2017. Wikipedia Talk Labels: Personal Attacks . (2 2017 ). https:\/\/doi.org\/10.6084\/m9.figshare.4054689.v6 10.6084\/m9.figshare.4054689.v6 Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Wikipedia Talk Labels: Personal Attacks . (2 2017). https:\/\/doi.org\/10.6084\/m9.figshare.4054689.v6"},{"key":"e_1_3_2_1_53_1","volume-title":"Modeling Task Complexity in Crowdsourcing. In HCOMP 2016 .","author":"Yang Jie","year":"2016","unstructured":"Jie Yang , Judith Redi , Gianluca Demartini , and Alessandro Bozzon . 2016 . Modeling Task Complexity in Crowdsourcing. In HCOMP 2016 . Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. 2016. Modeling Task Complexity in Crowdsourcing. In HCOMP 2016 ."}],"event":{"name":"WSDM '21: The Fourteenth ACM International Conference on Web Search and Data Mining","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Virtual Event Israel","acronym":"WSDM '21"},"container-title":["Proceedings of the 14th ACM International Conference on Web Search and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441831","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3437963.3441831","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:18Z","timestamp":1750193238000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3437963.3441831"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,8]]},"references-count":53,"alternative-id":["10.1145\/3437963.3441831","10.1145\/3437963"],"URL":"https:\/\/doi.org\/10.1145\/3437963.3441831","relation":{},"subject":[],"published":{"date-parts":[[2021,3,8]]},"assertion":[{"value":"2021-03-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}