{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:03:52Z","timestamp":1755839032069,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":29,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,17]],"date-time":"2020-10-17T00:00:00Z","timestamp":1602892800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Russian Science Foundation","award":["Project No. 19-18-00282"],"award-info":[{"award-number":["Project No. 19-18-00282"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,17]]},"DOI":"10.1145\/3406865.3418318","type":"proceedings-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T23:47:14Z","timestamp":1602805634000},"page":"377-382","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["DREC"],"prefix":"10.1145","author":[{"given":"Jorge","family":"Ram\u00edrez","sequence":"first","affiliation":[{"name":"University of Trento, Trento, Italy"}]},{"given":"Marcos","family":"Baez","sequence":"additional","affiliation":[{"name":"Universit\u00e9 Claude Bernard Lyon 1, Villeurbanne, France"}]},{"given":"Fabio","family":"Casati","sequence":"additional","affiliation":[{"name":"Tomsk Polytechnic University, Tomsk, Russian Fed."}]},{"given":"Luca","family":"Cernuzzi","sequence":"additional","affiliation":[{"name":"Catholic University of Asunci\u00f3n, Asuncion, Paraguay"}]},{"given":"Boualem","family":"Benatallah","sequence":"additional","affiliation":[{"name":"UNSW Sydney, Sydney, NSW, Australia"}]}],"member":"320","published-online":{"date-parts":[[2020,10,17]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Guidelines for performing Systematic Literature Reviews in Software Engineering. 2 (01","author":"Barbara Kitchenham","year":"2007","unstructured":"Kitchenham Barbara and Stuart Charters . 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. 2 (01 2007 ). Kitchenham Barbara and Stuart Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering. 2 (01 2007)."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300773"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"e_1_3_2_1_4_1","volume-title":"Its Rapid Rise, and Its Effective Use. Perspectives on Psychological Science 13 (03","author":"Buhrmester Michael","year":"2018","unstructured":"Michael Buhrmester , Sanaz Talaifar , and Samuel Gosling . 2018. An Evaluation of Amazon's Mechanical Turk , Its Rapid Rise, and Its Effective Use. Perspectives on Psychological Science 13 (03 2018 ), 149--154. https:\/\/doi.org\/10.1177\/1745691617706516 10.1177\/1745691617706516 Michael Buhrmester, Sanaz Talaifar, and Samuel Gosling. 2018. An Evaluation of Amazon's Mechanical Turk, Its Rapid Rise, and Its Effective Use. Perspectives on Psychological Science 13 (03 2018), 149--154. https:\/\/doi.org\/10.1177\/1745691617706516"},{"key":"e_1_3_2_1_5_1","volume-title":"Nonna\u00efvet\u00e9 Among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers. Behavior research methods 46 (07","author":"Chandler Jesse","year":"2013","unstructured":"Jesse Chandler , Pam Mueller , and Gabriele Paolacci . 2013. Nonna\u00efvet\u00e9 Among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers. Behavior research methods 46 (07 2013 ). https:\/\/doi.org\/10.3758\/s13428-013-0365--7 10.3758\/s13428-013-0365--7 Jesse Chandler, Pam Mueller, and Gabriele Paolacci. 2013. Nonna\u00efvet\u00e9 Among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers. Behavior research methods 46 (07 2013). https:\/\/doi.org\/10.3758\/s13428-013-0365--7"},{"key":"e_1_3_2_1_6_1","volume-title":"Assessment Techniques, and Assurance Actions. ACM Comput. Surv. 51, 1","author":"Daniel Florian","year":"2018","unstructured":"Florian Daniel , Pavel Kucherbaev , Cinzia Cappiello , Boualem Benatallah , and Mohammad Allahbakhsh . 2018. Quality Control in Crowdsourcing: A Survey of Quality Attributes , Assessment Techniques, and Assurance Actions. ACM Comput. Surv. 51, 1 ( 2018 ), 7:1--7:40. https:\/\/doi.org\/10.1145\/3148148 10.1145\/3148148 Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. ACM Comput. Surv. 51, 1 (2018), 7:1--7:40. https:\/\/doi.org\/10.1145\/3148148"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130914"},{"key":"e_1_3_2_1_8_1","volume-title":"Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments - Dagstuhl Seminar 15481","author":"Gadiraju Ujwal","year":"2015","unstructured":"Ujwal Gadiraju , Sebastian M\u00f6ller , Martin N\u00f6llenburg , Dietmar Saupe , Sebastian Egger-Lampl , Daniel W. Archambault , and Brian Fisher . 2015. Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd . In Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments - Dagstuhl Seminar 15481 , Dagstuhl Castle , Germany, November 22--27, 2015 , Revised Contributions . 6--26. https:\/\/doi.org\/10.1007\/978--3--319--66435--4_2 10.1007\/978--3--319--66435--4_2 Ujwal Gadiraju, Sebastian M\u00f6ller, Martin N\u00f6llenburg, Dietmar Saupe, Sebastian Egger-Lampl, Daniel W. Archambault, and Brian Fisher. 2015. Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd. In Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments - Dagstuhl Seminar 15481, Dagstuhl Castle, Germany, November 22--27, 2015, Revised Contributions. 6--26. https:\/\/doi.org\/10.1007\/978--3--319--66435--4_2"},{"key":"e_1_3_2_1_9_1","volume-title":"Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford.","author":"Gebru Timnit","year":"2018","unstructured":"Timnit Gebru , Jamie Morgenstern , Briana Vecchione , Jennifer Wortman Vaughan , Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018 . Datasheets for Datasets. CoRR abs\/1803.09010 (2018). arXiv:1803.09010 http:\/\/arxiv.org\/abs\/1803.09010 Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018. Datasheets for Datasets. CoRR abs\/1803.09010 (2018). arXiv:1803.09010 http:\/\/arxiv.org\/abs\/1803.09010"},{"key":"#cr-split#-e_1_3_2_1_10_1.1","doi-asserted-by":"crossref","unstructured":"Joseph Goodman and G. Paolacci. 2017. Crowdsourcing consumer research. Journal of Consumer Research 44 (06 2017) 196--210. https:\/\/doi.org\/10.1093\/jcr\/ucx047 10.1093\/jcr","DOI":"10.1093\/jcr\/ucx047"},{"key":"#cr-split#-e_1_3_2_1_10_1.2","doi-asserted-by":"crossref","unstructured":"Joseph Goodman and G. Paolacci. 2017. Crowdsourcing consumer research. Journal of Consumer Research 44 (06 2017) 196--210. https:\/\/doi.org\/10.1093\/jcr\/ucx047","DOI":"10.1093\/jcr\/ucx047"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741102"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_2_1_13_1","volume-title":"CSCW 2013","author":"Kittur Aniket","year":"2013","unstructured":"Aniket Kittur , Jeffrey V. Nickerson , Michael S. Bernstein , Elizabeth Gerber , Aaron D. Shaw , John Zimmerman , Matt Lease , and John J. Horton . 2013. The future of crowd work. In Computer Supported Cooperative Work , CSCW 2013 , San Antonio, TX, USA, February 23--27 , 2013 . 1301--1318. https:\/\/doi.org\/10.1145\/2441776.2441923 10.1145\/2441776.2441923 Aniket Kittur, Jeffrey V. Nickerson, Michael S. Bernstein, Elizabeth Gerber, Aaron D. Shaw, John Zimmerman, Matt Lease, and John J. Horton. 2013. The future of crowd work. In Computer Supported Cooperative Work, CSCW 2013, San Antonio, TX, USA, February 23--27, 2013. 1301--1318. https:\/\/doi.org\/10.1145\/2441776.2441923"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858115"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13284"},{"key":"e_1_3_2_1_16_1","volume-title":"Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (01","author":"Mason Winter","year":"2012","unstructured":"Winter Mason and Siddharth Suri . 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (01 Mar 2012 ), 1--23. https:\/\/doi.org\/10.3758\/s13428-011-0124--6 10.3758\/s13428-011-0124--6 Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (01 Mar 2012), 1--23. https:\/\/doi.org\/10.3758\/s13428-011-0124--6"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_1_18_1","volume-title":"Enhancing big data in the social sciences with crowdsourcing: Data augmentation practices, techniques, and opportunities. PLOS ONE 15, 6 (06","author":"Porter Nathaniel D.","year":"2020","unstructured":"Nathaniel D. Porter , Ashton M. Verdery , and S. Michael Gaddis . 2020. Enhancing big data in the social sciences with crowdsourcing: Data augmentation practices, techniques, and opportunities. PLOS ONE 15, 6 (06 2020 ), 1--21. https:\/\/doi.org\/10.1371\/journal.pone.0233154 10.1371\/journal.pone.0233154 Nathaniel D. Porter, Ashton M. Verdery, and S. Michael Gaddis. 2020. Enhancing big data in the social sciences with crowdsourcing: Data augmentation practices, techniques, and opportunities. PLOS ONE 15, 6 (06 2020), 1--21. https:\/\/doi.org\/10.1371\/journal.pone.0233154"},{"key":"e_1_3_2_1_19_1","volume-title":"Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks. In HCOMP","author":"Qarout Rehab Kamal","year":"2019","unstructured":"Rehab Kamal Qarout , Alessandro Checco , Gianluca Demartini , and Kalina Bontcheva . 2019 . Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks. In HCOMP 2019. Rehab Kamal Qarout, Alessandro Checco, Gianluca Demartini, and Kalina Bontcheva. 2019. Platform-Related Factors in Repeatability and Reproducibility of Crowdsourcing Tasks. In HCOMP 2019."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13104-019-4858-z"},{"key":"e_1_3_2_1_21_1","volume-title":"Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing, HCOMP","volume":"7","author":"Ram\u00edrez Jorge","year":"2019","unstructured":"Jorge Ram\u00edrez , Marcos Baez , Fabio Casati , and Boualem Benatallah . 2019 . Understanding the Impact of Text Highlighting in Crowdsourcing Tasks . In Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2019, Vol. 7 . AAAI, 144--152. Jorge Ram\u00edrez, Marcos Baez, Fabio Casati, and Boualem Benatallah. 2019. Understanding the Impact of Text Highlighting in Crowdsourcing Tasks. In Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2019, Vol. 7. AAAI, 144--152."},{"key":"e_1_3_2_1_22_1","volume-title":"Proceedings of the Fifth International Conference on Weblogs and Social Media","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius , Vassilis Kostakos , Aniket Kittur , Boris Smus , Jim Laredo , and Maja Vukovic . 2011 . An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets . In Proceedings of the Fifth International Conference on Weblogs and Social Media , Barcelona, Catalonia, Spain, July 17--21 , 2011. http:\/\/www.aaai.org\/ocs\/index.php\/ICWSM\/ICWSM11\/paper\/view\/2778 Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. In Proceedings of the Fifth International Conference on Weblogs and Social Media, Barcelona, Catalonia, Spain, July 17--21, 2011. http:\/\/www.aaai.org\/ocs\/index.php\/ICWSM\/ICWSM11\/paper\/view\/2778"},{"key":"e_1_3_2_1_23_1","volume-title":"Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014","author":"Sabou Marta","year":"2014","unstructured":"Marta Sabou , Kalina Bontcheva , Leon Derczynski , and Arno Scharl . 2014 . Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines . In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014 , Reykjavik, Iceland, May 26--31 , 2014. 859--866. http:\/\/www.lrec-conf.org\/proceedings\/lrec2014\/summaries\/497.html Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26--31, 2014. 859--866. http:\/\/www.lrec-conf.org\/proceedings\/lrec2014\/summaries\/497.html"},{"key":"e_1_3_2_1_24_1","volume-title":"CHI Conference on Human Factors in Computing Systems, CHI'14","author":"Sampath Harini Alagarai","year":"2014","unstructured":"Harini Alagarai Sampath , Rajeev Rajeshuni , and Bipin Indurkhya . 2014 . Cognitively inspired task design to improve user performance on crowdsourcing platforms . In CHI Conference on Human Factors in Computing Systems, CHI'14 , Toronto, ON, Canada - April 26 - May 01, 2014. 3665--3674. https:\/\/doi.org\/10.1145\/2556288.2557155 10.1145\/2556288.2557155 Harini Alagarai Sampath, Rajeev Rajeshuni, and Bipin Indurkhya. 2014. Cognitively inspired task design to improve user performance on crowdsourcing platforms. In CHI Conference on Human Factors in Computing Systems, CHI'14, Toronto, ON, Canada - April 26 - May 01, 2014. 3665--3674. https:\/\/doi.org\/10.1145\/2556288.2557155"},{"key":"e_1_3_2_1_25_1","volume-title":"Saturation in Qualitative Research: Exploring its Conceptualization and Operationalization. Quality & quantity 52, 4","author":"Saunders Benjamin","year":"2018","unstructured":"Benjamin Saunders , Julius Sim , Tom Kingstone , Shula Baker , Jackie Waterfield , Bernadette Bartlam , Heather Burroughs , and Clare Jinks . 2018. Saturation in Qualitative Research: Exploring its Conceptualization and Operationalization. Quality & quantity 52, 4 ( 2018 ), 1893--1907. Benjamin Saunders, Julius Sim, Tom Kingstone, Shula Baker, Jackie Waterfield, Bernadette Bartlam, Heather Burroughs, and Clare Jinks. 2018. Saturation in Qualitative Research: Exploring its Conceptualization and Operationalization. Quality & quantity 52, 4 (2018), 1893--1907."},{"key":"e_1_3_2_1_26_1","volume-title":"Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 349","author":"Shamseer Larissa","year":"2015","unstructured":"Larissa Shamseer , David Moher , Mike Clarke , Davina Ghersi , Alessandro Liberati , Mark Petticrew , Paul Shekelle , and Lesley A Stewart . 2015. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 349 ( 2015 ). https:\/\/doi.org\/10.1136\/bmj.g7647 arXiv: https:\/\/www.bmj.com\/content\/349\/bmj.g7647.full.pdf 10.1136\/bmj.g7647 Larissa Shamseer, David Moher, Mike Clarke, Davina Ghersi, Alessandro Liberati, Mark Petticrew, Paul Shekelle, and Lesley A Stewart. 2015. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 349 (2015). https:\/\/doi.org\/10.1136\/bmj.g7647 arXiv: https:\/\/www.bmj.com\/content\/349\/bmj.g7647.full.pdf"},{"key":"e_1_3_2_1_27_1","volume-title":"Proceedings of the 2011 ACM Conference on Computer Supported Cooperative Work, CSCW 2011","author":"Shaw Aaron D.","year":"2011","unstructured":"Aaron D. Shaw , John J. Horton , and Daniel L. Chen . 2011. Designing incentives for inexpert human raters . In Proceedings of the 2011 ACM Conference on Computer Supported Cooperative Work, CSCW 2011 , Hangzhou, China, March 19--23 , 2011 . 275--284. https:\/\/doi.org\/10.1145\/1958824.1958865 10.1145\/1958824.1958865 Aaron D. Shaw, John J. Horton, and Daniel L. Chen. 2011. Designing incentives for inexpert human raters. In Proceedings of the 2011 ACM Conference on Computer Supported Cooperative Work, CSCW 2011, Hangzhou, China, March 19--23, 2011. 275--284. https:\/\/doi.org\/10.1145\/1958824.1958865"},{"key":"e_1_3_2_1_28_1","volume-title":"Quinn","author":"Wu Meng-Han","year":"2017","unstructured":"Meng-Han Wu and Alexander J . Quinn . 2017 . Confusing the Crowd : Task Instruction Quality on Amazon Mechanical Turk. In HCOMP 2017. https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP17\/paper\/view\/15943 Meng-Han Wu and Alexander J. Quinn. 2017. Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk. In HCOMP 2017. https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP17\/paper\/view\/15943"}],"event":{"name":"CSCW '20: Computer Supported Cooperative Work and Social Computing","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"Virtual Event USA","acronym":"CSCW '20"},"container-title":["Companion Publication of the 2020 Conference on Computer Supported Cooperative Work and Social Computing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3406865.3418318","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3406865.3418318","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:37Z","timestamp":1750197697000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3406865.3418318"}},"subtitle":["towards a Datasheet for Reporting Experiments in Crowdsourcing"],"short-title":[],"issued":{"date-parts":[[2020,10,17]]},"references-count":29,"alternative-id":["10.1145\/3406865.3418318","10.1145\/3406865"],"URL":"https:\/\/doi.org\/10.1145\/3406865.3418318","relation":{},"subject":[],"published":{"date-parts":[[2020,10,17]]},"assertion":[{"value":"2020-10-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}