{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T17:49:05Z","timestamp":1772905745520,"version":"3.50.1"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T00:00:00Z","timestamp":1692316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Australian Research Council (ARC) Training Centre for Information Resilience","award":["IC200100022"],"award-info":[{"award-number":["IC200100022"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>To scale the size of Information Retrieval collections, crowdsourcing has become a common way to collect relevance judgments at scale. Crowdsourcing experiments usually employ 100\u201310,000 workers, but such a number is often decided in a heuristic way. The downside is that the resulting dataset does not have any guarantee of meeting predefined statistical requirements as, for example, have enough statistical power to be able to distinguish in a statistically significant way between the relevance of two documents.<\/jats:p>\n          <jats:p>We propose a methodology adapted from literature on sound topic set size design, based on t-test and ANOVA, which aims at guaranteeing the resulting dataset to meet a predefined set of statistical requirements. We validate our approach on several public datasets.<\/jats:p>\n          <jats:p>Our results show that we can reliably estimate the recommended number of workers needed to achieve statistical power, and that such estimation is dependent on the topic, while the effect of the relevance scale is limited. Furthermore, we found that such estimation is dependent on worker features such as agreement. Finally, we describe a set of practical estimation strategies that can be used to estimate the worker set size, and we also provide results on the estimation of document set sizes.<\/jats:p>","DOI":"10.1145\/3597201","type":"journal-article","created":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T12:03:43Z","timestamp":1684757023000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance Judgments"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9191-3280","authenticated-orcid":false,"given":"Kevin","family":"Roitero","sequence":"first","affiliation":[{"name":"University of Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8215-5502","authenticated-orcid":false,"given":"David La","family":"Barbera","sequence":"additional","affiliation":[{"name":"University of Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7337-7592","authenticated-orcid":false,"given":"Michael","family":"Soprano","sequence":"additional","affiliation":[{"name":"University of Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7311-3693","authenticated-orcid":false,"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2852-168X","authenticated-orcid":false,"given":"Stefano","family":"Mizzaro","sequence":"additional","affiliation":[{"name":"University of Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6720-963X","authenticated-orcid":false,"given":"Tetsuya","family":"Sakai","sequence":"additional","affiliation":[{"name":"Waseda University, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,8,18]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2012.01.004"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0081-0"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2499178.2499184"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3130348.3130373"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_3_2_7_2","first-page":"11","volume-title":"Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing","author":"Checco Alessandro","year":"2017","unstructured":"Alessandro Checco, Kevin Roitero, Eddy Maddalena, Stefano Mizzaro, and Gianluca Demartini. 2017. Let\u2019s agree to disagree: Fixing agreement measures for crowdsourcing. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing, Steven Dow and Adam Tauman Kalai (Eds.). AAAI Press, 11\u201320. Retrieved from https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP17\/paper\/view\/15927."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3409256.3409816"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539597.3572725"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411915"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3451161"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/C2013-0-10517-X"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277934"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/4235.996017"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532033"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511761676"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3310364"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3488560.3498406"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911530"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23910"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.5555\/2502692"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/1629096.1629099"},{"key":"e_1_3_2_26_2","series-title":"Proceedings of the Eighth Text REtrieval Conference","first-page":"1","volume":"500","author":"Hawking David","year":"1999","unstructured":"David Hawking, Ellen M. Voorhees, Nick Craswell, and Peter Bailey. 1999. Overview of the TREC-8 web track\u2014Results. In Proceedings of the Eighth Text REtrieval Conference(NIST Special Publication, Vol. 500-246), Ellen M. Voorhees and Donna K. Harman (Eds.). National Institute of Standards and Technology (NIST), 1\u201318. Retrieved from http:\/\/trec.nist.gov\/pubs\/trec8\/papers\/web_results.pdf."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1573-4412(01)05005-X"},{"issue":"6","key":"e_1_3_2_28_2","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe Jeff","year":"2006","unstructured":"Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1\u20134. Retrieved from https:\/\/www.wired.com\/2006\/06\/crowds\/.","journal-title":"Wired Mag."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.4097\/kja.d.18.00292"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_2_31_2","first-page":"43","article-title":"Computing Krippendorff\u2019s alpha-reliability","volume":"1","author":"Krippendorff Klaus","year":"2008","unstructured":"Klaus Krippendorff. 2008. Computing Krippendorff\u2019s alpha-reliability. UPENN Libr. 1 (2008), 43. Retrieved from https:\/\/repository.upenn.edu\/asc_papers\/43.","journal-title":"UPENN Libr."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210033"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1037\/bul0000220"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3002172"},{"issue":"1","key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1609\/hcomp.v4i1.13287","article-title":"Why is that relevant? Collecting annotator rationales for relevance judgments","volume":"4","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? Collecting annotator rationales for relevance judgments. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 4, 1 (Sep.2016), 139\u2013148. Retrieved from https:\/\/ojs.aaai.org\/index.php\/HCOMP\/article\/view\/13287.","journal-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277824"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.29173\/cais437"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.97.2.316"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.8.4.434"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1930297500002205"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3412841.3441945"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2011.5946971"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366424.3384362"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-019-09357-w"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210052"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_55"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102688"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3239573"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9273-z"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970398.2970443"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-1199-4_8"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9059-7"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/1937055.1937066"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3494833"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07692-8_38"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321528"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564433"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767760"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3471158.3472242"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458158"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531991"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291992.3291995"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331279"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597201","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597201","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:06Z","timestamp":1750182546000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597201"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,18]]},"references-count":64,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3597201"],"URL":"https:\/\/doi.org\/10.1145\/3597201","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,18]]},"assertion":[{"value":"2022-09-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}