{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,7]],"date-time":"2024-08-07T07:41:56Z","timestamp":1723016516250},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>In crowdsourcing, existing efforts mainly use real datasets collected from crowdsourcing as test datasets to evaluate the effectiveness of aggregation algorithms. However, these work ignore the fact that the datasets obtained by crowdsourcing are usually sparse and imbalanced due to limited budget. As a result, applying the same aggregation algorithm on different datasets often show contradicting conclusions. For example, on the RTE dataset, Dawid and Skene model performs significantly better than Majority Voting, while on the LableMe dataset, the experiments give the opposite conclusion. It is challenging to obtain comprehensive and balanced datasets at a low cost. To our best knowledge, little effort have been made to the fair evaluation of aggregation algorithms. To fill in this gap, we propose a novel method named TDG4Crowd  that can automatically generate comprehensive and balanced datasets. Using Kullback Leibler divergence and Kolmogorov\u2013Smirnov test, the experiment results show the superior of our method compared with others. Aggregation algorithms also perform more consistently on the synthetic datasets generated using our method.<\/jats:p>","DOI":"10.24963\/ijcai.2023\/333","type":"proceedings-article","created":{"date-parts":[[2023,8,11]],"date-time":"2023-08-11T08:31:30Z","timestamp":1691742690000},"page":"2984-2992","source":"Crossref","is-referenced-by-count":0,"title":["TDG4Crowd:Test Data Generation for Evaluation of Aggregation Algorithms in Crowdsourcing"],"prefix":"10.24963","author":[{"given":"Yili","family":"Fang","sequence":"first","affiliation":[{"name":"Zhejiang Gongshang University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chaojie","family":"Shen","sequence":"additional","affiliation":[{"name":"Zhejiang Gongshang University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Huamao","family":"Gu","sequence":"additional","affiliation":[{"name":"Zhejiang Gongshang University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tao","family":"Han","sequence":"additional","affiliation":[{"name":"Zhejiang Gongshang University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xinyi","family":"Ding","sequence":"additional","affiliation":[{"name":"Zhejiang Gongshang University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"10584","event":{"number":"32","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"acronym":"IJCAI-2023","name":"Thirty-Second International Joint Conference on Artificial Intelligence {IJCAI-23}","start":{"date-parts":[[2023,8,19]]},"theme":"Artificial Intelligence","location":"Macau, SAR China","end":{"date-parts":[[2023,8,25]]}},"container-title":["Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2023,8,11]],"date-time":"2023-08-11T08:45:57Z","timestamp":1691743557000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2023\/333"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2023,8]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2023\/333","relation":{},"subject":[],"published":{"date-parts":[[2023,8]]}}}