{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T09:51:20Z","timestamp":1747216280271,"version":"3.40.5"},"reference-count":0,"publisher":"IOS Press","isbn-type":[{"type":"print","value":"9781643684369"},{"type":"electronic","value":"9781643684376"}],"license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,9,28]]},"abstract":"<jats:p>The performance of classification models depends on the quality of training data labels. Unfortunately, acquiring good-quality annotations for many tasks is infeasible or too expensive. To address this challenge, active learning algorithms are commonly employed to select only the most relevant data for labeling. However, this is possible only when the quality and quantity of labels acquired from experts are sufficient. In many applications, a trade-off between annotating individual samples by multiple annotators to increase label quality vs. annotating new samples to increase the total number of labeled instances is necessary. In this paper, we address the issue of faulty data annotations in the context of active learning. In particular, we propose two novel annotation unification algorithms that utilize unlabeled parts of the sample space. The proposed methods require little to no intersection between samples annotated by different experts. Our experiments on four public datasets indicate the robustness and superiority of the proposed methods in both, the estimation of the annotator\u2019s reliability, and the assignment of actual labels, against the state-of-the-art algorithms and the simple majority voting.<\/jats:p>","DOI":"10.3233\/faia230397","type":"book-chapter","created":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T09:11:32Z","timestamp":1695978692000},"source":"Crossref","is-referenced-by-count":1,"title":["Robust Assignment of Labels for Active Learning with Sparse and Noisy Annotations"],"prefix":"10.3233","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2544-5052","authenticated-orcid":false,"given":"Daniel","family":"Ka\u0142u\u017ca","sequence":"first","affiliation":[{"name":"Institute of Informatics, University of Warsaw, Poland"},{"name":"QED Software, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9763-1399","authenticated-orcid":false,"given":"Andrzej","family":"Janusz","sequence":"additional","affiliation":[{"name":"Institute of Informatics, University of Warsaw, Poland"},{"name":"QED Software, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2453-4974","authenticated-orcid":false,"given":"Dominik","family":"\u015al\u0119zak","sequence":"additional","affiliation":[{"name":"Institute of Informatics, University of Warsaw, Poland"},{"name":"QED Software, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"7437","container-title":["Frontiers in Artificial Intelligence and Applications","ECAI 2023"],"original-title":[],"link":[{"URL":"https:\/\/ebooks.iospress.nl\/pdf\/doi\/10.3233\/FAIA230397","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T09:11:40Z","timestamp":1695978700000},"score":1,"resource":{"primary":{"URL":"https:\/\/ebooks.iospress.nl\/doi\/10.3233\/FAIA230397"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"ISBN":["9781643684369","9781643684376"],"references-count":0,"URL":"https:\/\/doi.org\/10.3233\/faia230397","relation":{},"ISSN":["0922-6389","1879-8314"],"issn-type":[{"type":"print","value":"0922-6389"},{"type":"electronic","value":"1879-8314"}],"subject":[],"published":{"date-parts":[[2023,9,28]]}}}