{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:40:10Z","timestamp":1750297210744,"version":"3.41.0"},"reference-count":0,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,6,1]],"date-time":"2024-06-01T00:00:00Z","timestamp":1717200000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p>Information retrieval systems play a crucial role in addressing users' information needs by aiding their exploration of vast collections of information. This thesis is framed in a critical information retrieval research aspect: evaluation. In particular, we propose new approaches for creating annotated test collections. Such collections are essential for evaluating retrieval systems' effectiveness in controlled experiments. Reflecting real-world conditions accurately in these test collections is pivotal for progress in the field.<\/jats:p>\n          <jats:p>We aim to introduce innovative techniques for efficiently assembling reliable test collections, facilitating broader research and development in information retrieval. The thesis first proposes a new method for building new pooled test collections without requiring costly evaluation campaigns [Otero et al., 2021b]. This approach simplifies and economizes the process of building new benchmarks. Then, we introduce a novel adjudication method for determining which pooled documents warrant human judgment, aiming to reduce the need for extensive expert assessments [Otero et al., 2023a]. This method is both cost-effective and efficient. Additionally, the thesis presents a fresh perspective on evaluating adjudication methods, emphasizing statistical significance, an aspect often overlooked in previous document adjudication research [Otero et al., 2023b]. As a demonstration of the methods explored in this thesis, we applied them to develop a new test collection whose construction process we describe here as an example of the use of reduced-budget methods [Otero et al., 2021a, 2020].<\/jats:p>\n          <jats:p>In summary, this thesis integrates established information retrieval knowledge with new methodologies to create annotated collections that are both cost-effective and reliable. This fusion is crucial for advancing the development of more effective retrieval systems.<\/jats:p>\n          <jats:p>\n            <jats:bold>Awarded by<\/jats:bold>\n            : Universidade da Coru\u00f1a, A Coru\u00f1a, Spain on 5 April 2024.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Supervised by<\/jats:bold>\n            : Javier Parapar, \u00c1lvaro Barreiro.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Available at<\/jats:bold>\n            : http:\/\/hdl.handle.net\/2183\/36378.\n          <\/jats:p>","DOI":"10.1145\/3687273.3687296","type":"journal-article","created":{"date-parts":[[2024,8,7]],"date-time":"2024-08-07T22:24:53Z","timestamp":1723069493000},"page":"1-2","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["New Algorithms and Methodologies for Building Information Retrieval Collections"],"prefix":"10.1145","volume":"58","author":[{"given":"David","family":"Otero","sequence":"first","affiliation":[{"name":"Universidade da Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,8,7]]},"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687273.3687296","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3687273.3687296","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:10:01Z","timestamp":1750295401000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687273.3687296"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6]]},"references-count":0,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1145\/3687273.3687296"],"URL":"https:\/\/doi.org\/10.1145\/3687273.3687296","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2024,6]]},"assertion":[{"value":"2024-08-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}