{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T00:29:52Z","timestamp":1777854592346,"version":"3.51.4"},"reference-count":58,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"Ministerio de Ciencia e Innovaci\u00f3n, Agencia Estatal de Investigaci\u00f3n","award":["PLEC2021-007662"],"award-info":[{"award-number":["PLEC2021-007662"]}]},{"DOI":"10.13039\/501100004837","name":"Ministerio de Ciencia e Innovaci\u00f3n","doi-asserted-by":"publisher","award":["PID2022-137061OB-C21"],"award-info":[{"award-number":["PID2022-137061OB-C21"]}],"id":[{"id":"10.13039\/501100004837","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Conseller\u00eda de Educaci\u00f3n, Universidade e Formaci\u00f3n Profesional & ERDF","award":["ED431-B 2022\/33"],"award-info":[{"award-number":["ED431-B 2022\/33"]}]},{"name":"Conseller\u00eda de Educaci\u00f3n, Universidade e Formaci\u00f3n Profesional & ERDF","award":["accreditation 2019-2022 ED431G 2019\/01"],"award-info":[{"award-number":["accreditation 2019-2022 ED431G 2019\/01"]}]},{"name":"FPU, Ministerio de Universidades","award":["FPU20\/02659"],"award-info":[{"award-number":["FPU20\/02659"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Information Science"],"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:p>Offline evaluation of information retrieval systems depends on test collections. These datasets provide the researchers with a corpus of documents, topics and relevance judgements indicating which documents are relevant for each topic. Gathering the latter is costly, requiring human assessors to judge the documents. Therefore, experts usually judge only a portion of the corpus. The most common approach for selecting that subset is pooling. By intelligently choosing which documents to assess, it is possible to optimise the number of positive labels for a given budget. For this reason, much work has focused on developing techniques to better select which documents from the corpus merit human assessments. In this article, we propose using relevance feedback to prioritise the documents when building new pooled test collections. We explore several state-of-the-art statistical feedback methods for prioritising the documents the algorithm presents to the assessors. A thorough comparison on eight Text Retrieval Conference (TREC) datasets against strong baselines shows that, among other results, our proposals improve in retrieving relevant documents with lower assessment effort than other state-of-the-art adjudicating methods without harming the reliability, fairness and reusability.<\/jats:p>","DOI":"10.1177\/01655515231171085","type":"journal-article","created":{"date-parts":[[2023,5,27]],"date-time":"2023-05-27T01:06:40Z","timestamp":1685149600000},"page":"1379-1396","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["Relevance feedback for building pooled test collections"],"prefix":"10.1177","volume":"51","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1139-0449","authenticated-orcid":false,"given":"David","family":"Otero","sequence":"first","affiliation":[{"name":"Information Retrieval Lab, CITIC, University of A Coru\u00f1a, Spain"},{"name":"Information Retrieval Lab, CITIC, University of A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Javier","family":"Parapar","sequence":"additional","affiliation":[{"name":"Information Retrieval Lab, CITIC, University of A Coru\u00f1a, Spain"},{"name":"Information Retrieval Lab, CITIC, University of A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"\u00c1lvaro","family":"Barreiro","sequence":"additional","affiliation":[{"name":"Information Retrieval Lab, CITIC, University of A Coru\u00f1a, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2023,5,26]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009"},{"key":"e_1_3_4_3_2","volume-title":"TREC: experiment and evaluation in information retrieval","author":"Voorhees EM","year":"2005","unstructured":"Voorhees EM, Harman DK. TREC: experiment and evaluation in information retrieval. Cambridge, MA: The MIT Press, 2005."},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.2200\/S00368ED1V01Y201105ICR019"},{"key":"e_1_3_4_5_2","first-page":"355","volume-title":"Evaluation of cross-language information retrieval systems: second workshop of the cross-language evaluation forum (CLEF \u201901)","author":"Voorhees EM","year":"2001","unstructured":"Voorhees EM. The philosophy of information retrieval evaluation. In: Peters C, Braschler M, Gonzalo J, et al. (eds) Evaluation of cross-language information retrieval systems: second workshop of the cross-language evaluation forum (CLEF \u201901). Berlin: Springer, 2001, pp. 355\u2013370."},{"key":"e_1_3_4_6_2","first-page":"162","volume-title":"Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201905","author":"Sanderson M","unstructured":"Sanderson M, Zobel J. Information retrieval system evaluation: effort, sensitivity, and reliability. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201905, Salvador, Brazil, 15\u201319 August 2005, pp. 162\u2013169. New York: Association for Computing Machinery."},{"key":"e_1_3_4_7_2","first-page":"282","volume-title":"Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. SIGIR \u201998","author":"Cormack GV","unstructured":"Cormack GV, Palmer CR, Clarke CLA. Efficient construction of large test collections. In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. SIGIR \u201998 (eds Croft WB, Moffat A, Van Rijsbergen CJ, et al.), Melbourne, VIC, Australia, 24\u201328 August 1998, pp. 282\u2013289. New York: Association for Computing Machinery."},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2017.04.005"},{"key":"e_1_3_4_9_2","first-page":"375","volume-title":"Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval. SIGIR \u201907","author":"Moffat A","unstructured":"Moffat A, Webber W, Zobel J. Strategic system comparisons via targeted relevance judgments. In: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval. SIGIR \u201907 (eds Kraaij W, De Vries AP, Clarke CLA, et al.), Amsterdam, 23\u201327 July 2007, pp. 375\u2013382. New York: Association for Computing Machinery."},{"key":"e_1_3_4_10_2","first-page":"484","volume-title":"Proceedings of the 12th ACM international conference on information and knowledge management. CIKM \u201903","author":"Aslam JA","unstructured":"Aslam JA, Pavlu V, Savell R. A unified model for metasearch, pooling, and system evaluation. In: Proceedings of the 12th ACM international conference on information and knowledge management. CIKM \u201903, New Orleans, LA, 3\u20138 November 2003, pp. 484\u2013491. New York: Association for Computing Machinery."},{"key":"e_1_3_4_11_2","first-page":"33","volume-title":"Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201904","author":"Sanderson M","unstructured":"Sanderson M, Joho H. Forming test collections with no system pooling. In: Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201904, Sheffield, 25\u201329 July 2004, pp. 33\u201340. New York: Association for Computing Machinery."},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1177\/0165551507086989"},{"key":"e_1_3_4_13_2","volume-title":"Report on the need for and provision of an \u2018ideal\u2019 information retrieval test collection","author":"Sp\u00e4rck Jones K","year":"1975","unstructured":"Sp\u00e4rck Jones K, van Rijsbergen CJ. Report on the need for and provision of an \u2018ideal\u2019 information retrieval test collection. Cambridge: University Computer Laboratory, 1975."},{"key":"e_1_3_4_14_2","first-page":"307","volume-title":"Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201998","author":"Zobel J","unstructured":"Zobel J. How reliable are the results of large-scale information retrieval experiments? In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201998, Melbourne, VIC, Australia, 24\u201328 August 1998, pp. 307\u2013314. New York: Association for Computing Machinery."},{"key":"e_1_3_4_15_2","first-page":"837","volume-title":"Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval. SIGIR \u201907","author":"Cormack GV","unstructured":"Cormack GV, Lynam TR. Power and bias of subset pooling strategies. In: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval. SIGIR \u201907, Amsterdam, 23\u201327 July 2007, pp. 837\u2013838. New York: Association for Computing Machinery."},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-016-9282-6"},{"key":"e_1_3_4_17_2","first-page":"1027","volume-title":"Proceedings of the 31st annual ACM symposium on applied computing SAC \u201916","author":"Losada DE","unstructured":"Losada DE, Parapar J, Barreiro A. Feeling lucky? Multi-armed bandits for ordering judgements in pooling-based evaluation. In: Proceedings of the 31st annual ACM symposium on applied computing SAC \u201916, Pisa, 4\u20138 April 2016, pp. 1027\u20131034. New York: Association for Computing Machinery."},{"key":"e_1_3_4_18_2","first-page":"49","volume-title":"Proceedings of the 2017 ACM on conference on information and knowledge management CIKM \u201917","author":"Li D","unstructured":"Li D, Kanoulas E. Active sampling for large-scale information retrieval evaluation. In: Proceedings of the 2017 ACM on conference on information and knowledge management CIKM \u201917 (eds Lim E, Winslett M, Sanderson M, et al.), Singapore, 6\u201310 November 2017, pp. 49\u201358. New York: Association for Computing Machinery."},{"key":"e_1_3_4_19_2","volume-title":"Proceedings of the 2020 ACM SIGIR on international conference on theory of information retrieval. ICTIR \u201920","author":"Rahman MM","unstructured":"Rahman MM, Kutlu M, Elsayed T, et al. Efficient test collection construction via active learning. In: Proceedings of the 2020 ACM SIGIR on international conference on theory of information retrieval. ICTIR \u201920, Oslo, 14\u201317 September 2020."},{"key":"e_1_3_4_20_2","first-page":"3158","volume-title":"The World Wide Web conference WWW \u201919","author":"Rahman MM","unstructured":"Rahman MM, Kutlu M, Lease M. Constructing test collections using multi-armed bandits and active learning. In: The World Wide Web conference WWW \u201919, San Francisco, CA, 13\u201317 May 2019, pp. 3158\u20133164. New York: Association for Computing Machinery."},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2019.2947049"},{"key":"e_1_3_4_22_2","doi-asserted-by":"crossref","unstructured":"Allan J Harman DK Kanoulas E et al. TREC 2017 Common core track overview. In: Proceedings of the twenty-sixth text REtrieval conference. TREC \u201917 Gaithersburg MD pp. 500\u2013324 https:\/\/trec.nist.gov\/pubs\/trec26\/papers\/Overview-CC.pdf","DOI":"10.6028\/NIST.SP.500-324.core-overview"},{"key":"e_1_3_4_23_2","first-page":"407","volume-title":"Proceedings of the 27th ACM international conference on information and knowledge management CIKM \u201918","author":"Voorhees EM","unstructured":"Voorhees EM. On building fair and reusable test collections using bandit techniques. In: Proceedings of the 27th ACM international conference on information and knowledge management CIKM \u201918, Torino, 22\u201326 October 2018, pp. 407\u2013416. New York: Association for Computing Machinery."},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463249."},{"key":"e_1_3_4_25_2","volume-title":"Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval SIGIR \u201922","author":"Voorhees EM","unstructured":"Voorhees EM, Craswell N, Lin J. Too many relevants, whither Cranfield test collections? In: Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval SIGIR \u201922, Madrid, 11\u201315 July 2022."},{"key":"e_1_3_4_26_2","unstructured":"Craswell N Mitra B Yilmaz E et al. Overview of the TREC 2019 deep learning track. In: Proceedings of the twenty-eight text REtrieval conference Gaithersburg MD 13\u201315November 2019 https:\/\/trec.nist.gov\/pubs\/trec28\/papers\/OVERVIEW.DL.pdf"},{"key":"e_1_3_4_27_2","doi-asserted-by":"crossref","unstructured":"Craswell N Mitra B Yilmaz E et al. Overview of the TREC 2020 deep learning track. In: Proceedings of the twenty-ninth text REtrieval conference Gaithersburg MD 16\u201320November 2020 https:\/\/trec.nist.gov\/pubs\/trec29\/papers\/OVERVIEW.DL.pdf","DOI":"10.6028\/NIST.SP.1266.deep-overview"},{"key":"e_1_3_4_28_2","doi-asserted-by":"crossref","unstructured":"Craswell N Mitra B Yilmaz E et al. Overview of the TREC 2021 deep learning track. In: Proceedings of the thirtieth text REtrieval conference Gaithersburg MD 15\u201319November 2021 https:\/\/trec.nist.gov\/pubs\/trec30\/papers\/Overview-DL.pdf","DOI":"10.6028\/NIST.SP.500-335.deep-overview"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3412841.3441947."},{"key":"e_1_3_4_30_2","first-page":"66","volume-title":"Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901","author":"Soboroff I","unstructured":"Soboroff I, Nicholas C, Cahan P. Ranking retrieval systems without relevance judgments. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901, New Orleans, LA, 9\u201313 September 2001, pp. 66\u201373. New York: Association for Computing Machinery."},{"key":"e_1_3_4_31_2","unstructured":"Sakai T Lin CY. Ranking retrieval systems without relevance assessments: revisited. In: Proceedings of the 3rd international workshop on evaluating information access EVIA \u201910 National Institute of Informatics (NII) pp. 25\u201333 http:\/\/research.nii.ac.jp\/ntcir\/workshop\/OnlineProceedings8\/EVIA\/05-EVIA2010-SakaiT.pdf"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.102149"},{"key":"e_1_3_4_33_2","first-page":"1169","volume-title":"Proceedings of ACM SIGIR","author":"Cormack GV","unstructured":"Cormack GV, Grossman MR. Beyond pooling. In: Proceedings of ACM SIGIR, Ann Arbor, MI, 8\u201312 July 2018, pp. 1169\u20131172. New York: ACM."},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3241064"},{"key":"e_1_3_4_35_2","first-page":"2","volume-title":"Proceedings of the 2014 Australasian document computing symposium ADCS \u201914","author":"Jayasinghe GK","unstructured":"Jayasinghe GK, Webber W, Sanderson M, et al. Improving test collection pools with machine learning. In: Proceedings of the 2014 Australasian document computing symposium ADCS \u201914, Melbourne, VIC, Australia, 27\u201328 November 2014, pp. 2\u20139. New York: Association for Computing Machinery."},{"key":"e_1_3_4_36_2","first-page":"915","volume-title":"Proceedings of the 37th international ACM SIGIR conference on research and development in information retrieval SIGIR \u201914","author":"Jayasinghe GK","unstructured":"Jayasinghe GK, Webber W, Sanderson M, et al. Extending test collection pools without manual runs. In: Proceedings of the 37th international ACM SIGIR conference on research and development in information retrieval SIGIR \u201914, Gold Coast, QLD, Australia, 6\u201311 July 2014, pp. 915\u2013918. New York: Association for Computing Machinery."},{"key":"e_1_3_4_37_2","first-page":"1759","volume-title":"Proceedings of the 24th ACM international on conference on information and knowledge management CIKM \u201915","author":"Moffat A","unstructured":"Moffat A, Scholer F, Thomas P, et al. Pooled evaluation over query variations: users are as diverse as systems. In: Proceedings of the 24th ACM international on conference on information and knowledge management CIKM \u201915, Melbourne, VIC, Australia, 18\u201323 October 2015, pp. 1759\u20131762. New York: Association for Computing Machinery."},{"key":"e_1_3_4_38_2","first-page":"1895","volume-title":"Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901","author":"Lv Y","unstructured":"Lv Y, Zhai C. A comparative study of methods for estimating query language models with pseudo feedback. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901, New Orleans, LA, 9\u201313 September 2001, pp. 1895\u20131898. New York: Association for Computing Machinery."},{"key":"e_1_3_4_39_2","first-page":"111","volume-title":"Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901","author":"Lafferty J","unstructured":"Lafferty J, Zhai C. Document language models, query models, and risk minimization for information retrieval. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901, New Orleans, LA, 9\u201313 September 2001, pp. 111\u2013119. New York: Association for Computing Machinery."},{"key":"e_1_3_4_40_2","first-page":"120","volume-title":"Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901","author":"Lavrenko V","unstructured":"Lavrenko V, Croft WB. Relevance based language models. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201901, New Orleans, LA, 9\u201313 September 2001, pp. 120\u2013127. New York: Association for Computing Machinery."},{"key":"e_1_3_4_41_2","doi-asserted-by":"crossref","unstructured":"Abdul-Jaleel N Allan J Croft WB et al. UMass at TREC 2004: novelty and hard. In: Proceedings of TREC 2004 Gaithersburg MD 16\u201319 November 204 pp. 1\u201313 https:\/\/trec.nist.gov\/pubs\/trec13\/papers\/umass.novelty.hard.pdf","DOI":"10.21236\/ADA460118"},{"key":"e_1_3_4_42_2","first-page":"403","volume-title":"Proceedings of the tenth international conference on information and knowledge management CIKM \u201901","author":"Zhai C","unstructured":"Zhai C, Lafferty J. Model-based feedback in the language modeling approach to information retrieval. In: Proceedings of the tenth international conference on information and knowledge management CIKM \u201901, Atlanta, GA, 5\u201310 October 2001, pp. 403\u2013410. New York: Association for Computing Machinery."},{"key":"e_1_3_4_43_2","first-page":"1863","volume-title":"Proceedings of the 23rd ACM international conference on conference on information and knowledge management CIKM \u201914","author":"Lv Y","unstructured":"Lv Y, Zhai C. Revisiting the divergence minimization feedback model. In: Proceedings of the 23rd ACM international conference on conference on information and knowledge management CIKM \u201914, Shanghai, China, 3\u20137 November 2014, pp. 1863\u20131866. New York: Association for Computing Machinery."},{"key":"e_1_3_4_44_2","first-page":"141","volume-title":"Proceedings of the 2015 international conference on the theory of information retrieval ICTIR \u201915","author":"Hazimeh H","unstructured":"Hazimeh H, Zhai C. Axiomatic analysis of smoothing methods in language models for pseudo-relevance feedback. In: Proceedings of the 2015 international conference on the theory of information retrieval ICTIR \u201915, Northampton, MA, 27\u201330 September 2015, pp. 141\u2013150. New York: Association for Computing Machinery."},{"key":"e_1_3_4_45_2","unstructured":"Voorhees EM Soboroff I Lin J. Can old TREC collections reliably evaluate modern neural retrieval models? arXiv 2022 https:\/\/arxiv.org\/abs\/2201.11086"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/33405313412121."},{"key":"e_1_3_4_47_2","volume-title":"Rank correlation methods","author":"George Kendall M","year":"1948","unstructured":"George Kendall M. Rank correlation methods. London: Charles Griffin, 1948."},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_4_49_2","first-page":"587","volume-title":"Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201908","author":"Yilmaz E","unstructured":"Yilmaz E, Aslam JA, Robertson S. A new rank correlation coefficient for information retrieval. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201908, Singapore, 20\u201324 July 2008, pp. 587\u2013594. New York: Association for Computing Machinery."},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-1199-4"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3190580.3190586"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3451964.3451976"},{"key":"e_1_3_4_54_2","first-page":"678","volume-title":"Proceedings of the 33rd annual ACM symposium on applied computing SAC \u201918","author":"Valcarce D","unstructured":"Valcarce D, Parapar J, Barreiro A. LiMe: linear methods for pseudo-relevance feedback. In: Proceedings of the 33rd annual ACM symposium on applied computing SAC \u201918, Pau, 9\u201313 April 2018, pp. 678\u2013687. New York: Association for Computing Machinery."},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3307624.3307626"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/33972713401099."},{"key":"e_1_3_4_57_2","first-page":"1253","volume-title":"Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval SIGIR\u201919","author":"Roy D","unstructured":"Roy D, Bhatia S, Mitra M. Selecting discriminative terms for relevance model. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval SIGIR\u201919, Paris, 21\u201325 July 2019, pp. 1253\u20131256. New York: Association for Computing Machinery."},{"key":"e_1_3_4_58_2","volume-title":"Proceedings of the third international conference on advances in information retrieval theory ICTIR\u201911Bertinoro","author":"Parapar J","unstructured":"Parapar J, Barreiro A. Promoting divergent terms in the estimation of relevance models. In: Proceedings of the third international conference on advances in information retrieval theory ICTIR\u201911, Bertinoro, 12\u201314 September 2011."},{"key":"e_1_3_4_59_2","first-page":"219","volume-title":"Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201908","author":"Wang X","unstructured":"Wang X, Fang H, Zhai C. A study of methods for negative relevance feedback. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval SIGIR \u201908, Singapore, 20\u201324 July 2008, pp. 219\u2013226. New York: Association for Computing Machinery."}],"container-title":["Journal of Information Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01655515231171085","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/01655515231171085","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/01655515231171085","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T23:10:07Z","timestamp":1777504207000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/01655515231171085"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,26]]},"references-count":58,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,12]]}},"alternative-id":["10.1177\/01655515231171085"],"URL":"https:\/\/doi.org\/10.1177\/01655515231171085","relation":{},"ISSN":["0165-5515","1741-6485"],"issn-type":[{"value":"0165-5515","type":"print"},{"value":"1741-6485","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,26]]}}}