{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T10:08:23Z","timestamp":1760609303347,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":28,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,9,14]],"date-time":"2020-09-14T00:00:00Z","timestamp":1600041600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Qatar National Research Fund","award":["NPRP 7-1313-1-245"],"award-info":[{"award-number":["NPRP 7-1313-1-245"]}]},{"name":"Good Systems a UT Austin Grand Challenge to develop responsible AI technologies"},{"name":"Wipro"},{"name":"Micron Foundation"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,9,14]]},"DOI":"10.1145\/3409256.3409837","type":"proceedings-article","created":{"date-parts":[[2020,9,5]],"date-time":"2020-09-05T10:13:32Z","timestamp":1599300812000},"page":"177-184","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Efficient Test Collection Construction via Active Learning"],"prefix":"10.1145","author":[{"given":"Md Mustafizur","family":"Rahman","sequence":"first","affiliation":[{"name":"The University of Texas at Austin, Austin, TX, USA"}]},{"given":"Mucahid","family":"Kutlu","sequence":"additional","affiliation":[{"name":"TOBB University of Economics and Technology, Ankara, Turkey"}]},{"given":"Tamer","family":"Elsayed","sequence":"additional","affiliation":[{"name":"Qatar University, Doha, Qatar"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"The University of Texas at Austin, Austin, TX, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,9,14]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_3_2_1_1_1","DOI":"10.1145\/1277741.1277755"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_2_1","DOI":"10.1145\/1321440.1321564"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_3_1","DOI":"10.1145\/1148170.1148219"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_4_1","DOI":"10.1007\/978-3-642-00958-7_27"},{"volume-title":"Aslib proceedings","author":"Cleverdon Cyril","unstructured":"Cyril Cleverdon . 1967. The Cranfield tests on index language devices . In Aslib proceedings , Vol. 19 . MCB UP Ltd , 173--194. Cyril Cleverdon. 1967. The Cranfield tests on index language devices. In Aslib proceedings, Vol. 19. MCB UP Ltd, 173--194.","key":"e_1_3_2_1_5_1"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_6_1","DOI":"10.1145\/2600428.2609601"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_7_1","DOI":"10.1145\/2983323.2983776"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_8_1","DOI":"10.1145\/3209978.3210119"},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of Second Conference on Email and Anti-Spam (CEAS).","author":"Cormack Gordon V","year":"2005","unstructured":"Gordon V Cormack and Thomas R Lynam . 2005 . Spam Corpus Creation for TREC . In Proceedings of Second Conference on Email and Anti-Spam (CEAS). Gordon V Cormack and Thomas R Lynam. 2005. Spam Corpus Creation for TREC. In Proceedings of Second Conference on Email and Anti-Spam (CEAS)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_10_1","DOI":"10.1145\/290941.291009"},{"volume-title":"Search Quality Rating Guidelines","year":"2016","unstructured":"Google. 2016. Search Quality Rating Guidelines . Inside Search : How Search Works (March 28 2016 ). www.google.com\/insidesearch\/. Google. 2016. Search Quality Rating Guidelines. Inside Search: How Search Works (March 28 2016). www.google.com\/insidesearch\/.","key":"e_1_3_2_1_11_1"},{"key":"e_1_3_2_1_12_1","volume-title":"TREC 2016 Total Recall Track Overview.. In TREC.","author":"Grossman Maura R","year":"2016","unstructured":"Maura R Grossman , Gordon V Cormack , and Adam Roegiest . 2016 . TREC 2016 Total Recall Track Overview.. In TREC. Maura R Grossman, Gordon V Cormack, and Adam Roegiest. 2016. TREC 2016 Total Recall Track Overview.. In TREC."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_13_1","DOI":"10.1007\/978-3-319-23826-5_14"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_14_1","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_2_1_15_1","volume-title":"Proceedings of the Medical Information Retrieval (MedIR) Workshop at the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval.","author":"Lease Matthew","year":"2016","unstructured":"Matthew Lease , Gordon V Cormack , An Thanh Nguyen , Thomas A Trikalinos , and Byron C Wallace . 2016 . Systematic Review is e-Discovery in Doctor's Clothing . In Proceedings of the Medical Information Retrieval (MedIR) Workshop at the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. Matthew Lease, Gordon V Cormack, An Thanh Nguyen, Thomas A Trikalinos, and Byron C Wallace. 2016. Systematic Review is e-Discovery in Doctor's Clothing. In Proceedings of the Medical Information Retrieval (MedIR) Workshop at the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_16_1","DOI":"10.5555\/188490.188495"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_17_1","DOI":"10.1145\/3132847.3133015"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_18_1","DOI":"10.5555\/2002472.2002491"},{"key":"e_1_3_2_1_19_1","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing.","author":"Nguyen An Thanh","year":"2015","unstructured":"An Thanh Nguyen , Byron C Wallace , and Matthew Lease . 2015 . Combining crowd and expert labels using decision theoretic active learning . In Third AAAI Conference on Human Computation and Crowdsourcing. An Thanh Nguyen, Byron C Wallace, and Matthew Lease. 2015. Combining crowd and expert labels using decision theoretic active learning. In Third AAAI Conference on Human Computation and Crowdsourcing."},{"volume-title":"Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 41--48","author":"Douglas","unstructured":"Douglas W Oard et al. 2004. Building an information retrieval test collection for spontaneous conversational speech . In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 41--48 . Douglas W Oard et al. 2004. Building an information retrieval test collection for spontaneous conversational speech. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 41--48.","key":"e_1_3_2_1_20_1"},{"key":"e_1_3_2_1_21_1","volume-title":"Constructing Test Collections Using Multi-Armed Bandits and Active Learning. In The World Wide Web Conference","author":"Rahman Md Mustafizur","year":"2019","unstructured":"Md Mustafizur Rahman , Mucahid Kutlu , and Matthew Lease . 2019 . Constructing Test Collections Using Multi-Armed Bandits and Active Learning. In The World Wide Web Conference ( San Francisco, CA, USA) (WWW '19). ACM, New York, NY, USA, 3158--3164. Md Mustafizur Rahman, Mucahid Kutlu, and Matthew Lease. 2019. Constructing Test Collections Using Multi-Armed Bandits and Active Learning. In The World Wide Web Conference (San Francisco, CA, USA) (WWW '19). ACM, New York, NY, USA, 3158--3164."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_22_1","DOI":"10.1145\/2396761.2396783"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_23_1","DOI":"10.1561\/1500000009"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_24_1","DOI":"10.1145\/1008992.1009001"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_25_1","DOI":"10.5555\/3019233"},{"key":"e_1_3_2_1_26_1","volume-title":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 1132--1132","author":"Soboroff Ian","year":"2013","unstructured":"Ian Soboroff . 2013 . Building Test Sollections (without running a community evaluation) . In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 1132--1132 . Ian Soboroff. 2013. Building Test Sollections (without running a community evaluation). In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 1132--1132."},{"key":"e_1_3_2_1_27_1","volume-title":"Variations in relevance judgments and the measurement of retrieval effectiveness. Information processing & management","author":"Voorhees Ellen","year":"2000","unstructured":"Ellen Voorhees . 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information processing & management , Vol. 36 , 5 ( 2000 ), 697--716. Ellen Voorhees. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information processing & management, Vol. 36, 5 (2000), 697--716."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_28_1","DOI":"10.1145\/1183614.1183633"}],"event":{"sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"],"acronym":"ICTIR '20","name":"ICTIR '20: The 2020 ACM SIGIR International Conference on the Theory of Information Retrieval","location":"Virtual Event Norway"},"container-title":["Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3409256.3409837","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3409256.3409837","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:33Z","timestamp":1750197693000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3409256.3409837"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,14]]},"references-count":28,"alternative-id":["10.1145\/3409256.3409837","10.1145\/3409256"],"URL":"https:\/\/doi.org\/10.1145\/3409256.3409837","relation":{},"subject":[],"published":{"date-parts":[[2020,9,14]]},"assertion":[{"value":"2020-09-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}