{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T02:36:25Z","timestamp":1783737385139,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":25,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,7,25]],"date-time":"2020-07-25T00:00:00Z","timestamp":1595635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100008982","name":"Qatar National Research Fund","doi-asserted-by":"publisher","award":["NPRP 7-1313-1-245"],"award-info":[{"award-number":["NPRP 7-1313-1-245"]}],"id":[{"id":"10.13039\/100008982","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,7,25]]},"DOI":"10.1145\/3397271.3401223","type":"proceedings-article","created":{"date-parts":[[2020,7,25]],"date-time":"2020-07-25T07:50:08Z","timestamp":1595663408000},"page":"2017-2020","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["ArTest: The First Test Collection for Arabic Web Search with Relevance Rationales"],"prefix":"10.1145","author":[{"given":"Maram","family":"Hasanain","sequence":"first","affiliation":[{"name":"Qatar University, Doha, Qatar"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yassmine","family":"Barkallah","sequence":"additional","affiliation":[{"name":"Mantu, Tunis, Tunisia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Reem","family":"Suwaileh","sequence":"additional","affiliation":[{"name":"Qatar University, Doha, Qatar"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mucahid","family":"Kutlu","sequence":"additional","affiliation":[{"name":"TOBB University of Economics and Technology, Ankara, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tamer","family":"Elsayed","sequence":"additional","affiliation":[{"name":"Qatar University, Doha, Qatar"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,7,25]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637002.2637025"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914681"},{"key":"e_1_3_2_2_3_1","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","volume":"15","author":"Alonso Omar","year":"2009","unstructured":"Omar Alonso and Stefano Mizzaro . 2009 . Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , Vol. 15 . 16. Omar Alonso and Stefano Mizzaro. 2009. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, Vol. 15. 16."},{"key":"e_1_3_2_2_4_1","first-page":"1053","article-title":"Using crowdsourcing for TREC relevance assessment","volume":"48","author":"Alonso Omar","year":"2012","unstructured":"Omar Alonso and Stefano Mizzaro . 2012 . Using crowdsourcing for TREC relevance assessment . IP&M , Vol. 48 , 6 (2012), 1053 -- 1066 . Omar Alonso and Stefano Mizzaro. 2012. Using crowdsourcing for TREC relevance assessment. IP&M, Vol. 48, 6 (2012), 1053--1066.","journal-title":"IP&M"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767728"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130348.3130373"},{"key":"e_1_3_2_2_7_1","volume-title":"Overview of the TREC 2012 web track. Technical Report. NATIONAL INST OF STANDARDS AND TECHNOLOGY GAITHERSBURG MD.","author":"Clarke Charles L","year":"2012","unstructured":"Charles L Clarke , Nick Craswell , and Ellen M Voorhees . 2012 . Overview of the TREC 2012 web track. Technical Report. NATIONAL INST OF STANDARDS AND TECHNOLOGY GAITHERSBURG MD. Charles L Clarke, Nick Craswell, and Ellen M Voorhees. 2012. Overview of the TREC 2012 web track. Technical Report. NATIONAL INST OF STANDARDS AND TECHNOLOGY GAITHERSBURG MD."},{"key":"e_1_3_2_2_8_1","volume-title":"Proceedings of the international conference on scientific information. 687--698","author":"Cleverdon Cyril W","year":"1959","unstructured":"Cyril W Cleverdon . 1959 . The evaluation of systems used in information retrieval . In Proceedings of the international conference on scientific information. 687--698 . Cyril W Cleverdon. 1959. The evaluation of systems used in information retrieval. In Proceedings of the international conference on scientific information. 687--698."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291009"},{"key":"e_1_3_2_2_10_1","volume-title":"Proceedings of TREC'01","author":"Gey F","year":"2001","unstructured":"F Gey and D Oard . 2001 . The TREC-2001 cross-language information retrieval track: Searching Arabic using English, French or Arabic queries . In Proceedings of TREC'01 . 78. F Gey and D Oard. 2001. The TREC-2001 cross-language information retrieval track: Searching Arabic using English, French or Arabic queries. In Proceedings of TREC'01. 78."},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-017-9325-7"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348403"},{"key":"e_1_3_2_2_13_1","volume-title":"TOIS","volume":"20","author":"Jaana Kalervo","year":"2002","unstructured":"Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002 . Cumulated gain-based evaluation of IR techniques . TOIS , Vol. 20 , 4 (2002), 422--446. Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002. Cumulated gain-based evaluation of IR techniques. TOIS, Vol. 20, 4 (2002), 422--446."},{"key":"e_1_3_2_2_14_1","first-page":"37","article-title":"Intelligent topic selection for low-cost information retrieval evaluation: A New perspective on deep vs. shallow judging","volume":"54","author":"Kutlu Mucahid","year":"2018","unstructured":"Mucahid Kutlu , Tamer Elsayed , and Matthew Lease . 2018 a. Intelligent topic selection for low-cost information retrieval evaluation: A New perspective on deep vs. shallow judging . IP&M , Vol. 54 , 1 (2018), 37 -- 59 . Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. 2018a. Intelligent topic selection for low-cost information retrieval evaluation: A New perspective on deep vs. shallow judging. IP&M, Vol. 54, 1 (2018), 37--59.","journal-title":"IP&M"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210033"},{"key":"e_1_3_2_2_16_1","first-page":"1005","article-title":"Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems","volume":"53","author":"Losada David E","year":"2017","unstructured":"David E Losada , Javier Parapar , and Alvaro Barreiro . 2017 . Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems . IP&M , Vol. 53 , 5 (2017), 1005 -- 1025 . David E Losada, Javier Parapar, and Alvaro Barreiro. 2017. Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. IP&M, Vol. 53, 5 (2017), 1005--1025.","journal-title":"IP&M"},{"key":"e_1_3_2_2_17_1","volume-title":"Fourth AAAI Conference on Human Computation and Crowdsourcing.","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell , Matthew Lease , Mucahid Kutlu , and Tamer Elsayed . 2016 . Why is that relevant? collecting annotator rationales for relevance judgments . In Fourth AAAI Conference on Human Computation and Crowdsourcing. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? collecting annotator rationales for relevance judgments. In Fourth AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806606"},{"key":"e_1_3_2_2_19_1","volume-title":"Fewer topics? A million topics? Both?! On topics subsets in test collections. IRJ","author":"Roitero Kevin","year":"2019","unstructured":"Kevin Roitero , J Shane Culpepper , Mark Sanderson , Falk Scholer , and Stefano Mizzaro . 2019. Fewer topics? A million topics? Both?! On topics subsets in test collections. IRJ ( 2019 ), 1--37. Kevin Roitero, J Shane Culpepper, Mark Sanderson, Falk Scholer, and Stefano Mizzaro. 2019. Fewer topics? A million topics? Both?! On topics subsets in test collections. IRJ (2019), 1--37."},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009001"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484090"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026616"},{"key":"e_1_3_2_2_23_1","volume-title":"Report on the need for and provision of an'ideal'information retrieval test collection. Computer Laboratory","author":"Spark-Jones K","year":"1975","unstructured":"K Spark-Jones . 1975. Report on the need for and provision of an'ideal'information retrieval test collection. Computer Laboratory ( 1975 ). K Spark-Jones. 1975. Report on the need for and provision of an'ideal'information retrieval test collection. Computer Laboratory (1975)."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914677"},{"key":"e_1_3_2_2_25_1","volume-title":"Proceedings of 3rd OSACT Workshop.","author":"Yasser Khaled","year":"2018","unstructured":"Khaled Yasser , Reem Suwaileh , Abdelrahman Shouman , Yassmine Barkallah , Mucahid Kutlu , and Tamer Elsayed . 2018 . iArabicWeb16: Making a Large Web Collection More Accessible for Research . In Proceedings of 3rd OSACT Workshop. Khaled Yasser, Reem Suwaileh, Abdelrahman Shouman, Yassmine Barkallah, Mucahid Kutlu, and Tamer Elsayed. 2018. iArabicWeb16: Making a Large Web Collection More Accessible for Research. In Proceedings of 3rd OSACT Workshop."}],"event":{"name":"SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval","location":"Virtual Event China","acronym":"SIGIR '20","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397271.3401223","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3397271.3401223","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:44Z","timestamp":1750200104000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397271.3401223"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,25]]},"references-count":25,"alternative-id":["10.1145\/3397271.3401223","10.1145\/3397271"],"URL":"https:\/\/doi.org\/10.1145\/3397271.3401223","relation":{},"subject":[],"published":{"date-parts":[[2020,7,25]]},"assertion":[{"value":"2020-07-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}