{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:32:45Z","timestamp":1750307565567,"version":"3.41.0"},"reference-count":14,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2009,6,25]],"date-time":"2009-06-25T00:00:00Z","timestamp":1245888000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2009,6,25]]},"abstract":"<jats:p>In the context of creating large scale test collections, the present paper discusses methods of constructing a patent test collection for evaluation of prior art search. In particular, it addresses criteria for topic selection and identification of recall bases. These issues arose while organizing the CLEF-IP evaluation track and were the subject of an online discussion among the track's organizers and its steering committee. Most literature on building test collections is concerned with minimizing the costs of obtaining relevance assessments. CLEF-IP can afford to have large topics sets since relevance assessments are generated by exploiting existing manually created information. In a cost-benefit analysis, the only issue seems to be the computing time required by participants to run (tens or hundreds of) thousands of queries. This document describes the data sets and decisions leading to the creation of the CLEF-IP collection.<\/jats:p>","DOI":"10.1145\/1670598.1670601","type":"journal-article","created":{"date-parts":[[2012,10,15]],"date-time":"2012-10-15T19:22:23Z","timestamp":1350328943000},"page":"9-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["So many topics, so little time"],"prefix":"10.1145","volume":"43","author":[{"given":"Giovanna","family":"Roda","sequence":"first","affiliation":[{"name":"Matrixware"}]},{"given":"Veronika","family":"Zenz","sequence":"additional","affiliation":[{"name":"Matrixware"}]},{"given":"Mihai","family":"Lupu","sequence":"additional","affiliation":[{"name":"Information Retrieval Facility"}]},{"given":"Kalervo","family":"J\u00e4rvelin","sequence":"additional","affiliation":[{"name":"University of Tampere"}]},{"given":"Mark","family":"Sanderson","sequence":"additional","affiliation":[{"name":"University of Shefield"}]},{"given":"Christa","family":"Womser-Hacker","sequence":"additional","affiliation":[{"name":"University of Hildesheim"}]}],"member":"320","published-online":{"date-parts":[[2009,6,25]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"European patent convention (EPC) http:\/\/www.epo.org\/patents\/law\/legal-texts.  European patent convention (EPC) http:\/\/www.epo.org\/patents\/law\/legal-texts."},{"key":"e_1_2_1_2_1","volume-title":"http:\/\/www.epo.org\/patents\/law\/legal-texts\/guidelines.html","author":"European Patent Office Guidelines","year":"2009","unstructured":"Guidelines for Examination in the European Patent Office , http:\/\/www.epo.org\/patents\/law\/legal-texts\/guidelines.html , 2009 . Guidelines for Examination in the European Patent Office, http:\/\/www.epo.org\/patents\/law\/legal-texts\/guidelines.html, 2009."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390419"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00958-7_27"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321530"},{"key":"e_1_2_1_6_1","volume-title":"Research Memoranda 017 11","author":"Criscuolo P.","year":"2005","unstructured":"P. Criscuolo and B. Verspagen , Does it matter where patent citations come from? inventor versus examiner citations in european patents , Research Memoranda 017 11 ( 2005 ), no. 5. P. Criscuolo and B. Verspagen, Does it matter where patent citations come from? inventor versus examiner citations in european patents, Research Memoranda 017 11 (2005), no. 5."},{"key":"e_1_2_1_7_1","volume-title":"IRF Symposium Exposition","author":"Croft W.B.","year":"2008","unstructured":"W.B. Croft and X. Xue , Prior art searching , IRF Symposium Exposition , 2008 . W.B. Croft and X. Xue, Prior art searching, IRF Symposium Exposition, 2008."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Second International Workshop on Evaluating Information Access (EVIA)","author":"Graf E.","year":"2008","unstructured":"E. Graf and L. Azzopardi , A methodology for building a patent test collection for prior art search , Proceedings of the Second International Workshop on Evaluating Information Access (EVIA) , 2008 . E. Graf and L. Azzopardi, A methodology for building a patent test collection for prior art search, Proceedings of the Second International Workshop on Evaluating Information Access (EVIA), 2008."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220835.1220885"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9059-7"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076064"},{"key":"e_1_2_1_12_1","volume-title":"personal communication","author":"Tomas Henk","year":"2008","unstructured":"Henk Tomas , personal communication , 2008 . Henk Tomas, personal communication, 2008."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564432"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1670598.1670601","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1670598.1670601","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T12:40:52Z","timestamp":1750250452000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1670598.1670601"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,6,25]]},"references-count":14,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2009,6,25]]}},"alternative-id":["10.1145\/1670598.1670601"],"URL":"https:\/\/doi.org\/10.1145\/1670598.1670601","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2009,6,25]]},"assertion":[{"value":"2009-06-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}