{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,27]],"date-time":"2025-03-27T09:49:41Z","timestamp":1743068981036,"version":"3.40.3"},"publisher-location":"Berlin, Heidelberg","reference-count":25,"publisher":"Springer Berlin Heidelberg","isbn-type":[{"type":"print","value":"9783642151804"},{"type":"electronic","value":"9783642151811"}],"license":[{"start":{"date-parts":[[2010,1,1]],"date-time":"2010-01-01T00:00:00Z","timestamp":1262304000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2010,1,1]],"date-time":"2010-01-01T00:00:00Z","timestamp":1262304000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2010]]},"DOI":"10.1007\/978-3-642-15181-1_4","type":"book-chapter","created":{"date-parts":[[2010,8,20]],"date-time":"2010-08-20T12:43:45Z","timestamp":1282308225000},"page":"63-80","source":"Crossref","is-referenced-by-count":0,"title":["Relevance Judgments for Image Retrieval Evaluation"],"prefix":"10.1007","author":[{"given":"Jayashree","family":"Kalpathy\u2013Cramer","sequence":"first","affiliation":[]},{"given":"Steven","family":"Bedrick","sequence":"additional","affiliation":[]},{"given":"William","family":"Hersh","sequence":"additional","affiliation":[]}],"member":"297","reference":[{"key":"4_CR1","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1145\/1008992.1009000","volume-title":"Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval","author":"C Buckley","year":"2004","unstructured":"Buckley C, Voorhees EM (2004) Retrieval evaluation with incomplete information. In: Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval. ACM press, New York, NY, USA, pp 25\u201332"},{"key":"4_CR2","doi-asserted-by":"publisher","first-page":"619","DOI":"10.1145\/1148170.1148284","volume-title":"Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval","author":"C Buckley","year":"2006","unstructured":"Buckley C, Dimmick D, Soboroff I, Voorhees E (2006) Bias and the limits of pooling. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval. ACM press, New York, NY, USA, pp 619\u2013620"},{"key":"4_CR3","unstructured":"Cleverdon CW (1962) Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. Tech. rep., Aslib Cranfield Research Project, Cranfield, USA"},{"key":"4_CR4","doi-asserted-by":"crossref","unstructured":"Cleverdon CW (1991) The significance of the cranfield tests on index languages. In: Proceedings of the 14th annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 3\u201312","DOI":"10.1145\/122860.122861"},{"key":"4_CR5","doi-asserted-by":"crossref","unstructured":"Clough P, Sanderson M, M\u00fcller H (2004) The CLEF Cross Language Image Retrieval Track (ImageCLEF) 2004. In: Image and Video Retrieval (CIVR 2004). Lecture Notes in Computer Science (LNCS), vol 3115. Springer, pp 243\u2013251","DOI":"10.1007\/978-3-540-27814-6_31"},{"key":"4_CR6","doi-asserted-by":"crossref","unstructured":"Cormack GV, Palmer CR, Clarke CLA (1998) Efficient construction of large test collections. In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 282\u2013289","DOI":"10.1145\/290941.291009"},{"key":"4_CR7","doi-asserted-by":"crossref","unstructured":"Di\u00a0Nunzio G, Ferro N (2004) DIRECT: a system for evaluating information access components of digital libraries. In: Research and Advanced Technology for Digital Libraries. Lecture Notes in Computer Science (LNCS), vol 3652. Springer, pp 483\u2013484","DOI":"10.1007\/11551362_46"},{"key":"4_CR8","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1145\/1390334.1390348","volume-title":"Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval","author":"B He","year":"2008","unstructured":"He B, Macdonald C, Ounis I (2008) Retrieval sensitivity under training using different measures. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, New York, NY, USA, pp 67\u201374"},{"key":"4_CR9","doi-asserted-by":"crossref","unstructured":"Hersh W, Buckley C, Leone TJ, Hickam D (1994) OHSUMED: an interactive retrieval evaluation and new large test collection for research. In: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 192\u2013201","DOI":"10.1007\/978-1-4471-2099-5_20"},{"key":"4_CR10","doi-asserted-by":"crossref","unstructured":"Hersh W, Turpin A, Price S, Chan B, Kramer D, Sacherek L, Olson D (2000) Do batch and user evaluations give the same results? In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 17\u201324","DOI":"10.1145\/345508.345539"},{"issue":"5","key":"4_CR11","doi-asserted-by":"publisher","first-page":"488","DOI":"10.1197\/jamia.M2082","volume":"13","author":"W Hersh","year":"2006","unstructured":"Hersh W, M\u00fcller H, Jensen J, Yang J, Gorman P, Ruch P (2006a) Advancing biomedical image retrieval: Development and analysis of a test collection. Journal of the American Medical Informatics Association 13(5):488\u2013496","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"6","key":"4_CR12","doi-asserted-by":"publisher","first-page":"648","DOI":"10.1007\/s10278-008-9154-8","volume":"22","author":"W Hersh","year":"2009","unstructured":"Hersh W, M\u00fcller H, Kalpathy-Cramer J, Kim E, Zhou X (2009) The consolidated ImageCLEFmed medical image retrieval task test collection. Journal of Digital Imaging 22(6):648\u2013655","journal-title":"Journal of Digital Imaging"},{"key":"4_CR13","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1186\/1747-5333-1-3","volume":"1","author":"WR Hersh","year":"2006","unstructured":"Hersh WR, Bhupatiraju RT, Ross L, Roberts P, Cohen AM, Kraemer DF (2006b) Enhancing access to the bibliome: the trec 2004 genomics track. Journal of Biomedical Discovery and Collaboration 1:3","journal-title":"Journal of Biomedical Discovery and Collaboration"},{"issue":"4","key":"4_CR14","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1145\/582415.582418","volume":"20","author":"K J\u00e4rvelin","year":"2002","unstructured":"J\u00e4rvelin K, Kek\u00e4l\u00e4inen J (2002) Cumulated gain\u2013based evaluation of ir techniques. ACM Transactions of Information Systems 20(4):422\u2013446","journal-title":"ACM Transactions of Information Systems"},{"key":"4_CR15","series-title":"Lecture Notes in Computer Science (LNCS)","first-page":"233","volume-title":"Adaptive Multimedia Retrieval (AMR)","author":"H M\u00fcller","year":"2007","unstructured":"M\u00fcller H, Clough P, Hersh W, Geissbuhler A (2007) Variations of relevance assessments for medical image retrieval. In: Adaptive Multimedia Retrieval (AMR). Lecture Notes in Computer Science (LNCS), vol 4398. Springer, Geneva, Switzerland, pp 233\u2013247"},{"key":"4_CR16","series-title":"Lecture Notes in Computer Science (LNCS)","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1007\/978-3-540-68555-5","volume-title":"CLEF 2007 Proceedings","author":"H M\u00fcller","year":"2008","unstructured":"M\u00fcller H, Deselaers T, Kim E, Kalpathy-Cramer J, Deserno TM, Clough PD, Hersh W (2008) Overview of the ImageCLEFmed 2007 medical retrieval and annotation tasks. In: CLEF 2007 Proceedings. Lecture Notes in Computer Science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 473\u2013491"},{"key":"4_CR17","doi-asserted-by":"crossref","unstructured":"M\u00fcller H, Kalpathy-Cramer J, Eggel I, Bedrick S, Sa\u00efd R, Bakke B, Kahn\u00a0Jr CE, Hersh W (2009) Overview of the CLEF 2009 medical image retrieval track. In: Working Notes of CLEF 2009, Corfu, Greece","DOI":"10.1007\/978-3-642-15751-6_8"},{"key":"4_CR18","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1145\/1743384.1743478","volume-title":"Proceedings of the international conference on multimedia information retrieval (MIR 2010)","author":"S Nowak","year":"2010","unstructured":"Nowak S, R\u00fcger S (2010) How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In: Proceedings of the international conference on multimedia information retrieval (MIR 2010). ACM press, New York, NY, USA, pp 557\u2013566"},{"issue":"1\u20133","key":"4_CR19","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1007\/s11263-007-0090-8","volume":"77","author":"B Russell","year":"2008","unstructured":"Russell B, Torralba A, Murphy K, Freeman W (2008) LabelMe: a database and web\u2013based tool for image annotation. International Journal of Computer Vision 77(1\u20133):157\u2013173","journal-title":"International Journal of Computer Vision"},{"key":"4_CR20","unstructured":"Sparck\u00a0Jones K, van Rijsbergen C (1975) Report on the need for and provision of an ideal information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge"},{"key":"4_CR21","series-title":"Lecture Notes in Computer Science (LNCS)","volume-title":"Evaluating Systems for Multilingual and Multimodal Information Access \u2014 9th Workshop of the Cross-Language Evaluation Forum","author":"T Tsikrika","year":"2009","unstructured":"Tsikrika T, Kludas J (2009) Overview of the wikipediaMM task at ImageCLEF 2008. In: Peters C, Giampiccolo D, Ferro N, Petras V, Gonzalo J, Pe\u00f1as A, Deselaers T, Mandl T, Jones G, Kurimo M (eds) Evaluating Systems for Multilingual and Multimodal Information Access \u2014 9th Workshop of the Cross-Language Evaluation Forum. Lecture Notes in Computer Science (LNCS). Springer, Aarhus, Denmark"},{"key":"4_CR22","series-title":"Lecture Notes in Computer Science (LNCS)","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1007\/3-540-45691-0_34","volume-title":"Revised Papers from the Second Workshop of the Cross\u2013Language Evaluation Forum on Evaluation of Cross\u2013Language Information Retrieval Systems \u2014 CLEF 2001","author":"EM Voorhees","year":"2002","unstructured":"Voorhees EM (2002) The philosophy of information retrieval evaluation. In: Revised Papers from the Second Workshop of the Cross\u2013Language Evaluation Forum on Evaluation of Cross\u2013Language Information Retrieval Systems \u2014 CLEF 2001. Lecture Notes in Computer Science (LNCS). Springer, London, UK, pp 355\u2013370"},{"key":"4_CR23","doi-asserted-by":"crossref","unstructured":"Voorhees EM, Harmann D (1998) Overview of the seventh Text REtrieval Conference (TREC\u20137). In: The Seventh Text Retrieval Conference, Gaithersburg, MD, USA, pp 1\u201323","DOI":"10.6028\/NIST.SP.500-242"},{"key":"4_CR24","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1145\/1183614.1183633","volume-title":"Proceedings of the 15th ACM international conference on Information and knowledge management","author":"E Yilmaz","year":"2006","unstructured":"Yilmaz E, Aslam JA (2006) Estimating average precision with incomplete and imperfect judgments. In: Proceedings of the 15th ACM international conference on Information and knowledge management. ACM press, New York, NY, USA, pp 102\u2013111"},{"key":"4_CR25","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1145\/290941.291014","volume-title":"Proceedings of the 21st Annual International ACM SIGIR conference on research and development in information retrieval","author":"J Zobel","year":"1998","unstructured":"Zobel J (1998) How reliable are the results of large\u2013scale information retrieval experiments? In: Croft WB, Moffat A, van Rijsbergen CJ, Wilkinson R, Zobel J (eds) Proceedings of the 21st Annual International ACM SIGIR conference on research and development in information retrieval. ACM press, Melbourne, Australia, pp 307\u2013314"}],"container-title":["The Information Retrieval Series","ImageCLEF"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-642-15181-1_4","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,18]],"date-time":"2023-02-18T04:07:17Z","timestamp":1676693237000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-642-15181-1_4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010]]},"ISBN":["9783642151804","9783642151811"],"references-count":25,"URL":"https:\/\/doi.org\/10.1007\/978-3-642-15181-1_4","relation":{},"ISSN":["1387-5264"],"issn-type":[{"type":"print","value":"1387-5264"}],"subject":[],"published":{"date-parts":[[2010]]}}}