{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T04:29:23Z","timestamp":1778300963613,"version":"3.51.4"},"publisher-location":"Berlin, Heidelberg","reference-count":26,"publisher":"Springer Berlin Heidelberg","isbn-type":[{"value":"9783540440420","type":"print"},{"value":"9783540456919","type":"electronic"}],"license":[{"start":{"date-parts":[[2002,1,1]],"date-time":"2002-01-01T00:00:00Z","timestamp":1009843200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2002,1,1]],"date-time":"2002-01-01T00:00:00Z","timestamp":1009843200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2002]]},"DOI":"10.1007\/3-540-45691-0_34","type":"book-chapter","created":{"date-parts":[[2007,5,19]],"date-time":"2007-05-19T10:53:02Z","timestamp":1179571982000},"page":"355-370","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":115,"title":["The Philosophy of Information Retrieval Evaluation"],"prefix":"10.1007","author":[{"given":"Ellen M.","family":"Voorhees","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2002,8,2]]},"reference":[{"key":"34_CR1","series-title":"Lect Notes Comput Sci","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1007\/3-540-44645-1_9","volume-title":"Cross-Language Information Retrieval and Evaluation","author":"M. Braschler","year":"2001","unstructured":"Martin Braschler. CLEF 200-Overview of results. In Carol Peters, editor, Cross-Language Information Retrieval and Evaluation; Lecture Notes in Computer Science2069, pages 89\u2013101. Springer, 2001."},{"key":"34_CR2","doi-asserted-by":"crossref","unstructured":"Chris Buckley and Ellen M. Voorhees. Evaluating evaluation measure stability. In N. Belkin, P. Ingwersen, and M.K. Leong, editors, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 33\u201340, 2000.","DOI":"10.1145\/345508.345543"},{"key":"34_CR3","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1108\/eb050097","volume":"19","author":"C. W. Cleverdon","year":"1967","unstructured":"C. W. Cleverdon. The Cranfield tests on index language devices. In Aslib Proceedings, volume 19, pages 173\u2013192, 1967. (Reprinted in Readings in Information Retrieval, K. Sparck-Jones and P. Willett, editors, Morgan Kaufmann, 1997).","journal-title":"Aslib Proceedings"},{"key":"34_CR4","doi-asserted-by":"crossref","unstructured":"Cyril W. Cleverdon. The significance of the Cranfield tests on index languages. In Proceedings of the Fourteenth Annual International ACM\/SIGIR Conference on Research and Development in Information Retrieval, pages 3\u201312, 1991.","DOI":"10.1145\/122860.122861"},{"key":"34_CR5","doi-asserted-by":"publisher","first-page":"282","DOI":"10.1145\/290941.291009","volume-title":"Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"G. V. Cormack","year":"1998","unstructured":"Gordon V. Cormack, Christopher R. Palmer, and Charles L.A. Clarke. Efficient construction of large test collections. In Alistair Moffat, C.J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, August 1998. ACM Press, New York Croft et al. [6], pages 282\u2013289."},{"key":"34_CR6","volume-title":"Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","year":"1998","unstructured":"W. Bruce Croft, Alistair Moffat, C.J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, August 1998. ACM Press, New York."},{"issue":"4","key":"34_CR7","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1108\/eb026436","volume":"23","author":"C. A. Cuadra","year":"1967","unstructured":"C. A. Cuadra and R. V. Katter. Opening the black box of relevance. Journal of Documentation, 23(4):291\u2013303, 1967.","journal-title":"Journal of Documentation"},{"key":"34_CR8","doi-asserted-by":"crossref","unstructured":"Donna Harman. Overview of the fourth Text REtrieval Conference (TREC-4). In D. K. Harman, editor, Proceedings of the Fourth Text REtrieval Conference (TREC-4), pages 1\u201323, October 1996. NIST Special Publication 500\u2013236.","DOI":"10.6028\/NIST.SP.500-236.overview-overview"},{"issue":"1","key":"34_CR9","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3","volume":"47","author":"S. P. Harter","year":"1996","unstructured":"Stephen P. Harter. Variations in relevance assessments and the measurement of retrieval effectiveness. Journal of the American Society for Information Science, 47(1):37\u201349, 1996.","journal-title":"Journal of the American Society for Information Science"},{"key":"34_CR10","doi-asserted-by":"crossref","unstructured":"William Hersh, Andrew Turpin, Susan Price, Benjamin Chan, Dale Kraemer, Lynetta Sacherek, and Daniel Olson. Do batch and user evaluations give the same results? In N. Belkin, P. Ingwersen, and M.K. Leong, editors, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 17\u201324, 2000.","DOI":"10.1145\/345508.345539"},{"key":"34_CR11","unstructured":"Noriko Kando, Kazuko Kuriyama, Toshihiko Nozue, Koji Eguchi, Hiroyuki Kato, and Souichiro Hidaka. Overview of IR tasks at the first NTCIR workshop. In Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, pages 11\u201344, 1999."},{"key":"34_CR12","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1016\/0020-0271(68)90029-6","volume":"4","author":"M.E. Lesk","year":"1969","unstructured":"M.E. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information Storage and Retrieval, 4:343\u2013359, 1969.","journal-title":"Information Storage and Retrieval"},{"key":"34_CR13","volume-title":"The SMART Retrieval System: Experiments in Automatic Document Processing","year":"1971","unstructured":"G. Salton, editor. The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice-Hall, Inc. Englewood Cliffs, New Jersey, 1971."},{"key":"34_CR14","first-page":"3","volume":"29","author":"L. Schamber","year":"1994","unstructured":"Linda Schamber. Relevance and information behavior. Annual Review of Information Science and Technology, 29:3\u201348, 1994.","journal-title":"Annual Review of Information Science and Technology"},{"key":"34_CR15","unstructured":"K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an \u201cideal\u201d information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975."},{"key":"34_CR16","first-page":"256","volume-title":"Information Retrieval Experiment","author":"K. S. Jones","year":"1981","unstructured":"Karen Sparck Jones. The Cranfield tests. In Karen Sparck Jones, editor, Information Retrieval Experiment, chapter 13, pages 256\u2013284. Butterworths, London, 1981."},{"key":"34_CR17","volume-title":"Information Retrieval Experiment","author":"K. S. Jones","year":"1981","unstructured":"Karen Sparck Jones. Information Retrieval Experiment. Butterworths, London, 1981."},{"key":"34_CR18","unstructured":"Karen Sparck Jones and Peter Willett. Evaluation. In Karen Sparck Jones and Peter Willett, editors, Readings in Information Retrieval, chapter 4, pages 167\u2013174. Morgan Kaufmann, 1997."},{"key":"34_CR19","unstructured":"Alan Stuart. Kendall\u2019s tau. In Samuel Kotz and Norman L. Johnson, editors, Encyclopedia of Statistical Sciences, volume 4, pages 367\u2013369. John Wiley & Sons, 1983."},{"issue":"2","key":"34_CR20","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1002\/asi.5090160204","volume":"16","author":"M. Taube","year":"1965","unstructured":"M. Taube. A note on the pseudomathematics of relevance. American Documentation, 16(2):69\u201372, April 1965.","journal-title":"American Documentation"},{"key":"34_CR21","doi-asserted-by":"crossref","unstructured":"Andrew H. Turpin and William Hersh. Why batch and user evaluations do not give the same results. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 225\u2013231, 2001.","DOI":"10.1145\/383952.383992"},{"key":"34_CR22","doi-asserted-by":"crossref","unstructured":"C.J. van Rijsbergen. Information Retrieval, chapter 7. Butterworths, 2 edition, 1979.","DOI":"10.1007\/978-3-642-23318-0_2"},{"key":"34_CR23","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1016\/S0306-4573(00)00010-8","volume":"36","author":"E. M. Voorhees","year":"2000","unstructured":"Ellen M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management, 36:697\u2013716, 2000.","journal-title":"Information Processing and Management"},{"key":"34_CR24","doi-asserted-by":"crossref","unstructured":"Ellen M. Voorhees and Donna Harman. Overview of the eighth Text REtrieval Conference (TREC-8). In E.M. Voorhees and D.K. Harman, editors, Proceedings of the Eighth Text REtrieval Conference (TREC-8), pages 1\u201324, 2000. NIST Special Publication 500\u2013246. Electronic version available at http:\/\/trec.nist.gov\/pubs.html.","DOI":"10.6028\/NIST.SP.500-246.overview"},{"key":"34_CR25","doi-asserted-by":"crossref","unstructured":"Ellen M. Voorhees and Donna Harman. Overview of TREC 2001. In Proceedings of TREC 2001 (Draft), 2001. To appear.","DOI":"10.6028\/NIST.SP.500-250.overview"},{"key":"34_CR26","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1145\/290941.291014","volume-title":"Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"J. Zobel","year":"1998","unstructured":"Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Alistair Moffat, C.J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, August 1998. ACM Press, New York Croft et al. [6], pages 307\u2013314."}],"container-title":["Lecture Notes in Computer Science","Evaluation of Cross-Language Information Retrieval Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/3-540-45691-0_34","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T13:50:30Z","timestamp":1778248230000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/3-540-45691-0_34"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2002]]},"ISBN":["9783540440420","9783540456919"],"references-count":26,"URL":"https:\/\/doi.org\/10.1007\/3-540-45691-0_34","relation":{},"ISSN":["0302-9743"],"issn-type":[{"value":"0302-9743","type":"print"}],"subject":[],"published":{"date-parts":[[2002]]},"assertion":[{"value":"2 August 2002","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}}]}}