{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,27]],"date-time":"2025-03-27T19:56:09Z","timestamp":1743105369001,"version":"3.40.3"},"publisher-location":"Berlin, Heidelberg","reference-count":47,"publisher":"Springer Berlin Heidelberg","isbn-type":[{"type":"print","value":"9783642151804"},{"type":"electronic","value":"9783642151811"}],"license":[{"start":{"date-parts":[[2010,1,1]],"date-time":"2010-01-01T00:00:00Z","timestamp":1262304000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2010,1,1]],"date-time":"2010-01-01T00:00:00Z","timestamp":1262304000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2010]]},"DOI":"10.1007\/978-3-642-15181-1_27","type":"book-chapter","created":{"date-parts":[[2010,8,20]],"date-time":"2010-08-20T12:43:45Z","timestamp":1282308225000},"page":"507-525","source":"Crossref","is-referenced-by-count":3,"title":["An Overview of Evaluation Campaigns in Multimedia Retrieval"],"prefix":"10.1007","author":[{"given":"Suzanne","family":"Little","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ainhoa","family":"Llorente","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefan","family":"R\u00fcger","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","reference":[{"issue":"10","key":"27_CR1","doi-asserted-by":"publisher","first-page":"913","DOI":"10.1002\/asi.10286","volume":"54","author":"P Borlund","year":"2003","unstructured":"Borlund P (2003) The concept of relevance in IR. Journal of the American Society for information Science and Technology 54(10):913\u2013925","journal-title":"Journal of the American Society for information Science and Technology"},{"key":"27_CR2","doi-asserted-by":"crossref","unstructured":"Bozzon A, Brambilla M, Fraternali P, Nucci F, Debald S, Moore E, Neidl W, Plu M, Aichroth P, Pihlajamaa O, Laurier C, Zagorac S, Backfried G, Weinland D, Croce V (2009) PHAROS: an audiovisual search platform. In: ACM International conference on research and development in information retrieval. ACM press, p 841","DOI":"10.1145\/1571941.1572161"},{"key":"27_CR3","unstructured":"Brookes B (1981) Information technology and the science of information. Information retrieval research. London: Butterworths 1\u20138"},{"key":"27_CR4","doi-asserted-by":"crossref","unstructured":"Buckley C, Voorhees E (2000) Evaluating evaluation measure stability. In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, p 40. ACM press","DOI":"10.1145\/345508.345543"},{"key":"27_CR5","doi-asserted-by":"crossref","unstructured":"Clarke C, Kolla M, Cormack G, Vechtomova O, Ashkan A, B\u00fcttcher S, MacKinnon I (2008) Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 659\u2013666","DOI":"10.1145\/1390334.1390446"},{"key":"27_CR6","unstructured":"Cleverdon C, Mills J, Keen E (1966) Factors determining the performance of indexing systems,(Volume 1: Design). Cranfield: College of Aeronautics"},{"key":"27_CR7","unstructured":"Cooniss L, Ashford A, Graham M (2000) Information seeking behaviour in image retrieval. VISOR\u00a01 final report. Technical report, Library and Information Commission Research Report, British Library"},{"key":"27_CR8","unstructured":"Cooniss L, Davis J, Graham M (2003) A user\u2013oriented evaluation framework for the development of electronic image retrieval systems in the workplace: VISOR\u00a02 final report. Technical report, Library and Information Commission Research Report, British Library"},{"issue":"4","key":"27_CR9","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1016\/S0306-4573(99)00072-2","volume":"36","author":"E Cosijn","year":"2000","unstructured":"Cosijn E, Ingwersen P (2000) Dimensions of relevance. Information Processing & Management 36(4):533\u2013550","journal-title":"Information Processing & Management"},{"issue":"4","key":"27_CR10","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1250\/ast.29.247","volume":"29","author":"JS Downie","year":"2008","unstructured":"Downie JS (2008) The music information retrieval evaluation exchange (2005\u20132007): a window into music information retrieval research. Acoustical Science and Technology 29(4):247\u2013255","journal-title":"Acoustical Science and Technology"},{"issue":"1","key":"27_CR11","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<23::AID-ASI3>3.0.CO;2-4","volume":"47","author":"D Ellis","year":"1996","unstructured":"Ellis D (1996) The dilemma of measurement in information retrieval research. Journal of the American Society for Information Science 47(1):23\u201336","journal-title":"Journal of the American Society for Information Science"},{"key":"27_CR12","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1007\/1-4020-3467-9_11","volume":"16","author":"D Harman","year":"2005","unstructured":"Harman D (2005) The importance of focused evaluations: A Case Study of TREC and DUC. Charting a New Course: Natural Language Processing and Information Retrieval 16:175\u2013194","journal-title":"Charting a New Course: Natural Language Processing and Information Retrieval"},{"key":"27_CR13","unstructured":"Heesch D, Pickering M, R\u00fcger S, Yavlinsky A (2003) Video retrieval using search and browsing with key frames. In: TREC Video Retrieval Evaluation"},{"key":"27_CR14","unstructured":"Heesch D, Howarth P, Magalh\u00e3es J, May A, Pickering M, Yavlinsky A, R\u00fcger S (2004) Video retrieval using search and browsing. In: TREC Video Retrieval Evaluation"},{"key":"27_CR15","doi-asserted-by":"crossref","unstructured":"Hersh W, Turpin A, Price S, Chan B, Kramer D, Sacherek L, Olson D (2000) Do batch and user evaluations give the same results? In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, p 24. ACM press","DOI":"10.1145\/345508.345539"},{"key":"27_CR16","doi-asserted-by":"crossref","unstructured":"Howarth P, Yavlinsky A, Heesch D, R\u00fcger S (2005) Medical image retrieval using texture, locality and colour. In: Proceedings of the Cross Language Evaluation Forum 2004. Lecture Notes in Computer Science (LNCS), vol 3491. Springer, pp 740\u2013749","DOI":"10.1007\/11519645_72"},{"key":"27_CR17","doi-asserted-by":"crossref","unstructured":"J\u00e4rvelin K (2009) Explaining user performance in information retrieval: Challenges to IR evaluation. Advances in Information Retrieval Theory 289\u2013296","DOI":"10.1007\/978-3-642-04417-5_28"},{"issue":"4","key":"27_CR18","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1145\/582415.582418","volume":"20","author":"K J\u00e4rvelin","year":"2002","unstructured":"J\u00e4rvelin K, Kek\u00e4l\u00e4inen J (2002) Cumulated gain\u2013based evaluation of ir techniques. ACM Transactions on Information Systems 20(4):422\u2013446","journal-title":"ACM Transactions on Information Systems"},{"key":"27_CR19","unstructured":"Jesus R, Magalh\u00e3es J, Yavlinsky A, R\u00fcger S (2005) Imperial college at trecvid. In: TREC Video Retrieval Evaluation, Gaithersburg, MD"},{"key":"27_CR20","unstructured":"Llorente A, Zagorac S, Little S, Hu R, Kumar A, Shaik S, Ma X, R\u00fcger S (2008) Semantic video annotation using background knowledge and similarity\u2013based video retrieval. In: TREC Video Retrieval Evaluation (TRECVid, Gaithersburg, MD)"},{"key":"27_CR21","unstructured":"Llorente A, Little S, R\u00fcger S (2009) MMIS at ImageCLEF 2009: Non\u2013parametric density estimation algorithms. In: Working notes of CLEF 2009, Corfu, Greece"},{"key":"27_CR22","unstructured":"Magalh\u00e3es J, Overell S, Yavlinsky A, R\u00fcger S (2006) Imperial college at TRECVID. In: TREC Video Retrieval Evaluation, Gaithersburg, MD"},{"key":"27_CR23","doi-asserted-by":"crossref","unstructured":"Mandl T, Gey F, Di Nunzio G, Ferro N, Larson R, Sanderson M, Santos D, Womser-Hacker C, Xie X (2008) GeoCLEF 2007: the CLEF 2007 cross\u2013language geographic information retrieval track overview. In: Advances in Multilingual and Multimodal Information Retrieval. Lecture Notes in Computer Science (LNCS), vol 5152. Springer, pp 745\u2013772","DOI":"10.1007\/978-3-540-85760-0_96"},{"issue":"1","key":"27_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1416950.1416952","volume":"27","author":"A Moffat","year":"2008","unstructured":"Moffat A, Zobel J (2008) Rank\u2013biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems 27(1):1\u201327","journal-title":"ACM Transactions on Information Systems"},{"key":"27_CR25","doi-asserted-by":"crossref","unstructured":"Nowak S, Lukashevich H, Dunker P, R\u00fcger S (2010) Performance measures for multilabel classification \u2014 a case study in the area of image classification. In: ACM SIGMM International Conference on Multimedia Information Retrieval, Philadelphia, Pennsylvania","DOI":"10.1145\/1743384.1743398"},{"key":"27_CR26","unstructured":"Over P, Smeaton A (eds) (2007) TVS 2007: proceedings of the international workshop on TRECVid video summarization. ACM press"},{"key":"27_CR27","unstructured":"Overell S, Magalh\u00e3es J, R\u00fcger S (2006) Place disambiguation with co\u2013occurrence models. In: Working notes CLEF 2006, Alicante, Spain"},{"key":"27_CR28","unstructured":"Overell S, Llorente A, Liu HM, Hu R, Rae A, Zhu J, Song D, R\u00fcger S (2008) MMIS at ImageCLEF 2008: Experiments combining different evidence sources. In: Working notes of CLEF 2008, Aarhus, Denmark"},{"key":"27_CR29","unstructured":"Pickering M, R\u00fcger S (2002) Multi\u2013timescale video shot\u2013change detection. In: Text Retrieval Conf, NIST (Trec, Gaithersburg, MD, Nov 2001), NIST Special Publication 500\u2013250, pp 275\u2013278"},{"key":"27_CR30","doi-asserted-by":"crossref","unstructured":"van Rijsbergen CJ (1989) Towards an information logic. In: Proceedings of the 12th annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 77\u201386","DOI":"10.1145\/75335.75344"},{"key":"27_CR31","doi-asserted-by":"crossref","unstructured":"R\u00fcger S (2010) Multimedia information retrieval. Lecture notes in the series Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan\u2013Claypool","DOI":"10.1007\/978-3-031-02269-2"},{"issue":"2","key":"27_CR32","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1016\/j.ipm.2006.07.020","volume":"43","author":"T Sakai","year":"2007","unstructured":"Sakai T (2007) On the reliability of information retrieval metrics based on graded relevance. Information Processing & Management 43(2):531\u2013548","journal-title":"Information Processing & Management"},{"issue":"4","key":"27_CR33","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1016\/0306-4573(92)90002-H","volume":"28","author":"G Salton","year":"1992","unstructured":"Salton G (1992) The state of retrieval system evaluation. Information Processing and Management 28(4):441\u2013449. Special Issue: Evaluation Issues in Information Retrieval","journal-title":"Information Processing and Management"},{"key":"27_CR34","doi-asserted-by":"crossref","unstructured":"Saracevic T (1995) Evaluation of evaluation in information retrieval. In: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval, pp 138\u2013146. ACM press","DOI":"10.1145\/215206.215351"},{"issue":"2","key":"27_CR35","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1145\/270886.270889","volume":"31","author":"T Saracevic","year":"1997","unstructured":"Saracevic T (1997) Users lost: Reflections on the past, future, and limits of information science. ACM SIGIR Forum 31(2):16\u201327","journal-title":"ACM SIGIR Forum"},{"key":"27_CR36","doi-asserted-by":"crossref","unstructured":"Smeaton A, Over P, Kraaij W (2006) Evaluation campaigns and TRECVid. In: ACM International Workshop on Multimedia Information Retrieval. ACM press","DOI":"10.1145\/1178677.1178722"},{"key":"27_CR37","doi-asserted-by":"crossref","unstructured":"Smeaton A, Over P, Kraaij W (2009) High\u2013level feature detection from video in TRECVid: a 5\u2013year retrospective of achievements. In: Divakaran A (ed) Multimedia Content Analysis: Theory and Applications. Springer, pp 151\u2013174","DOI":"10.1007\/978-0-387-76569-3_6"},{"issue":"12","key":"27_CR38","doi-asserted-by":"publisher","first-page":"1349","DOI":"10.1109\/34.895972","volume":"22","author":"A Smeulders","year":"2000","unstructured":"Smeulders A, Worring M, Santini S, Gupta A, Jain R (2000) Content\u2013based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1349\u20131380","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"1","key":"27_CR39","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1109\/MMUL.2008.21","volume":"15","author":"CGM Snoek","year":"2008","unstructured":"Snoek CGM, Worring M, de\u00a0Rooij O, van de Sande KEA, Yan R, Hauptmann AG (2008) VideOlympics: Real\u2013time evaluation of multimedia retrieval systems. IEEE MultiMedia 15(1):86\u201391","journal-title":"IEEE MultiMedia"},{"key":"27_CR40","doi-asserted-by":"crossref","unstructured":"Soboroff I, Nicholas C, Cahan P (2001) Ranking retrieval systems without relevance judgments. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, pp 66\u201373. ACM press","DOI":"10.1145\/383952.383961"},{"key":"27_CR41","doi-asserted-by":"crossref","unstructured":"Turpin A, Hersh W (2001) Why batch and user evaluations do not give the same results. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, pp 225\u2013231. ACM press","DOI":"10.1145\/383952.383992"},{"issue":"5","key":"27_CR42","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1016\/S0306-4573(00)00010-8","volume":"36","author":"EM Voorhees","year":"2000","unstructured":"Voorhees EM (2000) Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management 36(5):697\u2013716","journal-title":"Information Processing & Management"},{"key":"27_CR43","series-title":"Lecture Notes in Computer Science (LNCS)","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1007\/3-540-45691-0_34","volume-title":"Revised Papers from the Second Workshop of the Cross\u2013Language Evaluation Forum on Evaluation of Cross\u2013Language Information Retrieval Systems \u2014 CLEF 2001","author":"EM Voorhees","year":"2002","unstructured":"Voorhees EM (2002) The philosophy of information retrieval evaluation. In: Revised Papers from the Second Workshop of the Cross\u2013Language Evaluation Forum on Evaluation of Cross\u2013Language Information Retrieval Systems \u2014 CLEF 2001. Lecture Notes in Computer Science (LNCS). Springer, London, UK, pp 355\u2013370"},{"key":"27_CR44","doi-asserted-by":"crossref","unstructured":"Westerveld T, van Zwol R (2006) The INEX 2006 multimedia track. In: Comparative Evaluation of XML Information Retrieval Systems, International Workshop of the Initiative for the Evaluation of XML Retrieval. Lecture Notes in Computer Science (LNCS), vol 4518. Springer, pp 331\u2013344","DOI":"10.1007\/978-3-540-73888-6_33"},{"key":"27_CR45","unstructured":"Zagorac S, Llorente A, Little S, Liu HM, R\u00fcger S (2009) Automated content based video retrieval. In: TREC Video Retrieval Evaluation (TRECVid, Gaithersburg, MD)"},{"key":"27_CR46","doi-asserted-by":"crossref","unstructured":"Zobel J (1998) How reliable are the results of large\u2013scale information retrieval experiments? In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 307\u2013314","DOI":"10.1145\/290941.291014"},{"key":"27_CR47","doi-asserted-by":"crossref","unstructured":"van Zwol R, Kazai G, Lalmas M (2005) INEX 2005 multimedia track. In: Advances in XML Information Retrieval and Evaluation, International Workshop of the Initiative for the Evaluation of XML Retrieval. Lecture Notes in Computer Science (LNCS), vol 3977. Springer, pp 497\u2013510","DOI":"10.1007\/978-3-540-34963-1_38"}],"container-title":["The Information Retrieval Series","ImageCLEF"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-642-15181-1_27","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,10]],"date-time":"2023-02-10T23:31:48Z","timestamp":1676071908000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-642-15181-1_27"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010]]},"ISBN":["9783642151804","9783642151811"],"references-count":47,"URL":"https:\/\/doi.org\/10.1007\/978-3-642-15181-1_27","relation":{},"ISSN":["1387-5264"],"issn-type":[{"type":"print","value":"1387-5264"}],"subject":[],"published":{"date-parts":[[2010]]}}}