{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T18:45:04Z","timestamp":1774291504567,"version":"3.50.1"},"publisher-location":"Cham","reference-count":48,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783030229474","type":"print"},{"value":"9783030229481","type":"electronic"}],"license":[{"start":{"date-parts":[[2019,1,1]],"date-time":"2019-01-01T00:00:00Z","timestamp":1546300800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019]]},"DOI":"10.1007\/978-3-030-22948-1_13","type":"book-chapter","created":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T05:02:59Z","timestamp":1565672579000},"page":"307-329","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["About Sound and Vision: CLEF Beyond Text Retrieval Tasks"],"prefix":"10.1007","author":[{"given":"Gareth J. F.","family":"Jones","sequence":"first","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,8,14]]},"reference":[{"key":"13_CR1","unstructured":"Akiba T, Nishizaki H, Aikawa K, Kawahara T, Matsui T (2011) Overview of the IR for spoken documents task in NTCIR-9 workshop. In: Kando N, Ishikawa D, Sugimoto M (eds) Proceedings of the 9th NTCIR workshop meeting on evaluation of information access technologies: information retrieval, question answering and cross-lingual information access. National Institute of Informatics, Tokyo"},{"key":"13_CR2","unstructured":"Akiba T, Nishizaki H, Aikawa K, Hu X, Itoh Y, Kawahara T, Nakagawa S, Nanjo H, Yamashita Y (2013) Overview of the NTCIR-10 spokendoc-2 task. In: Kando N, Kishida K (eds) Proceedings of the 10th NTCIR conference on evaluation of information access technologies. National Institute of Informatics, Tokyo"},{"key":"13_CR3","unstructured":"Akiba T, Nishizaki H, Nanjo H, Jones GJF (2014) Overview of the NTCIR-11 spokenquery&doc task. In: Kando N, Joho H, Kishida K (eds) Proceedings of the 11th NTCIR conference on evaluation of information access technologies. National Institute of Informatics, Tokyo"},{"key":"13_CR4","unstructured":"Akiba T, Nishizaki H, Nanjo H, Jones GJF (2016) Overview of the ntcir-12 spokenquery&doc-2 task. In: Kando N, Sakai T, Sanderson M (eds) Proceedings of the 12th NTCIR conference on evaluation of information access technologies. National Institute of Informatics, Tokyo"},{"key":"13_CR5","doi-asserted-by":"crossref","unstructured":"Aly R, Ordelman R, Eskevich M, Jones GJF, Chen S (2013) Linking inside a video collection - what and how to measure? In: Proceedings of the first worldwide web workshop on linked media (LiME-2013), International World Wide Web Conference Committee (IW3C2), Geneva","DOI":"10.1145\/2487788.2487967"},{"key":"13_CR6","unstructured":"Awad G, Fiscus J, Joy D, Michel M, Smeaton AF, Kraaij W, Eskevich M, Aly R, Ordelman R, Jones GJF, Huet B, Larson M (2016) TRECVID 2016: evaluating video search, video event detection, localization, and hyperlinking. In: The sixteenth international workshop on video retrieval evaluation (TRECVID 2016). National Institute of Standards and Technology (NIST), Special Publication 500\u2013321, Washington"},{"key":"13_CR7","unstructured":"Awad G, Butt A, Fiscus J, Joy D, Delgado A, Mcclinton W, Michel M, Smeaton A, Graham Y, Kraaij W, Qu\u00e9not G, Eskevich M, Roeland Ordelman GJFJ, Huet B (2017) Trecvid 2017: evaluating ad-hoc and instance video search, events detection, video captioning, and hyperlinking. In: The seventeenth international workshop on video retrieval evaluation (TRECVID 2017). National Institute of Standards and Technology (NIST), Special Publication 500\u2013321, Washington"},{"issue":"4","key":"13_CR8","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1109\/TSA.2004.828702","volume":"12","author":"W. Byrne","year":"2004","unstructured":"Byrne W, Doermann D, Franz M, Member S, Gustman S, Soergel D, Ward T, jing Zhu W (2004) Automatic recognition of spontaneous speech for access to multilingual oral history archives. IEEE Trans Speech Audio Process 12(4):420\u2013435","journal-title":"IEEE Transactions on Speech and Audio Processing"},{"issue":"1","key":"13_CR9","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1145\/1147197.1147199","volume":"40","author":"P Clough","year":"2006","unstructured":"Clough P, Sanderson M, Reid N (2006) The Eurovision St Andrews collection of photographs. SIGIR Forum 40(1):21\u201330","journal-title":"SIGIR Forum"},{"issue":"5","key":"13_CR10","first-page":"1021","volume":"28","author":"M Eskevich","year":"2014","unstructured":"Eskevich M, Jones GJF (2014) Exploring speech retrieval from meetings using the AMI corpus. Comput Speech Lang (Special Issue on Information Extraction and Retrieval) 28(5):1021\u20131044","journal-title":"Comput Speech Lang (Special Issue on Information Extraction and Retrieval)"},{"key":"13_CR11","unstructured":"Eskevich M, Jones GJF, Chen S, Aly R, Ordelman R, Larson M (2012a) Search and hyperlinking task at mediaeval 2012. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working Notes Proceedings of the MediaEval 2012 multimedia benchmark workshop. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, \n                    http:\/\/ceur-ws.org\/Vol-927\/"},{"key":"13_CR12","doi-asserted-by":"crossref","unstructured":"Eskevich M, Jones GJF, Larson M, Wartena C, Aly R, Verschoor T, Ordelman R (2012b) Comparing retrieval effectiveness for alternative content segmentation methods for internet video. In: Proceedings of the 10th workshop on content-based multimedia indexing. IEEE, New Jersey, CBMI 2012","DOI":"10.1109\/CBMI.2012.6269810"},{"key":"13_CR13","unstructured":"Eskevich M, Jones GJF, Chen S, Aly R, Ordelman R (2013) The search and hyperlinking task at mediaeval 2013. In: Larson M, Anguera X, Reuter T, Jones GJF, Ionescu B, Schedl M, Piatrik T, Hauff C, Soleymani M (eds) Working notes proceedings of the MediaEval 2013 multimedia benchmark workshop. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, \n                    http:\/\/ceur-ws.org\/Vol-1043\/"},{"key":"13_CR14","unstructured":"Eskevich M, Aly R, Racca DN, Ordelman R, Chen S, Jones GJF (2014) The search and hyperlinking task at mediaeval 2014. In: Larson M, Ionescu B, Anguera X, Eskevich M, Korshunov P, Schedl M, Soleymani M, Petkos P, Sutcliffe R, Choi J, Jones GJF (eds) Working notes proceedings of the MediaEval 2014 multimedia benchmark workshop. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, \n                    http:\/\/ceur-ws.org\/Vol-1263\/"},{"key":"13_CR15","unstructured":"Eskevich M, Aly R, Ordelman R, Racca DN, Chen S, Jones GJF (2015) SAVA at Mediaeval 2015: Search and anchoring in video archives. In: Larson M, Ionescu B, Sj\u00f6berg M, Anguera X, Poignant J, Riegler M, Eskevich M, Hauff C, Sutcliffe R, Jones GJF, Yang YH, Soleymani M, Papadopoulos S (eds) Working notes proceedings of the MediaEval 2015 multimedia benchmark workshop. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, \n                    http:\/\/ceur-ws.org\/Vol-1436\/"},{"key":"13_CR16","doi-asserted-by":"publisher","first-page":"646","DOI":"10.1007\/978-3-540-30222-3_61","volume-title":"Comparative Evaluation of Multilingual Information Access Systems","author":"Marcello Federico","year":"2004","unstructured":"Federico M, Jones GJF (2004) The CLEF 2003 cross-language spoken document retrieval track. In: Peters C, Braschler M, Gonzalo J, Kluck M (eds) Comparative evaluation of multilingual information access systems: fourth workshop of the cross\u2013language evaluation forum (CLEF 2003) revised selected papers. Lecture notes in computer science (LNCS), vol 3237. Springer, Heidelberg, p 646"},{"key":"13_CR17","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"816","DOI":"10.1007\/11519645_79","volume-title":"Multilingual information access for text, speech and images: fifth workshop of the cross\u2013language evaluation forum (CLEF 2004) revised selected papers","author":"M Federico","year":"2005","unstructured":"Federico M, Bertoldi N, Levow GA, Jones GJF (2005) CLEF 2004 cross-language spoken document retrieval track. In: Peters C, Clough P, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual information access for text, speech and images: fifth workshop of the cross\u2013language evaluation forum (CLEF 2004) revised selected papers. Lecture notes in computer science (LNCS), vol 3491. Springer, Heidelberg, pp 816\u2013820"},{"key":"13_CR18","unstructured":"Garofolo JS, Auzanne CGP, Voorhees EM (2000) The trec spoken document retrieval track: a success story. In: Content-Based Multimedia Information Access - vol 1, LE CENTRE DE HAUTES ETUDES INTERNATIONALES D\u2019INFORMATIQUE DOCUMENTAIRE, Paris, France, France, RIAO \u201800, pp 1\u201320"},{"key":"13_CR19","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1145\/133160.133194","volume-title":"Proceedings of the 15th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1992)","author":"U Glavitsch","year":"1992","unstructured":"Glavitsch U, Sch\u00e4uble P (1992) A system for retrieving speech documents. In: Belkin NJ, Ingwersen P, Mark Pejtersen A, Fox EA (eds) Proceedings of the 15th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1992). ACM Press, New York, pp 168\u2013176"},{"key":"13_CR20","first-page":"215","volume-title":"Intelligent multimedia information retrieval","author":"AG Hauptmann","year":"1997","unstructured":"Hauptmann AG, Witbrock MJ (1997) Informedia: news-on-demand multimedia information acquisition and retrieval. In: Maybury MT (ed) Intelligent multimedia information retrieval. MIT Press, Cambridge, pp 215\u2013239"},{"key":"13_CR21","unstructured":"James DA (1995) The application of classical information retrieval techniques to spoken documents. PhD thesis, Cambridge University"},{"key":"13_CR22","unstructured":"Jones GJF (2000) Applying machine translation resources for cross-language information access from spoken documents. In: Proceedings of MT 2000: machine translation and multilingual applications in the new millennium. British Computer Society, pp 4-1\u20134-9"},{"key":"13_CR23","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1007\/3-540-44645-1_7","volume-title":"Cross-language information retrieval and evaluation: workshop of cross-language evaluation forum (CLEF 2000)","author":"GJF Jones","year":"2001","unstructured":"Jones GJF (2001) New challenges for cross-language information retrieval: multimedia data and the user experience. In: Peters C (ed) Cross-language information retrieval and evaluation: workshop of cross-language evaluation forum (CLEF 2000). Lecture notes in computer science (LNCS), vol 2069. Springer, Heidelberg, pp 71\u201381"},{"key":"13_CR24","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1007\/978-3-642-36415-0_9","volume-title":"Information retrieval meets information visualization \u2013 PROMISE Winter School 2012, Revised Tutorial Lectures","author":"GJF Jones","year":"2013","unstructured":"Jones GJF (2013) An introduction to crowdsourcing for language and multimedia technology research. In: Agosti M, Ferro N, Forner P, M\u00fcller H, Santucci G (eds) Information retrieval meets information visualization \u2013 PROMISE Winter School 2012, Revised Tutorial Lectures. Lecture notes in computer science (LNCS), vol 7757. Springer, Heidelberg, pp 132\u2013154"},{"key":"13_CR25","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1007\/978-3-540-45237-9_39","volume-title":"Advances in cross-language information retrieval: third workshop of the cross\u2013language evaluation forum (CLEF 2002) Revised Papers","author":"GJF Jones","year":"2003","unstructured":"Jones GJF, Federico M (2003) CLEF 2002 cross-language spoken document retrieval pilot track report. In: Peters C, Braschler M, Gonzalo J, Kluck M (eds) Advances in cross-language information retrieval: third workshop of the cross\u2013language evaluation forum (CLEF 2002) Revised Papers. Lecture notes in computer science (LNCS), vol 2785. Springer, Heidelberg, pp 446\u2013457"},{"key":"13_CR26","first-page":"30","volume-title":"Proceedings of the 19th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1996)","author":"GJF Jones","year":"1996","unstructured":"Jones GJF, Foote JT, Sp\u00e4rck Jones K, Young SJ (1996) Retrieving spoken documents by combining multiple index sources. In: Frei HP, Harman D, Scha\u00fcbie P, Wilkinson R (eds) Proceedings of the 19th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1996). ACM Press, New York, pp 30\u201338"},{"issue":"13","key":"13_CR27","doi-asserted-by":"publisher","first-page":"1120","DOI":"10.1002\/asi.10137","volume":"53","author":"J Kek\u00e4l\u00e4inen","year":"2002","unstructured":"Kek\u00e4l\u00e4inen J, J\u00e4rvelin K (2002) Using graded relevance assessments in IR evaluation. J Am Soc Inf Sci Technol 53(13):1120\u20131129","journal-title":"J Am Soc Inf Sci Technol"},{"key":"13_CR28","first-page":"1","volume":"2016","author":"A Khwileh","year":"2016","unstructured":"Khwileh A, Jones GJF (2016) Investigating segment-based query expansion for user-generated spoken content retrieval. In: 14th international workshop on content-based multimedia indexing, IEEE, CBMI 2016, pp 1\u20136","journal-title":"IEEE, CBMI"},{"key":"13_CR29","doi-asserted-by":"crossref","unstructured":"Khwileh A, Afli H, Jones GJF, Way A (2017) Identifying effective translations for cross-lingual arabic-to-english user-generated speech search. In: Proceedings of the third arabic natural language processing workshop. Association for Computational Linguistics, pp 100\u2013109","DOI":"10.18653\/v1\/W17-1313"},{"issue":"4\u20135","key":"13_CR30","first-page":"235","volume":"5","author":"M Larson","year":"2011","unstructured":"Larson M, Jones GJF (2011) Spoken content retrieval: a survey of techniques and technologies. Found Trends Inf Retr 5(4\u20135):235\u2014422","journal-title":"Found Trends Inf Retr"},{"key":"13_CR31","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1007\/978-3-642-04447-2_119","volume-title":"Evaluating systems for multilingual and multimodal information access: ninth workshop of the cross-language evaluation forum (CLEF 2008). Revised selected papers","author":"M Larson","year":"2009","unstructured":"Larson M, Newman E, Jones GJF (2009) Overview of VideoCLEF 2008: automatic generation of topic-based feeds for dual language audio-visual content. In: Peters C, Deselaers T, Ferro N, Gonzalo J, Jones GJF, Kurimo M, Mandl T, Pe\u00f1as A (eds) Evaluating systems for multilingual and multimodal information access: ninth workshop of the cross-language evaluation forum (CLEF 2008). Revised selected papers. Lecture notes in computer science (LNCS), vol 5706. Springer, Heidelberg, pp 906\u2013917"},{"key":"13_CR32","doi-asserted-by":"crossref","unstructured":"Larson M, Newman E, Jones GJF (2010) Overview of VideoCLEF 2009: new perspectives on speech-based multimedia content enrichment. In: Peters C, Tsikrika T, M\u00fcller H, Kalpathy-Cramer J, Jones GJF, Gonzalo J, Caputo B (eds) Multilingual information access evaluation Vol. II multimedia experiments \u2013 tenth workshop of the cross\u2013language evaluation forum (CLEF 2009). Revised selected papers. Lecture notes in computer science (LNCS). Springer, Heidelberg, pp 354\u2013368","DOI":"10.1007\/978-3-642-15751-6_46"},{"key":"13_CR33","unstructured":"Larson M, Eskevich M, Ordelman R, Kofler C, Schmiedeke S, Jones GJF (2011) Overview of mediaeval 2011 rich speech retrieval task and genre tagging task. In: Larson M, Rae A, Demarty CH, Kofler C, Metze F, Troncy R, Mezaris V, Jones GJF (eds) Working notes proceedings of the MediaEval 2011 multimedia benchmark workshop. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, \n                    http:\/\/ceur-ws.org\/Vol-807\/"},{"key":"13_CR34","volume-title":"Content-based video retrieval: an overview","author":"S Marchand-Maillet","year":"2000","unstructured":"Marchand-Maillet S (2000) Content-based video retrieval: an overview. Technical report, Computer Vision Group, Computing Science Center, University of Geneva"},{"key":"13_CR35","doi-asserted-by":"publisher","first-page":"5270","DOI":"10.1109\/ICASSP.2010.5494979","volume-title":"Proceedings of the IEEE international conference on acoustics, speech, and signal processing (ICASSP 2010)","author":"M Marge","year":"2010","unstructured":"Marge M, Banerjee S, Rudnicky AI (2010) Using the Amazon Mechanical Turk for transcription of spoken language. In: Proceedings of the IEEE international conference on acoustics, speech, and signal processing (ICASSP 2010). IEEE, Piscataway, pp 5270\u20135273"},{"key":"13_CR36","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"744","DOI":"10.1007\/978-3-540-74999-8_94","volume-title":"Evaluation of multilingual and multi-modal information retrieval: seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers","author":"DW Oard","year":"2007","unstructured":"Oard DW, Wang J, Jones GJF, White RW, Pecina P, Soergel D, Huang X, Shafran I (2007) Overview of the CLEF-2006 cross-language speech retrieval track. In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval: seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers. Lecture notes in computer science (LNCS), vol 4730. Springer, Heidelberg, pp 744\u2013758"},{"key":"13_CR37","doi-asserted-by":"crossref","unstructured":"Ordelman RJF, Eskevich M, Aly R, Huet B, Jones GJF (2015) Defining and evaluating video hyperlinking for navigating multimedia archives. In: Proceedings of the 24th international conference on world wide web. ACM, New York, WWW \u201815 Companion, pp 727\u2013732","DOI":"10.1145\/2740908.2742915"},{"key":"13_CR38","unstructured":"Over P, Fiscus J, Joy D, Michel M, Awad G, Smeaton A, Kraaij W, Qu\u00e9not G, Ordelman R, Aly R (2015) Trecvid 2015 \u2013 an overview of the goals, tasks, data, evaluation mechanisms and metrics. In: The fifteenth international workshop on video retrieval evaluation (TRECVID 2015). National Institute of Standards and Technology (NIST), Special Publication 500-321, Washington"},{"key":"13_CR39","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"674","DOI":"10.1007\/978-3-540-85760-0_86","volume-title":"Advances in multilingual and multimodal information retrieval: eighth workshop of the cross\u2013language evaluation forum (CLEF 2007). Revised selected papers","author":"P Pecina","year":"2008","unstructured":"Pecina P, Hoffmannov\u00e1 P, Jones GJF, Zhang Y, Oard DW (2008) Overview of the CLEF-2007 cross-language speech retrieval track. In: Peters C, Jijkoun V, Mandl T, M\u00fcller H, Oard DW, Pe\u00f1as A, Petras V, Santos D (eds) Advances in multilingual and multimodal information retrieval: eighth workshop of the cross\u2013language evaluation forum (CLEF 2007). Revised selected papers. Lecture notes in computer science (LNCS), vol 5152. Springer, Heidelberg, pp 674\u2013686"},{"key":"13_CR40","first-page":"933","volume-title":"Proceedings of the 39th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2016)","author":"DN Racca","year":"2016","unstructured":"Racca DN, Jones GJ (2016) On the effectiveness of contextualisation techniques in spoken query spoken content retrieval. In: Perego R, Sebastiani F, Aslam J, Ruthven I, Zobel J (eds) Proceedings of the 39th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2016). ACM Press, New York, pp 933\u2013936"},{"key":"13_CR41","unstructured":"Rashtchian C, Young P, Hodosh M, Hockenmaier J (2010) Collecting image annotations using Amazon\u2019s Mechanical Turk. In: Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon\u2019s Mechanical Turk, Association for Computational Linguistics, pp 139\u2013147"},{"key":"13_CR42","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1007\/978-3-540-71496-5_45","volume-title":"Advances in information retrieval. Proceedings of the 29th European conference on IR research (ECIR 2007)","author":"M Sanderson","year":"2007","unstructured":"Sanderson M, Shou XM (2007) Search of spoken documents retrieves well recognized transcripts. In: Amati G, Carpineto C, Romano G (eds) Advances in information retrieval. Proceedings of the 29th European conference on IR research (ECIR 2007). Lecture notes in computer science (LNCS), vol 4425. Springer, Heidelberg, pp 505\u2013516"},{"key":"13_CR43","doi-asserted-by":"crossref","unstructured":"Schmiedeke S, Xu P, Ferran\u00e9 I, Eskevich M, Kofler C, Larson M, Est\u00e8ve Y, Lamel L, Jones GJF, Sikora T (2013) Blip10000: a social video dataset containing SPUG content for tagging and retrieval. In: Proceedings of ACM multimedia systems. ACM, New York, MMSys\u201913","DOI":"10.1145\/2483977.2483988"},{"issue":"1","key":"13_CR44","first-page":"1","volume":"1","author":"K Schoeffmann","year":"2010","unstructured":"Schoeffmann K, Hopfgartner F, Marques O, B\u00f6sz\u00f6rmenyi L, Jose JM (2010) Video browsing interfaces and applications: a review. SPIE Rev 1(1):1\u201335","journal-title":"SPIE Rev"},{"key":"13_CR45","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1145\/258525.258544","volume-title":"Proceedings of the 20th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1997)","author":"P Sheridan","year":"1997","unstructured":"Sheridan P, Wechsler M, Sch\u00e4uble P (1997) Cross-language speech retrieval: establishing a baseline performance. In: Belkin NJ, Narasimhalu AD, Willett P, Hersh W, Can F, Voorhees EM (eds) Proceedings of the 20th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 1997). ACM Press, New York, pp 99\u2013108"},{"key":"13_CR46","doi-asserted-by":"crossref","unstructured":"Smeaton AF, Over P, Kraaij W (2006) Evaluation campaigns and trecvid. In: Proceedings of the 8th ACM international workshop on multimedia information retrieval. ACM, New York, MIR \u201806, pp 321\u2013330","DOI":"10.1145\/1178677.1178722"},{"issue":"12","key":"13_CR47","doi-asserted-by":"publisher","first-page":"1349","DOI":"10.1109\/34.895972","volume":"22","author":"AWM Smeulders","year":"2000","unstructured":"Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349\u20131380","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"13_CR48","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"744","DOI":"10.1007\/11878773_82","volume-title":"Accessing multilingual information repositories: sixth workshop of the cross\u2013language evaluation forum (CLEF 2005). Revised selected papers","author":"RW White","year":"2006","unstructured":"White RW, Oard DW, Jones GJF, Soergel D, Huang X (2006) Overview of the CLEF-2005 cross-language speech retrieval track. In: Peters C, Gey FC, Gonzalo J, Jones GJF, Kluck M, Magnini B, M\u00fcller H, de Rijke M (eds) Accessing multilingual information repositories: sixth workshop of the cross\u2013language evaluation forum (CLEF 2005). Revised selected papers. Lecture notes in computer science (LNCS), vol 4022. Springer, Heidelberg, pp 744\u2013759"}],"container-title":["The Information Retrieval Series","Information Retrieval Evaluation in a Changing World"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-22948-1_13","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T05:06:55Z","timestamp":1565672815000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/978-3-030-22948-1_13"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"ISBN":["9783030229474","9783030229481"],"references-count":48,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-22948-1_13","relation":{},"ISSN":["1387-5264"],"issn-type":[{"value":"1387-5264","type":"print"}],"subject":[],"published":{"date-parts":[[2019]]},"assertion":[{"value":"14 August 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}}]}}