{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,13]],"date-time":"2024-09-13T07:50:35Z","timestamp":1726213835002},"publisher-location":"Cham","reference-count":33,"publisher":"Springer International Publishing","isbn-type":[{"type":"print","value":"9783030229474"},{"type":"electronic","value":"9783030229481"}],"license":[{"start":{"date-parts":[[2019,1,1]],"date-time":"2019-01-01T00:00:00Z","timestamp":1546300800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019]]},"DOI":"10.1007\/978-3-030-22948-1_18","type":"book-chapter","created":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T01:02:59Z","timestamp":1565658179000},"page":"441-460","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Results and Lessons of the Question Answering Track at CLEF"],"prefix":"10.1007","author":[{"given":"Anselmo","family":"Pe\u00f1as","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"\u00c1lvaro","family":"Rodrigo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bernardo","family":"Magnini","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pamela","family":"Forner","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eduard","family":"Hovy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Richard","family":"Sutcliffe","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Danilo","family":"Giampiccolo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,8,14]]},"reference":[{"key":"18_CR1","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"300","DOI":"10.1007\/978-3-540-74999-8_37","volume-title":"Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers","author":"A Cassan","year":"2007","unstructured":"Cassan A, Figueira H, Martins A, Mendes A, Mendes P, Pinto C, Vidal D (2007) Priberam\u2019s question answering system in a cross-language environment. In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers. Lecture notes in computer science (LNCS) 4730. Springer, Heidelberg, pp 300\u2013309"},{"issue":"1","key":"18_CR2","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1609\/aimag.v37i1.2636","volume":"37","author":"P Clark","year":"2016","unstructured":"Clark P, Etzioni O (2016) My computer is an honor student - but how intelligent is it? Standardized tests as a measure of AI. AI Mag 37(1):5\u201312","journal-title":"AI Mag"},{"issue":"3","key":"18_CR3","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1609\/aimag.v31i3.2303","volume":"31","author":"DA Ferrucci","year":"2010","unstructured":"Ferrucci DA, Brown EW, Chu-Carroll J, Fan J, Gondek D, Kalyanpur A, Lally A, Murdock JW, Nyberg E, Prager JM, Schlaefer N, Welty CA (2010) Building Watson: an overview of the DeepQA project. AI Mag 31(3):59\u201379","journal-title":"AI Mag"},{"key":"18_CR4","first-page":"262","volume-title":"Lecture Notes in Computer Science","author":"Pamela Forner","year":"2009","unstructured":"Forner P, Pe\u00f1as A, Agirre E, Alegria I, Forascu C, Moreau N, Osenova P, Prokopidis P, Rocha P, Sacaleanu B, Sutcliffe RFE, Sang EFTK (2009) Overview of the Clef 2008 multilingual question answering track. In: Peters C, Deselaers T, Ferro N, Gonzalo J, Jones GJF, Kurimo M, Mandl T, Pe\u00f1as A (eds) Evaluating systems for multilingual and multimodal information access: ninth workshop of the cross\u2013language evaluation forum (CLEF 2008). Revised selected papers. Lecture notes in computer science (LNCS) 5706. Springer, Heidelberg, pp 262\u2013295"},{"key":"18_CR5","doi-asserted-by":"crossref","unstructured":"Giampiccolo D, Forner P, Herrera J, Pe\u00f1as A, Ayache C, Forascu C, Jijkoun V, Osenova P, Rocha P, Sacaleanu B, Sutcliffe RFE (2008) Overview of the CLEF 2007 multilingual question answering track. In: Peters C, Jijkoun V, Mandl T, M\u00fcller H, Oard DW, Pe\u00f1as A, Petras V, Santos D (eds) Advances in multilingual and multimodal information retrieval: eighth workshop of the cross\u2013language evaluation forum (CLEF 2007). Revised selected papers. Lecture notes in computer science (LNCS) 5152. Springer, Heidelberg, pp 200\u2013236","DOI":"10.1007\/978-3-540-85760-0_27"},{"key":"18_CR6","doi-asserted-by":"crossref","unstructured":"Harabagiu S, Lacatusu F, Hickl A (2006) Answering complex questions with random walk models. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR \u201906, pp 220\u2013227","DOI":"10.1145\/1148170.1148211"},{"key":"18_CR7","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1007\/11519645_57","volume-title":"Multilingual information access for text, speech and images: fifth workshop of the cross\u2013language evaluation forum (CLEF 2004) revised selected papers","author":"J Herrera","year":"2005","unstructured":"Herrera J, Pe\u00f1as A, Verdejo F (2005) Question answering pilot task at CLEF 2004. In: Peters C, Clough P, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual information access for text, speech and images: fifth workshop of the cross\u2013language evaluation forum (CLEF 2004) revised selected papers. Lecture notes in computer science (LNCS) 3491. Springer, Heidelberg, pp 581\u2013590"},{"key":"18_CR8","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1007\/978-3-540-74999-8_33","volume-title":"Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers","author":"V Jijkoun","year":"2007","unstructured":"Jijkoun V, de Rijke M (2007) Overview of the wiqa task at CLEF 2006. In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers. Lecture notes in computer science (LNCS) 4730. Springer, Heidelberg, pp 265\u2013274"},{"key":"18_CR9","unstructured":"Lamel L, Rosset S, Ayache C, Mostefa D, Turmo J, Comas P (2008) Question answering on speech transcriptions: the QAST evaluation in CLEF. In: Proceedings of the international conference on language resources and evaluation, LREC 2008, 26 May\u20131 June 2008, Marrakech, Morocco"},{"key":"18_CR10","unstructured":"Laurent D (2014) English run of synapse d\u00e9veloppement at entrance exams 2014. In: Cappellato L, Ferro N, Halvey M, Kraaij W (eds) CLEF 2014 labs and workshops, notebook papers. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. \n                    http:\/\/ceur-ws.org\/Vol-1180\/\n                    \n                  , pp 1404\u20131414"},{"key":"18_CR11","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1007\/978-3-540-74999-8_41","volume-title":"Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers","author":"D Laurent","year":"2007","unstructured":"Laurent D, S\u00e9gu\u00e9la P, N\u00e8gre S (2007) Cross lingual question answering using QRISTAL for CLEF 2006. In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers. Lecture notes in computer science (LNCS) 4730. Springer, Heidelberg, pp 339\u2013350"},{"key":"18_CR12","unstructured":"Laurent D, Chardon B, N\u00e8gre S (2014) French run of synapse d\u00e9veloppement at entrance exams 2014. In: Cappellato L, Ferro N, Halvey M, Kraaij W (eds) CLEF 2014 labs and workshops, notebook papers. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. \n                    http:\/\/ceur-ws.org\/Vol-1180\/\n                    \n                  , pp 1415\u20131426"},{"key":"18_CR13","unstructured":"Laurent D, Chardon B, N\u00e8gre S, Pradel C, S\u00e9gu\u00e9la P (2015) Reading comprehension at entrance exams 2015. In: Cappellato L, Ferro N, Jones GJF, SanJuan E (eds) CLEF 2015 labs and workshops, notebook papers. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. \n                    http:\/\/ceur-ws.org\/Vol-1391\/"},{"key":"18_CR14","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.websem.2013.05.006","volume":"21","author":"Vanessa Lopez","year":"2013","unstructured":"Lopez V, Unger C, Cimiano P, Motta E (2013) Evaluating question answering over linked data. Web Semantics: Sci Serv Agents World Wide Web 21(0):3\u201313. Special Issue on Evaluation of Semantic Technologies","journal-title":"Journal of Web Semantics"},{"key":"18_CR15","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1007\/978-3-540-30222-3_46","volume-title":"Comparative evaluation of multilingual information access systems: fourth workshop of the cross\u2013language evaluation forum (CLEF 2003) revised selected papers","author":"B Magnini","year":"2004","unstructured":"Magnini B, Romagnoli S, Vallin A, Herrera J, Pe\u00f1as A, Peinado V, Verdejo MF, de Rijke M (2004) The multiple language question answering track at CLEF 2003. In: Peters C, Braschler M, Gonzalo J, Kluck M (eds) Comparative evaluation of multilingual information access systems: fourth workshop of the cross\u2013language evaluation forum (CLEF 2003) revised selected papers. Lecture notes in computer science (LNCS) 3237. Springer, Heidelberg, pp 471\u2013486"},{"key":"18_CR16","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1007\/11519645_38","volume-title":"Multilingual information access for text, speech and images: fifth workshop of the cross\u2013language evaluation forum (CLEF 2004) Revised selected papers","author":"B Magnini","year":"2005","unstructured":"Magnini B, Vallin A, Ayache C, Erbach G, Pe\u00f1as A, de Rijke M, Rocha P, Simov KI, Sutcliffe RFE (2005) Overview of the CLEF 2004 multilingual question answering track. In: Peters C, Clough P, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual information access for text, speech and images: fifth workshop of the cross\u2013language evaluation forum (CLEF 2004) Revised selected papers. Lecture notes in computer science (LNCS) 3491. Springer, Heidelberg, pp 371\u2013391"},{"key":"18_CR17","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1007\/978-3-540-74999-8_31","volume-title":"Evaluation of Multilingual and Multi-modal Information Retrieval","author":"Bernardo Magnini","year":"2007","unstructured":"Magnini B, Giampiccolo D, Forner P, Ayache C, Jijkoun V, Osenova P, Pe\u00f1as A, Rocha P, Sacaleanu B, Sutcliffe RFE (2007) Overview of the CLEF 2006 multilingual question answering track. In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers. Lecture notes in computer science (LNCS) 4730. Springer, Heidelberg, pp 223\u2013256"},{"key":"18_CR18","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1007\/11878773_47","volume-title":"Accessing multilingual information repositories: sixth workshop of the cross\u2013language evaluation forum (CLEF 2005). Revised selected papers","author":"M Montes-y-G\u00f3mez","year":"2006","unstructured":"Montes-y-G\u00f3mez M, Pineda LV, P\u00e9rez-Couti\u00f1o MA, Soriano JMG, Arnal ES, Rosso P (2006) A full data-driven system for multiple language question answering. In: Peters C, Gey FC, Gonzalo J, Jones GJF, Kluck M, Magnini B, M\u00fcller H, de Rijke M (eds) Accessing multilingual information repositories: sixth workshop of the cross\u2013language evaluation forum (CLEF 2005). Revised selected papers. Lecture notes in computer science (LNCS) 4022. Springer, Heidelberg, pp 420\u2013428"},{"key":"18_CR19","unstructured":"Morante R, Daelemans W (2011) Overview of the QA4MRE pilot task: annotating modality and negation for a machine reading evaluation. In: Petras V, Forner P, Clough P, Ferro N (eds) CLEF 2011 working notes. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. \n                    http:\/\/ceur-ws.org\/Vol-1177\/"},{"key":"18_CR20","unstructured":"Morante R, Krallinger M, Valencia A, Daelemans W (2013) Machine reading of biomedical texts about alzheimer\u2019s disease. In: Forner P, Navigli R, Tufis D, Ferro N (eds) CLEF 2013 working notes. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. \n                    http:\/\/ceur-ws.org\/Vol-1179\/"},{"key":"18_CR21","doi-asserted-by":"crossref","unstructured":"Noguera E, Llopis F, Ferr\u00e1ndez A, Escapa A (2007) Evaluation of open-domain question answering systems within a time constraint. In: 21st International conference on advanced information networking and applications (AINA 2007). Workshops proceedings, vol 1, May 21\u201323, 2007, Niagara Falls, Canada, pp 260\u2013265","DOI":"10.1109\/AINAW.2007.176"},{"key":"18_CR22","unstructured":"Pe\u00f1as A, Rodrigo A (2011) A simple measure to assess non-response. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies - vol 1. Association for Computational Linguistics, HLT \u201911, pp 1415\u20131424"},{"key":"18_CR23","series-title":"Lecture notes in computer science (LNCS)","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1007\/978-3-540-74999-8_32","volume-title":"Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers","author":"A Pe\u00f1as","year":"2007","unstructured":"Pe\u00f1as A, Rodrigo \u00c1, Sama V, Verdejo F (2007) Overview of the answer validation exercise 2006. In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval : seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers. Lecture notes in computer science (LNCS) 4730. Springer, Heidelberg, pp 257\u2013264"},{"key":"18_CR24","unstructured":"Pe\u00f1as A, Forner P, Rodrigo A, Sutcliffe RFE, Forascu C, Mota C (2010a) Overview of ResPubliQA 2010: question answering evaluation over European legislation. In: Braschler M, Harman DK, Pianta E, Ferro N (eds) CLEF 2010 working notes. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073, \n                    http:\/\/ceur-ws.org\/Vol-1176\/"},{"key":"18_CR25","first-page":"174","volume-title":"Lecture Notes in Computer Science","author":"Anselmo Pe\u00f1as","year":"2010","unstructured":"Pe\u00f1as A, Forner P, Sutcliffe RFE, Rodrigo A, Forascu C, Alegria I, Giampiccolo D, Moreau N, Osenova P (2010b) Overview of ResPubliQA 2009: question answering evaluation over European legislation. In: Peters C, Di Nunzio GM, Kurimo M, Mandl T, Mostefa D, Pe\u00f1as A, Roda G (eds) Multilingual information access evaluation vol. I text retrieval experiments \u2013 tenth workshop of the cross\u2013language evaluation forum (CLEF 2009). Revised selected papers. Lecture notes in computer science (LNCS) 6241. Springer, Heidelberg, pp 174\u2013196"},{"key":"18_CR26","unstructured":"Pe\u00f1as A, Hovy EH, Forner P, Rodrigo A, Sutcliffe RFE, Forascu C, Sporleder C (2011) Overview of QA4MRE at CLEF 2011: question answering for machine reading evaluation. In: Petras V, Forner P, Clough P, Ferro N (eds) CLEF 2011 working notes. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. \n                    http:\/\/ceur-ws.org\/Vol-1177\/"},{"key":"18_CR27","unstructured":"Pe\u00f1as A, Miyao Y, Hovy E, Forner P, Kando N (2013) Overview of QA4MRE 2013 entrance exams task. In: Forner P, Navigli R, Tufis D, Ferro N (eds) CLEF 2013 working notes. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. \n                    http:\/\/ceur-ws.org\/Vol-1179\/"},{"key":"18_CR28","doi-asserted-by":"crossref","unstructured":"Santos D, Cabral LM (2010) GikiCLEF: expectations and lessons learned. In: Peters C, Di Nunzio GM, Kurimo M, Mandl T, Mostefa D, Pe\u00f1as A, Roda G (eds) Multilingual information access evaluation vol. I text retrieval experiments \u2013 tenth workshop of the cross\u2013language evaluation forum (CLEF 2009). Revised selected papers. Lecture notes in computer science (LNCS) 6241, Springer, Heidelberg, pp 212\u2013222","DOI":"10.1007\/978-3-642-15754-7_23"},{"key":"18_CR29","doi-asserted-by":"crossref","unstructured":"Tsatsaronis G, Balikas G, Malakasiotis P, Partalas I, Zschunke M, Alvers MR, Weissenborn D, Krithara A, Petridis S, Polychronopoulos D, Almirantis Y, Pavlopoulos J, Baskiotis N, Gallinari P, Arti\u00e8res T, Ngonga A, Heino N, Gaussier \u00c9, Barrio-Alvers L, Schroeder M, Androutsopoulos I, Paliouras G (2015) An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinf 16:138:1\u2013138:28","DOI":"10.1186\/s12859-015-0564-6"},{"key":"18_CR30","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1007\/11878773_36","volume-title":"Accessing Multilingual Information Repositories","author":"Alessandro Vallin","year":"2006","unstructured":"Vallin A, Magnini B, Giampiccolo D, Aunimo L, Ayache C, Osenova P, Pe\u00f1as A, de Rijke M, Sacaleanu B, Santos D, Sutcliffe RFE (2006) Overview of the CLEF 2005 multilingual question answering track. In: Peters C, Gey FC, Gonzalo J, Jones GJF, Kluck M, Magnini B, M\u00fcller H, de Rijke M (eds) Accessing multilingual information repositories: sixth workshop of the cross\u2013language evaluation forum (CLEF 2005). Revised selected papers. Lecture notes in computer science (LNCS) 4022. Springer, Heidelberg, pp 307\u2013331"},{"key":"18_CR31","first-page":"2000","volume":"13\u201316","author":"Voorhees EM (2000) Overview of the TREC-9 question answering track. In: Proceedings of the ninth text retrieval conference, TREC","year":"2000","unstructured":"Voorhees EM (2000) Overview of the TREC-9 question answering track. In: Proceedings of the ninth text retrieval conference, TREC 2000, Gaithersburg, Maryland, USA, November 13\u201316, 2000","journal-title":"Maryland, USA, November"},{"key":"18_CR32","unstructured":"Voorhees EM (2002) Overview of TREC 2002 question answering track. In: Voorhees EM, Buckland LP (eds) Proceedings of the eleventh text retrieval conference (TREC 2002). NIST Publication 500-251, pp 57\u201368"},{"key":"18_CR33","first-page":"83","volume":"TREC-8","author":"EM Voorhees","year":"1999","unstructured":"Voorhees EM, Tice DM (1999) The TREC-8 question answering track evaluation. In: Text retrieval conference TREC-8, pp 83\u2013105","journal-title":"Text retrieval conference"}],"container-title":["The Information Retrieval Series","Information Retrieval Evaluation in a Changing World"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-22948-1_18","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T01:09:08Z","timestamp":1565658548000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/978-3-030-22948-1_18"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"ISBN":["9783030229474","9783030229481"],"references-count":33,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-22948-1_18","relation":{},"ISSN":["1387-5264"],"issn-type":[{"type":"print","value":"1387-5264"}],"subject":[],"published":{"date-parts":[[2019]]},"assertion":[{"value":"14 August 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}}]}}