{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:19:15Z","timestamp":1750220355584,"version":"3.41.0"},"reference-count":17,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,12,1]],"date-time":"2019-12-01T00:00:00Z","timestamp":1575158400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:p>\n            This is a report on the tenth edition of the\n            <jats:italic>Conference and Labs of the Evaluation Forum<\/jats:italic>\n            (CLEF 2019), held from September 9--12, 2019, in Lugano, Switzerland.\n          <\/jats:p>\n          <jats:p>CLEF was a four day event combining a Conference and an Evaluation Forum. The Conference featured keynotes by Bruce Croft, Yair Neuman, and Miguel Mart\u00ednez, and presentation of peer reviewed research papers covering a wide range of topics in addition to many posters. The Evaluation Forum consisted to nine Labs: CENTRE, CheckThat, eHealth, eRisk, ImageCLEF, LifeCLEF, PAN, PIR-CLEF, and ProtestNews, addressing a wide range of tasks, media, languages, and ways to go beyond standard test collections.<\/jats:p>\n          <jats:p>CLEF 2019 marked the 20th anniversary of CLEF, which was celebrated with a dedicated session and a book on the lessons learnt in twenty years of evaluation activities and the future perspectives for CLEF.<\/jats:p>\n          <jats:p>CLEF 2019 also introduced the Industry Days to further extend the reach and impact of CLEF.<\/jats:p>","DOI":"10.1145\/3458553.3458571","type":"journal-article","created":{"date-parts":[[2021,3,23]],"date-time":"2021-03-23T22:17:49Z","timestamp":1616537869000},"page":"108-118","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Report on CLEF 2019"],"prefix":"10.1145","volume":"53","author":[{"given":"Martin","family":"Braschler","sequence":"first","affiliation":[{"name":"Zurich University of Applied Sciences (ZHAW), Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Linda","family":"Cappellato","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fabio","family":"Crestani","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana (USI), Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gundula Heinatz","family":"B\u00fcrki","sequence":"additional","affiliation":[{"name":"Swiss Alliance for Data-Intensive Services, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David E.","family":"Losada","sequence":"additional","affiliation":[{"name":"University of Santiago de Compostela, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Henning","family":"M\u00fcller","sequence":"additional","affiliation":[{"name":"University of Applied Sciences Western Switzerland (HES-SO), Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andreas","family":"Rauber","sequence":"additional","affiliation":[{"name":"Vienna University of Technology (TU Wien), Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jacques","family":"Savoy","sequence":"additional","affiliation":[{"name":"Universit\u00e9 de Neuch\u00e2tel, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,3,23]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"CLEF 2019 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073","author":"Cappellato L.","year":"2019","unstructured":"L. Cappellato , N. Ferro , D. E. Losada , and H. M\u00fcller , editors . CLEF 2019 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073 , http:\/\/ceur-ws.org\/Vol-2380\/, 2019 . L. Cappellato, N. Ferro, D. E. Losada, and H. M\u00fcller, editors. CLEF 2019 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http:\/\/ceur-ws.org\/Vol-2380\/, 2019."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019","author":"Crestani F.","year":"2019","unstructured":"F. Crestani , M. Braschler , J. Savoy , A. Rauber , H. M\u00fcller , D. E. Losada , G. Heinatz B\u00fcrki , L. Cappellato , and N. Ferro , editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction . Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019 ). Lecture Notes in Computer Science (LNCS) 11696, Springer, Heidelberg, Germany , 2019 . F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro, editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019). Lecture Notes in Computer Science (LNCS) 11696, Springer, Heidelberg, Germany, 2019."},{"key":"e_1_2_1_3_1","first-page":"402","volume-title":"Overview of PAN 2019: Bots and Gender Profiling, Celebrity Profiling, Cross-domain Authorship Attribution and Style Change Detection. In Crestani et al. [2]","author":"Daelemans W.","unstructured":"W. Daelemans , M. Kestemont , E. Manjavacas , M. Potthast , F. M. Rangel Pardo , P. Rosso , G. Specht , E. Stamatatos , B. Stein , M. Tschuggnall , M. Wiegmann , and E. Zangerle . Overview of PAN 2019: Bots and Gender Profiling, Celebrity Profiling, Cross-domain Authorship Attribution and Style Change Detection. In Crestani et al. [2] , pages 402 -- 416 . W. Daelemans, M. Kestemont, E. Manjavacas, M. Potthast, F. M. Rangel Pardo, P. Rosso, G. Specht, E. Stamatatos, B. Stein, M. Tschuggnall, M. Wiegmann, and E. Zangerle. Overview of PAN 2019: Bots and Gender Profiling, Celebrity Profiling, Cross-domain Authorship Attribution and Style Change Detection. In Crestani et al. [2], pages 402--416."},{"doi-asserted-by":"crossref","unstructured":"T. Elsayed P. Nakov A. Barr\u00f3n-Cede\u00f1o M. Hasanain R. Suwaileh G. Da San Martino and P. Atanasova. Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims. In Crestani et al. [2] pages 301--321.  T. Elsayed P. Nakov A. Barr\u00f3n-Cede\u00f1o M. Hasanain R. Suwaileh G. Da San Martino and P. Atanasova. Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims. In Crestani et al. [2] pages 301--321.","key":"e_1_2_1_4_1","DOI":"10.1007\/978-3-030-28577-7_25"},{"key":"e_1_2_1_5_1","first-page":"3","volume-title":"What Happened in CLEF \u2026 For a While? In Crestani et al. [2]","author":"Ferro N.","unstructured":"N. Ferro . What Happened in CLEF \u2026 For a While? In Crestani et al. [2] , pages 3 -- 45 . N. Ferro. What Happened in CLEF \u2026 For a While? In Crestani et al. [2], pages 3--45."},{"doi-asserted-by":"crossref","unstructured":"N. Ferro N. Fuhr M. Maistro T. Sakai and I. Soboroff. Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. In Crestani et al. [2] pages 287--300.  N. Ferro N. Fuhr M. Maistro T. Sakai and I. Soboroff. Overview of CENTRE@CLEF 2019: Sequel in the Systematic Reproducibility Realm. In Crestani et al. [2] pages 287--300.","key":"e_1_2_1_6_1","DOI":"10.1007\/978-3-030-28577-7_24"},{"doi-asserted-by":"publisher","key":"e_1_2_1_7_1","DOI":"10.1007\/978-3-030-22948-1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_8_1","DOI":"10.1016\/j.ipm.2016.08.001"},{"key":"e_1_2_1_9_1","first-page":"425","volume-title":"Overview of CLEF 2019 Lab ProtestNews: Extracting Protests from News in a Cross-context Setting. In Crestani et al. [2]","author":"H\u00fcrriyeto\u011flu A.","unstructured":"A. H\u00fcrriyeto\u011flu , E. Y\u00f6r\u00fck , D. Yuret , \u00c7. Yoltar, B. G\u00fcrel , F. Duru\u015fan , O. Mutlu , and A. Akdemir . Overview of CLEF 2019 Lab ProtestNews: Extracting Protests from News in a Cross-context Setting. In Crestani et al. [2] , pages 425 -- 432 . A. H\u00fcrriyeto\u011flu, E. Y\u00f6r\u00fck, D. Yuret, \u00c7. Yoltar, B. G\u00fcrel, F. Duru\u015fan, O. Mutlu, and A. Akdemir. Overview of CLEF 2019 Lab ProtestNews: Extracting Protests from News in a Cross-context Setting. In Crestani et al. [2], pages 425--432."},{"doi-asserted-by":"crossref","unstructured":"B. Ionescu H. M\u00fcller R. P\u00e9teri Y. Dicente Cid V. Liauchuk V. Kovalev D. Klimuk A. Tarasau A. Ben Abacha S. A. Hasan V. Datla J. Liu D. Demner-Fushman D.-T. Dang-Nguyen L. Piras M. Riegler M.-T. Tran M. Lux C. Gurrin O. Pelka C. M. Friedrich A. Garcia Seco de Herrera N. Garcia E. Kavallieratou C. R. del Blanco C. Cuevas Rodr\u00edguez N. Vasillopoulos K. Karampidis J. Chamberlain A. Clark and A. Campello. ImageCLEF 2019: Multimedia Retrieval in Medicine Lifelogging Security and Nature. In Crestani et al. [2] pages 358--386.  B. Ionescu H. M\u00fcller R. P\u00e9teri Y. Dicente Cid V. Liauchuk V. Kovalev D. Klimuk A. Tarasau A. Ben Abacha S. A. Hasan V. Datla J. Liu D. Demner-Fushman D.-T. Dang-Nguyen L. Piras M. Riegler M.-T. Tran M. Lux C. Gurrin O. Pelka C. M. Friedrich A. Garcia Seco de Herrera N. Garcia E. Kavallieratou C. R. del Blanco C. Cuevas Rodr\u00edguez N. Vasillopoulos K. Karampidis J. Chamberlain A. Clark and A. Campello. ImageCLEF 2019: Multimedia Retrieval in Medicine Lifelogging Security and Nature. In Crestani et al. [2] pages 358--386.","key":"e_1_2_1_10_1","DOI":"10.1007\/978-3-030-28577-7_28"},{"doi-asserted-by":"crossref","unstructured":"A. Joly H. Go\u00ebau C. Botella S. Kahl M. Servajean H. Glotin P. Bonnet W.-P. Vellinga R. Planqu\u00e9 F.-R. St\u00f6ter and H. M\u00fcller. Overview of LifeCLEF 2019: Identification of Amazonian Plants South & North American Birds and Niche Prediction. In Crestani et al. [2] pages 387--401.  A. Joly H. Go\u00ebau C. Botella S. Kahl M. Servajean H. Glotin P. Bonnet W.-P. Vellinga R. Planqu\u00e9 F.-R. St\u00f6ter and H. M\u00fcller. Overview of LifeCLEF 2019: Identification of Amazonian Plants South & North American Birds and Niche Prediction. In Crestani et al. [2] pages 387--401.","key":"e_1_2_1_11_1","DOI":"10.1007\/978-3-030-28577-7_29"},{"key":"e_1_2_1_12_1","first-page":"322","volume-title":"Overview of the CLEF eHealth Evaluation Lab","author":"Kelly L.","year":"2019","unstructured":"L. Kelly , H. Suominen , L. Goeuriot , M. Neves , E. Kanoulas , D. Li , L. Azzopardi , R. Spijker , G. Zuccon , H. Scells , and J. Palotti . Overview of the CLEF eHealth Evaluation Lab 2019 . In Crestani et al. [2], pages 322 -- 339 . L. Kelly, H. Suominen, L. Goeuriot, M. Neves, E. Kanoulas, D. Li, L. Azzopardi, R. Spijker, G. Zuccon, H. Scells, and J. Palotti. Overview of the CLEF eHealth Evaluation Lab 2019. In Crestani et al. [2], pages 322--339."},{"volume-title":"The Scholarly Impact of CLEF 2010-2017","author":"Larsen B.","unstructured":"B. Larsen . The Scholarly Impact of CLEF 2010-2017 . In Ferro and Peters [7], pages 547--554. B. Larsen. The Scholarly Impact of CLEF 2010-2017. In Ferro and Peters [7], pages 547--554.","key":"e_1_2_1_13_1"},{"key":"e_1_2_1_14_1","first-page":"340","volume-title":"Crestani et al. [2]","author":"Losada D. E.","unstructured":"D. E. Losada , F. Crestani , and J. Parapar . Overview of eRisk 2019: Early Risk Prediction on the Internet . In Crestani et al. [2] , pages 340 -- 357 . D. E. Losada, F. Crestani, and J. Parapar. Overview of eRisk 2019: Early Risk Prediction on the Internet. In Crestani et al. [2], pages 340--357."},{"key":"e_1_2_1_15_1","volume-title":"Overview of the CLEF 2019 Personalised Information Retrieval Lab (PIR-CLEF","author":"Pasi G.","year":"2019","unstructured":"G. Pasi , G. J. F. Jones , L. Goeuriot , L. Kelly , S. Marrara , and C. Sanvitto . Overview of the CLEF 2019 Personalised Information Retrieval Lab (PIR-CLEF 2019 ). In Crestani et al. [2], pages 417--424. G. Pasi, G. J. F. Jones, L. Goeuriot, L. Kelly, S. Marrara, and C. Sanvitto. Overview of the CLEF 2019 Personalised Information Retrieval Lab (PIR-CLEF 2019). In Crestani et al. [2], pages 417--424."},{"doi-asserted-by":"publisher","key":"e_1_2_1_16_1","DOI":"10.5555\/2045274.2045290"},{"key":"e_1_2_1_17_1","first-page":"1","volume-title":"Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013","author":"Tsikrika T.","year":"2013","unstructured":"T. Tsikrika , B. Larsen , H. M\u00fcller , S. Endrullis , and E. Rahm . The Scholarly Impact of CLEF (2000--2009). In P. Forner, H. M\u00fcller, R. Paredes, P. Rosso, and B. Stein, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization . Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013 ), pages 1 -- 12 . Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany , 2013 . T. Tsikrika, B. Larsen, H. M\u00fcller, S. Endrullis, and E. Rahm. The Scholarly Impact of CLEF (2000--2009). In P. Forner, H. M\u00fcller, R. Paredes, P. Rosso, and B. Stein, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013), pages 1--12. Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany, 2013."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3458553.3458571","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3458553.3458571","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:17Z","timestamp":1750191437000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3458553.3458571"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12]]},"references-count":17,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["10.1145\/3458553.3458571"],"URL":"https:\/\/doi.org\/10.1145\/3458553.3458571","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2019,12]]},"assertion":[{"value":"2021-03-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}