{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:20:05Z","timestamp":1750220405668,"version":"3.41.0"},"reference-count":20,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,12,1]],"date-time":"2020-12-01T00:00:00Z","timestamp":1606780800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p>\n            This is a report on the tenth edition of the\n            <jats:italic>Conference and Labs of the Evaluation Forum<\/jats:italic>\n            (CLEF 2020), (virtually) held from September 22--25, 2020, in Thessaloniki, Greece.\n          <\/jats:p>\n          <jats:p>CLEF was a four day event combining a Conference and an Evaluation Forum. The Conference featured keynotes by Ellen Voorhees and Yiannis Kompasiaris, and presentation of peer reviewed research papers covering a wide range of topics in addition to many posters. The Evaluation Forum consisted to twelve Labs: ARQMath, BioASQ, CheckThat!, ChEMU, CLEF eHealth, eRisk, HIPE, ImageCLEF, LifeCLEF, LiLAS, PAN, and Touch\u00e9, addressing a wide range of tasks, media, languages, and ways to go beyond standard test collections.<\/jats:p>","DOI":"10.1145\/3483382.3483396","type":"journal-article","created":{"date-parts":[[2021,8,20]],"date-time":"2021-08-20T18:04:52Z","timestamp":1629482692000},"page":"1-10","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Report on CLEF 2020"],"prefix":"10.1145","volume":"54","author":[{"given":"Avi","family":"Arampatzis","sequence":"first","affiliation":[{"name":"Democritus University of Thrace, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Linda","family":"Cappellato","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carsten","family":"Eickhoff","sequence":"additional","affiliation":[{"name":"Brown University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hideo","family":"Joho","sequence":"additional","affiliation":[{"name":"University of Tsukuba, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Evangelos","family":"Kanoulas","sequence":"additional","affiliation":[{"name":"University of Amsterdam, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christina","family":"Lioma","sequence":"additional","affiliation":[{"name":"University of Copenhagen, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aur\u00e9lie","family":"N\u00e9v\u00e9ol","sequence":"additional","affiliation":[{"name":"LIMSI, CNRS, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Theodora","family":"Tsikrika","sequence":"additional","affiliation":[{"name":"CERTH, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefanos","family":"Vrochidis","sequence":"additional","affiliation":[{"name":"CERTH, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,8,20]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020)","author":"Arampatzis A.","year":"2020","unstructured":"A. Arampatzis , E. Kanoulas , T. Tsikrika , S. Vrochidis , H. Joho , C. Lioma , K. Eickhoff , A. N\u00e9v\u00e9ol , L. Cappellato , and N. Ferro , editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction . Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020) , 2020 . Lecture Notes in Computer Science (LNCS) 12260, Springer, Heidelberg, Germany. A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis, H. Joho, C. Lioma, K. Eickhoff, A. N\u00e9v\u00e9ol, L. Cappellato, and N. Ferro, editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020), 2020. Lecture Notes in Computer Science (LNCS) 12260, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"A. Barr\u00f3n-Cede\u00f1o T. Elsayed P. Nakov G. Da San Martino M. Hasanain R. Suwaileh F. Haouari N. Babulkov B. Hamdan A. Nikolov S. Shaar and Z. Sheikh Ali. Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media. In Arampatzis et al. [2020] pages 215--236.  A. Barr\u00f3n-Cede\u00f1o T. Elsayed P. Nakov G. Da San Martino M. Hasanain R. Suwaileh F. Haouari N. Babulkov B. Hamdan A. Nikolov S. Shaar and Z. Sheikh Ali. Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media. In Arampatzis et al. [2020] pages 215--236.","DOI":"10.1007\/978-3-030-58219-7_17"},{"key":"e_1_2_1_3_1","first-page":"372","volume-title":"Overview of PAN 2020: Authorship Verification, Celebrity Profiling, Profiling Fake News Spreaders on Twitter, and Style Change Detection. In Arampatzis et al. [2020]","author":"Bevendorff J.","unstructured":"J. Bevendorff , B. Ghanem , A. Giachanou , M. Kestemont , E. Manjavacas , I. Markov , M. Mayerl , M. Potthast , F. M. Rangel Pardo , P. Rosso , G. Specht , E. Stamatatos , B. Stein , M. Wiegmann , and E. Zangerle . Overview of PAN 2020: Authorship Verification, Celebrity Profiling, Profiling Fake News Spreaders on Twitter, and Style Change Detection. In Arampatzis et al. [2020] , pages 372 -- 383 . J. Bevendorff, B. Ghanem, A. Giachanou, M. Kestemont, E. Manjavacas, I. Markov, M. Mayerl, M. Potthast, F. M. Rangel Pardo, P. Rosso, G. Specht, E. Stamatatos, B. Stein, M. Wiegmann, and E. Zangerle. Overview of PAN 2020: Authorship Verification, Celebrity Profiling, Profiling Fake News Spreaders on Twitter, and Style Change Detection. In Arampatzis et al. [2020], pages 372--383."},{"key":"e_1_2_1_4_1","first-page":"384","volume-title":"Arampatzis et al. [2020]","author":"Bondarenko A.","unstructured":"A. Bondarenko , M. Fr\u00f6be , M. Beloucif , L. Gienapp , Y. Ajjour , A. Panchenko , C. Biemann , B. Stein , H. Wachsmuth , M. Potthast , and M. Hagen . Overview of Toucho\u00e9 2020: Argument Retrieval - Extended Abstract . In Arampatzis et al. [2020] , pages 384 -- 395 . A. Bondarenko, M. Fr\u00f6be, M. Beloucif, L. Gienapp, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, and M. Hagen. Overview of Toucho\u00e9 2020: Argument Retrieval - Extended Abstract. In Arampatzis et al. [2020], pages 384--395."},{"volume-title":"CLEF 2020 Working Notes, 2020. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073","author":"Cappellato L.","key":"e_1_2_1_5_1","unstructured":"L. Cappellato , C. Eickhoff , N. Ferro , and A. N\u00e9v\u00e9ol , editors . CLEF 2020 Working Notes, 2020. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073 , http:\/\/ceur-ws.org\/Vol-2696\/. L. Cappellato, C. Eickhoff, N. Ferro, and A. N\u00e9v\u00e9ol, editors. CLEF 2020 Working Notes, 2020. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http:\/\/ceur-ws.org\/Vol-2696\/."},{"key":"e_1_2_1_6_1","first-page":"288","volume-title":"Overview of CLEF HIPE 2020: Named Entity Recognition and Linking on Historical Newspapers. In Arampatzis et al. [2020]","author":"Ehrmann M.","unstructured":"M. Ehrmann , M. Romanello , A. Fl\u00fcckiger , and S. Clematide . Overview of CLEF HIPE 2020: Named Entity Recognition and Linking on Historical Newspapers. In Arampatzis et al. [2020] , pages 288 -- 310 . M. Ehrmann, M. Romanello, A. Fl\u00fcckiger, and S. Clematide. Overview of CLEF HIPE 2020: Named Entity Recognition and Linking on Historical Newspapers. In Arampatzis et al. [2020], pages 288--310."},{"key":"e_1_2_1_7_1","first-page":"3","volume-title":"What Happened in CLEF... For a While? In F","author":"Ferro N.","year":"2019","unstructured":"N. Ferro . What Happened in CLEF... For a While? In F . Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro, editors, Experimental IR Meets Multilinguality, Multimodality , and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019 ), pages 3 -- 45 . Lecture Notes in Computer Science (LNCS) 11696, Springer , Heidelberg, Germany, 2019. N. Ferro. What Happened in CLEF... For a While? In F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019), pages 3--45. Lecture Notes in Computer Science (LNCS) 11696, Springer, Heidelberg, Germany, 2019."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2016.08.001"},{"key":"e_1_2_1_10_1","first-page":"255","volume-title":"Overview of the CLEF eHealth Evaluation Lab","author":"Goeuriot L.","year":"2020","unstructured":"L. Goeuriot , H. Suominen , L. Kelly , A. Miranda-Escalada , M. Krallinger , Z. Liu , G. Pasi , G. Gonzalez Saez , M. Viviani , and C. Xu . Overview of the CLEF eHealth Evaluation Lab 2020 . In Arampatzis et al. [2020], pages 255 -- 271 . L. Goeuriot, H. Suominen, L. Kelly, A. Miranda-Escalada, M. Krallinger, Z. Liu, G. Pasi, G. Gonzalez Saez, M. Viviani, and C. Xu. Overview of the CLEF eHealth Evaluation Lab 2020. In Arampatzis et al. [2020], pages 255--271."},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"J. He D. Q. Nguyen S. A. Akhondi C. Druckenbrodt C. Thorne R. Hoessel Z. Afzal Z. Zhai B. Fang H. Yoshikawa A. Albahem L. Cavedon T. Cohn T. Baldwin and K. Verspoor. Overview of ChEMU 2020: Named Entity Recognition and Event Extraction of Chemical Reactions from Patents. In Arampatzis et al. [2020] pages 237--254.  J. He D. Q. Nguyen S. A. Akhondi C. Druckenbrodt C. Thorne R. Hoessel Z. Afzal Z. Zhai B. Fang H. Yoshikawa A. Albahem L. Cavedon T. Cohn T. Baldwin and K. Verspoor. Overview of ChEMU 2020: Named Entity Recognition and Event Extraction of Chemical Reactions from Patents. In Arampatzis et al. [2020] pages 237--254.","DOI":"10.1007\/978-3-030-58219-7_18"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"B. Ionescu H. M\u00fcller R. P\u00e9teri A. Ben Abacha V. V. Datla S. A. Hasan D. Demner-Fushman S. Kozlovski V. Liauchuk Y. Dicente Cid V. Kovalev O. Pelka C. M. Friedrich A. Garc\u00eda Seco de Herrera V.-T. Ninh T.-K. Le L. Zhou L. Piras M. Riegler P. Halvorsen M.-T. Tran M. Lux C. Gurrin D.-T. Dang-Nguyen J. Chamberlain A. Clark A. Campello D. Fichou R. Berari P. Brie M. Dogariu L.-D. Stefan and M. G. Constantin. Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical Lifelogging Nature and Internet Applications. In Arampatzis et al. [2020] pages 311--341.  B. Ionescu H. M\u00fcller R. P\u00e9teri A. Ben Abacha V. V. Datla S. A. Hasan D. Demner-Fushman S. Kozlovski V. Liauchuk Y. Dicente Cid V. Kovalev O. Pelka C. M. Friedrich A. Garc\u00eda Seco de Herrera V.-T. Ninh T.-K. Le L. Zhou L. Piras M. Riegler P. Halvorsen M.-T. Tran M. Lux C. Gurrin D.-T. Dang-Nguyen J. Chamberlain A. Clark A. Campello D. Fichou R. Berari P. Brie M. Dogariu L.-D. Stefan and M. G. Constantin. Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical Lifelogging Nature and Internet Applications. In Arampatzis et al. [2020] pages 311--341.","DOI":"10.1007\/978-3-030-45442-5_69"},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","unstructured":"A. Joly H. Go\u00ebau S. Kahl B. Deneu M. Servajean E. Cole L. Picek R. L. Ruiz De Casta\u00f1eda I. Bolon A. Durso T. Lorieul C. Botella H. Glotin J. Champ I. Eggel W.-P. Vellinga P. Bonnet and H. M\u00fcller. Overview of LifeCLEF 2020: A System-Oriented Evaluation of Automated Species Identification and Species Distribution Prediction. In Arampatzis et al. [2020] pages 342--363.  A. Joly H. Go\u00ebau S. Kahl B. Deneu M. Servajean E. Cole L. Picek R. L. Ruiz De Casta\u00f1eda I. Bolon A. Durso T. Lorieul C. Botella H. Glotin J. Champ I. Eggel W.-P. Vellinga P. Bonnet and H. M\u00fcller. Overview of LifeCLEF 2020: A System-Oriented Evaluation of Automated Species Identification and Species Distribution Prediction. In Arampatzis et al. [2020] pages 342--363.","DOI":"10.1007\/978-3-030-58219-7_23"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1_22"},{"key":"e_1_2_1_15_1","first-page":"272","volume-title":"Arampatzis et al. [2020]","author":"Losada D. E.","unstructured":"D. E. Losada , F. Crestani , and J. Parapar . Overview of eRisk 2020: Early Risk Prediction on the Internet . In Arampatzis et al. [2020] , pages 272 -- 287 . D. E. Losada, F. Crestani, and J. Parapar. Overview of eRisk 2020: Early Risk Prediction on the Internet. In Arampatzis et al. [2020], pages 272--287."},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","unstructured":"A. Nentidis A. Krithara K. Bougiatiotis M. Krallinger C. Rodr\u00edguez Penagos M. Villegas and G. Paliouras. Overview of BioASQ 2020: The Eighth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In Arampatzis et al. [2020] pages 194--214.  A. Nentidis A. Krithara K. Bougiatiotis M. Krallinger C. Rodr\u00edguez Penagos M. Villegas and G. Paliouras. Overview of BioASQ 2020: The Eighth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In Arampatzis et al. [2020] pages 194--214.","DOI":"10.1007\/978-3-030-58219-7_16"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"P. Schaer J. Schaible and L. Jael Garc\u00eda Castro. Overview of LiLAS 2020 - Living Labs for Academic Search. In Arampatzis et al. [2020] pages 364--371.  P. Schaer J. Schaible and L. Jael Garc\u00eda Castro. Overview of LiLAS 2020 - Living Labs for Academic Search. In Arampatzis et al. [2020] pages 364--371.","DOI":"10.1007\/978-3-030-58219-7_24"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/2045274.2045290"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40802-1_1"},{"key":"e_1_2_1_20_1","first-page":"169","volume-title":"Arampatzis et al. [2020]","author":"Zanibbi R.","unstructured":"R. Zanibbi , D. W. Oard , A. Agarwal , and B. Mansouri . Overview of ARQMath 2020: CLEF Lab on Answer Retrieval for Questions on Math . In Arampatzis et al. [2020] , pages 169 -- 193 . R. Zanibbi, D. W. Oard, A. Agarwal, and B. Mansouri. Overview of ARQMath 2020: CLEF Lab on Answer Retrieval for Questions on Math. In Arampatzis et al. [2020], pages 169--193."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3483382.3483396","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3483382.3483396","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:58Z","timestamp":1750191538000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3483382.3483396"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":20,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["10.1145\/3483382.3483396"],"URL":"https:\/\/doi.org\/10.1145\/3483382.3483396","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2020,12]]},"assertion":[{"value":"2021-08-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}