{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:20:02Z","timestamp":1750220402159,"version":"3.41.0"},"reference-count":20,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p>\n            This is a report on the eleventh edition of the\n            <jats:italic>Conference and Labs of the Evaluation Forum<\/jats:italic>\n            (CLEF 2021), (virtually) held on September 21--24, 2021, in Bucharest, Romania. CLEF was a four day event combining a Conference and an Evaluation Forum. The Conference featured keynotes by Naila Murray and Mark Sanderson, and presentation of peer reviewed research papers covering a wide range of topics in addition to many posters. The Evaluation Forum consisted to thirteen Labs: ARQMath, BioASQ, CheckThat!, ChEMU, CLEF eHealth, eRisk, HIPE, ImageCLEF, LifeCLEF, LiLAS, PAN, SimpleText, and Touch\u00e9, addressing a wide range of tasks, media, languages, and ways to go beyond standard test collections.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Date<\/jats:bold>\n            : 21--24 September, 2021.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Website<\/jats:bold>\n            : http:\/\/clef2021.clef-initiative.eu\/.\n          <\/jats:p>","DOI":"10.1145\/3527546.3527564","type":"journal-article","created":{"date-parts":[[2022,3,17]],"date-time":"2022-03-17T23:24:36Z","timestamp":1647559476000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Report on the 12th conference and labs of the evaluation forum (CLEF 2021)"],"prefix":"10.1145","volume":"55","author":[{"given":"K. Sel\u00e7uk","family":"Candan","sequence":"first","affiliation":[{"name":"Arizona State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guglielmo","family":"Faggioli","sequence":"additional","affiliation":[{"name":"Univ. of Padua, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"Univ. of Padua, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lorraine","family":"Goeuriot","sequence":"additional","affiliation":[{"name":"Univ. Grenoble Alpes, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bogdan","family":"Ionescu","sequence":"additional","affiliation":[{"name":"Politehnica Univ. of Bucharest, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexis","family":"Joly","sequence":"additional","affiliation":[{"name":"INRIA Sophia-Antipolis, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Birger","family":"Larsen","sequence":"additional","affiliation":[{"name":"Aalborg Univ., Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria","family":"Maistro","sequence":"additional","affiliation":[{"name":"Univ. of Copenhagen, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Henning","family":"M\u00fcller","sequence":"additional","affiliation":[{"name":"Univ. of Applied Sciences, Western Switzerland, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Florina","family":"Piroi","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Wien, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,3,17]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"419","volume-title":"Zangerle. Overview of PAN 2021: Authorship Verification, Profiling Hate Speech Spreaders on Twitter, and Style Change Detection. In Candan et al. [2021]","author":"Bevendorff J.","unstructured":"J. Bevendorff , B. Chulvi , G. L. De La Pen\u00e3 Sarrac\u00e9n, M. Kestemont, E. Manjavacas, I. Markov, M. Mayerl, M. Potthast, F. Rangel, P. Rosso, E. Stamatatos, B. Stein, M. Wiegmann, M. Wolska, and E . Zangerle. Overview of PAN 2021: Authorship Verification, Profiling Hate Speech Spreaders on Twitter, and Style Change Detection. In Candan et al. [2021] , pages 419 -- 431 . J. Bevendorff, B. Chulvi, G. L. De La Pen\u00e3 Sarrac\u00e9n, M. Kestemont, E. Manjavacas, I. Markov, M. Mayerl, M. Potthast, F. Rangel, P. Rosso, E. Stamatatos, B. Stein, M. Wiegmann, M. Wolska, and E. Zangerle. Overview of PAN 2021: Authorship Verification, Profiling Hate Speech Spreaders on Twitter, and Style Change Detection. In Candan et al. [2021], pages 419--431."},{"key":"e_1_2_1_2_1","first-page":"450","volume-title":"Candan et al. [2021]","author":"Bondarenko A.","unstructured":"A. Bondarenko , L. Gienapp , M. Fr\u00f6be , M. Beloucif , Y. Ajjour , A. Panchenko , C. Biemann , B. Stein , J. Wachsmuth , M. Potthast , and M. Hagen . Overview of Touch\u00e9 2021: Argument Retrieval . In Candan et al. [2021] , pages 450 -- 467 . A. Bondarenko, L. Gienapp, M. Fr\u00f6be, M. Beloucif, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, J. Wachsmuth, M. Potthast, and M. Hagen. Overview of Touch\u00e9 2021: Argument Retrieval. In Candan et al. [2021], pages 450--467."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021)","author":"Candan K. S.","year":"2021","unstructured":"K. S. Candan , B. Ionescu , L. Goeuriot , B. Larsen , H. M\u00fcller , A. Joly , M. Maistro , F. Piroi , G. Faggioli , and N. Ferro , editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction . Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021) , 2021 . Lecture Notes in Computer Science (LNCS) 12880, Springer, Heidelberg, Germany. K. S. Candan, B. Ionescu, L. Goeuriot, B. Larsen, H. M\u00fcller, A. Joly, M. Maistro, F. Piroi, G. Faggioli, and N. Ferro, editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021), 2021. Lecture Notes in Computer Science (LNCS) 12880, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_4_1","first-page":"432","volume-title":"Overview of SimpleText 2021 - CLEF Workshop on Text Simplification for Scientific Information Access. In Candan et al. [2021]","author":"Ermakova L.","unstructured":"L. Ermakova , P. Bellot , P. Braslavski , J. Kamps , J. Mothe , D. Nurbakova , I. Ovchinnikova , and E. SanJuan . Overview of SimpleText 2021 - CLEF Workshop on Text Simplification for Scientific Information Access. In Candan et al. [2021] , pages 432 -- 449 . L. Ermakova, P. Bellot, P. Braslavski, J. Kamps, J. Mothe, D. Nurbakova, I. Ovchinnikova, and E. SanJuan. Overview of SimpleText 2021 - CLEF Workshop on Text Simplification for Scientific Information Access. In Candan et al. [2021], pages 432--449."},{"volume-title":"CLEF 2021 Working Notes, 2021. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073","author":"Faggioli G.","key":"e_1_2_1_5_1","unstructured":"G. Faggioli , N. Ferro , A. Joly , M. Maistro , and F. Piroi , editors . CLEF 2021 Working Notes, 2021. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073 , http:\/\/ceur-ws.org\/Vol-2936\/. G. Faggioli, N. Ferro, A. Joly, M. Maistro, and F. Piroi, editors. CLEF 2021 Working Notes, 2021. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, http:\/\/ceur-ws.org\/Vol-2936\/."},{"key":"e_1_2_1_6_1","first-page":"3","volume-title":"What Happened in CLEF... For a While? In F","author":"Ferro N.","year":"2019","unstructured":"N. Ferro . What Happened in CLEF... For a While? In F . Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro, editors, Experimental IR Meets Multilinguality, Multimodality , and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019 ), pages 3 -- 45 . Lecture Notes in Computer Science (LNCS) 11696, Springer , Heidelberg, Germany, 2019. N. Ferro. What Happened in CLEF... For a While? In F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019), pages 3--45. Lecture Notes in Computer Science (LNCS) 11696, Springer, Heidelberg, Germany, 2019."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2016.08.001"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"B. Ionescu H. M\u00fcller R. P\u00e9teri A. B. Abacha M. Sarrouti D. Demner-Fushman S. A. Hasan S. Kozlovski V. Liauchuk Y. D. Cid V. Kovalev O. Pelka A. Garc\u00eda Seco de Herrera J. Jacutprakart C. M. Friedrich R. Berari A. Tauteanu D. Fichou P. Brie M. Dogariu L. D. \u015etefan M. G. Constantin J. Chamberlain A. Campello A. Clark T. A. Oliver H. Moustahfid A. Popescu and J. Deshayes-Chossart. Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical Nature Internet and Social Media Applications. In Candan et al. [2021] pages 345--370.  B. Ionescu H. M\u00fcller R. P\u00e9teri A. B. Abacha M. Sarrouti D. Demner-Fushman S. A. Hasan S. Kozlovski V. Liauchuk Y. D. Cid V. Kovalev O. Pelka A. Garc\u00eda Seco de Herrera J. Jacutprakart C. M. Friedrich R. Berari A. Tauteanu D. Fichou P. Brie M. Dogariu L. D. \u015etefan M. G. Constantin J. Chamberlain A. Campello A. Clark T. A. Oliver H. Moustahfid A. Popescu and J. Deshayes-Chossart. Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical Nature Internet and Social Media Applications. In Candan et al. [2021] pages 345--370.","DOI":"10.1007\/978-3-030-85251-1_23"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"A. Joly H. Go\u00ebau S. Kahl L. Picek T. Lorieul E. Cole B. Deneu M. Servajean A. Durso I. Bolon H. Glotin R. Planqu\u00e9 R. L. Ruiz de Casta\u00f1eda W.-P. Vellinga H. Klinck T. Denton I. Eggel P. Bonnet and H. M\u00fcller. Overview of LifeCLEF 2021: An Evaluation of Machine-Learning Based Species Identification and Species Distribution Prediction. In Candan et al. [2021] pages 371--393.  A. Joly H. Go\u00ebau S. Kahl L. Picek T. Lorieul E. Cole B. Deneu M. Servajean A. Durso I. Bolon H. Glotin R. Planqu\u00e9 R. L. Ruiz de Casta\u00f1eda W.-P. Vellinga H. Klinck T. Denton I. Eggel P. Bonnet and H. M\u00fcller. Overview of LifeCLEF 2021: An Evaluation of Machine-Learning Based Species Identification and Species Distribution Prediction. In Candan et al. [2021] pages 371--393.","DOI":"10.1007\/978-3-030-85251-1_24"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1_22"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Y. Li B. Fang J. He H. Yoshikawa S. A. Akhondi C. Druckenbrodt C. Thorne Z. Afzal Z. Zhai T. Baldwin and K. Verspoor. Overview of ChEMU 2021: Reaction Reference Resolution and Anaphora Resolution in Chemical Patents. In Candan et al. [2021] pages 292--307.  Y. Li B. Fang J. He H. Yoshikawa S. A. Akhondi C. Druckenbrodt C. Thorne Z. Afzal Z. Zhai T. Baldwin and K. Verspoor. Overview of ChEMU 2021: Reaction Reference Resolution and Anaphora Resolution in Chemical Patents. In Candan et al. [2021] pages 292--307.","DOI":"10.1007\/978-3-030-85251-1_20"},{"key":"e_1_2_1_13_1","first-page":"215","volume-title":"Candan et al. [2021]","author":"Mansouri B.","unstructured":"B. Mansouri , R. Zanibbi , D. W. Oard , and A. Agarwal . Overview of ARQMath-2 (2021): Second CLEF Lab on Answer Retrieval for Questions on Math . In Candan et al. [2021] , pages 215 -- 238 . B. Mansouri, R. Zanibbi, D. W. Oard, and A. Agarwal. Overview of ARQMath-2 (2021): Second CLEF Lab on Answer Retrieval for Questions on Math. In Candan et al. [2021], pages 215--238."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"P. Nakov G. Da San Martino T. Elsayed A. Barr\u00f3n-Cede\u00f1o R. M\u00edguez S. Shaar F. Alam F. Haouari M. Hasanain W. Mansour B. Hamdan Z. S. Ali N. Babulkov A. Nikolov G. K. Shahi J. M. Stru\u00df T. Mandl M. Kutlu and Y. S. Kartal. Overview of the CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims Previously Fact-Checked Claims and Fake News. In Candan et al. [2021] pages 264--291.  P. Nakov G. Da San Martino T. Elsayed A. Barr\u00f3n-Cede\u00f1o R. M\u00edguez S. Shaar F. Alam F. Haouari M. Hasanain W. Mansour B. Hamdan Z. S. Ali N. Babulkov A. Nikolov G. K. Shahi J. M. Stru\u00df T. Mandl M. Kutlu and Y. S. Kartal. Overview of the CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims Previously Fact-Checked Claims and Fake News. In Candan et al. [2021] pages 264--291.","DOI":"10.1007\/978-3-030-85251-1_19"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"A. Nentidis G. Katsimpras E. Vandorou A. Krithara L. Gasco M. Krallinger and G. Paliouras. Overview of BioASQ 2021: The Ninth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In Candan et al. [2021] pages 239--263.  A. Nentidis G. Katsimpras E. Vandorou A. Krithara L. Gasco M. Krallinger and G. Paliouras. Overview of BioASQ 2021: The Ninth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In Candan et al. [2021] pages 239--263.","DOI":"10.1007\/978-3-030-85251-1_18"},{"key":"e_1_2_1_16_1","first-page":"324","volume-title":"Candan et al. [2021]","author":"Parapar J.","unstructured":"J. Parapar , P. Mart\u00edn-Rodilla , D. E. Losada , and F. Crestani . Overview of eRisk 2021: Early Risk Prediction on the Internet . In Candan et al. [2021] , pages 324 -- 344 . J. Parapar, P. Mart\u00edn-Rodilla, D. E. Losada, and F. Crestani. Overview of eRisk 2021: Early Risk Prediction on the Internet. In Candan et al. [2021], pages 324--344."},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"P. Schaer T. Breuer L. J. Castro B. Wolff J. Schaible and N. Tavakolpoursaleh. Overview of LiLAS 2021 - Living Labs for Academic Search. In Candan et al. [2021] pages 394--418.  P. Schaer T. Breuer L. J. Castro B. Wolff J. Schaible and N. Tavakolpoursaleh. Overview of LiLAS 2021 - Living Labs for Academic Search. In Candan et al. [2021] pages 394--418.","DOI":"10.1007\/978-3-030-85251-1_25"},{"key":"e_1_2_1_18_1","first-page":"308","volume-title":"Overview of the CLEF eHealth Evaluation Lab","author":"Suominen H.","year":"2021","unstructured":"H. Suominen , L. Goeuriot , L. Kelly , L. Alonso Alemany , E. Bassani , N. Brew-Sam , V. Cotik , D. Filippo , G. Gonz\u00e1lez-S\u00e1ez , F. Luque , P. Mulhem , G. Pasi , R. Roller , S. Seneviratne , R. Upadhyay , J. Vivaldi , M. Viviani , and C. Xu . Overview of the CLEF eHealth Evaluation Lab 2021 . In Candan et al. [2021], pages 308 -- 323 . H. Suominen, L. Goeuriot, L. Kelly, L. Alonso Alemany, E. Bassani, N. Brew-Sam, V. Cotik, D. Filippo, G. Gonz\u00e1lez-S\u00e1ez, F. Luque, P. Mulhem, G. Pasi, R. Roller, S. Seneviratne, R. Upadhyay, J. Vivaldi, M. Viviani, and C. Xu. Overview of the CLEF eHealth Evaluation Lab 2021. In Candan et al. [2021], pages 308--323."},{"key":"e_1_2_1_19_1","first-page":"95","volume-title":"Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011","author":"Tsikrika T.","year":"2011","unstructured":"T. Tsikrika , A. Garcia Seco de Herrera, and H. M\u00fcller. Assessing the Scholarly Impact of Image-CLEF. In P. Forner, J. Gonzalo, J. Kek\u00e4l\u00e4inen, M. Lalmas, and M. de Rijke, editors , Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011 ), pages 95 -- 106 . Lecture Notes in Computer Science (LNCS) 6941, Springer, Heidelberg, Germany , 2011 . T. Tsikrika, A. Garcia Seco de Herrera, and H. M\u00fcller. Assessing the Scholarly Impact of Image-CLEF. In P. Forner, J. Gonzalo, J. Kek\u00e4l\u00e4inen, M. Lalmas, and M. de Rijke, editors, Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011), pages 95--106. Lecture Notes in Computer Science (LNCS) 6941, Springer, Heidelberg, Germany, 2011."},{"key":"e_1_2_1_20_1","first-page":"1","volume-title":"Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013","author":"Tsikrika T.","year":"2013","unstructured":"T. Tsikrika , B. Larsen , H. M\u00fcller , S. Endrullis , and E. Rahm . The Scholarly Impact of CLEF (2000--2009). In P. Forner, H. M\u00fcller, R. Paredes, P. Rosso, and B. Stein, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization . Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013 ), pages 1 -- 12 . Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany , 2013 . T. Tsikrika, B. Larsen, H. M\u00fcller, S. Endrullis, and E. Rahm. The Scholarly Impact of CLEF (2000--2009). In P. Forner, H. M\u00fcller, R. Paredes, P. Rosso, and B. Stein, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013), pages 1--12. Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany, 2013."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527564","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3527546.3527564","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:53Z","timestamp":1750191533000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527564"}},"subtitle":["experimental IR meets multilinguality, multimodality, and interaction"],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":20,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1145\/3527546.3527564"],"URL":"https:\/\/doi.org\/10.1145\/3527546.3527564","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"2022-03-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}