{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T02:10:05Z","timestamp":1750299005985,"version":"3.41.0"},"reference-count":28,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,12,1]],"date-time":"2024-12-01T00:00:00Z","timestamp":1733011200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2024,12]]},"abstract":"<jats:p>\n            This is a report on the fifteenth edition of the\n            <jats:italic>Conference and Labs of the Evaluation Forum<\/jats:italic>\n            (CLEF 2024), held on September 9--12, 2024, in Grenoble, France. CLEF was a four-day hybrid event combining a conference and an evaluation forum. This edition also marked a special event, since we celebrated the 25th anniversary of CLEF. The conference featured keynotes by Paula Carvalho and Aur\u00e9lie N\u00e9v\u00e9ol, and presentation of peer-reviewed research papers covering a wide range of topics, in addition to many posters. The evaluation forum consisted of fourteen labs: BioASQ, CheckThat!, ELOQUENT, eRisk, EXIST, iDPP, Image-CLEF, JOKER, LifeCLEF, LongEval, PAN, qCLEF, SimpleText, and Touch\u00e9, addressing a wide range of tasks, media, languages, and ways to go beyond standard test collections.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Date<\/jats:bold>\n            : 9--12 September, 2024.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Website<\/jats:bold>\n            : https:\/\/clef2024.clef-initiative.eu\/.\n          <\/jats:p>","DOI":"10.1145\/3722449.3722465","type":"journal-article","created":{"date-parts":[[2025,3,6]],"date-time":"2025-03-06T17:24:20Z","timestamp":1741281860000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Report on the 15th Conference and Labs of the Evaluation Forum (CLEF 2024): Experimental IR Meets Multilinguality, Multimodality, and Interaction"],"prefix":"10.1145","volume":"58","author":[{"given":"Giorgio Maria","family":"Di Nunzio","sequence":"first","affiliation":[{"name":"University of Padua, Italy"}]},{"given":"Guglielmo","family":"Faggioli","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"University of Padua, Italy"}]},{"given":"Petra","family":"Galu\u0161\u010d\u00e1kov\u00e1","sequence":"additional","affiliation":[{"name":"University of Stavanger, Norway"}]},{"given":"Alba Garc\u00eda Seco","family":"de Herrera","sequence":"additional","affiliation":[{"name":"National Distance Education University (UNED), Spain"}]},{"given":"Lorraine","family":"Goeuriot","sequence":"additional","affiliation":[{"name":"Univ. Grenoble Alpes, CNRS, Grenoble, France"}]},{"given":"Philippe","family":"Mulhem","sequence":"additional","affiliation":[{"name":"Univ. Grenoble Alpes, CNRS, Grenoble, France"}]},{"given":"Georges","family":"Qu\u00e9not","sequence":"additional","affiliation":[{"name":"Univ. Grenoble Alpes, CNRS, Grenoble, France"}]},{"given":"Didier","family":"Schwab","sequence":"additional","affiliation":[{"name":"Univ. Grenoble Alpes, CNRS, Grenoble, France"}]},{"given":"Laure","family":"Soulier","sequence":"additional","affiliation":[{"name":"Sorbonne Universit\u00e9, CNRS, ISIR, Paris, France"}]}],"member":"320","published-online":{"date-parts":[[2025,3,6]]},"reference":[{"volume-title":"Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. In Goeuriot et al. [2024b].","author":"Alkhalifa R.","key":"e_1_2_1_1_1","unstructured":"R. Alkhalifa, H. Borkakoty, R. Deveaud, A. El-Ebshihy, L. Espinosa-Anke, T. Fink, P. Galu\u0161\u010d\u00e1kov\u00e1, G. N. Gonz\u00e1lez S\u00e1ez, L. Goeuriot, D. Iommi, M. Liakata, H. Tayyar Madabushi, P. Medina-Alias, P. Mulhem, F. Piroi, M. Popel, and A. Zubiaga. Overview of the CLEF 2024 LongEval Lab on Longitudinal Evaluation of Model Performance. In Goeuriot et al. [2024b]."},{"volume-title":"Overview of PAN 2024: Multi-Author Writing Style Analysis, Multilingual Text Detoxification, Oppositional Thinking Analysis, and Generative AI Authorship Verification. In Goeuriot et al. [2024b].","author":"Ayele A. A.","key":"e_1_2_1_2_1","unstructured":"A. A. Ayele, N. Babakov, J. Bevendorff, X. Bonet Casals, B. Chulvi, D. Dementieva, A. Elnagar, D. Freitag, M. Fr\u00f6be, D. Koren\u010di\u0107, M. Mayerl, D. Moskovskiy, A. Mukherjee, A. Panchenko, M. Potthast, F. Rangel Pardo, N. Rizwan, P. Rosso, F. Schneider, A. Smirnova, E. Stamatatos, E. Stakovskii, B. Stein, M. Taul\u00e9, D. Ustalov, X. Wang, N. Wiegmann, S. Muhie Yimam, and E. Zangerle. Overview of PAN 2024: Multi-Author Writing Style Analysis, Multilingual Text Detoxification, Oppositional Thinking Analysis, and Generative AI Authorship Verification. In Goeuriot et al. [2024b]."},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"A. Barr\u00f3n-Cede\u00f1o F. Alam J. M. Stru\u00df P. Nakov T. Chakraborty T. Elsayed P. Przyby\u0142a T. Caselli G. Da San Martino F. Haouari M. Hasanain C. Li J. Piskorski F. Ruggeri X. Song and R. Suwaileh. Overview of the CLEF-2024 CheckThat! Lab: Check-Worthiness Subjectivity Persuasion Roles Authorities and Adversarial Robustness. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-71908-0_2"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3636341.3636351"},{"key":"e_1_2_1_5_1","volume-title":"Dagstuhl Reports","author":"Bauer C.","year":"2023","unstructured":"C. Bauer, B. A. Carterette, N. Ferro, N. Fuhr, and G. Faggioli, editors. Report from Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education, Dagstuhl Reports, Volume 13, Number 1, 2023b. Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik, Germany."},{"key":"e_1_2_1_6_1","first-page":"118","volume-title":"Intelligent Disease Progression Prediction: Overview of iDPP@CLEF","author":"Birolo G.","year":"2024","unstructured":"G. Birolo, P. Bosoni, G. Faggioli, H. Aidos, R. Bergamaschi, P. Cavalla, A. Chi\u00f2, A. Dagliati, M. de Carvalho, G. M. Di Nunzio, P. Fariselli, J. M. Garcia Dominguez, M. Gromicho, A. Guazzo, E. Longato, S. Madeira, U. Manera, S. Marchesin, L. Menotti, G. Silvello, E. Tavazzi, E. Tavazzi, I. Trescato, M. Vettoretti, B. Di Camillo, and N. Ferro. Intelligent Disease Progression Prediction: Overview of iDPP@CLEF 2024. In Goeuriot et al. [2024b], pages 118--139."},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"L. Ermakova A.-G. Bosser T. Miller V. M. Palma Preciado G. Sidorov and A. Jatowt. Overview of JOKER @ CLEF-2024: Automatic Humour Analysis. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-56072-9_5"},{"volume-title":"Overview of the CLEF 2024 SimpleText Track: Improving Access to Scientific Texts for Everyone. In Goeuriot et al. [2024b].","author":"Ermakova L.","key":"e_1_2_1_8_1","unstructured":"L. Ermakova, E. Sanjuan, S. Huet, H. Azarbonyad, G. M. Di Nunzio, F. Vezzani, J. D'Souza, and J. Kamps. Overview of the CLEF 2024 SimpleText Track: Improving Access to Scientific Texts for Everyone. In Goeuriot et al. [2024b]."},{"volume-title":"CLEF 2024 Working Notes, 2024. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073","author":"Faggioli G.","key":"e_1_2_1_9_1","unstructured":"G. Faggioli, N. Ferro, P. Galu\u0161\u010d\u00e1kov\u00e1, and A. Garc\u00eda Seco de Herrera, editors. CLEF 2024 Working Notes, 2024. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, https:\/\/ceur-ws.org\/Vol-3740\/."},{"key":"e_1_2_1_10_1","first-page":"3","volume-title":"For a While? In F","author":"Ferro N.","year":"2019","unstructured":"N. Ferro. What Happened in CLEF... For a While? In F. Crestani, M. Braschler, J. Savoy, A. Rauber, H. M\u00fcller, D. E. Losada, G. Heinatz B\u00fcrki, L. Cappellato, and N. Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019), pages 3--45. Lecture Notes in Computer Science (LNCS) 11696, Springer, Heidelberg, Germany, 2019."},{"key":"e_1_2_1_11_1","first-page":"3","volume-title":"For Another While? In Goeuriot et al. [2024a]","author":"Ferro N.","unstructured":"N. Ferro. What Happened in CLEF... For Another While? In Goeuriot et al. [2024a], pages 3--57."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2016.08.001"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024)","author":"Goeuriot L.","year":"2024","unstructured":"L. Goeuriot, P. Mulhem, G. Q\u00faenot, D. Schwab, L. Soulier, G. M. Di Nunzio, P. Galu\u0161\u010d\u00e1kov\u00e1, A. Garc\u00eda Seco de Herrera, G. Faggioli, and N. Ferro, editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024) - Part I, 2024a. Lecture Notes in Computer Science (LNCS) 14958, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024)","author":"Goeuriot L.","year":"2024","unstructured":"L. Goeuriot, P. Mulhem, G. Q\u00faenot, D. Schwab, L. Soulier, G. M. Di Nunzio, P. Galu\u0161\u010d\u00e1kov\u00e1, A. Garc\u00eda Seco de Herrera, G. Faggioli, and N. Ferro, editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024) - Part II, 2024b. Lecture Notes in Computer Science (LNCS) 14959, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","unstructured":"B. Ionescu H. M\u00fcller A.-M. Dr\u0103gulinescu J. R\u00fcckert A. Ben Abacha A. Garc\u00eda Seco De Herrera L. Bloch R. Br\u00fcngel A. Idrissi-Yaghir H. Sch\u00e4fer C. S. Schmidt T. M. G. Pakull H. Damm B. Bracke C. M. Friedrich A.-G. Andrei Y. Prokopchuk D. Karapenka A. Radzhabov V. Kovalev C. Macaire D. Schwab B. Lecouteux E. Esperan\u00e7a-Rodier W.-W. Yim Y. Fu Z. Sun M. Yetisgen F. Xia S. A. Hicks M. A. Riegler V. Thambawita A. Stor\u00e5s P. Halvorsen M. Heinrich J. Kiesel M. Potthas and B. Stein. Overview of the ImageCLEF 2024: Multimedia retrieval in medical applications. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-71908-0_7"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"A. Joly L. Picek S. Kahl H. Go\u00ebau V. Espitalier C. Botella D. Marcos J. Estopinan C. Leblanc T. Larcher M. \u0160ulc M. Hr\u00faz M. Servajean H. Glotin R. Planqu\u00e9 W.-P. Vellinga H. Klinck T. Denton I. Eggel P. Bonnet and H. M\u00fcller. Overview of LifeCLEF 2024: Challenges on Species Distribution Prediction and Identification. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-71908-0_9"},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"J. Karlgren L. D\u00fcrlich E. Gogoulou L. Guillou J. Nivre M Sahlgren A. Talman and S. Zahra. Overview of ELOQUENT 2024 --- shared tasks for evaluating generative language model quality. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-56069-9_63"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"J. Kiesel C. \u00c7\u00f6ltekin H. Heinrich M. Fr\u00f6be M. Alshomary B. De Longueville T. Erjavec N. Handke M. Kopp N. Ljube\u0161si\u0107 K. Meden N. Mirzakhmedova V. Morkevi\u0107ius T. Reitis-M\u00fcnstermann M. Scharfbillig N. Stefanovitch H. Wachsmuth M. Potthast and B. Stein. Overview of Touch\u00e9 2024: Argumentation Systems. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-56069-9_64"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1_22"},{"key":"e_1_2_1_21_1","volume-title":"Advances in Information Retrieval. Proc. 46th European Conference on IR Research (ECIR 2024)","author":"Nazli G.","year":"2024","unstructured":"G. Nazli, N. Tonellotto, Y. He, A. Lipani, G. McDonald, C. Macdonald, and I. Ounis, editors. Advances in Information Retrieval. Proc. 46th European Conference on IR Research (ECIR 2024) - Part V, 2024a. Lecture Notes in Computer Science (LNCS) 14612, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_22_1","volume-title":"Advances in Information Retrieval. Proc. 46th European Conference on IR Research (ECIR 2024)","author":"Nazli G.","year":"2024","unstructured":"G. Nazli, N. Tonellotto, Y. He, A. Lipani, G. McDonald, C. Macdonald, and I. Ounis, editors. Advances in Information Retrieval. Proc. 46th European Conference on IR Research (ECIR 2024) - Part VI, 2024b. Lecture Notes in Computer Science (LNCS) 14613, Springer, Heidelberg, Germany."},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"A. Nentidis G. Katsimpras A. Krithara S. Lima L\u00f3pez E. Farr\u00e9-Maduell M. Krallinger N. Loukachevitch V. Davydova E. Tutubalina and G. Paliouras. Overview of BioASQ 2024: The Twelfth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-71908-0_1"},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"J. Parapar P. Mart\u00edn-Rodilla D. E. Losada and F. Crestani. Overview of eRisk 2024: Early Risk Prediction on the Internet. In Goeuriot et al. [2024b].","DOI":"10.1007\/978-3-031-71908-0_4"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"A. Pasin M. Ferrari Dacrema P. Cremonesi and N. Ferro. Overview of QuantumCLEF 2024: The Quantum Computing Challenge for Information Retrieval and Recommender Systems at CLEF. In Goeuriot et al. [2024b] pages 260--282.","DOI":"10.1007\/978-3-031-71908-0_12"},{"volume-title":"Overview of EXIST 2024 - Learning with Disagreement for Sexism Identification and Characterization in Tweets and Memes. In Goeuriot et al. [2024b].","author":"Plaza L.","key":"e_1_2_1_26_1","unstructured":"L. Plaza, J. Carrillo-de Albornoz, V. Ruiz, B. Maeso, B. Chulvi, P. Rosso, E. Amig\u00f3, J. Gonzalo, R. Morante, and D. Spina. Overview of EXIST 2024 - Learning with Disagreement for Sexism Identification and Characterization in Tweets and Memes. In Goeuriot et al. [2024b]."},{"key":"e_1_2_1_27_1","first-page":"95","volume-title":"Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011","author":"Tsikrika T.","year":"2011","unstructured":"T. Tsikrika, A. Garcia Seco de Herrera, and H. M\u00fcller. Assessing the Scholarly Impact of Image-CLEF. In P. Forner, J. Gonzalo, J. Kek\u00e4l\u00e4inen, M. Lalmas, and M. de Rijke, editors, Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011), pages 95--106. Lecture Notes in Computer Science (LNCS) 6941, Springer, Heidelberg, Germany, 2011."},{"key":"e_1_2_1_28_1","first-page":"1","volume-title":"Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013","author":"Tsikrika T.","year":"2013","unstructured":"T. Tsikrika, B. Larsen, H. M\u00fcller, S. Endrullis, and E. Rahm. The Scholarly Impact of CLEF (2000--2009). In P. Forner, H. M\u00fcller, R. Paredes, P. Rosso, and B. Stein, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013), pages 1--12. Lecture Notes in Computer Science (LNCS) 8138, Springer, Heidelberg, Germany, 2013."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3722449.3722465","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3722449.3722465","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:57:10Z","timestamp":1750298230000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3722449.3722465"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12]]},"references-count":28,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,12]]}},"alternative-id":["10.1145\/3722449.3722465"],"URL":"https:\/\/doi.org\/10.1145\/3722449.3722465","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2024,12]]},"assertion":[{"value":"2025-03-06","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}