{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,12]],"date-time":"2026-07-12T07:53:33Z","timestamp":1783842813951,"version":"3.55.0"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,12,23]],"date-time":"2024-12-23T00:00:00Z","timestamp":1734912000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Swiss National Science Foundation","doi-asserted-by":"crossref","award":["202125"],"award-info":[{"award-number":["202125"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2025,1,31]]},"abstract":"<jats:p>Performance evaluation in multimedia retrieval, as in the information retrieval domain at large, relies heavily on retrieval experiments, employing a broad range of techniques and metrics. These can involve human-in-the-loop and machine-only settings for the retrieval process itself and the subsequent verification of results. Such experiments can be elaborate and use-case-specific, which can make them difficult to compare or replicate. In this article, we present a formal model to express all relevant aspects of such retrieval experiments, as well as a flexible open-source evaluation infrastructure that implements the model. These contributions intend to make a step towards lowering the hurdles for conducting retrieval experiments and improving their reproducibility.<\/jats:p>","DOI":"10.1145\/3678881","type":"journal-article","created":{"date-parts":[[2024,10,14]],"date-time":"2024-10-14T12:27:19Z","timestamp":1728908839000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Performance Evaluation in Multimedia Retrieval"],"prefix":"10.1145","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8046-0362","authenticated-orcid":false,"given":"Loris","family":"Sauter","sequence":"first","affiliation":[{"name":"University of Basel, Basel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3016-1396","authenticated-orcid":false,"given":"Ralph","family":"Gasser","sequence":"additional","affiliation":[{"name":"University of Basel, Basel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9865-6371","authenticated-orcid":false,"given":"Heiko","family":"Schuldt","sequence":"additional","affiliation":[{"name":"University of Basel, Basel, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0128-4602","authenticated-orcid":false,"given":"Abraham","family":"Bernstein","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5389-9465","authenticated-orcid":false,"given":"Luca","family":"Rossetto","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,12,23]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Samuel Albanie Yang Liu Arsha Nagrani Antoine Miech Ernesto Coto Ivan Laptev Rahul Sukthankar Bernard Ghanem Andrew Zisserman Valentin Gabeur et al. 2020. The end-of-end-to-end: A video understanding pentathlon challenge. arXiv:2008.00744. Retrieved from https:\/\/arxiv.org\/abs\/2008.00744"},{"key":"e_1_3_2_3_2","unstructured":"George Awad Asad A. Butt Keith Curtis Jonathan Fiscus Afzal Godil Yooyoung Lee Andrew Delgado Jesse Zhang Eliot Godard Baptiste Chocot et al. 2021. Evaluating Multiple Video Understanding and Retrieval Tasks at TRECVID 2021. In Proceedings of TRECVID 2021. NIST 58. Retrieved from http:\/\/www-nlpir.nist.gov\/projects\/tvpubs\/tv21.papers\/tv21overview.pdf"},{"key":"e_1_3_2_4_2","volume-title":"Modern Information Retrieval - the Concepts and Technology Behind Search","author":"Baeza-Yates Ricardo","year":"2011","unstructured":"Ricardo Baeza-Yates and Berthier A. Ribeiro-Neto. 2011. Modern Information Retrieval - the Concepts and Technology Behind Search (2nd ed.). Pearson Education Ltd. Retrieved from http:\/\/www.mir2ed.org\/","edition":"2"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1108\/eb026726"},{"issue":"1","key":"e_1_3_2_6_2","first-page":"133","article-title":"Anomalous states of knowledge as a basis for information retrieval","volume":"5","author":"Belkin Nicholas J.","year":"1980","unstructured":"Nicholas J. Belkin. 1980. Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science 5, 1 (1980), 133\u2013143.","journal-title":"Canadian Journal of Information Science"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-72895-5"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1108\/eb050097"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581783.3612829"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1348246.1348248"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3628797.3628940"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3591106.3592304"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05710-7_26"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3652583.3658891"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3512527.3531439"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.2200\/S00368ED1V01Y201105ICR019"},{"key":"e_1_3_2_18_2","series-title":"CEUR Workshop Proceedings","first-page":"3","volume-title":"Working Notes Proceedings of the MediaEval 2021 Workshop","volume":"3181","author":"Hassan Syed Zohaib","year":"2021","unstructured":"Syed Zohaib Hassan, Kashif Ahmad, Michael Riegler, Steven Hicks, Nicola Conci, P\u00e5l Halvorsen, and Ala I. Al-Fuqaha. 2021. Visual sentiment analysis: A natural disaster use-case task at MediaEval 2021. In Working Notes Proceedings of the MediaEval 2021 Workshop CEUR Workshop Proceedings, Vol. 3181, CEUR-WS.org, Aachen, 3. Retrieved from https:\/\/ceur-ws.org\/Vol-3181\/paper5.pdf"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/S13735-021-00225-2"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-42448-9_25"},{"key":"e_1_3_2_21_2","first-page":"14","volume-title":"Proceedings of the 3rd NTCIR Workshop on Research in Information Retrieval, Automatic Text Summarization and Question Answering (NTCIR-3)","author":"Kando Noriko","year":"2002","unstructured":"Noriko Kando. 2002. Overview of the third NTCIR workshop. In Proceedings of the 3rd NTCIR Workshop on Research in Information Retrieval, Automatic Text Summarization and Question Answering (NTCIR-3). Keizo Oyama, Emi Ishida, and Noriko Kando (Eds.). National Institute of Informatics (NII), Tokyo, Japan, 14. DOI: http:\/\/research.nii.ac.jp\/ntcir\/workshop\/OnlineProceedings3\/NTCIR3-OV-KandoN.rev.pdf"},{"key":"e_1_3_2_22_2","first-page":"11","volume-title":"Proceedings of the 1st NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition","author":"Kando Noriko","year":"1999","unstructured":"Noriko Kando, Kazuko Kuriyama, Toshihiko Nozue, Koji Eguchi, Hiroyuki Kato, and Souichiro Hidaka. 1999. Overview of IR tasks at the first NTCIR workshop. In Proceedings of the 1st NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition. National Institute of Informatics (NII), Tokyo, Japan, 11\u201344. Retrieved from http:\/\/research.nii.ac.jp\/ntcir\/workshop\/OnlineProceedings\/IR-overview.pdf"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1108\/eb026547"},{"key":"e_1_3_2_24_2","volume-title":"On the Performance of Nine Printed Subject Index Entry Types","author":"Keen E. Michael","year":"1978","unstructured":"E. Michael Keen. 1978. On the Performance of Nine Printed Subject Index Entry Types. College of Librarianship, Dep. of Information Systems Studies, Aberystwyth."},{"key":"e_1_3_2_25_2","volume-title":"Measures and Averaging Methods Used in Performance Testing of Indexing Systems","author":"Keen Micheal","year":"1966","unstructured":"Micheal Keen. 1966. Measures and Averaging Methods Used in Performance Testing of Indexing Systems. Aslib Cranfield Research Project, Cranfield."},{"key":"e_1_3_2_26_2","series-title":"CEUR Workshop Proceedings","first-page":"3","volume-title":"Proceedings of the MediaEval 2021 Workshop","volume":"3181","author":"Kille Benjamin","year":"2021","unstructured":"Benjamin Kille, Andreas Lommatzsch, \u00d6zlem \u00d6zg\u00f6bek, Mehdi Elahi, and Duc-Tien Dang-Nguyen. 2021. News images in MediaEval 2021. In Proceedings of the MediaEval 2021 Workshop CEUR Workshop Proceedings, Vol. 3181, CEUR-WS.org, Aachen, 3. Retrieved from https:\/\/ceur-ws.org\/Vol-3181\/paper2.pdf"},{"key":"e_1_3_2_27_2","series-title":"CEUR Workshop Proceedings","first-page":"3","volume-title":"Proceedings of the MediaEval 2021 Workshop","volume":"3181","author":"Kiziltepe Rukiye Savran","year":"2021","unstructured":"Rukiye Savran Kiziltepe, Mihai Gabriel Constantin, Claire-H\u00e9l\u00e8ne Demarty, Graham Healy, Camilo Fosco, Alba Garc\u00eda Seco de Herrera, Sebastian Halder, Bogdan Ionescu, Ana Matran-Fernandez, Alan F. Smeaton, et al. 2021. Overview of the MediaEval 2021 predicting media memorability task. In Proceedings of the MediaEval 2021 Workshop. CEUR Workshop Proceedings, Vol. 3181, CEUR-WS.org, Aachen, 3. Retrieved from https:\/\/ceur-ws.org\/Vol-3181\/paper10.pdf"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/S00530-023-01143-5"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-98358-1_16"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2018.2830110"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3295663"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3445031"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3429457"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/s0306-4573(00)00053-4"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/MMUL.2021.3066779"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-67835-7_33"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05710-7_29"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3552467.3554797"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1007\/S13735-024-00325-9"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/MMUL.2014.56"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/CBMI.2019.8877397"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3444685.3450215"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68474-1_23"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-5661-9_11"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/1178677.1178722"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/MMUL.2008.21"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2006.886275"},{"key":"e_1_3_2_48_2","series-title":"CEUR Workshop Proceedings","first-page":"3","volume-title":"Proceedings of the MediaEval 2021 Workshop","volume":"3181","author":"Tovstogan Philip","year":"2021","unstructured":"Philip Tovstogan, Dmitry Bogdanov, and Alastair Porter. 2021. MediaEval 2021: Emotion and theme recognition in music using Jamendo. In Proceedings of the MediaEval 2021 Workshop, Online, CEUR Workshop Proceedings, Vol. 3181, CEUR-WS.org, Aachen, 3. Retrieved from https:\/\/ceur-ws.org\/Vol-3181\/paper6.pdf"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3248284"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-27077-2_42"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-22948-1_2"},{"key":"e_1_3_2_52_2","unstructured":"Nina Willis Abraham Bernstein and Luca Rossetto. 2024. Task presentation and human perception in interactive video retrieval. arXiv:2405.04279. Retrieved from https:\/\/arxiv.org\/abs\/2405.04279"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3446209"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678881","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678881","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:09:34Z","timestamp":1750295374000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678881"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,23]]},"references-count":52,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1,31]]}},"alternative-id":["10.1145\/3678881"],"URL":"https:\/\/doi.org\/10.1145\/3678881","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,23]]},"assertion":[{"value":"2023-12-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-23","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}