{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T22:07:57Z","timestamp":1759961277182},"publisher-location":"Cham","reference-count":38,"publisher":"Springer International Publishing","isbn-type":[{"type":"print","value":"9783030229474"},{"type":"electronic","value":"9783030229481"}],"license":[{"start":{"date-parts":[[2019,1,1]],"date-time":"2019-01-01T00:00:00Z","timestamp":1546300800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019]]},"DOI":"10.1007\/978-3-030-22948-1_4","type":"book-chapter","created":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T05:02:59Z","timestamp":1565672579000},"page":"105-122","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["An Innovative Approach to Data Management and Curation of Experimental Data Generated Through IR Test Collections"],"prefix":"10.1007","author":[{"given":"Maristella","family":"Agosti","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giorgio Maria","family":"Di Nunzio","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gianmaria","family":"Silvello","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,8,14]]},"reference":[{"key":"4_CR1","unstructured":"Agosti M, Di Nunzio GM, Ferro N (2007a) A proposal to extend and enrich the scientific data curation of evaluation campaigns. In: Sakay T, Sanderson M, Evans DK (eds) Proceedings of the 1st international workshop on evaluating information access (EVIA 2007). National Institute of Informatics, Tokyo, pp 62\u201373"},{"key":"4_CR2","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1007\/978-3-540-74999-8_2","volume-title":"Evaluation of multilingual and multi-modal information retrieval: seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers","author":"M Agosti","year":"2007","unstructured":"Agosti M, Di Nunzio GM, Ferro N (2007b) Scientific data of an evaluation campaign: do we properly deal with them? In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval: seventh workshop of the cross\u2013language evaluation forum (CLEF 2006). Revised selected papers. Lecture notes in computer science (LNCS), vol 4730. Springer, Heidelberg, pp 11\u201320"},{"key":"4_CR3","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1007\/978-3-540-77088-6_15","volume-title":"Digital libraries: research and development. First international DELOS conference. Revised selected papers","author":"M Agosti","year":"2007","unstructured":"Agosti M, Di Nunzio GM, Ferro N (2007c) The importance of scientific data curation for evaluation campaigns. In: Thanos C, Borri F, Candela L (eds) Digital libraries: research and development. First international DELOS conference. Revised selected papers. Lecture notes in computer science (LNCS), vol 4877. Springer, Heidelberg, pp 157\u2013166"},{"key":"4_CR4","first-page":"2631","volume-title":"Proceedings of the 20th international conference on information and knowledge management (CIKM 2011)","author":"M Agosti","year":"2009","unstructured":"Agosti M, Ferro N, Thanos C (2009) DESIRE 2011: first international workshop on data infrastructures for supporting information retrieval evaluation. In: Ounis I, Ruthven I, Berendt B, de Vries AP, Wenfei F (eds) Proceedings of the 20th international conference on information and knowledge management (CIKM 2011). ACM Press, New York, pp 2631\u20132632"},{"key":"4_CR5","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"88","DOI":"10.1007\/978-3-642-33247-0_11","volume-title":"Information access evaluation. Multilinguality, multimodality, and visual analytics. Proceedings of the third international conference of the CLEF initiative (CLEF 2012)","author":"M Agosti","year":"2012","unstructured":"Agosti M, Di Buccio E, Ferro N, Masiero I, Peruzzo S, Silvello G (2012) DIRECTions: design and specification of an IR evaluation infrastructure. In: Catarci T, Forner P, Hiemstra D, Pe\u00f1as A, Santucci G (eds) Information access evaluation. Multilinguality, multimodality, and visual analytics. Proceedings of the third international conference of the CLEF initiative (CLEF 2012). Lecture notes in computer science (LNCS), vol 7488. Springer, Heidelberg, pp 88\u201399"},{"issue":"10","key":"4_CR6","first-page":"92","volume":"3","author":"M Agosti","year":"2013","unstructured":"Agosti M, Fuhr N, Toms E, Vakkari P (2013) Evaluation methodologies in information retrieval (dagstuhl seminar 13441). Dagstuhl Rep 3(10):92\u2013126","journal-title":"Dagstuhl Rep"},{"issue":"1","key":"4_CR7","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/2641383.2641390","volume":"48","author":"M Agosti","year":"2014","unstructured":"Agosti M, Fuhr N, Toms EG, Vakkari P (2014) Evaluation methodologies in information retrieval Dagstuhl seminar 13441. SIGIR Forum 48(1):36\u201341. https:\/\/doi.org\/10.1145\/2641383.2641390","journal-title":"SIGIR Forum"},{"issue":"1","key":"4_CR8","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1145\/2215676.2215678","volume":"46","author":"J Allan","year":"2012","unstructured":"Allan J, Aslam J, Azzopardi L, Belkin N, Borlund P, Bruza P, Callan J, Carman C, Clarke M, Craswell N, Croft WB, Culpepper JS, Diaz F, Dumais S, Ferro N, Geva S, Gonzalo J, Hawking D, J\u00e4rvelin K, Jones G, Jones R, Kamps J, Kando N, Kanoulos E, Karlgren J, Kelly D, Lease M, Lin J, Mizzaro S, Moffat A, Murdock V, Oard DW, de Rijke M, Sakai T, Sanderson M, Scholer F, Si L, Thom J, Thomas P, Trotman A, Turpin A, de Vries AP, Webber W, Zhang X, Zhang Y (2012) Frontiers, challenges, and opportunities for information retrieval \u2013 report from SWIRL 2012, the second strategic workshop on information retrieval in Lorne, February 2012. SIGIR Forum 46(1):2\u201332","journal-title":"SIGIR Forum"},{"key":"4_CR9","first-page":"833","volume-title":"Proceedings of the 32nd annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2009)","author":"TG Armstrong","year":"2009","unstructured":"Armstrong TG, Moffat A, Webber W, Zobel J (2009) EvaluatIR: an online tool for evaluating and comparing IR systems. In: Allan J, Aslam JA, Sanderson M, Zhai C, Zobel J (eds) Proceedings of the 32nd annual international ACM SIGIR conference on research and development in information retrieval (SIGIR 2009). ACM Press, New York, p 833"},{"key":"4_CR10","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9963.001.0001","volume-title":"Big data, little data, no data","author":"CL Borgman","year":"2015","unstructured":"Borgman CL (2015) Big data, little data, no data. MIT Press, Cambridge"},{"issue":"1","key":"4_CR11","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1007\/s13740-012-0004-y","volume":"1","author":"S Bowers","year":"2012","unstructured":"Bowers S (2012) Scientific workflow, provenance, and data modeling challenges and approaches. J Data Semant 1(1):19\u201330. https:\/\/doi.org\/10.1007\/s13740-012-0004-y","journal-title":"J Data Semant"},{"key":"4_CR12","series-title":"Lecture notes in computer science","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/3-540-44450-5_6","volume-title":"Foundations of software technology and theoretical computer science, 20th conference, FST TCS 2000 New Delhi, India, December 13\u201315, 2000, Proceedings","author":"P Buneman","year":"2000","unstructured":"Buneman P, Khanna S, Tan WC (2000) Data provenance: some basic issues. In: Kapoor S, Prasad S (eds) Foundations of software technology and theoretical computer science, 20th conference, FST TCS 2000 New Delhi, India, December 13\u201315, 2000, Proceedings. Lecture notes in computer science, vol 1974. Springer, Berlin, pp 87\u201393. https:\/\/doi.org\/10.1007\/3-540-44450-5_6"},{"key":"4_CR13","unstructured":"Candela L, Castelli D, Ferro N, Ioannidis Y, Koutrika G, Meghini C, Pagano P, Ross S, Soergel D, Agosti M, Dobreva M, Katifori V, Schuldt H (2007) The DELOS digital library reference model. Foundations for digital libraries. ISTI-CNR at Gruppo ALI, Pisa, Italy. https:\/\/tinyurl.com\/y7fxsz2d"},{"key":"4_CR14","first-page":"47","volume-title":"Readings in information retrieval","author":"CW Cleverdon","year":"1997","unstructured":"Cleverdon CW (1997) The cranfield tests on index languages devices. In: Sp\u00e4rck Jones K, Willett P (eds) Readings in information retrieval. Morgan Kaufmann Publisher, San Francisco, pp 47\u201360"},{"key":"4_CR15","first-page":"1","volume-title":"Proceedings of the 36th ACM SIGMOD-SIGACT-SIGAI symposium on principles of database systems, PODS 2017, Chicago, IL, USA, May 14\u201319, 2017","author":"SB Davidson","year":"2017","unstructured":"Davidson SB, Buneman P, Deutch D, Milo T, Silvello G (2017) Data citation: a computational challenge. In: Sallinger E, den Bussche JV, Geerts F (eds) Proceedings of the 36th ACM SIGMOD-SIGACT-SIGAI symposium on principles of database systems, PODS 2017, Chicago, IL, USA, May 14\u201319, 2017. ACM, New York, pp 1\u20134. https:\/\/doi.org\/10.1145\/3034786.3056123"},{"key":"4_CR16","unstructured":"Di Nunzio GM, Ferro N (2005) DIRECT: a distributed tool for information retrieval evaluation campaigns. In: Ioannidis Y, Schek HJ, Weikum G (eds) Proceedings of the 8th DELOS thematic workshop on future digital library management systems: system architecture and information access, pp 58\u201363"},{"issue":"2","key":"4_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3020206","volume":"8","author":"Nicola Ferro","year":"2017","unstructured":"Ferro N (2017) Reproducibility challenges in information retrieval evaluation. ACM J Data Inf Qual 8(2):8:1\u20138:4. https:\/\/doi.org\/10.1145\/3020206","journal-title":"Journal of Data and Information Quality"},{"key":"4_CR18","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1016\/j.procs.2011.04.078","volume":"4","author":"N Ferro","year":"2011","unstructured":"Ferro N, Hanbury A, M\u00fcller H, Santucci G (2011) Harnessing the scientific data produced by the experimental evaluation of search engines and information access systems. Proc Comput Sci 4:740\u2013749","journal-title":"Proc Comput Sci"},{"issue":"1","key":"4_CR19","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1145\/2492189.2492198","volume":"47","author":"P Forner","year":"2013","unstructured":"Forner P, Bentivogli L, Braschler M, Choukri K, Ferro N, Hanbury A, Karlgren J, M\u00fcller H (2013) PROMISE technology transfer day: spreading the word on information access evaluation at an industrial event. SIGIR Forum 47(1):53\u201358","journal-title":"SIGIR Forum"},{"key":"4_CR20","doi-asserted-by":"crossref","unstructured":"Gollub T, Stein B, Burrows S, Hoppe D (2012) TIRA: configuring, executing, and disseminating information retrieval experiments. In: Hameurlain A, Tjoa AM, Wagner RR (eds) 23rd international workshop on database and expert systems applications, DEXA 2012, Vienna, Austria, September 3\u20137, 2012. IEEE Computer Society, Washington, pp 151\u2013155","DOI":"10.1109\/DEXA.2012.55"},{"issue":"2","key":"4_CR21","doi-asserted-by":"crossref","first-page":"101","DOI":"10.3233\/SW-2012-0088","volume":"5","author":"AJG Gray","year":"2014","unstructured":"Gray AJG, Groth P, Loizou A, Askjaer S, Brenninkmeijer CYA, Burger K, Chichester C, Evelo CTA, Goble CA, Harland L, Pettifer S, Thompson M, Waagmeester A, Williams AJ (2014) Applying linked data approaches to pharmacology. Architectural decisions and implementation. Seman Web 5(2):101\u2013113","journal-title":"Seman Web"},{"key":"4_CR22","doi-asserted-by":"crossref","unstructured":"Harman DK (ed) (1995) The fourth Text REtrieval Conference (TREC-4), National Institute of Standards and Technology (NIST), Special Publication 500\u2013236, Washington, USA. http:\/\/trec.nist.gov\/pubs\/trec4\/t4_proceedings.html","DOI":"10.6028\/NIST.SP.500-236.overview"},{"key":"4_CR23","doi-asserted-by":"publisher","DOI":"10.2200\/S00368ED1V01Y201105ICR019","volume-title":"Information retrieval evaluation","author":"DK Harman","year":"2011","unstructured":"Harman DK (2011) Information retrieval evaluation. Morgan & Claypool Publishers, San Rafael"},{"key":"4_CR24","doi-asserted-by":"crossref","unstructured":"Heath T, Bizer C (2011) Linked data: evolving the web into a global data space. Synthesis lectures on the semantic web: theory and technology. Morgan & Claypool Publishers, San Rafael","DOI":"10.2200\/S00334ED1V01Y201102WBE001"},{"issue":"1","key":"4_CR25","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1109\/TMM.2017.2716193","volume":"20","author":"G Ioannakis","year":"2018","unstructured":"Ioannakis G, Koutsoudis A, Pratikakis I, Chamzas C (2018) RETRIEVAL \u2013 an online performance evaluation tool for information retrieval methods. IEEE Trans Multimedia 20(1):119\u2013127. https:\/\/doi.org\/10.1109\/TMM.2017.2716193","journal-title":"IEEE Trans Multimedia"},{"issue":"4","key":"4_CR26","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1177\/0165551507086989","volume":"34","author":"SE Robertson","year":"2008","unstructured":"Robertson SE (2008) On the history of evaluation in IR. J Inf Sci 34(4):439\u2013456. https:\/\/doi.org\/10.1177\/0165551507086989","journal-title":"J Inf Sci"},{"key":"4_CR27","volume-title":"Introduction to modern information retrieval","author":"G Salton","year":"1983","unstructured":"Salton G, McGill MJ (1983) Introduction to modern information retrieval. McGraw-Hill, New York"},{"issue":"4","key":"4_CR28","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1561\/1500000009","volume":"4","author":"M Sanderson","year":"2010","unstructured":"Sanderson M (2010) Test collection based evaluation of information retrieval systems. Found Trends Inf Retr 4(4):247\u2013375","journal-title":"Found Trends Inf Retr"},{"key":"4_CR29","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1002\/asi.23917","volume":"69","author":"G Silvello","year":"2017","unstructured":"Silvello G (2017) Theory and practice of data citation. J Assoc Inf Sci Technol 69:6","journal-title":"J Assoc Inf Sci Technol"},{"issue":"2","key":"4_CR30","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1007\/s00799-016-0172-8","volume":"18","author":"G Silvello","year":"2017","unstructured":"Silvello G, Bordea G, Ferro N, Buitelaar P, Bogers T (2017) Semantic representation and enrichment of information retrieval experimental data. Int J Digit Libr 18(2):145\u2013172","journal-title":"Int J Digit Libr"},{"key":"4_CR31","unstructured":"Sp\u00e4rck Jones K, Bates RG (1977) Report on a design study for the \u2018ideal\u2019 information retrieval test collection. British Library Research and Development Report 5428, University Computer Laboratory, Cambridge"},{"key":"4_CR32","unstructured":"Sp\u00e4rck Jones K, van Rijsbergen CJ (1975) Report on the need for and provision of an \u2018ideal\u2019 information retrieval test collection. British Library Research and Development Report 5266, University Computer Laboratory, Cambridge"},{"key":"4_CR33","series-title":"Lecture notes in computer science (LNCS), vol","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1007\/3-540-45691-0_34","volume-title":"Evaluation of cross-language information retrieval systems: second workshop of the cross\u2013language evaluation forum (CLEF 2001) revised papers","author":"EM Voorhees","year":"2002","unstructured":"Voorhees EM (2002) The philosophy of information retrieval evaluation. In: Peters C, Braschler M, Gonzalo J, Kluck M (eds) Evaluation of cross-language information retrieval systems: second workshop of the cross\u2013language evaluation forum (CLEF 2001) revised papers. Lecture notes in computer science (LNCS), vol 2406. Springer, Heidelberg, pp 355\u2013370"},{"issue":"11","key":"4_CR34","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1145\/1297797.1297822","volume":"50","author":"EM Voorhees","year":"2007","unstructured":"Voorhees EM (2007) TREC: continuing information retrieval\u2019s tradition of experimentation. Commun ACM 50(11):51\u201354","journal-title":"Commun ACM"},{"key":"4_CR35","volume-title":"TREC: experiment and evaluation in information retrieval","author":"EM Voorhees","year":"2005","unstructured":"Voorhees EM, Harman DK (2005) TREC: experiment and evaluation in information retrieval. The MIT Press, Cambridge"},{"key":"4_CR36","unstructured":"W3C (2004) Resource description framework (RDF): concepts and abstract syntax \u2013 W3C recommendation 10 February 2004. https:\/\/www.w3.org\/TR\/rdf-concepts\/"},{"issue":"3","key":"4_CR37","doi-asserted-by":"crossref","first-page":"257","DOI":"10.3233\/SW-2012-0081","volume":"4","author":"B Zapilko","year":"2013","unstructured":"Zapilko B, Schaible J, Mayr P, Mathiak B (2013) TheSoz: a SKOS representation of the thesaurus for the social sciences. Seman Web 4(3):257\u2013263. https:\/\/doi.org\/10.3233\/SW-2012-0081","journal-title":"Seman Web"},{"key":"4_CR38","doi-asserted-by":"crossref","unstructured":"Zobel J, Webber W, Sanderson M, Moffat A (2011) Principles for robust evaluation infrastructure. In: Proceedings of the workshop on data infrastructures for supporting information retrieval evaluation (DESIRE 2011), pp 3\u20136","DOI":"10.1145\/2064227.2064247"}],"container-title":["The Information Retrieval Series","Information Retrieval Evaluation in a Changing World"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-22948-1_4","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,21]],"date-time":"2024-07-21T23:02:56Z","timestamp":1721602976000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/978-3-030-22948-1_4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"ISBN":["9783030229474","9783030229481"],"references-count":38,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-22948-1_4","relation":{},"ISSN":["1387-5264"],"issn-type":[{"type":"print","value":"1387-5264"}],"subject":[],"published":{"date-parts":[[2019]]},"assertion":[{"value":"14 August 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}}]}}