{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T23:42:59Z","timestamp":1774309379859,"version":"3.50.1"},"publisher-location":"Cham","reference-count":17,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783030229474","type":"print"},{"value":"9783030229481","type":"electronic"}],"license":[{"start":{"date-parts":[[2019,1,1]],"date-time":"2019-01-01T00:00:00Z","timestamp":1546300800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019]]},"DOI":"10.1007\/978-3-030-22948-1_5","type":"book-chapter","created":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T05:02:59Z","timestamp":1565672579000},"page":"123-160","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["TIRA Integrated Research Architecture"],"prefix":"10.1007","author":[{"given":"Martin","family":"Potthast","sequence":"first","affiliation":[]},{"given":"Tim","family":"Gollub","sequence":"additional","affiliation":[]},{"given":"Matti","family":"Wiegmann","sequence":"additional","affiliation":[]},{"given":"Benno","family":"Stein","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,8,14]]},"reference":[{"key":"5_CR1","doi-asserted-by":"crossref","unstructured":"Claerbout J, Karrenbach M (1992) Electronic documents give reproducible research a new meaning, pp 601\u2013604. \n                    https:\/\/doi.org\/10.1190\/1.1822162","DOI":"10.1190\/1.1822162"},{"key":"5_CR2","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1109\/MCSE.2009.15","volume":"11","author":"D Donoho","year":"2009","unstructured":"Donoho D, Maleki A, Rahman I, Shahram M, Stodden V (2009) Reproducible research in computational harmonic analysis. Comput Sci Eng 11:8\u201318. \n                    https:\/\/doi.org\/10.1109\/MCSE.2009.15","journal-title":"Comput Sci Eng"},{"key":"5_CR3","unstructured":"Fokkens A, van Erp M, Postma M, Pedersen T, Vossen P, Freire N (2013) Offspring from reproduction problems: What replication failure teaches us. In: Proceedings of the 51st annual meeting of the association for computational linguistics (Long papers), vol 1. Association for Computational Linguistics, Sofia, pp 1691\u20131701. \n                    http:\/\/www.aclweb.org\/anthology\/P13-1166"},{"issue":"1","key":"5_CR4","doi-asserted-by":"publisher","first-page":"108","DOI":"10.4230\/DagRep.6.1.108","volume":"6","author":"J Freire","year":"2016","unstructured":"Freire J, Fuhr N, Rauber A (2016) Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041). Dagstuhl Rep. 6(1):108\u2013159. \n                    https:\/\/doi.org\/10.4230\/DagRep.6.1.108","journal-title":"Dagstuhl Rep."},{"key":"5_CR5","first-page":"1125","volume-title":"35th international ACM conference on research and development in information retrieval (SIGIR 2012)","author":"T Gollub","year":"2012","unstructured":"Gollub T, Stein B, Burrows S (2012a) Ousting Ivory tower research: towards a web framework for providing experiments as a service. In: Hersh B, Callan J, Maarek Y, Sanderson M (eds) 35th international ACM conference on research and development in information retrieval (SIGIR 2012). ACM, New York, pp 1125\u20131126. \n                    https:\/\/doi.org\/10.1145\/2348283.2348501"},{"key":"5_CR6","doi-asserted-by":"publisher","unstructured":"Gollub T, Stein B, Burrows S, Hoppe D (2012b) TIRA: configuring, executing, and disseminating information retrieval experiments. In: Tjoa A, Liddle S, Schewe KD, Zhou X (eds) 9th international workshop on text-based information retrieval (TIR 2012) at DEXA. IEEE, Los Alamitos, pp 151\u2013155. \n                    https:\/\/doi.org\/10.1109\/DEXA.2012.55","DOI":"10.1109\/DEXA.2012.55"},{"key":"5_CR7","first-page":"282","volume-title":"Lecture Notes in Computer Science","author":"Tim Gollub","year":"2013","unstructured":"Gollub T, Potthast M, Beyer A, Busse M, Rangel Pardo F, Rosso P, Stamatatos E, Stenno B (2013) Recent trends in digital text forensics and its evaluation\u2014plagiarism detection, author identification, and author profiling. In: Forner P, M\u00fcller H, Paredes R, Rosso P, Stein B (eds) Information access evaluation meets multilinguality, multimodality, and visualization. Proceedings of the fourth international conference of the clef initiative (CLEF 2013). Lecture notes in computer science (LNCS), vol 8138, Springer, Heidelberg, pp 282\u2013302"},{"key":"5_CR8","unstructured":"Hanbury A, M\u00fcller H, Balog K, Brodt T, Cormack G, Eggel I, Gollub T, Hopfgartner F, Kalpathy-Cramer J, Kando N, Krithara A, Lin J, Mercer S, Potthast M (2015) Evaluation-as-a-service: overview and outlook. \n                    http:\/\/arxiv.org\/abs\/1512.07454"},{"issue":"3","key":"5_CR9","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1162\/coli.2008.34.3.465","volume":"34","author":"T Pedersen","year":"2008","unstructured":"Pedersen T (2008) Empiricism is not a matter of faith. Comput Linguist 34(3):465\u2013470 \n                    https:\/\/doi.org\/10.1162\/coli.2008.34.3.465","journal-title":"Comput Linguist"},{"key":"5_CR10","doi-asserted-by":"publisher","unstructured":"Plesser HE (2018) Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in Neuroinformatics 11:76. \n                    https:\/\/doi.org\/10.3389\/fninf.2017.00076","DOI":"10.3389\/fninf.2017.00076"},{"key":"5_CR11","first-page":"1004","volume-title":"35th international ACM conference on research and development in information retrieval (SIGIR 2012)","author":"M Potthast","year":"2012","unstructured":"Potthast M, Hagen M, Stein B, Gra\u00dfegger J, Michel M, Tippmann M, Welsch C (2012) ChatNoir: A search engine for the ClueWeb09 corpus. In: Hersh B, Callan J, Maarek Y, Sanderson M (eds) 35th international ACM conference on research and development in information retrieval (SIGIR 2012). ACM, New York, p 1004. \n                    https:\/\/doi.org\/10.1145\/2348283.2348429"},{"key":"5_CR12","unstructured":"Potthast M, Gollub T, Hagen M, Tippmann M, Kiesel J, Rosso P, Stamatatos E, Stein B (2013) Overview of the 5th international competition on plagiarism detection. In: Forner P, Navigli R, Tufis D (eds) Working notes papers of the CLEF 2013 evaluation labs. \n                    http:\/\/www.clef-initiative.eu\/publication\/working-notes"},{"key":"5_CR13","first-page":"268","volume-title":"Lecture Notes in Computer Science","author":"Martin Potthast","year":"2014","unstructured":"Potthast M, Gollub T, Rangel Pardo F, Rosso P, Stamatatos E, Stein B (2014) Improving the reproducibility of PAN\u2019s shared tasks: plagiarism detection, author identification, and author profiling. In: Kanoulas E, Lupu M, Clough P, Sanderson M, Hall M, Hanbury A, Toms E (eds) Information access evaluation\u2014multilinguality, multimodality, and interaction. Proceedings of the fifth international conference of the CLEF initiative (CLEF 2014). Lecture notes in computer science (LNCS), vol 8685, Springer, Heidelberg, pp 268\u2013299"},{"key":"5_CR14","unstructured":"Potthast M, G\u00f6ring S, Rosso P, Stein B (2015) Towards data submissions for shared tasks: first experiences for the task of text alignment. In: Working notes papers of the CLEF 2015 evaluation labs, CLEF and CEUR-WS.org, CEUR workshop proceedings. \n                    http:\/\/www.clef-initiative.eu\/publication\/working-notes"},{"key":"5_CR15","unstructured":"Potthast M, Hagen M, Stein B (2016) Author obfuscation: attacking the state of the art in authorship verification. In: Working notes papers of the CLEF 2016 evaluation labs, CLEF and CEUR-WS.org, CEUR workshop proceedings, vol 1609. \n                    http:\/\/ceur-ws.org\/Vol-1609\/"},{"key":"5_CR16","doi-asserted-by":"publisher","unstructured":"Stodden V (2010) The scientific method in practice: reproducibility in the computational sciences. Tech. Rep. MIT Sloan Research Paper No. 4773-10. \n                    https:\/\/doi.org\/10.2139\/ssrn.1550193","DOI":"10.2139\/ssrn.1550193"},{"key":"5_CR17","doi-asserted-by":"crossref","unstructured":"Zeman D, Popel M, Straka M, Hajic J, Nivre J, Ginter F, Luotolahti J, Pyysalo S, Petrov S, Potthast M, Tyers F, Badmaeva E, Gokirmak M, Nedoluzhko A, Cinkova S, Hajic jr J, Hlavacova J, Kettnerov\u00e1 V, Uresova Z, Kanerva J, Ojala S, Missil\u00e4 A, Manning C, Schuster S, Reddy S, Taji D, Habash N, Leung H, de Marneffe MC, Sanguinetti M, Simi M, Kanayama H, de Paiva V, Droganova K, Mart\u00ednez Alonso H, \u00c7\u00f6ltekin \u00c7, Sulubacak U, Uszkoreit H, Macketanz V, Burchardt A, Harris K, Marheinecke K, Rehm G, Kayadelen T, Attia M, Elkahky A, Yu Z, Pitler E, Lertpradit S, Mandl M, Kirchner J, Fernandez Alcalde H, Strnadov\u00e1 J, Banerjee E, Manurung R, Stella A, Shimada A, Kwak S, Mendonca G, Lando T, Nitisaroj R, Li J (2017) CoNLL 2017 shared task: multilingual parsing from raw text to universal dependencies. In: Proceedings of the CoNLL 2017 shared task: multilingual parsing from raw text to universal dependencies. Association for Computational Linguistics, pp 1\u201319. \n                    https:\/\/doi.org\/10.18653\/v1\/K17-3001","DOI":"10.18653\/v1\/K17-3001"}],"container-title":["The Information Retrieval Series","Information Retrieval Evaluation in a Changing World"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-22948-1_5","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,8,13]],"date-time":"2019-08-13T05:04:42Z","timestamp":1565672682000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/978-3-030-22948-1_5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"ISBN":["9783030229474","9783030229481"],"references-count":17,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-22948-1_5","relation":{},"ISSN":["1387-5264"],"issn-type":[{"value":"1387-5264","type":"print"}],"subject":[],"published":{"date-parts":[[2019]]},"assertion":[{"value":"14 August 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}}]}}