{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T00:41:13Z","timestamp":1767141673382,"version":"build-2238731810"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2017,2,16]],"date-time":"2017-02-16T00:00:00Z","timestamp":1487203200000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources & Evaluation"],"published-print":{"date-parts":[[2017,3]]},"DOI":"10.1007\/s10579-017-9380-0","type":"journal-article","created":{"date-parts":[[2017,2,16]],"date-time":"2017-02-16T03:43:15Z","timestamp":1487216595000},"page":"1-5","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Replicability and reproducibility of research results for human language technology: introducing an LRE special section"],"prefix":"10.1007","volume":"51","author":[{"given":"Ant\u00f3nio","family":"Branco","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kevin Bretonnel","family":"Cohen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6238-5941","authenticated-orcid":false,"given":"Piek","family":"Vossen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nancy","family":"Ide","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0994-6308","authenticated-orcid":false,"given":"Nicoletta","family":"Calzolari","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2017,2,16]]},"reference":[{"issue":"6251","key":"9380_CR1","doi-asserted-by":"crossref","first-page":"aac4716","DOI":"10.1126\/science.aac4716","volume":"349","author":"JE Anderson","year":"2015","unstructured":"Anderson, J. E., Aarts, A. A., Anderson, C. J., Attridge, P. R., Attwood, A., Axt, J., et al. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.","journal-title":"Science"},{"issue":"7391","key":"9380_CR2","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1038\/483531a","volume":"483","author":"CG Begley","year":"2012","unstructured":"Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531\u2013533.","journal-title":"Nature"},{"key":"9380_CR3","unstructured":"Begley, S. (2012). In cancer science, many \u201cdiscoveries\u201d don\u2019t hold up. Reuters. http:\/\/www.reuters.com\/article\/us-science-cancer-idUSBRE82R12P20120328"},{"key":"9380_CR4","doi-asserted-by":"crossref","unstructured":"Bohannon, J. (2013). Who\u2019s afraid of peer review. Science, 342(6154), 60\u201365.","DOI":"10.1126\/science.342.6154.60"},{"key":"9380_CR5","unstructured":"Branco, A. (2012). Reliability and meta-reliability of language resources: Ready to initiate the integrity debate? In The 12th workshop on treebanks and linguistic theories (TLT12)."},{"key":"9380_CR6","unstructured":"Branco, A., Calzolari, N. & Choukri, K. (Eds.). (2016). Workshop on research results reproducibility and resources citation in science and technology of language. 10th language resources and evaluation conference (LREC2016)."},{"key":"9380_CR7","unstructured":"Buchert, T. & Nussbaum, L. (2011). Leveraging business workflows in distributed systems research for the orchestration of reproducible and scalable experiments. In 9\u00e8me \u00e9dition de la conf\u00e9rence MAnifestation des JEunes Chercheurs en Sciences et Technologies de l\u2019Information et de la Communication-MajecSTIC 2012."},{"key":"9380_CR8","unstructured":"Caporaso, JG., Deshpande, N., Fink, JL., Bourne, PE., Cohen, KB. & Hunter, L. (2008). Intrinsic evaluation of text mining tools may not predict performance on realistic tasks. In Pacific symposium on biocomputing. Pacific symposium on biocomputing. NIH Public Access, (p. 640)."},{"key":"9380_CR9","doi-asserted-by":"crossref","unstructured":"Dalle, O. (2012). On reproducibility and traceability of simulations. In Proceedings of the 2012 winter simulation conference (WSC) (pp. 1\u201312). IEEE.","DOI":"10.1109\/WSC.2012.6465284"},{"key":"9380_CR10","unstructured":"Drummond, C. (2009). Replicability is not reproducibility: nor is itgood science. In Twenty-Sixth international conference on machinelearning: Workshop on evaluation methods for machine learning IV."},{"key":"9380_CR11","unstructured":"Editorial. (2013a). Announcement: Reducing our irreproducibility. Nature News Nature"},{"key":"9380_CR12","unstructured":"Editorial. (2013b). Unreliable research: Trouble at the lab. The Economist. http:\/\/www.economist.com\/news\/briefing\/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble ."},{"issue":"5","key":"9380_CR13","doi-asserted-by":"crossref","first-page":"e5738","DOI":"10.1371\/journal.pone.0005738","volume":"4","author":"D Fanelli","year":"2009","unstructured":"Fanelli, D. (2009). How many scientists fabricate and falsify research? a systematic review and meta-analysis of survey data. PloS ONE, 4(5), e5738.","journal-title":"PloS ONE"},{"key":"9380_CR14","unstructured":"Fokkens, A., Van\u00a0Erp, M., Postma, M., Pedersen, T., Vossen, P. & Freire, N. (2013). Offspring from reproduction problems: What replication failure teaches us. In Proceedings of the 51st annual meeting of the association for computational linguistics (Vol. 1, pp. 1691\u20131701)."},{"key":"9380_CR15","doi-asserted-by":"crossref","unstructured":"Gao, Q. & Vogel, S. (2008). Parallel implementations of word alignment tool. In Software engineering, testing, and quality assurance for natural language processing, association for computational linguistics (pp. 49\u201357).","DOI":"10.3115\/1622110.1622119"},{"key":"9380_CR16","doi-asserted-by":"crossref","unstructured":"Hagen, M., Potthast, M., B\u00fcchner, M. & Stein, B. (2015). Webis: An ensemble for twitter sentiment detection. Citeseer.","DOI":"10.18653\/v1\/S15-2097"},{"key":"9380_CR17","unstructured":"Howison, J. & Herbsleb, JD. (2013). Sharing the spoils: Incentives and collaboration in scientific software development. In Proceedings of the 2013 conference on computer supported cooperative work (pp. 459\u2013470)."},{"issue":"6162","key":"9380_CR18","doi-asserted-by":"crossref","first-page":"1035","DOI":"10.1126\/science.342.6162.1035","volume":"342","author":"M Hvistendahl","year":"2013","unstructured":"Hvistendahl, M. (2013). China\u2019s publication bazaar. Science, 342(6162), 1035\u20131039.","journal-title":"Science"},{"issue":"7386","key":"9380_CR19","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1038\/nature10836","volume":"482","author":"DC Ince","year":"2012","unstructured":"Ince, D. C., Hatton, L., & Graham-Cumming, J. (2012). The case for open computer programs. Nature, 482(7386), 485\u2013488.","journal-title":"Nature"},{"issue":"6","key":"9380_CR20","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1038\/scientificamerican0611-16","volume":"304","author":"JP Ioannidis","year":"2011","unstructured":"Ioannidis, J. P. (2011). An epidemic of false claims. Scientific American, 304(6), 16\u201316.","journal-title":"Scientific American"},{"key":"9380_CR21","unstructured":"Johnson, H. L., Cohen, K. B. & Hunter, L. (2007). A fault model for ontology mapping, alignment, and linking systems. In Pacific symposium on biocomputing. pacific symposium on biocomputing. NIH Public Access (p. 233)."},{"issue":"15","key":"9380_CR22","doi-asserted-by":"crossref","first-page":"1997","DOI":"10.1093\/bioinformatics\/btp289","volume":"25","author":"Y Kano","year":"2009","unstructured":"Kano, Y., Baumgartner, W. A., McCrohon, L., Ananiadou, S., Cohen, K. B., Hunter, L., et al. (2009). U-compare: Share and compare text mining tools with UIMA. Bioinformatics, 25(15), 1997\u20131998.","journal-title":"Bioinformatics"},{"key":"9380_CR29","unstructured":"Liberman, M. (2015). Replicability vs. reproducibility or is it the other way around? http:\/\/languagelog.ldc.upenn.edu\/nll\/?p=21956 , Accessed 31 Oct 2015."},{"issue":"5","key":"9380_CR23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2347696.2347706","volume":"37","author":"P Louridas","year":"2012","unstructured":"Louridas, P., & Gousios, G. (2012). A note on rigour and replicability. ACM SIGSOFT Software Engineering Notes, 37(5), 1\u20134.","journal-title":"ACM SIGSOFT Software Engineering Notes"},{"key":"9380_CR24","doi-asserted-by":"crossref","unstructured":"Mende, T. (2010). Replication of defect prediction studies: Problems, pitfalls and recommendations. In Proceedings of the 6th international conference on predictive models in software engineering, ACM.","DOI":"10.1145\/1868328.1868336"},{"issue":"130","key":"9380_CR25","first-page":"A1","volume":"258","author":"G Naik","year":"2011","unstructured":"Naik, G. (2011). Scientists\u2019 elusive goal: Reproducing study results. Wall Street Journal, 258(130), A1.","journal-title":"Wall Street Journal"},{"key":"9380_CR26","doi-asserted-by":"crossref","unstructured":"Poprat, M., Beisswanger, E. & Hahn, U. (2008). Building a biowordnet by using wordnet\u2019s data formats and wordnet\u2019s software infrastructure: a failure story. In Software engineering, testing, and quality assurance for natural language processing, association for computational linguistics (pp. 31\u201339).","DOI":"10.3115\/1622110.1622117"},{"issue":"9","key":"9380_CR27","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1038\/nrd3439-c1","volume":"10","author":"F Prinz","year":"2011","unstructured":"Prinz, F., Schlange, T., & Asadullah, K. (2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10(9), 712\u2013712.","journal-title":"Nature Reviews Drug Discovery"},{"key":"9380_CR30","doi-asserted-by":"crossref","unstructured":"Stodden, V., Leisch, F., Peng, R. D. (Eds.) (2014). Implementing reproducible research. CRC Press, http:\/\/www.crcpress.com\/product\/isbn\/9781466561595 .","DOI":"10.1201\/b16868"},{"key":"9380_CR28","unstructured":"Zimmer, C. (2012). A sharp rise in retractions prompts calls for reform. The New York Times 16. http:\/\/www.nytimes.com\/2012\/04\/17\/science\/rise-in-scientific-journal-retractions-prompts-calls-for-reform.html ."}],"updated-by":[{"DOI":"10.1007\/s10579-017-9386-7","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2017,3,17]],"date-time":"2017-03-17T00:00:00Z","timestamp":1489708800000}}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-017-9380-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10579-017-9380-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-017-9380-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,9,18]],"date-time":"2019-09-18T15:10:06Z","timestamp":1568819406000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10579-017-9380-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,2,16]]},"references-count":30,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2017,3]]}},"alternative-id":["9380"],"URL":"https:\/\/doi.org\/10.1007\/s10579-017-9380-0","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"value":"1574-020X","type":"print"},{"value":"1574-0218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,2,16]]}}}