{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,26]],"date-time":"2025-05-26T09:44:31Z","timestamp":1748252671324,"version":"3.37.3"},"reference-count":104,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,2,26]],"date-time":"2023-02-26T00:00:00Z","timestamp":1677369600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,2,26]],"date-time":"2023-02-26T00:00:00Z","timestamp":1677369600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2023,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Studies on the applicability of heterogeneous semantically interoperable corpora are rare. We investigate to what extent reusability (both of systems and of annotations) is entailed by corpora whose interoperability is based on compliance to standards. In particular, we look at event detection in English texts, supported by the ISO-TimeML annotation scheme. We run two sets of experiments using a common neural network architecture and extensively evaluate our results on both in-distribution and out-of-distribution settings. In all experimental settings, systems obtain state-of-the-art results on the in-distribution data and underperform out-of-distribution ones, setting limits to the benefits of semantically interoperable corpora. By means of a detailed error analysis, we show that while being compliant to a standard guarantees semantic interoperability, this becomes only a necessary condition for reusability, with factors such as differences in the quality of the annotations having a much stronger impact.<\/jats:p>","DOI":"10.1007\/s10579-023-09643-6","type":"journal-article","created":{"date-parts":[[2023,2,26]],"date-time":"2023-02-26T15:02:22Z","timestamp":1677423742000},"page":"1107-1137","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Investigating interoperable event corpora: limitations of reusability of resources and portability of models"],"prefix":"10.1007","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2936-0256","authenticated-orcid":false,"given":"Tommaso","family":"Caselli","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Johan","family":"Bos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,2,26]]},"reference":[{"key":"9643_CR1","doi-asserted-by":"crossref","unstructured":"Ahn, D. (2006). The stages of event extraction. In: Proceedings of the workshop on annotating and reasoning about time and events, association for computational linguistics, (pp 1\u20138).","DOI":"10.3115\/1629235.1629236"},{"key":"9643_CR2","unstructured":"Alex, B., Nissim, M., & Grover, C. (2006). The impact of annotation on the performance of protein tagging in biomedical text. In: Proceedings of the Fifth international conference on language resources and evaluation (LREC\u201906), European Language Resources Association (ELRA), Genoa, Italy, http:\/\/www.lrec-conf.org\/proceedings\/lrec2006\/pdf\/398_pdf.pdf"},{"key":"9643_CR3","unstructured":"Araki, J. (2018). Extraction of event structures from text. PhD thesis"},{"key":"9643_CR4","unstructured":"Araki, J., Mulaffer, L., Pandian, A., Yamakawa, Y., Oflazer, K., & Mitamura, T. (2018). Interoperable annotation of events and event relations across domains. In: Proceedings 14th Joint ACL-ISO Workshop on Interoperable Semantic Annotation, Association for Computational Linguistics, (pp 10\u201320). http:\/\/aclweb.org\/anthology\/W18-4702"},{"issue":"1","key":"9643_CR5","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/BF00627432","volume":"9","author":"E Bach","year":"1986","unstructured":"Bach, E. (1986). The algebra of events. Linguistics and Philosophy, 9(1), 5\u201316.","journal-title":"Linguistics and Philosophy"},{"key":"9643_CR6","unstructured":"Bethard, S. (2013). Cleartk-timeml: a minimalist approach to tempeval 2013. In: Second Joint Conference on Lexical and Computational Semantics (* SEM), (vol\u00a02, pp 10\u201314)."},{"key":"9643_CR7","doi-asserted-by":"crossref","unstructured":"Bethard, S., Savova, G., Chen, WT., Derczynski, L., Pustejovsky, J., & Verhagen, M. (2016). Semeval-2016 task 12: Clinical tempeval. In: Proceedings of the 10th international workshop on semantic evaluation (SemEval 2016), association for computational linguistics, San Diego, California.","DOI":"10.18653\/v1\/S16-1165"},{"key":"9643_CR8","doi-asserted-by":"crossref","unstructured":"Bethard, S., Savova, G., Palmer, M., & Pustejovsky, J. (2017). Semeval-2017 task 12: Clinical tempeval. In: Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), association for computational linguistics, Vancouver, Canada, (pp 565\u2013572) http:\/\/www.aclweb.org\/anthology\/S17-2093","DOI":"10.18653\/v1\/S17-2093"},{"key":"9643_CR9","doi-asserted-by":"crossref","unstructured":"Blitzer, J., McDonald, R., & Pereira, F. (2006). Domain adaptation with structural correspondence learning. In: Proceedings of the 2006 conference on empirical methods in natural language processing, Association for Computational Linguistics, (pp 120\u2013128).","DOI":"10.3115\/1610075.1610094"},{"key":"9643_CR10","unstructured":"Calzolari, N., Monachini, M., & Quochi, V. (2011). Interoperability framework: The FLaReNet action plan proposal. In: Proceedings of the workshop on language resources, technology and services in the sharing paradigm, Asian Federation of Natural Language Processing, Chiang Mai, Thailand, (pp 41\u201349) https:\/\/www.aclweb.org\/anthology\/W11-3306"},{"key":"9643_CR11","unstructured":"Caselli, T., & Morante, R. (2018). Systems\u2019 agreements and disagreements in temporal processing: An extensive error analysis of the tempeval-3 task. In: chair) NCC, Choukri K, Cieri C, Declerck T, Goggi S, Hasida K, Isahara H, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S, Tokunaga T (eds) Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Paris, France"},{"key":"9643_CR12","doi-asserted-by":"crossref","unstructured":"Caselli, T., & Sprugnoli, R. (2017). It-timeml and the ita-timebank: Language specific adaptations for temporal annotation. In: Pustejovsky J, Ide N (eds) Handbook of Linguistic Annotation - Vol. II, Springer, pp 969\u2013988","DOI":"10.1007\/978-94-024-0881-2_36"},{"key":"9643_CR13","unstructured":"Caselli, T., & \u00dcst\u00fcn, A. (2019). There and back again: Cross-lingual transfer learning for event detection. In: Proceedings of the Sixth Italian Conference on Computational Linguistics, CEUR Workshop Proceedings (CEUR-WS.org), vol 2481"},{"key":"9643_CR14","doi-asserted-by":"crossref","unstructured":"Chen, Z., & Ji, H. (2009). Language specific issue and feature exploration in Chinese event extraction. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, Association for Computational Linguistics, Boulder, Colorado, pp 209\u2013212, https:\/\/www.aclweb.org\/anthology\/N09-2053","DOI":"10.3115\/1620853.1620910"},{"key":"9643_CR15","doi-asserted-by":"crossref","unstructured":"Chiarcos, C. (2012). Interoperability of corpora and annotations. In: Linked Data in Linguistics, Springer, pp 161\u2013179","DOI":"10.1007\/978-3-642-28249-2_16"},{"key":"9643_CR16","doi-asserted-by":"crossref","unstructured":"Chiarcos, C., McCrae, J., Cimiano, P., & Fellbaum, C. (2013). Towards open data for linguistics: Linguistic linked data. In: New Trends of Research in Ontologies and Lexical Resources, Springer, pp 7\u201325","DOI":"10.1007\/978-3-642-31782-8_2"},{"issue":"4","key":"9643_CR17","doi-asserted-by":"publisher","first-page":"379","DOI":"10.3233\/SW-140167","volume":"6","author":"C Chiarcos","year":"2015","unstructured":"Chiarcos, C., & Sukhareva, M. (2015). OLiA-Ontologies of linguistic annotation. Semantic Web, 6(4), 379\u2013386.","journal-title":"Semantic Web"},{"key":"9643_CR18","unstructured":"Chinchor, NA. (1998). Overview of MUC-7. In: Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29\u2013May 1, 1998, https:\/\/www.aclweb.org\/anthology\/M98-1001"},{"key":"9643_CR19","doi-asserted-by":"crossref","unstructured":"Comeau, DC., Islamaj\u00a0Do\u011fan, R., Ciccarese, P., Cohen, KB., Krallinger, M., Leitner, F., Lu, Z., Peng, Y., Rinaldi, F., Torii, M., & et\u00a0al. (2013). Bioc: a minimalist approach to interoperability for biomedical text processing. Database 2013","DOI":"10.1093\/database\/bat064"},{"key":"9643_CR20","first-page":"256","volume":"2007","author":"H Daum\u00e9 III","year":"2007","unstructured":"Daum\u00e9, H., III. (2007). Frustratingly easy domain adaptation. ACL, 2007, 256.","journal-title":"ACL"},{"key":"9643_CR21","unstructured":"de\u00a0Jong, F., Maegaard, B., Fi\u0161er, D., Van\u00a0Uytvanck, D., Witt, A., & et\u00a0al. (2020). Interoperability in an infrastructure enabling multidisciplinary research: The case of clarin. In: Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association (ELRA), pp 3406\u20133413"},{"key":"9643_CR22","unstructured":"de\u00a0Marneffe, MC., MacCartney, B., & Manning, CD. (2006). Generating typed dependency parses from phrase structure parses. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC\u201906), European Language Resources Association (ELRA), Genoa, Italy, http:\/\/www.lrec-conf.org\/proceedings\/lrec2006\/pdf\/440_pdf.pdf"},{"key":"9643_CR23","unstructured":"Declerck, T. (2006). SynAF: Towards a standard for syntactic annotation. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC\u201906), European Language Resources Association (ELRA), Genoa, Italy, http:\/\/www.lrec-conf.org\/proceedings\/lrec2006\/pdf\/652_pdf.pdf"},{"key":"9643_CR24","unstructured":"Derczynski, L., & Gaizauskas, R. (2010). Analysing temporally annotated corpora with CAVaT. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC\u201910), European Language Resources Association (ELRA), Valletta, Malta, http:\/\/www.lrec-conf.org\/proceedings\/lrec2010\/pdf\/546_Paper.pdf"},{"key":"9643_CR25","doi-asserted-by":"publisher","unstructured":"Elsahar, H., & Gall\u00e9, M. (2019). To annotate or not? predicting performance drop under domain shift. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, pp 2163\u20132173, https:\/\/doi.org\/10.18653\/v1\/D19-1222,","DOI":"10.18653\/v1\/D19-1222"},{"key":"9643_CR26","doi-asserted-by":"crossref","unstructured":"Ettinger, A., Rao, S., Daum\u00e9\u00a0III, H., & Bender, EM. (2017). Towards linguistically generalizable nlp systems: A workshop and shared task. In: Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pp 1\u201310","DOI":"10.18653\/v1\/W17-5401"},{"key":"9643_CR27","doi-asserted-by":"crossref","unstructured":"Fan, W., & Davidson, I. (2006). Reverse testing: an efficient framework to select amongst classifiers under sample selection bias. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 147\u2013156","DOI":"10.1145\/1150402.1150422"},{"key":"#cr-split#-9643_CR28.1","doi-asserted-by":"crossref","unstructured":"Fillmore, C. (2006). Frame semantics. In: Geeraerts D","DOI":"10.1016\/B0-08-044854-2\/00424-7"},{"key":"#cr-split#-9643_CR28.2","unstructured":"(ed) Cognitive Linguistics: Basic Readings, De Gruyter Mouton, Berlin, Boston, pp 373-400, originally published in 1982."},{"key":"9643_CR29","unstructured":"Haji\u010dov\u00e1, E. (2014). \"three dimensions of the so-called \"\"interoperability\"\" of annotation schemes\". In: Chair) NCC, Choukri K, Declerck T, Loftsson H, Maegaard B, Mariani J, Moreno A, Odijk J, Piperidis S (eds) Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914), European Language Resources Association (ELRA), Reykjavik, Iceland"},{"key":"9643_CR30","doi-asserted-by":"crossref","unstructured":"Hellmann, S., Lehmann, J., Auer, S., & Br\u00fcmmer, M. (2013). Integrating nlp using linked data. In: International semantic web conference, Springer, pp 98\u2013113","DOI":"10.1007\/978-3-642-41338-4_7"},{"key":"9643_CR31","doi-asserted-by":"publisher","unstructured":"Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., & Song, D. (2020). Pretrained transformers improve out-of-distribution robustness. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, pp 2744\u20132751, https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.244","DOI":"10.18653\/v1\/2020.acl-main.244"},{"key":"9643_CR32","doi-asserted-by":"crossref","unstructured":"Huang, L., Cassidy, T., Feng, X., Ji, H., Voss, CR., Han, J., & Sil, A. (2016). Liberal event extraction and event schema induction. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol\u00a01, pp 258\u2013268","DOI":"10.18653\/v1\/P16-1025"},{"key":"9643_CR33","doi-asserted-by":"publisher","unstructured":"Huang, L., Ji, H., Cho, K., Dagan, I., Riedel, S., & Voss, C. (2018). Zero-shot transfer learning for event extraction. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Australia, pp 2160\u20132170, https:\/\/doi.org\/10.18653\/v1\/P18-1201,","DOI":"10.18653\/v1\/P18-1201"},{"key":"9643_CR34","unstructured":"Ide, N., & Pustejovsky, J. (2010). What does interoperability mean, anyway? toward an operational definition of interoperability for language technology. In: Proceedings of the Second International Conference on Global Interoperability for Language Resources. Hong Kong, China"},{"key":"9643_CR35","doi-asserted-by":"crossref","unstructured":"Ide, N., & Suderman, K. (2007). Graf: A graph-based format for linguistic annotations. In: proceedings of the Linguistic Annotation Workshop, pp 1\u20138","DOI":"10.3115\/1642059.1642060"},{"key":"9643_CR36","doi-asserted-by":"publisher","unstructured":"Inel, O., & Aroyo, L. (2019). Validation Methodology for Expert-Annotated Datasets: Event Annotation Case Study. In: Eskevich M, de\u00a0Melo G, F\u00e4th C, McCrae JP, Buitelaar P, Chiarcos C, Klimek B, Dojchinovski M (eds) 2nd Conference on Language, Data and Knowledge (LDK 2019), Schloss Dagstuhl\u2013Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, OpenAccess Series in Informatics (OASIcs), vol\u00a070, pp 12:1\u201312:15, https:\/\/doi.org\/10.4230\/OASIcs.LDK.2019.12,","DOI":"10.4230\/OASIcs.LDK.2019.12"},{"key":"9643_CR37","unstructured":"Ji, H., & Grishman, R. (2008). Refining event extraction through cross-document inference. Proceedings of ACL-08: HLT pp 254\u2013262"},{"key":"9643_CR38","unstructured":"Jung, H., & Stent, A. (2013). Att1: Temporal annotation using big windows and rich syntactic and semantic features. In: Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Association for Computational Linguistics, Atlanta, Georgia, USA, pp 20\u201324, http:\/\/www.aclweb.org\/anthology\/S13-2004"},{"key":"9643_CR39","doi-asserted-by":"publisher","unstructured":"Karan, M., & \u0160najder, J. (2018). Cross-domain detection of abusive language online. In: Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), Association for Computational Linguistics, Brussels, Belgium, pp 132\u2013137, https:\/\/doi.org\/10.18653\/v1\/W18-5117,","DOI":"10.18653\/v1\/W18-5117"},{"issue":"4","key":"9643_CR40","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1504\/IJMSO.2009.029230","volume":"4","author":"M Kemps-Snijders","year":"2009","unstructured":"Kemps-Snijders, M., Windhouwer, M., Wittenburg, P., & Wright, S. E. (2009). Isocat: remodelling metadata for language resources. International Journal of Metadata, Semantics and Ontologies, 4(4), 261\u2013276.","journal-title":"International Journal of Metadata, Semantics and Ontologies"},{"key":"9643_CR41","unstructured":"Kim, YB., Stratos, K., & Sarikaya, R. (2016). Frustratingly easy neural domain adaptation. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, The COLING 2016 Organizing Committee, Osaka, Japan, pp 387\u2013396, https:\/\/www.aclweb.org\/anthology\/C16-1038"},{"key":"9643_CR42","doi-asserted-by":"crossref","unstructured":"Komninos, A., & Manandhar, S. (2016). Dependency based embeddings for sentence classification tasks. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 1490\u20131500","DOI":"10.18653\/v1\/N16-1175"},{"key":"9643_CR43","unstructured":"Li, Q., Ji, H., & Huang, L. (2013). Joint event extraction via structured prediction with global features. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 73\u201382"},{"key":"9643_CR44","doi-asserted-by":"publisher","unstructured":"Liakata, M., Saha, S., Dobnik, S., Batchelor, C., & Rebholz-Schuhmann, D. (2012). Automatic recognition of conceptualization zones in scientific articles and two life science applications. Bioinformatics 28(7):991\u20131000, https:\/\/doi.org\/10.1093\/bioinformatics\/bts071,","DOI":"10.1093\/bioinformatics\/bts071"},{"key":"9643_CR45","unstructured":"Linguistic Data Consortium. (2005). ACE (Automatic Content Extraction) English Annotation Guidelines for Events. 5th edn"},{"key":"9643_CR46","doi-asserted-by":"crossref","unstructured":"Lyons, J. (1977). Semantics: Volume 2, vol\u00a02. Cambridge University Press","DOI":"10.1017\/CBO9780511620614"},{"key":"9643_CR47","doi-asserted-by":"crossref","unstructured":"Ma, X., & Hovy, E. (2016). End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354","DOI":"10.18653\/v1\/P16-1101"},{"key":"9643_CR48","doi-asserted-by":"crossref","unstructured":"Ma, J., Zhang, Y., & Zhu, J. (2014). Tagging the web: Building a robust web tagger with neural network. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol\u00a01, pp 144\u2013154","DOI":"10.3115\/v1\/P14-1014"},{"key":"9643_CR49","unstructured":"McCrae, JP., Bosque-Gil, J., Gracia, J., Buitelaar, P., & Cimiano, P. (2017). The Ontolex-Lemon model: development and applications. In: Proceedings of eLex 2017 conference, pp 19\u201321"},{"key":"9643_CR50","unstructured":"Minard, AL., Speranza, M., Urizar, R., Altuna, B., van Erp, M., Schoen, A., & van Son, C. (2016). Meantime, the newsreader multilingual event and time corpus. In: Chair) NCC, Choukri K, Declerck T, Goggi S, Grobelnik M, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S (eds) Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), European Language Resources Association (ELRA), Paris, France"},{"key":"9643_CR51","first-page":"308","volume":"14","author":"P Mirza","year":"2014","unstructured":"Mirza, P., & Tonelli, S. (2014). Classifying temporal relations with simple features. EACL, 14, 308\u2013317.","journal-title":"EACL"},{"key":"9643_CR52","doi-asserted-by":"publisher","unstructured":"Mitamura, T., Yamakawa, Y., Holm, S., Song, Z., Bies, A., Kulick, S., & Strassel, S. (2015). Event nugget annotation: Processes and issues. In: Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, Association for Computational Linguistics, Denver, Colorado, pp 66\u201376, https:\/\/doi.org\/10.3115\/v1\/W15-0809,","DOI":"10.3115\/v1\/W15-0809"},{"key":"9643_CR53","unstructured":"Muandet, K., Balduzzi, D., & Sch\u00f6lkopf, B. (2013). Domain generalization via invariant feature representation. In: International Conference on Machine Learning, PMLR, pp 10\u201318"},{"key":"9643_CR54","doi-asserted-by":"crossref","unstructured":"Nguyen, TH., & Grishman, R. (2015). Event detection and domain adaptation with convolutional neural networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), vol\u00a02, pp 365\u2013371","DOI":"10.3115\/v1\/P15-2060"},{"key":"9643_CR55","doi-asserted-by":"publisher","unstructured":"Nguyen, TH., Cho, K., & Grishman, R. (2016), Joint event extraction via recurrent neural networks. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, San Diego, California, pp 300\u2013309, https:\/\/doi.org\/10.18653\/v1\/N16-1034,","DOI":"10.18653\/v1\/N16-1034"},{"key":"9643_CR56","doi-asserted-by":"crossref","unstructured":"Niu, ZY., Wang, H., & Wu, H. (2009). Exploiting heterogeneous treebanks for parsing. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Association for Computational Linguistics, Suntec, Singapore, pp 46\u201354, https:\/\/www.aclweb.org\/anthology\/P09-1006","DOI":"10.3115\/1687878.1687887"},{"key":"9643_CR57","unstructured":"Nivre, J., de\u00a0Marneffe, MC., Ginter, F., Goldberg, Y., Haji\u010d, J., Manning, CD., McDonald, R., Petrov, S., Pyysalo, S., Silveira, N., Tsarfaty, R., & Zeman, D. (2016). Universal Dependencies v1: A multilingual treebank collection. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916), European Language Resources Association (ELRA), Portoro\u017e, Slovenia, pp 1659\u20131666, https:\/\/www.aclweb.org\/anthology\/L16-1262"},{"key":"9643_CR58","unstructured":"Nivre, J., de\u00a0Marneffe, MC., Ginter, F., Haji\u010d, J., Manning, CD., Pyysalo, S., Schuster, S., Tyers, F., & Zeman, D. (2020). Universal Dependencies v2: An evergrowing multilingual treebank collection. In: Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, pp 4034\u20134043, https:\/\/www.aclweb.org\/anthology\/2020.lrec-1.497"},{"key":"9643_CR59","doi-asserted-by":"crossref","unstructured":"Novielli, N., Girardi, D., & Lanubile, F. (2018). A benchmark study on sentiment analysis for software engineering research. In: 2018 IEEE\/ACM 15th International Conference on Mining Software Repositories (MSR), IEEE, pp 364\u2013375","DOI":"10.1145\/3196398.3196403"},{"key":"9643_CR60","unstructured":"O\u2019Gorman, T., Pradhan, S., Palmer, M., Bonn, J., Conger, K., & Gung, J. (2018). The new Propbank: Aligning Propbank with AMR through POS unification. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), European Language Resources Association (ELRA), Miyazaki, Japan, https:\/\/www.aclweb.org\/anthology\/L18-1231"},{"key":"9643_CR61","doi-asserted-by":"crossref","unstructured":"O\u2019Gorman, T., Wright-Bettner, K., & Palmer, M. (2016). Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In: Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), Association for Computational Linguistics, Austin, Texas, pp 47\u201356, http:\/\/aclweb.org\/anthology\/W16-5706","DOI":"10.18653\/v1\/W16-5706"},{"key":"9643_CR62","unstructured":"Plank, B. (2016). What to do about non-standard (or non-canonical) language in nlp. arXiv preprint arXiv:1608.07836"},{"key":"9643_CR63","unstructured":"Plank, B., & Van\u00a0Noord, G. (2011). Effective measures of domain similarity for parsing. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, Association for Computational Linguistics, pp 1566\u20131576"},{"key":"9643_CR64","unstructured":"Poch, M., & Bel\u00a0Rafecas, N. (2011). Interoperability and technology for a language resources factory. In: Proceedings of Workshop on Language Resources, Technology and Services in the Sharing Paradigm; 2011 Nov 12; Chiang Mai, Thailand. Chiang Mai (Thailand): ACL; 2011. p. 32-40., ACL (Association for Computational Linguistics)"},{"key":"9643_CR65","doi-asserted-by":"publisher","unstructured":"Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopoulos, I., Manandhar, S., AL-Smadi, M., Al-Ayyoub, M., Zhao, Y., Qin, B., De\u00a0Clercq, O., Hoste, V., Apidianaki, M., Tannier, X., Loukachevitch, N., Kotelnikov, E., Bel, N., Jim\u00e9nez-Zafra, SM., & Eryi\u011fit, G. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), Association for Computational Linguistics, San Diego, California, pp 19\u201330, https:\/\/doi.org\/10.18653\/v1\/S16-1002,","DOI":"10.18653\/v1\/S16-1002"},{"key":"9643_CR66","unstructured":"Pustejovsky, J., Castao, J., Ingria, R., Saur\u00ec, R., Gaizauskas, R., Setzer, A., & Katz, G. (2003a). TimeML: Robust Specification of Event and Temporal Expressions in Text. In: Fifth International Workshop on Computational Semantics (IWCS-5)"},{"key":"9643_CR67","unstructured":"Pustejovsky, J., Hanks, P., Sauri, R., See, A., Gaizauskas, R., Setzer, A., Radev, D., Sundheim, B., Day, D., Ferro, L., & et\u00a0al. (2003). The TimeBank corpus. In: Corpus linguistics, vol 2003, p\u00a040"},{"key":"9643_CR68","unstructured":"Pustejovsky, J., Lee, K., Bunt, H., & Romary, L. (2010). ISO-TimeML: An international standard for semantic annotation. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC\u201910), European Language Resources Association (ELRA), Valletta, Malta, http:\/\/www.lrec-conf.org\/proceedings\/lrec2010\/pdf\/55_Paper.pdf"},{"key":"9643_CR69","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/3225.001.0001","volume-title":"The Generative Lexicon","author":"J Pustejovsky","year":"1995","unstructured":"Pustejovsky, J. (1995). The Generative Lexicon. MIT Press."},{"key":"9643_CR70","doi-asserted-by":"publisher","unstructured":"Ramponi, A., & Plank, B. (2020). Neural unsupervised domain adaptation in NLP\u2014A survey. In: Proceedings of the 28th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Barcelona, Spain (Online), pp 6838\u20136855, https:\/\/doi.org\/10.18653\/v1\/2020.coling-main.603, https:\/\/www.aclweb.org\/anthology\/2020.coling-main.603","DOI":"10.18653\/v1\/2020.coling-main.603"},{"key":"9643_CR71","unstructured":"Rehm, G., Galanis, D., Labropoulou, P., Piperidis, S., Wel\u00df, M., Usbeck, R., K\u00f6hler, J., Deligiannis, M., Gkirtzou, K., Fischer, J., & et\u00a0al. (2020) Towards an interoperable ecosystem of ai and lt platforms: A roadmap for the implementation of different levels of interoperability. In: Proceedings of the 1st International Workshop on Language Technology Platforms, pp 96\u2013107"},{"key":"9643_CR72","unstructured":"Reimers, N., & Gurevych, I. (2017a). Optimal hyperparameters for deep lstm-networks for sequence labeling tasks. CoRR abs\/1707.06799, arxiv:1707.06799,"},{"key":"9643_CR73","doi-asserted-by":"crossref","unstructured":"Reimers, N., & Gurevych, I. (2017b). Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark, pp 338\u2013348, https:\/\/www.aclweb.org\/anthology\/D17-1035","DOI":"10.18653\/v1\/D17-1035"},{"key":"9643_CR74","doi-asserted-by":"crossref","unstructured":"Ritter, A., Etzioni, O., Clark, S., & et\u00a0al. (2012). Open domain event extraction from twitter. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, pp 1104\u20131112","DOI":"10.1145\/2339530.2339704"},{"key":"9643_CR75","doi-asserted-by":"crossref","unstructured":"Ruder, S., & Plank, B. (2017). Learning to select data for transfer learning with bayesian optimization. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark, pp 372\u2013382, https:\/\/www.aclweb.org\/anthology\/D17-1038","DOI":"10.18653\/v1\/D17-1038"},{"key":"9643_CR76","unstructured":"Ruder, S., Ghaffari, P., & Breslin, JG. (2017). Data selection strategies for multi-domain sentiment analysis. arXiv preprint arXiv:1702.02426"},{"key":"9643_CR77","unstructured":"Rupenhofer, J., Ellsworth, M., Petruck, MRL., Johnson, CR., & Schefczyk, J. (2010). FrameNet II: Extended theory and practice. https:\/\/framenet2.icsi.berkeley.edu\/docs\/r1.5\/book.pdf"},{"key":"9643_CR78","first-page":"495","volume":"1","author":"HJ Sasse","year":"2001","unstructured":"Sasse, H. J. (2001). Scales between nouniness and verbiness. Language typology and language universals, 1, 495\u2013509.","journal-title":"Language typology and language universals"},{"key":"9643_CR79","unstructured":"Saur\u0131, R., Littman, J., Knippen, B., Gaizauskas, R., Setzer, A., & Pustejovsky, J. (2006). Timeml annotation guidelines version 1.2. 1"},{"key":"9643_CR80","unstructured":"Simone, R. (2003). Masdar,\u2019ismu al-marrati et la fronti\u00e8re verbe\/nom. In: Estudios ofrecidos al Profesor Jos\u00e9 Jes\u00fas de Bustos Tovar, Servicio de Publicaciones, pp 901\u2013918"},{"issue":"2","key":"9643_CR81","first-page":"259","volume":"29","author":"R Simone","year":"2000","unstructured":"Simone, R. (2000). Cycles lexicaux. Studi italiani di linguistica teorica ed applicata, 29(2), 259\u2013287.","journal-title":"Studi italiani di linguistica teorica ed applicata"},{"key":"9643_CR82","doi-asserted-by":"publisher","unstructured":"Song, Z., Bies, A., Strassel, S., Ellis, J., Mitamura, T., Dang, HT., Yamakawa, Y., & Holm, S. (2016). Event nugget and event coreference annotation. In: Proceedings of the Fourth Workshop on Events, Association for Computational Linguistics, San Diego, California, pp 37\u201345, https:\/\/doi.org\/10.18653\/v1\/W16-1005,","DOI":"10.18653\/v1\/W16-1005"},{"key":"9643_CR83","doi-asserted-by":"publisher","unstructured":"Song, Z., Bies, A., Strassel, S., Riese, T., Mott, J., Ellis, J., Wright, J., Kulick, S., Ryant, N., & Ma, X. (2015). From light to rich ERE: Annotation of entities, relations, and events. In: Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, Association for Computational Linguistics, Denver, Colorado, pp 89\u201398, https:\/\/doi.org\/10.3115\/v1\/W15-0812,","DOI":"10.3115\/v1\/W15-0812"},{"key":"9643_CR84","doi-asserted-by":"publisher","unstructured":"Sprugnoli, R., & Tonelli, S. (2019). Novel event detection and classification for historical texts. Computational Linguistics 45(2):229\u2013265, https:\/\/doi.org\/10.1162\/coli_a_00347, https:\/\/www.aclweb.org\/anthology\/J19-2002","DOI":"10.1162\/coli_a_00347"},{"key":"9643_CR85","doi-asserted-by":"publisher","unstructured":"Stymne, S., de\u00a0Lhoneux, M., Smith, A., & Nivre, J. (2018). Parser training with heterogeneous treebanks. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Melbourne, Australia, pp 619\u2013625, https:\/\/doi.org\/10.18653\/v1\/P18-2098,","DOI":"10.18653\/v1\/P18-2098"},{"key":"9643_CR86","doi-asserted-by":"crossref","unstructured":"Sundheim, BM. (1992). Overview of the fourth Message Understanding Evaluation and Conference. In: Fourth Message Uunderstanding Conference (MUC-4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992, https:\/\/www.aclweb.org\/anthology\/M92-1001","DOI":"10.3115\/1072064.1072066"},{"key":"9643_CR87","unstructured":"Tonelli, S., Sprugnoli, R., Speranza, M., & Minard, AL. (2014). Newsreader guidelines for annotation at document level. Technical Re- port NWR2014-2-2, Fondazione Bruno Kessler"},{"key":"9643_CR88","doi-asserted-by":"publisher","unstructured":"Tu, L., Lalwani, G., Gella, S., & He, H. (2020). An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics 8:621\u2013633, https:\/\/doi.org\/10.1162\/tacl_a_00335, https:\/\/aclanthology.org\/2020.tacl-1.40","DOI":"10.1162\/tacl_a_00335"},{"key":"9643_CR89","unstructured":"UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., & Pustejovsky, J. (2013). Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In: Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Association for Computational Linguistics, Atlanta, Georgia, USA, pp 1\u20139, http:\/\/www.aclweb.org\/anthology\/S13-2001"},{"key":"9643_CR90","unstructured":"van Erp, M., Mendes, P., Paulheim, H., Ilievski, F., Plu, J., Rizzo, G., & Waitelonis, J. (2016). Evaluating entity linking: An analysis of current benchmark datasets and a roadmap for doing a better job. In: Chair) NCC, Choukri K, Declerck T, Goggi S, Grobelnik M, Maegaard B, Mariani J, Mazo H, Moreno A, Odijk J, Piperidis S (eds) Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), European Language Resources Association (ELRA), Paris, France"},{"key":"9643_CR91","unstructured":"van Son, C., Inel, O., Morante, R., Aroyo, L., & Vossen, P. (2018). Resource interoperability for sustainable benchmarking: The case of events. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018."},{"issue":"21","key":"9643_CR92","first-page":"704","volume":"64","author":"Z Vendler","year":"1967","unstructured":"Vendler, Z. (1967). Causal relations. The Journal of philosophy, 64(21), 704\u2013713.","journal-title":"Causal relations. The Journal of philosophy"},{"key":"9643_CR93","doi-asserted-by":"publisher","unstructured":"Venugopal D, Chen C, Gogate V, & Ng V (2014) Relieving the computational bottleneck: Joint inference for event extraction with high-dimensional features. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Doha, Qatar, pp 831\u2013843, https:\/\/doi.org\/10.3115\/v1\/D14-1090,","DOI":"10.3115\/v1\/D14-1090"},{"key":"9643_CR94","doi-asserted-by":"crossref","unstructured":"Verhagen M, Gaizauskas R, Schilder F, Hepple M, Katz G, & Pustejovsky J (2007) Semeval-2007 task 15: Tempeval temporal relation identification. In: Proceedings of the 4th International Workshop on Semantic Evaluations, Association for Computational Linguistics, pp 75\u201380","DOI":"10.3115\/1621474.1621488"},{"key":"9643_CR95","unstructured":"Verhagen M, Sauri R, Caselli T, & Pustejovsky J (2010) Semeval-2010 task 13: Tempeval-2. In: Proceedings of the 5th international workshop on semantic evaluation, Association for Computational Linguistics, pp 57\u201362"},{"key":"9643_CR96","doi-asserted-by":"crossref","unstructured":"Wang, X., Wang, H., & Yang, D. (2022). Measure and improve robustness in NLP models: A survey. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Seattle, United States, pp 4569\u20134586, https:\/\/aclanthology.org\/2022.naacl-main.339","DOI":"10.18653\/v1\/2022.naacl-main.339"},{"key":"9643_CR97","doi-asserted-by":"publisher","unstructured":"Wang, T., Wang, X., Qin, Y., Packer, B., Li, K., Chen, J., Beutel, A., & Chi, E. (2020). CAT-gen: Improving robustness in NLP models via controlled adversarial text generation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, pp 5141\u20135146, https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-main.417,","DOI":"10.18653\/v1\/2020.emnlp-main.417"},{"key":"9643_CR98","doi-asserted-by":"crossref","unstructured":"Windhouwer, M., & Wright, SE. (2012). Linking to linguistic data categories in isocat. In: Linked Data in Linguistics, Springer, pp 99\u2013107","DOI":"10.1007\/978-3-642-28249-2_10"},{"issue":"1","key":"9643_CR99","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10579-009-9088-x","volume":"43","author":"A Witt","year":"2009","unstructured":"Witt, A., Heid, U., Sasaki, F., & S\u00e9rasset, G. (2009). Multilingual language resources and interoperability. Language Resources and Evaluation, 43(1), 1\u201314.","journal-title":"Language Resources and Evaluation"},{"key":"9643_CR100","doi-asserted-by":"crossref","unstructured":"Wu, F., & Huang, Y. (2016). Sentiment domain adaptation with multiple sources. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol\u00a01, pp 301\u2013310","DOI":"10.18653\/v1\/P16-1029"},{"key":"9643_CR101","doi-asserted-by":"publisher","unstructured":"Yang, S., Feng, D., Qiao, L., Kan, Z., & Li, D. (2019). Exploring pre-trained language models for event extraction and generation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, pp 5284\u20135294, https:\/\/doi.org\/10.18653\/v1\/P19-1522,","DOI":"10.18653\/v1\/P19-1522"},{"key":"9643_CR102","doi-asserted-by":"crossref","unstructured":"Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & \u00c7\u00f6ltekin, \u00c7. (2020). SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), pp 1425\u20131447, https:\/\/www.aclweb.org\/anthology\/2020.semeval-1.188","DOI":"10.18653\/v1\/2020.semeval-1.188"},{"key":"9643_CR103","doi-asserted-by":"crossref","unstructured":"Zhong, E., Fan, W., Yang, Q., Verscheure, O., & Ren, J. (2010). Cross validation framework to choose amongst models and datasets for transfer learning. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer, pp 547\u2013562","DOI":"10.1007\/978-3-642-15939-8_35"}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-023-09643-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-023-09643-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-023-09643-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T05:28:54Z","timestamp":1728970134000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-023-09643-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,26]]},"references-count":104,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,9]]}},"alternative-id":["9643"],"URL":"https:\/\/doi.org\/10.1007\/s10579-023-09643-6","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"type":"print","value":"1574-020X"},{"type":"electronic","value":"1574-0218"}],"subject":[],"published":{"date-parts":[[2023,2,26]]},"assertion":[{"value":"30 January 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 February 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}