{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:25:33Z","timestamp":1760149533591,"version":"build-2065373602"},"reference-count":45,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2023,8,1]],"date-time":"2023-08-01T00:00:00Z","timestamp":1690848000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Amazon Alexa AI"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>This paper introduces a multi-layered cross-genre corpus, annotated for coreference resolution, causal relations, and temporal relations, comprising a variety of genres, from news articles and children\u2019s stories to Reddit posts. Our results reveal distinctive genre-specific characteristics at each layer of annotation, highlighting unique challenges for both annotators and machine learning models. Children\u2019s stories feature linear temporal structures and clear causal relations. In contrast, news articles employ non-linear temporal sequences with minimal use of explicit causal or conditional language and few first-person pronouns. Lastly, Reddit posts are author-centered explanations of ongoing situations, with occasional meta-textual reference. Our annotation schemes are adapted from existing work to better suit a broader range of text types. We argue that our multi-layered cross-genre corpus not only reveals genre-specific semantic characteristics but also indicates a rich contextual interplay between the various layers of semantic information. Our MLCG corpus is shared under the open-source Apache 2.0 license.<\/jats:p>","DOI":"10.3390\/info14080431","type":"journal-article","created":{"date-parts":[[2023,8,1]],"date-time":"2023-08-01T09:24:24Z","timestamp":1690881864000},"page":"431","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Exploring a Multi-Layered Cross-Genre Corpus of Document-Level Semantic Relations"],"prefix":"10.3390","volume":"14","author":[{"given":"Gregor","family":"Williamson","sequence":"first","affiliation":[{"name":"Department of Computer Science, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Angela","family":"Cao","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yingying","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuxin","family":"Ji","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liyan","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2693-6934","authenticated-orcid":false,"given":"Jinho D.","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Emory University, Atlanta, GA 30322, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,1]]},"reference":[{"key":"ref_1","unstructured":"Li, X., Palmer, M., Xue, N., Ramshaw, L., Maamouri, M., Bies, A., Conger, K., Grimes, S., and Strassel, S. (2016, January 23\u201328). Large Multi-lingual, Multi-level and Multi-genre Annotation Corpus. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916), Portoro\u017e, Slovenia."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zeldes, A., and Simonson, D. (2016, January 11). Different Flavors of GUM: Evaluating Genre and Sentence Type Effects on Multilayer Corpus Annotation Quality. Proceedings of the 10th Linguistic Annotation Workshop Held in Conjunction with ACL 2016 (LAW-X 2016), Berlin, Germany.","DOI":"10.18653\/v1\/W16-1709"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1007\/s10579-016-9343-x","article-title":"The GUM corpus: Creating multilayer resources in the classroom","volume":"51","author":"Zeldes","year":"2017","journal-title":"Lang. Resour. Eval."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zeldes, A. (2018). Multilayer Corpus Studies, Routledge.","DOI":"10.4324\/9781315112473"},{"key":"ref_5","unstructured":"Gessler, L., Peng, S., Liu, Y., Zhu, Y., Behzad, S., and Zeldes, A. (2020, January 11\u201316). AMALGUM\u2014A Free, Balanced, Multilayer English Web Corpus. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Grishman, R., and Sundheim, B.M. (1996, January 5\u20139). Message understanding conference-6: A brief history. Proceedings of the COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, Copenhagen, Denmark.","DOI":"10.3115\/992628.992709"},{"key":"ref_7","unstructured":"Doddington, G.R., Mitchell, A., Przybocki, M.A., Ramshaw, L.A., Strassel, S.M., and Weischedel, R.M. (2004, January 26\u201328). The automatic content extraction (ace) program-tasks, data, and evaluation. Proceedings of the Lrec, Lisbon, Portugal."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., and Weischedel, R. (2006, January 4\u20139). OntoNotes: The 90% solution. Proceedings of the Human Language Technology Conference of the NAACL, NewYork, NY, USA.","DOI":"10.3115\/1614049.1614064"},{"key":"ref_9","unstructured":"Silvano, P., Leal, A., Silva, F., Cantante, I., Oliveira, F., and Mario Jorge, A. (2021, January 14\u201318). Developing a multilayer semantic annotation scheme based on ISO standards for the visualization of a newswire corpus. Proceedings of the 17th Joint ACL\u2014ISO Workshop on Interoperable Semantic Annotation, Groningen, The Netherlands."},{"key":"ref_10","unstructured":"Bamman, D., Lewke, O., and Mansoor, A. (2020, January 11\u201316). An Annotated Dataset of Coreference in English Literature. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chen, H., Fan, Z., Lu, H., Yuille, A., and Rong, S. (November, January 31). PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.","DOI":"10.18653\/v1\/D18-1016"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Guha, A., Iyyer, M., Bouman, D., and Boyd-Graber, J. (June, January 31). Removing the training wheels: A coreference dataset that entertains humans and challenges computers. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, CO, USA.","DOI":"10.3115\/v1\/N15-1117"},{"key":"ref_13","unstructured":"Apostolova, E., Tomuro, N., Mongkolwat, P., and Demner-Fushman, D. (2012, January 3\u20138). Domain adaptation of coreference resolution for radiology reports. Proceedings of the BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing, Montreal, QC, Canada."},{"key":"ref_14","first-page":"28","article-title":"TimeML: Robust Specification of Event and Temporal Expressions in Text","volume":"3","author":"Pustejovsky","year":"2003","journal-title":"New Dir. Quest. Answ."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Verhagen, M., Gaizauskas, R., Schilder, F., Hepple, M., Katz, G., and Pustejovsky, J. (2007, January 23\u201324). SemEval-2007 Task 15: TempEval Temporal Relation Identification. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), Prague, Czech Republic.","DOI":"10.3115\/1621474.1621488"},{"key":"ref_16","unstructured":"Verhagen, M., Saur\u00ed, R., Caselli, T., and Pustejovsky, J. (2010, January 15\u201316). SemEval-2010 Task 13: TempEval-2. Proceedings of the 5th International Workshop on Semantic Evaluation, Uppsala, Sweden."},{"key":"ref_17","unstructured":"UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. (2013, January 14\u201315). SemEval-2013 Task 1: TempEval-3: Evaluating Time Expressions, Events, and Temporal Relations. Proceedings of the Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Atlanta, GA, USA."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ning, Q., Wu, H., and Roth, D. (2018, January 15\u201320). A Multi-Axis Annotation Scheme for Event Temporal Relations. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.","DOI":"10.18653\/v1\/P18-1122"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhang, Y., and Xue, N. (November, January 31). Neural Ranking Models for Temporal Dependency Structure Parsing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.","DOI":"10.18653\/v1\/D18-1371"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Yao, J., Qiu, H., Min, B., and Xue, N. (2020, January 16\u201320). Annotating Temporal Dependency Graphs via Crowdsourcing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.432"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1007\/s13218-021-00722-w","article-title":"Designing a Uniform Meaning Representation for Natural Language Processing","volume":"35","author":"Vigus","year":"2021","journal-title":"KI-K\u00fcnstliche Intell."},{"key":"ref_22","unstructured":"Bethard, S., Kolomiyets, O., and Moens, M.F. (2012, January 23\u201325). Annotating Story Timelines as Temporal Dependency Structures. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC\u201912), Istanbul, Turkey."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., and Allen, J. (2016, January 12\u201317). A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.","DOI":"10.18653\/v1\/N16-1098"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"O\u2019Gorman, T., Wright-Bettner, K., and Palmer, M. (2016, January 5). Richer Event Description: Integrating event coreference with temporal, causal and bridging annotation. Proceedings of the 2nd Workshop on Computing News Storylines (CNS 2016), Austin, TX, USA.","DOI":"10.18653\/v1\/W16-5706"},{"key":"ref_25","unstructured":"Araki, J. (2018). Extraction of Event Structures from Text. [Ph.D. Thesis, Carnegie Mellon University]."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1207\/s15516709cog1201_2","article-title":"Force dynamics in language and cognition","volume":"12","author":"Talmy","year":"1988","journal-title":"Cogn. Sci."},{"key":"ref_27","unstructured":"Wolff, P., Klettke, B., Ventura, T., and Song, G. (2005). Categorization Inside and Outside the Laboratory: Essays in Honor of Douglas L. Medin, American Psychological Association."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mirza, P., Sprugnoli, R., Tonelli, S., and Speranza, M. (2014, January 26). Annotating Causality in the TempEval-3 Corpus. Proceedings of the EACL 2014 Workshop on Computational Approaches to Causality in Language (CAtoCL), Gothenburg, Sweden.","DOI":"10.3115\/v1\/W14-0702"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Mostafazadeh, N., Grealish, A., Chambers, N., Allen, J., and Vanderwende, L. (2016, January 17). CaTeRS: Causal and Temporal Relation Scheme for Semantic Annotation of Event Structures. Proceedings of the Fourth Workshop on Events, San Diego, CA, USA.","DOI":"10.18653\/v1\/W16-1007"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1111\/j.1551-6709.2008.01002.x","article-title":"A Causal Model Theory of the Meaning of Cause, Enable, and Prevent","volume":"33","author":"Sloman","year":"2009","journal-title":"Cogn. Sci."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cao, A., Geiger, A., Kreiss, E., Icard, T., and Gerstenberg, T. (2023, January 26\u201329). A Semantics for Causing, Enabling, and Preventing Verbs Using Structural Causal Models. Proceedings of the 45th Annual Conference of the Cognitive Science Society, Sydney, Australia.","DOI":"10.31234\/osf.io\/kpu52"},{"key":"ref_32","unstructured":"Beller, A., Bennett, E., and Gerstenberg, T. (August, January 29). The language of causation. Proceedings of the 42nd Annual Conference of the Cognitive Science Society, Online."},{"key":"ref_33","unstructured":"Gerstenberg, T., Goodman, N.D., Lagnado, D., and Tenenbaum, J. (2015, January 22\u201325). How, whether, why: Causal judgements as counterfactual contrasts. Proceedings of the CogSci 2015, Pasadena, CA, USA."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"936","DOI":"10.1037\/rev0000281","article-title":"A counterfactual simulation model of causal judgments for physical Events","volume":"128","author":"Gerstenberg","year":"2021","journal-title":"Psychol. Rev."},{"key":"ref_35","unstructured":"Bonial, C., Babko-Malaya, O., Choi, J., Hwang, J., and Palmer, M. (2010). PropBank Annotation Guidelines, Center for Computational Language and Education Research Institute of Cognitive Science, University of Colorado at Boulder."},{"key":"ref_36","unstructured":"Dunietz, J. (2018). Annotating and Automatically Tagging Constructions of Causal Language. [Ph.D. Thesis, Carnegie Mellon University]."},{"key":"ref_37","unstructured":"Cao, A., Williamson, G., and Choi, J.D. (2022, January 24). A Cognitive Approach to Annotating Causal Constructions in a Cross-Genre Corpus. Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, Marseille, France."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Dunietz, J., Levin, L., and Carbonell, J. (2017, January 3). The BECauSE Corpus 2.0: Annotating Causality and Overlapping Relations. Proceedings of the 11th Linguistic Annotation Workshop, Valencia, Spain.","DOI":"10.18653\/v1\/W17-0812"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Lu, J., and Ng, V. (2018, January 13\u201319). Event coreference resolution: A survey of two decades of research. Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden.","DOI":"10.24963\/ijcai.2018\/773"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Vilain, M., Burger, J.D., Aberdeen, J., Connolly, D., and Hirschman, L. (1995, January 6\u20138). A model-theoretic coreference scoring scheme. Proceedings of the Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference, Columbia, MD, USA.","DOI":"10.3115\/1072399.1072405"},{"key":"ref_41","unstructured":"Bagga, A., and Baldwin, B. (1998, January 28\u201330). Algorithms for scoring coreference chains. Proceedings of the First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, Granada, Spain."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Luo, X. (2005, January 6\u20138). On coreference resolution performance metrics. Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Vancouver, BC, Canada.","DOI":"10.3115\/1220575.1220579"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Lewis, P., Oguz, B., Rinott, R., Riedel, S., and Schwenk, H. (2020, January 5\u201310). MLQA: Evaluating Cross-lingual Extractive Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.653"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s10579-005-7884-5","article-title":"Temporal closure in an annotation environment","volume":"39","author":"Verhagen","year":"2005","journal-title":"Lang. Resour. Eval."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Mani, I., Verhagen, M., Wellner, B., Lee, C.M., and Pustejovsky, J. (2006, January 17\u201318). Machine Learning of Temporal Relations. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia.","DOI":"10.3115\/1220175.1220270"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/8\/431\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:23:51Z","timestamp":1760127831000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/8\/431"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,1]]},"references-count":45,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["info14080431"],"URL":"https:\/\/doi.org\/10.3390\/info14080431","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2023,8,1]]}}}