{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T05:37:21Z","timestamp":1771306641725,"version":"3.50.1"},"reference-count":94,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2014,5,1]],"date-time":"2014-05-01T00:00:00Z","timestamp":1398902400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2015,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Events play an important role in natural language processing and information retrieval due to numerous event-oriented texts and information needs. Many natural language processing and information retrieval applications could benefit from a structured event-oriented document representation. In this paper, we propose<jats:italic>event graphs<\/jats:italic>as a novel way of structuring event-based information from text. Nodes in event graphs represent the individual mentions of events, whereas edges represent the temporal and coreference relations between mentions. Contrary to previous natural language processing research, which has mainly focused on individual event extraction tasks, we describe a complete end-to-end system for event graph extraction from text. Our system is a three-stage pipeline that performs anchor extraction, argument extraction, and relation extraction (temporal relation extraction and event coreference resolution), each at a performance level comparable with the state of the art. We present<jats:sc>EvExtra<\/jats:sc>, a large newspaper corpus annotated with event mentions and event graphs, on which we train and evaluate our models. To measure the overall quality of the constructed event graphs, we propose two metrics based on the tensor product between automatically and manually constructed graphs. Finally, we evaluate the overall quality of event graphs with the proposed evaluation metrics and perform a headroom analysis of the system.<\/jats:p>","DOI":"10.1017\/s1351324914000060","type":"journal-article","created":{"date-parts":[[2014,5,1]],"date-time":"2014-05-01T12:33:29Z","timestamp":1398947609000},"page":"607-652","source":"Crossref","is-referenced-by-count":21,"title":["Construction and evaluation of event graphs"],"prefix":"10.1017","volume":"21","author":[{"given":"GORAN","family":"GLAVA\u0160","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8942-5301","authenticated-orcid":false,"given":"JAN","family":"\u0160NAJDER","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2014,5,1]]},"reference":[{"key":"S1351324914000060_ref094","doi-asserted-by":"publisher","DOI":"10.3115\/992730.992782"},{"key":"S1351324914000060_ref092","doi-asserted-by":"publisher","DOI":"10.1109\/5254.784083"},{"key":"S1351324914000060_ref091","doi-asserted-by":"publisher","DOI":"10.3115\/981732.981751"},{"key":"S1351324914000060_ref088","first-page":"57","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval 2010)","author":"Verhagen","year":"2010"},{"key":"S1351324914000060_ref087","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621488"},{"key":"S1351324914000060_ref086","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-75989-8_2"},{"key":"S1351324914000060_ref084","volume-title":"Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013), in Conjunction With the Second Joint Conference on Lexical and Computational Semantcis (* SEM 2013), Association for Computational Linguistics","author":"UzZaman","year":"2013"},{"key":"S1351324914000060_ref083","first-page":"351","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT 2011)","author":"UzZaman","year":"2011"},{"key":"S1351324914000060_ref082","first-page":"276","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation","author":"UzZaman","year":"2010"},{"key":"S1351324914000060_ref078","doi-asserted-by":"publisher","DOI":"10.3115\/1706543.1706586"},{"key":"S1351324914000060_ref073","first-page":"25","volume-title":"Proceedings of the Fourth International Workshop on Inference in Computational Semantics (ICOS-4)","author":"Setzer","year":"2003"},{"key":"S1351324914000060_ref072","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00096"},{"key":"S1351324914000060_ref090","doi-asserted-by":"publisher","DOI":"10.1162\/0891201054223977"},{"key":"S1351324914000060_ref071","volume-title":"Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012) in Conjunction with the First Joint Conference on Lexical and Computational Semantics (*SEM 2012)","author":"\u0160ari\u0107","year":"2012"},{"key":"S1351324914000060_ref069","doi-asserted-by":"publisher","DOI":"10.1145\/361219.361220"},{"key":"S1351324914000060_ref067","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1613\/jair.514","article-title":"Semantic similarity in a taxonomy: an information-based measure and its application to problems of ambiguity in natural language","volume":"11","author":"Resnik","year":"1999","journal-title":"Journal of Artificial Intelligence Research"},{"key":"S1351324914000060_ref066","first-page":"162","volume-title":"Actions and Events: Perspectives on the Philosophy of Donald Davidson","author":"Quine","year":"1985"},{"key":"S1351324914000060_ref076","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2013-001628"},{"key":"S1351324914000060_ref065","first-page":"40","volume-title":"Proceedings of Corpus Linguistics","author":"Pustejovsky","year":"2003a"},{"key":"S1351324914000060_ref064","first-page":"28","article-title":"TimeML: robust specification of event and temporal expressions in text","volume":"2003","author":"Pustejovsky","year":"2003b","journal-title":"New Directions in Question Answering"},{"key":"S1351324914000060_ref063","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0277(91)90032-Y"},{"key":"S1351324914000060_ref060","doi-asserted-by":"publisher","DOI":"10.1162\/0891201053630264"},{"key":"S1351324914000060_ref059","first-page":"1683","volume-title":"Proceedings of the 20th International Joint Conference on Artifical Intelligence","author":"Navigli","year":"2007"},{"key":"S1351324914000060_ref058","first-page":"446","volume-title":"Proceedings of the 13th ACM International Conference on Information and Knowledge Management","author":"Nallapati","year":"2004"},{"key":"S1351324914000060_ref057","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1075\/li.30.1.03nad","article-title":"A survey of named entity recognition and classification","volume":"30","author":"Nadeau","year":"2007","journal-title":"Lingvisticae Investigationes"},{"key":"S1351324914000060_ref053","doi-asserted-by":"publisher","DOI":"10.3115\/1119176.1119206"},{"key":"S1351324914000060_ref052","doi-asserted-by":"publisher","DOI":"10.1023\/B:INRT.0000011210.12953.86"},{"key":"S1351324914000060_ref062","first-page":"1","volume-title":"Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task (CoNLL 2011)","author":"Pradhan","year":"2011"},{"key":"S1351324914000060_ref051","first-page":"43","volume-title":"Proceedings of the Student Research Workshop at Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL-HLT \u201903)","author":"Makkonen","year":"2003"},{"key":"S1351324914000060_ref050","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2012.05.005"},{"key":"S1351324914000060_ref006","doi-asserted-by":"publisher","DOI":"10.1145\/182.358434"},{"key":"S1351324914000060_ref077","first-page":"8","volume-title":"Proceedings of the 41st Annual Meeting on Association for Computational Linguistics","author":"Surdeanu","year":"2003"},{"key":"S1351324914000060_ref032","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013)","author":"Glava\u0161","year":"2013"},{"key":"S1351324914000060_ref014","first-page":"10","volume-title":"Second Joint Conference on Lexical and Computational Semantics (*SEM)","author":"Bethard","year":"2013"},{"key":"S1351324914000060_ref027","first-page":"1871","article-title":"LibLinear: a library for large linear classification","volume":"9","author":"Fan","year":"2008","journal-title":"Journal of Machine Learning Research"},{"key":"S1351324914000060_ref037","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1145\/345508.345582","volume-title":"Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Hatzivassiloglou","year":"2000"},{"key":"S1351324914000060_ref070","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1613\/jair.2805","article-title":"Enhancing QA systems with complex temporal question processing capabilities","volume":"35","author":"Saquete","year":"2009","journal-title":"Journal of Artificial Intelligence Research"},{"key":"S1351324914000060_ref044","first-page":"88","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012)","author":"Kolomiyets","year":"2012"},{"key":"S1351324914000060_ref013","unstructured":"Bethard S. 2008. Finding Event, Temporal and Causal Structure in Text: A Machine Learning Approach. PhD thesis. University of Colorado at Boulder, USA."},{"key":"S1351324914000060_ref024","first-page":"105","article-title":"The logical form of action sentences","volume":"5","author":"Davidson","year":"1967","journal-title":"Essays on Actions and Events"},{"key":"S1351324914000060_ref021","volume-title":"SUTIME: A Library for Recognizing and Normalizing Time Expressions","author":"Chang","year":"2012"},{"key":"S1351324914000060_ref031","doi-asserted-by":"publisher","DOI":"10.1162\/089120102760275983"},{"key":"S1351324914000060_ref089","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2006.886377"},{"key":"S1351324914000060_ref080","first-page":"3","article-title":"A history of events in linguistic theory","volume":"32","author":"Tenny","year":"2000","journal-title":"Events as Grammatical Objects"},{"key":"S1351324914000060_ref068","first-page":"3","article-title":"The syntactic representation of linguistic events","volume":"4","author":"Rosen","year":"1999","journal-title":"Glot International"},{"key":"S1351324914000060_ref035","doi-asserted-by":"publisher","DOI":"10.3115\/1706543.1706574"},{"key":"S1351324914000060_ref047","first-page":"28","volume-title":"Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task (CoNLL 2011)","author":"Lee","year":"2011"},{"key":"S1351324914000060_ref022","volume-title":"Proceedings of 5th International Joint Conference on Natural Language Processing (IJCNLP 2011)","author":"Chen","year":"2011"},{"key":"S1351324914000060_ref081","doi-asserted-by":"publisher","DOI":"10.1145\/1281192.1281271"},{"key":"S1351324914000060_ref048","first-page":"489","volume-title":"Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP 2012)","author":"Lee","year":"2012"},{"key":"S1351324914000060_ref085","doi-asserted-by":"publisher","DOI":"10.2307\/2182371"},{"key":"S1351324914000060_ref039","volume-title":"Proceedings of the Seventh Message Understanding Conferences (MUC-7)","author":"Humphreys","year":"1998"},{"key":"S1351324914000060_ref075","doi-asserted-by":"publisher","DOI":"10.1162\/089120101753342653"},{"key":"S1351324914000060_ref041","first-page":"2109","volume-title":"Proceedings of the International Conference on Computational Linguistics (COLING 2012)","author":"Jun","year":"2012"},{"key":"S1351324914000060_ref030","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-45167-9_11"},{"key":"S1351324914000060_ref018","unstructured":"Chambers N. , and Jurafsky D. 2008. Unsupervised learning of narrative event chains. In Proceedings of 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT \u201908), pp. 789\u201397."},{"key":"S1351324914000060_ref028","doi-asserted-by":"publisher","DOI":"10.1111\/j.1749-6632.1976.tb25467.x"},{"key":"S1351324914000060_ref093","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2009.2015885"},{"key":"S1351324914000060_ref074","doi-asserted-by":"publisher","DOI":"10.1023\/A:1005439826271"},{"key":"S1351324914000060_ref020","doi-asserted-by":"publisher","DOI":"10.1145\/1961189.1961199"},{"key":"S1351324914000060_ref054","volume-title":"Proceedings of the Document Understanding Conference","author":"Melli","year":"2006"},{"key":"S1351324914000060_ref016","unstructured":"Borgwardt K. M. 2007. Graph Kernels. PhD thesis, Ludwig-Maximilians-Universit\u00e4t M\u00fcnchen, Munich, Germany."},{"key":"S1351324914000060_ref010","volume-title":"Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008)","author":"Bejan","year":"2008"},{"key":"S1351324914000060_ref001","volume-title":"Evaluation of the Detection and Recognition of ACE: Entities, Values, Temporal Expressions, Relations, and Events","year":"2005"},{"key":"S1351324914000060_ref002","first-page":"33","volume-title":"Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL \u201909)","author":"Agirre","year":"2009"},{"key":"S1351324914000060_ref003","first-page":"1","volume-title":"Proceedings of COLING\/ACL 2006 Workshop on Annotating and Reasoning about Time and Events","author":"Ahn","year":"2006"},{"key":"S1351324914000060_ref004","first-page":"127","volume-title":"Proceedings of the 5th International Workshop on Inference in Computational Semantics","author":"Ahn","year":"2006"},{"key":"S1351324914000060_ref043","first-page":"1989","volume-title":"Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC \u201902)","author":"Kingsbury","year":"2002"},{"key":"S1351324914000060_ref005","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-0933-2"},{"key":"S1351324914000060_ref007","doi-asserted-by":"publisher","DOI":"10.3115\/974147.974158"},{"key":"S1351324914000060_ref008","doi-asserted-by":"publisher","DOI":"10.3115\/1608810"},{"key":"S1351324914000060_ref009","doi-asserted-by":"publisher","DOI":"10.3115\/980451.980860"},{"key":"S1351324914000060_ref015","volume-title":"Annotating, Extracting and Reasoning about Time and Events, Dagstuhl Seminar Proceedings","author":"Boguraev","year":"2005"},{"key":"S1351324914000060_ref011","first-page":"1412","volume-title":"Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010)","author":"Bejan","year":"2010"},{"key":"S1351324914000060_ref012","doi-asserted-by":"publisher","DOI":"10.3115\/1621474.1621576"},{"key":"S1351324914000060_ref017","doi-asserted-by":"publisher","DOI":"10.3115\/1610075.1610105"},{"key":"S1351324914000060_ref019","first-page":"602","volume-title":"Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP","author":"Chambers","year":"2009"},{"key":"S1351324914000060_ref023","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2004.75"},{"key":"S1351324914000060_ref025","first-page":"449","volume-title":"Proceedings of 5th International Conference on Language Resources and Evaluation (LREC 2006)","author":"De Marneffe","year":"2006"},{"key":"S1351324914000060_ref026","volume-title":"Proceedings of the Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria","author":"Derczynski","year":"2013"},{"key":"S1351324914000060_ref038","doi-asserted-by":"publisher","DOI":"10.3115\/1598819.1598830"},{"key":"S1351324914000060_ref029","doi-asserted-by":"publisher","DOI":"10.3115\/1219840.1219885"},{"key":"S1351324914000060_ref033","doi-asserted-by":"crossref","first-page":"466","DOI":"10.3115\/992628.992709","volume-title":"Proceedings of International Conference on Computational Linguistics (COLING 1996)","author":"Grishman","year":"1996"},{"key":"S1351324914000060_ref040","first-page":"336","volume-title":"Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics","author":"Jans","year":"2012"},{"key":"S1351324914000060_ref034","first-page":"333","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval 2010)","author":"Grover","year":"2010"},{"key":"S1351324914000060_ref036","doi-asserted-by":"crossref","DOI":"10.1201\/b10959","volume-title":"Handbook of Product Graphs. Discrete Mathematics and Its Applications","author":"Hammack","year":"2011"},{"key":"S1351324914000060_ref042","volume-title":"Annotating, Extracting and Reasoning about Time and Events, Dagstuhl Seminar Proceedings","author":"Karttunen","year":"2005"},{"key":"S1351324914000060_ref079","first-page":"150","volume-title":"Proceedings of the Conference on Language Resources and Evaluation","author":"Tannier","year":"2008"},{"key":"S1351324914000060_ref045","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1145\/1008992.1009044","volume-title":"Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Kumaran","year":"2004"},{"key":"S1351324914000060_ref055","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102425"},{"key":"S1351324914000060_ref046","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-009-9059-4"},{"key":"S1351324914000060_ref061","first-page":"1","article-title":"Contextual valence shifters","volume":"20","author":"Polanyi","year":"2006","journal-title":"Computing Attitude and Affect in Text: Theory and Applications"},{"key":"S1351324914000060_ref056","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2010.03.008"},{"key":"S1351324914000060_ref049","first-page":"284","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation","author":"Llorens","year":"2010"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324914000060","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,2]],"date-time":"2022-04-02T17:05:23Z","timestamp":1648919123000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324914000060\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,5,1]]},"references-count":94,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2015,8]]}},"alternative-id":["S1351324914000060"],"URL":"https:\/\/doi.org\/10.1017\/s1351324914000060","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,5,1]]}}}