{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T18:29:25Z","timestamp":1767637765638,"version":"3.48.0"},"reference-count":50,"publisher":"Maximum Academic Press","license":[{"start":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T00:00:00Z","timestamp":1576540800000},"content-version":"unspecified","delay-in-days":350,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["The Knowledge Engineering Review"],"published-print":{"date-parts":[[2019]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The extraction and processing of temporal expressions (TEs) in textual documents have been extensively studied in several domains; however, for the legal domain it remains an open challenge. This is possibly due to the scarcity of corpora in the domain and the particularities found in legal documents that are highlighted in this paper. Considering the pivotal role played by temporal information when it comes to analyzing legal cases, this paper presents TempCourt, a corpus of 30 legal documents from the European Court of Human Rights, the European Court of Justice, and the United States Supreme Court with manually annotated TEs. The corpus contains two different temporal annotation sets that adhere to the TimeML standard, the first one capturing all TEs and the second dedicated to TEs that are relevant for the case under judgment (thus excluding dates of previous court decisions). The proposed gold standards are subsequently used to compare ten state-of-the-art cross-domain temporal taggers, and to identify not only the limitations of cross-domain temporal taggers but also limitations of the TimeML standard when applied to legal documents. Finally, the paper identifies the need for dedicated resources and the adaptation of existing tools, and specific annotation guidelines that can be adapted to different types of legal documents.<\/jats:p>","DOI":"10.1017\/s0269888919000195","type":"journal-article","created":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T04:08:21Z","timestamp":1576555701000},"source":"Crossref","is-referenced-by-count":4,"title":["TempCourt: evaluation of temporal taggers on a new corpus of court decisions"],"prefix":"10.48130","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1011-5023","authenticated-orcid":false,"given":"Mar\u00eda","family":"Navas-Loro","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3445-0504","authenticated-orcid":false,"given":"Erwin","family":"Filtz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1076-2511","authenticated-orcid":false,"given":"V\u00edctor","family":"Rodr\u00edguez-Doncel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5670-1146","authenticated-orcid":false,"given":"Axel","family":"Polleres","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6955-7718","authenticated-orcid":false,"given":"Sabrina","family":"Kirrane","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"27968","published-online":{"date-parts":[[2019,12,17]]},"reference":[{"key":"S0269888919000195_ref50","first-page":"420","article-title":"Time expression analysis and recognition using syntactic token types and general heuristic rules","volume":"1","author":"Zhong","year":"2017","journal-title":"Proceedings of the 55th Annual Meeting of the ACL"},{"key":"S0269888919000195_ref48","doi-asserted-by":"publisher","DOI":"10.3115\/1225753.1225774"},{"key":"S0269888919000195_ref49","doi-asserted-by":"crossref","unstructured":"Vlek, C.\u00a0S. , Prakken, H. , Renooij, S. & Verheij, B. 2013. Representing and evaluating legal narratives with subscenarios in a Bayesian network. In OASIcs-OpenAccess Series in Informatics, 32. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.","DOI":"10.1145\/2514601.2514618"},{"key":"S0269888919000195_ref47","unstructured":"UzZaman, N. , Llorens, H. , Derczynski, L. , Allen, J. , Verhagen, M. & Pustejovsky, J. 2013. SemEval-2013 Task 1: TempEval-3: evaluating time expressions, events, and temporal relations. In Proceedings of the Workshop SemEval 2013, 1\u20139."},{"key":"S0269888919000195_ref22","unstructured":"Llorens, H. , Saquete, E. & Navarro, B. 2010. TIPSem (english and spanish): evaluating CRFs and semantic roles inTempEval-2. In Proceedings of the Workshop SemEval, 284\u2013291. ACL."},{"key":"S0269888919000195_ref18","unstructured":"Kajiwara, T. & Komachi, M. 2016. Building a monolingual parallel corpus for text simplification using sentence similarity based on alignment between word embeddings. In Proceedings of COLING 2016: Technical Papers, 1147\u20131158."},{"key":"S0269888919000195_ref21","first-page":"1437","article-title":"Context-dependent semantic parsing for time expressions","volume":"1","author":"Lee","year":"2014","journal-title":"Proceedings of the 52nd Annual Meeting of the ACL"},{"key":"S0269888919000195_ref17","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-013-0675-1"},{"key":"S0269888919000195_ref14","doi-asserted-by":"publisher","DOI":"10.1109\/TIME.2006.16"},{"key":"S0269888919000195_ref9","unstructured":"Derczynski, L. & Gaizauskas, R. 2010. USFD2: annotating temporal expressions and TLINKs for TempEval-2. In Proceedings of the Workshop SemEval, 337\u2013340. ACL."},{"key":"S0269888919000195_ref7","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1002854"},{"key":"S0269888919000195_ref13","doi-asserted-by":"publisher","DOI":"10.1162\/089120102760275983"},{"key":"S0269888919000195_ref36","unstructured":"Quinlan, J.\u00a0R. 1993. C4.5: Programs for Machine Learning. Morgan Kaufmann. ISBN 1-55860-238-0."},{"key":"S0269888919000195_ref11","doi-asserted-by":"publisher","DOI":"10.1016\/j.datak.2015.09.002"},{"key":"S0269888919000195_ref23","unstructured":"Llorens, H. , Derczynski, L. , Gaizauskas, R. J. & Saquete, E. 2012. TIMEN: an open temporal expression normalisation resource. In Proceedings of LREC 2012, 3044\u20133051."},{"key":"S0269888919000195_ref19","first-page":"64","article-title":"JU_CSE: a CRF based approach to annotation of temporal expression, event and temporal relations","volume":"2","author":"Kolya","year":"2013","journal-title":"Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)"},{"key":"S0269888919000195_ref16","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41278-3_60"},{"key":"S0269888919000195_ref4","unstructured":"Chang, A.\u00a0X. & Manning, C.\u00a0D . 2012. SUTime: a library for recognizing and normalizing time expressions. In Proceedings of LREC 2012."},{"key":"S0269888919000195_ref3","first-page":"273","volume":"2","author":"Chambers","year":"2014","journal-title":"Transactions of the ACL"},{"key":"S0269888919000195_ref20","unstructured":"Lafferty, J.\u00a0D. , McCallum, A. & Pereira, F.\u00a0C.\u00a0N. 2001. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001),Williams College, Williamstown, MA, USA, June 28\u2013July 1, 2001, Brodley, C.\u00a0E. & Danyluk, A.\u00a0P. (eds). Morgan Kaufmann, 282\u2013289. ISBN 1-55860-778-1."},{"key":"S0269888919000195_ref2","first-page":"73","article-title":"NavyTime: event and time ordering from raw text","volume":"2","author":"Chambers","year":"2013","journal-title":"Proceedings of the Workshop SemEval 2013"},{"key":"S0269888919000195_ref8","unstructured":"Dell\u2019Orletta, F. , Marchi, S. , Montemagni, S. , Plank, B. & Venturi, G. 2012. The splet\u20132012 shared task on dependency parsing of legal texts. In Semantic Processing of Legal Texts (SPLeT-2012) Workshop Programme, 42."},{"key":"S0269888919000195_ref5","unstructured":"Chang, A.\u00a0X. , et\u00a0al. 2014. TokensRegex: Defining Cascaded Regular Expressions Over Tokens. Technical report CSTR 2014-02, Department of Computer Science, Stanford University."},{"key":"S0269888919000195_ref46","doi-asserted-by":"publisher","DOI":"10.1142\/S1793351X10001097"},{"key":"S0269888919000195_ref15","unstructured":"Hellmann, S. Lehmann, J. & Auer, S. 2012. NIF: an ontology-based and linked-data-aware NLP Interchange Format. http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.232.1215&rep=rep1&type=pdf"},{"key":"S0269888919000195_ref1","unstructured":"Bethard, S. 2013. ClearTK-TimeML: a minimalist approach to TempEval 2013. In Proceedings of the Workshop SemEval 2013, 10\u201314. ACL."},{"key":"S0269888919000195_ref10","unstructured":"Ferro, L. , Gerber, L. , Mani, I. , Sundheim, B. & Wilson, G. 2005. Tides 2005 Standard for the Annotation of Temporal Expressions. Technical report, MITRE."},{"key":"S0269888919000195_ref24","doi-asserted-by":"publisher","DOI":"10.3115\/1118108.1118117"},{"key":"S0269888919000195_ref25","doi-asserted-by":"publisher","DOI":"10.3115\/1075218.1075228"},{"key":"S0269888919000195_ref30","doi-asserted-by":"publisher","DOI":"10.5120\/3402-4742"},{"key":"S0269888919000195_ref32","unstructured":"Northwood, C. 2010. TERNIP: Temporal Expression Recognition and Normalisation in Python. Master\u2019s thesis, University of Sheffield."},{"key":"S0269888919000195_ref12","first-page":"1612","volume":"14","author":"Freund","year":"1999","journal-title":"Journal-Japanese Society For Artificial Intelligence"},{"key":"S0269888919000195_ref26","unstructured":"Manning, C. , Surdeanu, M. , Bauer, J. , Finkel, J. , Bethard, S. & McClosky, D. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of the 52nd Annual Meeting of the ACL 2014, System Demonstrations, 55\u201360."},{"key":"S0269888919000195_ref27","unstructured":"Mazur, P. & Dale, R. 2009. The dante temporal expression tagger. In Human Language Technology. Challenges of the Information Society, Vetulani, Z. & Uszkoreit, H. (eds). 245\u2013257. Springer. ISBN 978-3-642-04235-5."},{"key":"S0269888919000195_ref28","unstructured":"Mazur, P. & Dale, R. 2010. Wikiwars: a new corpus for research on temporal expressions. In Proceedings of EMNLP 2010, 913\u2013922. ACL."},{"key":"S0269888919000195_ref29","unstructured":"Minard, A.-L. , Speranza, M. , Urizar, R. , Altuna, B. , Van Erp, M. , Schoen, A. & Van Son, C. 2016. Meantime, the newsreader multilingual event and time corpus. In Proceedings of LREC 2016, European Language Resources Association."},{"key":"S0269888919000195_ref31","unstructured":"Nivre, J. , et al. 2007. The CoNLL 2007 shared task on dependency parsing. In Proceedings of EMNLP-CoNLL 2007."},{"key":"S0269888919000195_ref38","unstructured":"Saur\u00ed, R. , Littman, J. , Knippen, B. , Gaizauskas, R. , Setzer, A. & Pustejovsky, J. TimeML Annotation Guidelines. Version 1.2.1, 2006."},{"key":"S0269888919000195_ref43","first-page":"3746","article-title":"Temporal tagging on different domains: challenges, strategies, and gold standards","volume":"12","author":"Str\u00f6tgen","year":"2012","journal-title":"Proceedings of LREC 2012"},{"key":"S0269888919000195_ref34","first-page":"28","article-title":"The TimeBank corpus","volume":"2003","author":"Pustejovsky","year":"2003","journal-title":"Corpus Linguistics"},{"key":"S0269888919000195_ref41","unstructured":"Steedman, M. Jason, B. 2011. Combinatory categorial grammar, In Non-Transformational Syntax: Formal and Explicit Models of Grammar, Borsley, R. & Borjars, K. (eds). Wiley-Blackwell, 181\u2013224."},{"key":"S0269888919000195_ref35","unstructured":"Pustejovsky, J. , Castano, J. M. , Ingria, R. , Sauri, R. , Gaizauskas, R. J. , Setzer, A. Katz, G. & Radev, D. R. 2003bb. TimeML: robust specification of event and temporal expressions intext. In New Directions in Question Answering, Papers from 2003 AAAI Spring Symposium, Maybury, M.\u00a0T. (ed). AAAI Press, 28\u201334. ISBN 1-57735-184-3."},{"key":"S0269888919000195_ref37","unstructured":"Rijsbergen, C.\u00a0J.\u00a0V. 1979. Information Retrieval, 2nd edition. Butterworth-Heinemann. ISBN 0408709294."},{"key":"S0269888919000195_ref39","unstructured":"Schilder, F. 2005. Event extraction and temporal reasoning in legal documents. In Annotating, Extracting and Reasoning about Time and Events,International Seminar, Dagstuhl Castle. Revised Papers, 59\u201371."},{"key":"S0269888919000195_ref40","doi-asserted-by":"publisher","DOI":"10.1086\/266577"},{"key":"S0269888919000195_ref42","doi-asserted-by":"publisher","DOI":"10.2200\/S00721ED1V01Y201606HLT036"},{"key":"S0269888919000195_ref44","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00172"},{"key":"S0269888919000195_ref6","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"S0269888919000195_ref33","unstructured":"Pellow, D. & Eskenazi, M. 2014. An open corpus of everyday documents for simplification tasks. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), 84\u201393."},{"key":"S0269888919000195_ref45","unstructured":"Tabassum, J. , Ritter, A. & Xu, W. TweeTime: a minimally supervised method for recognizing and normalizing time expressions in Twitter. arXiv preprint arXiv:1608.02904, 2016."}],"container-title":["The Knowledge Engineering Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0269888919000195","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T14:42:14Z","timestamp":1767624134000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0269888919000195\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019]]},"references-count":50,"alternative-id":["S0269888919000195"],"URL":"https:\/\/doi.org\/10.1017\/s0269888919000195","relation":{},"ISSN":["0269-8889","1469-8005"],"issn-type":[{"type":"print","value":"0269-8889"},{"type":"electronic","value":"1469-8005"}],"subject":[],"published":{"date-parts":[[2019]]},"article-number":"e24"}}