{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T18:47:18Z","timestamp":1773686838239,"version":"3.50.1"},"reference-count":47,"publisher":"Oxford University Press (OUP)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Objective To develop an open-source temporal relation discovery system for the clinical domain. The system is capable of automatically inferring temporal relations between events and time expressions using a multilayered modeling strategy. It can operate at different levels of granularity\u2014from rough temporality expressed as event relations to the document creation time (DCT) to temporal containment to fine-grained classic Allen-style relations.<\/jats:p>\n               <jats:p>Materials and Methods We evaluated our systems on 2 clinical corpora. One is a subset of the Temporal Histories of Your Medical Events (THYME) corpus, which was used in SemEval 2015 Task 6: Clinical TempEval. The other is the 2012 Informatics for Integrating Biology and the Bedside (i2b2) challenge corpus. We designed multiple supervised machine learning models to compute the DCT relation and within-sentence temporal relations. For the i2b2 data, we also developed models and rule-based methods to recognize cross-sentence temporal relations. We used the official evaluation scripts of both challenges to make our results comparable with results of other participating systems. In addition, we conducted a feature ablation study to find out the contribution of various features to the system\u2019s performance.<\/jats:p>\n               <jats:p>Results Our system achieved state-of-the-art performance on the Clinical TempEval corpus and was on par with the best systems on the i2b2 2012 corpus. Particularly, on the Clinical TempEval corpus, our system established a new F1 score benchmark, statistically significant as compared to the baseline and the best participating system.<\/jats:p>\n               <jats:p>Conclusion Presented here is the first open-source clinical temporal relation discovery system. It was built using a multilayered temporal modeling strategy and achieved top performance in 2 major shared tasks.<\/jats:p>","DOI":"10.1093\/jamia\/ocv113","type":"journal-article","created":{"date-parts":[[2015,11,1]],"date-time":"2015-11-01T00:38:21Z","timestamp":1446338301000},"page":"387-395","source":"Crossref","is-referenced-by-count":33,"title":["Multilayered temporal modeling for the clinical domain"],"prefix":"10.1093","volume":"23","author":[{"given":"Chen","family":"Lin","sequence":"first","affiliation":[{"name":"Boston Children\u2019s Hospital Boston, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dmitriy","family":"Dligach","sequence":"additional","affiliation":[{"name":"Boston Children\u2019s Hospital Boston, Boston, Massachusetts, USA"},{"name":"Harvard Medical School, Harvard University, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timothy A","family":"Miller","sequence":"additional","affiliation":[{"name":"Boston Children\u2019s Hospital Boston, Boston, Massachusetts, USA"},{"name":"Harvard Medical School, Harvard University, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Steven","family":"Bethard","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Sciences, University of Alabama at Birmingham, Birmingham, Alabama, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guergana K","family":"Savova","sequence":"additional","affiliation":[{"name":"Boston Children\u2019s Hospital Boston, Boston, Massachusetts, USA"},{"name":"Harvard Medical School, Harvard University, Boston, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2015,10,31]]},"reference":[{"issue":"3","key":"2020110612591539000_ocv113-B1","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1197\/jamia.M2922","article-title":"Syndromic surveillance using ambulatory electronic health records","volume":"16","author":"Hripcsak","year":"2009","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020110612591539000_ocv113-B2","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1016\/S0010-4825(96)00010-8","article-title":"Temporal reasoning and temporal data maintenance in medicine: issues and challenges","volume":"27","author":"Combi","year":"1997","journal-title":"Comput Biol Med"},{"key":"2020110612591539000_ocv113-B3","article-title":"A comparison of the temporal expressiveness of three database query methods","author":"Das","year":"1995","journal-title":"Annual Symposium on Computer Applications in Medical Care"},{"issue":"1","key":"2020110612591539000_ocv113-B4","first-page":"4","article-title":"Extensions to the time-oriented database model to support temporal reasoning in medical expert systems","volume":"30","author":"Kahn","year":"1990","journal-title":"Methods Inform Med"},{"issue":"4","key":"2020110612591539000_ocv113-B5","doi-asserted-by":"crossref","first-page":"610","DOI":"10.1002\/ana.20630","article-title":"White matter lesion progression, brain atrophy, and cognitive decline: the Austrian stroke prevention study","volume":"58","author":"Schmidt","year":"2005","journal-title":"Ann Neurol"},{"issue":"2","key":"2020110612591539000_ocv113-B6","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/j.jbi.2006.12.009","article-title":"Temporal reasoning with medical data\u2014a review with emphasis on medical natural language processing","volume":"40","author":"Zhou","year":"2007","journal-title":"J Biomed Inform"},{"key":"2020110612591539000_ocv113-B7","volume-title":"TimeML Annotation Guidelines, Version 1.2.1. \n              Citeseer","author":"Saur\u00ed","year":"2006"},{"key":"2020110612591539000_ocv113-B8","article-title":"The timebank corpus","volume-title":"Corpus Linguistics","author":"Pustejovsky"},{"key":"2020110612591539000_ocv113-B9","doi-asserted-by":"crossref","DOI":"10.3115\/1621474.1621488","article-title":"Semeval-2007 task 15: Tempeval temporal relation identification","volume-title":"Proceedings of the 4th International Workshop on Semantic Evaluations","author":"Verhagen","year":"2007"},{"key":"2020110612591539000_ocv113-B10","article-title":"SemEval-2010 task 13: TempEval-2","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation","author":"Verhagen","year":"2010"},{"key":"2020110612591539000_ocv113-B11","article-title":"SemEval-2013 task 1: TempEval-3: evaluating time expressions, events, and temporal relations","volume-title":"Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)","author":"UzZaman","year":"2013"},{"key":"2020110612591539000_ocv113-B12","article-title":"Towards temporal relation discovery from the clinical narrative","volume-title":"AMIA Annual Symposium Proceedings","author":"Savova","year":"2009"},{"key":"2020110612591539000_ocv113-B13","doi-asserted-by":"crossref","DOI":"10.1145\/2110363.2110446","article-title":"A corpus of clinical narratives annotated with temporal information","volume-title":"Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium","author":"Galescu","year":"2012"},{"issue":"5","key":"2020110612591539000_ocv113-B14","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1136\/amiajnl-2013-001628","article-title":"Evaluating temporal relations in clinical text: 2012 i2b2 Challenge","volume":"20","author":"Sun","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612591539000_ocv113-B15","article-title":"Increasing informativeness in temporal annotation","volume-title":"Proceedings of the 5th Linguistic Annotation Workshop","author":"Pustejovsky","year":"2011"},{"key":"2020110612591539000_ocv113-B16","author":"THYME"},{"key":"2020110612591539000_ocv113-B17","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1162\/tacl_a_00172","article-title":"Temporal annotations in the clinical domain","volume":"2","author":"Styler","year":"2014","journal-title":"Trans Assoc Comput Linguist"},{"key":"2020110612591539000_ocv113-B18","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/S15-2136","article-title":"Semeval-2015 task 6: Clinical tempeval","volume-title":"Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)","author":"Bethard","year":"2015"},{"key":"2020110612591539000_ocv113-B19","doi-asserted-by":"crossref","DOI":"10.3115\/v1\/P14-1094","article-title":"Cross-narrative temporal ordering of medical events","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Raghavan","year":"2014"},{"key":"2020110612591539000_ocv113-B20","first-page":"221","article-title":"An interval-based representation of temporal knowledge","volume-title":"IJCAI","author":"Allen","year":"1981"},{"key":"2020110612591539000_ocv113-B21","first-page":"18","article-title":"Discovering narrative containers in clinical text","author":"Miller","journal-title":"ACL 2013"},{"key":"2020110612591539000_ocv113-B22","volume-title":"Clinical TempEval","author":"Bethard","year":"2014"},{"key":"2020110612591539000_ocv113-B23","author":"TempEval"},{"key":"2020110612591539000_ocv113-B24","doi-asserted-by":"crossref","DOI":"10.1136\/amiajnl-2013-001624","article-title":"\u00c0 la Recherche du Temps Perdu: extracting temporal relations from medical text in the 2012 i2b2 NLP challenge","author":"Cherry","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612591539000_ocv113-B25","article-title":"Tipsem (English and Spanish): evaluating CRFs and semantic roles in tempeval-2","volume-title":"Proceedings of the 5th International Workshop on Semantic Evaluation","author":"Llorens","year":"2010"},{"key":"2020110612591539000_ocv113-B26","first-page":"4","article-title":"LIBLINEAR: A Library for Large Linear Classication","volume":"9","author":"Fan","year":"2008","journal-title":"J Mach Learn Res"},{"issue":"5","key":"2020110612591539000_ocv113-B27","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1136\/amiajnl-2012-001607","article-title":"An end-to-end system to identify temporal relation in discharge summaries: 2012 i2b2 challenge","volume":"20","author":"Xu","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612591539000_ocv113-B28","doi-asserted-by":"crossref","DOI":"10.1136\/amiajnl-2013-001635","article-title":"A hybrid system for temporal information extraction from clinical text","author":"Tang","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612591539000_ocv113-B29","doi-asserted-by":"crossref","first-page":"223","DOI":"10.1007\/978-1-60327-241-4_13","article-title":"A user's guide to support vector machines","volume":"609","author":"Ben-Hur","year":"2010","journal-title":"Methods Mol Biol"},{"key":"2020110612591539000_ocv113-B30","author":"Unified Medical Language System (UMLS)"},{"key":"2020110612591539000_ocv113-B31","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1162\/tacl_a_00172","article-title":"Temporal annotation in the clinical domain","volume":"2","author":"Styler","year":"2014","journal-title":"Trans Assoc Comput Linguist"},{"key":"2020110612591539000_ocv113-B32","article-title":"Discovering body site and severity modifiers in clinical texts","author":"Dligach","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2020110612591539000_ocv113-B33","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612591539000_ocv113-B34","author":"Apache Clinical Text Analysis and Knowledge Extraction System (cTAKES)"},{"key":"2020110612591539000_ocv113-B35","article-title":"Temporal evaluation","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2","author":"UzZaman","year":"2011"},{"issue":"7590","key":"2020110612591539000_ocv113-B36","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1016\/S0140-6736(69)91330-0","article-title":"The signed-rank (Wilcoxon) test","volume":"1","author":"Miller","year":"1969","journal-title":"Lancet"},{"key":"2020110612591539000_ocv113-B37","article-title":"Automatic identification of methotrexate-induced liver toxicity in patients with rheumatoid arthritis from the electronic medical record","author":"Lin","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612591539000_ocv113-B38","first-page":"328","article-title":"TN-TIES: A system for extracting temporal information from emergency department triage notes","author":"Irvine","year":"2008","journal-title":"AMIA Annu Symp Proc"},{"key":"2020110612591539000_ocv113-B39","first-page":"ocu041","article-title":"Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features","author":"Nikfarjam","year":"2015","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"2020110612591539000_ocv113-B40","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1111\/rssb.12060","article-title":"Joint modelling of repeated measurements and time-to-event outcomes: flexible model specification and exact likelihood inference","volume":"77","author":"Barrett","year":"2015","journal-title":"J R Stat Soc Series B Stat Methodol"},{"key":"2020110612591539000_ocv113-B41","article-title":"Bayesian inference for joint modelling of longitudinal continuous, binary and ordinal events","author":"Li","year":"2014","journal-title":"Stat Methods Med Res"},{"issue":"2","key":"2020110612591539000_ocv113-B42","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1093\/biostatistics\/kxm029","article-title":"Joint inference for nonlinear mixed-effects models and time to event at the presence of missing data","volume":"9","author":"Wu","year":"2008","journal-title":"Biostatistics"},{"key":"2020110612591539000_ocv113-B43","first-page":"81","article-title":"Descending-path convolution Kernel for syntactic structures","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Lin","year":"2014"},{"key":"2020110612591539000_ocv113-B44","article-title":"Discovering temporal narrative containers in clinical text","volume-title":"Proceedings of the 2013 Workshop on Biomedical Natural Language Processing","author":"Miller","year":"2013"},{"key":"2020110612591539000_ocv113-B45","article-title":"Convolution kernels for natural language","volume-title":"Advances in Neural Information Processing Systems","author":"Collins","year":"2001"},{"key":"2020110612591539000_ocv113-B46","volume-title":"Convolution Kernels on Discrete Structures","author":"Haussler","year":"1999"},{"key":"2020110612591539000_ocv113-B47","article-title":"A study on dependency tree kernels for automatic extraction of protein-protein interaction","volume-title":"Proceedings of BioNLP 2011 Workshop","author":"Chowdhury","year":"2011"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/2\/387\/34147812\/ocv113.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/2\/387\/34147812\/ocv113.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:36:49Z","timestamp":1604687809000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/23\/2\/387\/2572466"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,10,31]]},"references-count":47,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2015,10,31]]},"published-print":{"date-parts":[[2016,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocv113","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2016,3]]},"published":{"date-parts":[[2015,10,31]]}}}