{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T02:41:59Z","timestamp":1783737719601,"version":"3.55.0"},"reference-count":48,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2022,3,21]],"date-time":"2022-03-21T00:00:00Z","timestamp":1647820800000},"content-version":"vor","delay-in-days":79,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,3,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum\u2014those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently \u201crefreshed\u201d as new data arrives, without the need for retraining from scratch.<\/jats:p>","DOI":"10.1162\/tacl_a_00459","type":"journal-article","created":{"date-parts":[[2022,3,21]],"date-time":"2022-03-21T19:09:52Z","timestamp":1647889792000},"page":"257-273","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":77,"title":["Time-Aware Language Models as Temporal Knowledge Bases"],"prefix":"10.1162","volume":"10","author":[{"given":"Bhuwan","family":"Dhingra","sequence":"first","affiliation":[{"name":"Google Research. bdhingra@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeremy R.","family":"Cole","sequence":"additional","affiliation":[{"name":"Google Research. jrcole@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julian Martin","family":"Eisenschlos","sequence":"additional","affiliation":[{"name":"Google Research. eisenjulian@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Gillick","sequence":"additional","affiliation":[{"name":"Google Research. dgillick@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacob","family":"Eisenstein","sequence":"additional","affiliation":[{"name":"Google Research. jeisenstein@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William W.","family":"Cohen","sequence":"additional","affiliation":[{"name":"Google Research. wcohen@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,3,18]]},"reference":[{"key":"2022033118512799900_bib1","article-title":"Towards a human-like open-domain chatbot","volume":"abs\/2001.09977","author":"Adiwardana","year":"2020","journal-title":"CoRR"},{"key":"2022033118512799900_bib2","first-page":"380","article-title":"Dynamic word embeddings","volume-title":"International Conference on Machine Learning","author":"Bamler","year":"2017"},{"key":"2022033118512799900_bib3","doi-asserted-by":"publisher","first-page":"6491","DOI":"10.18653\/v1\/2021.emnlp-main.522","article-title":"Editing factual knowledge in language models","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Cao","year":"2021"},{"key":"2022033118512799900_bib4","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1109\/ICDM.2019.00022","article-title":"Learning dynamic author representations with temporal language models","volume-title":"2019 IEEE International Conference on Data Mining (ICDM)","author":"Delasalles","year":"2019"},{"key":"2022033118512799900_bib5","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2022033118512799900_bib6","doi-asserted-by":"publisher","first-page":"457","DOI":"10.18653\/v1\/P19-1044","article-title":"Time-out: Temporal referencing for robust modeling of lexical semantic change","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Dubossarsky","year":"2019"},{"issue":"12","key":"2022033118512799900_bib7","doi-asserted-by":"publisher","DOI":"10.3390\/app10124180","article-title":"Time of your hate: The challenge of time in hate speech detection on social media","volume":"10","author":"Florio","year":"2020","journal-title":"Applied Sciences"},{"key":"2022033118512799900_bib8","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1162\/tacl_a_00081","article-title":"A Bayesian model of diachronic meaning change","volume":"4","author":"Frermann","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022033118512799900_bib9","doi-asserted-by":"publisher","first-page":"3960","DOI":"10.18653\/v1\/2020.acl-main.365","article-title":"Analysing lexical semantic change with contextualised word representations","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Giulianelli","year":"2020"},{"key":"2022033118512799900_bib10","first-page":"3929","article-title":"Retrieval augmented language model pre-training","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Guu","year":"2020"},{"key":"2022033118512799900_bib11","doi-asserted-by":"publisher","first-page":"1489","DOI":"10.18653\/v1\/P16-1141","article-title":"Diachronic word embeddings reveal statistical laws of semantic change","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Hamilton","year":"2016"},{"key":"2022033118512799900_bib12","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467162","article-title":"Dynamic language models for continuously evolving content","volume-title":"Knowledge Discovery and Data Mining (KDD)","author":"Hombaiah","year":"2021"},{"key":"2022033118512799900_bib13","doi-asserted-by":"publisher","first-page":"694","DOI":"10.18653\/v1\/P18-2110","article-title":"Examining temporality in document classification","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Huang","year":"2018"},{"key":"2022033118512799900_bib14","doi-asserted-by":"publisher","first-page":"4113","DOI":"10.18653\/v1\/P19-1403","article-title":"Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Huang","year":"2019"},{"key":"2022033118512799900_bib15","article-title":"Cutting down on prompts and parameters: Simple few-shot learning with language models","author":"Logan","year":"2021","journal-title":"CoRR"},{"key":"2022033118512799900_bib16","doi-asserted-by":"publisher","first-page":"195","DOI":"10.18653\/v1\/P18-2032","article-title":"Diachronic degradation of language models: Insights from social media","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Jaidka","year":"2018"},{"key":"2022033118512799900_bib17","doi-asserted-by":"publisher","first-page":"1057","DOI":"10.1145\/3184558.3191536","article-title":"Tempquestions: A benchmark for temporal question answering","volume-title":"Companion Proceedings of the The Web Conference 2018","author":"Jia","year":"2018"},{"key":"2022033118512799900_bib18","doi-asserted-by":"publisher","first-page":"5943","DOI":"10.18653\/v1\/2020.emnlp-main.479","article-title":"X-FACTR: Multilingual factual knowledge retrieval from pretrained language models","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Jiang","year":"2020"},{"key":"2022033118512799900_bib19","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1162\/tacl_a_00324","article-title":"How can we know what language models know?","volume":"8","author":"Jiang","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022033118512799900_bib20","doi-asserted-by":"publisher","first-page":"3250","DOI":"10.18653\/v1\/2021.eacl-main.284","article-title":"Multilingual LAMA: Investigating knowledge in multilingual pretrained language models","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Kassner","year":"2021"},{"key":"2022033118512799900_bib21","first-page":"2766","article-title":"Dynamic evaluation of neural sequence models","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Krause","year":"2018"},{"key":"2022033118512799900_bib22","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1162\/tacl_a_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022033118512799900_bib23","article-title":"Mind the gap: Assessing temporal generalization in neural language models","volume":"34","author":"Lazaridou","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2022033118512799900_bib24","first-page":"1268","article-title":"Discovering biased news articles leveraging multiple human annotations","volume-title":"Proceedings of the 12th Language Resources and Evaluation Conference","author":"Lazaridou","year":"2020"},{"key":"2022033118512799900_bib25","doi-asserted-by":"crossref","first-page":"36","DOI":"10.18653\/v1\/2020.fever-1.5","article-title":"Language models as fact checkers?","volume-title":"Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER)","author":"Lee","year":"2020"},{"key":"2022033118512799900_bib26","doi-asserted-by":"publisher","first-page":"1000","DOI":"10.18653\/v1\/2021.eacl-main.86","article-title":"Question and answer test-train overlap in open-domain question answering datasets","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Lewis","year":"2021"},{"key":"2022033118512799900_bib27","doi-asserted-by":"publisher","first-page":"1076","DOI":"10.1145\/3447548.3467392","article-title":"Newsembed: Modeling news through pre-trained document representations","volume-title":"Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining","author":"Liu","year":"2021"},{"key":"2022033118512799900_bib28","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"CoRR"},{"key":"2022033118512799900_bib29","doi-asserted-by":"publisher","first-page":"65","DOI":"10.18653\/v1\/W18-6210","article-title":"Sentiment analysis under temporal shift","volume-title":"Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis","author":"Lukes","year":"2018"},{"key":"2022033118512799900_bib30","doi-asserted-by":"publisher","first-page":"1158","DOI":"10.18653\/v1\/2020.emnlp-main.88","article-title":"TORQUE: A reading comprehension dataset of temporal ordering questions","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Ning","year":"2020"},{"key":"2022033118512799900_bib31","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word representations","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Peters","year":"2018"},{"key":"2022033118512799900_bib32","doi-asserted-by":"crossref","first-page":"2463","DOI":"10.18653\/v1\/D19-1250","article-title":"Language models as knowledge bases?","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Petroni","year":"2019"},{"key":"2022033118512799900_bib33","article-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2019"},{"issue":"140","key":"2022033118512799900_bib34","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2022033118512799900_bib35","doi-asserted-by":"publisher","first-page":"784","DOI":"10.18653\/v1\/P18-2124","article-title":"Know what you don\u2019t know: Unanswerable questions for SQuAD","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Rajpurkar","year":"2018"},{"key":"2022033118512799900_bib36","article-title":"SLING: A framework for frame semantic parsing","author":"Ringgaard","year":"2017","journal-title":"CoRR"},{"key":"2022033118512799900_bib37","doi-asserted-by":"publisher","first-page":"5418","DOI":"10.18653\/v1\/2020.emnlp-main.437","article-title":"How much knowledge can you pack into the parameters of a language model?","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Roberts","year":"2020"},{"key":"2022033118512799900_bib38","doi-asserted-by":"publisher","first-page":"2400","DOI":"10.18653\/v1\/2021.findings-emnlp.206","article-title":"Temporal adaptation of BERT and performance on downstream document classification: Insights from social media","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"R\u00f6ttger","year":"2021"},{"key":"2022033118512799900_bib39","doi-asserted-by":"publisher","first-page":"6663","DOI":"10.18653\/v1\/2021.acl-long.520","article-title":"Question answering over temporal knowledge graphs","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Saxena","year":"2021"},{"key":"2022033118512799900_bib40","article-title":"Editable neural networks","volume-title":"International Conference on Learning Representations","author":"Sinitsin","year":"2020"},{"key":"2022033118512799900_bib41","doi-asserted-by":"publisher","first-page":"3645","DOI":"10.18653\/v1\/P19-1355","article-title":"Energy and policy considerations for deep learning in NLP","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Strubell","year":"2019"},{"key":"2022033118512799900_bib42","doi-asserted-by":"publisher","first-page":"142","DOI":"10.3115\/1119176.1119195","article-title":"Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition","volume-title":"Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003","author":"Sang","year":"2003"},{"key":"2022033118512799900_bib43","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"issue":"10","key":"2022033118512799900_bib44","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1145\/2629489","article-title":"Wikidata: A free collaborative knowledgebase","volume":"57","author":"Vrande\u010di\u0107","year":"2014","journal-title":"Commun. ACM"},{"key":"2022033118512799900_bib45","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1145\/2064448.2064475","article-title":"Understanding semantic change of words over centuries","volume-title":"Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural DiversiTy on the Social Web","author":"Wijaya","year":"2011"},{"key":"2022033118512799900_bib46","doi-asserted-by":"publisher","first-page":"3363","DOI":"10.18653\/v1\/D19-1332","article-title":"\u201cgoing on a vacation\u201d takes longer than \u201cgoing for a walk\u201d: A study of temporal commonsense understanding","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Zhou","year":"2019"},{"key":"2022033118512799900_bib47","doi-asserted-by":"publisher","first-page":"1361","DOI":"10.18653\/v1\/2021.naacl-main.107","article-title":"Temporal reasoning on implicit events from distant supervision","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Zhou","year":"2021"},{"key":"2022033118512799900_bib48","article-title":"Modifying memories in transformer models","author":"Zhu","year":"2020","journal-title":"CoRR"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00459\/2004543\/tacl_a_00459.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00459\/2004543\/tacl_a_00459.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,3,31]],"date-time":"2022-03-31T23:52:59Z","timestamp":1648770779000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00459\/110012\/Time-Aware-Language-Models-as-Temporal-Knowledge"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":48,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00459","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}