{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T18:13:58Z","timestamp":1783361638042,"version":"3.54.6"},"reference-count":66,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T00:00:00Z","timestamp":1783296000000},"content-version":"vor","delay-in-days":186,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We present an experimental recipe for studying the relationship between training data and language model (LM) behavior. We outline steps for intervening on data batches \u2013 i.e., \u201crewriting history\u201d \u2013 and then retraining model checkpoints over that data to test hypotheses relating data to behavior. Our intervention recipe\u2019s stages are (1) selecting evaluation items from a benchmark that measures model behavior, (2) matching relevant documents to those items, and (3) modifying those documents before retraining and measuring the effects. We demonstrate the utility of our recipe through case studies on factual knowledge acquisition and gender bias in LMs, using both cooccurrence statistics and information retrieval methods to identify documents that might contribute to model behavior. Our results supplement past observational analyses that link cooccurrence to model behavior, while demonstrating that extant methods for identifying relevant training documents do not fully explain an LM\u2019s abilities and biases. Researchers can follow the recipe to test further hypotheses about how training data affects model behavior. Our code is made publicly available to promote future work.1<\/jats:p>","DOI":"10.1162\/tacl.a.740","type":"journal-article","created":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T17:14:39Z","timestamp":1783358079000},"page":"1541-1567","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior"],"prefix":"10.1162","volume":"14","author":[{"given":"Rahul","family":"Nadkarni","sequence":"first","affiliation":[{"name":"Paul G. Allen School of Computer Science & Engineering, University of Washington, rahuln@cs.washington.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanai","family":"Elazar","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Bar-Ilan University, yanaiela@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hila","family":"Gonen","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of British Columbia, hilagnn@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Noah A.","family":"Smith","sequence":"additional","affiliation":[{"name":"Paul G. Allen School of Computer Science & Engineering, University of Washington"},{"name":"Allen Institute for Artificial Intelligence, nasmith@cs.washington.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,7,1]]},"reference":[{"key":"2026070613143212600_bib1","doi-asserted-by":"crossref","first-page":"2429","DOI":"10.18653\/v1\/2022.findings-emnlp.180","article-title":"Towards Tracing Knowledge in Language Models Back to the Training Data","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022","author":"Akyurek","year":"2022"},{"key":"2026070613143212600_bib2","first-page":"1067","article-title":"Physics of Language Models: Part 3.1, Knowledge Storage and Extraction","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Allen-Zhu","year":"2024"},{"key":"2026070613143212600_bib3","doi-asserted-by":"crossref","DOI":"10.2139\/ssrn.5250621","article-title":"Physics of Language Models: Part 3.2, Knowledge Manipulation","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Allen-Zhu","year":"2025"},{"issue":"4","key":"2026070613143212600_bib4","first-page":"335","article-title":"Distantly supervised Web relation extraction for knowledge base population","volume":"7","author":"Augenstein","year":"2016","journal-title":"Semantic Web"},{"key":"2026070613143212600_bib5","article-title":"Establishing Task Scaling Laws via Compute-Efficient Model Ladders","volume-title":"Second Conference on Language Modeling","author":"Bhagia","year":"2025"},{"key":"2026070613143212600_bib6","first-page":"2397","article-title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","volume-title":"Proceedings of the 40th International Conference on Machine Learning","author":"Biderman","year":"2023"},{"key":"2026070613143212600_bib7","first-page":"7","article-title":"Identifying and Reducing Gender Bias in WordLevel Language Models","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop","author":"Bordia","year":"2019"},{"key":"2026070613143212600_bib8","doi-asserted-by":"crossref","first-page":"9263","DOI":"10.18653\/v1\/2020.emnlp-main.745","article-title":"With Little Power Comes Great Responsibility","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Card","year":"2020"},{"key":"2026070613143212600_bib9","doi-asserted-by":"crossref","DOI":"10.52202\/079017-1939","article-title":"How Do Large Language Models Acquire Factual Knowledge During Pretraining?","volume-title":"Advances in Neural Information Processing Systems","author":"Chang","year":"2024"},{"key":"2026070613143212600_bib10","article-title":"Scalable Influence and Fact Tracing for Large Language Model Pretraining","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Chang","year":"2025"},{"issue":"70","key":"2026070613143212600_bib11","first-page":"1","article-title":"Scaling Instruction-Finetuned Language Models","volume":"25","author":"Chung","year":"2024","journal-title":"Journal of Machine Learning Research"},{"key":"2026070613143212600_bib12","volume-title":"coastal-cph\/pararel_patterns","author":"CoAStaL NLP Group","year":"2024"},{"key":"2026070613143212600_bib13","article-title":"What\u2019s In My Big Data?","volume-title":"The Twelfth International Conference on Learning Representations","author":"Elazar","year":"2024"},{"key":"2026070613143212600_bib14","article-title":"Measuring Causal Effects of Data Statistics on Language Model\u2019s \u2018Factual\u2019 Predictions","volume-title":"arXiv preprint arXiv:2207.14251v2","author":"Elazar","year":"2022"},{"key":"2026070613143212600_bib15","doi-asserted-by":"crossref","first-page":"1012","DOI":"10.1162\/tacl_a_00410","article-title":"Measuring and Improving Consistency in Pretrained Language Models","volume":"9","author":"Elazar","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613143212600_bib16","doi-asserted-by":"crossref","DOI":"10.63317\/2yu5nu76n7bu","article-title":"T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","author":"Elsahar","year":"2018"},{"key":"2026070613143212600_bib17","doi-asserted-by":"crossref","DOI":"10.1162\/tacl_a_00511","article-title":"Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond","volume":"10","author":"Feder","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613143212600_bib18","article-title":"The Llama 3 Herd of Models","volume-title":"arXiv preprint arXiv:2407.21783v3","author":"Grattafiori","year":"2024"},{"key":"2026070613143212600_bib19","doi-asserted-by":"crossref","first-page":"15789","DOI":"10.18653\/v1\/2024.acl-long.841","article-title":"OLMo: Accelerating the Science of Language Models","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Groeneveld","year":"2024"},{"key":"2026070613143212600_bib20","article-title":"Studying Large Language Model Generalization with Influence Functions","volume-title":"arXiv preprint arXiv:2308.03296v1","author":"Grosse","year":"2023"},{"key":"2026070613143212600_bib21","doi-asserted-by":"crossref","first-page":"5005","DOI":"10.18653\/v1\/2025.findings-naacl.282","article-title":"OLMES: A Standard for Language Model Evaluations","volume-title":"Findings of the Association for Computational Linguistics: NAACL 2025","author":"Gu","year":"2025"},{"key":"2026070613143212600_bib22","doi-asserted-by":"crossref","first-page":"8342","DOI":"10.18653\/v1\/2020.acl-main.740","article-title":"Don\u2019t Stop Pretraining: Adapt Language Models to Domains and Tasks","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Gururangan","year":"2020"},{"key":"2026070613143212600_bib23","doi-asserted-by":"crossref","first-page":"2351","DOI":"10.1007\/s10994-023-06495-7","article-title":"Training Data Influence Analysis and Estimation: A Survey","volume":"113","author":"Hammoudeh","year":"2022","journal-title":"Mach. Learn."},{"key":"2026070613143212600_bib24","doi-asserted-by":"crossref","first-page":"12660","DOI":"10.18653\/v1\/2023.acl-long.708","article-title":"Understanding In-Context Learning via Supportive Pretraining Data","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Han","year":"2023"},{"key":"2026070613143212600_bib25","article-title":"ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data","volume-title":"arXiv preprint arXiv:2205.12600v1","author":"Han","year":"2022"},{"key":"2026070613143212600_bib26","doi-asserted-by":"crossref","first-page":"5553","DOI":"10.18653\/v1\/2020.acl-main.492","article-title":"Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Han","year":"2020"},{"key":"2026070613143212600_bib27","article-title":"Measuring Massive Multitask Language Understanding","volume-title":"International Conference on Learning Representations","author":"Hendrycks","year":"2021"},{"key":"2026070613143212600_bib28","volume-title":"spaCy: Industrial-strength Natural Language Processing in Python","author":"Honnibal","year":"2020"},{"key":"2026070613143212600_bib29","doi-asserted-by":"crossref","first-page":"9036","DOI":"10.18653\/v1\/2023.findings-acl.576","article-title":"Data-Efficient Finetuning Using Cross-Task Nearest Neighbors","volume-title":"Findings of the Association for Computational Linguistics: ACL 2023","author":"Ivison","year":"2023"},{"key":"2026070613143212600_bib30","first-page":"15696","article-title":"Large Language Models Struggle to Learn Long-Tail Knowledge","volume-title":"Proceedings of the 40th International Conference on Machine Learning","author":"Kandpal","year":"2023"},{"key":"2026070613143212600_bib31","doi-asserted-by":"crossref","first-page":"7721","DOI":"10.18653\/v1\/2023.findings-emnlp.518","article-title":"Impact of Co-occurrence on Factual Knowledge of Large Language Models","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2023","author":"Kang","year":"2023"},{"key":"2026070613143212600_bib32","doi-asserted-by":"crossref","first-page":"6769","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense Passage Retrieval for Open-Domain Question Answering","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Karpukhin","year":"2020"},{"key":"2026070613143212600_bib33","doi-asserted-by":"crossref","first-page":"552","DOI":"10.18653\/v1\/2020.conll-1.45","article-title":"Are Pretrained Language Models Symbolic Reasoners over Knowledge?","volume-title":"Proceedings of the 24th Conference on Computational Natural Language Learning","author":"Kassner","year":"2020"},{"key":"2026070613143212600_bib34","article-title":"Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Kim","year":"2025"},{"key":"2026070613143212600_bib35","first-page":"1885","article-title":"Understanding Black-box Predictions via Influence Functions","volume-title":"Proceedings of the 34th International Conference on Machine Learning","author":"Koh","year":"2017"},{"key":"2026070613143212600_bib36","doi-asserted-by":"crossref","first-page":"3045","DOI":"10.18653\/v1\/2021.emnlp-main.243","article-title":"The Power of Scale for Parameter-Efficient Prompt Tuning","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Lester","year":"2021"},{"key":"2026070613143212600_bib37","doi-asserted-by":"crossref","DOI":"10.52202\/079017-0455","article-title":"DataComp-LM: In search of the next generation of training sets for language models","volume-title":"The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track","author":"Li","year":"2024"},{"key":"2026070613143212600_bib38","doi-asserted-by":"crossref","first-page":"3245","DOI":"10.18653\/v1\/2024.naacl-long.179","article-title":"A Pretrainer\u2019s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Longpre","year":"2024"},{"key":"2026070613143212600_bib39","article-title":"DataDecide: How to Predict Best Pretraining Data with Small Experiments","volume-title":"Forty-second International Conference on Machine Learning","author":"Magnusson","year":"2025"},{"key":"2026070613143212600_bib40","doi-asserted-by":"crossref","first-page":"9802","DOI":"10.18653\/v1\/2023.acl-long.546","article-title":"When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Mallen","year":"2023"},{"key":"2026070613143212600_bib41","article-title":"On Linear Representations and Pretraining Data Frequency in Language Models","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Merullo","year":"2025"},{"key":"2026070613143212600_bib42","doi-asserted-by":"crossref","first-page":"2381","DOI":"10.18653\/v1\/D18-1260","article-title":"Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Mihaylov","year":"2018"},{"key":"2026070613143212600_bib43","first-page":"1003","article-title":"Distant supervision for relation extraction without labeled data","volume-title":"Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP","author":"Mintz","year":"2009"},{"key":"2026070613143212600_bib44","first-page":"5356","article-title":"StereoSet: Measuring stereotypical bias in pretrained language models","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Nadeem","year":"2021"},{"issue":"2","key":"2026070613143212600_bib45","doi-asserted-by":"crossref","DOI":"10.1145\/3597307","article-title":"Biases in Large Language Models: Origins, Inventory, and Discussion","volume":"15","author":"Navigli","year":"2023","journal-title":"J. Data and Information Quality"},{"key":"2026070613143212600_bib46","volume-title":"Introducing GPT-5","author":"OpenAI","year":"2025"},{"key":"2026070613143212600_bib47","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511803161","volume-title":"Causality: Models, Reasoning, and Inference","author":"Pearl","year":"2009","edition":"2"},{"key":"2026070613143212600_bib48","doi-asserted-by":"crossref","first-page":"2463","DOI":"10.18653\/v1\/D19-1250","article-title":"Language Models as Knowledge Bases?","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Petroni","year":"2019"},{"key":"2026070613143212600_bib49","first-page":"4550","article-title":"Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Porada","year":"2022"},{"key":"2026070613143212600_bib50","article-title":"Backtracking Mathematical Reasoning of Language Models to the Pretraining Data","volume-title":"NeurIPS Workshop on Attributing Model Behavior at Scale","author":"Razeghi","year":"2023"},{"key":"2026070613143212600_bib51","doi-asserted-by":"crossref","first-page":"840","DOI":"10.18653\/v1\/2022.findings-emnlp.59","article-title":"Impact of Pre-training Term Frequencies on Few-Shot Numerical Reasoning","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022","author":"Razeghi","year":"2022"},{"key":"2026070613143212600_bib52","doi-asserted-by":"crossref","first-page":"5418","DOI":"10.18653\/v1\/2020.emnlp-main.437","article-title":"How Much Knowledge Can You Pack Into the Parameters of a Language Model?","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Roberts","year":"2020"},{"key":"2026070613143212600_bib53","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Rudinger","year":"2018"},{"key":"2026070613143212600_bib54","doi-asserted-by":"crossref","first-page":"6367","DOI":"10.18653\/v1\/2024.naacl-long.353","article-title":"The Bias Amplification Paradox in Text-to-Image Generation","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Seshadri","year":"2024"},{"key":"2026070613143212600_bib55","first-page":"75","article-title":"The Birth of Bias: A case study on the evolution of gender bias in an English language model","volume-title":"Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)","author":"Wal","year":"2022"},{"key":"2026070613143212600_bib56","article-title":"How Many Images Does It Take? Estimating Imitation Thresholds in Text-to-Image Models","volume-title":"Transactions on Machine Learning Research","author":"Verma","year":"2025"},{"key":"2026070613143212600_bib57","article-title":"2 OLMo 2 Furious (COLM\u2019s Version)","volume-title":"Second Conference on Language Modeling","author":"Walsh","year":"2025"},{"key":"2026070613143212600_bib58","article-title":"Generalization v.s. Memorization: Tracing Language Models\u2019 Capabilities Back to Pretraining Data","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Wang","year":"2025"},{"key":"2026070613143212600_bib59","doi-asserted-by":"crossref","first-page":"932","DOI":"10.18653\/v1\/2021.emnlp-main.72","article-title":"Frequency Effects on Syntactic Rule Learning in Transformers","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Wei","year":"2021"},{"key":"2026070613143212600_bib60","doi-asserted-by":"crossref","first-page":"94","DOI":"10.18653\/v1\/W17-4413","article-title":"Crowdsourcing Multiple Choice Science Questions","volume-title":"Proceedings of the 3rd Workshop on Noisy User-generated Text","author":"Welbl","year":"2017"},{"key":"2026070613143212600_bib61","first-page":"38","article-title":"Transformers: State-of-the-Art Natural Language Processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2026070613143212600_bib62","first-page":"54104","article-title":"LESS: Selecting Influential Data for Targeted Instruction Tuning","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Xia","year":"2024"},{"key":"2026070613143212600_bib63","doi-asserted-by":"crossref","first-page":"11295","DOI":"10.18653\/v1\/2023.emnlp-main.695","article-title":"Data Similarity is Not Enough to Explain Language Model Performance","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Yauney","year":"2023"},{"key":"2026070613143212600_bib64","first-page":"64889","article-title":"Cooccurrence is not Factual Association in Language Models","volume-title":"Advances in Neural Information Processing Systems","author":"Zhang","year":"2024"},{"key":"2026070613143212600_bib65","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/N18-2003","article-title":"Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2","author":"Zhao","year":"2018"},{"key":"2026070613143212600_bib66","article-title":"How do language models learn facts? Dynamics, curricula and hallucinations","volume-title":"Second Conference on Language Modeling","author":"Zucchet","year":"2025"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.740\/2612292\/tacl.a.740.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.740\/2612292\/tacl.a.740.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T17:14:53Z","timestamp":1783358093000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.740\/137427\/Rewriting-History-A-Recipe-for-Interventional"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":66,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.740","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}