{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T13:44:00Z","timestamp":1750340640925},"reference-count":52,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T00:00:00Z","timestamp":1699401600000},"content-version":"vor","delay-in-days":311,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,11,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The events in a narrative are understood as a coherent whole via the underlying states of their participants. Often, these participant states are not explicitly mentioned, instead left to be inferred by the reader. A model that understands narratives should likewise infer these implicit states, and even reason about the impact of changes to these states on the narrative. To facilitate this goal, we introduce a new crowdsourced English-language, Participant States dataset, PASTA. This dataset contains inferable participant states; a counterfactual perturbation to each state; and the changes to the story that would be necessary if the counterfactual were true. We introduce three state-based reasoning tasks that test for the ability to infer when a state is entailed by a story, to revise a story conditioned on a counterfactual state, and to explain the most likely state change given a revised story. Experiments show that today\u2019s LLMs can reason about states to some degree, but there is large room for improvement, especially in problems requiring access and ability to reason with diverse types of knowledge (e.g., physical, numerical, factual).1<\/jats:p>","DOI":"10.1162\/tacl_a_00600","type":"journal-article","created":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T15:07:46Z","timestamp":1699456066000},"page":"1283-1300","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["<tt>PASTA<\/tt>: A Dataset for Modeling PArticipant STAtes in Narratives"],"prefix":"10.1162","volume":"11","author":[{"given":"Sayontan","family":"Ghosh","sequence":"first","affiliation":[{"name":"Stony Brook University, USA. sagghosh@cs.stonybrook.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mahnaz","family":"Koupaee","sequence":"additional","affiliation":[{"name":"Stony Brook University, USA. mkoupaee@cs.stonybrook.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Isabella","family":"Chen","sequence":"additional","affiliation":[{"name":"Stony Brook University, USA. isabellachenusa@gmail.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francis","family":"Ferraro","sequence":"additional","affiliation":[{"name":"University of Maryland, Baltimore County, USA. ferraro@umbc.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nathanael","family":"Chambers","sequence":"additional","affiliation":[{"name":"United States Naval Academy, USA. nchamber@usna.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Niranjan","family":"Balasubramanian","sequence":"additional","affiliation":[{"name":"Stony Brook University, USA. niranjan@cs.stonybrook.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,11,2]]},"reference":[{"key":"2023110815073941300_bib1","first-page":"1721","article-title":"Generating coherent event schemas at scale","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Balasubramanian","year":"2013"},{"key":"2023110815073941300_bib2","article-title":"Abductive commonsense reasoning","author":"Bhagavatula","year":"2019","journal-title":"arXiv preprint arXiv:1908.05739"},{"key":"2023110815073941300_bib3","article-title":"Simulating action dynamics with neural process networks","volume":"abs\/1711.05313","author":"Bosselut","year":"2018","journal-title":"ArXiv"},{"key":"2023110815073941300_bib4","first-page":"588","article-title":"Cue me in: Content-inducing approaches to interactive story generation","volume-title":"Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing","author":"Brahman","year":"2020"},{"issue":"9","key":"2023110815073941300_bib5","doi-asserted-by":"publisher","first-page":"1109","DOI":"10.1080\/17512786.2017.1363657","article-title":"How journalists and social media users perceive online fact-checking and verification services","volume":"12","author":"Brandtzaeg","year":"2018","journal-title":"Journalism Practice"},{"key":"2023110815073941300_bib6","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"5","key":"2023110815073941300_bib7","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1016\/0895-4356(93)90018-V","article-title":"Bias, prevalence and kappa","volume":"46","author":"Byrt","year":"1993","journal-title":"Journal of Clinical Epidemiology"},{"key":"2023110815073941300_bib8","first-page":"789","article-title":"Unsupervised learning of narrative event chains","volume-title":"Proceedings of ACL-08: HLT","author":"Chambers","year":"2008"},{"key":"2023110815073941300_bib9","doi-asserted-by":"publisher","first-page":"602","DOI":"10.3115\/1690219.1690231","article-title":"Unsupervised learning of narrative schemas and their participants","volume-title":"Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP","author":"Chambers","year":"2009"},{"key":"2023110815073941300_bib10","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1144","article-title":"Tracking state changes in procedural text: A challenge dataset and models for process paragraph comprehension","volume-title":"NAACL","author":"Dalvi","year":"2018"},{"key":"2023110815073941300_bib11","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"issue":"6","key":"2023110815073941300_bib12","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1016\/0895-4356(90)90158-L","article-title":"High agreement but low kappa: I. The problems of two paradoxes","volume":"43","author":"Feinstein","year":"1990","journal-title":"Journal of Clinical Epidemiology"},{"key":"2023110815073941300_bib13","doi-asserted-by":"publisher","first-page":"2601","DOI":"10.1609\/aaai.v30i1.10328","article-title":"A unified bayesian model of scripts, frames and language","volume-title":"Proceedings of the 30th Conference on Artificial Intelligence (AAAI)","author":"Ferraro","year":"2016"},{"key":"2023110815073941300_bib14","doi-asserted-by":"publisher","first-page":"817","DOI":"10.1016\/j.procs.2017.11.106","article-title":"The current state of fake news: Challenges and opportunities","volume":"121","author":"Figueira","year":"2017","journal-title":"Procedia Computer Science"},{"issue":"2","key":"2023110815073941300_bib15","first-page":"222","article-title":"Frames and the semantics of understanding","volume":"6","author":"Fillmore","year":"1985","journal-title":"Quaderni di Semantica"},{"issue":"2\u20133","key":"2023110815073941300_bib16","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/0004-3702(90)90053-3","article-title":"A critical examination of allen\u2019s theory of action and time","volume":"42","author":"Galton","year":"1990","journal-title":"Artificial Intelligence"},{"key":"2023110815073941300_bib17","article-title":"Automated fake news detection using cross-checking with reliable sources","author":"Ghadiri","year":"2022","journal-title":"arXiv preprint arXiv: 2201.00083"},{"key":"2023110815073941300_bib18","doi-asserted-by":"publisher","first-page":"89","DOI":"10.18653\/v1\/N19-4016","article-title":"Plan, write, and revise: An interactive system for open-domain story generation","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)","author":"Goldfarb-Tarrant","year":"2019"},{"key":"2023110815073941300_bib19","volume-title":"Handbook of Inter-rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters","author":"Gwet","year":"2014"},{"issue":"1","key":"2023110815073941300_bib20","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","year":"2008","journal-title":"British Journal of Mathematical and Statistical Psychology"},{"key":"2023110815073941300_bib21","doi-asserted-by":"publisher","first-page":"103","DOI":"10.18653\/v1\/W18-5516","article-title":"UKP-athene: Multi-sentence textual entailment for claim verification","volume-title":"Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)","author":"Hanselowski","year":"2018"},{"key":"2023110815073941300_bib22","doi-asserted-by":"publisher","first-page":"905","DOI":"10.3115\/1220175.1220289","article-title":"Methods for using textual entailment in open-domain question answering","volume-title":"Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics","author":"Harabagiu","year":"2006"},{"key":"2023110815073941300_bib23","doi-asserted-by":"publisher","first-page":"9106","DOI":"10.18653\/v1\/2020.emnlp-main.732","article-title":"An analysis of natural language inference benchmarks through the lens of negation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Md","year":"2020"},{"key":"2023110815073941300_bib24","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i7.16792","article-title":"Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs","volume-title":"AAAI","author":"Hwang","year":"2021"},{"key":"2023110815073941300_bib25","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2023110815073941300_bib26","doi-asserted-by":"publisher","first-page":"100","DOI":"10.18653\/v1\/2022.deelio-1.10","article-title":"What makes good in-context examples for GPT-3?","volume-title":"Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures","author":"Liu","year":"2022"},{"key":"2023110815073941300_bib27","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"ArXiv"},{"key":"2023110815073941300_bib28","article-title":"Decoupled weight decay regularization","author":"Loshchilov","year":"2017","journal-title":"arXiv preprint arXiv:1711.05101"},{"key":"2023110815073941300_bib29","doi-asserted-by":"publisher","first-page":"839","DOI":"10.18653\/v1\/N16-1098","article-title":"A corpus and cloze evaluation for deeper understanding of commonsense stories","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Mostafazadeh","year":"2016"},{"key":"2023110815073941300_bib30","doi-asserted-by":"publisher","first-page":"4569","DOI":"10.18653\/v1\/2020.emnlp-main.370","article-title":"GLUCOSE: GeneraLized and COntextualized story explanations","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Mostafazadeh","year":"2020"},{"key":"2023110815073941300_bib31","unstructured":"OpenAI. 2022. Chatgpt. https:\/\/openai.com\/blog\/chatgpt"},{"key":"2023110815073941300_bib32","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2023110815073941300_bib33","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1509","article-title":"Counterfactual story reasoning and generation","volume-title":"EMNLP","author":"Qin","year":"2019"},{"issue":"140","key":"2023110815073941300_bib34","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2023110815073941300_bib35","doi-asserted-by":"publisher","first-page":"2289","DOI":"10.18653\/v1\/P18-1213","article-title":"Modeling naive psychology of characters in simple commonsense stories","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Rashkin","year":"2018"},{"key":"2023110815073941300_bib36","doi-asserted-by":"publisher","first-page":"3027","DOI":"10.1609\/aaai.v33i01.33013027","article-title":"Atomic: An atlas of machine commonsense for if-then reasoning","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Sap","year":"2019"},{"key":"2023110815073941300_bib37","first-page":"151","article-title":"Scripts, plans, and knowledge","volume-title":"IJCAI","author":"Schank","year":"1975"},{"key":"2023110815073941300_bib38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1049","article-title":"Joint learning templates and slots for event schema induction","author":"Sha","year":"2016","journal-title":"arXiv preprint arXiv:1603.01333"},{"key":"2023110815073941300_bib39","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2019.332","article-title":"Can machines learn to detect fake news? A survey focused on social media","volume-title":"Hawaii International Conference on System Sciences","author":"da Silva","year":"2019"},{"key":"2023110815073941300_bib40","doi-asserted-by":"publisher","first-page":"4506","DOI":"10.18653\/v1\/D19-1458","article-title":"CLUTRR: A diagnostic benchmark for inductive reasoning from text","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Sinha","year":"2019"},{"key":"2023110815073941300_bib41","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.11164","article-title":"Conceptnet 5.5: An open multilingual graph of general knowledge","volume-title":"Thirty-First AAAI Conference on Artificial Intelligence","author":"Speer","year":"2017"},{"key":"2023110815073941300_bib42","doi-asserted-by":"publisher","first-page":"6076","DOI":"10.18653\/v1\/D19-1629","article-title":"WIQA: A dataset for \u201cwhat if\u2026\u201d reasoning over procedural text","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Tandon","year":"2019"},{"key":"2023110815073941300_bib43","doi-asserted-by":"publisher","first-page":"6408","DOI":"10.18653\/v1\/2020.emnlp-main.520","article-title":"A dataset for tracking entities in open domain procedural text","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Tandon","year":"2020"},{"key":"2023110815073941300_bib44","article-title":"Lamda: Language models for dialog applications","author":"Thoppilan","year":"2022","journal-title":"arXiv preprint arXiv:2201.08239"},{"key":"2023110815073941300_bib45","doi-asserted-by":"publisher","first-page":"2948","DOI":"10.18653\/v1\/N19-1302","article-title":"Repurposing entailment for multi-hop question answering tasks","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Trivedi","year":"2019"},{"issue":"2","key":"2023110815073941300_bib46","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1016\/j.jclinepi.2011.05.008","article-title":"Development of the rti item bank on risk of bias and precision of observational studies","volume":"65","author":"Viswanathan","year":"2012","journal-title":"Journal of Clinical Epidemiology"},{"key":"2023110815073941300_bib47","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1363","article-title":"Dialogue natural language inference","volume-title":"Annual Meeting of the Association for Computational Linguistics","author":"Welleck","year":"2018"},{"key":"2023110815073941300_bib48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.341","article-title":"Symbolic knowledge distillation: From general language models to commonsense models","volume-title":"North American Chapter of the Association for Computational Linguistics","author":"West","year":"2022"},{"issue":"1","key":"2023110815073941300_bib49","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of cohen\u2019s kappa and gwet\u2019s ac1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples","volume":"13","author":"Wongpakaran","year":"2013","journal-title":"BMC Medical Research Methodology"},{"key":"2023110815073941300_bib50","article-title":"Google\u2019s neural machine translation system: Bridging the gap between human and machine translation","author":"Yonghui","year":"2016","journal-title":"ArXiv"},{"key":"2023110815073941300_bib51","doi-asserted-by":"publisher","first-page":"105","DOI":"10.18653\/v1\/D18-1010","article-title":"TwoWingOS: A two-wing optimization strategy for evidential claim verification","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Yin","year":"2018"},{"key":"2023110815073941300_bib52","article-title":"Bertscore: Evaluating text generation with BERT","author":"Zhang","year":"2019","journal-title":"arXiv preprint arXiv:1904.09675"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00600\/2173956\/tacl_a_00600.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00600\/2173956\/tacl_a_00600.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T15:08:04Z","timestamp":1699456084000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00600\/118075\/PASTA-A-Dataset-for-Modeling-PArticipant-STAtes-in"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":52,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00600","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}