{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T12:32:54Z","timestamp":1787315574090,"version":"build-2736575974"},"reference-count":74,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T00:00:00Z","timestamp":1719360000000},"content-version":"vor","delay-in-days":177,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,6,25]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We address a growing debate about the extent to which large language models (LLMs) produce behavior consistent with Theory of Mind (ToM) in humans. We present EPITOME: a battery of six experiments that tap diverse ToM capacities, including belief attribution, emotional inference, and pragmatic reasoning. We elicit a performance baseline from human participants for each task. We use the dataset to ask whether distributional linguistic information learned by LLMs is sufficient to explain ToM in humans. We compare performance of five LLMs to a baseline of responses from human comprehenders. Results are mixed. LLMs display considerable sensitivity to mental states and match human performance in several tasks. Yet, they commit systematic errors in others, especially those requiring pragmatic reasoning on the basis of mental state information. Such uneven performance indicates that human-level ToM may require resources beyond distributional information.<\/jats:p>","DOI":"10.1162\/tacl_a_00674","type":"journal-article","created":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T14:30:07Z","timestamp":1719412207000},"page":"803-819","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Comparing Humans and Large Language Models on an Experimental Protocol Inventory for Theory of Mind Evaluation (EPITOME)"],"prefix":"10.1162","volume":"12","author":[{"given":"Cameron R.","family":"Jones","sequence":"first","affiliation":[{"name":"Department of Cognitive Science, UC San Diego, 9500 Gilman Dr, La Jolla, CA 92093, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sean","family":"Trott","sequence":"additional","affiliation":[{"name":"Department of Cognitive Science, UC San Diego, 9500 Gilman Dr, La Jolla, CA 92093, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Bergen","sequence":"additional","affiliation":[{"name":"Department of Cognitive Science, UC San Diego, 9500 Gilman Dr, La Jolla, CA 92093, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,6,25]]},"reference":[{"issue":"5","key":"2024062618293935900_bib1","doi-asserted-by":"publisher","first-page":"825","DOI":"10.1080\/17470218.2012.676055","article-title":"What is \u201ctheory of mind\u201d? Concepts, cognitive processes and individual differences","volume":"65","author":"Apperly","year":"2012","journal-title":"Quarterly Journal of Experimental Psychology"},{"key":"2024062618293935900_bib2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2019.02905","article-title":"Systematic review and inventory of theory of mind measures for young children","volume":"10","author":"Beaudoin","year":"2020","journal-title":"Frontiers in Psychology"},{"issue":"27","key":"2024062618293935900_bib3","doi-asserted-by":"publisher","first-page":"11312","DOI":"10.1073\/pnas.0900010106","article-title":"Growing up blind does not change the neural bases of Theory of Mind","volume":"106","author":"Bedny","year":"2009","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024062618293935900_bib4","doi-asserted-by":"publisher","first-page":"5185","DOI":"10.18653\/v1\/2020.acl-main.463","article-title":"Climbing towards NLU: On meaning, form, and understanding in the age of data","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Bender","year":"2020"},{"issue":"3","key":"2024062618293935900_bib5","doi-asserted-by":"publisher","first-page":"836","DOI":"10.1111\/j.1467-8624.1996.tb01767.x","article-title":"Why talk about mental states? The significance of children\u2019s conversations with friends, siblings, and mothers","volume":"67","author":"Brown","year":"1996","journal-title":"Child Development"},{"key":"2024062618293935900_bib6","first-page":"1877","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2024062618293935900_bib7","article-title":"Language model behavior: A comprehensive survey","author":"Chang","year":"2023"},{"issue":"4","key":"2024062618293935900_bib8","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1097\/TLD.0000000000000037","article-title":"The role of language in theory of mind development","volume":"34","author":"de Villiers","year":"2014","journal-title":"Topics in Language Disorders"},{"issue":"4","key":"2024062618293935900_bib9","doi-asserted-by":"publisher","first-page":"568","DOI":"10.1017\/S0140525X00076664","article-title":"Beliefs about beliefs [P&W, SR&B]","volume":"1","author":"Dennett","year":"1978","journal-title":"Behavioral and Brain Sciences"},{"issue":"24","key":"2024062618293935900_bib10","doi-asserted-by":"publisher","first-page":"17817","DOI":"10.1109\/JIOT.2021.3081556","article-title":"IoT-enabled social relation ships meet artificial social intelligence","volume":"8","author":"Dhelim","year":"2021","journal-title":"IEEE Internet of Things Journal"},{"issue":"11","key":"2024062618293935900_bib11","doi-asserted-by":"publisher","first-page":"e81279","DOI":"10.1371\/journal.pone.0081279","article-title":"Using fiction to assess mental state understanding: A new task for assessing theory of mind in adults","volume":"8","author":"Dodell-Feder","year":"2013","journal-title":"PLOS ONE"},{"key":"2024062618293935900_bib12","volume-title":"A Synopsis of Linguistic Theory","author":"Firth","year":"1957"},{"issue":"1","key":"2024062618293935900_bib13","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1146\/annurev-psych-120710-100449","article-title":"Mechanisms of social cognition","volume":"63","author":"Frith","year":"2012","journal-title":"Annual Review of Psychology"},{"key":"2024062618293935900_bib14","article-title":"Understanding social reasoning in language models with language models","author":"Gandhi","year":"2023"},{"issue":"1","key":"2024062618293935900_bib15","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1037\/arc0000067","article-title":"Empirical failures of the claim that autistic people lack a theory of mind","volume":"7","author":"Gernsbacher","year":"2019","journal-title":"Archives of Scientific Psychology"},{"key":"2024062618293935900_bib16","article-title":"Time travel in LLMs: Tracing data contamination in large language models","author":"Golchin","year":"2023"},{"issue":"1","key":"2024062618293935900_bib17","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1111\/tops.12007","article-title":"Knowledge and implicature: Modeling language understanding as social cognition","volume":"5","author":"Goodman","year":"2013","journal-title":"Topics in Cognitive Science"},{"key":"2024062618293935900_bib18","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1163\/9789004368811_003","article-title":"Logic and conversation","volume-title":"Speech Acts","author":"Grice","year":"1975"},{"key":"2024062618293935900_bib19","article-title":"Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods","author":"Hagendorff","year":"2023"},{"issue":"3","key":"2024062618293935900_bib20","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1111\/1467-7687.00289","article-title":"The influence of language on theory of mind: A training study","volume":"6","author":"Hale","year":"2003","journal-title":"Developmental Science"},{"issue":"3","key":"2024062618293935900_bib21","doi-asserted-by":"publisher","first-page":"473","DOI":"10.1086\/681627","article-title":"There is no special problem of mindreading in nonhuman animals","volume":"82","author":"Halina","year":"2015","journal-title":"Philosophy of Science"},{"issue":"2","key":"2024062618293935900_bib22","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1007\/BF02172093","article-title":"An advanced test of theory of mind: Understanding of story characters\u2019 thoughts and feelings by able autistic, mentally handicapped, and normal children and adults","volume":"24","author":"Happ\u00e9","year":"1994","journal-title":"Journal of Autism and Developmental Disorders"},{"key":"2024062618293935900_bib23","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1093\/acprof:oso\/9780195159912.003.0004","article-title":"Conversation, pretense, and theory of mind","volume-title":"Why Language Matters for Theory of Mind","author":"Harris","year":"2005"},{"issue":"2\u20133","key":"2024062618293935900_bib24","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1080\/00437956.1954.11659520","article-title":"Distributional structure","volume":"10","author":"Harris","year":"1954","journal-title":"Word"},{"issue":"3","key":"2024062618293935900_bib25","doi-asserted-by":"publisher","first-page":"454","DOI":"10.1111\/bjdp.12186","article-title":"Reliability and validity of advanced theory-of-mind measures in middle childhood and adolescence","volume":"35","author":"Hayward","year":"2017","journal-title":"British Journal of Developmental Psychology"},{"issue":"2","key":"2024062618293935900_bib26","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1177\/1745691613518076","article-title":"Submentalizing: I am not really reading your mind","volume":"9","author":"Heyes","year":"2014","journal-title":"Perspectives on Psychological Science"},{"key":"2024062618293935900_bib27","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.564","article-title":"Surface Form Competition: Why the Highest Probability Answer Isn\u2019t Always Right","author":"Holtzman","year":"2022"},{"key":"2024062618293935900_bib28","article-title":"A fine-grained comparison of pragmatic language understanding in humans and language models","author":"Jennifer","year":"2022"},{"issue":"2","key":"2024062618293935900_bib29","doi-asserted-by":"publisher","first-page":"356","DOI":"10.1111\/j.1467-8624.2005.00850_a.x","article-title":"Origins of individual differences in theory of mind: From nature to nurture?","volume":"76","author":"Hughes","year":"2005","journal-title":"Child Development"},{"key":"2024062618293935900_bib30","article-title":"AI is mastering language. Should we trust what it says","author":"Johnson","year":"2022","journal-title":"The New York Times"},{"key":"2024062618293935900_bib31","article-title":"Distrubutional semantics still can\u2019t account for affordances","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","author":"Jones","year":"2022"},{"key":"2024062618293935900_bib32","article-title":"Scaling laws for neural language models","author":"Kaplan","year":"2020","journal-title":"arXiv:2001.08361 [cs, stat]"},{"key":"2024062618293935900_bib33","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2023.emnlp-main.890","article-title":"FANToM: A benchmark for stress-testing machine theory of mind in interactions","author":"Kim","year":"2023"},{"key":"2024062618293935900_bib34","article-title":"Theory of mind may have spontaneously emerged in large language models","author":"Kosinski","year":"2023"},{"issue":"6","key":"2024062618293935900_bib35","doi-asserted-by":"publisher","first-page":"e1503","DOI":"10.1002\/wcs.1503","article-title":"Theory of mind in animals: Current and future directions","volume":"10","author":"Krupenye","year":"2019","journal-title":"WIREs Cognitive Science"},{"issue":"6308","key":"2024062618293935900_bib36","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1126\/science.aaf8110","article-title":"Great apes anticipate that other individuals will act according to false beliefs","volume":"354","author":"Krupenye","year":"2016","journal-title":"Science (New York, N.Y.)"},{"key":"2024062618293935900_bib37","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2022.778852","article-title":"Theory of mind and preference learning at the interface of cognitive science, neuroscience, and AI: A review","volume":"5","author":"Langley","year":"2022","journal-title":"Frontiers in Artificial Intelligence"},{"key":"2024062618293935900_bib38","doi-asserted-by":"publisher","first-page":"5872","DOI":"10.18653\/v1\/D19-1598","article-title":"Revisiting the evaluation of theory of mind through question answering","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Le","year":"2019"},{"key":"2024062618293935900_bib39","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2022.3176783","article-title":"So cloze yet so far: N400 amplitude is better predicted by distributional information than human predictability judgements","author":"Michaelov","year":"2022","journal-title":"IEEE Transactions on Cognitive and Developmental Systems"},{"issue":"13","key":"2024062618293935900_bib40","doi-asserted-by":"publisher","first-page":"e2215907120","DOI":"10.1073\/pnas.2215907120","article-title":"The debate over understanding in AI\u2019s large language models","volume":"120","author":"Mitchell","year":"2023","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024062618293935900_bib41","unstructured":"Shima Rahimi\n              Moghaddam\n             and Christopher J.Honey. 2023. Boosting theory-of-mind performance in large language models via prompting."},{"key":"2024062618293935900_bib42","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1261","article-title":"Evaluating theory of mind in question answering","author":"Nematzadeh","year":"2018","journal-title":"arXiv:1808.09352 [cs]"},{"key":"2024062618293935900_bib43","doi-asserted-by":"publisher","first-page":"4658","DOI":"10.18653\/v1\/P19-1459","article-title":"Probing neural network comprehension of natural language arguments","volume-title":"ACL 2019-57th Annual Meeting of the Association for Computational Linguistics","author":"Niven","year":"2020"},{"issue":"4","key":"2024062618293935900_bib44","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1016\/j.evolhumbehav.2015.01.004","article-title":"The ease and extent of recursive mindreading, across implicit and explicit tasks","volume":"36","author":"O\u2019Grady","year":"2015","journal-title":"Evolution and Human Behavior"},{"key":"2024062618293935900_bib45","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"1480","key":"2024062618293935900_bib46","doi-asserted-by":"publisher","first-page":"731","DOI":"10.1098\/rstb.2006.2023","article-title":"On the lack of evidence that non-human animals possess anything remotely resembling a \u2018theory of mind\u2019","volume":"362","author":"Penn","year":"2007","journal-title":"Philosophical Transactions of the Royal Society B: Biological Sciences"},{"issue":"2","key":"2024062618293935900_bib47","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1111\/j.2044-835X.1987.tb01048.x","article-title":"Three-year-olds\u2019 difficulty with false belief: The case for a conceptual deficit","volume":"5","author":"Perner","year":"1987","journal-title":"British Journal of Developmental Psychology"},{"issue":"4","key":"2024062618293935900_bib48","doi-asserted-by":"publisher","first-page":"515","DOI":"10.1017\/S0140525X00076512","article-title":"Does the chimpanzee have a theory of mind?","volume":"1","author":"Premack","year":"1978","journal-title":"Behavioral and Brain Sciences"},{"key":"2024062618293935900_bib49","first-page":"4218","article-title":"Machine theory of mind","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Rabinowitz","year":"2018"},{"key":"2024062618293935900_bib50","article-title":"AI and the everything in the whole wide world benchmark","author":"Raji","year":"2021","journal-title":"arXiv preprint arXiv:2111.15366"},{"key":"2024062618293935900_bib51","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2021.682578","article-title":"The singleton fallacy: Why current critiques of language models miss the point","volume":"4","author":"Sahlgren","year":"2021","journal-title":"Frontiers in Artificial Intelligence"},{"key":"2024062618293935900_bib52","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2022.emnlp-main.248","article-title":"Neural theory-of-mind? On the limits of social intelligence in large LMs","author":"Sap","year":"2022"},{"key":"2024062618293935900_bib53","doi-asserted-by":"publisher","first-page":"4463","DOI":"10.18653\/v1\/D19-1454","article-title":"Social IQa: Commonsense reasoning about social interactions","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Sap","year":"2019"},{"issue":"45","key":"2024062618293935900_bib54","doi-asserted-by":"publisher","first-page":"e2105646118","DOI":"10.1073\/pnas.2105646118","article-title":"The neural architecture of language: Integrative modeling converges on predictive processing","volume":"118","author":"Schrimpf","year":"2021","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024062618293935900_bib55","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1057\/9781137026521_5","article-title":"A dispositional approach to attitudes: Thinking outside of the belief box","author":"Schwitzgebel","year":"2013","journal-title":"New Essays on Belief: Constitution, Content and Structure"},{"issue":"3","key":"2024062618293935900_bib56","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1017\/S0140525X00005756","article-title":"Minds, brains, and programs","volume":"3","author":"Searle","year":"1980","journal-title":"Behavioral and Brain Sciences"},{"issue":"2","key":"2024062618293935900_bib57","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1016\/j.tics.2005.12.009","article-title":"Joint action: Bodies and minds moving together","volume":"10","author":"Sebanz","year":"2006","journal-title":"Trends in Cognitive Sciences"},{"key":"2024062618293935900_bib58","article-title":"Clever Hans or neural theory of mind? Stress testing social reasoning in large language models","author":"Shapira","year":"2023"},{"key":"2024062618293935900_bib59","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1145\/3442381.3450097","article-title":"Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach","volume-title":"Proceedings of the Web Conference 2021","author":"Sharma","year":"2021"},{"key":"2024062618293935900_bib60","article-title":"Uncanny believers: Chatbots, beliefs, and folk psychology","author":"Shevlin","year":""},{"issue":"1\u20132","key":"2024062618293935900_bib61","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1111\/1468-0017.00186","article-title":"Pragmatics, modularity","volume":"17","author":"Sperber","year":"2002","journal-title":"Mind & Language"},{"issue":"7","key":"2024062618293935900_bib62","doi-asserted-by":"publisher","first-page":"580","DOI":"10.1111\/j.1467-9280.2007.01943.x","article-title":"Attribution of beliefs by 13-month-old infants","volume":"18","author":"Surian","year":"2007","journal-title":"Psychological Science"},{"issue":"5","key":"2024062618293935900_bib63","doi-asserted-by":"publisher","first-page":"675","DOI":"10.1017\/S0140525X05000129","article-title":"Understanding and sharing intentions: The origins of cultural cognition","volume":"28","author":"Tomasello","year":"2005","journal-title":"Behavioral and Brain Sciences"},{"issue":"8","key":"2024062618293935900_bib64","doi-asserted-by":"publisher","first-page":"675","DOI":"10.1080\/0163853X.2018.1548219","article-title":"Individual differences in mentalizing capacity predict indirect request comprehension","volume":"56","author":"Trott","year":"2018","journal-title":"Discourse Processes"},{"issue":"10","key":"2024062618293935900_bib65","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1080\/0163853X.2020.1822709","article-title":"When do comprehenders mentalize for pragmatic inference?","volume":"57","author":"Trott","year":"2020","journal-title":"Discourse Processes"},{"issue":"7","key":"2024062618293935900_bib66","doi-asserted-by":"publisher","first-page":"e13309","DOI":"10.1111\/cogs.13309","article-title":"Do large language models know what humans know?","volume":"47","author":"Trott","year":"2023","journal-title":"Cognitive Science"},{"key":"2024062618293935900_bib67","article-title":"Large language models fail on trivial alterations to theory-of-mind tasks","author":"Ullman","year":"2023"},{"key":"2024062618293935900_bib68","doi-asserted-by":"publisher","first-page":"5635","DOI":"10.18653\/v1\/P19-1566","article-title":"Persuasion for good: Towards a personalized persuasive dialogue system for social good","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Wang","year":"2019"},{"key":"2024062618293935900_bib69","doi-asserted-by":"publisher","DOI":"10.1201\/9781003205388-2","article-title":"What artificial neural networks can tell us about human language acquisition","author":"Warstadt","year":"2022"},{"issue":"3","key":"2024062618293935900_bib70","doi-asserted-by":"publisher","first-page":"655","DOI":"10.1111\/1467-8624.00304","article-title":"Meta-analysis of theory-of-mind development: The truth about false belief","volume":"72","author":"Wellman","year":"2001","journal-title":"Child Development"},{"issue":"1","key":"2024062618293935900_bib71","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1016\/0010-0277(83)90004-5","article-title":"Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children\u2019s understanding of deception","volume":"13","author":"Wimmer","year":"1983","journal-title":"Cognition"},{"issue":"2","key":"2024062618293935900_bib72","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1162\/daed_a_01909","article-title":"Do Large Language Models Understand Us?","volume":"151","author":"Arcas","year":"2022","journal-title":"Daedalus"},{"issue":"2","key":"2024062618293935900_bib73","doi-asserted-by":"publisher","first-page":"e13238","DOI":"10.1111\/cogs.13238","article-title":"Scalar implicature is sensitive to contextual alternatives","volume":"47","author":"Zhang","year":"2023","journal-title":"Cognitive Science"},{"key":"2024062618293935900_bib74","article-title":"Few-shot language coordination by modeling theory of mind","author":"Zhu","year":"2021"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00674\/2387988\/tacl_a_00674.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00674\/2387988\/tacl_a_00674.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T14:30:15Z","timestamp":1719412215000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00674\/122721\/Comparing-Humans-and-Large-Language-Models-on-an"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":74,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00674","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}