{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T18:32:45Z","timestamp":1792607565691,"version":"4.5.0"},"reference-count":68,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T00:00:00Z","timestamp":1688515200000},"content-version":"vor","delay-in-days":185,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,6,29]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply these analyses to four neural language models trained on English (an LSTM, a Transformer, Transformer-XL, and GPT-2). For local structure\u2014e.g., individual dependencies\u2014text generated with a standard sampling scheme is substantially less novel than our baseline of human-generated text from each model\u2019s test set. For larger-scale structure\u2014e.g., overall sentence structure\u2014model-generated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also perform extensive manual analysis, finding evidence that GPT-2 uses both compositional and analogical generalization mechanisms and showing that GPT-2\u2019s novel text is usually well-formed morphologically and syntactically but has reasonably frequent semantic issues (e.g., being self-contradictory).<\/jats:p>","DOI":"10.1162\/tacl_a_00567","type":"journal-article","created":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T13:33:45Z","timestamp":1688564025000},"page":"652-670","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":58,"title":["How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN"],"prefix":"10.1162","volume":"11","author":[{"given":"R. Thomas","family":"McCoy","sequence":"first","affiliation":[{"name":"Princeton University, USA. tom.mccoy@princeton.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paul","family":"Smolensky","sequence":"additional","affiliation":[{"name":"Microsoft Research, USA. psmo@microsoft.com"},{"name":"Johns Hopkins University, USA. psmo@microsoft.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tal","family":"Linzen","sequence":"additional","affiliation":[{"name":"New York University, USA. linzen@nyu.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianfeng","family":"Gao","sequence":"additional","affiliation":[{"name":"Microsoft Research, USA. jfgao@microsoft.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Asli","family":"Celikyilmaz","sequence":"additional","affiliation":[{"name":"Meta AI, USA. aslic@meta.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2023,6,29]]},"reference":[{"key":"2023070517334390300_bib1","doi-asserted-by":"crossref","first-page":"2429","DOI":"10.18653\/v1\/2022.findings-emnlp.180","article-title":"Towards tracing knowledge in language models back to the training data","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2022","author":"Akyurek","year":"2022"},{"issue":"2","key":"2023070517334390300_bib2","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/S0010-0277(03)00146-X","article-title":"Rules vs. analogy in English past tenses: A computational\/ experimental study","volume":"90","author":"Albright","year":"2003","journal-title":"Cognition"},{"key":"2023070517334390300_bib3","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922","article-title":"On the dangers of stochastic parrots: Can language models be too big?","volume-title":"Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency","author":"Bender","year":"2021"},{"key":"2023070517334390300_bib4","doi-asserted-by":"publisher","first-page":"5185","DOI":"10.18653\/v1\/2020.acl-main.463","article-title":"Climbing towards NLU: On meaning, form, and understanding in the age of data","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Bender","year":"2020"},{"issue":"2\u20133","key":"2023070517334390300_bib5","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1080\/00437956.1958.11659661","article-title":"The child\u2019s learning of English morphology","volume":"14","author":"Berko","year":"1958","journal-title":"Word"},{"key":"2023070517334390300_bib6","first-page":"1877","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2023070517334390300_bib7","article-title":"Quantifying memorization across neural language models","volume-title":"The Eleventh International Conference on Learning Representations","author":"Carlini","year":"2023"},{"key":"2023070517334390300_bib8","first-page":"267","article-title":"The secret sharer: Evaluating and testing unintended memorization in neural networks","volume-title":"28th USENIX Security Symposium","author":"Carlini","year":"2019"},{"key":"2023070517334390300_bib9","first-page":"2633","article-title":"Extracting training data from large language models","volume-title":"30th USENIX Security Symposium","author":"Carlini","year":"2021"},{"key":"2023070517334390300_bib10","article-title":"Evaluation of text generation: A survey","author":"Celikyilmaz","year":"2020","journal-title":"arXiv preprint arXiv:2006.14799v2"},{"key":"2023070517334390300_bib11","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2107.03374","article-title":"Evaluating large language models trained on code","author":"Chen","year":"2021","journal-title":"arXiv preprint arXiv:2107.03374v2"},{"key":"2023070517334390300_bib12","doi-asserted-by":"publisher","first-page":"2978","DOI":"10.18653\/v1\/P19-1285","article-title":"Transformer-XL: Attentive language models beyond a fixed-length context","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Dai","year":"2019"},{"key":"2023070517334390300_bib13","first-page":"4816","article-title":"Distinguishing rule and exemplar-based generalization in learning systems","volume-title":"Proceedings of the 39th International Conference on Machine Learning","author":"Dasgupta","year":"2022"},{"key":"2023070517334390300_bib14","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional Transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2023070517334390300_bib15","doi-asserted-by":"publisher","first-page":"7250","DOI":"10.18653\/v1\/2022.acl-long.501","article-title":"Is GPT-3 text indistinguishable from human text? Scarecrow: A framework for scrutinizing machine text","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Dou","year":"2022"},{"key":"2023070517334390300_bib16","doi-asserted-by":"publisher","first-page":"199","DOI":"10.18653\/v1\/N16-1024","article-title":"Recurrent neural network grammars","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Dyer","year":"2016"},{"key":"2023070517334390300_bib17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2207.14251","article-title":"Measuring causal effects of data statistics on language model\u2019s \u2018factual\u2019 predictions","author":"Elazar","year":"2022","journal-title":"arXiv preprint arXiv: 2207.14251v1"},{"key":"2023070517334390300_bib18","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1613\/jair.5477","article-title":"Survey of the state of the art in natural language generation: Core tasks, applications and evaluation","volume":"61","author":"Gatt","year":"2018","journal-title":"Journal of Artificial Intelligence Research"},{"issue":"3","key":"2023070517334390300_bib19","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1111\/j.1468-0017.1994.tb00225.x","article-title":"Systematicity in connectionist language learning","volume":"9","author":"Hadley","year":"1994","journal-title":"Mind & Language"},{"key":"2023070517334390300_bib20","doi-asserted-by":"publisher","first-page":"333","DOI":"10.18653\/v1\/2020.blackboxnlp-1.31","article-title":"This is a BERT. Now there are several of them. Can they generalize to novel words?","volume-title":"Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP","author":"Haley","year":"2020"},{"key":"2023070517334390300_bib21","article-title":"ORCA: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data","author":"Han","year":"2022","journal-title":"arXiv preprint arXiv:2205.12600v1"},{"key":"2023070517334390300_bib22","doi-asserted-by":"publisher","first-page":"1689","DOI":"10.18653\/v1\/N19-1169","article-title":"Unifying human and statistical evaluation for natural language generation","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Hashimoto","year":"2019"},{"issue":"8","key":"2023070517334390300_bib23","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"key":"2023070517334390300_bib24","article-title":"The curious case of neural text degeneration","volume-title":"International Conference on Learning Representations","author":"Holtzman","year":"2020"},{"key":"2023070517334390300_bib25","doi-asserted-by":"publisher","first-page":"1725","DOI":"10.18653\/v1\/2020.acl-main.158","article-title":"A systematic assessment of syntactic generalization in neural language models","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Jennifer","year":"2020"},{"key":"2023070517334390300_bib26","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1613\/jair.1.11674","article-title":"Compositionality decomposed: How do neural networks generalise?","volume":"67","author":"Hupkes","year":"2020","journal-title":"Journal of Artificial Intelligence Research"},{"key":"2023070517334390300_bib27","first-page":"10697","article-title":"Deduplicating training data mitigates privacy risks in language models","volume-title":"Proceedings of the 39th International Conference on Machine Learning","author":"Kandpal","year":"2022"},{"key":"2023070517334390300_bib28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2001.08361","article-title":"Scaling laws for neural language models","author":"Kaplan","year":"2020","journal-title":"arXiv preprint arXiv:2001.08361v1"},{"key":"2023070517334390300_bib29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.05858","article-title":"CTRL: A conditional Transformer language model for controllable generation","author":"Keskar","year":"2019","journal-title":"arXiv preprint arXiv:1909.05858v2"},{"key":"2023070517334390300_bib30","article-title":"Measuring compositional generalization: A comprehensive method on realistic data","volume-title":"International Conference on Learning Representations","author":"Keysers","year":"2020"},{"key":"2023070517334390300_bib31","article-title":"Generalization through memorization: Nearest neighbor language models","volume-title":"International Conference on Learning Representations","author":"Khandelwal","year":"2020"},{"key":"2023070517334390300_bib32","doi-asserted-by":"publisher","first-page":"9087","DOI":"10.18653\/v1\/2020.emnlp-main.731","article-title":"COGS: A compositional generalization challenge based on semantic interpretation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Kim","year":"2020"},{"key":"2023070517334390300_bib33","doi-asserted-by":"publisher","first-page":"2676","DOI":"10.18653\/v1\/P18-1249","article-title":"Constituency parsing with a self-attentive encoder","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Kitaev","year":"2018"},{"key":"2023070517334390300_bib34","first-page":"177","article-title":"Moses: Open source toolkit for statistical machine translation","volume-title":"Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions","author":"Koehn","year":"2007"},{"key":"2023070517334390300_bib35","doi-asserted-by":"publisher","first-page":"9332","DOI":"10.18653\/v1\/2020.emnlp-main.750","article-title":"Evaluating the factual consistency of abstractive text summarization","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Kry\u015bci\u0144ski","year":"2020"},{"issue":"5","key":"2023070517334390300_bib36","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1016\/S0022-5371(77)80021-2","article-title":"The acquisition of regular and irregular past tense forms","volume":"16","author":"Stan","year":"1977","journal-title":"Journal of Verbal Learning and Verbal Behavior"},{"key":"2023070517334390300_bib37","first-page":"2873","article-title":"Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks","volume-title":"International Conference on Machine Learning","author":"Lake","year":"2018"},{"key":"2023070517334390300_bib38","article-title":"Evaluating distributional distortion in neural language modeling","volume-title":"International Conference on Learning Representations","author":"LeBrun","year":"2022"},{"key":"2023070517334390300_bib39","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583199","article-title":"Do language models plagiarize?","author":"Lee","year":"2022","journal-title":"arXiv preprint arXiv:2203.07618v2"},{"key":"2023070517334390300_bib40","doi-asserted-by":"publisher","first-page":"8424","DOI":"10.18653\/v1\/2022.acl-long.577","article-title":"Deduplicating training data makes language models better","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Lee","year":"2022"},{"key":"2023070517334390300_bib41","doi-asserted-by":"publisher","first-page":"4767","DOI":"10.18653\/v1\/2021.acl-long.368","article-title":"On compositional generalization of neural machine translation","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Li","year":"2021"},{"key":"2023070517334390300_bib42","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2002.06177","article-title":"The next decade in AI: Four steps towards robust artificial intelligence","author":"Marcus","year":"2020","journal-title":"arXiv preprint arXiv:2002.06177v3"},{"key":"2023070517334390300_bib43","doi-asserted-by":"publisher","first-page":"i\u2013178","DOI":"10.2307\/1166115","article-title":"Overregularization in language acquisition","author":"Marcus","year":"1992","journal-title":"Monographs of the Society for Research in Child Development"},{"key":"2023070517334390300_bib44","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1162\/tacl_a_00304","article-title":"Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks","volume":"8","author":"Thomas McCoy","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023070517334390300_bib45","doi-asserted-by":"publisher","first-page":"5328","DOI":"10.18653\/v1\/2021.acl-long.414","article-title":"Language model evaluation beyond perplexity","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Meister","year":"2021"},{"key":"2023070517334390300_bib46","article-title":"Pointer sentinel mixture models","volume-title":"International Conference on Learning Representations","author":"Merity","year":"2017"},{"issue":"3","key":"2023070517334390300_bib47","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1111\/j.1755-2567.1970.tb00434.x","article-title":"Universal grammar","volume":"36","author":"Montague","year":"1970","journal-title":"Theoria"},{"key":"2023070517334390300_bib48","first-page":"344","article-title":"GLEU: Automatic evaluation of sentence-level fluency","volume-title":"Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics","author":"Mutton","year":"2007"},{"key":"2023070517334390300_bib49","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262028844.001.0001","volume-title":"Productivity and Reuse in Language: A Theory of Linguistic Computation and Storage","author":"O\u2019Donnell","year":"2015"},{"key":"2023070517334390300_bib50","doi-asserted-by":"publisher","first-page":"165","DOI":"10.18653\/v1\/2020.conll-1.13","article-title":"Recurrent babbling: Evaluating the acquisition of grammar from limited input data","volume-title":"Proceedings of the 24th Conference on Computational Natural Language Learning","author":"Pannitto","year":"2020"},{"issue":"1","key":"2023070517334390300_bib51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/01690969308406948","article-title":"Generalisation of regular and irregular morphological patterns","volume":"8","author":"Prasada","year":"1993","journal-title":"Language and Cognitive Processes"},{"key":"2023070517334390300_bib52","article-title":"Language models are unsupervised multitask learners","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2023070517334390300_bib53","doi-asserted-by":"publisher","first-page":"4274","DOI":"10.18653\/v1\/2020.emnlp-main.349","article-title":"PlotMachines: Outline-conditioned generation with dynamic plot state tracking","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Rashkin","year":"2020"},{"key":"2023070517334390300_bib54","doi-asserted-by":"publisher","first-page":"3532","DOI":"10.18653\/v1\/N19-1356","article-title":"Studying the inductive biases of RNNs with synthetic variations of natural languages","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Ravfogel","year":"2019"},{"key":"2023070517334390300_bib55","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1910.06611","article-title":"Enhancing the Transformer with explicit relational encoding for math problem solving","volume-title":"NeurIPS Workshop on Context and Composition in Biological and Artificial Neural Systems","author":"Schlag","year":"2019"},{"key":"2023070517334390300_bib56","doi-asserted-by":"publisher","first-page":"1715","DOI":"10.18653\/v1\/P16-1162","article-title":"Neural machine translation of rare words with subword units","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich","year":"2016"},{"key":"2023070517334390300_bib57","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/SP.2017.41","article-title":"Membership inference attacks against machine learning models","volume-title":"2017 IEEE Symposium on Security and Privacy (SP)","author":"Shokri","year":"2017"},{"key":"2023070517334390300_bib58","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2023070517334390300_bib59","doi-asserted-by":"publisher","first-page":"932","DOI":"10.18653\/v1\/2021.emnlp-main.72","article-title":"Frequency effects on syntactic rule learning in Transformers","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Wei","year":"2021"},{"key":"2023070517334390300_bib60","doi-asserted-by":"publisher","first-page":"3977","DOI":"10.18653\/v1\/2022.findings-acl.314","article-title":"Probing BERT\u2019s priors with serial reproduction chains","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Yamakoshi","year":"2022"},{"key":"2023070517334390300_bib61","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262035323.001.0001","volume-title":"The Price of Linguistic Productivity: How Children Learn to Break the Rules of Language","author":"Yang","year":"2016"},{"issue":"3","key":"2023070517334390300_bib62","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1145\/3446776","article-title":"Understanding deep learning (still) requires rethinking generalization","volume":"64","author":"Zhang","year":"2021","journal-title":"Communications of the ACM"},{"key":"2023070517334390300_bib63","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2205.01068","article-title":"OPT: Open pre-trained Transformer language models","author":"Zhang","year":"2022","journal-title":"arXiv preprint arXiv:2205.01068v4"},{"key":"2023070517334390300_bib64","article-title":"BERTScore: Evaluating text generation with BERT","volume-title":"International Conference on Learning Representations","author":"Zhang","year":"2020"},{"key":"2023070517334390300_bib65","doi-asserted-by":"publisher","first-page":"1112","DOI":"10.18653\/v1\/2021.acl-long.90","article-title":"When do you need billions of words of pretraining data?","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Zhang","year":"2021"},{"key":"2023070517334390300_bib66","doi-asserted-by":"crossref","first-page":"3295","DOI":"10.18653\/v1\/2020.acl-main.302","article-title":"Efficient second-order TreeCRF for neural dependency parsing","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Zhang","year":"2020"},{"key":"2023070517334390300_bib67","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1145\/3209978.3210080","article-title":"Texygen: A benchmarking platform for text generation models","volume-title":"The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval","author":"Zhu","year":"2018"},{"key":"2023070517334390300_bib68","article-title":"A first look at rote learning in GitHub Copilot suggestions.","author":"Ziegler","year":"2021","journal-title":"GitHub Docs"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00567\/2141008\/tacl_a_00567.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00567\/2141008\/tacl_a_00567.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,16]],"date-time":"2023-12-16T08:16:08Z","timestamp":1702714568000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00567\/116616\/How-Much-Do-Language-Models-Copy-From-Their"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":68,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00567","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}