{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T16:07:10Z","timestamp":1784736430158,"version":"3.55.0"},"reference-count":96,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T00:00:00Z","timestamp":1726704000000},"content-version":"vor","delay-in-days":262,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,9,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Storytelling is an integral part of human experience and plays a crucial role in social interactions. Thus, Automatic Story Evaluation (ASE) and Generation (ASG) could benefit society in multiple ways, but they are challenging tasks which require high-level human abilities such as creativity, reasoning, and deep understanding. Meanwhile, Large Language Models (LLMs) now achieve state-of-the-art performance on many NLP tasks. In this paper, we study whether LLMs can be used as substitutes for human annotators for ASE. We perform an extensive analysis of the correlations between LLM ratings, other automatic measures, and human annotations, and we explore the influence of prompting on the results and the explainability of LLM behaviour. Most notably, we find that LLMs outperform current automatic measures for system-level evaluation but still struggle at providing satisfactory explanations for their answers.<\/jats:p>","DOI":"10.1162\/tacl_a_00689","type":"journal-article","created":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T19:44:50Z","timestamp":1726775090000},"page":"1122-1142","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":23,"title":["Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation"],"prefix":"10.1162","volume":"12","author":[{"given":"Cyril","family":"Chhun","sequence":"first","affiliation":[{"name":"LTCI, T\u00e9l\u00e9com Paris, Institut Polytechnique de Paris. cyril.chhun@telecom-paris.fr"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabian M.","family":"Suchanek","sequence":"additional","affiliation":[{"name":"LTCI, T\u00e9l\u00e9com Paris, Institut Polytechnique de Paris. fabian.suchanek@telecom-paris.fr"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chlo\u00e9","family":"Clavel","sequence":"additional","affiliation":[{"name":"ALMAnaCH, INRIA Paris. chloe.clavel@inria.fr"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,9,18]]},"reference":[{"key":"2024091919443324600_bib1","doi-asserted-by":"publisher","first-page":"72","DOI":"10.18653\/v1\/2021.nuse-1.8","article-title":"Automatic story generation: Challenges and attempts","volume-title":"Proceedings of the Third Workshop on Narrative Understanding","author":"Alabdulkarim","year":"2021"},{"issue":"7748","key":"2024091919443324600_bib2","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1038\/d41586-019-00857-9","article-title":"Scientists rise up against statistical significance","volume":"567","author":"Amrhein","year":"2019","journal-title":"Nature"},{"key":"2024091919443324600_bib3","article-title":"Ask me anything: A simple strategy for prompting language models","volume-title":"The Eleventh International Conference on Learning Representations","author":"Arora","year":"2023"},{"key":"2024091919443324600_bib4","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1007\/978-3-030-92300-6_45","article-title":"A preliminary survey on story interestingness: Focusing on cognitive and emotional interest","volume-title":"International Conference on Interactive Digital Storytelling","author":"Bae","year":"2021"},{"issue":"1","key":"2024091919443324600_bib5","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: A practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"key":"2024091919443324600_bib6","doi-asserted-by":"publisher","first-page":"9347","DOI":"10.18653\/v1\/2020.emnlp-main.751","article-title":"Re-evaluating evaluation in text summarization","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Bhandari","year":"2020"},{"key":"2024091919443324600_bib7","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\u201312, 2020, virtual","author":"Brown","year":"2020"},{"key":"2024091919443324600_bib8","article-title":"Sparks of artificial general intelligence: Early experiments with GPT-4","author":"Bubeck","year":"2023","journal-title":"arXiv preprint arXiv:2303.12712v5"},{"key":"2024091919443324600_bib9","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1016\/j.ins.2019.09.013","article-title":"Yake! Keyword extraction from single documents using multiple local features","volume":"509","author":"Campos","year":"2020","journal-title":"Information Sciences"},{"key":"2024091919443324600_bib10","article-title":"Evaluation of text generation: A survey","author":"Celikyilmaz","year":"2020","journal-title":"ArXiv preprint"},{"key":"2024091919443324600_bib11","article-title":"Art or artifice? Large language models and the false promise of creativity","author":"Chakrabarty","year":"2023","journal-title":"arXiv preprint arXiv:2309.14556v1"},{"key":"2024091919443324600_bib12","first-page":"5794","article-title":"Of human criteria and automatic metrics: A benchmark of the evaluation of story generation","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics","author":"Chhun","year":"2022"},{"issue":"240","key":"2024091919443324600_bib13","first-page":"1","article-title":"PaLM: Scaling language modeling with pathways","volume":"24","author":"Chowdhery","year":"2023","journal-title":"Journal of Machine Learning Research"},{"key":"2024091919443324600_bib14","doi-asserted-by":"publisher","first-page":"7282","DOI":"10.18653\/v1\/2021.acl-long.565","article-title":"All that\u2019s \u2018human\u2019 is not gold: Evaluating human evaluation of generated text","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Clark","year":"2021"},{"issue":"1","key":"2024091919443324600_bib15","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educational and Psychological Measurement"},{"key":"2024091919443324600_bib16","doi-asserted-by":"publisher","first-page":"178","DOI":"10.18653\/v1\/2023.findings-ijcnlp.16","article-title":"The glass ceiling of automatic evaluation in natural language generation","volume-title":"Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings)","author":"Colombo","year":"2023"},{"key":"2024091919443324600_bib17","doi-asserted-by":"publisher","first-page":"10450","DOI":"10.18653\/v1\/2021.emnlp-main.817","article-title":"Automatic text evaluation through the lens of Wasserstein barycenters","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Colombo","year":"2021"},{"key":"2024091919443324600_bib18","doi-asserted-by":"publisher","first-page":"10960","DOI":"10.18653\/v1\/2022.emnlp-main.753","article-title":"On the limitations of reference-free evaluations of generated text","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Deutsch","year":"2022"},{"issue":"3","key":"2024091919443324600_bib19","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1162\/15241730360580212","article-title":"The four elements of every successful story","volume":"4","author":"Dickman","year":"2003","journal-title":"Reflections - Society for Organizational Learning"},{"key":"2024091919443324600_bib20","doi-asserted-by":"publisher","first-page":"11173","DOI":"10.18653\/v1\/2023.acl-long.626","article-title":"Is GPT-3 a good data annotator?","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ding","year":"2023"},{"key":"2024091919443324600_bib21","doi-asserted-by":"publisher","first-page":"7250","DOI":"10.18653\/v1\/2022.acl-long.501","article-title":"Is GPT-3 text indistinguishable from human text? Scarecrow: A framework for scrutinizing machine text","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Dou","year":"2022"},{"key":"2024091919443324600_bib22","doi-asserted-by":"publisher","first-page":"1203","DOI":"10.3758\/s13423-017-1344-2","article-title":"A Bayesian perspective on Likert scales and central tendency","volume":"25","author":"Douven","year":"2018","journal-title":"Psychonomic Bulletin & Review"},{"key":"2024091919443324600_bib23","article-title":"Concours de nouvelles 2023","author":"Edilivre","year":"2023"},{"key":"2024091919443324600_bib24","doi-asserted-by":"publisher","first-page":"695","DOI":"10.1007\/s10115-020-01532-6","article-title":"Auto-labelling entities in low-resource text: A geological case study","volume":"63","author":"Enkhsaikhan","year":"2021","journal-title":"Knowledge and Information Systems"},{"key":"2024091919443324600_bib25","doi-asserted-by":"publisher","first-page":"889","DOI":"10.18653\/v1\/P18-1082","article-title":"Hierarchical neural story generation","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Fan","year":"2018"},{"key":"2024091919443324600_bib26","article-title":"Chance-corrected agreement coefficients","author":"Fergadis","year":"2022"},{"issue":"5","key":"2024091919443324600_bib27","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychological Bulletin"},{"key":"2024091919443324600_bib28","doi-asserted-by":"publisher","first-page":"1347","DOI":"10.18653\/v1\/2020.acl-main.124","article-title":"SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Gao","year":"2020"},{"issue":"3","key":"2024091919443324600_bib29","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1177\/1471301212468732","article-title":"How a creative storytelling intervention can improve medical student attitude towards persons with dementia: A mixed methods study","volume":"13","author":"George","year":"2014","journal-title":"Dementia"},{"key":"2024091919443324600_bib30","doi-asserted-by":"publisher","first-page":"4319","DOI":"10.18653\/v1\/2020.emnlp-main.351","article-title":"Content planning for neural story generation with Aristotelian rescoring","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Goldfarb-Tarrant","year":"2020"},{"key":"2024091919443324600_bib31","doi-asserted-by":"publisher","first-page":"172","DOI":"10.3115\/v1\/D14-1020","article-title":"Testing for significance of increased correlation with human judgment","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Graham","year":"2014"},{"key":"2024091919443324600_bib32","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1162\/tacl_a_00302","article-title":"A knowledge-enhanced pretraining model for commonsense story generation","volume":"8","author":"Guan","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024091919443324600_bib33","doi-asserted-by":"publisher","first-page":"6379","DOI":"10.18653\/v1\/2021.acl-long.499","article-title":"Long text generation by modeling sentence-level and discourse-level coherence","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Guan","year":"2021"},{"issue":"1","key":"2024091919443324600_bib34","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","year":"2008","journal-title":"British Journal of Mathematical and Statistical Psychology"},{"issue":"1","key":"2024091919443324600_bib35","doi-asserted-by":"publisher","first-page":"23","DOI":"10.20982\/tqmp.08.1.p023","article-title":"Computing inter-rater reliability for observational data: An overview and tutorial","volume":"8","author":"Hallgren","year":"2012","journal-title":"Tutorials in Quantitative Methods for Psychology"},{"issue":"1","key":"2024091919443324600_bib36","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1080\/19312450709336664","article-title":"Answering the call for a standard reliability measure for coding data","volume":"1","author":"Hayes","year":"2007","journal-title":"Communication Methods and Measures"},{"issue":"4","key":"2024091919443324600_bib37","doi-asserted-by":"publisher","first-page":"604","DOI":"10.22074\/cellj.2019.5992","article-title":"Why, when and how to adjust your P values?","volume":"20","author":"Jafari","year":"2019","journal-title":"Cell Journal (Yakhteh)"},{"issue":"2","key":"2024091919443324600_bib38","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1002\/mar.21758","article-title":"A story to sell: The influence of storytelling on consumers\u2019 purchasing behavior","volume":"40","author":"de Oliveira J\u00fanior","year":"2023","journal-title":"Psychology & Marketing"},{"key":"2024091919443324600_bib39","volume-title":"Thinking, Fast and Slow","author":"Kahneman","year":"2011"},{"issue":"1\/2","key":"2024091919443324600_bib40","doi-asserted-by":"publisher","first-page":"81","DOI":"10.2307\/2332226","article-title":"A new measure of rank correlation","volume":"30","author":"Kendall","year":"1938","journal-title":"Biometrika"},{"key":"2024091919443324600_bib41","article-title":"CTRL: A conditional transformer language model for controllable generation","author":"Keskar","year":"2019","journal-title":"ArXiv preprint"},{"key":"2024091919443324600_bib42","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume-title":"Advances in Neural Information Processing Systems","author":"Kojima","year":"2022"},{"issue":"1","key":"2024091919443324600_bib43","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1609\/aaai.v27i1.8649","article-title":"Story generation with crowdsourced plot graphs","volume":"27","author":"Li","year":"2013","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2024091919443324600_bib44","article-title":"OpenOrca: An open dataset of GPT augmented FLAN reasoning traces","author":"Lian","year":"2023"},{"key":"2024091919443324600_bib45","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2024091919443324600_bib46","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"ArXiv preprint"},{"issue":"1-2","key":"2024091919443324600_bib47","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1080\/13614568.2012.617846","article-title":"Storytelling on mobile devices for cultural heritage","volume":"18","author":"Lombardo","year":"2012","journal-title":"New Review of Hypermedia and Multimedia"},{"key":"2024091919443324600_bib48","doi-asserted-by":"publisher","first-page":"62","DOI":"10.18653\/v1\/W19-5302","article-title":"Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Ma","year":"2019"},{"key":"2024091919443324600_bib49","doi-asserted-by":"publisher","first-page":"157","DOI":"10.18653\/v1\/2022.acl-short.18","article-title":"Data contamination: From memorization to exploitation","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Magar","year":"2022"},{"issue":"6","key":"2024091919443324600_bib50","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1016\/j.tics.2024.01.011","article-title":"Dissociating language and thought in large language models","volume":"28","author":"Mahowald","year":"2024","journal-title":"Trends in Cognitive Sciences"},{"issue":"6","key":"2024091919443324600_bib51","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/BF01068179","article-title":"What makes a good story","volume":"13","author":"McCabe","year":"1984","journal-title":"Journal of Psycholinguistic Research"},{"issue":"29","key":"2024091919443324600_bib52","doi-asserted-by":"publisher","first-page":"861","DOI":"10.21105\/joss.00861","article-title":"UMAP: Uniform manifold approximation and projection","volume":"3","author":"McInnes","year":"2018","journal-title":"Journal of Open Source Software"},{"issue":"sup1","key":"2024091919443324600_bib53","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1080\/00031305.2018.1527253","article-title":"Abandon statistical significance","volume":"73","author":"McShane","year":"2019","journal-title":"The American Statistician"},{"issue":"1","key":"2024091919443324600_bib54","first-page":"36","article-title":"The power of story: Using storytelling to improve literacy learning","volume":"1","author":"Miller","year":"2008","journal-title":"Journal of Cross-Disciplinary Perspectives in Education"},{"key":"2024091919443324600_bib55","article-title":"Significance test of increase in correlation for NLP evaluations in Python","author":"Moon","year":"2019"},{"issue":"3","key":"2024091919443324600_bib56","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1016\/j.tree.2021.10.009","article-title":"Rewriting results sections in the language of evidence","volume":"37","author":"Muff","year":"2022","journal-title":"Trends in Ecology & Evolution"},{"key":"2024091919443324600_bib57","article-title":"Orca: Progressive learning from complex explanation traces of GPT-4","author":"Mukherjee","year":"2023","journal-title":"arXiv preprint arXiv:2306.02707v1"},{"key":"2024091919443324600_bib58","doi-asserted-by":"publisher","first-page":"2241","DOI":"10.18653\/v1\/D17-1238","article-title":"Why we need new evaluation metrics for NLG","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Novikova","year":"2017"},{"key":"2024091919443324600_bib59","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume-title":"Advances in Neural Information Processing Systems","author":"Ouyang","year":"2022"},{"key":"2024091919443324600_bib60","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"issue":"347\u2013352","key":"2024091919443324600_bib61","doi-asserted-by":"publisher","first-page":"240","DOI":"10.1098\/rspl.1895.0041","article-title":"VII. Note on regression and inheritance in the case of two parents","volume":"58","author":"Pearson","year":"1895","journal-title":"Proceedings of the Royal Society of London"},{"key":"2024091919443324600_bib62","doi-asserted-by":"publisher","first-page":"392","DOI":"10.18653\/v1\/W15-3049","article-title":"chrF: Character n-gram F-score for automatic MT evaluation","volume-title":"Proceedings of the Tenth Workshop on Statistical Machine Translation","author":"Popovi\u0107","year":"2015"},{"issue":"3","key":"2024091919443324600_bib63","doi-asserted-by":"publisher","first-page":"4987","DOI":"10.32604\/cmc.2022.020544","article-title":"A novel auto-annotation technique for aspect level sentiment analysis","volume":"70","author":"Qureshi","year":"2022","journal-title":"Computers, Materials and Continua"},{"issue":"8","key":"2024091919443324600_bib64","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI blog"},{"key":"2024091919443324600_bib65","doi-asserted-by":"publisher","first-page":"4274","DOI":"10.18653\/v1\/2020.emnlp-main.349","article-title":"PlotMachines: Outline-conditioned generation with dynamic plot state tracking","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Rashkin","year":"2020"},{"key":"2024091919443324600_bib66","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-BERT: Sentence embeddings using Siamese BERT-networks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Reimers","year":"2019"},{"key":"2024091919443324600_bib67","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3411763.3451760","article-title":"Prompt programming for large language models: Beyond the few-shot paradigm","volume-title":"Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems","author":"Reynolds","year":"2021"},{"key":"2024091919443324600_bib68","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1162\/tacl_a_00313","article-title":"Leveraging pre-trained checkpoints for sequence generation tasks","volume":"8","author":"Rothe","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"314","key":"2024091919443324600_bib69","first-page":"121","article-title":"Storytelling in science","volume":"86","author":"Rowcliffe","year":"2004","journal-title":"School Science Review"},{"key":"2024091919443324600_bib70","article-title":"Detecting pretraining data from large language models","volume-title":"The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\u201311, 2024","author":"Shi","year":"2024"},{"key":"2024091919443324600_bib71","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1037\/11491-005","article-title":"The proof and measurement of association between two things","volume-title":"Studies in Individual Differences: The Search for Intelligence","author":"Spearman","year":"1961"},{"issue":"2","key":"2024091919443324600_bib72","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1037\/0033-2909.87.2.245","article-title":"Tests for comparing elements of a correlation matrix","volume":"87","author":"Steiger","year":"1980","journal-title":"Psychological Bulletin"},{"issue":"5","key":"2024091919443324600_bib73","doi-asserted-by":"publisher","first-page":"426","DOI":"10.1037\/h0031324","article-title":"Issues in psychophysical measurement","volume":"78","author":"Stevens","year":"1971","journal-title":"Psychological Review"},{"key":"2024091919443324600_bib74","article-title":"LaMDA: Language models for dialog applications","author":"Thoppilan","year":"2022","journal-title":"ArXiv preprint"},{"key":"2024091919443324600_bib75","article-title":"Llama: Open and efficient foundation language models","author":"Touvron","year":"2023","journal-title":"arXiv preprint arXiv:2302.13971v1"},{"key":"2024091919443324600_bib76","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"arXiv preprint arXiv:2307.09288v2"},{"key":"2024091919443324600_bib77","doi-asserted-by":"crossref","DOI":"10.4324\/9781315806464","volume-title":"The Creative Process: A Computer Model of Storytelling and Creativity","author":"Turner","year":"2014"},{"issue":"31","key":"2024091919443324600_bib78","doi-asserted-by":"publisher","first-page":"1026","DOI":"10.21105\/joss.01026","article-title":"Pingouin: Statistics in Python","volume":"3","author":"Vallat","year":"2018","journal-title":"Journal of Open Source Software"},{"key":"2024091919443324600_bib79","doi-asserted-by":"publisher","first-page":"11","DOI":"10.18653\/v1\/2020.eval4nlp-1.2","article-title":"Fill in the BLANC: Human-free quality estimation of document summaries","volume-title":"Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems","author":"Vasilyev","year":"2020"},{"key":"2024091919443324600_bib80","first-page":"333","article-title":"Automated event annotation in literary texts","volume-title":"Proceedings of the Conference on Computational Humanities Research, CHR2021, Amsterdam, The Netherlands, November 17\u201319, 2021","author":"Vauth","year":"2021"},{"key":"2024091919443324600_bib81","doi-asserted-by":"publisher","first-page":"4195","DOI":"10.18653\/v1\/2021.findings-emnlp.354","article-title":"Want to reduce labeling cost? GPT-3 can help","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Wang","year":"2021"},{"issue":"sup1","key":"2024091919443324600_bib82","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/00031305.2019.1583913","article-title":"Moving to a world beyond \u201cp &lt;0.05\u201d","volume":"73","author":"Wasserstein","year":"2019","journal-title":"The American Statistician"},{"key":"2024091919443324600_bib83","article-title":"Emergent abilities of large language models","volume":"2022","author":"Wei","year":"2022","journal-title":"Transactions on Machine Learning Research"},{"key":"2024091919443324600_bib84","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024091919443324600_bib85","article-title":"A prompt pattern catalog to enhance prompt engineering with ChatGPT","author":"White","year":"2023","journal-title":"ArXiv preprint"},{"key":"2024091919443324600_bib86","volume-title":"Regression Analysis","author":"Williams","year":"1959"},{"key":"2024091919443324600_bib87","article-title":"A temporal variational model for story generation","author":"Wilmot","year":"2021","journal-title":"ArXiv preprint"},{"key":"2024091919443324600_bib88","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2024091919443324600_bib89","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of Cohen\u2019s Kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples","volume":"13","author":"Wongpakaran","year":"2013","journal-title":"BMC Medical Research Methodology"},{"key":"2024091919443324600_bib90","first-page":"5754","article-title":"XLNet: Generalized autoregressive pretraining for language understanding","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\u201314, 2019, Vancouver, BC, Canada","author":"Yang","year":"2019"},{"key":"2024091919443324600_bib91","first-page":"27263","article-title":"BARTScore: Evaluating generated text as text generation","volume-title":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6\u201314, 2021, virtual","author":"Yuan","year":"2021"},{"key":"2024091919443324600_bib92","article-title":"BERTScore: Evaluating text generation with BERT","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Zhang","year":"2020"},{"key":"2024091919443324600_bib93","article-title":"Interpreting BLEU\/NIST scores: How much improvement do we need to have a better system?","volume-title":"Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC\u201904)","author":"Zhang","year":"2004"},{"key":"2024091919443324600_bib94","first-page":"12697","article-title":"Calibrate before use: Improving few-shot performance of language models","volume-title":"Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18\u201324 July 2021, Virtual Event","author":"Zhao","year":"2021"},{"key":"2024091919443324600_bib95","article-title":"Least-to-most prompting enables complex reasoning in large language models","volume-title":"The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\u20135, 2023","author":"Zhou","year":"2023"},{"key":"2024091919443324600_bib96","article-title":"Large language models are human-level prompt engineers","volume-title":"The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\u20135, 2023","author":"Zhou","year":"2023"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00689\/2470807\/tacl_a_00689.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00689\/2470807\/tacl_a_00689.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,19]],"date-time":"2024-09-19T19:45:03Z","timestamp":1726775103000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00689\/124460\/Do-Language-Models-Enjoy-Their-Own-Stories"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":96,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00689","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}