{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T04:05:37Z","timestamp":1777089937057,"version":"3.51.4"},"reference-count":50,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,6,22]],"date-time":"2023-06-22T00:00:00Z","timestamp":1687392000000},"content-version":"vor","delay-in-days":172,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,6,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Current work on image-based story generation suffers from the fact that the existing image sequence collections do not have coherent plots behind them. We improve visual story generation by producing a new image-grounded dataset, Visual Writing Prompts (VWP). VWP contains almost 2K selected sequences of movie shots, each including 5-10 images. The image sequences are aligned with a total of 12K stories which were collected via crowdsourcing given the image sequences and a set of grounded characters from the corresponding image sequence. Our new image sequence collection and filtering process has allowed us to obtain stories that are more coherent, diverse, and visually grounded compared to previous work. We also propose a character-based story generation model driven by coherence as a strong baseline. Evaluations show that our generated stories are more coherent, visually grounded, and diverse than stories generated with the current state-of-the-art model. Our code, image features, annotations and collected stories are available at https:\/\/vwprompt.github.io\/.<\/jats:p>","DOI":"10.1162\/tacl_a_00553","type":"journal-article","created":{"date-parts":[[2023,6,22]],"date-time":"2023-06-22T22:23:33Z","timestamp":1687472613000},"page":"565-581","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":18,"title":["Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences"],"prefix":"10.1162","volume":"11","author":[{"given":"Xudong","family":"Hong","sequence":"first","affiliation":[{"name":"Dept. of Computer Vision and Machine Learning, MPI Informatics, Germany. xhong@lst.uni-saarland.de"},{"name":"Dept. of Language Science and Technology and Dept. of Computer Science, Saarland University, Germany. xhong@lst.uni-saarland.de"},{"name":"Saarland Informatics Campus, Saarbr\u00fccken, Germany. xhong@lst.uni-saarland.de"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Asad","family":"Sayeed","sequence":"additional","affiliation":[{"name":"Dept. of Philosophy, Linguistics, and Theory of Science, University of Gothenburg, Sweden. asad.sayeed@gu.se"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Khushboo","family":"Mehra","sequence":"additional","affiliation":[{"name":"Dept. of Language Science and Technology and Dept. of Computer Science, Saarland University, Germany. kmehra@lst.uni-saarland.de"},{"name":"Saarland Informatics Campus, Saarbr\u00fccken, Germany. kmehra@lst.uni-saarland.de"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vera","family":"Demberg","sequence":"additional","affiliation":[{"name":"Dept. of Language Science and Technology and Dept. of Computer Science, Saarland University, Germany. vera@lst.uni-saarland.de"},{"name":"Saarland Informatics Campus, Saarbr\u00fccken, Germany. vera@lst.uni-saarland.de"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bernt","family":"Schiele","sequence":"additional","affiliation":[{"name":"Dept. of Computer Vision and Machine Learning, MPI Informatics, Germany. schiele@mpi-inf.mpg.de"},{"name":"Saarland Informatics Campus, Saarbr\u00fccken, Germany. schiele@mpi-inf.mpg.de"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,6,20]]},"reference":[{"key":"2023062214082517500_bib1","doi-asserted-by":"publisher","first-page":"6470","DOI":"10.18653\/v1\/2020.emnlp-main.525","article-title":"STORIUM: A dataset and evaluation platform for machine-in-the-loop story generation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Akoury","year":"2020"},{"key":"2023062214082517500_bib2","first-page":"65","article-title":"METEOR: An automatic metric for MT evaluation with improved correlation with human judgments","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee","year":"2005"},{"issue":"5","key":"2023062214082517500_bib3","doi-asserted-by":"publisher","first-page":"1483","DOI":"10.1109\/TPAMI.2019.2956516","article-title":"Cascade R-CNN: High quality object detection and instance segmentation","volume":"43","author":"Cai","year":"2021","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2023062214082517500_bib4","doi-asserted-by":"publisher","first-page":"6040","DOI":"10.18653\/v1\/P19-1606","article-title":"Storyboarding of recipes: Grounded contextual generation","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Chandu","year":"2019"},{"key":"2023062214082517500_bib5","doi-asserted-by":"publisher","first-page":"2250","DOI":"10.18653\/v1\/N18-1204","article-title":"Neural text generation in stories using entity representations as context","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Clark","year":"2018"},{"issue":"3","key":"2023062214082517500_bib6","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1111\/cogs.12016","article-title":"Visual narrative structure","volume":"37","author":"Cohn","year":"2013","journal-title":"Cognitive Science"},{"issue":"2","key":"2023062214082517500_bib7","doi-asserted-by":"publisher","first-page":"266","DOI":"10.3758\/s13423-019-01670-1","article-title":"Visual narrative comprehension: Universal or not?","volume":"27","author":"Cohn","year":"2020","journal-title":"Psychonomic Bulletin & Review"},{"issue":"1","key":"2023062214082517500_bib8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.cogpsych.2012.01.003","article-title":"(Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension","volume":"65","author":"Cohn","year":"2012","journal-title":"Cognitive Psychology"},{"key":"2023062214082517500_bib9","doi-asserted-by":"crossref","first-page":"889","DOI":"10.18653\/v1\/P18-1082","article-title":"Hierarchical neural story generation","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Fan","year":"2018"},{"key":"2023062214082517500_bib10","doi-asserted-by":"publisher","first-page":"96","DOI":"10.18653\/v1\/2021.gem-1.10","article-title":"The GEM benchmark: Natural language generation, its evaluation and metrics","volume-title":"Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)","author":"Gehrmann","year":"2021"},{"key":"2023062214082517500_bib11","doi-asserted-by":"publisher","first-page":"4319","DOI":"10.18653\/v1\/2020.emnlp-main.351","article-title":"Content planning for neural story generation with aristotelian rescoring","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Goldfarb-Tarrant","year":"2020"},{"issue":"2","key":"2023062214082517500_bib12","doi-asserted-by":"publisher","first-page":"203","DOI":"10.21236\/ADA324949","article-title":"Centering: A framework for modeling the local coherence of discourse","volume":"21","author":"Grosz","year":"1995","journal-title":"Computational Linguistics"},{"key":"2023062214082517500_bib13","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1162\/tacl_a_00302","article-title":"A knowledge-enhanced pretraining model for commonsense story generation","volume":"8","author":"Guan","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023062214082517500_bib14","doi-asserted-by":"publisher","first-page":"9157","DOI":"10.18653\/v1\/2020.emnlp-main.736","article-title":"UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Guan","year":"2020"},{"issue":"1","key":"2023062214082517500_bib15","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1016\/0167-2789(90)90087-6","article-title":"The symbol grounding problem","volume":"42","author":"Harnad","year":"1990","journal-title":"Physica D: Nonlinear Phenomena"},{"issue":"2","key":"2023062214082517500_bib16","doi-asserted-by":"publisher","first-page":"386","DOI":"10.1109\/TPAMI.2018.2844175","article-title":"Mask R-CNN","volume":"42","author":"He","year":"2020","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2023062214082517500_bib17","article-title":"The curious case of neural text degeneration","volume-title":"International Conference on Learning Representations","author":"Holtzman","year":"2020"},{"key":"2023062214082517500_bib18","doi-asserted-by":"publisher","first-page":"420","DOI":"10.18653\/v1\/2020.conll-1.34","article-title":"Diverse and relevant visual storytelling with scene graph embeddings","volume-title":"Proceedings of the 24th Conference on Computational Natural Language Learning","author":"Hong","year":"2020"},{"issue":"05","key":"2023062214082517500_bib19","doi-asserted-by":"publisher","first-page":"7952","DOI":"10.1609\/aaai.v34i05.6303","article-title":"Knowledge-enriched visual storytelling","volume":"34","author":"Hsu","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2023062214082517500_bib20","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1007\/978-3-030-58548-8_41","article-title":"Movienet: A holistic dataset for movie understanding","volume-title":"Computer Vision \u2013 ECCV 2020","author":"Huang","year":"2020"},{"key":"2023062214082517500_bib21","first-page":"1233","article-title":"Visual storytelling","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Huang","year":"2016"},{"key":"2023062214082517500_bib22","doi-asserted-by":"publisher","DOI":"10.1515\/9783110316469","volume-title":"Handbook of Narratology","author":"H\u00fchn","year":"2014"},{"key":"2023062214082517500_bib23","first-page":"1085","article-title":"Automatic evaluation of text coherence: Models and representations","volume-title":"IJCAI\u201905 Proceedings of the 19th International Joint Conference on Artificial Intelligence","author":"Lapata","year":"2005"},{"key":"2023062214082517500_bib24","first-page":"687","article-title":"Higher-order coreference resolution with coarse-to-fine inference","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Lee","year":"2018"},{"key":"2023062214082517500_bib25","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2023062214082517500_bib26","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1007\/978-3-319-10602-1_48","article-title":"Microsoft COCO: Common objects in context","volume-title":"Computer Vision \u2013 ECCV 2014","author":"Lin","year":"2014"},{"key":"2023062214082517500_bib27","doi-asserted-by":"publisher","first-page":"9992","DOI":"10.1109\/ICCV48922.2021.00986","article-title":"Swin transformer: Hierarchical vision transformer using shifted windows","volume-title":"2021 IEEE\/CVF International Conference on Computer Vision (ICCV)","author":"Ze","year":"2021"},{"issue":"1","key":"2023062214082517500_bib28","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11430","article-title":"Event representations for automated story generation with deep neural nets","volume":"32","author":"Martin","year":"2018","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2023062214082517500_bib29","volume-title":"Proceedings of the First Workshop on Storytelling","author":"Mitchell","year":"2018"},{"key":"2023062214082517500_bib30","doi-asserted-by":"publisher","first-page":"839","DOI":"10.18653\/v1\/N16-1098","article-title":"A corpus and cloze evaluation for deeper understanding of commonsense stories","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Mostafazadeh","year":"2016"},{"key":"2023062214082517500_bib31","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2023062214082517500_bib32","article-title":"Expressing an image stream with a sequence of natural sentences","volume-title":"Advances in Neural Information Processing Systems","author":"Park","year":"2015"},{"key":"2023062214082517500_bib33","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1109\/ICPR.2000.903548","article-title":"Diatom autofocusing in brightfield microscopy: A comparative study","volume-title":"Proceedings 15th International Conference on Pattern Recognition. ICPR-2000","author":"Luis Pech-Pacheco","year":"2000"},{"key":"2023062214082517500_bib34","doi-asserted-by":"crossref","first-page":"43","DOI":"10.18653\/v1\/W18-1505","article-title":"Towards controllable story generation","volume-title":"Proceedings of the First Workshop on Storytelling","author":"Peng","year":"2018"},{"key":"2023062214082517500_bib35","doi-asserted-by":"publisher","first-page":"1756","DOI":"10.18653\/v1\/P17-1161","article-title":"Semi-supervised sequence tagging with bidirectional language models","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Peters","year":"2017"},{"key":"2023062214082517500_bib36","doi-asserted-by":"publisher","first-page":"298","DOI":"10.18653\/v1\/2021.emnlp-main.26","article-title":"Narrative theory for computational narrative understanding","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Piper","year":"2021"},{"key":"2023062214082517500_bib37","first-page":"57","article-title":"Two-stage movie script summarization: An efficient method for low-resource long document summarization","volume-title":"Proceedings of the Workshop on Automatic Summarization for Creative Writing","author":"Dongqi","year":"2022"},{"key":"2023062214082517500_bib38","doi-asserted-by":"crossref","first-page":"5043","DOI":"10.18653\/v1\/D19-1509","article-title":"Counterfactual story reasoning and generation","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Qin","year":"2019"},{"issue":"8","key":"2023062214082517500_bib39","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2023062214082517500_bib40","doi-asserted-by":"publisher","first-page":"2289","DOI":"10.18653\/v1\/P18-1213","article-title":"Modeling naive psychology of characters in simple commonsense stories","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Rashkin","year":"2018"},{"key":"2023062214082517500_bib41","doi-asserted-by":"publisher","first-page":"4274","DOI":"10.18653\/v1\/2020.emnlp-main.349","article-title":"PlotMachines: Outline-conditioned generation with dynamic plot state tracking","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Rashkin","year":"2020"},{"key":"2023062214082517500_bib42","article-title":"Simple BERT models for relation extraction and semantic role labeling","volume":"abs\/1904.05255","author":"Shi","year":"2019","journal-title":"CoRR"},{"key":"2023062214082517500_bib43","first-page":"4111","article-title":"Cohere: A toolkit for local coherence","volume-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916)","author":"Smith","year":"2016"},{"key":"2023062214082517500_bib44","doi-asserted-by":"publisher","first-page":"4566","DOI":"10.1109\/CVPR.2015.7299087","article-title":"CIDEr: Consensus-based image description evaluation","volume-title":"2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Vedantam","year":"2015"},{"key":"2023062214082517500_bib45","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-020-0772-5","article-title":"SciPy 1.0: Fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nature Methods"},{"key":"2023062214082517500_bib46","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2023062214082517500_bib47","doi-asserted-by":"publisher","first-page":"4591","DOI":"10.1109\/ICCV.2019.00469","article-title":"A graph-based framework to bridge movies and synopses","volume-title":"2019 IEEE\/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 \u2013 November 2, 2019","author":"Xiong","year":"2019"},{"key":"2023062214082517500_bib48","first-page":"2831","article-title":"MEGATRON-CNTRL: Controllable story generation with external knowledge using large-scale language models","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Peng","year":"2020"},{"issue":"01","key":"2023062214082517500_bib49","doi-asserted-by":"publisher","first-page":"7378","DOI":"10.1609\/aaai.v33i01.33017378","article-title":"Plan-and-write: Towards better automatic storytelling","volume":"33","author":"Yao","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2023062214082517500_bib50","first-page":"12653","article-title":"Transitional adaptation of pretrained models for visual storytelling","volume-title":"2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Youngjae","year":"2021"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00553\/2134487\/tacl_a_00553.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00553\/2134487\/tacl_a_00553.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,22]],"date-time":"2023-06-22T22:23:58Z","timestamp":1687472638000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00553\/116468\/Visual-Writing-Prompts-Character-Grounded-Story"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":50,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00553","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}