{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T04:30:47Z","timestamp":1783571447493,"version":"3.55.0"},"reference-count":75,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,9,4]],"date-time":"2024-09-04T00:00:00Z","timestamp":1725408000000},"content-version":"vor","delay-in-days":247,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,9,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Multi-document summarization entails producing concise synopses of collections of inputs. For some applications, the synopsis should accurately synthesize inputs with respect to a key aspect, e.g., a synopsis of film reviews written about a particular movie should reflect the average critic consensus. As a more consequential example, narrative summaries that accompany biomedical systematic reviews of clinical trial results should accurately summarize the potentially conflicting results from individual trials. In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this sort of synthesis? We run experiments over opinion and evidence synthesis datasets using a suite of summarization models, from fine-tuned transformers to GPT-4. We find that existing models partially perform synthesis, but imperfectly: Even the best performing models are over-sensitive to changes in input ordering and under-sensitive to changes in input compositions (e.g., ratio of positive to negative reviews). We propose a simple, general, effective method for improving model synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or abstaining when the model produces no good candidate.<\/jats:p>","DOI":"10.1162\/tacl_a_00687","type":"journal-article","created":{"date-parts":[[2024,9,4]],"date-time":"2024-09-04T13:35:27Z","timestamp":1725456927000},"page":"1043-1062","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["Do Multi-Document Summarization Models <i>Synthesize<\/i>?"],"prefix":"10.1162","volume":"12","author":[{"given":"Jay","family":"DeYoung","sequence":"first","affiliation":[{"name":"Northeastern University, Boston, MA, USA. deyoung.j@northeastern.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie C.","family":"Martinez","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, MA, USA. martinez.s@northeastern.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iain J.","family":"Marshall","sequence":"additional","affiliation":[{"name":"King\u2019s College London, London, UK. iain.marshall@kcl.ac.uk"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Byron C.","family":"Wallace","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, MA. b.wallace@northeastern.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,9,4]]},"reference":[{"key":"2024090413342185600_bib1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i14.17481","article-title":"Unsupervised opinion summarization with content planning","volume-title":"AAAI Conference on Artificial Intelligence","author":"Amplayo","year":"2020"},{"key":"2024090413342185600_bib2","doi-asserted-by":"publisher","first-page":"6578","DOI":"10.18653\/v1\/2021.emnlp-main.528","article-title":"Aspect-controllable opinion summarization","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Amplayo","year":"2021"},{"key":"2024090413342185600_bib3","doi-asserted-by":"publisher","first-page":"1934","DOI":"10.18653\/v1\/2020.acl-main.175","article-title":"Unsupervised opinion summarization with noising and denoising","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Amplayo","year":"2020"},{"key":"2024090413342185600_bib4","doi-asserted-by":"publisher","first-page":"2662","DOI":"10.18653\/v1\/2021.eacl-main.229","article-title":"Informative and controllable opinion summarization","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Amplayo","year":"2021"},{"key":"2024090413342185600_bib5","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1162\/tacl_a_00366","article-title":"Extractive opinion summarization in quantized transformer spaces","volume":"9","author":"Angelidis","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024090413342185600_bib6","doi-asserted-by":"publisher","first-page":"3675","DOI":"10.18653\/v1\/D18-1403","article-title":"Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Angelidis","year":"2018"},{"key":"2024090413342185600_bib7","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau","year":"2014","journal-title":"CoRR"},{"key":"2024090413342185600_bib8","article-title":"Neural machine translation by jointly learning to align and translate","volume-title":"International Conference on Learning Representations (ICLR)","author":"Bahdanau","year":"2015"},{"key":"2024090413342185600_bib9","article-title":"Attacked by Rotten Tomatoes","author":"Barnes","year":"2017","journal-title":"The New York Times"},{"issue":"3","key":"2024090413342185600_bib10","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1162\/089120105774321091","article-title":"Sentence fusion for multidocument news summarization","volume":"31","author":"Barzilay","year":"2005","journal-title":"Computational Linguistics"},{"key":"2024090413342185600_bib11","article-title":"Longformer: The long-document transformer","author":"Iz","year":"2020","journal-title":"ArXiv"},{"key":"2024090413342185600_bib12","first-page":"1223","article-title":"MeanSum: A neural model for unsupervised multi-document abstractive summarization","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Chu","year":"2019"},{"key":"2024090413342185600_bib13","article-title":"Scaling instruction-finetuned language models","author":"Chung","year":"2022","journal-title":"ArXiv"},{"key":"2024090413342185600_bib14","article-title":"Overview of DUC 2005","volume-title":"Document Understand Conference","author":"Dang","year":"2005"},{"key":"2024090413342185600_bib15","article-title":"Overview of DUC","volume-title":"Proceedings of HLT-NAACL","author":"Dang","year":"2006"},{"key":"2024090413342185600_bib16","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2024090413342185600_bib17","doi-asserted-by":"publisher","first-page":"7494","DOI":"10.18653\/v1\/2021.emnlp-main.594","article-title":"MS2^: Multi-document summarization of medical studies","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"DeYoung","year":"2021"},{"key":"2024090413342185600_bib18","doi-asserted-by":"publisher","first-page":"123","DOI":"10.18653\/v1\/2020.bionlp-1.13","article-title":"Evidence inference 2.0: More data, better models","volume-title":"Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing","author":"DeYoung","year":"2020"},{"key":"2024090413342185600_bib19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3115\/1614025.1614026","article-title":"Columbia newsblaster: Multilingual news summarization on the web","volume-title":"Demonstration Papers at HLT-NAACL 2004","author":"Evans","year":"2004"},{"key":"2024090413342185600_bib20","doi-asserted-by":"publisher","first-page":"1074","DOI":"10.18653\/v1\/P19-1102","article-title":"Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Fabbri","year":"2019"},{"key":"2024090413342185600_bib21","doi-asserted-by":"publisher","first-page":"2214","DOI":"10.18653\/v1\/P19-1213","article-title":"Ranking generated summaries by correctness: An interesting but challenging application for natural language inference","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Falke","year":"2019"},{"key":"2024090413342185600_bib22","doi-asserted-by":"publisher","first-page":"page 37","DOI":"10.1145\/1015330.1015395","article-title":"Delegating classifiers","volume-title":"Proceedings of the twenty-first international conference on Machine learning","author":"Ferri","year":"2004"},{"key":"2024090413342185600_bib23","doi-asserted-by":"publisher","first-page":"1302","DOI":"10.18653\/v1\/2020.acl-main.120","article-title":"A large-scale multi-document summarization dataset from the Wikipedia current events portal","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Ghalandari","year":"2020"},{"key":"2024090413342185600_bib24","article-title":"Exploring the challenges of open domain multi-document summarization","author":"Giorgi","year":"2022","journal-title":"ArXiv"},{"key":"2024090413342185600_bib25","doi-asserted-by":"publisher","first-page":"388","DOI":"10.18653\/v1\/P17-1036","article-title":"An unsupervised neural attention model for aspect extraction","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"He","year":"2017"},{"key":"2024090413342185600_bib26","article-title":"Cautious deep learning","author":"Hechtlinger","year":"2018","journal-title":"arXiv preprint arXiv:1805.09460"},{"key":"2024090413342185600_bib27","doi-asserted-by":"publisher","first-page":"8488","DOI":"10.18653\/v1\/2023.acl-long.473","article-title":"Attributable and scalable opinion summarization","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Hosking","year":"2023"},{"key":"2024090413342185600_bib28","doi-asserted-by":"publisher","first-page":"28","DOI":"10.18653\/v1\/2021.mrqa-1.3","article-title":"What would it take to get biomedical QA systems into practice?","volume-title":"Proceedings of the 3rd Workshop on Machine Reading for Question Answering","author":"Kell","year":"2021"},{"key":"2024090413342185600_bib29","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"CoRR"},{"key":"2024090413342185600_bib30","doi-asserted-by":"publisher","first-page":"540","DOI":"10.18653\/v1\/D19-1051","article-title":"Neural text summarization: A critical evaluation","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Kryscinski","year":"2019"},{"key":"2024090413342185600_bib31","doi-asserted-by":"publisher","first-page":"9332","DOI":"10.18653\/v1\/2020.emnlp-main.750","article-title":"Evaluating the factual consistency of abstractive text summarization","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Kryscinski","year":"2020"},{"key":"2024090413342185600_bib32","article-title":"Rotten Tomatoes movies and critic reviews dataset","author":"Leone","year":"2020"},{"key":"2024090413342185600_bib33","doi-asserted-by":"publisher","first-page":"7871","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Lewis","year":"2020"},{"key":"2024090413342185600_bib34","first-page":"74","article-title":"Rouge: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2024090413342185600_bib35","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1162\/tacl_a_00115","article-title":"Assessing the ability of LSTMs to learn syntax-sensitive dependencies","volume":"4","author":"Linzen","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024090413342185600_bib36","article-title":"Generating Wikipedia by summarizing long sequences","volume-title":"International Conference on Learning Representations","author":"Liu","year":"2018"},{"key":"2024090413342185600_bib37","doi-asserted-by":"publisher","first-page":"5070","DOI":"10.18653\/v1\/P19-1500","article-title":"Hierarchical transformers for multi-document summarization","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Liu","year":"2019"},{"key":"2024090413342185600_bib38","doi-asserted-by":"publisher","first-page":"8068","DOI":"10.18653\/v1\/2020.emnlp-main.648","article-title":"Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Yao","year":"2020"},{"key":"2024090413342185600_bib39","first-page":"142","article-title":"Learning word vectors for sentiment analysis","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","author":"Maas","year":"2011"},{"key":"2024090413342185600_bib40","doi-asserted-by":"publisher","first-page":"7","DOI":"10.18653\/v1\/P17-4002","article-title":"Automating biomedical evidence synthesis: RobotReviewer","volume-title":"Proceedings of ACL 2017, System Demonstrations","author":"Marshall","year":"2017"},{"key":"2024090413342185600_bib41","volume-title":"Advances in Automatic Text Summarization","author":"Maybury","year":"1999"},{"key":"2024090413342185600_bib42","doi-asserted-by":"publisher","first-page":"1906","DOI":"10.18653\/v1\/2020.acl-main.173","article-title":"On faithfulness and factuality in abstractive summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Maynez","year":"2020"},{"key":"2024090413342185600_bib43","article-title":"Locating and editing factual associations in GPT","volume":"36","author":"Meng","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"9","key":"2024090413342185600_bib44","doi-asserted-by":"crossref","first-page":"503","DOI":"10.4066\/AMJ.2012.1375","article-title":"Creation of a corpus for evidence based medicine summarisation","volume":"5","author":"Moll\u00e1","year":"2012","journal-title":"The Australasian Medical Journal"},{"key":"2024090413342185600_bib45","doi-asserted-by":"publisher","first-page":"2727","DOI":"10.18653\/v1\/2021.eacl-main.235","article-title":"Entity-level factual consistency of abstractive text summarization","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Nan","year":"2021"},{"key":"2024090413342185600_bib46","doi-asserted-by":"publisher","first-page":"6881","DOI":"10.18653\/v1\/2021.acl-long.536","article-title":"Improving factual consistency of abstractive summarization via question answering","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Nan","year":"2021"},{"key":"2024090413342185600_bib47","doi-asserted-by":"publisher","DOI":"10.1561\/9781601984715","volume-title":"Automatic summarization","author":"Nenkova","year":"2011"},{"key":"2024090413342185600_bib48","doi-asserted-by":"publisher","first-page":"4658","DOI":"10.18653\/v1\/P19-1459","article-title":"Probing neural network comprehension of natural language arguments","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Niven","year":"2019"},{"key":"2024090413342185600_bib49","article-title":"Gpt-4 technical report","author":"OpenAI","year":"2023"},{"key":"2024090413342185600_bib50","doi-asserted-by":"publisher","first-page":"351","DOI":"10.18653\/v1\/2021.acl-long.30","article-title":"PASS: Perturb-and-select summarizer for product reviews","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Oved","year":"2021"},{"key":"2024090413342185600_bib51","doi-asserted-by":"publisher","first-page":"4812","DOI":"10.18653\/v1\/2021.naacl-main.383","article-title":"Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Pagnoni","year":"2021"},{"key":"2024090413342185600_bib52","doi-asserted-by":"publisher","first-page":"115","DOI":"10.3115\/1219840.1219855","article-title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","volume-title":"Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL\u201905)","author":"Bo","year":"2005"},{"key":"2024090413342185600_bib53","doi-asserted-by":"publisher","first-page":"2463","DOI":"10.18653\/v1\/D19-1250","article-title":"Language models as knowledge bases?","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Petroni","year":"2019"},{"issue":"1","key":"2024090413342185600_bib54","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2024090413342185600_bib55","doi-asserted-by":"publisher","first-page":"1179","DOI":"10.1109\/CVPR.2017.131","article-title":"Self-critical sequence training for image captioning","volume-title":"2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Rennie","year":"2017"},{"key":"2024090413342185600_bib56","doi-asserted-by":"publisher","first-page":"6594","DOI":"10.18653\/v1\/2021.emnlp-main.529","article-title":"QuestEval: Summarization asks for fact-based evaluation","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Scialom","year":"2021"},{"key":"2024090413342185600_bib57","doi-asserted-by":"publisher","first-page":"5213","DOI":"10.18653\/v1\/2021.naacl-main.411","article-title":"Nutri-bullets hybrid: Consensual multi-document summarization","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Shah","year":"2021"},{"key":"2024090413342185600_bib58","doi-asserted-by":"publisher","first-page":"13780","DOI":"10.1609\/aaai.v35i15.17624","article-title":"Nutri-bullets: Summarizing health studies by composing segments","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Shah","year":"2021"},{"key":"2024090413342185600_bib59","first-page":"4596","article-title":"Adafactor: Adaptive learning rates with sublinear memory cost","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Shazeer","year":"2018"},{"key":"2024090413342185600_bib60","first-page":"1631","article-title":"Recursive deep models for semantic compositionality over a sentiment treebank","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Socher","year":"2013"},{"key":"2024090413342185600_bib61","doi-asserted-by":"publisher","first-page":"1667","DOI":"10.18653\/v1\/2022.naacl-main.120","article-title":"Improving multi-document summarization through referenced flexible extraction with credit-awareness","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Song","year":"2022"},{"key":"2024090413342185600_bib62","article-title":"What do you learn from context? Probing for sentence structure in contextualized word representations","volume-title":"International Conference on Learning Representations","author":"Tenney","year":"2019"},{"key":"2024090413342185600_bib63","first-page":"1410","article-title":"Supervised sentence fusion with single-stage inference","volume-title":"Proceedings of the Sixth International Joint Conference on Natural Language Processing","author":"Thadani","year":"2013"},{"key":"2024090413342185600_bib64","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024090413342185600_bib65","article-title":"Diverse beam search: Decoding diverse solutions from neural sequence models","author":"Vijayakumar","year":"2016","journal-title":"arXiv preprint arXiv:1610.02424"},{"key":"2024090413342185600_bib66","article-title":"Pointer networks","volume-title":"Advances in Neural Information Processing Systems","author":"Vinyals","year":"2015"},{"key":"2024090413342185600_bib67","article-title":"Generating (Factual?) Narrative summaries of RCTs: Experiments with neural multi-document summarization","volume-title":"Proceedings of AMIA Informatics Summit","author":"Wallace","year":"2021"},{"key":"2024090413342185600_bib68","doi-asserted-by":"publisher","first-page":"5008","DOI":"10.18653\/v1\/2020.acl-main.450","article-title":"Asking and answering questions to evaluate the factual consistency of summaries","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Wang","year":"2020"},{"key":"2024090413342185600_bib69","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2024090413342185600_bib70","doi-asserted-by":"publisher","first-page":"5761","DOI":"10.18653\/v1\/2022.emnlp-main.389","article-title":"How \u201cmulti\u201d is multi-document summarization?","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Wolhandler","year":"2022"},{"key":"2024090413342185600_bib71","doi-asserted-by":"publisher","first-page":"5245","DOI":"10.18653\/v1\/2022.acl-long.360","article-title":"PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Xiao","year":"2022"},{"key":"2024090413342185600_bib72","doi-asserted-by":"publisher","first-page":"6275","DOI":"10.18653\/v1\/2020.emnlp-main.508","article-title":"Understanding neural abstractive summarization models via uncertainty","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Jiacheng","year":"2020"},{"key":"2024090413342185600_bib73","doi-asserted-by":"publisher","first-page":"6925","DOI":"10.18653\/v1\/2021.acl-long.539","article-title":"Dissecting generation modes for abstractive summarization models via ablation and attribution","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Jiacheng","year":"2021"},{"key":"2024090413342185600_bib74","doi-asserted-by":"publisher","first-page":"5071","DOI":"10.18653\/v1\/2020.acl-main.455","article-title":"Fact-based content weighting for evaluating abstractive summarisation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Xinnuo","year":"2020"},{"key":"2024090413342185600_bib75","first-page":"11328","article-title":"PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Zhang","year":"2020"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00687\/2468722\/tacl_a_00687.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00687\/2468722\/tacl_a_00687.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,4]],"date-time":"2024-09-04T13:35:49Z","timestamp":1725456949000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00687\/124262\/Do-Multi-Document-Summarization-Models-Synthesize"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":75,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00687","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}