{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T02:49:48Z","timestamp":1783738188360,"version":"3.55.0"},"reference-count":79,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2021,4,28]],"date-time":"2021-04-28T00:00:00Z","timestamp":1619568000000},"content-version":"vor","delay-in-days":117,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,26]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations; 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics; 3) we assemble the largest collection of summaries generated by models trained on the CNN\/DailyMail news dataset and share it in a unified format; 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics; and 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN\/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.<\/jats:p>","DOI":"10.1162\/tacl_a_00373","type":"journal-article","created":{"date-parts":[[2021,4,28]],"date-time":"2021-04-28T23:52:02Z","timestamp":1619653922000},"page":"391-409","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":281,"title":["SummEval: Re-evaluating Summarization Evaluation"],"prefix":"10.1162","volume":"9","author":[{"given":"Alexander R.","family":"Fabbri","sequence":"first","affiliation":[{"name":"Yale University, United States. alexander.fabbri@yale.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wojciech","family":"Kry\u015bci\u0144ski","sequence":"additional","affiliation":[{"name":"Salesforce Research, United States. kryscinski@salesforce.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bryan","family":"McCann","sequence":"additional","affiliation":[{"name":"Salesforce Research, United States. bryan.mccann.is@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caiming","family":"Xiong","sequence":"additional","affiliation":[{"name":"Salesforce Research, United States. cxiong@salesforce.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Socher","sequence":"additional","affiliation":[{"name":"Salesforce Research, United States. richard@socher.org"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dragomir","family":"Radev","sequence":"additional","affiliation":[{"name":"Yale University, United States"},{"name":"Salesforce Research, United States. dragomir.radev@yale.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,4,26]]},"reference":[{"key":"2021060823395851900_bib1","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau","year":"2014","journal-title":"arXiv preprint arXiv:1409.0473"},{"key":"2021060823395851900_bib2","doi-asserted-by":"crossref","first-page":"3110","DOI":"10.18653\/v1\/D19-1307","article-title":"Better rewards yield better summaries: Learning to summarise without references","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"B\u00f6hm","year":"2019"},{"key":"2021060823395851900_bib3","doi-asserted-by":"crossref","first-page":"243","DOI":"10.18653\/v1\/P19-2034","article-title":"STRASS: A light and effective method for extractive summarization based on sentence embeddings","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop","author":"Bouscarrat","year":"2019"},{"key":"2021060823395851900_bib4","doi-asserted-by":"crossref","first-page":"643","DOI":"10.18653\/v1\/P18-1060","article-title":"The price of debiasing automatic metrics in natural language evaluation","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Chaganty","year":"2018"},{"key":"2021060823395851900_bib5","doi-asserted-by":"crossref","first-page":"675","DOI":"10.18653\/v1\/P18-1063","article-title":"Fast abstractive summarization with reinforce-selected sentence rewriting","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Chen","year":"2018"},{"key":"2021060823395851900_bib6","doi-asserted-by":"crossref","first-page":"2748","DOI":"10.18653\/v1\/P19-1264","article-title":"Sentence mover\u2019s similarity: Automatic evaluation for multi-sentence texts","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Clark","year":"2019"},{"key":"2021060823395851900_bib7","first-page":"806","article-title":"Revisiting summarization evaluation for scientific articles","volume-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916)","author":"Cohan","year":"2016"},{"key":"2021060823395851900_bib8","first-page":"1","article-title":"Overview of DUC 2005","volume-title":"Proceedings of the document understanding conference","author":"Dang","year":"2005"},{"key":"2021060823395851900_bib9","article-title":"Overview of the TAC 2008 update summarization task.","volume-title":"TAC","author":"Dang","year":"2008"},{"key":"2021060823395851900_bib10","article-title":"Overview of the TAC 2009 summarization track","volume-title":"Proceedings of the Text Analysis Conference","author":"Dang","year":"2009"},{"key":"2021060823395851900_bib11","article-title":"A repository of corpora for summarization","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","author":"Dernoncourt","year":"2018"},{"key":"2021060823395851900_bib12","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.nlposs-1.17","article-title":"SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metrics","author":"Deutsch","year":"2020"},{"key":"2021060823395851900_bib13","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2021060823395851900_bib14","first-page":"13042","article-title":"Unified language model pre-training for natural language understanding and generation","volume-title":"Advances in Neural Information Processing Systems","author":"Li","year":"2019"},{"key":"2021060823395851900_bib15","doi-asserted-by":"crossref","first-page":"3739","DOI":"10.18653\/v1\/D18-1409","article-title":"BanditSum: Extractive summarization as a contextual bandit","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Dong","year":"2018"},{"key":"2021060823395851900_bib16","doi-asserted-by":"crossref","first-page":"5055","DOI":"10.18653\/v1\/2020.acl-main.454","article-title":"FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Durmus","year":"2020"},{"key":"2021060823395851900_bib17","article-title":"Rouge 2.0: Updated and improved measures for evaluation of summarization tasks","author":"Ganesan","year":"2015"},{"key":"2021060823395851900_bib18","first-page":"1347","article-title":"SUPERT: towards new frontiers in unsupervised evaluation metrics for multi-document summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020","author":"Gao","year":"2020"},{"key":"2021060823395851900_bib19","doi-asserted-by":"crossref","first-page":"4098","DOI":"10.18653\/v1\/D18-1443","article-title":"Bottom-up abstractive summarization","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Gehrmann","year":"2018"},{"key":"2021060823395851900_bib20","first-page":"148","article-title":"Non-expert evaluation of summarization systems is risky","volume-title":"Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk","author":"Gillick","year":"2010"},{"key":"2021060823395851900_bib21","doi-asserted-by":"crossref","first-page":"128","DOI":"10.18653\/v1\/D15-1013","article-title":"Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Graham","year":"2015"},{"key":"2021060823395851900_bib22","first-page":"708","article-title":"Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Grusky","year":"2018"},{"key":"2021060823395851900_bib23","doi-asserted-by":"crossref","first-page":"687","DOI":"10.18653\/v1\/P18-1064","article-title":"Soft layer-specific multi-task summarization with entailment and question generation","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Guo","year":"2018"},{"key":"2021060823395851900_bib24","doi-asserted-by":"crossref","first-page":"3381","DOI":"10.18653\/v1\/P19-1330","article-title":"HighRES: Highlight-based reference-less evaluation of summarization","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Hardy","year":"2019"},{"key":"2021060823395851900_bib25","first-page":"1693","article-title":"Teaching machines to read and comprehend","volume-title":"Advances in Neural Information Processing Systems","author":"Hermann","year":"2015"},{"key":"2021060823395851900_bib26","doi-asserted-by":"crossref","first-page":"132","DOI":"10.18653\/v1\/P18-1013","article-title":"A unified model for extractive and abstractive summarization using inconsistency loss","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Hsu","year":"2018"},{"key":"2021060823395851900_bib27","doi-asserted-by":"crossref","first-page":"4067","DOI":"10.18653\/v1\/D18-1440","article-title":"Closed-book training to improve summarization encoder memory","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Jiang","year":"2018"},{"key":"2021060823395851900_bib28","doi-asserted-by":"crossref","first-page":"1818","DOI":"10.18653\/v1\/D18-1208","article-title":"Content selection in deep learning models of summarization","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Kedzie","year":"2018"},{"key":"2021060823395851900_bib29","article-title":"Computing krippendorff\u2019s alpha-reliability","author":"Krippendorff","year":"2011"},{"key":"2021060823395851900_bib30","doi-asserted-by":"crossref","first-page":"540","DOI":"10.18653\/v1\/D19-1051","article-title":"Neural text summarization: A critical evaluation","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Kry\u015bci\u0144ski","year":"2019"},{"key":"2021060823395851900_bib31","doi-asserted-by":"crossref","first-page":"9332","DOI":"10.18653\/v1\/2020.emnlp-main.750","article-title":"Evaluating the factual consistency of abstractive text summarization","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Kry\u015bci\u0144ski","year":"2020"},{"key":"2021060823395851900_bib32","doi-asserted-by":"crossref","first-page":"1808","DOI":"10.18653\/v1\/D18-1207","article-title":"Improving abstraction in text summarization","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Kry\u015bci\u0144ski","year":"2018"},{"key":"2021060823395851900_bib33","first-page":"957","article-title":"From word embeddings to document distances","volume-title":"International Conference on Machine Learning","author":"Kusner","year":"2015"},{"key":"2021060823395851900_bib34","doi-asserted-by":"crossref","first-page":"228","DOI":"10.3115\/1626355.1626389","article-title":"METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments","volume-title":"Proceedings of the Second Workshop on Statistical Machine Translation","author":"Lavie","year":"2007"},{"key":"2021060823395851900_bib35","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","author":"Lewis","year":"2019","journal-title":"arXiv preprint arXiv:1910.13461"},{"key":"2021060823395851900_bib36","article-title":"Looking for a few good metrics: Automatic summarization evaluation-how many samples are enough?","volume-title":"NTCIR","author":"Lin","year":"2004"},{"key":"2021060823395851900_bib37","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2021060823395851900_bib38","first-page":"201","article-title":"Correlation between ROUGE and human evaluation of extractive meeting summaries","volume-title":"Proceedings of ACL-08: HLT, Short Papers","author":"Liu","year":"2008"},{"key":"2021060823395851900_bib39","first-page":"3730","article-title":"Text summarization with pretrained encoders","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Liu","year":"2019"},{"issue":"2","key":"2021060823395851900_bib40","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1162\/COLI_a_00123","article-title":"Automatically assessing machine summary content without a gold standard","volume":"39","author":"Louis","year":"2013","journal-title":"Computational Linguistics"},{"key":"2021060823395851900_bib41","first-page":"1906","article-title":"On faithfulness and factuality in abstractive summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\u201310, 2020","author":"Maynez","year":"2020"},{"key":"2021060823395851900_bib42","first-page":"3111","article-title":"Distributed representations of words and phrases and their compositionality","volume-title":"Advances in Neural Information Processing Systems 26","author":"Mikolov","year":"2013"},{"key":"2021060823395851900_bib43","article-title":"Abstractive text summarization using sequence-to-sequence rnns and beyond","author":"Nallapati","year":"2016","journal-title":"arXiv preprint arXiv: 1602.06023"},{"key":"2021060823395851900_bib44","first-page":"1747","article-title":"Ranking sentences for extractive summarization with reinforcement learning","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Narayan","year":"2018"},{"key":"2021060823395851900_bib45","doi-asserted-by":"crossref","first-page":"1925","DOI":"10.18653\/v1\/D15-1222","article-title":"Better summarization evaluation with word embeddings for ROUGE","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Ng","year":"2015"},{"key":"2021060823395851900_bib46","first-page":"359","article-title":"Assessing the effect of inconsistent assessors on summarization evaluation","volume-title":"The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8\u201314, 2012, Jeju Island, Korea - Volume 2: Short Papers","author":"Owczarzak","year":"2012"},{"key":"2021060823395851900_bib47","first-page":"311","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2021060823395851900_bib48","first-page":"646","article-title":"Multi-reward reinforced summarization with saliency and entailment","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Pasunuru","year":"2018"},{"key":"2021060823395851900_bib49","article-title":"A deep reinforced model for abstractive summarization","author":"Paulus","year":"2017","journal-title":"arXiv preprint arXiv:1705.04304"},{"key":"2021060823395851900_bib50","doi-asserted-by":"crossref","first-page":"5093","DOI":"10.18653\/v1\/P19-1502","article-title":"Studying summarization evaluation metrics in the appropriate scoring range","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Peyrard","year":"2019"},{"key":"2021060823395851900_bib51","doi-asserted-by":"crossref","first-page":"74","DOI":"10.18653\/v1\/W17-4510","article-title":"Learning to score system summaries for better content selection evaluation.","volume-title":"Proceedings of the Workshop on New Frontiers in Summarization","author":"Peyrard","year":"2017"},{"key":"2021060823395851900_bib52","doi-asserted-by":"crossref","first-page":"392","DOI":"10.18653\/v1\/W15-3049","article-title":"chrF: character n-gram F-score for automatic MT evaluation","volume-title":"Proceedings of the Tenth Workshop on Statistical Machine Translation","author":"Popovi\u0107","year":"2015"},{"issue":"8","key":"2021060823395851900_bib53","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2021060823395851900_bib54","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel","year":"2019","journal-title":"arXiv e-prints"},{"key":"2021060823395851900_bib55","first-page":"131","article-title":"A decade of automatic content evaluation of news summaries: Reassessing the state of the art","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Rankel","year":"2013"},{"issue":"12","key":"2021060823395851900_bib56","first-page":"e26752","article-title":"The New York Times annotated corpus","volume":"6","author":"Sandhaus","year":"2008","journal-title":"Linguistic Data Consortium, Philadelphia"},{"key":"2021060823395851900_bib57","doi-asserted-by":"crossref","first-page":"3246","DOI":"10.18653\/v1\/D19-1320","article-title":"Answers unite! unsupervised metrics for reinforced summarization models","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Scialom","year":"2019"},{"key":"2021060823395851900_bib58","doi-asserted-by":"crossref","first-page":"1073","DOI":"10.18653\/v1\/P17-1099","article-title":"Get to the point: Summarization with pointer-generator networks","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"See","year":"2017"},{"key":"2021060823395851900_bib59","doi-asserted-by":"crossref","first-page":"762","DOI":"10.18653\/v1\/D18-1085","article-title":"A graph-theoretic summary evaluation for ROUGE","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"ShafieiBavani","year":"2018"},{"key":"2021060823395851900_bib60","doi-asserted-by":"crossref","first-page":"3280","DOI":"10.18653\/v1\/D19-1323","article-title":"An entity-driven framework for abstractive summarization","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Sharma","year":"2019"},{"key":"2021060823395851900_bib61","article-title":"Learning to summarize from human feedback","author":"Stiennon","year":"2020","journal-title":"CoRR"},{"key":"2021060823395851900_bib62","first-page":"3104","article-title":"Sequence to sequence learning with neural networks","volume-title":"Advances in Neural Information processing Systems","author":"Sutskever","year":"2014"},{"key":"2021060823395851900_bib63","article-title":"Fill in the BLANC: human-free quality estimation of document summaries","author":"Vasilyev","year":"2020","journal-title":"CoRR"},{"key":"2021060823395851900_bib64","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2021060823395851900_bib65","first-page":"4566","article-title":"CIDEr: Consensus-based image description evaluation","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Vedantam","year":"2015"},{"key":"2021060823395851900_bib66","first-page":"2692","article-title":"Pointer networks","volume-title":"Advances in Neural Information Processing Systems","author":"Vinyals","year":"2015"},{"key":"2021060823395851900_bib67","doi-asserted-by":"crossref","first-page":"5008","DOI":"10.18653\/v1\/2020.acl-main.450","article-title":"Asking and answering questions to evaluate the factual consistency of summaries","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Wang","year":"2020"},{"issue":"3\u20134","key":"2021060823395851900_bib68","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1007\/BF00992696","article-title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Machine Learning"},{"key":"2021060823395851900_bib69","article-title":"Learning to extract coherent summary via deep reinforcement learning","volume-title":"Thirty-Second AAAI Conference on Artificial Intelligence","author":"Yuxiang","year":"2018"},{"key":"2021060823395851900_bib70","first-page":"3292","article-title":"Neural extractive text summarization with syntactic compression","volume-title":"EMNLP-IJCNLP 2019","author":"Jiacheng","year":"2019"},{"key":"2021060823395851900_bib71","doi-asserted-by":"crossref","first-page":"785","DOI":"10.18653\/v1\/D18-1089","article-title":"On the abstractiveness of neural document summarization","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Zhang","year":"2018"},{"key":"2021060823395851900_bib72","article-title":"Pegasus: Pre-training with extracted gap-sentences for abstractive summarization","author":"Zhang","year":"2019","journal-title":"arXiv preprint arXiv:1912.08777"},{"key":"2021060823395851900_bib73","article-title":"Bertscore: Evaluating text generation with BERT","volume-title":"International Conference on Learning Representations","author":"Zhang","year":"2020"},{"key":"2021060823395851900_bib74","doi-asserted-by":"crossref","first-page":"779","DOI":"10.18653\/v1\/D18-1088","article-title":"Neural latent extractive document summarization","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Zhang","year":"2018"},{"key":"2021060823395851900_bib75","doi-asserted-by":"crossref","first-page":"5059","DOI":"10.18653\/v1\/P19-1499","article-title":"HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Zhang","year":"2019"},{"key":"2021060823395851900_bib76","first-page":"563","article-title":"MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance","volume-title":"EMNLP-IJCNLP 2019","author":"Zhao","year":"2019"},{"key":"2021060823395851900_bib77","first-page":"447","article-title":"ParaEval: Using paraphrases to evaluate summaries automatically","volume-title":"Proceedings of the Human Language Technology Conference of the NAACL, Main Conference","author":"Zhou","year":"2006"},{"key":"2021060823395851900_bib78","first-page":"654","article-title":"Neural document summarization by jointly learning to score and select sentences","volume-title":"ACL 2018","author":"Zhou","year":"2018"},{"key":"2021060823395851900_bib79","article-title":"Fine-tuning language models from human preferences","author":"Ziegler","year":"2019","journal-title":"arXiv preprint arXiv:1909.08593"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00373\/1923949\/tacl_a_00373.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00373\/1923949\/tacl_a_00373.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,25]],"date-time":"2022-12-25T19:35:51Z","timestamp":1671996951000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00373\/100686\/SummEval-Re-evaluating-Summarization-Evaluation"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":79,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00373","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}