{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T10:15:01Z","timestamp":1784888101100,"version":"3.55.0"},"reference-count":43,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,11,5]],"date-time":"2021-11-05T00:00:00Z","timestamp":1636070400000},"content-version":"vor","delay-in-days":308,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,10,27]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The quality of a summarization evaluation metric is quantified by calculating the correlation between its scores and human annotations across a large number of summaries. Currently, it is unclear how precise these correlation estimates are, nor whether differences between two metrics\u2019 correlations reflect a true difference or if it is due to mere chance. In this work, we address these two problems by proposing methods for calculating confidence intervals and running hypothesis tests for correlations using two resampling methods, bootstrapping and permutation. After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations. We find that the confidence intervals are rather wide, demonstrating high uncertainty in the reliability of automatic metrics. Further, although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do so in some evaluation settings.1<\/jats:p>","DOI":"10.1162\/tacl_a_00417","type":"journal-article","created":{"date-parts":[[2021,11,5]],"date-time":"2021-11-05T12:55:56Z","timestamp":1636116956000},"page":"1132-1146","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":18,"title":["A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods"],"prefix":"10.1162","volume":"9","author":[{"given":"Daniel","family":"Deutsch","sequence":"first","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, USA. ddeutsch@seas.upenn.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rotem","family":"Dror","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, USA. rtmdrr@seas.upenn.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dan","family":"Roth","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, USA. danroth@seas.upenn.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,10,27]]},"reference":[{"key":"2021110512555008200_bib1","doi-asserted-by":"publisher","first-page":"5715","DOI":"10.18653\/v1\/2020.acl-main.506","article-title":"Not all claims are created equal: Choosing the right statistical approach to assess hypotheses","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Azer","year":"2020"},{"key":"2021110512555008200_bib2","doi-asserted-by":"publisher","first-page":"9347","DOI":"10.18653\/v1\/2020.emnlp-main.751","article-title":"Re-evaluating evaluation in text summarization","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Bhandari","year":"2020"},{"key":"2021110512555008200_bib3","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1007\/BF02294183","article-title":"Sample size requirements for estimating Pearson, Kendall and Spearman correlations","volume":"65","author":"Bonett","year":"2000","journal-title":"Psychometrika"},{"key":"2021110512555008200_bib4","volume-title":"Teoria Statistica Delle Classi e Calcolo Delle Probabilita","author":"Bonferroni","year":"1936"},{"key":"2021110512555008200_bib5","article-title":"Overview of the TAC 2008 Update Summarization Task.","volume-title":"Proc. of the Text Analysis Conference (TAC)","author":"Dang","year":"2008"},{"key":"2021110512555008200_bib6","article-title":"Overview of the TAC 2009 Summarization track","volume-title":"Text Analysis Conference (TAC)","author":"Dang","year":"2009"},{"key":"2021110512555008200_bib7","first-page":"454","volume-title":"OCCAMS\u2013An optimal combinatorial covering algorithm for multi- document summarization","author":"Davis","year":"2012"},{"key":"2021110512555008200_bib8","doi-asserted-by":"publisher","first-page":"376","DOI":"10.3115\/v1\/w14-3348","article-title":"Meteor universal: Language specific translation evaluation for any target language","volume-title":"Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26\u201327, 2014, Baltimore, Maryland, USA","author":"Denkowski","year":"2014"},{"key":"2021110512555008200_bib9","first-page":"9","article-title":"Towards question-answering as an automatic metric for evaluating the content quality of a summary","author":"Deutsch","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021110512555008200_bib10","doi-asserted-by":"publisher","first-page":"120","DOI":"10.18653\/v1\/2020.nlposs-1.17","article-title":"SacreROUGE: An open-source library for using and developing summarization evaluation metrics","volume-title":"Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)","author":"Deutsch","year":"2020"},{"key":"2021110512555008200_bib11","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1162\/tacl_a_00074","article-title":"Replicability analysis for natural language processing: Testing significance with multiple datasets","volume":"5","author":"Dror","year":"2017","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021110512555008200_bib12","doi-asserted-by":"publisher","first-page":"1383","DOI":"10.18653\/v1\/P18-1128","article-title":"The hitchhiker\u2019s guide to testing statistical significance in natural language processing","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Dror","year":"2018"},{"issue":"2","key":"2021110512555008200_bib13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2200\/S00994ED1V01Y202002HLT045","article-title":"Statistical significance testing for natural language processing","volume":"13","author":"Dror","year":"2020","journal-title":"Synthesis Lectures on Human Language Technologies"},{"issue":"336","key":"2021110512555008200_bib14","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1080\/01621459.1971.10482369","article-title":"Comparison of tests of the equality of dependent correlation coefficients","volume":"66","author":"Dunn","year":"1971","journal-title":"Journal of the American Statistical Association"},{"key":"2021110512555008200_bib15","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1971.10482369","volume-title":"An Introduction to the Bootstrap","author":"Efron","year":"1993"},{"key":"2021110512555008200_bib16","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1162\/tacl_a_00373","article-title":"SummEval: Re-evaluating summarization evaluation","volume":"9","author":"Fabbri","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021110512555008200_bib17","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1007\/978-1-4612-4380-9_6","volume-title":"Statistical methods for research workers","author":"Fisher","year":"1992"},{"key":"2021110512555008200_bib18","article-title":"Summarization system evaluation variations based on n-gram graphs","volume-title":"Proceedings of the Third Text Analysis Conference, TAC 2010, Gaithersburg, Maryland, USA, November 15\u201316, 2010","author":"Giannakopoulos","year":"2010"},{"key":"2021110512555008200_bib19","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1007\/978-3-642-37256-8_36","article-title":"Summary evaluation: Together we stand NPowER-ed","volume-title":"Computational Linguistics and Intelligent Text Processing - 14th International Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part II","author":"Giannakopoulos","year":"2013"},{"issue":"3","key":"2021110512555008200_bib20","doi-asserted-by":"publisher","first-page":"5:1","DOI":"10.1145\/1410358.1410359","article-title":"Summarization system evaluation revisited: N-gram graphs","volume":"5","author":"Giannakopoulos","year":"2008","journal-title":"ACM Transactions on Audio, Speech, and Language Processing"},{"key":"2021110512555008200_bib21","first-page":"148","article-title":"Non-expert evaluation of summarization systems is risky","volume-title":"Proceedings of the 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk, Los Angeles, USA, June 6, 2010","author":"Gillick","year":"2010"},{"key":"2021110512555008200_bib22","doi-asserted-by":"publisher","first-page":"128","DOI":"10.18653\/v1\/D15-1013","article-title":"Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Graham","year":"2015"},{"key":"2021110512555008200_bib23","doi-asserted-by":"crossref","first-page":"172","DOI":"10.3115\/v1\/D14-1020","article-title":"Testing for significance of increased correlation with human judgment","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Graham","year":"2014"},{"key":"2021110512555008200_bib24","doi-asserted-by":"publisher","first-page":"74","DOI":"10.3115\/v1\/D14-1020","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2021110512555008200_bib25","doi-asserted-by":"crossref","first-page":"62","DOI":"10.18653\/v1\/W19-5302","article-title":"Results of the WMT19 Metrics Shared Task: Segment-level and strong MT systems pose big challenges","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Ma","year":"2019"},{"key":"2021110512555008200_bib26","first-page":"688","article-title":"Results of the WMT20 Metrics Shared Task","volume-title":"Proceedings of the Fifth Conference on Machine Translation","author":"Mathur","year":"2020"},{"key":"2021110512555008200_bib27","first-page":"280","article-title":"Abstractive text summarization using sequence-to-sequence RNNs and beyond","volume-title":"Proceedings of CoNLL","author":"Nallapati","year":"2016"},{"key":"2021110512555008200_bib28","doi-asserted-by":"publisher","first-page":"21","DOI":"10.18653\/v1\/K16-1028","article-title":"Computer Intensive Methods for Hypothesis Testing: An Introduction","volume":"19","author":"Noreen","year":"1989"},{"key":"2021110512555008200_bib29","first-page":"1","article-title":"An assessment of the accuracy of automatic evaluation in summarization","volume-title":"Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization@NACCL-HLT 2012, Montr\u00e8al, Canada, June 2012, 2012","author":"Owczarzak","year":"2012"},{"key":"2021110512555008200_bib30","article-title":"Overview of the TAC 2011 Summarization Track: Guided task and AESOP task","volume-title":"Proceedings of the Text Analysis Conference (TAC 2011), Gaithersburg, Maryland, USA, November","author":"Owczarzak","year":"2011"},{"key":"2021110512555008200_bib31","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"ACL","author":"Papineni","year":"2011"},{"key":"2021110512555008200_bib32","doi-asserted-by":"publisher","first-page":"74","DOI":"10.18653\/v1\/w17-4510","article-title":"Learning to score system summaries for better content selection evaluation","volume-title":"Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017","author":"Peyrard","year":"2017"},{"key":"2021110512555008200_bib33","first-page":"467","article-title":"Ranking human and machine summarization systems","volume-title":"Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing","author":"Rankel","year":"2011"},{"key":"2021110512555008200_bib34","first-page":"131","article-title":"A decade of automatic content evaluation of news summaries: Reassessing the state of the art","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Rankel","year":"2013"},{"issue":"4","key":"2021110512555008200_bib35","doi-asserted-by":"publisher","first-page":"398","DOI":"10.3390\/a5040398","article-title":"Better metrics to automatically predict the quality of a text summary","volume":"5","author":"Rankel","year":"2012","journal-title":"Algorithms"},{"issue":"1","key":"2021110512555008200_bib36","first-page":"21","article-title":"Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson- Darling Tests","volume":"2","author":"Razali","year":"2011","journal-title":"Journal of Statistical Modeling and Analytics"},{"key":"2021110512555008200_bib37","doi-asserted-by":"publisher","first-page":"682","DOI":"10.18653\/v1\/n19-1072","article-title":"Crowdsourcing lightweight pyramids for manual summary evaluation","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers)","author":"Shapira","year":"2019"},{"issue":"3\/4","key":"2021110512555008200_bib38","doi-asserted-by":"crossref","first-page":"591","DOI":"10.2307\/2333709","article-title":"An Analysis of variance test for normality (complete samples)","volume":"52","author":"Shapiro","year":"1965","journal-title":"Biometrika"},{"key":"2021110512555008200_bib39","article-title":"Summarization evaluation using transformed basic elements","volume-title":"Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17\u201319, 2008","author":"Tratz","year":"2008"},{"key":"2021110512555008200_bib40","doi-asserted-by":"publisher","first-page":"196","DOI":"10.1093\/biomet\/52.3-4.591","article-title":"Individual comparisons by ranking methods","volume-title":"Breakthroughs in Statistics","author":"Wilcoxon","year":"1992"},{"key":"2021110512555008200_bib41","volume-title":"Regression Analysis","author":"Williams","year":"1959"},{"key":"2021110512555008200_bib42","article-title":"BERTScore: Evaluating text generation with BERT","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Zhang","year":"2020"},{"key":"2021110512555008200_bib43","doi-asserted-by":"publisher","first-page":"563","DOI":"10.18653\/v1\/D19-1053","article-title":"MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3\u20137, 2019","author":"Zhao","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00417\/1970948\/tacl_a_00417.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00417\/1970948\/tacl_a_00417.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,5]],"date-time":"2021-11-05T12:56:16Z","timestamp":1636116976000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00417\/107833\/A-Statistical-Analysis-of-Summarization-Evaluation"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":43,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00417","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}