{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,26]],"date-time":"2026-04-26T05:26:34Z","timestamp":1777181194152,"version":"3.51.4"},"reference-count":30,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,8,6]],"date-time":"2021-08-06T00:00:00Z","timestamp":1628208000000},"content-version":"vor","delay-in-days":217,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary\u2019s information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.1<\/jats:p>","DOI":"10.1162\/tacl_a_00397","type":"journal-article","created":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T20:09:31Z","timestamp":1632168571000},"page":"774-789","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":27,"title":["Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary"],"prefix":"10.1162","volume":"9","author":[{"given":"Daniel","family":"Deutsch","sequence":"first","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, United States. ddeutsch@seas.upenn.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tania","family":"Bedrax-Weiss","sequence":"additional","affiliation":[{"name":"Google Research, United States. tbedrax@google.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dan","family":"Roth","sequence":"additional","affiliation":[{"name":"Department of Computer and Information Science, University of Pennsylvania, United States. danroth@seas.upenn.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2021,8,2]]},"reference":[{"key":"2021080620253931500_bib1","doi-asserted-by":"publisher","first-page":"9347","DOI":"10.18653\/v1\/2020.emnlp-main.751","article-title":"Re-evaluating evaluation in text summarization","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Bhandari","year":"2020"},{"key":"2021080620253931500_bib2","doi-asserted-by":"publisher","first-page":"6521","DOI":"10.18653\/v1\/2020.emnlp-main.528","article-title":"MOCHA: A dataset for training and evaluating generative reading comprehension metrics","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16\u201320, 2020","author":"Chen","year":"2020"},{"key":"2021080620253931500_bib3","article-title":"ELECTRA: Pre-training text encoders as discriminators rather than generators","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Clark","year":"2020"},{"key":"2021080620253931500_bib4","article-title":"Overview of the TAC 2008 Update Summarization Task","volume-title":"Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17\u201319, 2008","author":"Dang","year":"2008"},{"key":"2021080620253931500_bib5","article-title":"Overview of the TAC 2009 Summarization Track","volume-title":"Proceedings of the Text Analysis Conference","author":"Dang","year":"2009"},{"key":"2021080620253931500_bib6","article-title":"Transforming question answering datasets into natural language inference datasets","author":"Demszky","year":"2018","journal-title":"CoRR"},{"key":"2021080620253931500_bib7","doi-asserted-by":"publisher","first-page":"376","DOI":"10.3115\/v1\/W14-3348","article-title":"Meteor Universal: Language specific translation evaluation for any target language","volume-title":"Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26\u201327, 2014, Baltimore, Maryland, USA","author":"Denkowski","year":"2014"},{"key":"2021080620253931500_bib8","article-title":"A Statistical analysis of summarization evaluation metrics using resampling methods","author":"Deutsch","year":"2021","journal-title":"CoRR"},{"key":"2021080620253931500_bib9","article-title":"Understanding the extent to which summarization evaluation metrics measure the information quality of summaries","author":"Deutsch","year":"2020","journal-title":"CoRR"},{"key":"2021080620253931500_bib10","doi-asserted-by":"publisher","first-page":"5055","DOI":"10.18653\/v1\/2020.acl-main.454","article-title":"FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\u201310, 2020","author":"Durmus","year":"2020"},{"key":"2021080620253931500_bib11","first-page":"3938","article-title":"Question answering as an automatic evaluation metric for news article summarization","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers)","author":"Eyal","year":"2019"},{"key":"2021080620253931500_bib12","article-title":"SummEval: Re-evaluating summarization evaluation","author":"Fabbri","year":"2020","journal-title":"CoRR"},{"key":"2021080620253931500_bib13","first-page":"404","article-title":"Automated Pyramid summarization evaluation","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3\u20134, 2019","author":"Gao","year":"2019"},{"issue":"3","key":"2021080620253931500_bib14","doi-asserted-by":"publisher","first-page":"5:1","DOI":"10.1145\/1410358.1410359","article-title":"Summarization system evaluation revisited: N-gram graphs","volume":"5","author":"Giannakopoulos","year":"2008","journal-title":"ACM Transactions on Speech and Language Processing"},{"key":"2021080620253931500_bib15","first-page":"899","article-title":"Automated summarization evaluation with basic elements","volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation, LREC 2006, Genoa, Italy, May 22\u201328, 2006","author":"Hovy","year":"2006"},{"key":"2021080620253931500_bib16","doi-asserted-by":"publisher","first-page":"7871","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\u201310, 2020","author":"Lewis","year":"2020"},{"key":"2021080620253931500_bib17","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"issue":"2","key":"2021080620253931500_bib18","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1162\/COLI_a_00123","article-title":"Automatically assessing machine summary content without a gold standard","volume":"39","author":"Louis","year":"2013","journal-title":"Computational Linguistics"},{"key":"2021080620253931500_bib19","doi-asserted-by":"publisher","first-page":"280","DOI":"10.18653\/v1\/K16-1028","article-title":"Abstractive text summarization using sequence-to-sequence RNNs and beyond","volume-title":"Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11\u201312, 2016","author":"Nallapati","year":"2016"},{"key":"2021080620253931500_bib20","doi-asserted-by":"publisher","first-page":"1747","DOI":"10.18653\/v1\/N18-1158","article-title":"Ranking sentences for extractive summarization with reinforcement learning","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Narayan","year":"2018"},{"key":"2021080620253931500_bib21","first-page":"145","article-title":"Evaluating content selection in summarization: The pyramid method","volume-title":"Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May 2\u20137, 2004","author":"Nenkova","year":"2004"},{"key":"2021080620253931500_bib22","doi-asserted-by":"publisher","first-page":"74","DOI":"10.18653\/v1\/W17-4510","article-title":"Learning to score system summaries for better content selection evaluation","volume-title":"Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017","author":"Peyrard","year":"2017"},{"key":"2021080620253931500_bib23","doi-asserted-by":"publisher","first-page":"784","DOI":"10.18653\/v1\/P18-2124","article-title":"Know what you don\u2019t know: Unanswerable questions for SQuAD","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15\u201320, 2018, Volume 2: Short Papers","author":"Rajpurkar","year":"2018"},{"key":"2021080620253931500_bib24","first-page":"2383","article-title":"SQuAD: 100,000+ questions for machine comprehension of text","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1\u20134, 2016","author":"Rajpurkar","year":"2016"},{"key":"2021080620253931500_bib25","first-page":"682","article-title":"Crowdsourcing lightweight Pyramids for manual summary evaluation","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers)","author":"Shapira","year":"2019"},{"key":"2021080620253931500_bib26","article-title":"Summarization evaluation using transformed basic elements","volume-title":"Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17\u201319, 2008","author":"Tratz","year":"2008"},{"key":"2021080620253931500_bib27","doi-asserted-by":"publisher","first-page":"5008","DOI":"10.18653\/v1\/2020.acl-main.450","article-title":"Asking and answering questions to evaluate the factual consistency of summaries","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\u201310, 2020","author":"Wang","year":"2020"},{"key":"2021080620253931500_bib28","first-page":"2673","article-title":"PEAK: Pyramid Evaluation via Automated Knowledge Extraction","volume-title":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12\u201317, 2016, Phoenix, Arizona, USA","author":"Yang","year":"2016"},{"key":"2021080620253931500_bib29","article-title":"BERTScore: Evaluating Text Generation with BERT","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Zhang","year":"2020"},{"key":"2021080620253931500_bib30","doi-asserted-by":"publisher","first-page":"563","DOI":"10.18653\/v1\/D19-1053","article-title":"MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3\u20137, 2019","author":"Zhao","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00397\/1955159\/tacl_a_00397.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00397\/1955159\/tacl_a_00397.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T20:09:56Z","timestamp":1632168596000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00397\/106792\/Towards-Question-Answering-as-an-Automatic-Metric"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":30,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00397","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}