{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T22:11:09Z","timestamp":1760220669308,"version":"build-2065373602"},"reference-count":21,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2012,9,26]],"date-time":"2012-09-26T00:00:00Z","timestamp":1348617600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>In this paper we demonstrate a family of metrics for estimating the quality of a text summary relative to one or more human-generated summaries. The improved metrics are based on features automatically computed from the summaries to measure content and linguistic quality. The features are combined using one of three methods\u2014robust regression, non-negative least squares, or canonical correlation, an eigenvalue method. The new metrics significantly outperform the previous standard for automatic text summarization evaluation, ROUGE.<\/jats:p>","DOI":"10.3390\/a5040398","type":"journal-article","created":{"date-parts":[[2012,9,27]],"date-time":"2012-09-27T19:44:07Z","timestamp":1348775047000},"page":"398-420","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Better Metrics to Automatically Predict the Quality of a Text Summary"],"prefix":"10.3390","volume":"5","author":[{"given":"Peter A.","family":"Rankel","sequence":"first","affiliation":[{"name":"Statistics Program, Department of Mathematics, University of Maryland, College Park, MD 20742, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John M.","family":"Conroy","sequence":"additional","affiliation":[{"name":"Center for Computing Sciences, Institute for Defense Analyses, 17100 Science Drive, Bowie, MD 20715, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Judith D.","family":"Schlesinger","sequence":"additional","affiliation":[{"name":"Center for Computing Sciences, Institute for Defense Analyses, 17100 Science Drive, Bowie, MD 20715, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2012,9,26]]},"reference":[{"key":"ref_1","unstructured":"Luhn, H.P. (1956). Advances in Automatic Text Summarization, The MIT Press."},{"key":"ref_2","unstructured":"McKeown, K., and Radev, D.R. Generating Summaries of Multiple News Articles. Proceedings of the 18th Annual International ACM SIGIR Conference On Research and Development in Information Retrieval, SIGIR \u201995."},{"key":"ref_3","unstructured":"Text Analysis Conference, NIST, 2011. Available online:http:\/\/www.nist.gov\/tac."},{"key":"ref_4","unstructured":"Lin, C.Y. ROUGE: A Package for Automatic Evaluation of Summaries. Proceedings of the ACL-04 Workshop: Text Summarization Branches Out."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Conroy, J.M., and Dang, H.T. (2008, January 18\u201322). Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality. Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), Stroudsburg, PA, USA.","DOI":"10.3115\/1599081.1599100"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/coli_a_00033","article-title":"Nouveau-ROUGE: A novelty metric for update summarization","volume":"37","author":"Conroy","year":"2011","journal-title":"Comput. Linguist."},{"key":"ref_7","unstructured":"De Oliveira, P.C.F., Torrens, E.W., Cidral, A., Schossland, S., and Bittencourt, E. Evaluating Summaries Automatically-A system Proposal. Available online:http:\/\/www.lrecconf. org\/proceedings\/lrec2008\/."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1410358.1410359","article-title":"Summarization system evaluation revisited: N-gram graphs","volume":"5","author":"Giannakopoulos","year":"2008","journal-title":"Trans. Speech Lang. Process."},{"key":"ref_9","unstructured":"Giannakopoulos, G., Vouros, G.A., and Karkaletsis, V. MUDOS-NG: Multi-Document Summaries Using N-gram Graphs (Technical Report). arXiv:1012.2042."},{"key":"ref_10","unstructured":"Giannakopoulos, G., and Karkaletsis, V. (2011, January 14\u201315). AutoSummENG and MeMoG in Evaluating Guided Summaries. Proceedings of the Text Analysis Conference (TAC 2011); NIST, Gaithersburg, MD, USA."},{"key":"ref_11","unstructured":"Pitler, E., Louis, A., and Nenkova, A. (2010, January 11\u201316). Automatic Evaluation of Linguistic Quality in Multi-Document Summarization. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics, Uppsala, Sweden."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1007\/978-3-642-28601-8_30","article-title":"Using graph based mapping of co-occurring words and closeness centrality score for summarization evaluation","volume":"7182","author":"Kumar","year":"2012","journal-title":"Comput. Linguist. Intell. Text Process."},{"key":"ref_13","first-page":"251","article-title":"Evaluation measures for text summarization","volume":"28","author":"Steinberger","year":"2012","journal-title":"Comput. Inf."},{"key":"ref_14","unstructured":"Saggion, H., Torres-Moreno, J., Cunha, I., and SanJuan, E. (2010, January 23\u201327). Multilingual Summarization Evaluation Without Human Models. Proceedings of the 23rd International Conference on Computational Linguistics: Posters. Association for Computational Linguistics, Stroudsburg, PA, USA."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Louis, A., and Nenkova, A. (2009, January 6\u20137). Automatically Evaluating Content Selection in Summarization without Human Models. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics, Singapore, Singapore.","DOI":"10.3115\/1699510.1699550"},{"key":"ref_16","unstructured":"Document Understanding Conference, NIST, 2004. Available online:http:\/\/duc.nist.gov."},{"key":"ref_17","unstructured":"Over, P. Introduction to DUC-2001: An Intrinsic Evaluation of Generic News Text Summarization Systems, Technical report, Retrieval Group, Information Access Division, National Institute of Standards and Technology, 2001."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1233912.1233913","article-title":"The pyramid method: Incorporating human content selection variation in summarization evaluation","volume":"4","author":"Nenkova","year":"2007","journal-title":"ACM Trans. Speech Lang. Process."},{"key":"ref_19","unstructured":"Conroy, J.M., Schlesinger, J.D., Rankel, P.A., and O\u2019Leary, D.P. Guiding CLASSY Toward More Responsive Summaries. Available online:http:\/\/www.nist.gov\/tac\/publications\/index.html."},{"key":"ref_20","unstructured":"Seber, G. (2004). Multivariate Observations (Wiley series in probability and statistics), Wiley-Interscience."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tavernier, J., and Bellot, P. Combining Relevance and Readability for INEX 2011 Question-Answering Track. Pre-Proceedings of INEX 2011.","DOI":"10.1007\/978-3-642-35734-3_22"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/5\/4\/398\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:52:34Z","timestamp":1760219554000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/5\/4\/398"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,9,26]]},"references-count":21,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2012,12]]}},"alternative-id":["a5040398"],"URL":"https:\/\/doi.org\/10.3390\/a5040398","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2012,9,26]]}}}