{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T13:36:03Z","timestamp":1770816963034,"version":"3.50.1"},"reference-count":48,"publisher":"MIT Press - Journals","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2013,6]]},"abstract":"<jats:p> The most widely adopted approaches for evaluation of summary content follow some protocol for comparing a summary with gold-standard human summaries, which are traditionally called model summaries. This evaluation paradigm falls short when human summaries are not available and becomes less accurate when only a single model is available. We propose three novel evaluation techniques. Two of them are model-free and do not rely on a gold standard for the assessment. The third technique improves standard automatic evaluations by expanding the set of available model summaries with chosen system summaries. <\/jats:p><jats:p> We show that quantifying the similarity between the source text and its summary with appropriately chosen measures produces summary scores which replicate human assessments accurately. We also explore ways of increasing evaluation quality when only one human model summary is available as a gold standard. We introduce pseudomodels, which are system summaries deemed to contain good content according to automatic evaluation. Combining the pseudomodels with the single human model to form the gold-standard leads to higher correlations with human judgments compared to using only the one available model. Finally, we explore the feasibility of another measure\u2014similarity between a system summary and the pool of all other system summaries for the same input. This method of comparison with the consensus of systems produces impressively accurate rankings of system summaries, achieving correlation with human rankings above 0.9. <\/jats:p>","DOI":"10.1162\/coli_a_00123","type":"journal-article","created":{"date-parts":[[2012,8,22]],"date-time":"2012-08-22T13:28:44Z","timestamp":1345642124000},"page":"267-300","source":"Crossref","is-referenced-by-count":61,"title":["Automatically Assessing Machine Summary Content Without a Gold Standard"],"prefix":"10.1162","volume":"39","author":[{"given":"Annie","family":"Louis","sequence":"first","affiliation":[{"name":"University of Pennsylvania"}]},{"given":"Ani","family":"Nenkova","sequence":"additional","affiliation":[{"name":"University of Pennsylvania"}]}],"member":"281","reference":[{"key":"R1","first-page":"296","volume-title":"Proceedings of ACL","author":"Albrecht Joshua","year":"2007"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.3115\/1626394.1626424"},{"issue":"3","key":"R3","first-page":"377","volume":"24","author":"Best D. J.","year":"1975","journal-title":"Journal of the Royal Statistical Society. Series C (Applied Statistics)"},{"key":"R4","first-page":"17","volume-title":"Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR","author":"Callison-Burch Chris","year":"2010"},{"key":"R5","first-page":"22","volume-title":"Proceedings of the Sixth Workshop on Statistical Machine Translation","author":"Callison-Burch Chris","year":"2011"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291025"},{"key":"R7","volume-title":"Proceedings of the 4th Document Understanding Conference (DUC'04)","author":"Conroy John M.","year":"2004"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1145\/383952.384042"},{"key":"R9","volume-title":"Proceedings of TAC","author":"Conroy John M.","year":"2011"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.3115\/1273073.1273093"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.3115\/1117575.1117583"},{"key":"R13","first-page":"365","volume-title":"Proceedings of EMNLP","author":"Erkan G\u00fcne\u015f","year":"2004"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.3115\/1611638.1611640"},{"key":"R15","first-page":"148","volume-title":"Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk","author":"Gillick Dan","year":"2010"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383955"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.3115\/1620754.1620807"},{"key":"R18","first-page":"10","volume-title":"Proceedings of the ACL-04 Workshop: Text Summarization Branches Out","author":"Harman Donna","year":"2004"},{"key":"R19","first-page":"60","volume-title":"AAAI Symposium on Intelligent Summarization","author":"Jing Hongyan","year":"1998"},{"key":"R20","first-page":"169","volume-title":"Proceedings of HLT-NAACL","author":"Kumar Shankar","year":"2004"},{"key":"R21","first-page":"1","volume-title":"Proceedings of the NTCIR Workshop","volume":"4","author":"Lin Chin-Yew","year":"2004"},{"key":"R22","first-page":"74","volume-title":"Proceedings of the ACL Text Summarization Workshop","author":"Lin Chin-Yew","year":"2004"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.3115\/1220835.1220894"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.3115\/990820.990892"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.3115\/1073445.1073465"},{"key":"R26","volume-title":"Proceedings of TAC","author":"Louis Annie","year":"2008"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.3115\/1699510.1699550"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.3115\/1609067.1609127"},{"key":"R29","volume-title":"Proceedings of TAC","author":"Louis Annie","year":"2009"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.3115\/1626355.1626371"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-71496-5_51"},{"key":"R32","volume-title":"Proceedings of DUC","author":"McKeown Kathy","year":"2001"},{"key":"R33","volume-title":"Proceedings of the First International Conference on Intelligent Analysis Methods and Tools (IA 2005)","author":"Mihalcea Rada","year":"2005"},{"key":"R34","first-page":"825","volume-title":"Proceedings of ACL-HLT","author":"Nenkova Ani","year":"2008"},{"key":"R35","first-page":"145","volume-title":"Proceedings of HLT-NAACL","author":"Nenkova Ani","year":"2004"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1145\/1233912.1233913"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148269"},{"key":"R38","first-page":"23","volume-title":"Proceedings of the Workshop on Language Generation and Summarisation","author":"Owkzarzak Karolina","year":"2009"},{"key":"R39","volume-title":"R: A Language and Environment for Statistical Computing.","author":"R Development Core Team","year":"2011"},{"key":"R40","first-page":"1","volume-title":"Proceedings of LREC 2004","author":"Radev Dragomir","year":"2004"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2003.10.006"},{"key":"R42","first-page":"508","volume-title":"Proceedings of CIKM","author":"Radev Dragomir","year":"2003"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1002\/asi.5090120210"},{"key":"R44","first-page":"1,059","volume-title":"Proceedings of COLING","author":"Saggion Horacio","year":"2010"},{"key":"R45","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613792"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.3115\/1119467.1119475"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-010-9073-6"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:27:17Z","timestamp":1615584437000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/39\/2\/267-300\/1425"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,6]]},"references-count":48,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2013,6]]}},"alternative-id":["10.1162\/COLI_a_00123"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00123","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,6]]}}}