{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T16:29:08Z","timestamp":1771000148447,"version":"3.50.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2008,10,1]],"date-time":"2008-10-01T00:00:00Z","timestamp":1222819200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2008,10]]},"abstract":"<jats:p>This article presents a novel automatic method (AutoSummENG) for the evaluation of summarization systems, based on comparing the character n-gram graphs representation of the extracted summaries and a number of model summaries. The presented approach is language neutral, due to its statistical nature, and appears to hold a level of evaluation performance that matches and even exceeds other contemporary evaluation methods. Within this study, we measure the effectiveness of different representation methods, namely, word and character n-gram graph and histogram, different n-gram neighborhood indication methods as well as different comparison methods between the supplied representations. A theory for the a priori determination of the methods' parameters along with supporting experiments concludes the study to provide a complete alternative to existing methods concerning the automatic summary system evaluation process.<\/jats:p>","DOI":"10.1145\/1410358.1410359","type":"journal-article","created":{"date-parts":[[2008,10,14]],"date-time":"2008-10-14T13:15:19Z","timestamp":1223990119000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":77,"title":["Summarization system evaluation revisited"],"prefix":"10.1145","volume":"5","author":[{"given":"George","family":"Giannakopoulos","sequence":"first","affiliation":[{"name":"National Centre for Scientific Research Demokritos, Demokritos, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vangelis","family":"Karkaletsis","sequence":"additional","affiliation":[{"name":"National Centre for Scientific Research Demokritos, Demokritos, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"George","family":"Vouros","sequence":"additional","affiliation":[{"name":"University of the Aegean"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Panagiotis","family":"Stamatopoulos","sequence":"additional","affiliation":[{"name":"University of Athens"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2008,10,13]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Cognitive Psychology and Its Implications","author":"Anderson J.","unstructured":"Anderson , J. 1985. Cognitive Psychology and Its Implications , 2 nd ed. WH Freeman . Anderson, J. 1985. Cognitive Psychology and Its Implications, 2nd ed. WH Freeman.","edition":"2"},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Banko M. and Vanderwende L. 2004. Using n-grams to understand the nature of summaries. In HLT-NAACL 2004: Short Papers D. M. Susan Dumais and S. Roukos Eds. Association for Computational Linguistics Boston Massachusetts USA 1--4.   Banko M. and Vanderwende L. 2004. Using n-grams to understand the nature of summaries. In HLT-NAACL 2004: Short Papers D. M. Susan Dumais and S. Roukos Eds. Association for Computational Linguistics Boston Massachusetts USA 1--4.","DOI":"10.3115\/1613984.1613985"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.5555\/645888.673056"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of SDAIR-94","author":"Cavnar W. B.","unstructured":"Cavnar , W. B. and Trenkle , J. M . 1994. N-gram-based text categorization . In Proceedings of SDAIR-94 , 3rd Annual Symposium on Document Analysis and Information Retrieval. Las Vegas, US, 161--175. Cavnar, W. B. and Trenkle, J. M. 1994. N-gram-based text categorization. In Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval. Las Vegas, US, 161--175."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.2307\/2683591"},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Conroy J. and Dang H. T. 2008. Mind the gap: dangers of divorcing evaluations of summary content from linguistic quality. In Coling 2008 (to appear).   Conroy J. and Dang H. T. 2008. Mind the gap: dangers of divorcing evaluations of summary content from linguistic quality. In Coling 2008 (to appear).","DOI":"10.3115\/1599081.1599100"},{"key":"e_1_2_1_7_1","volume-title":"Vocabulary Usage in Newswire Summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. Association for Computational Linguistics, 19--26","author":"Copeck T.","unstructured":"Copeck , T. and Szpakowicz , S . 2004 . Vocabulary Usage in Newswire Summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. Association for Computational Linguistics, 19--26 . Copeck, T. and Szpakowicz, S. 2004. Vocabulary Usage in Newswire Summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. Association for Computational Linguistics, 19--26."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 9th Portuguese Conference on Artificial Intelligence: Progress in Artificial Intelligence, 113--132","author":"da Silva J.","unstructured":"da Silva , J. , Dias , G. , Guillor\u00e9 , S. , and Lopes , J . 1999. Using LocalMaxs algorithm for the extraction of contiguous and non-contiguous multiword lexical units . Proceedings of the 9th Portuguese Conference on Artificial Intelligence: Progress in Artificial Intelligence, 113--132 . da Silva, J., Dias, G., Guillor\u00e9, S., and Lopes, J. 1999. Using LocalMaxs algorithm for the extraction of contiguous and non-contiguous multiword lexical units. Proceedings of the 9th Portuguese Conference on Artificial Intelligence: Progress in Artificial Intelligence, 113--132."},{"key":"e_1_2_1_9_1","volume-title":"Overview of DUC","author":"Dang H.","year":"2005","unstructured":"Dang , H. 2005. Overview of DUC 2005 . In Proceedings of the Document Understanding Conference Workkshop. (DUC'05) at the Human Language Technology Conference\/Conference on Empirical Methods in Natural Language Processing (HLT\/EMNLP '05). Dang, H. 2005. Overview of DUC 2005. In Proceedings of the Document Understanding Conference Workkshop. (DUC'05) at the Human Language Technology Conference\/Conference on Empirical Methods in Natural Language Processing (HLT\/EMNLP'05)."},{"key":"e_1_2_1_10_1","volume-title":"Overview of DUC 2006. In Proceedings of HLT-NAACL'06","author":"Dang H.","year":"2006","unstructured":"Dang , H. 2006 . Overview of DUC 2006. In Proceedings of HLT-NAACL'06 . Dang, H. 2006. Overview of DUC 2006. In Proceedings of HLT-NAACL'06."},{"key":"e_1_2_1_11_1","unstructured":"Duda R. Hart P. and Stork D. 2001. Pattern Classification. John Wiley New York NY.   Duda R. Hart P. and Stork D. 2001. Pattern Classification. John Wiley New York NY."},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Efron B. and Tibshirani R. 1993. An Introduction to the Bootstrap. Chapman & Hall\/CRC.  Efron B. and Tibshirani R. 1993. An Introduction to the Bootstrap. Chapman & Hall\/CRC.","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"e_1_2_1_13_1","unstructured":"Endres-Niggemeyer B. 2000. Human-style www summarization. http:\/\/endres.niggemeyer.fh-hannover.de\/paps.html  Endres-Niggemeyer B. 2000. Human-style www summarization. http:\/\/endres.niggemeyer.fh-hannover.de\/paps.html"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622487.1622501"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Document Understanding Conferences.","author":"Erkan G.","unstructured":"Erkan , G. and Radev , D . 2004b. Michigan at DUC 2004\u2014Using sentence prestige for document summarization . In Proceedings of the Document Understanding Conferences. Erkan, G. and Radev, D. 2004b. Michigan at DUC 2004\u2014Using sentence prestige for document summarization. In Proceedings of the Document Understanding Conferences."},{"key":"e_1_2_1_16_1","unstructured":"Giannakopoulos G. Karkaletsis V. and Vouros G. 2006. Automatic Multi-document summarization and prior knowledge: Past present and vision. Tech. rep. (DEMO-2006-2). NCSR Demokritos.  Giannakopoulos G. Karkaletsis V. and Vouros G. 2006. Automatic Multi-document summarization and prior knowledge: Past present and vision. Tech. rep. (DEMO-2006-2). NCSR Demokritos."},{"key":"e_1_2_1_17_1","unstructured":"Hollander M. and Wolfe D. 1973. Nonparametric Statistical Inference. John Wiley New York NY.  Hollander M. and Wolfe D. 1973. Nonparametric Statistical Inference. John Wiley New York NY."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/11861461_10"},{"key":"e_1_2_1_19_1","volume-title":"Evaluating DUC 2005 using basic elements. In Proceedings of the Document Understanding Conference (DUC'05)","author":"Hovy E.","unstructured":"Hovy , E. , Lin , C. , and Zhou , L . 2005a . Evaluating DUC 2005 using basic elements. In Proceedings of the Document Understanding Conference (DUC'05) . Hovy, E., Lin, C., and Zhou, L. 2005a. Evaluating DUC 2005 using basic elements. In Proceedings of the Document Understanding Conference (DUC'05)."},{"key":"e_1_2_1_20_1","unstructured":"Hovy E. Lin C. Zhou L. and Fukumoto J. 2005b. Basic Elements. http:\/\/haydn.isi.edu\/BE\/.  Hovy E. Lin C. Zhou L. and Fukumoto J. 2005b. Basic Elements. http:\/\/haydn.isi.edu\/BE\/."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 5th Conference on Language Resources and Evaluation (LREC).","author":"Hovy E.","unstructured":"Hovy , E. , Lin , C. , Zhou , L. , and Fukumoto , J . 2006. Automated summarization evaluation with basic elements . In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC). Hovy, E., Lin, C., Zhou, L., and Fukumoto, J. 2006. Automated summarization evaluation with basic elements. In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC)."},{"key":"e_1_2_1_22_1","volume-title":"Rank Correlation Methods","author":"Kendall M.","unstructured":"Kendall , M. 1962. Rank Correlation Methods . Hafner, New York, NY. Kendall, M. 1962. Rank Correlation Methods. Hafner, New York, NY."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the Workshop on Text Summarization Branches Out (WAS'04)","author":"Lin C.","year":"2004","unstructured":"Lin , C. 2004 . Rouge: A package for automatic evaluation of summaries . In Proceedings of the Workshop on Text Summarization Branches Out (WAS'04) . 25--26. Lin, C. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS'04). 25--26."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.3115\/1118162.1118168"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073445.1073465"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1147\/rd.22.0159"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of AAAI. AAAI, 622--628","author":"Mani I.","unstructured":"Mani , I. and Bloedorn , E . 1997. Multi-document summarization by graph search and matching . In Proceedings of AAAI. AAAI, 622--628 . Mani, I. and Bloedorn, E. 1997. Multi-document summarization by graph search and matching. In Proceedings of AAAI. AAAI, 622--628."},{"key":"e_1_2_1_29_1","unstructured":"Manning C. and Sch\u00fctze H. 1999. Foundations of Statistical Natural Language Processing. The MIT Press.   Manning C. and Sch\u00fctze H. 1999. Foundations of Statistical Natural Language Processing. The MIT Press."},{"key":"e_1_2_1_30_1","volume-title":"The Theory and Practice of Discourse Parsing and Summarization","author":"Marcu D.","unstructured":"Marcu , D. 2000. The Theory and Practice of Discourse Parsing and Summarization . The MIT Press . Marcu, D. 2000. The Theory and Practice of Discourse Parsing and Summarization. The MIT Press."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1951.10500769"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings the 5th World Multi-Conference on Systemics, Cybenetics and Infomatics (SCI'01)","volume":"8","author":"Matsuo Y.","unstructured":"Matsuo , Y. , Ohsawa , Y. , and Ishizuka , M . 2001. A Document as a Small World . In Proceedings the 5th World Multi-Conference on Systemics, Cybenetics and Infomatics (SCI'01) Vol. 8 . 410--414. Matsuo, Y., Ohsawa, Y., and Ishizuka, M. 2001. A Document as a Small World. In Proceedings the 5th World Multi-Conference on Systemics, Cybenetics and Infomatics (SCI'01) Vol. 8. 410--414."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.3115\/1219044.1219064"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 1st International Conference on Intelligent Analysis Methods and Tools (IA'05)","author":"Mihalcea R.","year":"2005","unstructured":"Mihalcea , R. 2005 . Multi-document Summarization with iterative graph-based algorithms . In Proceedings of the 1st International Conference on Intelligent Analysis Methods and Tools (IA'05) . Mihalcea, R. 2005. Multi-document Summarization with iterative graph-based algorithms. In Proceedings of the 1st International Conference on Intelligent Analysis Methods and Tools (IA'05)."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the Document Understanding Conference (DUC'06)","author":"Mohamed A.","unstructured":"Mohamed , A. and Rajasekaran , S . 2006. Query-based summarization based on document graphs . In Proceedings of the Document Understanding Conference (DUC'06) . Mohamed, A. and Rajasekaran, S. 2006. Query-based summarization based on document graphs. In Proceedings of the Document Understanding Conference (DUC'06)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/375360.375365"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220575.1220690"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the Document Understanding Conference (DUC'06)","author":"Passonneau R.","unstructured":"Passonneau , R. , McKeown , K. , Sigelman , S. , and Goodkind , A . 2006. Applying the Pyramid method . In Proceedings of the Document Understanding Conference (DUC'06) . Passonneau, R., McKeown, K., Sigelman, S., and Goodkind, A. 2006. Applying the Pyramid method. In Proceedings of the Document Understanding Conference (DUC'06)."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.3115\/1117736.1117745"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.3115\/1117575.1117578"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/45.6.631"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671963"},{"key":"e_1_2_1_45_1","first-page":"89","article-title":"Footrule for measuring correlation","volume":"2","author":"Spearman C.","year":"1906","unstructured":"Spearman , C. 1906 . Footrule for measuring correlation . British J. Psych. 2 , 89 -- 108 . Spearman, C. 1906. Footrule for measuring correlation. British J. Psych. 2, 89--108.","journal-title":"British J. Psych."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1974.10480196"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/0304-3975(92)90143-4"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.3115\/1119467.1119475"},{"key":"e_1_2_1_49_1","volume-title":"Overview of the TREC 2003 question answering track. In Proceedings of the 12th Text REtrieval Conference (TREC'03)","author":"Voorhees E.","year":"2003","unstructured":"Voorhees , E. 2003 . Overview of the TREC 2003 question answering track. In Proceedings of the 12th Text REtrieval Conference (TREC'03) . Voorhees, E. 2003. Overview of the TREC 2003 question answering track. In Proceedings of the 12th Text REtrieval Conference (TREC'03)."},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of Document Understanding Workshop (DUC)","author":"Witte R.","unstructured":"Witte , R. , Krestel , R. , and Bergler , S . 2006. Context-based multi-document summarization using fuzzy coreference cluster graphs . In Proceedings of Document Understanding Workshop (DUC) . New York, NY. Witte, R., Krestel, R., and Bergler, S. 2006. Context-based multi-document summarization using fuzzy coreference cluster graphs. In Proceedings of Document Understanding Workshop (DUC). New York, NY."},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the Human Language Technology Conference Workshop on Statistical Machine Translation. 72--77","author":"Zens R.","unstructured":"Zens , R. and Ney , H . 2006. N-gram posterior probabilities for statistical machine translation . In Proceedings of the Human Language Technology Conference Workshop on Statistical Machine Translation. 72--77 . Zens, R. and Ney, H. 2006. N-gram posterior probabilities for statistical machine translation. In Proceedings of the Human Language Technology Conference Workshop on Statistical Machine Translation. 72--77."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220835.1220892"}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1410358.1410359","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1410358.1410359","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T13:29:36Z","timestamp":1750253376000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1410358.1410359"}},"subtitle":["N-gram graphs"],"short-title":[],"issued":{"date-parts":[[2008,10]]},"references-count":50,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2008,10]]}},"alternative-id":["10.1145\/1410358.1410359"],"URL":"https:\/\/doi.org\/10.1145\/1410358.1410359","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"value":"1550-4875","type":"print"},{"value":"1550-4883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2008,10]]},"assertion":[{"value":"2008-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2008-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2008-10-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}