{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T05:20:49Z","timestamp":1784092849578,"version":"3.55.0"},"reference-count":28,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2007,5,1]],"date-time":"2007-05-01T00:00:00Z","timestamp":1177977600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2007,5]]},"abstract":"<jats:p>Human variation in content selection in summarization has given rise to some fundamental research questions: How can one incorporate the observed variation in suitable evaluation measures? How can such measures reflect the fact that summaries conveying different content can be equally good and informative? In this article, we address these very questions by proposing a method for analysis of multiple human abstracts into semantic content units. Such analysis allows us not only to quantify human variation in content selection, but also to assign empirical importance weight to different content units. It serves as the basis for an evaluation method, the Pyramid Method, that incorporates the observed variation and is predictive of different equally informative summaries. We discuss the reliability of content unit annotation, the properties of Pyramid scores, and their correlation with other evaluation methods.<\/jats:p>","DOI":"10.1145\/1233912.1233913","type":"journal-article","created":{"date-parts":[[2007,6,6]],"date-time":"2007-06-06T14:37:11Z","timestamp":1181140631000},"page":"4","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":123,"title":["The Pyramid Method"],"prefix":"10.1145","volume":"4","author":[{"given":"Ani","family":"Nenkova","sequence":"first","affiliation":[{"name":"Columbia University, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rebecca","family":"Passonneau","sequence":"additional","affiliation":[{"name":"Columbia University, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathleen","family":"McKeown","sequence":"additional","affiliation":[{"name":"Columbia University, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2007,5]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"crossref","unstructured":"Dice L. R. 1945. Measures of the amount of ecologic association between species. Ecology 297--302.  Dice L. R. 1945. Measures of the amount of ecologic association between species. Ecology 297--302.","DOI":"10.2307\/1932409"},{"key":"e_1_2_1_2_1","unstructured":"Dixon W. and Massey F. 1969. Introduction to Statistical Analysis. McGraw-Hill New York.  Dixon W. and Massey F. 1969. Introduction to Statistical Analysis. McGraw-Hill New York."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the NAACL-ANLP Workshop on Automatic Summarization. 10","author":"Donaway R.","unstructured":"Donaway , R. , Drummey , K. , and Mather , L . 2000. A comparison of rankings produced by summarization evaluation measures . In Proceedings of the NAACL-ANLP Workshop on Automatic Summarization. 10 .3115\/1117575.1117583 Donaway, R., Drummey, K., and Mather, L. 2000. A comparison of rankings produced by summarization evaluation measures. In Proceedings of the NAACL-ANLP Workshop on Automatic Summarization. 10.3115\/1117575.1117583"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1080\/00220671.1982.10885394","article-title":"Efficient text summarization","volume":"75","author":"Garner R.","year":"1982","unstructured":"Garner , R. 1982 . Efficient text summarization . J. Educat. Res. 75 , 275 -- 279 . Garner, R. 1982. Efficient text summarization. J. Educat. Res. 75, 275--279.","journal-title":"J. Educat. Res."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of ACM SIGIR'99","author":"Goldstein J.","unstructured":"Goldstein , J. , Kantrowitz , M. , Mittal , V. , and Carbonell , J . 1999. Summarizing text documents: Sentence selection and evaluation metrics . In Proceedings of ACM SIGIR'99 . ACM, New York, 121--128. 10.1145\/312624.312665 Goldstein, J., Kantrowitz, M., Mittal, V., and Carbonell, J. 1999. Summarizing text documents: Sentence selection and evaluation metrics. In Proceedings of ACM SIGIR'99. ACM, New York, 121--128. 10.1145\/312624.312665"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the AAAI Symposium on Intelligent Summarization.","author":"Jing H.","unstructured":"Jing , H. , Barzilay , R. , McKeown , K. , and Elhadad , M . 1998. Summarization evaluation methods: Experiments and analysis . In Proceedings of the AAAI Symposium on Intelligent Summarization. Jing, H., Barzilay, R., McKeown, K., and Elhadad, M. 1998. Summarization evaluation methods: Experiments and analysis. In Proceedings of the AAAI Symposium on Intelligent Summarization."},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/S0022-5371(70)80003-2","article-title":"Recall of prose as a function of structural importance of linguistic units","volume":"9","author":"Johnson R.","year":"1970","unstructured":"Johnson , R. 1970 . Recall of prose as a function of structural importance of linguistic units . J. Verb. Learn. Verb. Behav. 9 , 12 -- 20 . Johnson, R. 1970. Recall of prose as a function of structural importance of linguistic units. J. Verb. Learn. Verb. Behav. 9, 12--20.","journal-title":"J. Verb. Learn. Verb. Behav."},{"key":"e_1_2_1_8_1","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff K.","year":"1980","unstructured":"Krippendorff , K. 1980 . Content Analysis: An Introduction to Its Methodology . Sage Publications , Beverly Hills, CA . Krippendorff, K. 1980. Content Analysis: An Introduction to Its Methodology. Sage Publications, Beverly Hills, CA."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the ACL Text Summarization Workshop.","author":"Lin C. Y.","year":"2004","unstructured":"Lin , C. Y. 2004 . ROUGE: A package for automatic evaluation of summaries . In Proceedings of the ACL Text Summarization Workshop. Lin, C. Y. 2004. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL Text Summarization Workshop."},{"key":"e_1_2_1_10_1","volume-title":"Evaluating summaries and answers: Two sides of the same coin&quest","author":"Lin J.","unstructured":"Lin , J. and Demner-Fushman , D. 2005. Evaluating summaries and answers: Two sides of the same coin&quest ; In Proceedings of the ACL Workshop on Measures for MT and Summarization . Lin, J. and Demner-Fushman, D. 2005. Evaluating summaries and answers: Two sides of the same coin&quest; In Proceedings of the ACL Workshop on Measures for MT and Summarization."},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the NAACL 2001 Workshop on Automatic Summarization.","author":"Mani I.","year":"2001","unstructured":"Mani , I. 2001 . Summarization evaluation: An overview . In Proceedings of the NAACL 2001 Workshop on Automatic Summarization. Mani, I. 2001. Summarization evaluation: An overview. In Proceedings of the NAACL 2001 Workshop on Automatic Summarization."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of ACL\/EACL-97 summarization workshop. 82--88","author":"Marcu D.","year":"1997","unstructured":"Marcu , D. 1997 . From discourse structure to text summaries . In Proceedings of ACL\/EACL-97 summarization workshop. 82--88 . Marcu, D. 1997. From discourse structure to text summaries. In Proceedings of ACL\/EACL-97 summarization workshop. 82--88."},{"key":"e_1_2_1_13_1","volume-title":"The Theory and Practice of Discourse and Summarization","author":"Marcu D.","unstructured":"Marcu , D. 2000. The Theory and Practice of Discourse and Summarization . The MIT Press , Cambridge, MA . Marcu, D. 2000. The Theory and Practice of Discourse and Summarization. The MIT Press, Cambridge, MA."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the Document Understanding Conference","author":"Marcu D.","year":"2001","unstructured":"Marcu , D. 2001 . Discourse-based summarization in DUC 2001 . In Proceedings of the Document Understanding Conference 2001. Marcu, D. 2001. Discourse-based summarization in DUC 2001. In Proceedings of the Document Understanding Conference 2001."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Document Understanding Conference","author":"McKeown K.","year":"2001","unstructured":"McKeown , K. , Barzilay , R. , Evans , D. , Hatzivassiloglou , V. , Schiffman , B. , and Teufel , S . 2001. Columbia multi-document summarization: Approach and evaluation . In Proceedings of the Document Understanding Conference 2001 . McKeown, K., Barzilay, R., Evans, D., Hatzivassiloglou, V., Schiffman, B., and Teufel, S. 2001. Columbia multi-document summarization: Approach and evaluation. In Proceedings of the Document Understanding Conference 2001."},{"key":"e_1_2_1_16_1","unstructured":"Minel J.-L. Nugier S. and Piat G. 1997. How to appreciate the quality of automatic text summarization&quest; In Proceedings of the ACL\/ECL'97 Workshop on Intelligent Scalable Text Summarization. 25--30.  Minel J.-L. Nugier S. and Piat G. 1997. How to appreciate the quality of automatic text summarization&quest; In Proceedings of the ACL\/ECL'97 Workshop on Intelligent Scalable Text Summarization. 25--30."},{"key":"e_1_2_1_17_1","unstructured":"Nenkova A. and Passonneau R. 2004. Evaluating content selection in summarization: The pyramid method. In HLT\/NAACL.  Nenkova A. and Passonneau R. 2004. Evaluating content selection in summarization: The pyramid method. In HLT\/NAACL."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of ACL. 10","author":"Papineni K.","unstructured":"Papineni , K. , Roukos , S. , Ward , T. , and Zhu , W . 2002. BLEU: A method for automatic evaluation of machine translation . In Proceedings of ACL. 10 .3115\/1073083.1073135 Papineni, K., Roukos, S., Ward, T., and Zhu, W. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of ACL. 10.3115\/1073083.1073135"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the International Conference on Language Resources and Evaluation (LREC).","author":"Passonneau R.","year":"2006","unstructured":"Passonneau , R. 2006 . Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation . In Proceedings of the International Conference on Language Resources and Evaluation (LREC). Passonneau, R. 2006. Measuring agreement on set-valued items (masi) for semantic and pragmatic annotation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC)."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Document Understanding Conference (DUC'05)","author":"Passonneau R.","unstructured":"Passonneau , R. , Nenkova , A. , McKeown , K. , and Sigleman , S . 2005. Applying the pyramid method in DUC 2005 . In Proceedings of the Document Understanding Conference (DUC'05) . Passonneau, R., Nenkova, A., McKeown, K., and Sigleman, S. 2005. Applying the pyramid method in DUC 2005. In Proceedings of the Document Understanding Conference (DUC'05)."},{"key":"e_1_2_1_21_1","volume-title":"ANLP\/NAACL Workshop on Summarization. 10","author":"Radev D.","unstructured":"Radev , D. , Jing , H. , and Budzikowska , M . 2000. Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies . In ANLP\/NAACL Workshop on Summarization. 10 .3115\/1117575.1117578 Radev, D., Jing, H., and Budzikowska, M. 2000. Centroid-based summarization of multiple documents: sentence extraction, utility-based evaluation, and user studies. In ANLP\/NAACL Workshop on Summarization. 10.3115\/1117575.1117578"},{"key":"e_1_2_1_22_1","volume-title":"CIKM'03","author":"Radev D.","unstructured":"Radev , D. and Tam , D . 2003. Single-document and multi-document summary evaluation via relative utility. In Poster session , CIKM'03 . 10.1145\/956863.956960 Radev, D. and Tam, D. 2003. Single-document and multi-document summary evaluation via relative utility. In Poster session, CIKM'03. 10.1145\/956863.956960"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1002\/asi.5090120210","article-title":"The formation of abstracts by the selection of sentences","volume":"2","author":"Rath G. J.","year":"1961","unstructured":"Rath , G. J. , Resnick , A. , and Savage , R. 1961 . The formation of abstracts by the selection of sentences : Part 1: Sentence selection by man and machines. Amer. Document. 2 , 12, 139 -- 208 . Rath, G. J., Resnick, A., and Savage, R. 1961. The formation of abstracts by the selection of sentences: Part 1: Sentence selection by man and machines. Amer. Document. 2, 12, 139--208.","journal-title":"Amer. Document."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(96)00062-3"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Soricut R. and Marcu D. 2003. Sentence level discourse parsing using syntactic and lexical information. In HLT-NAACL. 10.3115\/1073445.1073475   Soricut R. and Marcu D. 2003. Sentence level discourse parsing using syntactic and lexical information. In HLT-NAACL. 10.3115\/1073445.1073475","DOI":"10.3115\/1073445.1073475"},{"key":"e_1_2_1_26_1","unstructured":"Teufel S. and van Halteren H. 2004. Evaluating information content by factoid analysis: human annotation and stability. In EMNLP-04.  Teufel S. and van Halteren H. 2004. Evaluating information content by factoid analysis: human annotation and stability. In EMNLP-04."},{"key":"e_1_2_1_27_1","volume-title":"HLT-NAACL DUC Workshop. 10","author":"van Halteren H.","year":"1946","unstructured":"van Halteren , H. and Teufel , S . 2003. Examining the consensus between human summaries: initial experiments with factoid analysis . In HLT-NAACL DUC Workshop. 10 .3115\/11 1946 7.1119475 van Halteren, H. and Teufel, S. 2003. Examining the consensus between human summaries: initial experiments with factoid analysis. In HLT-NAACL DUC Workshop. 10.3115\/1119467.1119475"},{"key":"e_1_2_1_28_1","volume-title":"Overview of the TREC 2003 question answering track. In Proceedings of the 12th Text REtrieval Conference (TREC","author":"Voorhees E. M.","year":"2004","unstructured":"Voorhees , E. M. 2004 . Overview of the TREC 2003 question answering track. In Proceedings of the 12th Text REtrieval Conference (TREC 2003). 54--68. Voorhees, E. M. 2004. Overview of the TREC 2003 question answering track. In Proceedings of the 12th Text REtrieval Conference (TREC 2003). 54--68."}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1233912.1233913","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1233912.1233913","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T15:06:38Z","timestamp":1750259198000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1233912.1233913"}},"subtitle":["Incorporating human content selection variation in summarization evaluation"],"short-title":[],"issued":{"date-parts":[[2007,5]]},"references-count":28,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2007,5]]}},"alternative-id":["10.1145\/1233912.1233913"],"URL":"https:\/\/doi.org\/10.1145\/1233912.1233913","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"value":"1550-4875","type":"print"},{"value":"1550-4883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2007,5]]},"assertion":[{"value":"2007-05-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}