{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T17:51:23Z","timestamp":1778867483511,"version":"3.51.4"},"reference-count":0,"publisher":"Cambridge University Press (CUP)","issue":"1","license":[{"start":{"date-parts":[[2002,6,17]],"date-time":"2002-06-17T00:00:00Z","timestamp":1024272000000},"content-version":"unspecified","delay-in-days":108,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2002,3]]},"abstract":"<jats:p>The TIPSTER Text Summarization Evaluation (SUMMAC) has developed several new \nextrinsic and intrinsic methods for evaluating summaries. It has established definitively that \nautomatic text summarization is very effective in relevance assessment tasks on news articles. \nSummaries as short as 17% of full text length sped up decision-making by almost a factor \nof 2 with no statistically significant degradation in accuracy. Analysis of feedback forms \nfilled in after each decision indicated that the intelligibility of present-day machine-generated \nsummaries is high. Systems that performed most accurately in the production of indicative \nand informative topic-related summaries used term frequency and co-occurrence statistics, \nand vocabulary overlap comparisons between text passages. However, in the absence of a \ntopic, these statistical methods do not appear to provide any additional leverage: in the case \nof generic summaries, the systems were indistinguishable in accuracy. The paper discusses \nsome of the tradeoffs and challenges faced by the evaluation, and also lists some of the \nlessons learned, impacts, and possible future directions. The evaluation methods used in the \nSUMMAC evaluation are of interest to both summarization evaluation as well as evaluation \nof other \u2018output-related\u2019 NLP technologies, where there may be many potentially acceptable \noutputs, with no automatic way to compare them.<\/jats:p>","DOI":"10.1017\/s1351324901002741","type":"journal-article","created":{"date-parts":[[2002,7,28]],"date-time":"2002-07-28T23:18:00Z","timestamp":1027898280000},"page":"43-68","source":"Crossref","is-referenced-by-count":79,"title":["SUMMAC: a text summarization evaluation"],"prefix":"10.1017","volume":"8","author":[{"given":"INDERJEET","family":"MANI","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"GARY","family":"KLEIN","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"DAVID","family":"HOUSE","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"LYNETTE","family":"HIRSCHMAN","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"THERESE","family":"FIRMIN","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"BETH","family":"SUNDHEIM","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2002,6,17]]},"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324901002741","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,3,30]],"date-time":"2019-03-30T18:58:30Z","timestamp":1553972310000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324901002741\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2002,3]]},"references-count":0,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2002,3]]}},"alternative-id":["S1351324901002741"],"URL":"https:\/\/doi.org\/10.1017\/s1351324901002741","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2002,3]]}}}