{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:32:27Z","timestamp":1750307547214,"version":"3.41.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2009,10,1]],"date-time":"2009-10-01T00:00:00Z","timestamp":1254355200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100004965","name":"Sixth Framework Programme","doi-asserted-by":"publisher","award":["FP6-033812 (AMIDA)"],"award-info":[{"award-number":["FP6-033812 (AMIDA)"]}],"id":[{"id":"10.13039\/501100004965","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Speech Lang. Process."],"published-print":{"date-parts":[[2009,10]]},"abstract":"<jats:p>In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech. The particular task is a decision audit, wherein a user must satisfy a complex information need, navigating several meetings in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive technologies in satisfying this information need, and assess the impact of automatic speech recognition (ASR) errors on user performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective and objective judgments, and a detailed analysis of participant browsing behavior. We find that while ASR errors affect user satisfaction on an information retrieval task, users can adapt their browsing behavior to complete the task satisfactorily. Results also indicate that users consider extractive summaries to be intuitive and useful tools for browsing multimodal meeting data. We discuss areas in which automatic summarization techniques can be improved in comparison with gold-standard meeting abstracts.<\/jats:p>","DOI":"10.1145\/1596517.1596518","type":"journal-article","created":{"date-parts":[[2009,10,9]],"date-time":"2009-10-09T19:06:16Z","timestamp":1255115176000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Extrinsic summarization evaluation"],"prefix":"10.1145","volume":"6","author":[{"given":"Gabriel","family":"Murray","sequence":"first","affiliation":[{"name":"University of British Columbia"}]},{"given":"Thomas","family":"Kleinbauer","sequence":"additional","affiliation":[{"name":"German Research Center for Artificial Intelligence (DFKI)"}]},{"given":"Peter","family":"Poller","sequence":"additional","affiliation":[{"name":"German Research Center for Artificial Intelligence (DFKI)"}]},{"given":"Tilman","family":"Becker","sequence":"additional","affiliation":[{"name":"German Research Center for Artificial Intelligence (DFKI)"}]},{"given":"Steve","family":"Renals","sequence":"additional","affiliation":[{"name":"University of Edinburgh"}]},{"given":"Jonathan","family":"Kilgour","sequence":"additional","affiliation":[{"name":"University of Edinburgh"}]}],"member":"320","published-online":{"date-parts":[[2009,10,14]]},"reference":[{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/244754.244758"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-007-9040-x"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/11677482_3"},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Christensen H. Kolluru B. Gotoh Y. and \n      Renals S\n  . \n  2004\n  . From text summarisation to style-specific summarisation for broadcast news. In Advances in Information Retrieval Lecture Notes in Computer Science vol. \n  2997 Springer Berlin 223--237.  Christensen H. Kolluru B. Gotoh Y. and Renals S. 2004. From text summarisation to style-specific summarisation for broadcast news. In Advances in Information Retrieval Lecture Notes in Computer Science vol. 2997 Springer Berlin 223--237.","DOI":"10.1007\/978-3-540-24752-4_17"},{"volume-title":"Proceedings of the Document Understanding Conference.","author":"Daum\u00e9 H.","key":"e_1_2_1_6_1"},{"volume-title":"Strategies for Natural Language Processing","author":"DeJong G.","key":"e_1_2_1_7_1"},{"volume-title":"Tech. Rep. LAMP-TR-115,CAR-TR-999,CS-TR-4610,UMIACS-TR-2004-48","year":"2004","author":"Dorr B.","key":"e_1_2_1_8_1"},{"key":"e_1_2_1_9_1","unstructured":"Dorr B. Monz C. President S. Schwartz R. and Zajic D. 2005. A methodology for extrinsic evaluation of text summarization: Does ROUGE correlate&quest; In Proceedings of the ACL05 Workshop.  Dorr B. Monz C. President S. Schwartz R. and Zajic D. 2005. A methodology for extrinsic evaluation of text summarization: Does ROUGE correlate&quest; In Proceedings of the ACL05 Workshop."},{"volume-title":"Summarizing Information","author":"Endres-Niggemeyer B.","key":"e_1_2_1_10_1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-72025-3"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/1610075.1610126"},{"key":"e_1_2_1_12_1","unstructured":"Hahn U. and Reimer U. 1999. Knowledge-based text summarization: Salience and generalization operators for knowledge base abstraction. In Advances in Automatic Text Summarization I. Mani and M. Maybury Eds. MIT Press Cambridge MA 215--232.  Hahn U. and Reimer U. 1999. Knowledge-based text summarization: Salience and generalization operators for knowledge base abstraction. In Advances in Automatic Text Summarization I. Mani and M. Maybury Eds. MIT Press Cambridge MA 215--232."},{"volume-title":"Proceedings of the Document Understanding Conference.","author":"Harman D.","key":"e_1_2_1_13_1"},{"volume-title":"Proceedings of the 7th European Conference on Speech Communication and Technology. 1299--1302","author":"Hirschberg J.","key":"e_1_2_1_14_1"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.3115\/1034678.1034731"},{"volume-title":"Proceedings of the International Conference in Acoustics Speech and Signal Processing. 9--12","author":"Hori C.","key":"e_1_2_1_16_1"},{"volume-title":"Proceedings of the AAAI Symposium on Intelligent Summarization. 60--68","author":"Jing H.","key":"e_1_2_1_17_1"},{"volume":"1083","volume-title":"Lecture Notes in Artificial Intelligence","author":"Jones K. S.","key":"e_1_2_1_18_1"},{"volume":"2","volume-title":"Proceedings of the 4th International Conference on Spoken Language Processing (ICSLP'96)","author":"Kameyama M.","key":"e_1_2_1_19_1"},{"volume-title":"Proceedings of the European Natural Language Generation Workshop. 151--154","author":"Kleinbauer T.","key":"e_1_2_1_20_1"},{"volume-title":"Proceedings of the Interspeech Conference. 69--72","author":"Kolluru B.","key":"e_1_2_1_21_1"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1075389.1075390"},{"volume-title":"Proceedings of the SIGIR Workshop, Evaluation Exploratory Search Systems. ACM","author":"Kraaij W.","key":"e_1_2_1_23_1"},{"volume-title":"Proceedings of the NTCIR-5 Workshop. 1765--1776","year":"2004","author":"Lin C.-Y.","key":"e_1_2_1_24_1"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073445.1073465"},{"volume-title":"Proceedings of the NTCIR Workshop 2 Meeting on Evaluation of Chinese and Japanese Text Retrieval and Text Summarization. 77--85","year":"2001","author":"Mani I.","key":"e_1_2_1_26_1"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.3115\/977035.977047"},{"volume-title":"Proceedings of the Interspeech Conference. 621--624","author":"Maskey S.","key":"e_1_2_1_28_1"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.3.1.17"},{"volume-title":"Proceedings of the MLMI Conference. 155--166","author":"Murray G.","key":"e_1_2_1_30_1"},{"volume-title":"Proceedings of the ACL MTSE Workshop. 33--40","author":"Murray G.","key":"e_1_2_1_31_1"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220835.1220882"},{"volume-title":"Proceedings of the HLT-NAACL Conference. 145--152","author":"Nenkova A.","key":"e_1_2_1_33_1"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1233912.1233913"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/160688.160696"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671963"},{"volume-title":"Advances in Automatic Text Summarization","author":"Sparck-Jones K.","key":"e_1_2_1_38_1"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30568-2_1"},{"volume-title":"Proceedings. of the ESCA Workshop on Accessing Information in Spoken Audio. 111--116","author":"Valenza R.","key":"e_1_2_1_40_1"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30568-2_2"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1056808.1057082"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/503376.503426"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-007-0146-3"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671945"},{"volume-title":"Proceedings of the NAACL Conference. 186--193","author":"Zechner K.","key":"e_1_2_1_46_1"},{"volume-title":"Proceedings of the Interspeech Conference. 1531--1534","author":"Zhu X.","key":"e_1_2_1_47_1"}],"container-title":["ACM Transactions on Speech and Language Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1596517.1596518","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1596517.1596518","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T12:23:32Z","timestamp":1750249412000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1596517.1596518"}},"subtitle":["A decision audit task"],"short-title":[],"issued":{"date-parts":[[2009,10]]},"references-count":46,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2009,10]]}},"alternative-id":["10.1145\/1596517.1596518"],"URL":"https:\/\/doi.org\/10.1145\/1596517.1596518","relation":{},"ISSN":["1550-4875","1550-4883"],"issn-type":[{"type":"print","value":"1550-4875"},{"type":"electronic","value":"1550-4883"}],"subject":[],"published":{"date-parts":[[2009,10]]},"assertion":[{"value":"2008-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2009-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2009-10-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}