{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,16]],"date-time":"2025-06-16T18:10:19Z","timestamp":1750097419707,"version":"3.41.0"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"4","funder":[{"name":"Mohamed bin Zayed University of Artificial Intelligence","award":["8481000078"],"award-info":[{"award-number":["8481000078"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2025,7,31]]},"abstract":"<jats:p>\n            Evaluation of summary quality is particularly crucial within the scientific domain, because it facilitates efficient knowledge dissemination and automated scientific information retrieval. This article presents conceptual and experimental analyses of scientific summarization, highlighting the inadequacies of traditional evaluation methods. These methods, including\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\( n \\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            -gram overlap calculations, embedding comparisons, verification, and QA-based approaches, often fall short in providing explanations, grasping scientific concepts, or identifying key content. Correspondingly, we introduce the Facet-aware Metric (FM), employing LLMs for advanced semantic matching to evaluate summaries based on different facets. The\n            <jats:italic toggle=\"yes\">facet granularity<\/jats:italic>\n            is tailored to the structure of scientific abstracts, offering an integrated evaluation approach that is not fragmented, while also providing fine-grained interpretability. Recognizing the absence of an evaluation benchmark in the scientific domain, we curate a Scientific abstract summary evaluation Dataset (ScholarSum) with facet-level annotations. Our findings confirm that FM offers a more logical approach to evaluating scientific summaries. In addition, fine-tuned smaller models can compete with LLMs in scientific contexts, while LLMs have limitations in learning from in-context information in scientific domains. We hope our benchmark inspires better evaluation metrics and future enhancements to LLMs:\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/iriscxy\/ScholarSum\">https:\/\/github.com\/iriscxy\/ScholarSum<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3733597","type":"journal-article","created":{"date-parts":[[2025,5,6]],"date-time":"2025-05-06T15:56:09Z","timestamp":1746546969000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["New Paradigm for Evaluating Scholar Summaries: A Facet-aware Metric and a Meta-evaluation Benchmark"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7680-8992","authenticated-orcid":false,"given":"Tairan","family":"Wang","sequence":"first","affiliation":[{"name":"King Abdullah University of Science and Technology, Thuwal, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6633-0796","authenticated-orcid":false,"given":"Xiuying","family":"Chen","sequence":"additional","affiliation":[{"name":"Mohamed bin Zayed, University of Artificial Intelligence, Abu Dhabi, UAE"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7326-0098","authenticated-orcid":false,"given":"Qingqing","family":"Zhu","sequence":"additional","affiliation":[{"name":"National Institutes of Health, Bethesda, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7919-6912","authenticated-orcid":false,"given":"Taicheng","family":"Guo","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, Indiana, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1301-3700","authenticated-orcid":false,"given":"Shen","family":"Gao","sequence":"additional","affiliation":[{"name":"Shandong University, Jinan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9998-916X","authenticated-orcid":false,"given":"Zhiyong","family":"Lu","sequence":"additional","affiliation":[{"name":"National Institutes of Health, Bethesda, Maryland, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7108-3574","authenticated-orcid":false,"given":"Xin","family":"Gao","sequence":"additional","affiliation":[{"name":"King Abdullah University of Science and Technology, Thuwal, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3574-5665","authenticated-orcid":false,"given":"Xiangliang","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, Indiana, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,6,16]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. Gpt-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"issue":"4","key":"e_1_3_2_3_2","doi-asserted-by":"crossref","first-page":"1011","DOI":"10.1016\/j.jksuci.2020.04.020","article-title":"Automatic summarization of scientific articles: A survey","volume":"34","author":"Altmami Nouf Ibrahim","year":"2022","unstructured":"Nouf Ibrahim Altmami and Mohamed El Bachir Menai. 2022. Automatic summarization of scientific articles: A survey. Journal of King Saud University-Computer and Information Sciences 34, 4 (2022), 1011\u20131028.","journal-title":"Journal of King Saud University-Computer and Information Sciences"},{"key":"e_1_3_2_4_2","volume-title":"Proc. of AAAI","author":"An Chenxin","year":"2021","unstructured":"Chenxin An, Ming Zhong, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2021. Enhancing scientific papers summarization with citation graph. In Proc. of AAAI."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3464299"},{"key":"e_1_3_2_6_2","volume-title":"Proc. ACL Workshop on Eval Measures for MT and Summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. ACL Workshop on Eval Measures for MT and Summarization."},{"key":"e_1_3_2_7_2","volume-title":"Proc. of EMNLP","author":"Bhandari Manik","year":"2020","unstructured":"Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proc. of EMNLP."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094075"},{"issue":"2","key":"e_1_3_2_9_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3298987","article-title":"ELSA: A multilingual document summarization algorithm based on frequent itemsets and latent semantic analysis","volume":"37","author":"Cagliero Luca","year":"2019","unstructured":"Luca Cagliero, Paolo Garza, and Elena Baralis. 2019. ELSA: A multilingual document summarization algorithm based on frequent itemsets and latent semantic analysis. ACM Transactions on Information Systems 37, 2 (2019), 1\u201333.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_2_10_2","volume-title":"Proc. of ICLR","author":"Chan Chi-Min","year":"2024","unstructured":"Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. Chateval: Towards better LLM-based evaluators through multi-agent debate. In Proc. of ICLR."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3517221"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.270"},{"key":"e_1_3_2_13_2","volume-title":"Proc. of EMNLP Findings","author":"Chen Yiran","year":"2021","unstructured":"Yiran Chen, Pengfei Liu, and Xipeng Qiu. 2021. Are factuality checkers reliable? Adversarial meta-evaluation of factuality in summarization. In Proc. of EMNLP Findings."},{"key":"e_1_3_2_14_2","unstructured":"Steffi Chern Ethan Chern Graham Neubig and Pengfei Liu. 2024. Can large language models be trusted for evaluation? Scalable meta-evaluation of LLMs as evaluators via agent debate. arXiv:2401.16788. Retrieved from https:\/\/arxiv.org\/abs\/2401.16788"},{"key":"e_1_3_2_15_2","volume-title":"Proc. of AACL","author":"Cohan Arman","year":"2018","unstructured":"Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proc. of AACL."},{"key":"e_1_3_2_16_2","volume-title":"Proc. of LREC","author":"Cohan Arman","year":"2016","unstructured":"Arman Cohan and Nazli Goharian. 2016. Revisiting summarization evaluation for scientific articles. In Proc. of LREC."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00799-017-0216-8"},{"key":"e_1_3_2_18_2","volume-title":"Proc. of TAC","author":"Dang Hoa Trang","year":"2008","unstructured":"Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the TAC 2008 update summarization task. In Proc. of TAC."},{"key":"e_1_3_2_19_2","volume-title":"Proc. of EMNLP","author":"Dernoncourt Franck","year":"2017","unstructured":"Franck Dernoncourt and Ji-Young Lee. 2017. PubMed 200k RCT: A dataset for sequential sentence classification in medical abstracts. In Proc. of EMNLP."},{"key":"e_1_3_2_20_2","volume-title":"Proc. of EMNLP","author":"DeYoung Jay","year":"2021","unstructured":"Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, and Lucy Lu Wang. 2021. MS\u02c62: Multi-document summarization of medical studies. In Proc. of EMNLP."},{"key":"e_1_3_2_21_2","volume-title":"Proc. of AACL","author":"Fabbri Alexander Richard","year":"2022","unstructured":"Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proc. of AACL."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/1961209.1961210"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-020-09548-1"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","unstructured":"Marcio Fonseca and Shay B. Cohen. 2024. Can large language model summarizers adapt to diverse scientific communication goals? arXiv:2401.10415. Retrieved from https:\/\/arxiv.org\/abs\/2401.10415","DOI":"10.18653\/v1\/2024.findings-acl.508"},{"key":"e_1_3_2_25_2","volume-title":"Proc. of EMNLP","author":"Fonseca Marcio","year":"2022","unstructured":"Marcio Fonseca, Yftah Ziser, and Shay B. Cohen. 2022. Factorizing content and budget decisions in abstractive summarization of long documents. In Proc. of EMNLP."},{"key":"e_1_3_2_26_2","first-page":"478","volume-title":"Proc. of ACL-IJCNLP","author":"Gabriel Saadia","year":"2021","unstructured":"Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021. GO FIGURE: A meta evaluation of factuality in summarization. In Proc. of ACL-IJCNLP, 478\u2013487."},{"key":"e_1_3_2_27_2","unstructured":"Lang Gao Xiangliang Zhang Preslav Nakov and Xiuying Chen. 2024. Shaping the safety boundaries: Understanding and defending against jailbreaks in large language models. arXiv:2412.17034. Retrieved from https:\/\/arxiv.org\/abs\/2412.17034"},{"key":"e_1_3_2_28_2","volume-title":"Proc. of ACL Findings","author":"Guo Mandy","year":"2022","unstructured":"Mandy Guo, Joshua Ainslie, David C. Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Proc. of ACL Findings."},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","unstructured":"Yue Guo Tal August Gondy Leroy Trevor Cohen and Lucy Lu Wang. 2024. APPLS: Evaluating evaluation metrics for plain language summarization. arXiv:2305.14341. Retrieved from https:\/\/arxiv.org\/abs\/2305.14341","DOI":"10.18653\/v1\/2024.emnlp-main.519"},{"key":"e_1_3_2_30_2","volume-title":"Information Retrieval: A Health and Biomedical Perspective","author":"Hersh William","year":"2008","unstructured":"William Hersh. 2008. Information Retrieval: A Health and Biomedical Perspective. Springer Science & Business Media."},{"key":"e_1_3_2_31_2","first-page":"359","volume-title":"AMIA Annual Symposium Proceedings","volume":"2006","author":"Huang Xiaoli","year":"2006","unstructured":"Xiaoli Huang, Jimmy Lin, and Dina Demner-Fushman. 2006. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annual Symposium Proceedings 2006 (2006), 359."},{"key":"e_1_3_2_32_2","unstructured":"Dongfu Jiang Yishan Li Ge Zhang Wenhao Huang Bill Yuchen Lin and Wenhu Chen. 2023. TIGERScore: Towards building explainable metric for all text generation tasks. arXiv:2310.00752. Retrieved from https:\/\/arxiv.org\/abs\/2310.00752"},{"key":"e_1_3_2_33_2","volume-title":"Proc. of EMNLP","author":"Jin Di","year":"2018","unstructured":"Di Jin and Peter Szolovits. 2018. Hierarchical neural networks for sequential sentence classification in medical scientific abstracts. In Proc. of EMNLP."},{"key":"e_1_3_2_34_2","volume-title":"Proc. of EMNLP","author":"Jin Qiao","year":"2019","unstructured":"Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proc. of EMNLP."},{"key":"e_1_3_2_35_2","unstructured":"Waleed Kadous. 2023. Llama 2 is about as factually accurate as GPT-4 for summaries and is 30X cheaper. Retrieved from https:\/\/www.anyscale.com\/blog\/llama-2-is-about-as-factually-accurate-as-gpt-4-for-summaries-and-is-30x-cheaper"},{"issue":"8","key":"e_1_3_2_36_2","first-page":"1","article-title":"An empirical survey on long document summarization: Datasets, models, and metrics","volume":"55","author":"Koh Huan Yee","year":"2022","unstructured":"Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. An empirical survey on long document summarization: Datasets, models, and metrics. ACM Computing Surveys 55, 8 (2022), 1\u201335.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_37_2","volume-title":"Proc. of EACL","author":"Krishna Kalpesh","year":"2023","unstructured":"Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proc. of EACL."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.750"},{"key":"e_1_3_2_39_2","volume-title":"Proc. of ACL","author":"Lewis Mike","year":"2020","unstructured":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proc. of ACL."},{"key":"e_1_3_2_40_2","volume-title":"Proc. of the First Workshop on Text Summarization Branches Out (TextSumm2020)","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. of the First Workshop on Text Summarization Branches Out (TextSumm2020)."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.228"},{"key":"e_1_3_2_42_2","doi-asserted-by":"crossref","unstructured":"Yixin Liu Alexander R. Fabbri Yilun Zhao Pengfei Liu Shafiq Joty Chien-Sheng Wu Caiming Xiong and Dragomir Radev. 2023. Towards interpretable and efficient automatic reference-based summarization evaluation. arXiv:2303.03608. Retrieved from https:\/\/arxiv.org\/abs\/2303.03608","DOI":"10.18653\/v1\/2023.emnlp-main.1018"},{"key":"e_1_3_2_43_2","unstructured":"Yu Liu Lang Gao Mingxin Yang Yu Xie Ping Chen Xiaojin Zhang and Wei Chen. 2024. Vuldetectbench: Evaluating the deep capability of vulnerability detection with large language models. arXiv:2406.07595. Retrieved from https:\/\/arxiv.org\/abs\/2406.07595"},{"key":"e_1_3_2_44_2","doi-asserted-by":"crossref","unstructured":"Yang Liu Dan Iter Yichong Xu Shuohang Wang Ruochen Xu and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment (2023). arXiv:2303.16634. Retrieved from http:\/\/arxiv.org\/abs\/2303.16634","DOI":"10.18653\/v1\/2023.emnlp-main.153"},{"key":"e_1_3_2_45_2","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1907.11692"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3445029"},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","unstructured":"Mihir Parmar Hanieh Deilamsalehy Franck Dernoncourt Seunghyun Yoon Ryan A. Rossi and Trung Bui. 2024. Towards enhancing coherence in extractive summarization: Dataset and experiments with LLMs. arXiv:2407.04855. Retrieved from https:\/\/arxiv.org\/abs\/2407.04855","DOI":"10.18653\/v1\/2024.emnlp-main.1106"},{"issue":"10","key":"e_1_3_2_48_2","first-page":"875","article-title":"Contribution and performance of ChatGPT and other large language models (LLM) for scientific and research advancements: A double-edged sword","volume":"5","author":"Liladhar Rane Nitin","year":"2023","unstructured":"Nitin Liladhar Rane, Abhijeet Tawde, Saurabh P. Choudhary, and Jayesh Rane. 2023. Contribution and performance of ChatGPT and other large language models (LLM) for scientific and research advancements: A double-edged sword. International Research Journal of Modernization in Engineering Technology and Science 5, 10 (2023), 875\u2013899.","journal-title":"International Research Journal of Modernization in Engineering Technology and Science"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3200864"},{"issue":"2","key":"e_1_3_2_50_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3431813","article-title":"Retrieval evaluation measures that agree with users\u2019 SERP preferences: Traditional, preference-based, and diversity measures","volume":"39","author":"Sakai Tetsuya","year":"2021","unstructured":"Tetsuya Sakai and Zhaohao Zeng. 2021. Retrieval evaluation measures that agree with users\u2019 SERP preferences: Traditional, preference-based, and diversity measures. ACM Transactions on Information Systems 39, 2 (2020), 1\u201335.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_2_51_2","volume-title":"Proc. of EMNLP","author":"Scialom Thomas","year":"2021","unstructured":"Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021. QuestEval: Summarization asks for fact-based evaluation. In Proc. of EMNLP."},{"key":"e_1_3_2_52_2","volume-title":"Proc. of AACL","author":"Shapira Ori","year":"2019","unstructured":"Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Proc. of AACL."},{"issue":"3","key":"e_1_3_2_53_2","first-page":"364","article-title":"The introduction, methods, results, and discussion (IMRAD) structure: A fifty-year survey","volume":"92","author":"Sollaci Luciana B.","year":"2004","unstructured":"Luciana B. Sollaci and Mauricio G. Pereira. 2004. The introduction, methods, results, and discussion (IMRAD) structure: A fifty-year survey. Journal of the Medical Library Association 92, 3 (2004), 364.","journal-title":"Journal of the Medical Library Association"},{"key":"e_1_3_2_54_2","doi-asserted-by":"crossref","unstructured":"Derek Tam Anisha Mascarenhas Shiyue Zhang Sarah Kwan Mohit Bansal and Colin Raffel. 2022. Evaluating the factual consistency of large language models through summarization. arXiv:2211.08412. Retrieved from https:\/\/arxiv.org\/abs\/2211.08412","DOI":"10.18653\/v1\/2023.findings-acl.322"},{"key":"e_1_3_2_55_2","unstructured":"Xiangru Tang Qiao Jin Kunlun Zhu Tongxin Yuan Yichi Zhang Wangchunshu Zhou Meng Qu Yilun Zhao Jian Tang Zhuosheng Zhang et al. 2024. Prioritizing safeguarding over autonomy: Risks of LLM agents for science. arXiv:2402.04247. Retrieved from https:\/\/arxiv.org\/abs\/2402.04247"},{"key":"e_1_3_2_56_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_2_57_2","first-page":"605","volume-title":"AMIA Summits on Translational Science Proceedings","author":"Wallace Byron C.","year":"2021","unstructured":"Byron C. Wallace, Sayantan Saha, Frank Soboczenski, and Iain J. Marshall. 2021. Generating (factual?) narrative summaries of RCTS: Experiments with neural multi-document summarization. AMIA Summits on Translational Science Proceedings (2021), 605."},{"key":"e_1_3_2_58_2","unstructured":"Chenxi Wang Tianle Gu Zhongyu Wei Lang Gao Zirui Song and Xiuying Chen. 2025. Word form matters: LLMs\u2019 semantic reconstruction under typoglycemia. arXiv:2503.01714. Retrieved from https:\/\/arxiv.org\/abs\/2503.01714"},{"key":"e_1_3_2_59_2","volume-title":"Proc. of ACL","author":"Wang Lucy Lu","year":"2023","unstructured":"Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey E. Kuehl, Erin Bransom, and Byron C. Wallace. 2023. Automated metrics for medical multi-document summarization disagree with human evaluations. In Proc. of ACL."},{"key":"e_1_3_2_60_2","volume-title":"Proc. of EMNLP","author":"Xiao Wen","year":"2019","unstructured":"Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In Proc. of EMNLP."},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2960611"},{"key":"e_1_3_2_62_2","volume-title":"Proc. of NeurIPS","author":"Yuan Weizhe","year":"2021","unstructured":"Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In Proc. of NeurIPS."},{"key":"e_1_3_2_63_2","volume-title":"Proc. of NeurIPS","author":"Zaheer Manzil","year":"2020","unstructured":"Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. In Proc. of NeurIPS."},{"key":"e_1_3_2_64_2","volume-title":"Proc. of EMNLP","author":"Zhang Haopeng","year":"2022","unstructured":"Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2022. HEGEL: Hypergraph transformer for long document summarization. In Proc. of EMNLP."},{"key":"e_1_3_2_65_2","volume-title":"Proc. of ICLR","author":"Zhang Tianyi","year":"2020","unstructured":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proc. of ICLR."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2022.118335"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3733597","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,16]],"date-time":"2025-06-16T17:28:22Z","timestamp":1750094902000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3733597"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,16]]},"references-count":65,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,7,31]]}},"alternative-id":["10.1145\/3733597"],"URL":"https:\/\/doi.org\/10.1145\/3733597","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2025,6,16]]},"assertion":[{"value":"2024-09-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-06-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}