{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T14:23:24Z","timestamp":1760711004818,"version":"3.41.2"},"reference-count":54,"publisher":"World Scientific Pub Co Pte Ltd","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Patt. Recogn. Artif. Intell."],"published-print":{"date-parts":[[2022,8]]},"abstract":"<jats:p> With an increasing number of new summarization systems proposed in recent years, an automatic text evaluation metric that can accurately and reliably rate the performance of summarization systems has been a pressing need. However, current automatic text evaluation metrics can only measure one or certain aspects of the quality between two summary texts and do not agree with human judgments consistently. In this paper, we show that combining multiple well-chosen evaluation metrics and training predictive models using human annotated datasets can lead to more reliable evaluation scores than using any individual automatic metric. Our predictive models trained on a human annotated subset of the CNN\/DailyMail corpus demonstrate significant improvements (e.g. approximately 25% along coherence dimension) over selected individual metrics. Furthermore, a concise meta-evaluation on automatic metrics is provided along with an analysis of the performance of 12 predictive models. We also investigate the sensitivity of automatic metrics when mixed together for training these models. We have made the code, the instructions for experiment setup, and the trained models available as a tool for comparing and evaluating text summarization systems. <jats:sup>a<\/jats:sup> <\/jats:p><jats:p> <jats:sup>a<\/jats:sup> https:\/\/github.com\/bzhao2718\/ReliableSummEvalReg . <\/jats:p>","DOI":"10.1142\/s0218001422510119","type":"journal-article","created":{"date-parts":[[2022,6,4]],"date-time":"2022-06-04T15:48:34Z","timestamp":1654357714000},"source":"Crossref","is-referenced-by-count":2,"title":["Towards a Reliable Text Summarization Evaluation Metric Using Predictive Models"],"prefix":"10.1142","volume":"36","author":[{"given":"Bo","family":"Zhao","sequence":"first","affiliation":[{"name":"School of Computer Science and Mathematics, University of Central Missouri, Warrensburg, MO 64093, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5156-0310","authenticated-orcid":false,"given":"Yui Man","family":"Lui","sequence":"additional","affiliation":[{"name":"School of Computer Science and Mathematics, University of Central Missouri, Warrensburg, MO 64093, USA"}]}],"member":"219","published-online":{"date-parts":[[2022,7,22]]},"reference":[{"key":"S0218001422510119BIB001","first-page":"1039","volume-title":"The 26th Int. Conf. Computational Linguistics","author":"Benikova D.","year":"2016"},{"key":"S0218001422510119BIB005","doi-asserted-by":"publisher","DOI":"10.1007\/BF00058655"},{"key":"S0218001422510119BIB006","first-page":"1683","volume":"26","author":"Breiman L.","year":"1998","journal-title":"Ann. Prob."},{"key":"S0218001422510119BIB007","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007563306331"},{"volume-title":"Classification and Regression Trees","year":"1984","author":"Breiman L.","key":"S0218001422510119BIB008"},{"key":"S0218001422510119BIB013","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1264"},{"key":"S0218001422510119BIB014","first-page":"145","volume-title":"Proceedings of the 22nd International Conference on Computational Linguistics","author":"Conroy J. M.","year":"2008"},{"volume-title":"Proceedings of the Fifth Document Understanding Conference (DUC)","year":"2005","author":"Dang H.","key":"S0218001422510119BIB015"},{"key":"S0218001422510119BIB016","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(05)80023-1"},{"key":"S0218001422510119BIB017","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488420"},{"key":"S0218001422510119BIB022","doi-asserted-by":"publisher","DOI":"10.20532\/cit.2017.1003398"},{"key":"S0218001422510119BIB023","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.04.001"},{"volume-title":"Straightforward Statistics for the Behavioral Sciences","year":"1996","author":"Evans J. D.","key":"S0218001422510119BIB024"},{"key":"S0218001422510119BIB027","doi-asserted-by":"publisher","DOI":"10.1006\/jcss.1997.1504"},{"key":"S0218001422510119BIB028","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"S0218001422510119BIB030","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(89)90049-0"},{"volume-title":"Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems","year":"2019","author":"Geron A.","key":"S0218001422510119BIB031"},{"key":"S0218001422510119BIB034","first-page":"235","volume-title":"Proc. Twelfth Int. Florida Artificial Intelligence Research Society Conf.","author":"Hall M. A.","year":"1999"},{"key":"S0218001422510119BIB037","first-page":"278","volume-title":"Proc. 3rd Int. Conf. Document Analysis and Recognition","author":"Ho T. K.","year":"1995"},{"key":"S0218001422510119BIB038","first-page":"81","volume-title":"Advances in Automatic Text Summarization","volume":"14","author":"Hovy E.","year":"1999"},{"volume-title":"Deep Learning","year":"2016","author":"Ian G.","key":"S0218001422510119BIB041"},{"volume-title":"An Introduction to Statistical Learning: With Applications in R","year":"2014","author":"James G.","key":"S0218001422510119BIB042"},{"key":"S0218001422510119BIB045","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(97)00043-X"},{"key":"S0218001422510119BIB047","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1958.10501481"},{"key":"S0218001422510119BIB050","doi-asserted-by":"publisher","DOI":"10.3115\/1626355.1626389"},{"key":"S0218001422510119BIB051","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.1988.10475524"},{"key":"S0218001422510119BIB053","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin C.-Y.","year":"2004"},{"key":"S0218001422510119BIB054","doi-asserted-by":"publisher","DOI":"10.3115\/1220835.1220894"},{"key":"S0218001422510119BIB056","first-page":"1552","volume-title":"Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics","author":"Mairesse F.","year":"2010"},{"key":"S0218001422510119BIB057","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324901002741"},{"key":"S0218001422510119BIB058","first-page":"957","volume-title":"ICML\u201915: Proc. 32nd Int. Conf. Machine Learning","author":"Matt J. K.","year":"2015"},{"key":"S0218001422510119BIB060","first-page":"6690","volume-title":"Proc. 12th Language Resources and Evaluation Conf.","author":"Mieskes M.","year":"2020"},{"key":"S0218001422510119BIB062","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPCT.2016.7530193"},{"key":"S0218001422510119BIB063","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K16-1028"},{"key":"S0218001422510119BIB066","first-page":"145","volume-title":"HLT-NAACL","author":"Nenkova A.","year":"2004"},{"key":"S0218001422510119BIB068","first-page":"243","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop","author":"Nova E.","year":"2019"},{"key":"S0218001422510119BIB069","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1238"},{"key":"S0218001422510119BIB070","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics","author":"Papineni K.","year":"2002"},{"key":"S0218001422510119BIB072","first-page":"2825","volume":"12","author":"Pedregosa F.","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"S0218001422510119BIB074","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4510"},{"key":"S0218001422510119BIB075","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671927"},{"key":"S0218001422510119BIB077","doi-asserted-by":"publisher","DOI":"10.3390\/a5040398"},{"key":"S0218001422510119BIB078","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28569-1_1"},{"key":"S0218001422510119BIB079","first-page":"69","volume":"58","author":"Ellouze S.","year":"2017","journal-title":"Proces. Leng. Natural"},{"key":"S0218001422510119BIB080","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2014.09.003"},{"issue":"24","key":"S0218001422510119BIB081","first-page":"018","volume":"3","author":"Sebastian R.","journal-title":"J. Open Source Softw."},{"key":"S0218001422510119BIB082","volume":"349","author":"Sedgwick P.","year":"2014","journal-title":"BMJ"},{"key":"S0218001422510119BIB086","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/9365340"},{"volume-title":"Introduction to Data Mining","year":"2019","author":"Tan P.","key":"S0218001422510119BIB087"},{"key":"S0218001422510119BIB088","doi-asserted-by":"publisher","DOI":"10.3115\/1073445.1073478"},{"key":"S0218001422510119BIB089","first-page":"2692","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems","author":"Vinyals O.","year":"2015"},{"key":"S0218001422510119BIB094","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.455"},{"key":"S0218001422510119BIB095","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10336"},{"key":"S0218001422510119BIB099","doi-asserted-by":"crossref","unstructured":"W. Zhao,  M. Peyrard,  F. Liu,  Y. Gao,  C. M. Meyer and  S. Eger ,  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance  (Association for Computational Linguistics,  2019).","DOI":"10.18653\/v1\/D19-1053"}],"container-title":["International Journal of Pattern Recognition and Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0218001422510119","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,30]],"date-time":"2022-08-30T03:38:48Z","timestamp":1661830728000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0218001422510119"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,22]]},"references-count":54,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2022,8]]}},"alternative-id":["10.1142\/S0218001422510119"],"URL":"https:\/\/doi.org\/10.1142\/s0218001422510119","relation":{},"ISSN":["0218-0014","1793-6381"],"issn-type":[{"type":"print","value":"0218-0014"},{"type":"electronic","value":"1793-6381"}],"subject":[],"published":{"date-parts":[[2022,7,22]]},"article-number":"2251011"}}