{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T04:02:28Z","timestamp":1784865748123,"version":"3.55.0"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T00:00:00Z","timestamp":1784851200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T00:00:00Z","timestamp":1784851200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004909","name":"Universidade Federal Do Rio Grande Do Sul","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004909","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Evaluating the quality of long-form answers generated by Question Answering systems presents significant challenges. Traditional metrics, such as BLEU and ROUGE, often reduce the assessment to a single similarity score with a reference answer, failing to capture semantic and specific aspects of answer quality. This reliance on an aggregated score not only overlooks important dimensions but also depends heavily on the availability of reference answers, which may not always be practical or sufficient. Developing metrics capable of individually assessing specific criteria, particularly completeness and relevance, is crucial for identifying weaknesses and guiding improvements in these systems. To address these limitations, this paper introduces specialized metrics designed to evaluate completeness and relevance of long answers without the need for reference texts. We present a new dataset comprising long answers to instructional questions in Computer Science, annotated by human experts based on completeness and relevance. Building upon this, we propose three novel metric models: (1) a prompt-based strategy utilizing Large Language Models to assess answers, (2) an approach that adapts precision and recall concepts by segmenting answers into discrete information units, and (3) a regression model trained on synthetic data to predict completeness and relevance scores. Experimental results demonstrate that the proposed metrics closely align with human judgments and provide more detailed evaluations of completeness and relevance compared to traditional metrics. By enabling a more granular assessment, these metrics facilitate targeted refinements in QA systems, enhancing their ability to meet users\u2019 informational needs more effectively.<\/jats:p>","DOI":"10.1007\/s10579-026-09936-6","type":"journal-article","created":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T03:22:31Z","timestamp":1784863351000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Beyond accuracy: completeness and relevance metrics for evaluating the quality of long answers"],"prefix":"10.1007","volume":"60","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0272-1982","authenticated-orcid":false,"given":"Eduardo G.","family":"Cortes","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renata","family":"Vieira","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dante A. C.","family":"Barone","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,7,24]]},"reference":[{"key":"9936_CR1","doi-asserted-by":"crossref","unstructured":"Aharoni, R., Narayan, S., Maynez, J., Herzig, J., Clark, E., & Lapata, M. (2023). Multilingual summarization with factual consistency evaluation. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Findings of the association for computational linguistics: ACL 2023, pp. 3562\u20133591. Association for Computational Linguistics, Toronto, Canada. https:\/\/doi.org\/10.18653\/v1\/2023.findings-acl.220. https:\/\/aclanthology.org\/2023.findings-acl.220","DOI":"10.18653\/v1\/2023.findings-acl.220"},{"key":"9936_CR2","unstructured":"Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., & Wang, T. (2018). MS MARCO: A human generated machine reading comprehension dataset"},{"key":"9936_CR3","doi-asserted-by":"publisher","unstructured":"Bidgoly, A.J., Amirkhani, H., & Baradaran, R. (2022). Clustering-based sequence to sequence model for generative question answering in a low-resource language. ACM Trans. Asian Low-Resour. Lang. Inf. Process. 22(2) https:\/\/doi.org\/10.1145\/3563036","DOI":"10.1145\/3563036"},{"key":"9936_CR4","doi-asserted-by":"publisher","unstructured":"Bolotova, V., Blinov, V., Scholer, F., Croft, W.B., & Sanderson, M. (2022). A non-factoid question-answering taxonomy. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3477495.3531926.","DOI":"10.1145\/3477495.3531926"},{"key":"9936_CR5","doi-asserted-by":"crossref","unstructured":"Bolotova-Baranova, V., Blinov, V., Filippova, S., Scholer, F., & Sanderson, M. (2023). WikiHowQA: A comprehensive benchmark for multi-document non-factoid question answering. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp. 5291\u20135314. Association for Computational Linguistics, Toronto, Canada. https:\/\/doi.org\/10.18653\/v1\/2023.acl-long.290. https:\/\/aclanthology.org\/2023.acl-long.290","DOI":"10.18653\/v1\/2023.acl-long.290"},{"key":"9936_CR6","doi-asserted-by":"publisher","unstructured":"Cambazoglu, B.B., Baranova, V., Scholer, F., Sanderson, M., Tavakoli, L., & Croft, B. (2021). Quantifying human-perceived answer utility in non-factoid question answering. In: Proceedings of the 2021 conference on human information interaction and retrieval. CHIIR \u201921, pp. 75\u201384. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3406522.3446028.","DOI":"10.1145\/3406522.3446028"},{"key":"9936_CR7","doi-asserted-by":"crossref","unstructured":"Cegin, J., Simko, J., & Brusilovsky, P. (2023). ChatGPT to replace crowdsourcing of paraphrases for intent classification: Higher diversity and comparable model robustness. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 1889\u20131905. Association for Computational Linguistics, Singapore. https:\/\/doi.org\/10.18653\/v1\/2023.emnlp-main.117. https:\/\/aclanthology.org\/2023.emnlp-main.117","DOI":"10.18653\/v1\/2023.emnlp-main.117"},{"key":"9936_CR8","doi-asserted-by":"publisher","unstructured":"Chen, Y., & Eger, S. (2023). MENLI: Robust evaluation metrics from natural language inference. Transactions of the Association for Computational Linguistics, 11, 804\u2013825 https:\/\/doi.org\/10.1162\/tacl_a_00576https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00576","DOI":"10.1162\/tacl_a_00576"},{"issue":"3","key":"9936_CR9","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.lisr.2018.09.005","volume":"40","author":"SKW Chu","year":"2018","unstructured":"Chu, S. K. W., Huang, H., Wong, W. N. M., van Ginneken, W. F., Wu, K. M., & Hung, M. Y. (2018). Quality and clarity of health information on q&a sites. Library & Information Science Research, 40(3), 237\u2013244. https:\/\/doi.org\/10.1016\/j.lisr.2018.09.005","journal-title":"Library & Information Science Research"},{"issue":"3","key":"9936_CR10","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1007\/s10844-021-00655-8","volume":"58","author":"EG Cortes","year":"2022","unstructured":"Cortes, E. G., Woloszyn, V., Barone, D., M\u00f6ller, S., & Vieira, R. (2022). A systematic review of question answering systems for non-factoid questions. Journal of Intelligent Information Systems, 58(3), 453\u2013480. https:\/\/doi.org\/10.1007\/s10844-021-00655-8","journal-title":"Journal of Intelligent Information Systems"},{"key":"9936_CR11","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2023.101486","volume":"80","author":"K Darvishi","year":"2023","unstructured":"Darvishi, K., Shahbodaghkhan, N., Abbasiantaeb, Z., & Momtazi, S. (2023). Pquad: A persian question answering dataset. Computer Speech & Language, 80, Article 101486. https:\/\/doi.org\/10.1016\/j.csl.2023.101486","journal-title":"Computer Speech & Language"},{"key":"9936_CR12","doi-asserted-by":"publisher","unstructured":"Deng, Y., Zhang, W., Xu, W., Shen, Y., & Lam, W. (2023). Nonfactoid question answering as query-focused summarization with graph-enhanced multihop inference. IEEE Transactions on Neural Networks and Learning Systems, 1\u201315. https:\/\/doi.org\/10.1109\/TNNLS.2023.3258413","DOI":"10.1109\/TNNLS.2023.3258413"},{"key":"9936_CR13","doi-asserted-by":"publisher","unstructured":"Deutsch, D., Bedrax-Weiss, T., & Roth, D. (2021). Towards question-answering as an automatic metric for evaluating the content quality of a summary. Transactions of the Association for Computational Linguistics, 9, 774\u2013789 https:\/\/doi.org\/10.1162\/tacl_a_00397https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00397","DOI":"10.1162\/tacl_a_00397"},{"key":"9936_CR14","doi-asserted-by":"publisher","unstructured":"Deutsch, D., & Roth, D. (2021). Understanding the extent to which content quality metrics measure the information quality of summaries. In A. Bisazza & O. Abend (Eds.), Proceedings of the 25th conference on computational natural language learning (pp. 300\u2013309). Online: Association for Computational Linguistics. https:\/\/doi.org\/10.18653\/v1\/2021.conll-1.24","DOI":"10.18653\/v1\/2021.conll-1.24"},{"key":"9936_CR15","unstructured":"Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs\/1810.04805 arXiv:1810.04805"},{"key":"9936_CR16","unstructured":"Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding"},{"key":"9936_CR17","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-023-09702-y","author":"K Doxolodeo","year":"2024","unstructured":"Doxolodeo, K., & Krisnadhi, A. A. (2024). AC-IQuAD: Automatically constructed indonesian question answering dataset by leveraging wikidata. Language Resources and Evaluation. https:\/\/doi.org\/10.1007\/s10579-023-09702-y","journal-title":"Language Resources and Evaluation"},{"key":"9936_CR18","doi-asserted-by":"crossref","unstructured":"Durmus, E., He, H., & Diab, M. (2020). FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 5055\u20135070. Association for Computational Linguistics, Online. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.454. https:\/\/aclanthology.org\/2020.acl-main.454","DOI":"10.18653\/v1\/2020.acl-main.454"},{"key":"9936_CR19","doi-asserted-by":"publisher","unstructured":"Fabbri, A.R., Kry\u015bci\u0144ski, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021). Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9, 391\u2013409 https:\/\/doi.org\/10.1162\/tacl_a_00373https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00373","DOI":"10.1162\/tacl_a_00373"},{"key":"9936_CR20","doi-asserted-by":"crossref","unstructured":"Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022). QAFactEval: Improved QA-based factual consistency evaluation for summarization. In: Carpuat, M., Marneffe, M.-C., Meza\u00a0Ruiz, I.V. (eds.) Proceedings of the 2022 conference of the north American chapter of the association for computational linguistics: human language technologies, pp. 2587\u20132601. Association for Computational Linguistics, Seattle, United States. https:\/\/doi.org\/10.18653\/v1\/2022.naacl-main.187. https:\/\/aclanthology.org\/2022.naacl-main.187","DOI":"10.18653\/v1\/2022.naacl-main.187"},{"key":"9936_CR21","doi-asserted-by":"crossref","unstructured":"Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019). Eli5: Long form question answering. Proceedings of ACL 2019","DOI":"10.18653\/v1\/P19-1346"},{"key":"9936_CR22","doi-asserted-by":"crossref","unstructured":"Gao, M., Hu, X., Ruan, J., Pu, X., & Wan, X. (2024). LLM-based NLG evaluation: Current status and challenges","DOI":"10.1162\/coli_a_00561"},{"key":"9936_CR23","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1613\/jair.1.13715","volume":"77","author":"S Gehrmann","year":"2023","unstructured":"Gehrmann, S., Clark, E., & Sellam, T. (2023). Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77, 103\u2013166.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"9936_CR24","unstructured":"Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding arXiv:2009.03300 arXiv preprint."},{"key":"9936_CR25","doi-asserted-by":"crossref","unstructured":"Hengle, A., Padhi, A.K., Bandhakavi, A., & Chakraborty, T. (2025). CSEval: Towards automated, multi-dimensional, and reference-free counterspeech evaluation using auto-calibrated LLMs. In: Chiruzzo, L., Ritter, A., Wang, L. (eds.) Proceedings of the 2025 conference of the nations of the Americas chapter of the association for computational linguistics: human language technologies (Volume 1: Long Papers), pp. 5402\u20135419. Association for Computational Linguistics, Albuquerque, New Mexico. https:\/\/doi.org\/10.18653\/v1\/2025.naacl-long.279. https:\/\/aclanthology.org\/2025.naacl-long.279\/","DOI":"10.18653\/v1\/2025.naacl-long.279"},{"key":"9936_CR26","doi-asserted-by":"crossref","unstructured":"Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y.(2022). TRUE: Re-evaluating factual consistency evaluation. In: Carpuat, M., Marneffe, M.-C., Meza\u00a0Ruiz, I.V. (eds.) Proceedings of the 2022 conference of the north American chapter of the association for computational linguistics: human language technologies, pp. 3905\u20133920. Association for Computational Linguistics, Seattle, United States. https:\/\/doi.org\/10.18653\/v1\/2022.naacl-main.287. https:\/\/aclanthology.org\/2022.naacl-main.287","DOI":"10.18653\/v1\/2022.naacl-main.287"},{"key":"9936_CR27","doi-asserted-by":"crossref","unstructured":"Isabelle, P., Cherry, C., & Foster, G. (2017). A challenge set approach to evaluating machine translation","DOI":"10.18653\/v1\/D17-1263"},{"key":"9936_CR28","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/978-3-642-22688-5_10","volume-title":"Conceptual structures for discovering knowledge","author":"M Keeler","year":"2011","unstructured":"Keeler, M. (2011). Crowdsourced knowledge: Peril and promise for conceptual structures research. In S. Andrews, S. Polovina, R. Hill, & B. Akhgar (Eds.), Conceptual structures for discovering knowledge (pp. 131\u2013144). Berlin, Heidelberg: Springer."},{"issue":"4","key":"9936_CR29","doi-asserted-by":"publisher","first-page":"3701","DOI":"10.1007\/s13369-020-05236-5","volume":"46","author":"AFUR Khilji","year":"2021","unstructured":"Khilji, A. F. U. R., Manna, R., Laskar, S. R., Pakray, P., Das, D., Bandyopadhyay, S., & Gelbukh, A. (2021). Cookingqa: answering questions and recommending recipes based on ingredients. Arabian Journal for Science and Engineering, 46(4), 3701\u20133712.","journal-title":"Arabian Journal for Science and Engineering"},{"key":"9936_CR30","doi-asserted-by":"publisher","unstructured":"Kim, Y., Oh, D., & Huang, H. H. (2025). Syncse: syntax graph-based contrastive learning of sentence embeddings. Expert Systems with Applications, 287, 128047. https:\/\/doi.org\/10.1016\/j.eswa.2025.128047","DOI":"10.1016\/j.eswa.2025.128047"},{"key":"9936_CR31","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1162\/tacl_a_00023","volume":"6","author":"T Ko\u010disk\u00fd","year":"2018","unstructured":"Ko\u010disk\u00fd, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018). The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6, 317\u2013328. https:\/\/doi.org\/10.1162\/tacl_a_00023","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"9936_CR32","unstructured":"Kocmi, T., & Federmann, C. (2023a). Large language models are state-of-the-art evaluators of translation quality. In: Nurminen, M., Brenner, J., Koponen, M., Latomaa, S., Mikhailov, M., Schierl, F., Ranasinghe, T., Vanmassenhove, E., Vidal, S.A., Aranberri, N., Nunziatini, M., Escart\u00edn, C.P., Forcada, M., Popovic, M., Scarton, C., Moniz, H. (eds.) Proceedings of the 24th annual conference of the European association for machine translation, pp. 193\u2013203. European Association for Machine Translation, Tampere, Finland. https:\/\/aclanthology.org\/2023.eamt-1.19"},{"key":"9936_CR33","doi-asserted-by":"crossref","unstructured":"Kocmi, T., & Federmann, C. (2023b). GEMBA-MQM: Detecting translation quality error spans with GPT-4","DOI":"10.18653\/v1\/2023.wmt-1.64"},{"key":"9936_CR34","doi-asserted-by":"crossref","unstructured":"Krishna, K., Chang, Y., Wieting, J., & Iyyer, M. (2022). Rankgen: Improving text generation with large ranking models. Empirical methods in natural language processing","DOI":"10.18653\/v1\/2022.emnlp-main.15"},{"key":"9936_CR35","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1162\/tacl_a_00276","volume":"7","author":"T Kwiatkowski","year":"2019","unstructured":"Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, 453\u2013466.","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"9936_CR36","doi-asserted-by":"publisher","unstructured":"Li, B., Yang, P., Zhao, H., Zhang, P., & Liu, Z. (2023). Hierarchical sliding inference generator for question-driven abstractive answer summarization. ACM Trans. Inf. Syst. 41(1) https:\/\/doi.org\/10.1145\/3511891","DOI":"10.1145\/3511891"},{"issue":"6","key":"9936_CR40","doi-asserted-by":"publisher","first-page":"887","DOI":"10.1108\/AJIM-03-2020-0085","volume":"72","author":"L Li","year":"2020","unstructured":"Li, L., Zhang, C., & He, D. (2020a). Factors influencing the importance of criteria for judging answer quality on academic social q&a platforms. Aslib Journal of Information Management, 72(6), 887\u2013907.","journal-title":"Aslib Journal of Information Management"},{"key":"9936_CR37","doi-asserted-by":"publisher","unstructured":"Li, L., Zhang, L., Wang, A., & Huang, K. (2020b). Investigating factors for assessing the quality of academic user-generated content on social media. In Proceedings of the ACM\/IEEE joint conference on digital libraries in 2020. JCDL \u201920, pp. 511\u2013512. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3383583.3398588.","DOI":"10.1145\/3383583.3398588"},{"issue":"8","key":"9936_CR41","doi-asserted-by":"publisher","first-page":"8688","DOI":"10.1609\/aaai.v38i8.28714","volume":"38","author":"X Li","year":"2024","unstructured":"Li, X., Zhou, Y., & Dou, Z. (2024). Unigen: A unified generative framework for retrieval and question answering with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(8), 8688\u20138696. https:\/\/doi.org\/10.1609\/aaai.v38i8.28714","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"9936_CR38","unstructured":"Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In Text summarization branches Out, pp. 74\u201381. Association for Computational Linguistics, Barcelona, Spain. https:\/\/aclanthology.org\/W04-1013"},{"key":"9936_CR39","doi-asserted-by":"crossref","unstructured":"Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human alignment","DOI":"10.18653\/v1\/2023.emnlp-main.153"},{"key":"9936_CR42","doi-asserted-by":"crossref","unstructured":"Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2024). ExpertQA: Expert-curated questions and attributed answers","DOI":"10.18653\/v1\/2024.naacl-long.167"},{"key":"9936_CR43","doi-asserted-by":"crossref","unstructured":"Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023). FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 12076\u201312100. Association for Computational Linguistics, Singapore. https:\/\/doi.org\/10.18653\/v1\/2023.emnlp-main.741. https:\/\/aclanthology.org\/2023.emnlp-main.741","DOI":"10.18653\/v1\/2023.emnlp-main.741"},{"key":"9936_CR44","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-024-09745-9","author":"M Mountantonakis","year":"2024","unstructured":"Mountantonakis, M., Mertzanis, L., Bastakis, M., & Tzitzikas, Y. (2024). A comparative evaluation for question answering over greek texts by using machine translation and bert. Language Resources and Evaluation. https:\/\/doi.org\/10.1007\/s10579-024-09745-9","journal-title":"Language Resources and Evaluation"},{"key":"9936_CR45","unstructured":"Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., & Schulman, J. (2022). WebGPT: Browser-assisted question-answering with human feedback"},{"key":"9936_CR46","doi-asserted-by":"crossref","unstructured":"Ostyakova, L., Smilga, V., Petukhova, K., Molchanova, M., & Kornev, D. (2023). ChatGPT vs. crowdsourcing vs. experts: Annotating open-domain conversations with speech functions. In: Stoyanchev, S., Joty, S., Schlangen, D., Dusek, O., Kennington, C., Alikhani, M. (eds.) Proceedings of the 24th annual meeting of the special interest group on discourse and dialogue, pp. 242\u2013254. Association for Computational Linguistics, Prague, Czechia. https:\/\/doi.org\/10.18653\/v1\/2023.sigdial-1.23. https:\/\/aclanthology.org\/2023.sigdial-1.23","DOI":"10.18653\/v1\/2023.sigdial-1.23"},{"key":"9936_CR47","doi-asserted-by":"publisher","unstructured":"Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. ACL \u201902, pp. 311\u2013318. Association for Computational Linguistics, USA. https:\/\/doi.org\/10.3115\/1073083.1073135.","DOI":"10.3115\/1073083.1073135"},{"key":"9936_CR48","doi-asserted-by":"crossref","unstructured":"Pu, A., Chung, H. W., Parikh, A. P., Gehrmann, S., & Sellam, T. (2021). Learning compact metrics for mt. Proceedings of EMNLP","DOI":"10.18653\/v1\/2021.emnlp-main.58"},{"key":"9936_CR49","doi-asserted-by":"crossref","unstructured":"Sellam, T., Das, D., & Parikh, A. P. (2020). Bleurt: Learning robust metrics for text generation. Proceedings of ACL","DOI":"10.18653\/v1\/2020.acl-main.704"},{"key":"9936_CR50","unstructured":"Srivastava, A., & Rastogi, A., et.al. (2023) A.R.: Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615"},{"key":"9936_CR51","doi-asserted-by":"crossref","unstructured":"Sulem, E., Abend, O., & Rappoport, A. (2018). BLEU is not suitable for the evaluation of text simplification. In: Riloff, E., Chiang, D., Hockenmaier, J., Tsujii, J. (eds.) Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 738\u2013744. Association for Computational Linguistics, Brussels, Belgium. https:\/\/doi.org\/10.18653\/v1\/D18-1081. https:\/\/aclanthology.org\/D18-1081","DOI":"10.18653\/v1\/D18-1081"},{"key":"9936_CR52","unstructured":"T\u00f6rnberg, P. (2023). ChatGPT-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning"},{"key":"9936_CR53","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-031-40498-6_1","volume-title":"Text, speech, and dialogue","author":"X Wang","year":"2023","unstructured":"Wang, X., Li, T., Tamura, T., Nishida, S., Zhu, F., & Utsuro, T. (2023). Japanese how-to tip machine reading comprehension by multi-task learning based on generative model. In K. Ek\u0161tein, F. P\u00e1rtl, & M. Konop\u00edk (Eds.), Text, speech, and dialogue (pp. 3\u201314). Cham: Springer."},{"key":"9936_CR54","doi-asserted-by":"crossref","unstructured":"Xu, F., Song, Y., Iyyer, M., & Choi, E. (2023). A critical evaluation of evaluations for long-form question answering. Association of Computational Linguistics","DOI":"10.18653\/v1\/2023.acl-long.181"},{"key":"9936_CR55","unstructured":"Yuan, W., Neubig, G., & Liu, P. (2021). BARTScore: Evaluating generated text as text generation"},{"key":"9936_CR56","unstructured":"Zhang*, T., Kishore*, V., Wu*, F., Weinberger, K.Q., & Artzi, Y. (2020). BERTScore: Evaluating text generation with BERT. https:\/\/openreview.net\/forum?id=SkeHuCVFDr"}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-026-09936-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-026-09936-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-026-09936-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T03:22:51Z","timestamp":1784863371000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-026-09936-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,24]]},"references-count":56,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,9]]}},"alternative-id":["9936"],"URL":"https:\/\/doi.org\/10.1007\/s10579-026-09936-6","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"value":"1574-020X","type":"print"},{"value":"1574-0218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,24]]},"assertion":[{"value":"6 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 July 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no conflict of interest.","order":1,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"58"}}