{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T22:58:20Z","timestamp":1781650700611,"version":"3.54.5"},"reference-count":84,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T00:00:00Z","timestamp":1776729600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T00:00:00Z","timestamp":1776729600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100022075","name":"Universidade Federal Do Esp\u00edrito Santo","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100022075","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2026,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Evaluating open-ended question answering (QA) remains challenging, as traditional metrics often fail to reflect semantic correctness, especially in cases with paraphrastic variation or multiple valid answers. To deal with this challenge, we propose\n                    <jats:italic>Score2Choice<\/jats:italic>\n                    , a structured evaluation framework that reformulates QA evaluation as a multiple-choice selection task. This setup enables similarity-based metrics to be interpreted via accuracy, enhancing transparency and comparability. To support this approach, we introduce\n                    <jats:italic>WikiTrapQA<\/jats:italic>\n                    , a new MCQA dataset built from recent Wikipedia content and enriched with paraphrased and adversarial answers. Alongside a reformulated version of TruthfulQA, this dataset allows us to systematically compare lexical, semantic, and LLM-based metrics. A preliminary score distribution analysis reveals that many metrics struggle to distinguish correct from incorrect answers based on similarity scores alone. Experimental results show that LLM-based methods, in our case LLaMA 3, achieve the highest discriminative performance, while Sentence-BERT and BARTScore emerge as strong non-LLM alternatives. Our findings highlight the limitations of surface-level metrics and demonstrate the value of Score2Choice as a reproducible and interpretable framework for QA evaluation.\n                  <\/jats:p>","DOI":"10.1007\/s00521-026-12053-8","type":"journal-article","created":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T02:35:08Z","timestamp":1776738908000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Exploring question answering: metric analysis and evaluation framework for enhanced interpretability"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-9200-257X","authenticated-orcid":false,"given":"Let\u00edcia C.","family":"Navarro","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"S\u00e9rgio S.","family":"Mucciaccia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Filipe","family":"Mutz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thiago M.","family":"Paix\u00e4o","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claudine","family":"Badue","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alberto F.","family":"De Souza","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thiago","family":"Oliveira-Santos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,4,21]]},"reference":[{"key":"12053_CR1","doi-asserted-by":"crossref","unstructured":"Agarwal A, Sachdeva N, Yadav RK, Udandarao V, Mittal V, Gupta A, Mathur A (2019) Eduqa: Educational domain question answering system using conceptual network mapping. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8137\u20138141. IEEE","DOI":"10.1109\/ICASSP.2019.8683538"},{"key":"12053_CR2","doi-asserted-by":"publisher","unstructured":"Gautre TA, Khan TH (2018) An analysis of question answering system for education empowered by crowdsourcing. In: 2018 2nd International Conference on Inventive Systems and Control (ICISC), pp. 955\u2013958. https:\/\/doi.org\/10.1109\/ICISC.2018.8398942","DOI":"10.1109\/ICISC.2018.8398942"},{"key":"12053_CR3","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2024.100252","volume":"6","author":"J Ling","year":"2024","unstructured":"Ling J, Afzaal M (2024) Automatic question-answer pairs generation using pre-trained large language models in higher education. Computers and Education: Artificial Intelligence 6:100252. https:\/\/doi.org\/10.1016\/j.caeai.2024.100252","journal-title":"Computers and Education: Artificial Intelligence"},{"issue":"11","key":"12053_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0242061","volume":"15","author":"Y Yan","year":"2020","unstructured":"Yan Y, Zhang B-W, Li X-F, Liu Z (2020) List-wise learning to rank biomedical question-answer pairs with deep ranking recursive autoencoders. PLoS One 15(11):1\u201319. https:\/\/doi.org\/10.1371\/journal.pone.0242061","journal-title":"PLoS One"},{"key":"12053_CR5","doi-asserted-by":"crossref","unstructured":"Nishida K, Nishida K, Yoshida S (2021) Task-adaptive pre-training of language models with word embedding regularization. arXiv preprint arXiv:2109.08354","DOI":"10.18653\/v1\/2021.findings-acl.398"},{"issue":"1","key":"12053_CR6","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1109\/TCBB.2022.3161032","volume":"20","author":"J Bai","year":"2023","unstructured":"Bai J, Yin C, Zhang J, Wang Y, Dong Y, Rong W, Xiong Z (2023) Adversarial knowledge distillation based biomedical factoid question answering. IEEE ACM Trans Comput Biol Bioinform 20(1):106\u2013118. https:\/\/doi.org\/10.1109\/TCBB.2022.3161032","journal-title":"IEEE ACM Trans Comput Biol Bioinform"},{"key":"12053_CR7","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-022-04751-6","volume":"23","author":"S Raza","year":"2022","unstructured":"Raza S, Schwartz B, Rosella L (2022) Coquad: a covid-19 question answering dataset system, facilitating research, benchmarking, and practice. BMC Bioinformatics 23:210. https:\/\/doi.org\/10.1186\/s12859-022-04751-6","journal-title":"BMC Bioinformatics"},{"key":"12053_CR8","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2022.773329","volume":"16","author":"X Zhu","year":"2022","unstructured":"Zhu X, Chen Y, Gu Y, Xiao Z (2022) Sentimedqaer: a transfer learning-based sentiment-aware model for biomedical question answering. Front Neurorobot 16:773329. https:\/\/doi.org\/10.3389\/fnbot.2022.773329","journal-title":"Front Neurorobot"},{"key":"12053_CR9","unstructured":"Wang YJ, Li Y, Qin H, Guan Y, Chen S (2022) A novel deberta-based model for financial question answering task. arXiv preprint arXiv:2207.05875"},{"key":"12053_CR10","unstructured":"Wang B, Ju J, Mao Y, Dai X-Y, Huang S, Chen J (2022) A numerical reasoning question answering system with fine-grained retriever and the ensemble of multiple generators for finqa. arXiv preprint arXiv:2206.08506"},{"key":"12053_CR11","doi-asserted-by":"publisher","unstructured":"Nararatwong R, Kertkeidkachorn N, Ichise R (2022) KIQA: Knowledge-infused question answering model for financial table-text data. In: Agirre E, Apidianaki M, Vuli\u0107 I (eds.) Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 53\u201361. Association for Computational Linguistics, Dublin, Ireland and Online. https:\/\/doi.org\/10.18653\/v1\/2022.deelio-1.6. https:\/\/aclanthology.org\/2022.deelio-1.6\/","DOI":"10.18653\/v1\/2022.deelio-1.6"},{"key":"12053_CR12","doi-asserted-by":"crossref","unstructured":"Kamalloo E, Dziri N, Clarke CL, Rafiei D (2023) Evaluating open-domain question answering in the era of large language models. arXiv preprint arXiv:2305.06984","DOI":"10.18653\/v1\/2023.acl-long.307"},{"key":"12053_CR13","unstructured":"Gu J, Jiang X, Shi Z, Tan H, Zhai X, Xu C, Li W, Shen Y, Ma S, Liu H, et al. (2024) A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594"},{"key":"12053_CR14","first-page":"211","volume":"2","author":"A Allam","year":"2012","unstructured":"Allam A, Haggag M (2012) The question answering systems: a survey. Int J Res Rev Inf Sci 2:211\u2013221","journal-title":"Int J Res Rev Inf Sci"},{"issue":"24","key":"12053_CR15","doi-asserted-by":"publisher","first-page":"5412","DOI":"10.1016\/j.ins.2011.07.047","volume":"181","author":"O Kolomiyets","year":"2011","unstructured":"Kolomiyets O, Moens M-F (2011) A survey on question answering technology from an information retrieval perspective. Inf Sci 181(24):5412\u20135434","journal-title":"Inf Sci"},{"key":"12053_CR16","doi-asserted-by":"crossref","unstructured":"Li Z, Mondal I, Liang Y, Nghiem H, Boyd-Graber JL (2024) Pedants: Cheap but effective and interpretable answer equivalence. arXiv preprint arXiv:2402.11161","DOI":"10.18653\/v1\/2024.findings-emnlp.548"},{"key":"12053_CR17","doi-asserted-by":"crossref","unstructured":"Papineni K, Roukos S, Ward T, Zhu W-J (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311\u2013318","DOI":"10.3115\/1073083.1073135"},{"key":"12053_CR18","unstructured":"Chin-Yew L (2004) Rouge: A package for automatic evaluation of summaries. In: Proceedings of the Workshop on Text Summarization Branches Out, 2004"},{"key":"12053_CR19","unstructured":"Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y (2019) Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675"},{"key":"12053_CR20","doi-asserted-by":"crossref","unstructured":"Sellam T, Das D, Parikh AP (2020) Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696","DOI":"10.18653\/v1\/2020.acl-main.704"},{"key":"12053_CR21","unstructured":"Hendrycks D, Burns C, Basart S, Zou A, Mazeika M, Song D, Steinhardt J (2020) Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300"},{"key":"12053_CR22","unstructured":"Clark P, Cowhey I, Etzioni O, Khot T, Sabharwal A, Schoenick C, Tafjord O (2018) Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457"},{"key":"12053_CR23","doi-asserted-by":"crossref","unstructured":"Lin S, Hilton J, Evans O (2021) Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958","DOI":"10.18653\/v1\/2022.acl-long.229"},{"key":"12053_CR24","doi-asserted-by":"crossref","unstructured":"Lai G, Xie Q, Liu H, Yang Y, Hovy E (2017) Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683","DOI":"10.18653\/v1\/D17-1082"},{"key":"12053_CR25","doi-asserted-by":"crossref","unstructured":"Schmidtov\u00e1 P, Mahamood S, Balloccu S, Du\u0161ek O, Gatt A, Gkatzia D, Howcroft DM, Pl\u00e1tek O, Sivaprasad A (2024) Automatic metrics in natural language generation: A survey of current evaluation practices. arXiv preprint arXiv:2408.09169","DOI":"10.18653\/v1\/2024.inlg-main.44"},{"key":"12053_CR26","doi-asserted-by":"crossref","unstructured":"Shi Z, Chen X, Qiu X, Huang X (2018) Toward diverse text generation with inverse reinforcement learning. arXiv preprint arXiv:1804.11258","DOI":"10.24963\/ijcai.2018\/606"},{"key":"12053_CR27","doi-asserted-by":"crossref","unstructured":"Post M (2018) A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771","DOI":"10.18653\/v1\/W18-6319"},{"key":"12053_CR28","unstructured":"Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, et al. (2016) Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144"},{"key":"12053_CR29","unstructured":"Banerjee S, Lavie A (2005) Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation And\/or Summarization, pp. 65\u201372"},{"key":"12053_CR30","first-page":"27263","volume":"34","author":"W Yuan","year":"2021","unstructured":"Yuan W, Neubig G, Liu P (2021) Bartscore: evaluating generated text as text generation. Adv Neural Inf Process Syst 34:27263\u201327277","journal-title":"Adv Neural Inf Process Syst"},{"key":"12053_CR31","doi-asserted-by":"crossref","unstructured":"Zhao W, Peyrard M, Liu F, Gao Y, Meyer CM, Eger S (2019) Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622","DOI":"10.18653\/v1\/D19-1053"},{"key":"12053_CR32","first-page":"4816","volume":"34","author":"K Pillutla","year":"2021","unstructured":"Pillutla K, Swayamdipta S, Zellers R, Thickstun J, Welleck S, Choi Y, Harchaoui Z (2021) Mauve: measuring the gap between neural text and human text using divergence frontiers. Adv Neural Inf Process Syst 34:4816\u20134828","journal-title":"Adv Neural Inf Process Syst"},{"key":"12053_CR33","doi-asserted-by":"crossref","unstructured":"Ansch\u00fctz M, Lozano DM, Groh G (2023) This is not correct! negation-aware evaluation of language generation systems. arXiv preprint arXiv:2307.13989","DOI":"10.18653\/v1\/2023.inlg-main.12"},{"key":"12053_CR34","doi-asserted-by":"crossref","unstructured":"Zhao W, Strube M, Eger S (2022) Discoscore: Evaluating text generation with bert and discourse coherence. arXiv preprint arXiv:2201.11176","DOI":"10.18653\/v1\/2023.eacl-main.278"},{"key":"12053_CR35","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I (2019) Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084","DOI":"10.18653\/v1\/D19-1410"},{"key":"12053_CR36","doi-asserted-by":"crossref","unstructured":"Reimers N, Gurevych I (2020) Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv preprint arXiv:2004.09813","DOI":"10.18653\/v1\/2020.emnlp-main.365"},{"key":"12053_CR37","doi-asserted-by":"crossref","unstructured":"Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250","DOI":"10.18653\/v1\/D16-1264"},{"key":"12053_CR38","doi-asserted-by":"crossref","unstructured":"Chen D, Fisch A, Weston J, Bordes A (2017) Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051","DOI":"10.18653\/v1\/P17-1171"},{"key":"12053_CR39","unstructured":"Sun H, Zhou M (2012) Joint learning of a dual smt system for paraphrase generation. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 38\u201342"},{"key":"12053_CR40","doi-asserted-by":"crossref","unstructured":"Zhu Y, Lu S, Zheng L, Guo J, Zhang W, Wang J, Yu Y (2018) Texygen: A benchmarking platform for text generation models. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097\u20131100","DOI":"10.1145\/3209978.3210080"},{"key":"12053_CR41","doi-asserted-by":"crossref","unstructured":"Chen A, Stanovsky G, Singh S, Gardner M (2019) Evaluating question answering evaluation. In: Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 119\u2013124","DOI":"10.18653\/v1\/D19-5817"},{"key":"12053_CR42","doi-asserted-by":"crossref","unstructured":"Ng J-P, Abrecht V (2015) Better summarization evaluation with word embeddings for rouge. arXiv preprint arXiv:1508.06034","DOI":"10.18653\/v1\/D15-1222"},{"key":"12053_CR43","doi-asserted-by":"crossref","unstructured":"Xiang J, Liu Y, Cai D, Li H, Lian D, Liu L (2021) Assessing dialogue systems with distribution distances. arXiv preprint arXiv:2105.02573","DOI":"10.18653\/v1\/2021.findings-acl.193"},{"key":"12053_CR44","doi-asserted-by":"crossref","unstructured":"Ma\u00f1as O, Krojer B, Agrawal A (2024) Improving automatic vqa evaluation using large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 4171\u20134179","DOI":"10.1609\/aaai.v38i5.28212"},{"key":"12053_CR45","unstructured":"Krolik J, Mahal H, Ahmad F, Trivedi G, Saket B (2024) Towards leveraging large language models for automated medical q&a evaluation. arXiv preprint arXiv:2409.01941"},{"key":"12053_CR46","doi-asserted-by":"crossref","unstructured":"Fan Y, Liu Y, Yao Z, Yu J, Hou L, Li J (2024) Evaluating generative language models in information extraction as subjective question correction. arXiv preprint arXiv:2404.03532","DOI":"10.63317\/2rbzu6ev8x4m"},{"key":"12053_CR47","doi-asserted-by":"crossref","unstructured":"Bulian J, Buck C, Gajewski W, Boerschinger B, Schuster T (2022) Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. arXiv preprint arXiv:2202.07654","DOI":"10.18653\/v1\/2022.emnlp-main.20"},{"key":"12053_CR48","doi-asserted-by":"crossref","unstructured":"Si C, Zhao C, Boyd-Graber J (2021) What\u2019s in a name? answer equivalence for open-domain question answering. arXiv preprint arXiv:2109.05289","DOI":"10.18653\/v1\/2021.emnlp-main.757"},{"key":"12053_CR49","doi-asserted-by":"crossref","unstructured":"Ke P, Huang F, Mi F, Wang Y, Liu Q, Zhu X, Huang M (2023) Decompeval: Evaluating generated texts as unsupervised decomposed question answering. arXiv preprint arXiv:2307.06869","DOI":"10.18653\/v1\/2023.acl-long.539"},{"key":"12053_CR50","unstructured":"Lee Y, Kim J, Kim J, Cho H, Kang P (2024) Checkeval: Robust evaluation framework using large language model via checklist. arXiv preprint arXiv:2403.18771"},{"key":"12053_CR51","doi-asserted-by":"crossref","unstructured":"Zhong M, Liu Y, Yin D, Mao Y, Jiao Y, Liu P, Zhu C, Ji H, Han J (2022) Towards a unified multi-dimensional evaluator for text generation. arXiv preprint arXiv:2210.07197","DOI":"10.18653\/v1\/2022.emnlp-main.131"},{"key":"12053_CR52","doi-asserted-by":"crossref","unstructured":"Lin Y-T, Chen Y-N (2023) Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711","DOI":"10.18653\/v1\/2023.nlp4convai-1.5"},{"key":"12053_CR53","unstructured":"Fu J, Ng S-K, Jiang Z, Liu P (2023) Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166"},{"key":"12053_CR54","doi-asserted-by":"crossref","unstructured":"Xu F, Li JJ, Choi E (2022) How do we answer complex questions: Discourse structure of long-form answers. arXiv preprint arXiv:2203.11048","DOI":"10.18653\/v1\/2022.acl-long.249"},{"key":"12053_CR55","doi-asserted-by":"crossref","unstructured":"Liu Y, Su Y, Shareghi E, Collier N (2024) Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence. https:\/\/arxiv.org\/abs\/2402.10175","DOI":"10.18653\/v1\/2024.naacl-short.9"},{"key":"12053_CR56","doi-asserted-by":"crossref","unstructured":"Durmus E, He H, Diab M (2020) Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. arXiv preprint arXiv:2005.03754","DOI":"10.18653\/v1\/2020.acl-main.454"},{"key":"12053_CR57","doi-asserted-by":"crossref","unstructured":"Scialom T, Dray P-A, Lamprier S, Piwowarski B, Staiano J, Wang A, Gallinari P (2021) Questeval: Summarization asks for fact-based evaluation. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6594\u20136604","DOI":"10.18653\/v1\/2021.emnlp-main.529"},{"key":"12053_CR58","doi-asserted-by":"crossref","unstructured":"Fabbri AR, Wu C-S, Liu W, Xiong C (2022) Qafacteval: Improved qa-based factual consistency evaluation for summarization. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2587\u20132601","DOI":"10.18653\/v1\/2022.naacl-main.187"},{"key":"12053_CR59","doi-asserted-by":"crossref","unstructured":"Zha Y, Yang Y, Li R, Hu Z (2023) Alignscore: Evaluating factual consistency with a unified alignment function. arXiv preprint arXiv:2305.16739","DOI":"10.18653\/v1\/2023.acl-long.634"},{"key":"12053_CR60","doi-asserted-by":"crossref","unstructured":"Mustafazade F, Ebbinghaus PF (2022) Evaluation of semantic answer similarity metrics. arXiv preprint arXiv:2206.12664","DOI":"10.5121\/csit.2022.121109"},{"key":"12053_CR61","doi-asserted-by":"crossref","unstructured":"Chen A, Stanovsky G, Singh S, Gardner M (2020) Mocha: A dataset for training and evaluating generative reading comprehension metrics. arXiv preprint arXiv:2010.03636","DOI":"10.18653\/v1\/2020.emnlp-main.528"},{"key":"12053_CR62","doi-asserted-by":"crossref","unstructured":"Risch J, M\u00f6ller T, Gutsch J, Pietsch M (2021) Semantic answer similarity for evaluating question answering models. arXiv preprint arXiv:2108.06130","DOI":"10.18653\/v1\/2021.mrqa-1.15"},{"key":"12053_CR63","unstructured":"Zhu L, Wang X, Wang X (2023) Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631"},{"key":"12053_CR64","unstructured":"Li J, Sun S, Yuan W, Fan R-Z, Zhao H, Liu P (2023) Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470"},{"key":"12053_CR65","doi-asserted-by":"crossref","unstructured":"Xiong T, Wang X, Guo D, Ye Q, Fan H, Gu Q, Huang H, Li C (2024) Llava-critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02712","DOI":"10.1109\/CVPR52734.2025.01271"},{"key":"12053_CR66","doi-asserted-by":"crossref","unstructured":"Jones J, Mo L, Fosler-Lussier E, Sun H (2024) A multi-aspect framework for counter narrative evaluation using large language models. arXiv preprint arXiv:2402.11676","DOI":"10.18653\/v1\/2024.naacl-short.14"},{"key":"12053_CR67","unstructured":"Yuan Z, Yuan H, Tan C, Wang W, Huang S, Huang F (2023) Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302"},{"key":"12053_CR68","doi-asserted-by":"crossref","unstructured":"Qin Z, Jagerman R, Hui K, Zhuang H, Wu J, Yan L, Shen J, Liu T, Liu J, Metzler D, et al. (2023) Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306.17563","DOI":"10.18653\/v1\/2024.findings-naacl.97"},{"key":"12053_CR69","unstructured":"Liu Y, Zhou H, Guo Z, Shareghi E, Vuli\u0107 I, Korhonen A, Collier N (2024) Aligning with human judgement: The role of pairwise preference in large language model evaluators. arXiv preprint arXiv:2403.16950"},{"key":"12053_CR70","doi-asserted-by":"crossref","unstructured":"Liusie A, Manakul P, Gales MJ (2023) Llm comparative assessment: Zero-shot nlg evaluation through pairwise comparisons using large language models. arXiv preprint arXiv:2307.07889","DOI":"10.18653\/v1\/2024.eacl-long.8"},{"key":"12053_CR71","unstructured":"Sun J, Xu C, Tang L, Wang S, Lin C, Gong Y, Ni LM, Shum H-Y, Guo J (2023) Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697"},{"key":"12053_CR72","first-page":"8634","volume":"36","author":"N Shinn","year":"2023","unstructured":"Shinn N, Cassano F, Gopinath A, Narasimhan K, Yao S (2023) Reflexion: language agents with verbal reinforcement learning. Adv Neural Inf Process Syst 36:8634\u20138652","journal-title":"Adv Neural Inf Process Syst"},{"issue":"3\/4","key":"12053_CR73","doi-asserted-by":"publisher","first-page":"324","DOI":"10.2307\/2334029","volume":"39","author":"RA Bradley","year":"1952","unstructured":"Bradley RA, Terry ME (1952) Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39(3\/4):324\u2013345","journal-title":"Biometrika"},{"key":"12053_CR74","volume-title":"Individual Choice Behavior","author":"RD Luce","year":"1959","unstructured":"Luce RD et al (1959) Individual Choice Behavior, vol 4. Wiley, New York"},{"key":"12053_CR75","unstructured":"Negahban S, Oh S, Shah D (2012) Iterative ranking from pair-wise comparisons. Adv Neural Inf Process Syst 25"},{"key":"12053_CR76","doi-asserted-by":"crossref","unstructured":"Pokharel G, Farabi S, Fowler PJ, Das S (2025) Street-level ai: Are large language models ready for real-world judgments? In: Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, vol. 8, pp. 2043\u20132054","DOI":"10.1609\/aies.v8i3.36694"},{"key":"12053_CR77","doi-asserted-by":"crossref","unstructured":"Chen L, Li B, Zheng L, Wang H, Meng Z, Shi R, Fei H, Zhou J, Li F, Teng C, et al. (2024) What factors influence llms\u2019 judgments? a case study on question answering. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 17473\u201317485","DOI":"10.63317\/2ecq8riwk5ay"},{"key":"12053_CR78","doi-asserted-by":"crossref","unstructured":"Laskar MTR, Alqahtani S, Bari MS, Rahman M, Khan MAM, Khan H, Jahan I, Bhuiyan A, Tan CW, Parvez MR, et al. (2024) A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13785\u201313816","DOI":"10.18653\/v1\/2024.emnlp-main.764"},{"key":"12053_CR79","doi-asserted-by":"crossref","unstructured":"Zellers R, Holtzman A, Bisk Y, Farhadi A, Choi Y (2019) Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830","DOI":"10.18653\/v1\/P19-1472"},{"key":"12053_CR80","doi-asserted-by":"crossref","unstructured":"Mucciaccia SS, Meireles Paix\u00e3o T, Wall Mutz F, Santos Badue C, Souza A, Oliveira-Santos T (2025) Automatic multiple-choice question generation and evaluation systems based on LLM: A study case with university resolutions. In: Proceedings of the 31st International Conference on Computational Linguistics. Association for Computational Linguistics, Abu Dhabi, UAE. https:\/\/aclanthology.org\/2025.coling-main.154\/","DOI":"10.5753\/jbcs.2025.5801"},{"key":"12053_CR81","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.1010","volume":"8","author":"F Maheen","year":"2022","unstructured":"Maheen F, Asif M, Ahmad H, Ahmad S, Alturise F, Asiry O, Ghadi YY (2022) Automatic computer science domain multiple-choice questions generation based on informative sentences. PeerJ Comput Sci 8:1010. https:\/\/doi.org\/10.7717\/peerj-cs.1010","journal-title":"PeerJ Comput Sci"},{"issue":"8","key":"12053_CR82","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0290691","volume":"18","author":"BHH Cheung","year":"2023","unstructured":"Cheung BHH, Lau GKK, Wong GTC, Lee EYP, Kulkarni D, Seow CS, Wong R, Co M-H (2023) Chatgpt versus human in generating medical graduate exam multiple choice questions-a multinational prospective study (Hong Kong SAR, Singapore, Ireland, and the United Kingdom). PLoS One 18(8):0290691. https:\/\/doi.org\/10.1371\/journal.pone.0290691","journal-title":"PLoS One"},{"key":"12053_CR83","unstructured":"Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Vaughan A, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783"},{"key":"12053_CR84","unstructured":"Team G, Riviere M, Pathak S, Sessa PG, Hardin C, Bhupatiraju S, Hussenot L, Mesnard T, Shahriari B, Ram\u00e9 A, et al. (2024) Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-12053-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-026-12053-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-12053-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T22:27:23Z","timestamp":1781648843000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-026-12053-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,21]]},"references-count":84,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,5]]}},"alternative-id":["12053"],"URL":"https:\/\/doi.org\/10.1007\/s00521-026-12053-8","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,21]]},"assertion":[{"value":"16 December 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 February 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 April 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest related to this work.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"303"}}