{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T19:32:47Z","timestamp":1793129567486,"version":"4.5.0"},"reference-count":47,"publisher":"Wiley","issue":"8","license":[{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"},{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["WUT: 104972025RSCbs0184"],"award-info":[{"award-number":["WUT: 104972025RSCbs0184"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Quality &amp; Reliability Eng"],"published-print":{"date-parts":[[2026,12]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>A subjective\u2010objective information \u2010 based decision \u2010 making fusion framework is proposed to assess the generative performance of large language models (LLMs) by combining objective evidence and expert\u2010based subjective judgement. The Analytic Hierarchy Process (AHP) is first used to quantify the objective weights of sub\u2010indicators and compute the objective performance based on the standardised benchmarks. The Best\u2010Worst Method is then used to quantify the subjective weights of sub\u2010indicators based on domain experts. After that, an iterative solution method is introduced to align the divergence domain expert opinion, update the subjective weights, and compute the subjective performance of LLMs. Finally, the performance of LLM can be quantified by the weighted sum of subjective and objective performance. Comparative and sensitivity analyses are carried out to validate the feasibility and superiority of the proposed method. Overall, the proposed method can deal with the potential subjective uncertainty in expert opinion and supporting final decision\u2010making.<\/jats:p>","DOI":"10.1002\/qre.70358","type":"journal-article","created":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T17:23:50Z","timestamp":1786987430000},"page":"4548-4560","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["An Information Fusion Based Decision\u2010Making Framework for Generative Performance Assessment of Large Language Models"],"prefix":"10.1002","volume":"42","author":[{"given":"Yongqing","family":"Huang","sequence":"first","affiliation":[{"name":"National College for Excellent Engineers Beihang University  Beijing P. R. China"},{"name":"Information Science Academy China Electronics Technology Group Corporation  Beijing P. R. China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Jiang","sequence":"additional","affiliation":[{"name":"Information Science Academy China Electronics Technology Group Corporation  Beijing P. R. China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2797-2304","authenticated-orcid":false,"given":"Yunling","family":"Ye","sequence":"additional","affiliation":[{"name":"School of Naval Architecture Ocean and Energy Power Engineering Wuhan University of Technology  Wuhan P. R. China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,8,17]]},"reference":[{"key":"e_1_2_7_2_1","doi-asserted-by":"crossref","unstructured":"C.Qin A.Zhang Z.Zhang J.Chen M.Yasunaga andD.Yang Is ChatGPT a General\u2010Purpose Natural Language Processing Task Solver?Singapore:Association for Computational Linguistics 2023 1339\u20131384.","DOI":"10.18653\/v1\/2023.emnlp-main.85"},{"key":"e_1_2_7_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.energy.2023.128442"},{"key":"e_1_2_7_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.marstruc.2026.104042"},{"key":"e_1_2_7_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.oceaneng.2026.125389"},{"key":"e_1_2_7_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.apor.2025.104796"},{"key":"e_1_2_7_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2025.111113"},{"key":"e_1_2_7_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.marstruc.2026.104152"},{"key":"e_1_2_7_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.oceaneng.2026.125854"},{"key":"e_1_2_7_10_1","doi-asserted-by":"crossref","unstructured":"Y.Bang S.Cahyawijaya N.Lee et\u00a0al. \u201cMultitask Multilingual Multimodal Evaluation of ChatGPT on Reasoning Hallucination and Interactivity \u201d Nusa Dua Bali: Association for Computational Linguistics (2023):675\u2013718.","DOI":"10.18653\/v1\/2023.ijcnlp-main.45"},{"key":"e_1_2_7_11_1","doi-asserted-by":"crossref","unstructured":"Y.\u2010T.LinandY.\u2010N.Chen \u201cLlm\u2010eval: Unified Multi\u2010Dimensional Automatic Evaluation for Open\u2010Domain Conversations With Large Language Models \u201d inProceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023)2023 47\u201358.","DOI":"10.18653\/v1\/2023.nlp4convai-1.5"},{"key":"e_1_2_7_12_1","unstructured":"Y.Wang Z.Yu Z.Zeng et\u00a0al. \u201cPandaLm: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization \u201dInternational Coeference on Learning Representations(2024)."},{"key":"e_1_2_7_13_1","unstructured":"N.Jain K.Saifullah Y.Wen et\u00a0al. \u201cBring Your Own Data! Self\u2010sensitivity Evaluation for Large Language Models \u201dConference on Language Modeling(2024)."},{"key":"e_1_2_7_14_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00502"},{"key":"e_1_2_7_15_1","article-title":"Holistic Evaluation of Language Models","author":"Liang P.","year":"2023","journal-title":"Transaction on Machine Learning Research"},{"key":"e_1_2_7_16_1","first-page":"675","article-title":"A Multitask, Multilingual, Multimodal Evaluation of Chatgpt on Reasoning, Hallucination, and Interactivity","volume":"1","author":"Bang Y.","year":"2023","journal-title":"Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia\u2010Pacific Chapter of the Association for Computational Linguistics"},{"key":"e_1_2_7_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.oceaneng.2025.123353"},{"key":"e_1_2_7_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.oceaneng.2024.117435"},{"key":"e_1_2_7_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2026.112238"},{"key":"e_1_2_7_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2026.112378"},{"key":"e_1_2_7_21_1","unstructured":"D.Hendrycks C.Burns S.Basart et\u00a0al. \u201cMeasuring Massive Multitask Language Understanding \u201d inProceeding of the International Conference on Learning Representations(2021)."},{"key":"e_1_2_7_22_1","doi-asserted-by":"crossref","unstructured":"A. P.Gema J. O. J.Leang andG.Hong \u201cAre We Done with MMLU? \u201d inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies(Volume 1: Long Papers) (2025) 5069\u20135096.","DOI":"10.18653\/v1\/2025.naacl-long.262"},{"key":"e_1_2_7_23_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-3018"},{"key":"e_1_2_7_24_1","unstructured":"D.Rein B. L.Hou A. C.Stickland et\u00a0al. \u201cGPQA: A Graduate\u2010Level Google\u2010Proof Q&A Benchmark \u201dConference on Language Modeling(2024)."},{"key":"e_1_2_7_25_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2749"},{"key":"e_1_2_7_26_1","unstructured":"J.Wei K.Nguyen H. W.Chung et\u00a0al. \u201cMeasuring Short\u2010Form Factuality in Large Language Models \u201d arXiv preprint arXiv:2411.04368 (2024)."},{"key":"e_1_2_7_27_1","doi-asserted-by":"crossref","unstructured":"Y.He S.Li andJ.Liu \u201cChinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models \u201d inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics(2025) 19182\u201319208.","DOI":"10.18653\/v1\/2025.acl-long.941"},{"key":"e_1_2_7_28_1","unstructured":"Mathematical Association of America American Invitational Mathematics Examination (AIME) [Internet]Washington DC:Mathematical Association of America https:\/\/maa.org\/maa\u2010invitational\u2010competitions\/."},{"key":"e_1_2_7_29_1","unstructured":"Jia M. AIME_2024.Hugging Face;2024.https:\/\/huggingface.co\/datasets\/Maxwell\u2010Jia\/AIME_2024."},{"key":"e_1_2_7_30_1","unstructured":"D.Hendrycks C.Burns S.Kadavath et\u00a0al. \u201cMeasuring Mathematical Problem Solving With the MATH Dataset \u201d inThirty\u2010fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track(2021)."},{"key":"e_1_2_7_31_1","unstructured":"H.Lightman V.Kosaraju andY.Burda \u201cLet's Verify Step by Step \u201d inInternational Conference on Learning Representations(2024) 39578\u201339601."},{"key":"e_1_2_7_32_1","unstructured":"Chinese Mathematical Society Mathematical Competition (Chinese Mathematical Society) https:\/\/www.cms.org.cn\/en\/Home\/comp\/comp.html."},{"key":"e_1_2_7_33_1","doi-asserted-by":"crossref","unstructured":"D.Dua Y.Wang P.Dasigi G.Stanovsky S.Singh andM.Gardner \u201cDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning over Paragraphs \u201d inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics(Human Language Technologies 2019) 2368\u20132378.","DOI":"10.18653\/v1\/N19-1246"},{"key":"e_1_2_7_34_1","doi-asserted-by":"crossref","unstructured":"S.Krishna K.Krishna A.Mohananey et\u00a0al. \u201cFact Fetch and Reason: A Unified Evaluation of Retrieval\u2010Augmented Generation \u201d inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Huamn Language Technologies(Volume 1: Long Papers) (2025) 4745\u20134759.","DOI":"10.18653\/v1\/2025.naacl-long.243"},{"key":"e_1_2_7_35_1","doi-asserted-by":"crossref","unstructured":"L.Xu H.Hu X.Zhang L.Li C.Cao andY.Li \u201cCLUE: A Chinese Language Understanding Evaluation Benchmark \u201d inProceedings of the 28th International Conference on Computational Linguistics(2020) 4762\u20134772.","DOI":"10.18653\/v1\/2020.coling-main.419"},{"key":"e_1_2_7_36_1","unstructured":"J.Zhou T.Lu S.Mishra et\u00a0al. \u201cInstruction\u2010Following Evaluation for Large Language Models \u201d aiXiv preprint arXiv:2311.07911 (2023)."},{"key":"e_1_2_7_37_1","unstructured":"N.Jain K.Han A.Gu et\u00a0al. \u201cLiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code \u201dInternational Conference on Learning Representations(2025)."},{"key":"e_1_2_7_38_1","unstructured":"Codeforces.Programming Competitions and Contests Programming Community.Codeforceshttps:\/\/codeforces.com\/."},{"key":"e_1_2_7_39_1","unstructured":"Y.Dubois B.Galambosi P.Liang andT. B.Hashimoto \u201cLength\u2010Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators \u201dFirst Conference on Language Modeling(2024)."},{"key":"e_1_2_7_40_1","unstructured":"T.Li W. L.Chiang E.Frick et\u00a0al. \u201cFrom Crowdsourced Data to High\u2010quality Benchmarks: Arena\u2010Hard and BenchBuilder Pipeline \u201dPreceedings of the 42nd International Conference on Machine Learning PMLR267(2025):34209\u201334231."},{"key":"e_1_2_7_41_1","unstructured":"B.Vidgen N.Scherrer H. R.Kirk et\u00a0al. \u201cSimpleSafetyTests: A Test Suite for Identifying Critical Safety Risks in Large Language Models \u201d arXiv preprint November 14 2023 arXiv:2311.08370."},{"key":"e_1_2_7_42_1","doi-asserted-by":"crossref","unstructured":"A.Parrish A.Chen N.Nangia V.Padmakumar J.Phang andJ.Thompson Findings of the Association for Computational Linguistics: ACL 2022(2022) 2086\u20132105.","DOI":"10.18653\/v1\/2022.findings-acl.165"},{"key":"e_1_2_7_43_1","doi-asserted-by":"crossref","unstructured":"P. R\u00f6ttger H.Kirk B.Vidgen G.Attanasio F.Bianchi andD.Hovy \u201cXSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models \u201d inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics(Human Language Technologies 2024) 5377\u20135400.","DOI":"10.18653\/v1\/2024.naacl-long.301"},{"key":"e_1_2_7_44_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-eacl.61"},{"key":"e_1_2_7_45_1","first-page":"35181","article-title":"HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal","volume":"235","author":"Mazeika M.","year":"2024","journal-title":"Proceedings of Machine Learning Research"},{"key":"e_1_2_7_46_1","unstructured":"Aider.Aider LLM Leaderboards: Aider Polyglot coding leaderboard."},{"key":"e_1_2_7_47_1","unstructured":"SWE\u2010bench Team.SWE\u2010Bench Verified: A Human\u2010Validated Subset of SWE\u2010bench."},{"key":"e_1_2_7_48_1","unstructured":"C. E.Jimenez J.Yang andA.Wettig \u201cSWE\u2010bench: Can Language Models Resolve Real\u2010World GitHub Issues? \u201d inThe Twelfth International Conference on Learning Representations(2024)."}],"container-title":["Quality and Reliability Engineering International"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/qre.70358","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/qre.70358","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/qre.70358","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T18:45:09Z","timestamp":1793126709000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/qre.70358"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,17]]},"references-count":47,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2026,12]]}},"alternative-id":["10.1002\/qre.70358"],"URL":"https:\/\/doi.org\/10.1002\/qre.70358","archive":["Portico"],"relation":{},"ISSN":["0748-8017","1099-1638"],"issn-type":[{"value":"0748-8017","type":"print"},{"value":"1099-1638","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,17]]},"assertion":[{"value":"2026-02-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-05","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}