{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T04:55:53Z","timestamp":1782363353912,"version":"3.54.5"},"reference-count":28,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2025,7,31]],"date-time":"2025-07-31T00:00:00Z","timestamp":1753920000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The evaluation of large language models (LLMs) increasingly relies on other LLMs acting as automated judges. While this approach offers scalability and efficiency, it raises serious concerns regarding evaluator reliability, positional bias, and ranking stability. This paper presents a scalable framework for diagnosing positional bias and instability in LLM-based evaluation by using controlled pairwise comparisons judged by multiple independent language models. The system supports mirrored comparisons with reversed response order, prompt injection, and surface-level perturbations (e.g., paraphrasing, lexical noise), enabling fine-grained analysis of evaluator consistency and verdict robustness. Over 3600 pairwise comparisons were conducted across five instruction-tuned open-weight models using ten open-ended prompts. The top-performing model (gemma:7b-instruct) achieved a 66.5% win rate. Evaluator agreement was uniformly high, with 100% consistency across judges, yet 48.4% of verdicts reversed under mirrored response order, indicating strong positional bias. Kendall\u2019s Tau analysis further showed that local model rankings varied substantially across prompts, suggesting that semantic context influences evaluator judgment. All evaluation traces were stored in a graph database (Neo4j), enabling structured querying and longitudinal analysis. The proposed framework provides not only a diagnostic lens for benchmarking models but also a blueprint for fairer and more interpretable LLM-based evaluation. These findings underscore the need for structure-aware, perturbation-resilient evaluation pipelines when benchmarking LLMs. The proposed framework offers a reproducible path for diagnosing evaluator bias and ranking instability in open-ended language tasks. Future work will apply this methodology to educational assessment tasks, using rubric-based scoring and graph-based traceability to evaluate student responses in technical domains.<\/jats:p>","DOI":"10.3390\/info16080652","type":"journal-article","created":{"date-parts":[[2025,8,5]],"date-time":"2025-08-05T08:46:55Z","timestamp":1754383615000},"page":"652","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Diagnosing Bias and Instability in LLM Evaluation: A Scalable Pairwise Meta-Evaluator"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1849-3072","authenticated-orcid":false,"given":"Catalin","family":"Anghel","sequence":"first","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2537-6713","authenticated-orcid":false,"given":"Andreea Alexandra","family":"Anghel","sequence":"additional","affiliation":[{"name":"Faculty of Automation, Computer Science, Electrical and Electronic Engineering, \u201cDun\u0103rea de Jos\u201d University of Galati, 800008 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1155-5274","authenticated-orcid":false,"given":"Emilia","family":"Pecheanu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0935-4713","authenticated-orcid":false,"given":"Adina","family":"Cocu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1317-6676","authenticated-orcid":false,"given":"Adrian","family":"Istrate","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5595-1135","authenticated-orcid":false,"given":"Constantin Adrian","family":"Andrei","sequence":"additional","affiliation":[{"name":"\u201cFoisor\u201d Clinical Hospital of Orthopaedics, Traumatology and Osteoarticular TB, 021382 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,7,31]]},"reference":[{"key":"ref_1","unstructured":"Google AI (2025, July 14). Gemma Formatting and System Instructions. Available online: https:\/\/ai.google.dev\/gemma\/docs\/formatting."},{"key":"ref_2","unstructured":"Mistral AI (2025, July 14). Latest Updates from Mistral AI. Available online: https:\/\/mistral.ai\/news\/."},{"key":"ref_3","unstructured":"Cognitive Computations (2025, July 14). Dolphin-2.8-Mistral-7B. Available online: https:\/\/huggingface.co\/cognitivecomputations\/dolphin-2.8-mistral-7b-v02."},{"key":"ref_4","unstructured":"Hugging Face (2025, July 14). Zephyr-7B-Beta Model Card. Available online: https:\/\/huggingface.co\/HuggingFaceH4\/zephyr-7b-beta."},{"key":"ref_5","unstructured":"DeepSeek AI (2025, July 14). DeepSeek-VL: Vision\u2013Language Models by DeepSeek. Available online: https:\/\/github.com\/deepseek-ai\/DeepSeek-VL."},{"key":"ref_6","unstructured":"Bernard, R., Raza, S., Das, S., and Murugan, R. (2024). EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. arXiv, arXiv:2501.00257."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., and Kedzie, C. (2024, January 11\u201316). LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Bangkok, Thailand.","DOI":"10.18653\/v1\/2024.acl-long.745"},{"key":"ref_8","unstructured":"Bhat, V. (2025, July 14). RubricEval: A Scalable Human\u2013LLM Evaluation Framework for Open-Ended Tasks. Stanford CS224N Final Report, Available online: https:\/\/web.stanford.edu\/class\/cs224n\/final-reports\/256846781.pdf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s10462-024-10824-0","article-title":"A Survey of Safety and Trustworthiness of Large Language Models Through the Lens of Verification and Validation","volume":"57","author":"Huang","year":"2024","journal-title":"Artif. Intell. Rev."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Huang, X., Ruan, W., Huang, W., Jin, G., and Dong, Y. (2024). A Survey of Safety and Trustworthiness of Large Language Models Through the Lens of Verification and Validation. arXiv.","DOI":"10.1007\/s10462-024-10824-0"},{"key":"ref_11","unstructured":"Chan, C., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. (2024, January 7\u201311). ChatEval: Towards Better LLM-Based Evaluators Through Multi-Agent Debate. Proceedings of the 12th International Conference on Learning Representations (ICLR 2024), Vienna Austria. Available online: https:\/\/openreview.net\/forum?id=FQepisCUWu."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Jin, Y., Zhao, Q., Wang, Y., Chen, H., Zhu, K., Xiao, Y., and Wang, J. (2024, January 6\u201310). AgentReview: Exploring Peer Review Dynamics with LLM Agents. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), Singapore.","DOI":"10.18653\/v1\/2024.emnlp-main.70"},{"key":"ref_13","unstructured":"Li, T., Chiang, W., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J.E., and Stoica, I. (2025, July 14). From Live Data to High-Quality Benchmarks: The ArenaHard Pipeline. LMSYS Blog, 19 April 2024. Available online: https:\/\/lmsys.org\/blog\/2024-04-19-arena-hard."},{"key":"ref_14","unstructured":"Scherrer, N., Shi, C., Feder, A., and Blei, D. (2023, January 10\u201316). Evaluating the Moral Beliefs Encoded in LLMs. Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA. Available online: https:\/\/openreview.net\/forum?id=O06z2G18me."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Feng, Z., Zhang, Y., Li, H., Wu, B., Liao, J., Liu, W., Lang, J., Feng, Y., Wu, J., and Liu, Z. (2025, January 16\u201321). TEaR: Improving LLM-Based Machine Translation with Systematic Self-Refinement. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2025, Mexico City, Mexico.","DOI":"10.18653\/v1\/2025.findings-naacl.218"},{"key":"ref_16","unstructured":"Shinn, N., Cassano, F., Gopinath, A., Naranghan, K., and Yao, S. (2023, January 10\u201316). Reflexion: Language Agents with Verbal Reinforcement Learning. Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, LA, USA. Available online: https:\/\/papers.nips.cc\/paper_files\/paper\/2023\/file\/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf."},{"key":"ref_17","unstructured":"Panickssery, A., Bowman, S.R., and Feng, S. (2024, January 9\u201315). LLM Evaluators Recognize and Favor Their Own Generations. Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), Vancouver, BC, Canada. Available online: https:\/\/papers.nips.cc\/paper_files\/paper\/2024\/hash\/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html."},{"key":"ref_18","unstructured":"Li, G., Hammoud, H.A.A.K., Itani, H., Khizbullin, D., and Ghanem, B. (2023). CAMEL: Communicative Agents for \u2018Mind\u2019 Exploration of Large Language Model Society. arXiv, arXiv:2303.17760."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B., Fu, J., and Shi, Y. (2023). AutoAgents: A Framework for Automatic Agent Generation. arXiv.","DOI":"10.24963\/ijcai.2024\/3"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kim, A., Kim, K., and Yoon, S. (2024). DEBATE: Devil\u2019s Advocate-Based Assessment and Text Evaluation. Findings of the Association for Computational Linguistics, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2024.findings-acl.112"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Perin, G., Chen, X., Liu, S., Kailkhura, B., Wang, Z., and Gallagher, B. (2024). RankMean: Module-Level Importance Score for Merging Fine-Tuned LLM Models. Findings of the Association for Computational Linguistics, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2024.findings-acl.104"},{"key":"ref_22","unstructured":"Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., and Mordatch, I. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv, arXiv:2305.14325."},{"key":"ref_23","unstructured":"Ni, H. (2025, July 14). Extracting Insights from Unstructured Data with LLMs & Neo4j. Medium, Available online: https:\/\/watchsound.medium.com\/extracting-insights-from-unstructured-data-with-llms-neo4j-914b1f193c64."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Rebmann, A., Schmidt, F.D., Glava\u0161, G., and van der Aa, H. (2025). On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks. arXiv.","DOI":"10.1007\/s44311-025-00019-3"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1093\/biomet\/30.1-2.81","article-title":"Kendall. A New Measure of Rank Correlation","volume":"30","author":"Maurice","year":"1938","journal-title":"Biometrika"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Susnea, I., Pecheanu, E., Cocu, A., Istrate, A., Anghel, C., and Iacobescu, P. (2025). Non-Intrusive Monitoring and Detection of Mobility Loss in Older Adults Using Binary Sensors. Sensors, 25.","DOI":"10.20944\/preprints202503.1358.v1"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Iacobescu, P., Marina, V., Anghel, C., and Anghele, A.-D. (2024). Evaluating Binary Classifiers for Cardiovascular Disease Prediction: Enhancing Early Diagnostic Capabilities. J. Cardiovasc. Dev. Dis., 11.","DOI":"10.3390\/jcdd11120396"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Anghele, A.-D., Marina, V., Dragomir, L., Moscu, C.A., Anghele, M., and Anghel, C. (2024). Predicting Deep Venous Thrombosis Using Artificial Intelligence: A Clinical Data Approach. Bioengineering, 11.","DOI":"10.3390\/bioengineering11111067"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/8\/652\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:19:46Z","timestamp":1760033986000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/8\/652"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,31]]},"references-count":28,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2025,8]]}},"alternative-id":["info16080652"],"URL":"https:\/\/doi.org\/10.3390\/info16080652","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,31]]}}}