{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T16:52:13Z","timestamp":1780505533232,"version":"3.54.1"},"reference-count":45,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T00:00:00Z","timestamp":1754006400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>(1) Background and objectives: Large language models (LLMs) such as GPT, Mistral, and LLaMA exhibit strong capabilities in text generation, yet assessing the quality of their reasoning\u2014particularly in open-ended and argumentative contexts\u2014remains a persistent challenge. This study introduces Dialectical Agent, an internally developed modular framework designed to evaluate reasoning through a structured three-stage process: opinion, counterargument, and synthesis. The framework enables transparent and comparative analysis of how different LLMs handle dialectical reasoning. (2) Methods: Each stage is executed by a single model, and final syntheses are scored via two independent LLM evaluators (LLaMA 3.1 and GPT-4o) based on a rubric with four dimensions: clarity, coherence, originality, and dialecticality. In parallel, a rule-based semantic analyzer detects rhetorical anomalies and ethical values. All outputs and metadata are stored in a Neo4j graph database for structured exploration. (3) Results: The system was applied to four open-weight models (Gemma 7B, Mistral 7B, Dolphin-Mistral, Zephyr 7B) across ten open-ended prompts on ethical, political, and technological topics. The results show consistent stylistic and semantic variation across models, with moderate inter-rater agreement. Semantic diagnostics revealed differences in value expression and rhetorical flaws not captured by rubric scores. (4) Originality: The framework is, to our knowledge, the first to integrate multi-stage reasoning, rubric-based and semantic evaluation, and graph-based storage into a single system. It enables replicable, interpretable, and multidimensional assessment of generative reasoning\u2014supporting researchers, developers, and educators working with LLMs in high-stakes contexts.<\/jats:p>","DOI":"10.3390\/informatics12030076","type":"journal-article","created":{"date-parts":[[2025,8,6]],"date-time":"2025-08-06T07:45:11Z","timestamp":1754466311000},"page":"76","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Multi-Model Dialectical Evaluation of LLM Reasoning Chains: A Structured Framework with Dual Scoring Agents"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1849-3072","authenticated-orcid":false,"given":"Catalin","family":"Anghel","sequence":"first","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2537-6713","authenticated-orcid":false,"given":"Andreea Alexandra","family":"Anghel","sequence":"additional","affiliation":[{"name":"Faculty of Automation, Computer Science, Electrical and Electronic Engineering, \u201cDun\u0103rea de Jos\u201d University of Galati, 800008 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1155-5274","authenticated-orcid":false,"given":"Emilia","family":"Pecheanu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4795-4699","authenticated-orcid":false,"given":"Ioan","family":"Susnea","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0935-4713","authenticated-orcid":false,"given":"Adina","family":"Cocu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1317-6676","authenticated-orcid":false,"given":"Adrian","family":"Istrate","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, \u201cDun\u0103rea de Jos\u201d University of Galati, \u0218tiin\u021bei St. 2, 800146 Galati, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,1]]},"reference":[{"key":"ref_1","unstructured":"Kishore, P., Salim, R., Todd, W., and Zhu, W.-J. (2002, January 6\u201312). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th ACL, Philadelphia, PA, USA. Available online: https:\/\/www.aclweb.org\/anthology\/P02-1040."},{"key":"ref_2","unstructured":"Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries, ACL. Available online: https:\/\/www.aclweb.org\/anthology\/W04-1013."},{"key":"ref_3","unstructured":"Pranav, R., Zhang, J., Konstantin, L., and Percy, L. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text, EMNLP. Available online: https:\/\/arxiv.org\/abs\/1606.05250."},{"key":"ref_4","unstructured":"Jason, W., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q.V., and Zhou, D. (2022). Chain of Thought Prompting Elicits Reasoning in Large Language Models. arXiv, Available online: https:\/\/arxiv.org\/abs\/2201.11903."},{"key":"ref_5","unstructured":"Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., and Yang, Y. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv, Available online: https:\/\/arxiv.org\/abs\/2303.17651."},{"key":"ref_6","unstructured":"Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv, Available online: https:\/\/arxiv.org\/abs\/2303.11366."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Park, J.S., Joseph, O.B., Cai, C.J., Morris, M.R., Liang, P., and Bernstein, M.S. (2023). Generative Agents: Interactive Simulacra of Human Behavior. arXiv, Available online: https:\/\/arxiv.org\/abs\/2304.03442.","DOI":"10.1145\/3586183.3606763"},{"key":"ref_8","unstructured":"Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., and Mordatch, I. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv, Available online: https:\/\/arxiv.org\/abs\/2305.14325."},{"key":"ref_9","unstructured":"Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., and Narasimhan, K. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv, Available online: https:\/\/arxiv.org\/abs\/2305.10601."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Li, Y., Yang, C., and Ettinger, A. (2024). When Hindsight is Not 20\/20: Testing Limits on Reflective Thinking in Large Language Models. arXiv, Available online: https:\/\/arxiv.org\/abs\/2404.09129.","DOI":"10.18653\/v1\/2024.findings-naacl.237"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ioan, S., Pecheanu, E., Cocu, A., Istrate, A., Anghel, C., and Iacobescu, P. (2025). Non-Intrusive Monitoring and Detection of Mobility Loss in Older Adults Using Binary Sensors. Sensors, 25.","DOI":"10.3390\/s25092755"},{"key":"ref_12","unstructured":"Zhu, K., Zhao, Q., Chen, H., Wang, J., and Xie, X. (2023). PromptBench: A Unified Library for Evaluation of Large Language Models. arXiv, Available online: https:\/\/arxiv.org\/abs\/2312.07910."},{"key":"ref_13","unstructured":"Bhat, V. (2023). RubricEval: A Scalable Human-LLM Evaluation Framework for Open-Ended Tasks, Stanford University. Available online: https:\/\/web.stanford.edu\/class\/cs224n\/final-reports\/256846781.pdf."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., and Kedzie, C. (2024, January 11\u201316). LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Bangkok, Thailand. Available online: https:\/\/aclanthology.org\/2024.acl-long.745.","DOI":"10.18653\/v1\/2024.acl-long.745"},{"key":"ref_15","first-page":"11","article-title":"An Overview of the Schwartz Theory of Basic Values","volume":"2","author":"Schwartz","year":"2012","journal-title":"Online Read. Psychol. Cult."},{"key":"ref_16","unstructured":"Zheng, C., Liu, Z., Xie, E., Li, Z., and Li, Y. (2023). Progressive-Hint Prompting Improves Reasoning in Large Language Models. arXiv, Available online: https:\/\/arxiv.org\/abs\/2304.09797."},{"key":"ref_17","unstructured":"Webb, T., Mondal, S.S., and Momennejad, I. (2023). Improving Planning with Large Language Models: A Modular Agentic Architecture. arXiv, Available online: https:\/\/arxiv.org\/abs\/2310.00194."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Gu, Y., Tafjord, O., Kuehl, B., Haddad, D., Dodge, J., and Hajishirzi, H. (2024). OLMES: A Standard for Language Model Evaluations. arXiv, Available online: https:\/\/arxiv.org\/abs\/2406.08446.","DOI":"10.18653\/v1\/2025.findings-naacl.282"},{"key":"ref_19","unstructured":"(2025, May 25). Structured Outputs: Making LLMs Reliable for Document Processing. Generative AI Newsroom. Available online: https:\/\/generative-ai-newsroom.com\/structured-outputs-making-llms-reliable-for-document-processing-c3b6b2baed36."},{"key":"ref_20","unstructured":"Kuchnik, M., Smith, V., and Amvrosiadis, G. (2022). Validating Large Language Models with ReLM. arXiv, Available online: https:\/\/arxiv.org\/abs\/2211.15458."},{"key":"ref_21","unstructured":"Scherrer, N., Shi, C., Feder, A., and Blei, D. (2023, January 10\u201316). Evaluating the Moral Beliefs Encoded in LLMs. Proceedings of the NeurIPS 2023, New Orleans, Louisiana. Available online: https:\/\/openreview.net\/forum?id=O06z2G18me."},{"key":"ref_22","unstructured":"Sidana, N. (2025, May 25). Running Models with Ollama Step-by-Step. Medium. Available online: https:\/\/medium.com\/@nsidana123\/running-models-with-ollama-step-by-step-b3bdbfd91e8e."},{"key":"ref_23","unstructured":"(2025, May 25). Prompt Engineering Guide. Gemma 7B. Available online: https:\/\/www.promptingguide.ai\/models\/gemma."},{"key":"ref_24","unstructured":"Google AI for Developers (2025, May 25). Gemma Formatting and System Instructions. Available online: https:\/\/ai.google.dev\/gemma\/docs\/formatting."},{"key":"ref_25","unstructured":"Hugging Face (2025, May 25). Cognitivecomputations\/Dolphin-2.8-Mistral-7b-v02. Available online: https:\/\/huggingface.co\/cognitivecomputations\/dolphin-2.8-mistral-7b-v02."},{"key":"ref_26","unstructured":"Hugging Face (2025, May 25). HuggingFaceH4\/zephyr-7b-beta. Available online: https:\/\/huggingface.co\/HuggingFaceH4\/zephyr-7b-beta."},{"key":"ref_27","unstructured":"Meta AI (2025, May 25). Introducing Llama 3.1: Our Most Capable Models to Date. Available online: https:\/\/ai.meta.com\/blog\/meta-llama-3-1\/."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Asprino, L., De Giorgis, S., Gangemi, A., Bulla, L., Marinucci, L., and Mongiov\u00ec, M. (2022, January 27). Uncovering Values: Detecting Latent Moral Content from Natural Language with Explainable and Non-Trained Methods. Proceedings of the Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Dublin, Ireland. Available online: https:\/\/aclanthology.org\/2022.deelio-1.4.pdf.","DOI":"10.18653\/v1\/2022.deelio-1.4"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B.F., Fu, J., and Shi, Y. (2023). AutoAgents: A Framework for Automatic Agent Generation. arXiv, Available online: https:\/\/arxiv.org\/abs\/2309.17288.","DOI":"10.24963\/ijcai.2024\/3"},{"key":"ref_30","unstructured":"Liu, X., Chen, J., Li, C., Song, X., and Wang, Y. (2023). CAMEL: Communicative Agents for Mind Exploration of Large-Scale Language Model Society. arXiv, Available online: https:\/\/arxiv.org\/abs\/2303.17760."},{"key":"ref_31","unstructured":"Ni, H. (2025, May 25). Extracting Insights from Unstructured Data with LLMs & Neo4j. Medium, 15 January 2025. Available online: https:\/\/watchsound.medium.com\/extracting-insights-from-unstructured-data-with-llms-neo4j-914b1f193c64."},{"key":"ref_32","unstructured":"Sahoo, P., Singh, A.K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv, Available online: https:\/\/arxiv.org\/abs\/2402.07927."},{"key":"ref_33","unstructured":"Lum, K., Anthis, J.R., Robinson, K., Nagpal, C., and Alexander, D.A. (2024). Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation. arXiv, Available online: https:\/\/arxiv.org\/abs\/2402.12649."},{"key":"ref_34","unstructured":"Microsoft Learn (2025, May 25). Evaluation and Monitoring Metrics for Generative AI. Available online: https:\/\/learn.microsoft.com\/en-us\/azure\/ai-foundry\/concepts\/evaluation-metrics-built-in."},{"key":"ref_35","unstructured":"Confident AI (2025, May 25). G-Eval: The Definitive Guide. Available online: https:\/\/www.confident-ai.com\/blog\/g-eval-the-definitive-guide."},{"key":"ref_36","unstructured":"Padmakumar, V., Yueh-Han, C., Pan, J., Chen, V., and He, H. (2025). Beyond Memorization: Mapping the Originality-Quality Frontier of Language Models. arXiv, Available online: https:\/\/arxiv.org\/abs\/2504.09389."},{"key":"ref_37","unstructured":"Cohn, A.G., and Hernandez-Orallo, J. (2023). Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs. arXiv, Available online: https:\/\/arxiv.org\/abs\/2304.11164."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Pathak, A., Gandhi, R., Uttam, V., Nakka, Y., Jindal, A.R., Ghosh, P., Ramamoorthy, A., Verma, S., and Mittal, A. (2025). Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics. arXiv, Available online: https:\/\/arxiv.org\/abs\/2503.23989.","DOI":"10.1145\/3702652.3744220"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Rebmann, A., Schmidt, F.D., Glava\u0161, G., and van der Aa, H. (2025). On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks. arXiv, Available online: https:\/\/arxiv.org\/abs\/2504.21074.","DOI":"10.1007\/s44311-025-00019-3"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s10462-024-10824-0","article-title":"A Survey of Safety and Trustworthiness of Large Language Models Through the Lens of Verification and Validation","volume":"57","author":"Huang","year":"2024","journal-title":"Artif. Intell. Rev."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1162\/coli_a_00418","article-title":"The (Un) Suitability of Automatic Evaluation Metrics for Text Simplification","volume":"47","author":"Scarton","year":"2021","journal-title":"Comput. Linguist."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Iacobescu, P., Marina, V., Anghel, C., and Anghele, A.-D. (2024). Evaluating Binary Classifiers for Cardiovascular Disease Prediction: Enhancing Early Diagnostic Capabilities. J. Cardiovasc. Dev. Dis., 11.","DOI":"10.3390\/jcdd11120396"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Anghele, A.-D., Marina, V., Dragomir, L., Moscu, C.A., Anghele, M., and Anghel, C. (2024). Predicting Deep Venous Thrombosis Using Artificial Intelligence: A Clinical Data Approach. Bioengineering, 11.","DOI":"10.3390\/bioengineering11111067"},{"key":"ref_44","first-page":"77","article-title":"A Framework of Rhetorical Moves Designed to Scaffold the Development of Research Proposals","volume":"18","author":"Reddy","year":"2023","journal-title":"Int. J. Dr. Stud."},{"key":"ref_45","unstructured":"Vengal, T. (2025, May 25). LLMs: A Review of Their Capabilities, Limitations and Evaluation. LinkedIn. Available online: https:\/\/www.linkedin.com\/pulse\/llms-review-capabilities-limitations-evaluation-thomas-vengal."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/3\/76\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:20:37Z","timestamp":1760034037000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/3\/76"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,1]]},"references-count":45,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["informatics12030076"],"URL":"https:\/\/doi.org\/10.3390\/informatics12030076","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,1]]}}}