{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T17:03:39Z","timestamp":1778864619675,"version":"3.51.4"},"reference-count":46,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2024,11,11]],"date-time":"2024-11-11T00:00:00Z","timestamp":1731283200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>This study investigates the effectiveness of a proposed version of Meta\u2019s LLaMA 3 model in detecting fake claims across bilingual (English and Romanian) datasets, focusing on a multi-class approach beyond traditional binary classifications in order to better mimic real-world scenarios. The research employs a proposed version of the LLaMA 3 model, optimized for identifying nuanced categories such as \u201cMostly True\u201d and \u201cMostly False\u201d, and compares its performance against leading large language models (LLMs) including Open AI\u2019s ChatGPT versions, Google\u2019s Gemini, and similar LLaMA models. The analysis reveals that the proposed LLaMA 3 model consistently outperforms its base version and older LLaMA models, particularly in the Romanian dataset, achieving the highest accuracy of 39% and demonstrating superior capabilities in identifying nuanced claims, over all the compared large language models. However, the model\u2019s performance across both languages highlights some challenges, with generally low accuracy and difficulties in handling ambiguous categories by all the LLMs. The study also underscores the impact of language and cultural context on model reliability, noting that even state-of-the-art models like ChatGPT 4.o and Gemini exhibit inconsistencies when applied to Romanian text and more than a binary true\/false approach.<\/jats:p>","DOI":"10.3390\/computers13110292","type":"journal-article","created":{"date-parts":[[2024,11,11]],"date-time":"2024-11-11T08:46:54Z","timestamp":1731314814000},"page":"292","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["LLaMA 3 vs. State-of-the-Art Large Language Models: Performance in Detecting Nuanced Fake News"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1735-6092","authenticated-orcid":false,"given":"Stefan Emil","family":"Repede","sequence":"first","affiliation":[{"name":"Department of Computer Science, Electrical and Electronics Engineering, University of Sibiu, 4 Emil Cioran Street, 550025 Sibiu, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8100-1379","authenticated-orcid":false,"given":"Remus","family":"Brad","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Electrical and Electronics Engineering, University of Sibiu, 4 Emil Cioran Street, 550025 Sibiu, Romania"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,11,11]]},"reference":[{"key":"ref_1","unstructured":"Shearer, E., and Mitchell, A. (2024, August 28). News Use Across Social Media Platforms in 2021. Pew Research Center. Available online: https:\/\/www.pewresearch.org\/journalism\/2021\/01\/12\/news-use-across-social-media-platforms-in-2020\/."},{"key":"ref_2","unstructured":"Lorenz, T. (2024, August 28). Why TikTok Videos on the Israel-Hamas War Have Drawn Billions of Views. The Washington Post, 10 October 2023. Available online: https:\/\/www.washingtonpost.com\/technology\/2023\/10\/10\/tiktok-hamas-israel-war-videos\/."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"114","DOI":"10.53477\/2284-9378-23-10","article-title":"A comparison of artificial intelligence models used for fake news detection","volume":"12","author":"Repede","year":"2023","journal-title":"Bull. \u201dCarol I\u201d Natl. Def. Univ."},{"key":"ref_4","unstructured":"Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., and Ganapathy, R. (2024). The llama 3 herd of models. arXiv."},{"key":"ref_5","first-page":"139","article-title":"Misinformation, disinformation, and fake news: Lessons from an interdisciplinary, systematic literature review","volume":"48","author":"Broda","year":"2024","journal-title":"Ann. Int. Commun. Assoc."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"11765","DOI":"10.1007\/s11042-020-10183-2","article-title":"FakeBERT: Fake news detection in social media with a BERT-based deep learning approach","volume":"80","author":"Kaliyar","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"69","DOI":"10.53477\/2284-9378-23-21","article-title":"Researching disinformation using artificial intelligence techniques: Challenges","volume":"12","author":"Repede","year":"2023","journal-title":"Bull. \u201dCarol I\u201d Natl. Def. Univ."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5557784","DOI":"10.1155\/2021\/5557784","article-title":"Fake detect: A deep learning ensemble model for fake news detection","volume":"2021","author":"Aslam","year":"2021","journal-title":"Complexity"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebr\u00f3n, F., and Sanghai, S. (2023). Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.298"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wang, Y.A., and Chen, Y.N. (2020). What do position embeddings learn? An empirical study of pre-trained language model positional encoding. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.555"},{"key":"ref_11","unstructured":"Silva, E.C.D.M., and Vaz, J.C. (2024). What characteristics define disinformation and fake news?: Review of taxonomies and definitions. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"51009","DOI":"10.1007\/s11042-023-17470-8","article-title":"A comprehensive survey on machine learning approaches for fake news detection","volume":"83","author":"Alghamdi","year":"2024","journal-title":"Multimed. Tools Appl."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Farhoudinia, B., Ozturkcan, S., and Kasap, N. (2023). Fake news in business and management literature: A systematic review of definitions, theories, methods and implications. Aslib J. Inf. Manag., ahead-of-print.","DOI":"10.1108\/AJIM-09-2022-0418"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Vishnupriya, G., Jeriel K, A., RNS, A., Ajay, G., and Giftson J, A. (2024, January 5\u20137). Combating Fake News in the Digital Age: A Review of AI-Based Approaches. Proceedings of the IEEE 2024 IEEE 9th International Conference for Convergence in Technology (I2CT), Pune, India.","DOI":"10.1109\/I2CT61223.2024.10544008"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"76133","DOI":"10.1109\/ACCESS.2024.3406258","article-title":"A Review of Deep Learning Techniques for Multimodal Fake News and Harmful Languages Detection","volume":"12","author":"Ayetiran","year":"2024","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Kumar, S., Malhotra, N., Garg, N., and Shakil, M.A. (2024, January 3\u20134). An In-depth Analysis of Transformer Models for Enhanced Performance in Fake News Detection. Proceedings of the 2024 5th International Conference on Image Processing and Capsule Networks (ICIPCN), Dhulikhel, Nepal.","DOI":"10.1109\/ICIPCN63822.2024.00034"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Saleh, A.O., Karao\u011flan, K.M., and \u00c7akmak, M. (2024, January 21\u201322). A Comprehensive Survey on Automatic Detection of Fake News Using Natural Language Processing: Challenges and Limitations. Proceedings of the 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), Malatya, Turkiye.","DOI":"10.1109\/IDAP64064.2024.10710923"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"110642","DOI":"10.1016\/j.knosys.2023.110642","article-title":"Towards COVID-19 fake news detection using transformer-based models","volume":"274","author":"Alghamdi","year":"2023","journal-title":"Knowl.-Based Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"30367","DOI":"10.1109\/ACCESS.2022.3159651","article-title":"A taxonomy of fake news classification techniques: Survey and implementation aspects","volume":"10","author":"Rohera","year":"2022","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Cui, L., Wang, S., and Lee, D. (2019, January 27\u201330). Same: Sentiment-aware multi-modal embedding for detecting fake news. Proceedings of the 2019 IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining, Vancouver, BC, Canada.","DOI":"10.1145\/3341161.3342894"},{"key":"ref_21","first-page":"22105","article-title":"Bad actor, good advisor: Exploring the role of large language models in fake news detection","volume":"38","author":"Hu","year":"2024","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_22","first-page":"41","article-title":"Artificial intelligence\u2014Friend or foe in fake news campaigns. Economics and Business Review","volume":"9","author":"Lewoniewski","year":"2023","journal-title":"Sciendo"},{"key":"ref_23","unstructured":"Yi, Z., Ouyang, J., Liu, Y., Liao, T., Xu, Z., and Shen, Y. (2024). A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"100619","DOI":"10.1016\/j.simpa.2024.100619","article-title":"LangTest: A comprehensive evaluation library for custom LLM and NLP models","volume":"19","author":"Nazir","year":"2024","journal-title":"Softw. Impacts"},{"key":"ref_25","unstructured":"Tanvir, A.A., Mahir, E.M., Akhter, S., and Huq, M.R. (2019, January 28\u201330). Detecting Fake News using Machine Learning and Deep Learning Algorithms. Proceedings of the 2019 7th International Conference on Smart Computing & Communications (ICSCC), Sarawak, Malaysia."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"12320","DOI":"10.1007\/s11227-021-03743-2","article-title":"Considerations about learning Word2Vec","volume":"77","author":"Buonanno","year":"2021","journal-title":"J. Supercomput."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Liu, Y., and Wu, Y.-F. (2018, January 2\u20137). Early Detection of Fake News on Social Media Through Propagation Path Classification with Recurrent and Convolutional Networks. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11268"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Taherdoost, H., and Madanchian, M. (2023). Artificial Intelligence and Sentiment Analysis: A Review in Competitive Research. Computers, 12.","DOI":"10.3390\/computers12020037"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s41060-021-00302-z","article-title":"Fake news detection based on news content and social contexts: A transformer-based approach","volume":"13","author":"Raza","year":"2022","journal-title":"Int. J. Data Sci. Anal."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Alotaibi, A., and Nadeem, F. (2024). Leveraging Social Media and Deep Learning for Sentiment Analysis for Smart Governance: A Case Study of Public Reactions to Educational Reforms in Saudi Arabia. Computers, 13.","DOI":"10.3390\/computers13110280"},{"key":"ref_32","unstructured":"Zong, M., and Krishnamachari, B. (2022). A survey on GPT-3. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"26839","DOI":"10.1109\/ACCESS.2024.3365742","article-title":"A review on large Language Models: Architectures, applications, taxonomies, open issues and challenges","volume":"12","author":"Raiaan","year":"2024","journal-title":"IEEE Access"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"108072","DOI":"10.1109\/ACCESS.2024.3438353","article-title":"Enhanced Sentiment Intensity Regression Through LoRA Fine-Tuning on Llama 3","volume":"12","author":"Lin","year":"2024","journal-title":"IEEE Access"},{"key":"ref_35","unstructured":"Repede, \u0218.E. (2024, August 20). Dataset Compiled for the Article. Available online: https:\/\/huggingface.co\/datasets\/Phoenyx83\/Politifact-fake-news-6-categories-for-llama3-1."},{"key":"ref_36","unstructured":"Repede, \u0218.E. (2024, August 20). Fine Tuned Version of Meta-Llama-3-8B, Trained on 2 Datasets. Available Online on Hugging Face Hub. Available online: https:\/\/huggingface.co\/Phoenyx83\/Meta-Llama-3-8B-Politifact-fake-news."},{"key":"ref_37","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., and Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv."},{"key":"ref_38","unstructured":"Li, X., Zhang, Y., and Malthouse, E.C. (2023). A preliminary study of chatgpt on news recommendation: Personalization, provider fairness, fake news. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Pang, S., Nol, E., and Heng, K. (2023). ChatGPT-4o for English language teaching and learning: Features, applications, and future prospects. SSRN Electron. J.","DOI":"10.2139\/ssrn.4837988"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Islam, R., and Ahmed, I. (2024, January 10\u201312). Gemini-the most powerful LLM: Myth or Trut. Proceedings of the 5th Information Communication Technologies Conference (ICTC), Nanjing, China.","DOI":"10.1109\/ICTC61510.2024.10602253"},{"key":"ref_41","unstructured":"Repede, \u0218.E. (2024, August 28). Ro and En Datasets. Available Online on Kaggle Datasets. Available online: https:\/\/www.kaggle.com\/datasets\/restem\/en-ro-datasets-for-llm-testing."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Caramancion, K.M. (2023). News Verifiers Showdown: A Comparative Performance Evaluation of ChatGPT 3.5, ChatGPT 4.0, Bing AI, and Bard in News Fact-Checking. arXiv.","DOI":"10.1109\/FNWF58287.2023.10520446"},{"key":"ref_43","unstructured":"Hu, T., and Zhou, X.H. (2024). Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"187","DOI":"10.4103\/picr.PICR_123_17","article-title":"Common pitfalls in statistical analysis: Measures of agreement","volume":"8","author":"Ranganathan","year":"2017","journal-title":"Perspect. Clin. Res."},{"key":"ref_45","unstructured":"Wang, X., Zhang, W., and Rajtmajer, S. (2024). Monolingual and Multilingual Misinformation Detection for Low-Resource Languages: A Comprehensive Survey. arXiv."},{"key":"ref_46","unstructured":"Kuntur, S., Wr\u00f3blewska, A., Paprzycki, M., and Ganzha, M. (2024). Fake News Detection: It\u2019s All in the Data!. arXiv."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/13\/11\/292\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:30:10Z","timestamp":1760113810000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/13\/11\/292"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,11]]},"references-count":46,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2024,11]]}},"alternative-id":["computers13110292"],"URL":"https:\/\/doi.org\/10.3390\/computers13110292","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,11]]}}}