{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T11:36:45Z","timestamp":1787917005959,"version":"build-2784847793"},"reference-count":82,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:00:00Z","timestamp":1760140800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>To explore the potential of Large Language Models (LLMs) as AI Annotators in the domain of sustainability reporting, this study establishes a systematic evaluation methodology. We use the specific case of European football clubs, quantifying their sustainability reports based on the sport Positive matrix as a benchmark to compare the performance of three state-of-the-art models (i.e., GPT-4o, Qwen-2-72b-instruct, and Llama-3-70b-instruct) against human expert scores. The evaluation is benchmarked on dimensions including accuracy, mean absolute error (MAE), and hallucination rates. The results indicate that GPT-4o is the top performer, yet its average accuracy of approximately 56% shows it cannot fully replace human experts at present. The study also reveals significant issues with overconfidence and factual hallucinations in models like Qwen-2-72b-instructon. Critically, we find that by implementing further data processing, specifically a Chain-of-Verification (CoVe) self-correction method, GPT-4o\u2019s initial hallucination rate is successfully reduced from 16% to 10%, while accuracy improved to 58%. In conclusion, while LLMs demonstrate immense potential to streamline and democratize sustainability ratings, inherent risks like hallucinations remain a primary obstacle. Adopting verification strategies such as CoVe is a crucial pathway to enhancing model reliability and advancing their effective application in this field.<\/jats:p>","DOI":"10.3390\/systems13100899","type":"journal-article","created":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T11:39:34Z","timestamp":1760701174000},"page":"899","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["The AI Annotator: Large Language Models\u2019 Potential in Scoring Sustainability Reports"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-1465-9515","authenticated-orcid":false,"given":"Yue","family":"Wu","sequence":"first","affiliation":[{"name":"School of Business Administration, Capital University of Economics and Business, 121 Zhangjialukou, Beijing 100070, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2866-8180","authenticated-orcid":false,"given":"Peng","family":"Hu","sequence":"additional","affiliation":[{"name":"School of Business Administration, Capital University of Economics and Business, 121 Zhangjialukou, Beijing 100070, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Derek D.","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Business Administration, Capital University of Economics and Business, 121 Zhangjialukou, Beijing 100070, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Schaltegger, S., Bennett, M., and Burritt, R. (2006). Sustainability Accounting and Reporting: Development, Linkages and Reflection. An Introduction. Sustainability Accounting and Reporting, Springer.","DOI":"10.1007\/978-1-4020-4974-3"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.bushor.2019.09.003","article-title":"Rulers of the World, Unite! The Challenges and Opportunities of Artificial Intelligence","volume":"63","author":"Kaplan","year":"2020","journal-title":"Bus. Horiz."},{"key":"ref_3","unstructured":"(2024, July 14). Sustainability Rate the Raters 2020: Investor Survey and Interview Results. Available online: https:\/\/www.sustainability.com\/globalassets\/sustainability.com\/thinking\/pdfs\/sustainability-ratetheraters2020-report.pdf."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1426","DOI":"10.1002\/csr.2177","article-title":"Inside the ESG Ratings: (Dis)Agreement and Performance","volume":"28","author":"Billio","year":"2021","journal-title":"Corp. Soc. Responsib. Environ. Manag."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Hughes, A., Urban, M.A., and W\u00f3jcik, D. (2021). Alternative ESG Ratings: How Technological Innovation Is Reshaping Sustainable Investment. Sustainability, 13.","DOI":"10.3390\/su13063551"},{"key":"ref_6","unstructured":"(2024, July 07). OpenAI Hello GPT-4o. Available online: https:\/\/openai.com\/index\/hello-gpt-4o\/."},{"key":"ref_7","unstructured":"European Club Association (2025, September 07). Sustainability Strategy. Available online: https:\/\/online.flippingbook.com\/view\/561335824\/."},{"key":"ref_8","unstructured":"(2024, July 07). Sport Positive Environmental Sustainability Matrix. Available online: https:\/\/www.sportpositiveleagues.com\/wp-content\/uploads\/2023\/03\/Sport-Positive-Leagues-EPL-Enviro-Sustainability-Matrix-202223-Points-Key.pdf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"851","DOI":"10.5465\/amj.2019.0238","article-title":"Why Activist Hedge Funds Target Socially Responsible Firms: The Reaction Costs of Signaling Corporate Social Responsibility","volume":"64","author":"DesJardine","year":"2021","journal-title":"Acad. Manag. J."},{"key":"ref_10","unstructured":"Zhang, M., Press, O., Merrill, W., Liu, A., and Smith, N.A. (2023). How Language Model Hallucinations Can Snowball. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"101593","DOI":"10.1016\/j.frl.2020.101593","article-title":"Does ESG Certification Add Firm Value?","volume":"39","author":"Wong","year":"2021","journal-title":"Financ. Res. Lett."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1016\/j.ribaf.2015.05.001","article-title":"The Effects of Social Ratings on Firm Value","volume":"36","author":"Cellier","year":"2016","journal-title":"Res. Int. Bus. Financ."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1975","DOI":"10.1002\/bse.2482","article-title":"How Do Corporate Environmental Policy and Corporate Reputation Affect Risk-adjusted Financial Performance?","volume":"29","author":"Gangi","year":"2020","journal-title":"Bus. Strategy Environ."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"102302","DOI":"10.1016\/j.frl.2021.102302","article-title":"When ESG Meets AAA: The Effect of ESG Rating Changes on Stock Returns","volume":"46","author":"Shanaev","year":"2022","journal-title":"Financ. Res. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1315","DOI":"10.1093\/rof\/rfac033","article-title":"Aggregate Confusion: The Divergence of ESG Ratings","volume":"26","author":"Berg","year":"2022","journal-title":"Rev. Financ."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1597","DOI":"10.1002\/smj.2407","article-title":"Do Ratings of Firms Converge? Implications for Managers, Investors and Strategy Researchers","volume":"37","author":"Chatterji","year":"2016","journal-title":"Strateg. Manag. J."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Hou, C., Zhu, G., Zheng, J., Zhang, L., Huang, X., Zhong, T., Li, S., Du, H., and Ker, C.L. (2024, January 18\u201322). Prompt-Based and Fine-Tuned GPT Models for Context-Dependent and -Independent Deductive Coding in Social Annotation. Proceedings of the ACM International Conference Proceeding Series, Kyoto, Japan.","DOI":"10.1145\/3636555.3636910"},{"key":"ref_18","first-page":"342","article-title":"ChatGPT: Bullshit Spewer or the End of Traditional Assessments in Higher Education?","volume":"6","author":"Rudolph","year":"2023","journal-title":"J. Appl. Learn. Teach."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1002\/hcs2.61","article-title":"Large Language Models in Health Care: Development, Applications, and Challenges","volume":"2","author":"Yang","year":"2023","journal-title":"Health Care Sci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"109713","DOI":"10.1016\/j.isci.2024.109713","article-title":"The Application of Large Language Models in Medicine: A Scoping Review","volume":"27","author":"Meng","year":"2024","journal-title":"iScience"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yan, L., Echeverria, V., Nieto, G.F., Jin, Y., Swiecki, Z., Zhao, L., Ga\u0161evi\u0107, D., and Martinez-Maldonado, R. (2023). Human-AI Collaboration in Thematic Analysis Using ChatGPT: A User Study and Design Recommendations. arXiv.","DOI":"10.1145\/3613905.3650732"},{"key":"ref_22","first-page":"42","article-title":"ChatGPT for Next Generation Science Learning","volume":"29","author":"Zhai","year":"2023","journal-title":"ACM Mag. Stud."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1111\/1911-3846.12832","article-title":"FinBERT: A Large Language Model for Extracting Information from Financial Text","volume":"40","author":"Huang","year":"2023","journal-title":"Contemp. Account. Res."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Kim, A.G., Muhn, M., Nikolaev, V.V., Baik, B., Bradshaw, M., Dou, Y., Gassen, J., Han, S.-Y., Jain, K., and Koijen, R. (2024). Financial Statement Analysis with Large Language Models. arXiv.","DOI":"10.2139\/ssrn.4835311"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"47","DOI":"10.2308\/JETA-2023-041","article-title":"A Demonstration of How ChatGPT Can Be Used in the Internal Auditing Process","volume":"22","author":"Eulerich","year":"2025","journal-title":"J. Emerg. Technol. Account."},{"key":"ref_26","unstructured":"F\u00f6hr, T.L., Schreyer, M., Juppe, T.A., and Marten, K.-U. (2023). Assuring Sustainable Futures: Auditing Sustainability Reports Using AI Foundation Models. SSRN Electron. J."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"125","DOI":"10.2308\/HORIZONS-2023-111","article-title":"Leveraging ChatGPT for Enhancing the Internal Audit Process\u2014A Real-World Example from Uniper, a Large Multinational Company","volume":"39","author":"Emett","year":"2025","journal-title":"Account. Horiz."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1080\/14765284.2023.2212434","article-title":"Bridging the Gap\u2014The Impact of ChatGPT on Financial Research","volume":"21","author":"Cao","year":"2023","journal-title":"J. Chin. Econ. Bus. Stud."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"471","DOI":"10.1080\/14765284.2023.2245279","article-title":"From Fiction to Fact: The Growing Role of Generative AI in Business and Finance","volume":"21","author":"Chen","year":"2023","journal-title":"J. Chin. Econ. Bus. Stud."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Bernard, D., Blankespoor, E., de Kok, T., and Toynbee, S. (2023). Confused Readers: A Modular Measure of Business Complexity. SSRN Electron. J.","DOI":"10.2139\/ssrn.4480309"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Faggioli, G., Dietz, L., Clarke, C.L.A., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., and Stein, B. (2023, January 23). Perspectives on Large Language Models for Relevance Judgment. Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, Taipei, Taiwan.","DOI":"10.1145\/3578337.3605136"},{"key":"ref_32","first-page":"27730","article-title":"Training Language Models to Follow Instructions with Human Feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv. Neural Inf. Process Syst."},{"key":"ref_33","unstructured":"Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Bermeo, J.D., Korobeynikova, M., and Gilardi, F. (2023). Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text-Annotation Tasks. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Huang, F., Kwak, H., and An, J. (2023, January 30). Is ChatGPT Better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech. Proceedings of the ACM Web Conference 2023\u2014Companion of the World Wide Web Conference, WWW 2023, Austin, TX, USA.","DOI":"10.1145\/3543873.3587368"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Gielens, E., and Sowula, J. (2024). Goodbye Human Annotators? Content Analysis of Policy Debates Using ChatGPT. SocArXiv.","DOI":"10.31235\/osf.io\/m5r3h"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"e2305016120","DOI":"10.1073\/pnas.2305016120","article-title":"ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks","volume":"120","author":"Gilardi","year":"2023","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"102440","DOI":"10.1016\/j.techsoc.2023.102440","article-title":"The Road to Corporate Sustainability: The Importance of Artificial Intelligence","volume":"76","author":"Chen","year":"2024","journal-title":"Technol. Soc."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1108\/ARA-07-2023-0201","article-title":"Renovation in Environmental, Social and Governance (ESG) Research: The Application of Machine Learning","volume":"32","author":"Zhang","year":"2023","journal-title":"Asian Rev. Account."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"144572","DOI":"10.1016\/j.jclepro.2024.144572","article-title":"ESGReveal: An LLM-Based Approach for Extracting Structured Data from ESG Reports","volume":"489","author":"Zou","year":"2023","journal-title":"J. Clean. Prod."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1080\/00014788.2019.1609346","article-title":"Retrieving, Classifying and Analysing Narrative Commentary in Unstructured (Glossy) Annual Reports Published as PDF Files","volume":"50","author":"Alves","year":"2020","journal-title":"Account. Bus. Res."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1049","DOI":"10.1111\/j.1475-679X.2010.00382.x","article-title":"The Information Content of Forward-Looking Statements in Corporate Filings\u2014A Na\u00efve Bayesian Machine Learning Approach","volume":"48","author":"LI","year":"2010","journal-title":"J. Account. Res."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Lin, L.H.-M., Ting, F.-K., Chang, T.-J., Wu, J.-W., and Tsai, R.T.-H. (2024, January 19\u201321). GPT4ESG: Streamlining Environment, Society, and Governance Analysis with Custom AI Models. Proceedings of the 2024 IEEE 4th International Conference on Electronic Communications, Internet of Things and Big Data (ICEIB), Taipei, Taiwan.","DOI":"10.1109\/ICEIB61477.2024.10602567"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Kim, A., Muhn, M., and Nikolaev, V. (2023). Bloated Disclosures: Can ChatGPT Help Investors Process Information?. arXiv.","DOI":"10.2139\/ssrn.4425527"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1187","DOI":"10.1111\/1475-679X.12123","article-title":"Textual Analysis in Accounting and Finance: A Survey","volume":"54","author":"Loughran","year":"2016","journal-title":"J. Account. Res."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Ni, J., Bingler, J., Colesanti-Senni, C., Kraus, M., Gostlow, G., Schimanski, T., Stammbach, D., Vaghefi, S.A., Wang, Q., and Webersinke, N. (2023, January 6\u201310). CHATREPORT: Democratizing Sustainability Disclosure Analysis through LLM-Based Tools. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Singapore.","DOI":"10.18653\/v1\/2023.emnlp-demo.3"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1140\/epjds\/s13688-024-00481-2","article-title":"Glitter or Gold? Deriving Structured Insights from Sustainability Reports via Large Language Models","volume":"13","author":"Bronzini","year":"2023","journal-title":"EPJ Data Sci."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"319","DOI":"10.14207\/ejsd.2023.v12n4p319","article-title":"A Conceptual Framework for Subdomain Specific Pre-Training of Large Language Models for Green Claim Detection","volume":"12","author":"Moodaley","year":"2023","journal-title":"Eur. J. Sustain. Dev."},{"key":"ref_48","first-page":"104027","article-title":"Evaluating the Impact of Report Readability on ESG Scores: A Generative AI Approach","volume":"101","author":"Managi","year":"2024","journal-title":"Int. Rev. Financ. Anal."},{"key":"ref_49","unstructured":"Luccioni, A., Baylor, E., and Duchene, N. (2020). Analyzing Sustainability Reports Using Natural Language Processing. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"102776","DOI":"10.1016\/j.frl.2022.102776","article-title":"Cheap Talk and Cherry-Picking: What ClimateBert Has to Say on Corporate Climate Risk Disclosures","volume":"47","author":"Bingler","year":"2022","journal-title":"Financ. Res. Lett."},{"key":"ref_51","first-page":"96","article-title":"How Will AI Text Generation and Processing Impact Sustainability Reporting? Critical Analysis, a Conceptual Framework and Avenues for Future Research","volume":"15","author":"Dimes","year":"2024","journal-title":"Sustain. Account. Manag. Policy J."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"e26404","DOI":"10.1016\/j.heliyon.2024.e26404","article-title":"ESG2PreEM: Automated ESG Grade Assessment Framework Using Pre-Trained Ensemble Models","volume":"10","author":"Lee","year":"2024","journal-title":"Heliyon"},{"key":"ref_53","unstructured":"Kannan, N., and Seki, Y. (2023, January 19\u201325). Textual Evidence Extraction for ESG Scores. Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial Forecasting, Macao, China."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Lanza, A., Bernardini, E., and Faiella, I. (2020). Mind the Gap! Machine Learning, ESG Metrics and Sustainable Investment. SSRN Electron. J.","DOI":"10.2139\/ssrn.3659584"},{"key":"ref_55","unstructured":"(2025, September 17). Meta Llama-3-70B-Instruct. Available online: https:\/\/huggingface.co\/meta-llama\/Meta-Llama-3-70B-Instruct."},{"key":"ref_56","unstructured":"(2025, September 17). Qwen Qwen2-72B-Instruct. Available online: https:\/\/huggingface.co\/Qwen\/Qwen2-72B-Instruct."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1007\/s10551-019-04164-1","article-title":"The Influence of Firm Size on the ESG Score: Corporate Sustainability Ratings Under Review","volume":"167","author":"Drempetic","year":"2020","journal-title":"J. Bus. Ethics"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1111\/jacf.12346","article-title":"Four Things No One Will Tell You About ESG Data","volume":"31","author":"Kotsantonis","year":"2019","journal-title":"J. Appl. Corp. Financ."},{"key":"ref_59","first-page":"7","article-title":"Preprocessing Techniques for Text Mining\u2014An Overview","volume":"5","author":"Vijayarani","year":"2015","journal-title":"Int. J. Comput. Sci. Commun. Netw."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Clavi\u00e9, B., Ciceu, A., Naylor, F., Souli\u00e9, G., and Brightwell, T. (2023, January 21\u201323). Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification. Proceedings of the International Conference on Applications of Natural Language to Information Systems, Derby, UK.","DOI":"10.1007\/978-3-031-35320-8_1"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Ekin, S. (2023). Prompt Engineering for ChatGPT: A Quick Guide to Techniques, Tips, and Best Practices. Authorea Prepr.","DOI":"10.36227\/techrxiv.22683919"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Wu, Y., and Hu, G. (2023, January 6\u20137). Exploring Prompt Engineering with GPT Language Models for Document-Level Machine Translation: Insights and Findings. Proceedings of the Eighth Conference on Machine Translation, Singapore.","DOI":"10.18653\/v1\/2023.wmt-1.15"},{"key":"ref_63","first-page":"e75748","article-title":"The Impact of Temperature on Extracting Information from Clinical Trial Publications Using Large Language Models","volume":"16","author":"Windisch","year":"2024","journal-title":"Cureus"},{"key":"ref_64","unstructured":"Borchers, C., Shahrokhian, B., Balzan, F., Tajik, E., Sankaranarayanan, S., and Simon, S. (2025). Temperature and Persona Shape LLM Agent Consensus with Minimal Accuracy Gains in Qualitative Coding. arXiv."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"e53559","DOI":"10.2196\/53559","article-title":"The Temperature Feature of ChatGPT: Modifying Creativity for Clinical Research","volume":"11","author":"Davis","year":"2024","journal-title":"JMIR Hum. Factors"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Liyanage, C., Gokani, R., and Mago, V. (2023). GPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification Task. Authorea Prepr.","DOI":"10.36227\/techrxiv.24143706"},{"key":"ref_67","first-page":"1","article-title":"Multi-Label Classifier Performance Evaluation with Confusion Matrix","volume":"1","year":"2020","journal-title":"Comput. Sci. Inf. Technol."},{"key":"ref_68","first-page":"1","article-title":"Confusion Matrix in Binary Classification Problems: A Step-by-Step Tutorial","volume":"6","author":"Amin","year":"2022","journal-title":"J. Eng. Res."},{"key":"ref_69","unstructured":"Li, Z., Shi, Y., Liu, Z., Yang, F., Liu, N., and Du, M. (2024). Quantifying Multilingual Performance of Large Language Models Across Languages. arXiv."},{"key":"ref_70","first-page":"34161","article-title":"Mindmerger: Efficiently boosting LLM reasoning in non-english languages","volume":"37","author":"Huang","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"e53164","DOI":"10.2196\/53164","article-title":"Hallucination Rates and Reference Accuracy in ChatGPT and Bard for Systematic Reviews: A Comparative Analysis","volume":"26","author":"Chelli","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of Hallucination in Natural Language Generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., and Chung, W. (2023). A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. arXiv.","DOI":"10.18653\/v1\/2023.ijcnlp-main.45"},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"1500","DOI":"10.1162\/tacl_a_00615","article-title":"Hallucinations in Large Multilingual Translation Models","volume":"11","author":"Guerreiro","year":"2023","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"De Cao, N., Aziz, W., and Titov, I. (2021). Editing Factual Knowledge in Language Models. arXiv.","DOI":"10.18653\/v1\/2021.emnlp-main.522"},{"key":"ref_76","unstructured":"Zheng, S., Huang, J., and Chang, K.C.-C. (2023). Why Does ChatGPT Fall Short in Providing Truthful Answers?. arXiv."},{"key":"ref_77","first-page":"e35179","article-title":"Artificial Hallucinations in ChatGPT: Implications in Scientific Writing","volume":"15","author":"Alkaissi","year":"2023","journal-title":"Cureus"},{"key":"ref_78","first-page":"1","article-title":"A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions","volume":"43","author":"Huang","year":"2025","journal-title":"ACM Trans. Inf. Syst."},{"key":"ref_79","doi-asserted-by":"crossref","unstructured":"Wang, C., and Sennrich, R. (2020). On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.326"},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Azaria, A., and Mitchell, T. (2023). The Internal State of an LLM Knows When It\u2019s Lying. arXiv.","DOI":"10.18653\/v1\/2023.findings-emnlp.68"},{"key":"ref_81","unstructured":"Cotra, A. (2025, June 15). Why AI Alignment Could Be Hard with Modern Deep Learning. Cold Takes. Available online: https:\/\/www.cold-takes.com\/why-ai-alignment-could-be-hard-with-modern-deep-learning\/."},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models. arXiv.","DOI":"10.18653\/v1\/2024.findings-acl.212"}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/13\/10\/899\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T04:16:53Z","timestamp":1760761013000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/13\/10\/899"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,11]]},"references-count":82,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["systems13100899"],"URL":"https:\/\/doi.org\/10.3390\/systems13100899","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,11]]}}}