{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,13]],"date-time":"2026-10-13T18:37:53Z","timestamp":1791916673577,"version":"4.3.4"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:00:00Z","timestamp":1750291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/publication-rights-and-licensing-policy"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>Large Language Models (LLMs) are prone to hallucinations, e.g., factually incorrect information, in their responses. These hallucinations present challenges for LLM-based applications that demand high factual accuracy. Existing hallucination detection methods primarily depend on external resources, which can suffer from issues such as low availability, incomplete coverage, privacy concerns, high latency, low reliability, and poor scalability. There are also methods depending on output probabilities, which are often inaccessible for closed-source LLMs like GPT models. This paper presents MetaQA, a self-contained hallucination detection approach that leverages metamorphic relation and prompt mutation. Unlike existing methods, MetaQA operates without any external resources and is compatible with both open-source and closed-source LLMs. MetaQA is based on the hypothesis that if an LLM\u2019s response is a hallucination, the designed metamorphic relations will be violated. We compare MetaQA with the state-of-the-art zero-resource hallucination detection method, SelfCheckGPT, across multiple datasets, and on two open-source and two closed-source LLMs. Our results reveal that MetaQA outperforms SelfCheckGPT in terms of precision, recall, and f1 score. For the four LLMs we study, MetaQA outperforms SelfCheckGPT with a superiority margin ranging from 0.041 - 0.113 (for precision), 0.143 - 0.430 (for recall), and 0.154 - 0.368 (for F1-score). For instance, with Mistral-7B, MetaQA achieves an average F1-score of 0.435, compared to SelfCheckGPT\u2019s F1-score of 0.205, representing an improvement rate of 112.2%. MetaQA also demonstrates superiority across all different categories of questions.<\/jats:p>","DOI":"10.1145\/3715735","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T11:15:34Z","timestamp":1750331734000},"page":"425-445","source":"Crossref","is-referenced-by-count":34,"title":["Hallucination Detection in Large Language Models with Metamorphic Relations"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-3482-7667","authenticated-orcid":false,"given":"Borui","family":"Yang","sequence":"first","affiliation":[{"name":"King's College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9319-3483","authenticated-orcid":false,"given":"Md Afif","family":"Al Mamun","sequence":"additional","affiliation":[{"name":"University of Calgary, Calgary, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0481-7264","authenticated-orcid":false,"given":"Jie M.","family":"Zhang","sequence":"additional","affiliation":[{"name":"King's College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1376-095X","authenticated-orcid":false,"given":"Gias","family":"Uddin","sequence":"additional","affiliation":[{"name":"York University, Toronto, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2022. OpenAI ChatGPT. https:\/\/openai.com\/index\/chatgpt\/"},{"issue":"2020","key":"e_1_3_1_3_2","first-page":"9649","article-title":"Boxe: A box embedding model for knowledge base completion","volume":"33","author":"Abboud Ralph","year":"2020","unstructured":"Ralph Abboud, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. 2020. Boxe: A box embedding model for knowledge base completion. Advances in Neural Information Processing Systems 33 (2020), 9649\u20139661.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","unstructured":"Ayush Agrawal Mirac Suzgun Lester Mackey and Adam Tauman Kalai. 2023. Do Language Models Know When They\u2019re Hallucinating References? arXiv preprint arXiv:2305.18248 (2023). doi:10.48550\/arXiv.2305.18248","DOI":"10.48550\/arXiv.2305.18248"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","unstructured":"Nicholas Asher and Swarnadeep Bhar. 2024. Strong hallucinations from negation and how to fix them. arXiv preprint arXiv:2402.10543 (2024). doi:10.48550\/arXiv.2402.10543","DOI":"10.48550\/arXiv.2402.10543"},{"key":"e_1_3_1_6_2","doi-asserted-by":"crossref","unstructured":"Pepa Atanasova Jakob Grue Simonsen Christina Lioma and Isabelle Augenstein. 2020. Generating Fact Checking Explanations. In Annual Meeting of the Association for Computational Linguistics. https:\/\/api.semanticscholar.org\/CorpusID:215744944","DOI":"10.18653\/v1\/2020.acl-main.656"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","unstructured":"Isabelle Augenstein Christina Lioma Dongsheng Wang Lucas Chaves Lima Casper Hansen Christian Hansen and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. arXiv preprint arXiv:1909.03242 (2019). doi:10.48550\/arXiv.1909.03242","DOI":"10.48550\/arXiv.1909.03242"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","unstructured":"Jifan Chen Grace Kim Aniruddh Sriram Greg Durrett and Eunsol Choi. 2023. Complex claim verification with evidence retrieved in the wild. arXiv preprint arXiv:2305.11859 (2023). doi:10.48550\/arXiv.2305.11859","DOI":"10.48550\/arXiv.2305.11859"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","unstructured":"I Chern Steffi Chern Shiqi Chen Weizhe Yuan Kehua Feng Chunting Zhou Junxian He Graham Neubig Pengfei Liu et al. 2023. FacTool: Factuality Detection in Generative AI\u2013A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios. arXiv preprint arXiv:2307.13528 (2023). doi:10.48550\/arXiv.2307.13528","DOI":"10.48550\/arXiv.2307.13528"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","unstructured":"Roi Cohen May Hamri Mor Geva and Amir Globerson. 2023. Lm vs lm: Detecting factual errors via cross examination. arXiv preprint arXiv:2305.13281 (2023). doi:10.48550\/arXiv.2305.13281","DOI":"10.48550\/arXiv.2305.13281"},{"key":"e_1_3_1_12_2","doi-asserted-by":"crossref","unstructured":"Shehzaad Dhuliawala Mojtaba Komeili Jing Xu Roberta Raileanu Xian Li Asli Celikyilmaz and Jason Weston. 2024. Chain-of-Verification Reduces Hallucination in Large Language Models. In Findings of the Association for Computational Linguistics ACL 2024 Lun-Wei Ku Andre Martins and Vivek Srikumar (Eds.). Association for Computational Linguistics Bangkok Thailand and virtual meeting 3563\u20133578. https:\/\/aclanthology.org\/2024.findings-acl.212","DOI":"10.18653\/v1\/2024.findings-acl.212"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","unstructured":"Christopher Foster Abhishek Gulati Mark Harman Inna Harper Ke Mao Jillian Ritchey Herv\u00e9 Robert and Shubho Sengupta. 2025. Mutation-Guided LLM-based Test Generation at Meta. arXiv preprint arXiv:2501.12862 (2025). doi:10.48550\/arXiv.2501.12862","DOI":"10.48550\/arXiv.2501.12862"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","unstructured":"Boris A. Galitsky. 2023. Truth-O-Meter: Collaborating with LLM in Fighting its Hallucinations. Preprints (July 2023). doi:10.20944\/preprints202307.1723.v1","DOI":"10.20944\/preprints202307.1723.v1"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00454"},{"key":"e_1_3_1_16_2","unstructured":"Andreas Hanselowski Christian Stab Claudia Schulz Zile Li and Iryna Gurevych. 2019. A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking. ArXiv abs\/1911.01214 (2019). https:\/\/api.semanticscholar.org\/CorpusID:207779874"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","unstructured":"Lei Huang Weijiang Yu Weitao Ma Weihong Zhong Zhangyin Feng Haotian Wang Qianglong Chen Weihua Peng Xiaocheng Feng Bing Qin et al. 2023. A survey on hallucination in large language models: Principles taxonomy challenges and open questions. arXiv preprint arXiv:2311.05232 (2023). doi:10.1145\/3703155","DOI":"10.1145\/3703155"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","unstructured":"Siqing Huo Negar Arabzadeh and Charles LA Clarke. 2023. Retrieving supporting evidence for llms generated answers. arXiv preprint arXiv:2306.13781 (2023). doi:10.48550\/arXiv.2306.13781","DOI":"10.48550\/arXiv.2306.13781"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","unstructured":"Sangwon Hyun Mingyu Guo and M. Ali Babar. 2024. METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities. In 2024 IEEE Conference on Software Testing Verification and Validation (ICST). 117\u2013128. doi:10.1109\/ICST60714.2024.00019","DOI":"10.1109\/ICST60714.2024.00019"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3571730"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","unstructured":"Saurav Kadavath Tom Conerly Amanda Askell Tom Henighan Dawn Drain Ethan Perez Nicholas Schiefer Zac Hatfield-Dodds Nova DasSarma Eli Tran-Johnson et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 (2022). doi:10.48550\/arXiv.2207.05221","DOI":"10.48550\/arXiv.2207.05221"},{"issue":"2024","key":"e_1_3_1_22_2","article-title":"REALTIME QA: what\u2019s the answer right now?","volume":"36","author":"Kasai Jungo","year":"2024","unstructured":"Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. 2024. REALTIME QA: what\u2019s the answer right now? Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","unstructured":"Wojciech Kryscinski Bryan McCann Caiming Xiong and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840 (2019). doi:10.48550\/arXiv.1910.12840","DOI":"10.48550\/arXiv.1910.12840"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","unstructured":"Vivian Lai Alison Smith-Renner Ke Zhang Ruijia Cheng Wenjuan Zhang Joel Tetreault and Alejandro Jaimes. 2022. An exploration of post-editing effectiveness in text summarization. arXiv preprint arXiv:2206.06383 (2022). doi:10.48550\/arXiv.2206.06383","DOI":"10.48550\/arXiv.2206.06383"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","unstructured":"Junyi Li Xiaoxue Cheng Wayne Xin Zhao Jian-Yun Nie and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747 (2023). doi:10.48550\/arXiv.2305.11747","DOI":"10.48550\/arXiv.2305.11747"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","unstructured":"Ningke Li Yuekang Li Yi Liu Ling Shi Kailong Wang and Haoyu Wang. 2024. HalluVault: A Novel Logic Programming-aided Metamorphic Testing Framework for Detecting Fact-Conflicting Hallucinations in Large Language Models. arXiv preprint arXiv:2405.00648 (2024). doi:10.48550\/arXiv.2405.00648","DOI":"10.48550\/arXiv.2405.00648"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","unstructured":"Wei Li Wenhao Wu Moye Chen Jiachen Liu Xinyan Xiao and Hua Wu. 2022. Faithfulness in natural language generation: A systematic survey of analysis evaluation and optimization methods. arXiv preprint arXiv:2203.05227 (2022). doi:10.48550\/arXiv.2203.05227","DOI":"10.48550\/arXiv.2203.05227"},{"key":"e_1_3_1_28_2","doi-asserted-by":"crossref","unstructured":"Ke Liang Lingyuan Meng Meng Liu Yue Liu Wenxuan Tu Siwei Wang Sihang Zhou Xinwang Liu Fuchun Sun and Kunlun He. 2024. A survey of knowledge graph reasoning on graph types: Static dynamic and multi-modal. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).","DOI":"10.1109\/TPAMI.2024.3417451"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","unstructured":"Stephanie Lin Jacob Hilton and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 (2021). doi:10.48550\/arXiv.2109.07958","DOI":"10.48550\/arXiv.2109.07958"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","unstructured":"Zheheng Luo Qianqian Xie and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization. arXiv preprint arXiv:2303.15621 (2023). doi:10.48550\/arXiv.2303.15621","DOI":"10.48550\/arXiv.2303.15621"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","unstructured":"Potsawee Manakul Adian Liusie and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896 (2023). doi:10.48550\/arXiv.2303.08896","DOI":"10.48550\/arXiv.2303.08896"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","unstructured":"Sewon Min Kalpesh Krishna Xinxi Lyu Mike Lewis Wen-tau Yih Pang Wei Koh Mohit Iyyer Luke Zettlemoyer and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251 (2023). doi:10.48550\/arXiv.2305.14251","DOI":"10.48550\/arXiv.2305.14251"},{"key":"e_1_3_1_33_2","unstructured":"Seungwhan Moon Pararth Shah Anuj Kumar and Rajen Subba. 2019. OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs. In Annual Meeting of the Association for Computational Linguistics. https:\/\/api.semanticscholar.org\/CorpusID:196176000"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","unstructured":"Francesco Moramarco Alex Papadopoulos Korfiatis Aleksandar Savkov and Ehud Reiter. 2021. A preliminary study on evaluating consultation notes with post-editing. arXiv preprint arXiv:2104.04402 (2021). doi:10.48550\/arXiv.2104.04402","DOI":"10.48550\/arXiv.2104.04402"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","unstructured":"R OpenAI et al. 2023. GPT-4 technical report. ArXiv 2303 (2023) 08774. doi:10.48550\/arXiv.2303.08774","DOI":"10.48550\/arXiv.2303.08774"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","unstructured":"Shuyin Ouyang Jie M. Zhang Mark Harman and Meng Wang. 2025. An Empirical Study of the Non-Determinism of ChatGPT in Code Generation. ACM Trans. Softw. Eng. Methodol. 34 2 Article 42 (Jan. 2025) 28 pages.","DOI":"10.1145\/3697010"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","unstructured":"Libo Qin Qiguang Chen Xiachong Feng Yang Wu Yongheng Zhang Yinghui Li Min Li Wanxiang Che and Philip S Yu. 2024. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819 (2024). doi:10.48550\/arXiv.2405.12819","DOI":"10.48550\/arXiv.2405.12819"},{"issue":"2020","key":"e_1_3_1_38_2","first-page":"19716","article-title":"Beta embeddings for multi-hop logical reasoning in knowledge graphs","volume":"33","author":"Ren Hongyu","year":"2020","unstructured":"Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances in Neural Information Processing Systems 33 (2020), 19716\u201319726.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","unstructured":"Abigail See Peter J Liu and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368 (2017). doi:10.48550\/arXiv.1704.04368","DOI":"10.48550\/arXiv.1704.04368"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","unstructured":"Zeyu Sun Jie M Zhang Mark Harman Mike Papadakis and Lu Zhang. 2020. Automatic testing and improvement of machine translation. In Proceedings of the ACM\/IEEE 42nd international conference on software engineering. 974\u2013985.","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnlest.2022.100159"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","unstructured":"Neeraj Varshney Satyam Raj Venkatesh Mishra Agneet Chatterjee Ritika Sarkar Amir Saeidi and Chitta Baral. 2024. Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negation. arXiv preprint arXiv:2406.05494 (2024). doi:10.48550\/arXiv.2406.05494","DOI":"10.48550\/arXiv.2406.05494"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","unstructured":"Neeraj Varshney Wenlin Yao Hongming Zhang Jianshu Chen and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987 (2023). doi:10.48550\/arXiv.2307.03987","DOI":"10.48550\/arXiv.2307.03987"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","unstructured":"Tu Vu Mohit Iyyer Xuezhi Wang Noah Constant Jerry Wei Jason Wei Chris Tar Yun-Hsuan Sung Denny Zhou Quoc Le et al. 2023. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214 (2023). doi:10.48550\/arXiv.2310.03214","DOI":"10.48550\/arXiv.2310.03214"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","unstructured":"Miao Xiong Zhiyuan Hu Xinyang Lu Yifei Li Jie Fu Junxian He and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063 (2023). doi:10.48550\/arXiv.2306.13063","DOI":"10.48550\/arXiv.2306.13063"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","unstructured":"Ziwei Xu Sanjay Jain and Mohan Kankanhalli. 2024. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817 (2024). doi:10.48550\/arXiv.2401.11817","DOI":"10.48550\/arXiv.2401.11817"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","unstructured":"Zhilin Yang Peng Qi Saizheng Zhang Yoshua Bengio William W Cohen Ruslan Salakhutdinov and Christopher D Manning. 2018. HotpotQA: A dataset for diverse explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 (2018). doi:10.48550\/arXiv.1809.09600","DOI":"10.48550\/arXiv.1809.09600"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","unstructured":"Jia-Yu Yao Kun-Peng Ning Zhen-Hui Liu Mu-Nan Ning and Li Yuan. 2023. Llm lies: Hallucinations are not bugs but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023). doi:10.48550\/arXiv.2310.01469","DOI":"10.48550\/arXiv.2310.01469"},{"key":"e_1_3_1_49_2","doi-asserted-by":"crossref","unstructured":"Yifan Yao Jinhao Duan Kaidi Xu Yuanfang Cai Zhibo Sun and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good the bad and the ugly. High-Confidence Computing (2024) 100211.","DOI":"10.1016\/j.hcc.2024.100211"},{"key":"e_1_3_1_50_2","unstructured":"Chan Xing Yu Chan Si Yu James and Poh Hui-Li Phyllis David. [n.d.]. CAN LLMS HAVE A FEVER? INVESTIGATING THE EFFECTS OF TEMPERATURE ON LLM SECURITY. ([n. d.])."},{"key":"e_1_3_1_51_2","doi-asserted-by":"crossref","unstructured":"Jie Zhang Junjie Chen Dan Hao Yingfei Xiong Bing Xie Lu Zhang and Hong Mei. 2014. Search-based inference of polynomial metamorphic relations. In Proceedings of the 29th ACM\/IEEE international conference on Automated software engineering. 701\u2013712.","DOI":"10.1145\/2642937.2642994"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_1_53_2","doi-asserted-by":"crossref","unstructured":"Xiaoyu Zhang Jianping Li Po-Wei Chi Senthil Chandrasegaran and Kwan-Liu Ma. 2023. ConceptEVA: concept-based interactive exploration and customization of document summaries. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1\u201316.","DOI":"10.1145\/3544548.3581260"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","unstructured":"Yue Zhang Yafu Li Leyang Cui Deng Cai Lemao Liu Tingchen Fu Xinting Huang Enbo Zhao Yu Zhang Yulong Chen et al. 2023. Siren\u2019s song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219 (2023). doi:10.48550\/arXiv.2309.01219","DOI":"10.48550\/arXiv.2309.01219"},{"key":"e_1_3_1_55_2","doi-asserted-by":"crossref","unstructured":"Zili Zhou Shaowu Liu Guandong Xu and Wu Zhang. 2019. On completing sparse knowledge base with transitive relation embedding. In Proceedings of the AAAI Conference on Artificial Intelligence Vol. 33. 3125\u20133132.","DOI":"10.1609\/aaai.v33i01.33013125"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715735","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T11:04:05Z","timestamp":1787483045000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715735"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":54,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3715735"],"URL":"https:\/\/doi.org\/10.1145\/3715735","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}