{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T17:11:44Z","timestamp":1772039504736,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":42,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,8,24]],"date-time":"2024-08-24T00:00:00Z","timestamp":1724457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,8,25]]},"DOI":"10.1145\/3637528.3671796","type":"proceedings-article","created":{"date-parts":[[2024,8,25]],"date-time":"2024-08-25T04:55:12Z","timestamp":1724561712000},"page":"2721-2732","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["On Early Detection of Hallucinations in Factual Question Answering"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1584-1961","authenticated-orcid":false,"given":"Ben","family":"Snyder","sequence":"first","affiliation":[{"name":"Amazon Web Services, Santa Clara, CA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4084-8748","authenticated-orcid":false,"given":"Marius","family":"Moisescu","sequence":"additional","affiliation":[{"name":"Amazon Web Services, Seattle, WA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8347-7813","authenticated-orcid":false,"given":"Muhammad Bilal","family":"Zafar","sequence":"additional","affiliation":[{"name":"Ruhr-Universit\u00e4t Bochum & Research Center for Trustworthy Data Science and Security, University Alliance Ruhr, Bochum, Nordrhein-Westfalen, Germany"}]}],"member":"320","published-online":{"date-parts":[[2024,8,24]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00667"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"crossref","unstructured":"Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734.","DOI":"10.18653\/v1\/2023.findings-emnlp.68"},{"key":"e_1_3_2_2_3_1","unstructured":"Jon Christian. 2018. Why Is Google Translate Spitting Out Sinister Religious Prophecies? https:\/\/www.vice.com\/en\/article\/j5npeg\/why-is-google-translat e-spitting-out-sinister-religious-prophecies. Accessed: 2023-08--16. (2018)."},{"key":"e_1_3_2_2_4_1","unstructured":"2023. Confirmed: the new Bing runs on OpenAI's GPT-4. https:\/\/blogs.bing.co m\/search\/march_2023\/Confirmed-the-new-Bing-runs-on-OpenAI%E2%80 %99s-GPT-4. Accessed: 2023--10--11. (2023)."},{"key":"e_1_3_2_2_5_1","volume-title":"Costa-juss\u00e0","author":"Dale David","year":"2023","unstructured":"David Dale, Elena Voita, Loic Barrault, and Marta R. Costa-juss\u00e0. 2023. Detecting and mitigating hallucinations in machine translation: model internal workings alone do well, sentence similarity Even better. In ACL. (July 2023)."},{"key":"e_1_3_2_2_6_1","unstructured":"Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608."},{"key":"e_1_3_2_2_7_1","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018","author":"Elsahar Hady","year":"2018","unstructured":"Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: a large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan, (May 2018). https:\/\/aclanthology.org\/L18--1544."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00330"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.eacl-main.75"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"crossref","unstructured":"Riccardo Guidotti Anna Monreale Salvatore Ruggieri Franco Turini Fosca Giannotti and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51 5 1--42.","DOI":"10.1145\/3236009"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"crossref","unstructured":"Lei Huang et al. 2023. A survey on hallucination in large language models: principles taxonomy challenges and open questions. arXiv preprint arXiv:2311.05232.","DOI":"10.1145\/3703155"},{"key":"e_1_3_2_2_14_1","first-page":"1","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ziwei Ji","year":"2023","unstructured":"Ziwei Ji et al. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55, 12, 1--38.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1147"},{"key":"e_1_3_2_2_16_1","unstructured":"Saurav Kadavath et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_3_2_2_18_1","unstructured":"Narine Kokhlikyan et al. 2020. Captum: a unified and generic model interpretability library for pytorch. (2020). arXiv: 2009.07896 [cs.LG]."},{"key":"e_1_3_2_2_19_1","first-page":"9459","article-title":"Retrieval-augmented generation for knowledgeintensive nlp tasks","volume":"33","author":"Patrick Lewis","year":"2020","unstructured":"Patrick Lewis et al. 2020. Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459--9474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_2_20_1","unstructured":"Percy Liang et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_2_2_23_1","unstructured":"Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems 30."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.557"},{"key":"e_1_3_2_2_25_1","first-page":"17359","article-title":"Locating and editing factual associations in gpt","volume":"35","author":"Meng Kevin","year":"2022","unstructured":"Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35, 17359--17372.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"crossref","unstructured":"Gr\u00e9goire Montavon Alexander Binder Sebastian Lapuschkin Wojciech Samek and Klaus-Robert M\u00fcller. 2019. Layer-wise relevance propagation: an overview. Explainable AI: interpreting explaining and visualizing deep learning 193--209.","DOI":"10.1007\/978-3-030-28954-6_10"},{"key":"e_1_3_2_2_27_1","unstructured":"2023. Need a Last Minute Mother's Day Gift? AI Is Here to Help. https:\/\/about .you.com\/need-a-last-minute-mothers-day-gift-ai-is-here-to-help-d363b17 e76b4\/. Accessed: 2023--10--11. (2023)."},{"key":"e_1_3_2_2_28_1","unstructured":"OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774."},{"key":"e_1_3_2_2_29_1","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Long Ouyang","year":"2022","unstructured":"Long Ouyang et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730--27744.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"crossref","unstructured":"Artidoro Pagnoni Vidhisha Balachandran and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: a benchmark for factuality metrics. arXiv preprint arXiv:2104.13346.","DOI":"10.18653\/v1\/2021.naacl-main.383"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","unstructured":"Fabio Petroni Tim Rockt\u00e4schel Sebastian Riedel Patrick Lewis Anton Bakhtin YuxiangWu and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics Hong Kong China (Nov. 2019) 2463--2473. doi: 10.18653\/v1\/D19--1250.","DOI":"10.18653\/v1"},{"key":"e_1_3_2_2_32_1","unstructured":"Sundar Pichai. 2023. An important next step on our AI journey. https:\/\/blog.g oogle\/technology\/ai\/bard-google-ai-search-updates\/. Accessed: 2023--10--11. (2023)."},{"key":"e_1_3_2_2_33_1","unstructured":"Nathalie Rauschmayr et al. 2021. Amazon sagemaker debugger: a system for real-time insights into machine learning model training. In MLSys 2021. https: \/\/www.amazon.science\/publications\/amazon-sagemaker-debugger-a-system -for-real-time-insights-into-machine-learning-model-training."},{"key":"e_1_3_2_2_34_1","unstructured":"Stephen Roller et al. 2020. Open-domain conversational agents: current progress open problems and future directions. arXiv preprint arXiv:2006.12442."},{"key":"e_1_3_2_2_35_1","volume-title":"International conference on machine learning. PMLR, 3319--3328","author":"Sundararajan Mukund","year":"2017","unstructured":"Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning. PMLR, 3319--3328."},{"key":"e_1_3_2_2_36_1","volume-title":"Alpaca: a strong, replicable instruction-following model","author":"Taori Rohan","year":"2023","unstructured":"Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: a strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https:\/\/crfm. stanford. edu\/2023\/03\/13\/alpaca. html, 3, 6, 7."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12599-023-00795-x"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_3_2_2_39_1","unstructured":"Qinyuan Wu et al. 2024. Towards reliable latent knowledge estimation in llms: in-context learning vs. prompting based factual knowledge extraction. arXiv preprint arXiv:2404.12957."},{"key":"e_1_3_2_2_40_1","unstructured":"Muhammad Bilal Zafar Michele Donini Dylan Slack C\u00e9dric Archambeau Sanjiv Das and Krishnaram Kenthapadi. 2021. On the lack of robust interpretability of neural text classifiers. arXiv preprint arXiv:2106.04631."},{"key":"e_1_3_2_2_41_1","volume-title":"Proceedings of the Sixth Conference on Machine Translation, 961--972","author":"Chrysoula","unstructured":"Chrysoula Zerva et al. 2021. Ist-unbabel 2021 submission for the quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation, 961--972."},{"key":"e_1_3_2_2_42_1","unstructured":"Muru Zhang Ofir Press William Merrill Alisa Liu and Noah A Smith. 2023. Howlanguage model hallucinations can snowball. arXiv preprint arXiv:2305.13534."}],"event":{"name":"KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Barcelona Spain","acronym":"KDD '24","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671796","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637528.3671796","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:14Z","timestamp":1750291454000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671796"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,24]]},"references-count":42,"alternative-id":["10.1145\/3637528.3671796","10.1145\/3637528"],"URL":"https:\/\/doi.org\/10.1145\/3637528.3671796","relation":{},"subject":[],"published":{"date-parts":[[2024,8,24]]},"assertion":[{"value":"2024-08-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}