{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T00:46:18Z","timestamp":1788482778303,"version":"build-2803163510"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2025,2,10]],"date-time":"2025-02-10T00:00:00Z","timestamp":1739145600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2025,2,10]]},"abstract":"<jats:p>\n                    Dialogue benchmarks are crucial in training and evaluating chatbots engaging in domain-specific conversations. Knowledge graphs (KGs) represent semantically rich and well-organized data spanning various domains, such as DBLP, DBpedia, and YAGO. Traditionally, dialogue benchmarks have been manually created from documents, neglecting the potential of KGs in automating this process. Some question-answering benchmarks are automatically generated using extensive preprocessing from KGs, but they do not support dialogue generation. This paper introduces Chatty-Gen, a novel multi-stage retrieval-augmented generation platform for automatically generating high-quality dialogue benchmarks tailored to a specific domain using a KG. Chatty-Gen decomposes the generation process into manageable stages and uses assertion rules for automatic validation between stages. Our approach enables control over intermediate results to prevent time-consuming restarts due to hallucinations. It also reduces reliance on costly and more powerful commercial LLMs. Chatty-Gen eliminates upfront processing of the entire KG using efficient query-based retrieval to find representative subgraphs based on the dialogue context. Our experiments with several real and large KGs demonstrate that C\n                    <jats:sc>hatty<\/jats:sc>\n                    -G\n                    <jats:sc>en<\/jats:sc>\n                    significantly outperforms state-of-the-art systems and ensures consistent model and system performance across multiple LLMs of diverse capabilities, such as GPT-4o, Gemini 1.5, Llama 3, and Mistral.\n                  <\/jats:p>","DOI":"10.1145\/3709681","type":"journal-article","created":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T15:45:06Z","timestamp":1739288706000},"page":"1-26","source":"Crossref","is-referenced-by-count":5,"title":["Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6233-2092","authenticated-orcid":false,"given":"Reham","family":"Omar","sequence":"first","affiliation":[{"name":"Concordia University, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2364-3511","authenticated-orcid":false,"given":"Omij","family":"Mangukiya","sequence":"additional","affiliation":[{"name":"Concordia University, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6851-6351","authenticated-orcid":false,"given":"Essam","family":"Mansour","sequence":"additional","affiliation":[{"name":"Concordia University, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,2,11]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"AI@Meta. 2024. Llama 3 Model Card. (2024). https:\/\/github.com\/meta-llama\/llama3\/blob\/main\/MODEL_CARD.md"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the International Workshop on Bibliometric-enhanced Information Retrieval","volume":"3617","author":"Banerjee Debayan","year":"2023","unstructured":"Debayan Banerjee, Sushil Awale, Ricardo Usbeck, and Chris Biemann. 2023. DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph. In Proceedings of the International Workshop on Bibliometric-enhanced Information Retrieval, Vol. 3617. 37--51. https:\/\/ceur-ws.org\/Vol-3617\/paper-05.pdf"},{"key":"e_1_2_1_3_1","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan and et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS). https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i16.29728"},{"key":"e_1_2_1_5_1","volume-title":"Premise Order Matters in Reasoning with Large Language Models. In International Conference on Machine Learning, ICML. https:\/\/openreview.net\/forum?id=4zAHgkiCQg","author":"Chen Xinyun","year":"2024","unstructured":"Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. 2024a. Premise Order Matters in Reasoning with Large Language Models. In International Conference on Machine Learning, ICML. https:\/\/openreview.net\/forum?id=4zAHgkiCQg"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D18--1241nolinkurl10.18653\/V1\/D18--1241"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358016"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531815nolinkurl10.1145\/3477495.3531815"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.10053"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/N19--1423nolinkurl10.18653\/V1\/N19--1423"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30796-7_5"},{"key":"e_1_2_1_12_1","volume-title":"The International Conference on Learning Representations, ICLR. https:\/\/openreview.net\/forum?id=IuXR1CCrSi","author":"Fatemi Bahare","year":"2024","unstructured":"Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. 2024. Talk like a Graph: Encoding Graphs for Large Language Models. In The International Conference on Learning Representations, ICLR. https:\/\/openreview.net\/forum?id=IuXR1CCrSi"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2312.10997nolinkurl10.48550\/ARXIV.2312.10997"},{"key":"e_1_2_1_14_1","unstructured":"Google. 2023. https:\/\/gemini.google.com."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-88361-4_8"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2310.06825nolinkurl10.48550\/ARXIV.2310.06825"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","unstructured":"Yohan Jo Xinyan Zhao Arijit Biswas Nikoletta Basiou Vincent Auvray and et al. 2023. Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users. In Findings of the Association for Computational Linguistics: (EMNLP). 3237--3269. hrefhttps:\/\/doi.org\/10.18653\/V1\/2023.FINDINGS-EMNLP.213doi:nolinkurl10.18653\/V1\/2023.FINDINGS-EMNLP.213","DOI":"10.18653\/V1\/2023.FINDINGS-EMNLP.213nolinkurl10.18653\/V1\/2023.FINDINGS-EMNLP.213"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.EMNLP-MAIN.280nolinkurl10.18653\/V1\/2020.EMNLP-MAIN.280"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3618260.3649777nolinkurl10.1145\/3618260.3649777"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.FINDINGS-ACL.339nolinkurl10.18653\/V1\/2021.FINDINGS-ACL.339"},{"key":"e_1_2_1_21_1","volume-title":"Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems : Annual Conference on Neural Information Processing Systems (NeurIPS). http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.ACL-DEMO.18nolinkurl10.18653\/V1\/2023.ACL-DEMO.18"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3233\/SW-140134nolinkurl10.3233\/SW-140134"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2024.ACL-LONG.471nolinkurl10.18653\/V1\/2024.ACL-LONG.471"},{"key":"e_1_2_1_25_1","volume-title":"Yuyao Wang, and Lingming Zhang.","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS). http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3588911"},{"key":"e_1_2_1_27_1","unstructured":"OpenAI. 2022. https:\/\/openai.com\/blog\/chatgpt."},{"key":"e_1_2_1_28_1","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023). hrefhttps:\/\/doi.org\/10.48550\/ARXIV.2303.08774doi:nolinkurl10.48550\/ARXIV.2303.08774"},{"key":"e_1_2_1_29_1","volume-title":"https:\/\/openai.com\/index\/hello-gpt-4o\/","author":"AI.","year":"2024","unstructured":"OpenAI. 2024. GPT-4o. (2024). https:\/\/openai.com\/index\/hello-gpt-4o\/"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3589322"},{"key":"e_1_2_1_31_1","unstructured":"Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll L. Wainwright and et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS). http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/b1efde53be364a73914f58805a001731-Abstract-Conference.html"},{"key":"e_1_2_1_32_1","unstructured":"Aaron Pham Chaoyu Yang Sean Sheng Shenyang Zhao Sauyon Lee Bo Jiang Fog Dong Xipeng Guan and Frost Ming. 2023. OpenLLM: Operating LLMs in production. https:\/\/github.com\/bentoml\/OpenLLM"},{"key":"e_1_2_1_33_1","unstructured":"Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS). http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/72223cc66f63ca1aa59edaec1b3670e6-Abstract-Conference.html"},{"key":"e_1_2_1_34_1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, Vol. 21 (2020), 140:1--140:67. http:\/\/jmlr.org\/papers\/v21\/20-074.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00266nolinkurl10.1162\/TACL_A_00266"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2308.12950nolinkurl10.48550\/ARXIV.2308.12950"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1609\/AAAI.V32I1.11332nolinkurl10.1609\/AAAI.V32I1.11332"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2024.NAACL-LONG.18nolinkurl10.18653\/V1\/2024.NAACL-LONG.18"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00264nolinkurl10.1162\/TACL_A_00264"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.EMNLP-MAIN.327nolinkurl10.18653\/V1\/2023.EMNLP-MAIN.327"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.11805"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","unstructured":"Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. https:\/\/doi.org\/10.48550\/arXiv.2403.05530","DOI":"10.48550\/arXiv.2403.05530"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux and et al. 2023a. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv 2302.13971 (2023). hrefhttps:\/\/doi.org\/10.48550\/ARXIV.2302.13971doi:nolinkurl10.48550\/ARXIV.2302.13971","DOI":"10.48550\/ARXIV.2302.13971nolinkurl10.48550\/ARXIV.2302.13971"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi and et al. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv 2307.09288 (2023). hrefhttps:\/\/doi.org\/10.48550\/ARXIV.2307.09288doi:nolinkurl10.48550\/ARXIV.2307.09288","DOI":"10.48550\/ARXIV.2307.09288nolinkurl10.48550\/ARXIV.2307.09288"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-68204-4_22"},{"key":"e_1_2_1_46_1","unstructured":"Ricardo Usbeck Ria Gusmita Axel-Cyrille Ngomo and Muhammad Saleem. 2018. 9th Challenge on Question Answering over Linked Data (QALD-9). In Joint proceedings of the Workshop on Semantic Deep Learning (SemDeep-4) and NLIWoD4: Natural Language Interfaces for the Web of Data (NLIWOD-4) and 9th Question Answering over Linked Data challenge (QALD-9) co-located with International Semantic Web Conference (ISWC) Vol. 2241. 58--64. http:\/\/ceur-ws.org\/Vol-2241\/paper-06.pdf"},{"key":"e_1_2_1_47_1","unstructured":"Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter and et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS). http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","unstructured":"Xiao Yang Kai Sun Hao Xin Yushi Sun Nikita Bhalla and et al. 2024. CRAG - Comprehensive RAG Benchmark. arXiv preprint arXiv:2406.04744 (2024). hrefhttps:\/\/doi.org\/10.48550\/ARXIV.2406.04744doi:nolinkurl10.48550\/ARXIV.2406.04744","DOI":"10.48550\/ARXIV.2406.04744nolinkurl10.48550\/ARXIV.2406.04744"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P16--2033nolinkurl10.18653\/V1\/P16--2033"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.ACL-DEMO.58nolinkurl10.18653\/V1\/2023.ACL-DEMO.58"},{"key":"e_1_2_1_51_1","unstructured":"Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric P. Xing Hao Zhang Joseph E. Gonzalez and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS). http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i19.30185"},{"key":"e_1_2_1_53_1","volume-title":"Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The International Conference on Learning Representations, (ICLR). https:\/\/openreview.net\/pdf?id=WZH7099tgfM","author":"Zhou Denny","unstructured":"Denny Zhou, Nathanael Sch\u00e4rli, Le Hou, Jason Wei, Nathan Scales, and et al. 2023b. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In The International Conference on Learning Representations, (ICLR). https:\/\/openreview.net\/pdf?id=WZH7099tgfM"},{"key":"e_1_2_1_54_1","volume-title":"The International Conference on Learning Representations (ICLR). https:\/\/openreview.net\/pdf?id=92gvk82DE-","author":"Zhou Yongchao","unstructured":"Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, and et al. 2023a. Large Language Models are Human-Level Prompt Engineers. In The International Conference on Learning Representations (ICLR). https:\/\/openreview.net\/pdf?id=92gvk82DE-"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3709681","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3709681","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T18:23:08Z","timestamp":1774981388000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3709681"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,10]]},"references-count":54,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,2,10]]}},"alternative-id":["10.1145\/3709681"],"URL":"https:\/\/doi.org\/10.1145\/3709681","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,10]]}}}