{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,23]],"date-time":"2025-11-23T08:21:22Z","timestamp":1763886082317,"version":"3.45.0"},"reference-count":0,"publisher":"Association for the Advancement of Artificial Intelligence (AAAI)","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AAAI-SS"],"abstract":"<jats:p>Generative AI (GEN AI) models have revolutionized diverse\napplication domains but present substantial challenges due\nto reliability concerns, including hallucinations, semantic\ndrift, and inherent biases. These models typically operate\nas black-boxes, complicating transparent and objective\nevaluation. Current evaluation methods primarily depend on\nsubjective human assessment, limiting scalability,\ntransparency, and effectiveness. This research proposes a\nsystematic methodology using deterministic and Large\nLanguage Model (LLM)-generated Knowledge Graphs (KGs) to\ncontinuously monitor and evaluate GEN AI reliability. We\nconstruct two parallel KGs: a deterministic KG built using\nexplicit rule-based methods, predefined ontologies,\ndomain-specific dictionaries, and structured\nentity-relation extraction rules; and an LLM-generated KG\ndynamically derived from real-time textual data streams\nsuch as live news articles. Utilizing real-time news\nstreams ensures authenticity, mitigates biases from\nrepetitive training, and prevents adaptive LLMs from\nbypassing predefined benchmarks through feedback\nmemorization. To quantify structural deviations and\nsemantic discrepancies, we employ several established KG\nmetrics including Instantiated Class Ratio (ICR),\nInstantiated Property Ratio (IPR), and Class Instantiation\n(CI). These metrics systematically evaluate critical\nstructural properties, including class and property\ninstantiation ratios, class depth and complexity, and\ninheritance patterns. An automated real-time monitoring\nframework continuously computes deviations between\ndeterministic and LLM-generated KGs. By establishing\ndynamic anomaly thresholds based on historical structural\nmetric distributions, our method proactively identifies and\nflags significant deviations, thus promptly detecting\nsemantic anomalies or hallucinations. This structured,\nmetric-driven comparison between deterministic and\ndynamically generated KGs delivers a robust and scalable\nevaluation framework. A demo website is currently live at (\nanonymous ).<\/jats:p>","DOI":"10.1609\/aaaiss.v7i1.36883","type":"journal-article","created":{"date-parts":[[2025,11,23]],"date-time":"2025-11-23T08:17:23Z","timestamp":1763885843000},"page":"169-176","source":"Crossref","is-referenced-by-count":0,"title":["Continuous Monitoring of Large-Scale Generative AI via\nDeterministic Knowledge Graph Structures"],"prefix":"10.1609","volume":"7","author":[{"given":"Kishor Datta","family":"Gupta","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohd Ariful","family":"Haque","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hasmot","family":"Ali","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marufa","family":"Kamal","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Syed Bahauddin","family":"Alam","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammad Ashiqur","family":"Rahman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"9382","published-online":{"date-parts":[[2025,11,23]]},"container-title":["Proceedings of the AAAI Symposium Series"],"original-title":[],"link":[{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI-SS\/article\/download\/36883\/39021","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI-SS\/article\/download\/36883\/39021","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,23]],"date-time":"2025-11-23T08:17:23Z","timestamp":1763885843000},"score":1,"resource":{"primary":{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI-SS\/article\/view\/36883"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,23]]},"references-count":0,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,11,23]]}},"URL":"https:\/\/doi.org\/10.1609\/aaaiss.v7i1.36883","relation":{},"ISSN":["2994-4317"],"issn-type":[{"value":"2994-4317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,23]]}}}