{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T13:10:35Z","timestamp":1790082635337,"version":"4.0.1"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2025,8,14]],"date-time":"2025-08-14T00:00:00Z","timestamp":1755129600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,8,14]],"date-time":"2025-08-14T00:00:00Z","timestamp":1755129600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100014840","name":"Gemeinsame Bundesausschuss","doi-asserted-by":"publisher","award":["01NVF21116"],"award-info":[{"award-number":["01NVF21116"]}],"id":[{"id":"10.13039\/501100014840","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Albert-Ludwigs-Universit\u00e4t Freiburg im Breisgau"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J CARS"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>The goal of our work is to develop a multi-view validation framework for evaluating LLM-generated knowledge graph (KG) triples. The proposed approach aims to address the lack of established validation procedure in the context of LLM-supported KG construction.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>The proposed framework evaluates the LLM-generated triples across three dimensions: semantic plausibility, ontology-grounded type compatibility, and structural importance. We demonstrate the performance for GPT-4 generated concept-specific (e.g., for medications, diagnosis, procedures) triples in the context of chronic kidney disease (CKD).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The proposed approach consistently achieves high-quality results across evaluated GPT-4 generated triples, strong semantic plausibility (semantic score mean: 0.79), excellent type compatibility (type score mean: 0.84), and high structural importance of entities within the CKD knowledge domain (ResourceRank mean: 0.94).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The validation framework offers a reliable and scalable method for evaluating quality and validity of LLM-generated triples across three views: semantic plausibility, type compatibility, and structural importance. The framework demonstrates robust performance in filtering high-quality triples and lays a strong foundation for fast and reliable medical KG construction and validation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s11548-025-03495-x","type":"journal-article","created":{"date-parts":[[2025,8,14]],"date-time":"2025-08-14T07:12:36Z","timestamp":1755155556000},"page":"2523-2528","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A multi-view validation framework for LLM-generated knowledge graphs of chronic kidney disease"],"prefix":"10.1007","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-0325-2505","authenticated-orcid":false,"given":"Aditya","family":"Kumar","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dilpreet","family":"Singh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mario","family":"Cypko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oliver","family":"Amft","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,8,14]]},"reference":[{"issue":"1","key":"3495_CR1","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1186\/s40537-023-00774-9","volume":"10","author":"B Abu-Salih","year":"2023","unstructured":"Abu-Salih B, Al-Qurishi M, Alweshah M, Al-Smadi M, Alfayez R, Saadeh H (2023) Healthcare knowledge graph construction: a systematic review of the state-of-the-art, open issues, and opportunities. J Big Data 10(1):81","journal-title":"J Big Data"},{"issue":"8","key":"3495_CR2","doi-asserted-by":"publisher","first-page":"396","DOI":"10.3390\/info13080396","volume":"13","author":"M Zamini","year":"2022","unstructured":"Zamini M, Reza H, Rabiei M (2022) A review of knowledge graph completion. Information 13(8):396","journal-title":"Information"},{"key":"3495_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/2193-8636-1-8","volume":"1","author":"L Piovesan","year":"2014","unstructured":"Piovesan L, Molino G, Terenziani P (2014) An ontological knowledge and multiple abstraction level decision support system in healthcare. Decis Anal 1:1\u201324","journal-title":"Decis Anal"},{"key":"3495_CR4","doi-asserted-by":"crossref","unstructured":"Gyrard A, Jaimini U, Gaur M, Shekharpour S, Thirunarayan K, Sheth A (2022) Reasoning over personalized healthcare knowledge graph: a case study of patients with allergies and symptoms. In:\u00a0Semantic models in IoT and Ehealth applications. Academic Press, pp 199\u2013225","DOI":"10.1016\/B978-0-32-391773-5.00016-9"},{"issue":"01","key":"3495_CR5","first-page":"606","volume":"34","author":"E Choi","year":"2020","unstructured":"Choi E, Xu Z, Li Y, Dusenberry M, Flores G, Xue E, Dai A (2020) Learning the graphical structure of electronic health records with graph convolutional transformer. Proc AAAI Conf Artif Intell 34(01):606\u2013613","journal-title":"Proc AAAI Conf Artif Intell"},{"key":"3495_CR6","unstructured":"Choi E, Xiao C, Stewart W, Sun J (2018) Mime: multilevel medical embedding of electronic health records for predictive healthcare.\u00a0Advances in neural information processing systems,\u00a031, 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada. Paper URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2018\/file\/934b535800b1cba8f96a5d72f72f1611-Paper.pdf"},{"key":"3495_CR7","doi-asserted-by":"crossref","unstructured":"Choi E, Bahadori MT, Song L, Stewart WF, Sun J (2017) GRAM: graph-based attention model for healthcare representation learning. In:\u00a0Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 787\u2013795","DOI":"10.1145\/3097983.3098126"},{"issue":"Suppl_1","key":"3495_CR8","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","volume":"32","author":"O Bodenreider","year":"2004","unstructured":"Bodenreider O (2004) The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res 32(Suppl_1):D267\u2013D270","journal-title":"Nucleic Acids Res"},{"key":"3495_CR9","first-page":"279","volume":"121","author":"K Donnelly","year":"2006","unstructured":"Donnelly K (2006) SNOMED-CT: the advanced terminology and coding system for eHealth. Stud Health Technol Inform 121:279","journal-title":"Stud Health Technol Inform"},{"issue":"8","key":"3495_CR10","doi-asserted-by":"publisher","first-page":"509","DOI":"10.3390\/info15080509","volume":"15","author":"M Hofer","year":"2024","unstructured":"Hofer M, Obraczka D, Saeedi A, K\u00f6pcke H, Rahm E (2024) Construction of knowledge graphs: current state and challenges. Information 15(8):509","journal-title":"Information"},{"key":"3495_CR11","doi-asserted-by":"crossref","unstructured":"Kim E, Shrestha M, Foty R, DeLay T, Seyfert-Margolis V (2024) Structured extraction of real world medical knowledge using LLMs for summarization and search.\u00a0arXiv preprint arXiv:2412.15256","DOI":"10.1109\/BigData62323.2024.10825160"},{"key":"3495_CR12","unstructured":"Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, Almeida D, Altenschmidt J, Altman S, Anadkat S, Avila R (2023) Gpt-4 technical report.\u00a0arXiv preprint arXiv:2303.08774"},{"key":"3495_CR13","unstructured":"Jiang P, Xiao C, Cross A, Sun J (2023) Graphcare: enhancing healthcare predictions with personalized knowledge graphs. arXiv preprint arXiv:2305.12788"},{"key":"3495_CR14","doi-asserted-by":"crossref","unstructured":"Lv X, Lin Y, Cao Y, Hou L, Li J, Liu Z, Li P, Zhou J (2022) Do pre-trained models benefit knowledge graph completion? A reliable evaluation and a reasonable approach. Findings of the Association for Computational Linguistics: ACL 2022, pp. 3570\u20133581. Paper URL: https:\/\/ink.library.smu.edu.sg\/cgi\/viewcontent.cgi?article=8449&context=sis_research","DOI":"10.18653\/v1\/2022.findings-acl.282"},{"key":"3495_CR15","doi-asserted-by":"crossref","unstructured":"Petroni F, Rockt\u00e4schel T, Lewis P, Bakhtin A, Wu Y, Miller AH, Riedel S (2019) Language models as knowledge bases? arXiv preprint arXiv:1909.01066","DOI":"10.18653\/v1\/D19-1250"},{"key":"3495_CR16","unstructured":"AlKhamissi B, Li M, Celikyilmaz A, Diab M, Ghazvininejad M (2022) A review on language models as knowledge bases.\u00a0arXiv preprint arXiv:2204.06031"},{"key":"3495_CR17","doi-asserted-by":"crossref","unstructured":"Arsenyan V, Bughdaryan S, Shaya F, Small K, Shahnazaryan D (2023) Large language models for biomedical knowledge graph construction: information extraction from EMR notes.\u00a0arXiv preprint arXiv:2301.12473","DOI":"10.18653\/v1\/2024.bionlp-1.23"},{"key":"3495_CR18","doi-asserted-by":"publisher","DOI":"10.2196\/65537","volume":"27","author":"H Yang","year":"2025","unstructured":"Yang H, Li J, Zhang C, Sierra AP, Shen B (2025) Large language model-driven knowledge graph construction in sepsis care using multicenter clinical databases: development and usability study. J Med Internet Res 27:e65537","journal-title":"J Med Internet Res"},{"key":"3495_CR19","doi-asserted-by":"crossref","unstructured":"Diaz Gonzalez AD, Hughes KS, Yue S, Hayes ST (2023) Applying BioBERT to extract germline gene-disease associations for building a knowledge graph from the biomedical literature. In:\u00a0Proceedings of the 2023 7th international conference on information system and data mining, pp 37\u201342","DOI":"10.1145\/3603765.3603771"},{"key":"3495_CR20","doi-asserted-by":"crossref","unstructured":"Yuan Y, Xu Y, Zhang W (2024) Is large language model good at triple set prediction? An empirical study. arXiv preprint arXiv:2412.18443","DOI":"10.1109\/ICKG63256.2024.00066"},{"issue":"1","key":"3495_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41597-022-01899-x","volume":"10","author":"AE Johnson","year":"2023","unstructured":"Johnson AE, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, Pollard TJ, Hao S, Moody B, Gow B, Lehman LWH (2023) MIMIC-IV, a freely accessible electronic health record dataset. Sci Data 10(1):1","journal-title":"Sci Data"},{"key":"3495_CR22","unstructured":"Gupta M, Gallamoza B, Cutrona N, Dhakal P, Poulain R, Beheshti R (2022) An extensive data processing pipeline for mimic-iv. In: Machine learning for health. PMLR, pp 311\u2013325"},{"key":"3495_CR23","doi-asserted-by":"crossref","unstructured":"Jia S, Xiang Y, Chen X, Wang K (2019) Triple trustworthiness measurement for knowledge graph. In: The World Wide Web conference, pp 2865\u20132871","DOI":"10.1145\/3308558.3313586"},{"issue":"1","key":"3495_CR24","first-page":"1","volume":"3","author":"Y Gu","year":"2021","unstructured":"Gu Y, Tinn R, Cheng H, Lucas M, Usuyama N, Liu X, Naumann T, Gao J, Poon H (2021) Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comput Healthc (HEALTH) 3(1):1\u201323","journal-title":"ACM Trans Comput Healthc (HEALTH)"},{"key":"3495_CR25","doi-asserted-by":"crossref","unstructured":"Liu S, d\u2019Aquin M, Motta E (2017) Measuring accuracy of triples in knowledge graphs. In: Language, data, and knowledge: first international conference, LDK 2017, Galway, Ireland, June 19\u201320, 2017, Proceedings 1. Springer, pp 343\u2013357","DOI":"10.1007\/978-3-319-59888-8_29"},{"key":"3495_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12882-018-0882-9","volume":"19","author":"NI Cole","year":"2018","unstructured":"Cole NI, Liyanage H, Suckling RJ, Swift PA, Gallagher H, Byford R, Williams J, Kumar S, de Lusignan S (2018) An ontological approach to identifying cases of chronic kidney disease from routine primary care data: a cross-sectional study. BMC Nephrol 19:1\u20136","journal-title":"BMC Nephrol"},{"key":"3495_CR27","doi-asserted-by":"crossref","unstructured":"Varma S, Shivam S, Thumu A, Bhushanam A, Sarkar D (2022) Jaccard based similarity index in graphs: a multi-hop approach. In: 2022 IEEE Delhi section conference (DELCON). IEEE, pp 1\u20134","DOI":"10.1109\/DELCON54057.2022.9753316"},{"issue":"3","key":"3495_CR28","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1109\/51.932728","volume":"20","author":"GB Moody","year":"2001","unstructured":"Moody GB, Mark RG, Goldberger AL (2001) PhysioNet: a web-based resource for the study of physiologic signals. IEEE Eng Med Biol Mag 20(3):70\u201375","journal-title":"IEEE Eng Med Biol Mag"}],"container-title":["International Journal of Computer Assisted Radiology and Surgery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-025-03495-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11548-025-03495-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11548-025-03495-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,9]],"date-time":"2025-12-09T13:39:36Z","timestamp":1765287576000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11548-025-03495-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,14]]},"references-count":28,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["3495"],"URL":"https:\/\/doi.org\/10.1007\/s11548-025-03495-x","relation":{},"ISSN":["1861-6429"],"issn-type":[{"value":"1861-6429","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,14]]},"assertion":[{"value":"20 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 July 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 August 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"For MIMIC-IV, the collection of patient information and creation of the research resource was reviewed by the Institutional Review Board at the Beth Israel Deaconess Medical Center, who granted a waiver of informed consent and approved the data sharing initiative.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical statement"}}]}}