{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T22:56:07Z","timestamp":1782514567328,"version":"3.54.5"},"reference-count":60,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,7,2]],"date-time":"2024-07-02T00:00:00Z","timestamp":1719878400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Neurosci."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>Constructing an accurate and comprehensive knowledge graph of specific diseases is critical for practical clinical disease diagnosis and treatment, reasoning and decision support, rehabilitation, and health management. For knowledge graph construction tasks (such as named entity recognition, relation extraction), classical BERT-based methods require a large amount of training data to ensure model performance. However, real-world medical annotation data, especially disease-specific annotation samples, are very limited. In addition, existing models do not perform well in recognizing out-of-distribution entities and relations that are not seen in the training phase.<\/jats:p><\/jats:sec><jats:sec><jats:title>Method<\/jats:title><jats:p>In this study, we present a novel and practical pipeline for constructing a heart failure knowledge graph using large language models and medical expert refinement. We apply prompt engineering to the three phases of schema design: schema design, information extraction, and knowledge completion. The best performance is achieved by designing task-specific prompt templates combined with the TwoStepChat approach.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Experiments on two datasets show that the TwoStepChat method outperforms the Vanillia prompt and outperforms the fine-tuned BERT-based baselines. Moreover, our method saves 65% of the time compared to manual annotation and is better suited to extract the out-of-distribution information in the real world.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fncom.2024.1389475","type":"journal-article","created":{"date-parts":[[2024,7,2]],"date-time":"2024-07-02T04:37:05Z","timestamp":1719895025000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["Knowledge graph construction for heart failure using large language models with prompt engineering"],"prefix":"10.3389","volume":"18","author":[{"given":"Tianhan","family":"Xu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yixun","family":"Gu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mantian","family":"Xue","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renjie","family":"Gu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiang","family":"Gu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,7,2]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1186\/s40537-023-00774-9","article-title":"Healthcare knowledge graph construction: a systematic review of the state-of-the-art, open issues, and opportunities","volume":"10","author":"Abu-Salih","year":"2023","journal-title":"J. Big Data"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.08774","article-title":"Gpt-4 technical report","author":"Achiam","year":"2023","journal-title":"arXiv"},{"key":"B3","doi-asserted-by":"crossref","first-page":"1998","DOI":"10.18653\/v1\/2022.emnlp-main.130","article-title":"\u201cLarge language models are few-shot clinical information extractors,\u201d","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Agrawal","year":"2022"},{"key":"B4","first-page":"23716","article-title":"Flamingo: a visual language model for few-shot learning","volume":"35","author":"Alayrac","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-1909","article-title":"Publicly available clinical bert embeddings","author":"Alsentzer","year":"2019","journal-title":"arXiv"},{"key":"B6","doi-asserted-by":"publisher","first-page":"2705","DOI":"10.3390\/math9212705","article-title":"Performance of a novel chaotic firefly algorithm with enhanced exploration for tackling global optimization problems: application for dropout regularization","volume":"9","author":"Bacanin","year":"2021","journal-title":"Mathematics"},{"key":"B7","doi-asserted-by":"publisher","first-page":"6302","DOI":"10.1038\/s41598-022-09744-2","article-title":"Hybridized sine cosine algorithm with convolutional neural networks dropout regularization application","volume":"12","author":"Bacanin","year":"2022","journal-title":"Sci. Rep"},{"key":"B8","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B9","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1038\/s41597-023-01960-3","article-title":"Building a knowledge graph to enable precision medicine","volume":"10","author":"Chandak","year":"2023","journal-title":"Sci. Data"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2204.02311","article-title":"Palm: scaling language modeling with pathways","author":"Chowdhery","year":"2022","journal-title":"arXiv"},{"key":"B11","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ. Psychol. Meas"},{"key":"B12","article-title":"\u201cBert: pre-training of deep bidirectional transformers for language understanding,\u201d","volume-title":"North American Chapter of the Association for Computational Linguistics","author":"Devlin","year":"2019"},{"key":"B13","doi-asserted-by":"publisher","first-page":"13063","DOI":"10.5555\/3454287.3455457","article-title":"Unified language model pre-training for natural language understanding and generation","volume":"32","author":"Dong","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.03378","article-title":"Palm-e: an embodied multimodal language model","author":"Driess","year":"2023","journal-title":"arXiv"},{"key":"B15","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1016\/j.neunet.2019.08.032","article-title":"Named entity recognition in electronic health records using transfer learning bootstrapped neural networks","volume":"121","author":"Gligic","year":"2019","journal-title":"Neural Netw"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2021","journal-title":"ACM Transact. Comp. Healthc"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1072","DOI":"10.1002\/psp4.12975","article-title":"Knowledge graph aids comprehensive explanation of drug and chemical toxicity","volume":"12","author":"Hao","year":"2023","journal-title":"CPT Pharmacometr. Syst. Pharmacol"},{"key":"B18","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume":"35","author":"Kojima","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B19","doi-asserted-by":"publisher","first-page":"5678","DOI":"10.1093\/bioinformatics\/btaa1087","article-title":"BERT-GT: cross-sentence n-ary relation extraction with bert and graph transformer","volume":"36","author":"Lai","year":"2020","journal-title":"Bioinformatics"},{"key":"B20","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"Biobert: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"B21","author":"Liu","year":"2024","journal-title":"Qasystemonmedicalkg"},{"key":"B22","doi-asserted-by":"publisher","first-page":"bbac282","DOI":"10.1093\/bib\/bbac282","article-title":"Biored: a rich biomedical relation extraction dataset","volume":"23","author":"Luo","year":"2022","journal-title":"Brief. Bioinf"},{"key":"B23","doi-asserted-by":"publisher","first-page":"103384","DOI":"10.1016\/j.jbi.2020.103384","article-title":"A neural network-based joint learning approach for biomedical entity and relation extraction from biomedical literature","volume":"103","author":"Luo","year":"2020","journal-title":"J. Biomed. Inform"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3605943","article-title":"Recent advances in natural language processing via large pre-trained language models: a survey","volume":"56","author":"Min","year":"2023","journal-title":"ACM Comp. Surv"},{"key":"B25","year":"2024"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.06435","article-title":"A comprehensive overview of large language models","author":"Naveed","year":"2023","journal-title":"arXiv"},{"key":"B27","unstructured":"Chatgpt: A Large-Scale Language Model2023"},{"key":"B28","volume-title":"Best Practices for Prompt Engineering With OpenAI API","author":"Shieh","year":"2023"},{"key":"B29","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"B30","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat. Med"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2201.08239","article-title":"Lamda: language models for dialog applications","author":"Thoppilan","year":"2022","journal-title":"arXiv"},{"key":"B32","doi-asserted-by":"publisher","first-page":"8","DOI":"10.3969\/j.issn.1673-6036.2016.04.002","article-title":"Construction and application of traditional chinese medicine knowledge graph","volume":"37","author":"Tong","year":"2016","journal-title":"J. Med. Inf"},{"key":"B33","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1197\/jamia.M3115","article-title":"Recognizing obesity and comorbidities in sparse data","volume":"16","author":"Uzuner","year":"2009","journal-title":"J. Am. Med. Inform. Assoc"},{"key":"B34","doi-asserted-by":"publisher","first-page":"4052","DOI":"10.1007\/s12325-022-02254-7","article-title":"Application of medical knowledge graphs in cardiology and cardiovascular medicine: a brief literature review","volume":"39","author":"Wang","year":"","journal-title":"Adv. Ther"},{"key":"B35","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1109\/TLT.2023.3324714","article-title":"Unleashing chatgpt's power: a case study on optimizing information retrieval in flipped classrooms via prompt engineering","volume":"17","author":"Wang","year":"2024","journal-title":"IEEE Transact. Learn. Technol"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2207.00747","article-title":"Rationale-augmented ensembles in language models","author":"Wang","year":"","journal-title":"arXiv"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.10205","article-title":"Zero-shot information extraction via chatting with chatgpt","author":"Wei","year":"2023","journal-title":"arXiv"},{"key":"B38","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B39","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1038\/s41746-023-00879-8","article-title":"The shaky foundations of large language models and foundation models for electronic health records","volume":"6","author":"Wornow","year":"2023","journal-title":"npj Digit. Med"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.04437","article-title":"Structured entity extraction using large language models","author":"Wu","year":"2024","journal-title":"arXiv"},{"key":"B41","first-page":"59","article-title":"\u201cAsdKB: A Chinese knowledge base for the early screening and diagnosis of autism spectrum disorder,\u201d","author":"Wu","year":"2023","journal-title":"The Semantic Web\u2014ISWC 2023\u201422nd International Semantic Web Conference, Athens, Greece, November 6-10, 2023, Proceedings, Part II, volume 14266 of Lecture Notes in Computer Science"},{"key":"B42","doi-asserted-by":"publisher","first-page":"201","DOI":"10.26599\/BDMA.2022.9020021","article-title":"Medical knowledge graph: data sources, construction, reasoning, and applications","volume":"6","author":"Wu","year":"2023","journal-title":"Big Data Mining Anal"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-demos.15","article-title":"LingYi: medical conversational question answering system based on multi-modal knowledge graphs","author":"Xia","year":"2022","journal-title":"arXiv"},{"key":"B44","first-page":"25","article-title":"\u201cLambdaKG: a library for pre-trained language model-based knowledge graph embeddings,\u201d","volume-title":"Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations","author":"Xie","year":"2023"},{"key":"B45","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.01097","article-title":"DoctorGLM: fine-tuning your chinese doctor is not a herculean task","author":"Xiong","year":"2023","journal-title":"arXiv"},{"key":"B46","doi-asserted-by":"publisher","first-page":"4969","DOI":"10.1109\/TKDE.2022.3150080","article-title":"Knowledge graph quality management: a comprehensive survey","volume":"35","author":"Xue","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng"},{"key":"B47","doi-asserted-by":"publisher","first-page":"bbac543","DOI":"10.1093\/bib\/bbac543","article-title":"A comprehensive review on knowledge graphs for complex diseases","volume":"24","author":"Yang","year":"2023","journal-title":"Brief. Bioinf"},{"key":"B48","doi-asserted-by":"publisher","first-page":"8693937","DOI":"10.1155\/2022\/8693937","article-title":"Decision-making system for the diagnosis of syndrome based on traditional chinese medicine knowledge graph","volume":"2022","author":"Yang","year":"2022","journal-title":"Evid. Based Comp. Altern Med"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.06794","article-title":"Cognitive mirage: a review of hallucinations in large language models","author":"Ye","year":"2023","journal-title":"arXiv"},{"key":"B50","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1016\/j.artmed.2017.04.001","article-title":"Knowledge graph for tcm health preservation: Design, construction, and applications","volume":"77","author":"Yu","year":"2017","journal-title":"Artif. Intell. Med"},{"key":"B51","doi-asserted-by":"publisher","first-page":"2219","DOI":"10.3778\/j.issn.1673-9418.2112118","article-title":"Research and application progress of chinese medical knowledge graph","volume":"16","author":"Yuanyuan","year":"2022","journal-title":"J. Front. Comp. Sci. Technol"},{"key":"B52","first-page":"15476","article-title":"Star: Bootstrapping reasoning with reasoning","volume":"35","author":"Zelikman","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-demo.1","article-title":"Human-in-the-loop schema induction","author":"Zhang","year":"2023","journal-title":"arXiv"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.06671","article-title":"Making large language models perform better in knowledge graph completion","author":"Zhang","year":"2023","journal-title":"arXiv"},{"key":"B55","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.18223","article-title":"A survey of large language models","author":"Zhao","year":"2023","journal-title":"arXiv"},{"key":"B56","doi-asserted-by":"crossref","first-page":"560","DOI":"10.1109\/ICBK50248.2020.00084","article-title":"\u201cTCMKG: a deep learning based traditional chinese medicine knowledge graph platform,\u201d","volume-title":"2020 IEEE International Conference on Knowledge Graph (ICKG)","author":"Zheng","year":"2020"},{"key":"B57","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2205.10625","article-title":"Least-to-most prompting enables complex reasoning in large language models","author":"Zhou","year":"2022","journal-title":"arXiv"},{"key":"B58","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.03279","article-title":"Universalner: targeted distillation from large language models for open named entity recognition","author":"Zhou","year":"2023","journal-title":"arXiv"},{"key":"B59","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.13168","article-title":"LLMs for knowledge graph construction and reasoning: Recent capabilities and future opportunities","author":"Zhu","year":"2023","journal-title":"arXiv"},{"key":"B60","doi-asserted-by":"publisher","first-page":"3798","DOI":"10.3390\/electronics11223798","article-title":"Hybrid cnn and XGBoost model tuned by modified arithmetic optimization algorithm for Covid-19 early diagnostics from x-ray images","volume":"11","author":"Zivkovic","year":"2022","journal-title":"Electronics"}],"container-title":["Frontiers in Computational Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2024.1389475\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,2]],"date-time":"2024-07-02T04:37:32Z","timestamp":1719895052000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2024.1389475\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,2]]},"references-count":60,"alternative-id":["10.3389\/fncom.2024.1389475"],"URL":"https:\/\/doi.org\/10.3389\/fncom.2024.1389475","relation":{},"ISSN":["1662-5188"],"issn-type":[{"value":"1662-5188","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,2]]},"article-number":"1389475"}}