{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:18:17Z","timestamp":1776082697911,"version":"3.50.1"},"reference-count":41,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2024,10,23]],"date-time":"2024-10-23T00:00:00Z","timestamp":1729641600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["52305033"],"award-info":[{"award-number":["52305033"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Zhongnan University of Economics and Law and Beijing Borui Tongyun Technology Co., Ltd.","award":["52305033"],"award-info":[{"award-number":["52305033"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Legal knowledge involves multidimensional heterogeneous knowledge such as legal provisions, judicial interpretations, judicial cases, and defenses, which requires extremely high relevance and accuracy of knowledge. Meanwhile, the construction of a legal knowledge reasoning system also faces challenges in obtaining, processing, and sharing multisource heterogeneous knowledge. The knowledge graph technology, which is a knowledge organization form with triples as the basic unit, is able to efficiently transform multisource heterogeneous information into a knowledge representation form close to human cognition. Taking the automated construction of the Chinese legal knowledge graph (CLKG) as a case scenario, this paper presents a joint knowledge enhancement model (JKEM), where prior knowledge is embedded into a large language model (LLM), and the LLM is fine-tuned through the prefix of the prior knowledge data. Under the condition of freezing most parameters of the LLM, this fine-tuning scheme adds continuous deep prompts as prefix tokens to the input sequences of different layers, which can significantly improve the accuracy of knowledge extraction. The results show that the knowledge extraction accuracy of the JKEM in this paper reaches 90.92%. Based on the superior performance of this model, the CLKG is further constructed, which contains 3480 knowledge triples composed of 9 entities and 2 relationships, providing strong support for an in-depth understanding of the complex relationships in the legal field.<\/jats:p>","DOI":"10.3390\/info15110666","type":"journal-article","created":{"date-parts":[[2024,10,23]],"date-time":"2024-10-23T12:04:22Z","timestamp":1729685062000},"page":"666","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Construction of Legal Knowledge Graph Based on Knowledge-Enhanced Large Language Models"],"prefix":"10.3390","volume":"15","author":[{"given":"Jun","family":"Li","sequence":"first","affiliation":[{"name":"Criminal Justice School, Zhongnan University of Economics and Law, Wuhan 430064, China"}]},{"given":"Lu","family":"Qian","sequence":"additional","affiliation":[{"name":"The School of Transportation and Logistics Engineering, Wuhan University of Technology (WHUT), Wuhan 430063, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5589-1662","authenticated-orcid":false,"given":"Peifeng","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute for Innovation and Development, Tsinghua University, Beijing 100084, China"}]},{"given":"Taoxiong","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute for Innovation and Development, Tsinghua University, Beijing 100084, China"}]}],"member":"1968","published-online":{"date-parts":[[2024,10,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1109\/TNNLS.2021.3070843","article-title":"A Survey on Knowledge Graphs: Representation, Acquisition, and Applications","volume":"33","author":"Ji","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kejriwal, M. (2022). Knowledge Graphs: A Practical Review of the Research Landscape. Information, 13.","DOI":"10.3390\/info13040161"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"8160","DOI":"10.1109\/TII.2024.3366977","article-title":"Joint Knowledge Graph and Large Language Model for Fault Diagnosis and Its Application in Aviation Assembly","volume":"20","author":"Liu","year":"2024","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/2629489","article-title":"Wikidata: A Free Collaborative Knowledgebase","volume":"57","year":"2014","journal-title":"Commun. ACM"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"167","DOI":"10.3233\/SW-140134","article-title":"DBpedia\u2014A large-scale, multilingual knowledge base extracted from Wikipedia","volume":"6","author":"Lehmann","year":"2015","journal-title":"Semant. Web"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"26483","DOI":"10.1109\/ACCESS.2023.3254132","article-title":"The Construction of Knowledge Graphs in the Aviation Assembly Domain Based on a Joint Knowledge Extraction Model","volume":"11","author":"Liu","year":"2023","journal-title":"IEEE Access"},{"key":"ref_7","unstructured":"Hubauer, T., Lamparter, S., Haase, P., and Herzig, D.M. (2018, January 8\u201312). Use Cases of the Industrial Knowledge Graph at Siemens. Proceedings of the Semantic Web\u2014ISWC 2018, Cham, Switzerland."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"32862","DOI":"10.1109\/ACCESS.2020.2973928","article-title":"Named Entity Extraction for Knowledge Graphs: A Literature Overview","volume":"8","author":"Opdahl","year":"2020","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1109\/TKDE.2020.2981314","article-title":"A Survey on Deep Learning for Named Entity Recognition","volume":"34","author":"Li","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"102265","DOI":"10.1016\/j.datak.2023.102265","article-title":"Towards deep understanding of graph convolutional networks for relation extraction","volume":"149","author":"Wu","year":"2024","journal-title":"Data Knowl. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.neucom.2016.12.075","article-title":"Joint entity and relation extraction based on a hybrid neural network","volume":"257","author":"Zheng","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"87571","DOI":"10.1109\/ACCESS.2023.3305259","article-title":"Knowledge-Enriched Multi-Cross Attention Network for Legal Judgment Prediction","volume":"11","author":"He","year":"2023","journal-title":"IEEE Access"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Vuong, T.H.Y., Hoang, M.Q., Nguyen, T.M., Nguyen, H.T., and Nguyen, H.T. (2023, January 18\u201320). Constructing a Knowledge Graph for Vietnamese Legal Cases with Heterogeneous Graphs. Proceedings of the 2023 15th International Conference on Knowledge and Systems Engineering (KSE), Hanoi, Vietnam.","DOI":"10.1109\/KSE59128.2023.10299526"},{"key":"ref_14","unstructured":"State Council Legislative Affairs Office (Compiler) (2018). Criminal Law Code of the People\u2019s Republic of China: Annotated Edition (Fourth New Edition), China Legal Publishing House. Annotated Edition."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101981","DOI":"10.1016\/j.is.2022.101981","article-title":"Managing, Mining and Learning in the Legal Data Domain","volume":"106","author":"Tagarelli","year":"2022","journal-title":"Inf. Syst."},{"key":"ref_16","first-page":"242","article-title":"Developing Artificially Intelligent Justice","volume":"22","author":"Re","year":"2019","journal-title":"Stanf. Technol. Law Rev."},{"key":"ref_17","first-page":"501","article-title":"Can Robots Be Lawyers? Computers, Lawyers, and the Practice of Law","volume":"30","author":"Remus","year":"2015","journal-title":"Georget. J. Leg. Ethics"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Yao, S., Ke, Q., Wang, Q., Li, K., and Hu, J. (2024, January 5\u20137). Lawyer GPT: A Legal Large Language Model with Enhanced Domain Knowledge and Reasoning Capabilities. Proceedings of the 2024 3rd International Symposium on Robotics, Artificial Intelligence and Information Engineering (RAIIE \u201924), Singapore.","DOI":"10.1145\/3689299.3689319"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Savelka, J. (2023, January 19\u201323). Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts. Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (ICAIL \u201923), Braga, Portugal.","DOI":"10.1145\/3594536.3595161"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Ammar, A., Koubaa, A., Benjdira, B., Nacar, O., and Sibaee, S. (2024). Prediction of Arabic Legal Rulings Using Large Language Models. Electronics, 13.","DOI":"10.3390\/electronics13040764"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Licari, D., Bushipaka, P., Marino, G., Comand\u00e9, G., and Cucinotta, T. (2023, January 19\u201323). Legal Holding Extraction from Italian Case Documents using Italian-LEGAL-BERT Text Summarization. Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (ICAIL \u201923), Braga, Portugal.","DOI":"10.1145\/3594536.3595177"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"101966","DOI":"10.1016\/j.is.2021.101966","article-title":"Lynx: A knowledge-based AI service platform for content processing, enrichment and analysis for the legal domain","volume":"106","author":"Rehm","year":"2022","journal-title":"Inf. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"103663","DOI":"10.1016\/j.ipm.2024.103663","article-title":"Legal Judgment Prediction via graph boosting with constraints","volume":"61","author":"Tong","year":"2024","journal-title":"Inf. Process. Manag."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"123626","DOI":"10.1016\/j.eswa.2024.123626","article-title":"Knowledge-enhanced model with dual-graph interaction for confusing legal charge prediction","volume":"249","author":"Bi","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Zou, L., Huang, R., Wang, H., Yu, J.X., He, W., and Zhao, D. (2014, January 22\u201327). Natural language question answering over RDF: A graph data driven approach. Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, Snowbird, UT, USA.","DOI":"10.1145\/2588555.2610525"},{"key":"ref_26","first-page":"1918","article-title":"Joint entity and relation extraction model based on reinforcement learning","volume":"39","author":"Chen","year":"2019","journal-title":"J. Comput. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3580","DOI":"10.1109\/TKDE.2024.3352100","article-title":"Unifying Large Language Models and Knowledge Graphs: A Roadmap","volume":"36","author":"Pan","year":"2024","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"112155","DOI":"10.1016\/j.knosys.2024.112155","article-title":"Enhancing text-based knowledge graph completion with zero-shot large language models: A focus on semantic enhancement","volume":"300","author":"Yang","year":"2024","journal-title":"Knowl.-Based Syst."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Kumar, A., Pandey, A., Gadia, R., and Mishra, M. (2020, January 2\u20134). Building Knowledge Graph using Pre-trained Language Model for Learning Entity-aware Relationships. Proceedings of the 2020 IEEE International Conference on Computing, Power and Communication Technologies (GUCON), Greater Noida, India.","DOI":"10.1109\/GUCON48875.2020.9231227"},{"key":"ref_30","unstructured":"Cohn, T., He, Y., and Liu, Y. (2020). Pretrain-KGE: Learning Knowledge Representation from Pretrained Language Models. Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.aiopen.2021.03.001","article-title":"Neural, symbolic and neural-symbolic reasoning on knowledge graphs","volume":"2","author":"Zhang","year":"2021","journal-title":"AI Open"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"103076","DOI":"10.1016\/j.jnca.2021.103076","article-title":"Domain-specific knowledge graphs: A survey","volume":"185","year":"2021","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1145\/3191513","article-title":"Never-ending learning","volume":"61","author":"Mitchell","year":"2018","journal-title":"Commun. ACM"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Cadeddu, A., Chessa, A., De Leo, V., Fenu, G., Motta, E., Osborne, F., Reforgiato Recupero, D., Salatino, A., and Secchi, L. (2024). Optimizing Tourism Accommodation Offers by Integrating Language Models and Knowledge Graph Technologies. Information, 15.","DOI":"10.3390\/info15070398"},{"key":"ref_35","unstructured":"Nakayama, H., Kubo, T., Kamura, J., Taniguchi, Y., and Liang, X. (2020, May 01). Doccano: Text Annotation Tool for Human. Available online: https:\/\/github.com\/doccano\/doccano."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Li, X.L., and Liang, P. (2021, January 1\u20136). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online.","DOI":"10.18653\/v1\/2021.acl-long.353"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. (2022, January 22\u201327). GLM: General Language Model Pretraining with Autoregressive Blank Infilling. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.","DOI":"10.18653\/v1\/2022.acl-long.26"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1016\/j.cosrev.2018.06.001","article-title":"Recent Named Entity Recognition and Classification techniques: A systematic review","volume":"29","author":"Goyal","year":"2018","journal-title":"Comput. Sci. Rev."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016, January 16). Neural Architectures for Named Entity Recognition. Proceedings of the NAACL, San Diego, CA, USA.","DOI":"10.18653\/v1\/N16-1030"},{"key":"ref_40","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA."},{"key":"ref_41","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS\u201917), Red Hook, NY, USA."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/11\/666\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:18:53Z","timestamp":1760113133000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/11\/666"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,23]]},"references-count":41,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2024,11]]}},"alternative-id":["info15110666"],"URL":"https:\/\/doi.org\/10.3390\/info15110666","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,23]]}}}