{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,11]],"date-time":"2025-09-11T20:27:50Z","timestamp":1757622470194,"version":"3.44.0"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,8,9]],"date-time":"2025-08-09T00:00:00Z","timestamp":1754697600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,8,9]],"date-time":"2025-08-09T00:00:00Z","timestamp":1754697600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001775","name":"University of Technology Sydney","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001775","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Discov Computing"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Nowadays, the volume of literature in each field is huge and is growing rapidly, which posts challenge to researchers\u2019 literature review. In this circumstance, developing useful tool for achieving efficient literature management is of high value. Traditional literature management tools, such as tools for key word searching, paper recommendation, relation visualization, and keyword cloud drawing, are not suitable for conducting content-level literature review. To address the issues of traditional literature management tools, a novel problem and method-oriented fine-grained academic graph is proposed to facilitate the exploration of research questions, methodologies, study perspectives, and their connections hidden in massive literature. For building such graph, a new ontology dedicated for describing the features of research paper is developed, an innovative multi-relation join extraction model is proposed, and a creative approach for leveraging the Large Language Models (LLM) to augment the triplet extraction results generated by supervised-learning model is developed. Experiments on widely used benchmark datasets show that the proposed multi-relation extraction model is able to achieve at least 8.01% and 8.65% improvement on entity identification and relation classification respectively, compared with state-of-the-art models. The visualized demonstration of the proposed graph shows that our graph is capable of accurately capturing the problem network, method network and hot topics hidden in massive literature. The Q&amp;A system supported by the proposed graph demonstrates that our graph is really helpful for conducting literature review.<\/jats:p>","DOI":"10.1007\/s10791-025-09675-2","type":"journal-article","created":{"date-parts":[[2025,8,9]],"date-time":"2025-08-09T10:55:55Z","timestamp":1754736955000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Research on the construction and application of problem-method-oriented academic graph empowered by LLM"],"prefix":"10.1007","volume":"28","author":[{"given":"Qigang","family":"Liu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yinfan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lifeng","family":"Mu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jun","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,8,9]]},"reference":[{"issue":"1","key":"9675_CR1","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1162\/dint_a_00006","volume":"1","author":"H Wan","year":"2019","unstructured":"Wan H, Zhang Y, Zhang J, Tang J, Aminer. Search and mining of academic social networks. Data Intell. 2019;1(1):58\u201376.","journal-title":"Data Intell"},{"key":"9675_CR2","doi-asserted-by":"crossref","unstructured":"Sinha A, Shen Z, Song Y, Ma H, Eide D, Hsu BJ, Wang K. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, ACM, Florence, Italy; 2015. pp. 243\u2013246.","DOI":"10.1145\/2740908.2742839"},{"key":"9675_CR3","doi-asserted-by":"crossref","unstructured":"Wang R, Yan Y, Wang J, Jia Y, Zhang Y, Zhang W, Wang X. Acekg: A large-scale knowledge graph for academic data mining. In Proceedings of the 27th ACM international conference on information and knowledge management, ACM, Torino, Italy; 2018. pp. 1487\u20131490.","DOI":"10.1145\/3269206.3269252"},{"key":"9675_CR4","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1812.06722","author":"F Ren","year":"2018","unstructured":"Ren F, Hou Y, Li Y, Pan L, Zhang Y, Liang X, Liu Y, Guo Y, Zhao R, Ming R, Wu H. Techkg: A large-scale Chinese technology-oriented knowledge graph. Comput Sc. 2018. https:\/\/doi.org\/10.48550\/arXiv.1812.06722.","journal-title":"Comput Sc"},{"key":"9675_CR5","doi-asserted-by":"crossref","unstructured":"Zhong Z, Chen D. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, online; 2021. pp. 50\u201361.","DOI":"10.18653\/v1\/2021.naacl-main.5"},{"key":"9675_CR6","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K, BERT. Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, Minnesota; 2019. pp. 4171\u20134186."},{"key":"9675_CR7","doi-asserted-by":"crossref","unstructured":"Sun K, Zhang R, Mensah S, Mao Y, Liu X. Recurrent interaction network for jointly extracting entities and classifying relations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), ACL, online; 2020. pp. 3722\u20133732.","DOI":"10.18653\/v1\/2020.emnlp-main.304"},{"key":"9675_CR8","doi-asserted-by":"crossref","unstructured":"Carbonell M, Riba P, Villegas M, Forn\u00e9s A, Llad\u00f3s J. Named entity recognition and relation extraction with graph neural networks in semi structured documents. In 2020 25th International Conference on Pattern Recognition (ICPR), IEEE, Milan, Italy; 2021. pp. 9622\u20139627.","DOI":"10.1109\/ICPR48806.2021.9412669"},{"key":"9675_CR9","doi-asserted-by":"crossref","unstructured":"Zheng S, Hao Y, Lu D, Bao H, Xu J, Hao H, Xu B. 2017. Joint entity and relation extraction based on a hybrid neural network. Neurocomputing. 2017; 257:59\u201366.","DOI":"10.1016\/j.neucom.2016.12.075"},{"key":"9675_CR10","doi-asserted-by":"crossref","unstructured":"Zheng S, Wang F, Bao H, Hao Y, Zhou P, Xu B. Joint extraction of entities and relations based on a novel tagging scheme. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL, Vancouver, Canada; 2017. pp. 1227\u20131236.","DOI":"10.18653\/v1\/P17-1113"},{"key":"9675_CR11","doi-asserted-by":"crossref","unstructured":"Wei Z, Su J, Wang Y, Tian Y, Chang Y. A novel cascade binary tagging framework for relational triple extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, online; 2020. pp. 1476\u20131488.","DOI":"10.18653\/v1\/2020.acl-main.136"},{"key":"9675_CR12","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.10428","author":"S Wang","year":"2023","unstructured":"Wang S, Sun X, Li X, Ouyang R, Wu F, Zhang T, Li J, Wang G. Gpt-ner: named entity recognition via large Language models. Comput Sc. 2023. https:\/\/doi.org\/10.48550\/arXiv.2304.10428.","journal-title":"Comput Sc"},{"key":"9675_CR13","doi-asserted-by":"crossref","unstructured":"Kim S, Seo K, Chae H, Yeo J, Lee D, VerifiNER. Verification-augmented NER via Knowledge-grounded Reasoning with Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL, Bangkok, Thailand 2024. pp. 2441\u20132461.","DOI":"10.18653\/v1\/2024.acl-long.134"},{"key":"9675_CR14","unstructured":"Jiang G, Luo Z, Shi Y, Wang D, Liang J, Yang D. ToNER: Type-oriented Named Entity Recognition with Generative Language Model. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), ELRA and ICCL, Torino, Italia; 2024. pp. 16251\u201316262."},{"key":"9675_CR15","doi-asserted-by":"crossref","unstructured":"Wan Z, Cheng F, Mao Z, Liu Q, Song H, Li J, Kurohashi S. Gpt-re: In-context learning for relation extraction using large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, ACL, Singapore; 2023. pp. 3534\u20133547.","DOI":"10.18653\/v1\/2023.emnlp-main.214"},{"issue":"2","key":"9675_CR16","first-page":"167","volume":"7","author":"A Constantin","year":"2016","unstructured":"Constantin A, Peroni S, Pettifer S, Shotton D, Vitali F. The document components ontology (DoCO). Semantic Web. 2016;7(2):167\u201381.","journal-title":"Semantic Web"},{"key":"9675_CR17","unstructured":"Soldatova L, Liakata M. An ontology methodology and cisp-the proposed core information about scientific papers. JISC Project Report; 2007."},{"key":"9675_CR18","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1007\/s007990000034","volume":"3","author":"S Buckingham Shum","year":"2000","unstructured":"Buckingham Shum S, Motta E, Domingue J. ScholOnto: an ontology-based digital library server for research documents and discourse. Int J Digit Libr. 2000;3:237\u201348.","journal-title":"Int J Digit Libr"},{"key":"9675_CR19","doi-asserted-by":"crossref","unstructured":"Beltagy I, Lo K, Cohan A. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), ACL, Hong Kong, China; 2019. pp. 3615\u20133620.","DOI":"10.18653\/v1\/D19-1371"},{"key":"9675_CR20","doi-asserted-by":"crossref","unstructured":"Tao Q, Luo X, Wang H, Xu R. Enhancing relation extraction using syntactic indicators and sentential contexts. In 2019 IEEE 31st international conference on tools with artificial intelligence (ICTAI), IEEE, Portland, OR, USA; 2019. pp. 1574\u20131580.","DOI":"10.1109\/ICTAI.2019.00227"},{"key":"9675_CR21","doi-asserted-by":"publisher","first-page":"104144","DOI":"10.1016\/j.jbi.2022.104144","volume":"133","author":"J Shi","year":"2022","unstructured":"Shi J, Sun M, Sun Z, Li M, Gu Y, Zhang W. Multi-level semantic fusion network for Chinese medical named entity recognition. J Biomed Inform. 2022;133:104144.","journal-title":"J Biomed Inform"},{"key":"9675_CR22","doi-asserted-by":"publisher","first-page":"116682","DOI":"10.1016\/j.eswa.2022.116682","volume":"196","author":"D Li","year":"2022","unstructured":"Li D, Yan L, Yang J, Ma Z. Dependency syntax guided bert-bilstm-gam-crf for Chinese Ner. Expert Syst Appl. 2022;196:116682.","journal-title":"Expert Syst Appl"},{"key":"9675_CR23","doi-asserted-by":"crossref","unstructured":"Yan R, Jiang X, Dang D. 2021. Named entity recognition by using XLNet-BiLSTM-CRF. Neural Processing Letters. 2021; 53(5):3339\u20133356.","DOI":"10.1007\/s11063-021-10547-1"},{"key":"9675_CR24","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1711.05101","author":"I Loshchilov","year":"2017","unstructured":"Loshchilov I. Decoupled weight decay regularization. Comput Sci. 2017. https:\/\/doi.org\/10.48550\/arXiv.1711.05101.","journal-title":"Comput Sci"},{"key":"9675_CR25","doi-asserted-by":"crossref","unstructured":"Ji Z, Lee N, Frieske R, Yu T, Su D, Xu Y, Ishii E, Bang YJ, Madotto A, Fung P. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys. 2023; 55(12):1\u201338.","DOI":"10.1145\/3571730"},{"key":"9675_CR26","doi-asserted-by":"crossref","unstructured":"Kim J, Park S, Jeong K, Lee S, Han SH, Lee J, Kang P. Which is better? exploring prompting strategy for llm-based metrics. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems; 2023. pp. 164\u2013183.","DOI":"10.18653\/v1\/2023.eval4nlp-1.14"},{"key":"9675_CR27","doi-asserted-by":"crossref","unstructured":"Cheng Q, Chen L, Hu Z, Tang J, Xu Q, Ning B. 2024. A novel prompting method for few-shot ner via llms. Natural Language Processing Journal. 2024; 8:100099.","DOI":"10.1016\/j.nlp.2024.100099"},{"key":"9675_CR28","doi-asserted-by":"publisher","first-page":"120841","DOI":"10.1016\/j.eswa.2023.120841","volume":"231","author":"C Liu","year":"2023","unstructured":"Liu C, Ji X, Dong Y, He M, Yang M, Wang Y. Chinese mineral question and answering system based on knowledge graph. Expert Syst Appl. 2023;231:120841.","journal-title":"Expert Syst Appl"},{"key":"9675_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s44196-021-00010-3","volume":"14","author":"S Liu","year":"2021","unstructured":"Liu S, Tan N, Yang H, Luka\u010d N. An intelligent question answering system of the Liao dynasty based on knowledge graph. Int J Comput Intell Syst. 2021;14:1\u201312.","journal-title":"Int J Comput Intell Syst"},{"key":"9675_CR30","doi-asserted-by":"crossref","unstructured":"Feng G, Zhu G, Shi S, Sun Y, Fan Z, Gao S, Hu J. August. Robust NL-to-Cypher Translation for KBQA: Harnessing Large Language Model with Chain of Prompts. In China Conference on Knowledge Graph and Semantic Computing, Springer, Shenyang, China; 2023. pp. 317\u2013326.","DOI":"10.1007\/978-981-99-7224-1_25"},{"key":"9675_CR31","doi-asserted-by":"crossref","unstructured":"Shang YM, Huang H, Mao X, Onerel. Joint entity and relation extraction with one module in one step. In Proceedings of the AAAI conference on artificial intelligence. 2022; 36(10):11285\u201311293.","DOI":"10.1609\/aaai.v36i10.21379"},{"key":"9675_CR32","doi-asserted-by":"crossref","unstructured":"Han D, Zheng Z, Zhao H, Feng S, Pang H. 2023. Span-based single-stage joint entity-relation extraction model. Plos one. 2023; 18(2):0281055.","DOI":"10.1371\/journal.pone.0281055"},{"key":"9675_CR33","doi-asserted-by":"crossref","unstructured":"Yan Z, Zhang C, Fu J, Zhang Q, Wei Z. A partition filter network for joint entity and relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, ACL, Punta Cana, Dominican Republic; 2021. pp. 185\u2013197.","DOI":"10.18653\/v1\/2021.emnlp-main.17"},{"key":"9675_CR34","doi-asserted-by":"publisher","unstructured":"Ashok D, Lipton ZC, Promptner. Prompting for named entity recognition. Computation and Language. 2023. https:\/\/doi.org\/10.48550\/arXiv.2305.15444","DOI":"10.48550\/arXiv.2305.15444"},{"key":"9675_CR35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.02105","author":"Z Wan","year":"2023","unstructured":"Wan Z, Cheng F, Mao Z, et al. Gpt-re: In-context learning for relation extraction using large Language models. Comput Lang. 2023. https:\/\/doi.org\/10.48550\/arXiv.2305.02105.","journal-title":"Comput Lang"},{"key":"9675_CR36","doi-asserted-by":"crossref","unstructured":"Takanobu R, Zhang T, Liu J, Huang M. A hierarchical framework for relation extraction with reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, Honolulu AAAI. Hawaii, USA; 2019. pp. 7072\u20137079.","DOI":"10.1609\/aaai.v33i01.33017072"},{"key":"9675_CR37","doi-asserted-by":"publisher","unstructured":"Bassignana E, Plank B, CrossRE:. A Cross-Domain Dataset for Relation Extraction. 2022. https:\/\/doi.org\/10.48550\/arXiv.2210.09345","DOI":"10.48550\/arXiv.2210.09345"},{"key":"9675_CR38","doi-asserted-by":"publisher","unstructured":"Ofosu-Ampong K. Artificial intelligence research: A review on dominant themes, methods, frameworks and future research directions. Telematics Inf Rep. 2024;14:100127. https:\/\/doi.org\/10.1016\/j.teler.2024.10012.","DOI":"10.1016\/j.teler.2024.10012"},{"key":"9675_CR39","doi-asserted-by":"crossref","unstructured":"Guler N, Kirshner SN, Vidgen R. A literature review of artificial intelligence research in business and management using machine learning and ChatGPT. Data Inform Manage. 2024;8(3):100076.","DOI":"10.1016\/j.dim.2024.100076"},{"issue":"15","key":"9675_CR40","doi-asserted-by":"publisher","first-page":"2552","DOI":"10.3390\/math10152552","volume":"10","author":"RI Mukhamediev","year":"2022","unstructured":"Mukhamediev RI, Popova Y, Kuchin Y, et al. Review of artificial intelligence and machine learning technologies: classification, restrictions, opportunities and challenges. Mathematics. 2022;10(15):2552.","journal-title":"Mathematics"},{"issue":"1","key":"9675_CR41","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1007\/s44163-023-00089-x","volume":"3","author":"M Elahi","year":"2023","unstructured":"Elahi M, Afolaranmi SO, Martinez Lastra JL, et al. A comprehensive literature review of the applications of AI techniques through the lifecycle of industrial equipment. Discover Artif Intell. 2023;3(1):43.","journal-title":"Discover Artif Intell"}],"container-title":["Discover Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10791-025-09675-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10791-025-09675-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10791-025-09675-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,8]],"date-time":"2025-09-08T20:03:01Z","timestamp":1757361781000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10791-025-09675-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,9]]},"references-count":41,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["9675"],"URL":"https:\/\/doi.org\/10.1007\/s10791-025-09675-2","relation":{},"ISSN":["2948-2992"],"issn-type":[{"type":"electronic","value":"2948-2992"}],"subject":[],"published":{"date-parts":[[2025,8,9]]},"assertion":[{"value":"11 December 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 July 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 August 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All authors abide by academic ethics and agree to publish a statement of compliance with academic ethics.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"The authors affirm that this manuscript contains original work and has not been published or submitted elsewhere. All authors have given their consent for the publication of this research.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to publish"}}],"article-number":"168"}}