{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T15:55:01Z","timestamp":1791388501907,"version":"4.3.3"},"reference-count":49,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62077027"],"award-info":[{"award-number":["62077027"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002855","name":"Ministry of Science and Technology of the People's Republic of China","doi-asserted-by":"publisher","award":["2018YFC2002500"],"award-info":[{"award-number":["2018YFC2002500"]}],"id":[{"id":"10.13039\/501100002855","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Jilin Province Development and Reform Commission, China","award":["2019C053-1"],"award-info":[{"award-number":["2019C053-1"]}]},{"name":"Education Department of Jilin Province, China","award":["JJKH20200993K"],"award-info":[{"award-number":["JJKH20200993K"]}]},{"name":"Department of Science and Technology of Jilin Province, China","award":["20200801002GH"],"award-info":[{"award-number":["20200801002GH"]}]},{"name":"European Union's Horizon 2020 FET Proactive project \u201cWeNet-The Internet of us","award":["No. 823783"],"award-info":[{"award-number":["No. 823783"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The popularity of information technology has given rise to a growing interest in smart education and has provided the possibility of combining online and offline education. Knowledge graphs, an effective technology for knowledge representation and management, have been successfully utilized to manage massive educational resources. However, the existing research on constructing educational knowledge graphs ignores multiple modalities and their relationships, such as teacher speeches and their relationship with knowledge. To tackle this problem, we propose an automatic approach to construct multi-modal educational knowledge graphs that integrate speech as a modal resource to facilitate the reuse of educational resources. Specifically, we first propose a fine-tuned Bidirectional Encoder Representation from Transformers (BERT) model based on education lexicon, called EduBERT, which can adaptively capture effective information in the education field. We also add a Bidirectional Long Short-Term Memory-Conditional Random Field (BiLSTM-CRF) to effectively identify educational entities. Then, the locational information of the entity is incorporated into BERT to extract the educational relationship. In addition, to cover the shortage of traditional text-based knowledge graphs, we focus on collecting teacher speech to construct a multi-modal knowledge graph. We propose a speech-fusion method that links these data into the graph as a class of entities. The numeric results show that our proposed approach can manage and present various modes of educational resources and that it can provide better education services.<\/jats:p>","DOI":"10.3390\/info13020091","type":"journal-article","created":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T22:44:47Z","timestamp":1644965087000},"page":"91","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":51,"title":["MEduKG: A Deep-Learning-Based Approach for Multi-Modal Educational Knowledge Graph Construction"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9011-9213","authenticated-orcid":false,"given":"Nan","family":"Li","sequence":"first","affiliation":[{"name":"College of Computer Science and Technology, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Shen","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Song","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Chi","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hao","family":"Xu","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology, Jilin University, Changchun 130012, China"},{"name":"School of Artificial Intelligence, Jilin University, Changchun 130012, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Mart\u00edn, A.C., Alario-Hoyos, C., and Kloos, C.D. (2019). Smart Education: A Review and Future Research Directions. Proceedings, 31.","DOI":"10.3390\/proceedings2019031057"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"D\u2019Mello, S.K., Olney, A.M., Blanchard, N., Samei, B., Sun, X., Ward, B., and Kelly, S. (2015). Multimodal Capture of Teacher-Student Interactions for Automated Dialogic Analysis in Live Classrooms. ICMI\u201915, Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Association for Computing Machinery.","DOI":"10.1145\/2818346.2830602"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9721","DOI":"10.1609\/aaai.v33i01.33019721","article-title":"Automating Analysis and Feedback to Improve Mathematics Teachers\u2019 Classroom Discourse","volume":"Volume 33","author":"Suresh","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"ref_4","unstructured":"Anand, R., Ottmar, E., Crouch, J.L., and Whitehill, J. (2019, January 14\u201318). Toward Automated Classroom Observation: Predicting Positive and Negative Climate. Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France."},{"key":"ref_5","unstructured":"Heiko, P. (2017, January 22). Towards Profiling Knowledge Graphs. Proceedings of the PROFILES@ISWC, Vienna, Austria."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Wang, S., Liang, C., Wu, Z., Williams, K., Pursel, B., Brautigam, B., Saul, S., Williams, H., Bowen, K., and Giles, C.L. (2015, January 8\u201311). Concept Hierarchy Extraction from Textbooks. Proceedings of the DocEng\u201915\u2014ACM Symposium on Document Engineering 2015, Lusanne, Switzerland.","DOI":"10.1145\/2682571.2797062"},{"key":"ref_7","first-page":"9678","article-title":"Concept Extraction and Prerequisite Relation Learning from Educational Data","volume":"33","author":"Lu","year":"2019","journal-title":"Proc. Conf. AAAI Artif. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hu, J., Zheng, L., and Xu, B. (2016, January 1\u20133). An Approach of Ontology Based Knowledge Base Construction for Chinese K12 Education. Proceedings of the 2016 First International Conference on Multimedia and Image Processing (ICMIP), Bandar Seri Begawan, Brunei.","DOI":"10.1109\/ICMIP.2016.10"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1613\/jair.5002","article-title":"Learning Concept Graphs from Online Educational Data","volume":"55","author":"Liu","year":"2016","journal-title":"J. Artif. Intell. Res."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chen, P., Yu, L., Zheng, V.W., Chen, X., and Li, X. (2018, January 26\u201328). An Automatic Knowledge Graph Construction System for K-12 Education. Proceedings of the Fifth Annual ACM Conference on Learning at Scale 2018, London, UK.","DOI":"10.1145\/3231644.3231698"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Biega, J.A., Kuzey, E., and Suchanek, F.M. (2013, January 13\u201317). Inside YAGO2s: A Transparent Information Extraction Architecture. Proceedings of the 22nd International Conference on World Wide Web\u2014WWW\u201913 Companion, Rio de Janeiro, Brazil.","DOI":"10.1145\/2487788.2487935"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.websem.2009.07.002","article-title":"DBpedia\u2014A crystallization point for the Web of Data","volume":"7","author":"Bizer","year":"2009","journal-title":"J. Web Semant."},{"key":"ref_13","unstructured":"Fredo, E., G\u00fcnther, M., Kr\u00f6tzsch, M., Mendez, J., and Vrande\u010di\u0107, D. (2014). Introducing Wikidata to the Linked Data Web, Springer."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liu, S., Yang, H., Li, J., and Kolmani\u010d, S. (2020). Preliminary Study on the Knowledge Graph Construction of Chinese Ancient History and Culture. Information, 11.","DOI":"10.3390\/info11040186"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Su, Y., and Zhang, Y. (2020, January 1\u20133). Automatic Construction of Subject Knowledge Graph based on Educational Big Data. Proceedings of the ICBDE\u201920\u20143rd International Conference on Big Data and Education, London, UK.","DOI":"10.1145\/3396452.3396458"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Yang, Y., Liu, H., Carbonell, J.G., and Ma, W. (2015, January 2\u20136). Concept Graph Learning from Educational Data. Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, Shanghai, China.","DOI":"10.1145\/2684822.2685292"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Senthilkumar, R.D. (2017, January 25\u201328). Concept Maps in Teaching Physics Concepts Applied to Engineering Education: An Explorative Study at The Middle East College, Sultanate of Oman. Proceedings of the 2017 IEEE Global Engineering Education Conference, EDUCON 2017, Athens, Greece.","DOI":"10.1109\/EDUCON.2017.7942832"},{"key":"ref_18","first-page":"10550","article-title":"Recovering Concept Prerequisite Relations from University Course Dependencies","volume":"31","author":"Chen","year":"2017","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_19","first-page":"3","article-title":"Visualization for Knowledge Graph Based on Education Data","volume":"10","author":"Kai","year":"2017","journal-title":"Int. J. Softw. Inform."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zheng, Y., Liu, R., and Hou, J. (2017, January 15\u201317). The Construction of High Educational Knowledge Graph Based on MOOC. Proceedings of the 2017 IEEE 2nd Information Technology, Networking, Electronic and Automation Control Conference, ITNEC, Chengdu, China.","DOI":"10.1109\/ITNEC.2017.8284984"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1007\/s11390-020-0328-2","article-title":"Constructing an Educational Knowledge Graph with Concepts Linked to Wikipedia","volume":"36","author":"Dang","year":"2021","journal-title":"J. Comput. Sci. Technol."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yao, S., Wang, R., Sun, S., Bu, D., and Liu, J. (2019). Joint Embedding Learning of Educational Knowledge Graphs. arXiv.","DOI":"10.1007\/978-3-030-41099-5_12"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural Architectures for Named Entity Recognition. NAACL. arXiv.","DOI":"10.18653\/v1\/N16-1030"},{"key":"ref_24","unstructured":"Yadav, V., and Bethard, S. (2018). A Survey on Recent Advances in Named Entity Recognition from Deep Learning models. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1075\/li.30.1.03nad","article-title":"A Survey of Named Entity Recognition and Classification","volume":"30","author":"Nadeau","year":"2007","journal-title":"Lingvist. Investig."},{"key":"ref_26","first-page":"2455","article-title":"A Morpheme-based Lexical Chunking System for Chinese","volume":"5","author":"Fu","year":"2008","journal-title":"Int. Conf. Mach. Learn. Cybern."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: A literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J. Biomed. Inform."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1561\/2200000013","article-title":"An Introduction to Conditional Random Fields","volume":"4","author":"Sutton","year":"2012","journal-title":"Found. Trends Mach. Learn."},{"key":"ref_29","first-page":"2493","article-title":"Natural Language Processing (Almost) from Scratch","volume":"12","author":"Collobert","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"ref_30","unstructured":"Zhiheng, H., Xu, W., and Yu, K. (2015). Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv."},{"key":"ref_31","first-page":"933","article-title":"Language Modeling with Gated Convolutional Networks","volume":"70","author":"Yann","year":"2017","journal-title":"Proc. Mach. Learn. Res."},{"key":"ref_32","unstructured":"Hang, Y., Deng, B., Li, X., and Qiu, X. (2019). TENER: Adapting Transformer Encoder for Named Entity Recognition. arXiv."},{"key":"ref_33","unstructured":"Hui, C., Lin, Z., Ding, G., Lou, J.-G., Zhang, Y., and B\u00f6rje, F.K. (2019). GRN: Gated Relation Network to Enhance Convolutional Neural Network for Named Entity Recognition. arXiv."},{"key":"ref_34","unstructured":"Zhu, Y., Wang, G., and B\u00f6rje, F.K. (2019). CAN-NER: Convolutional Attention Network for Chinese Named Entity Recognition. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep Contextualized Word Representations. NAACL. arXiv.","DOI":"10.18653\/v1\/N18-1202"},{"key":"ref_36","unstructured":"Radford, A., and Karthik, N. (2018). Improving Language Understanding by Generative Pre-Training, OpenAI."},{"key":"ref_37","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_38","unstructured":"Aydar, M., Ozge, B., and \u00d6zbay, F. (2020). Neural Relation Extraction: A Survey. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Vu, N.T., Adel, H., Gupta, P., and Sch\u00fctze, H. (2016). Combining Recurrent and Convolutional Neural Networks for Relation Classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics.","DOI":"10.18653\/v1\/N16-1065"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"He, H., Ganjam, K., Jain, N., Lundin, J., White, R., and Lin, J. (2017). An Insight Extraction System on BioMedical Literature with Deep Neural Networks. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics.","DOI":"10.18653\/v1\/D17-1285"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Zeng, D., Liu, K., Chen, Y., and Zhao, J. (2015). Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks, EMNLP.","DOI":"10.18653\/v1\/D15-1203"},{"key":"ref_42","unstructured":"Zhang, S., Zheng, D., Hu, X., and Yang, M. (2015). Bidirectional Long Short-Term Memory Networks for Relation Classification, Fujitsu Research and Development Center."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1109\/TNN.2008.2005605","article-title":"The Graph Neural Network Model","volume":"20","author":"Scarselli","year":"2009","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Qi, P., and Manning, C.D. (2018). Graph Convolution over Pruned Dependency Trees Improves Relation Extraction. arXiv.","DOI":"10.18653\/v1\/D18-1244"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Guo, Z., Zhang, Y., and Lu, W. (2019). Attention Guided Graph Convolutional Networks for Relation Extraction. arXiv.","DOI":"10.18653\/v1\/P19-1024"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Fu, T.-J., Li, P.-H., and Ma, W.-Y. (2019). GraphRel: Modeling Text as Relational Graphs for Joint Entity and Relation Extraction, Association for Computational Linguistics.","DOI":"10.18653\/v1\/P19-1136"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Wu, S., and He, Y. (2019). Enriching Pre-trained Language Model with Entity Information for Relation Classification. arXiv.","DOI":"10.1145\/3357384.3358119"},{"key":"ref_48","unstructured":"Li, C., and Ye, T. (2020). Downstream Model Design of Pre-trained Language Model for Relation Extraction Task. arXiv."},{"key":"ref_49","unstructured":"Mikolov, T., Chen, K., Corrado, G.S., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/2\/91\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:19:44Z","timestamp":1760134784000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/2\/91"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,15]]},"references-count":49,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["info13020091"],"URL":"https:\/\/doi.org\/10.3390\/info13020091","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,15]]}}}