{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,16]],"date-time":"2026-05-16T15:36:02Z","timestamp":1778945762625,"version":"3.51.4"},"reference-count":46,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,9,19]],"date-time":"2025-09-19T00:00:00Z","timestamp":1758240000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JST SPRING","award":["JPMJSP2136"],"award-info":[{"award-number":["JPMJSP2136"]}]},{"name":"JST SPRING","award":["JP20H00622"],"award-info":[{"award-number":["JP20H00622"]}]},{"name":"JST SPRING","award":["JP24K20903"],"award-info":[{"award-number":["JP24K20903"]}]},{"DOI":"10.13039\/501100001691","name":"JSPS KAKENHI","doi-asserted-by":"publisher","award":["JPMJSP2136"],"award-info":[{"award-number":["JPMJSP2136"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"JSPS KAKENHI","doi-asserted-by":"publisher","award":["JP20H00622"],"award-info":[{"award-number":["JP20H00622"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"JSPS KAKENHI","doi-asserted-by":"publisher","award":["JP24K20903"],"award-info":[{"award-number":["JP24K20903"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Students must navigate large catalogs of courses and make appropriate enrollment decisions in many online learning environments. In this context, identifying key concepts and their relationships is essential for understanding course content and informing course recommendations. However, identifying and extracting concepts can be an extremely labor-intensive and time-consuming task when it has to be done manually. Traditional NLP-based methods to extract relevant concepts from courses heavily rely on resource-intensive preparation of detailed course materials, thereby failing to minimize labor. As recent advances in large language models (LLMs) offer a promising alternative for automating concept identification and relationship inference, we thoroughly investigate the potential of LLMs in automatically generating course concepts and their relations. Specifically, we systematically evaluate three LLM variants (GPT-3.5, GPT-4o-mini, and GPT-4o) across three distinct educational tasks, which are concept generation, concept extraction, and relation identification, using six systematically designed prompt configurations that range from minimal context (course title only) to rich context (course description, seed concepts, and subtitles). We systematically assess model performance through extensive automated experiments using standard metrics (Precision, Recall, F1, and Accuracy) and human evaluation by four domain experts, providing a comprehensive analysis of how prompt design and model choice influence the quality and reliability of the generated concepts and their interrelations. Our results show that GPT-3.5 achieves the highest scores on quantitative metrics, whereas GPT-4o and GPT-4o-mini often generate concepts that are more educationally meaningful despite lexical divergence from the ground truth. Nevertheless, LLM outputs still require expert revision, and performance is sensitive to prompt complexity. Overall, our experiments demonstrate the viability of LLMs as a tool for supporting educational content selection and delivery.<\/jats:p>","DOI":"10.3390\/make7030103","type":"journal-article","created":{"date-parts":[[2025,9,19]],"date-time":"2025-09-19T12:33:32Z","timestamp":1758285212000},"page":"103","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Leveraging LLMs for Automated Extraction and Structuring of Educational Concepts and Relationships"],"prefix":"10.3390","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-5555-3306","authenticated-orcid":false,"given":"Tianyuan","family":"Yang","sequence":"first","affiliation":[{"name":"Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Baofeng","family":"Ren","sequence":"additional","affiliation":[{"name":"Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6699-9388","authenticated-orcid":false,"given":"Chenghao","family":"Gu","sequence":"additional","affiliation":[{"name":"Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tianjia","family":"He","sequence":"additional","affiliation":[{"name":"Graduate School of Information Science and Electrical Engineering, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1566-880X","authenticated-orcid":false,"given":"Boxuan","family":"Ma","sequence":"additional","affiliation":[{"name":"Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5831-2152","authenticated-orcid":false,"given":"Shin\u2019ichi","family":"Konomi","sequence":"additional","affiliation":[{"name":"Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,9,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/s41039-021-00167-7","article-title":"CourseQ: The impact of visual and interactive course recommendation in university environments","volume":"16","author":"Ma","year":"2021","journal-title":"Res. Pract. Technol. Enhanc. Learn."},{"key":"ref_2","unstructured":"Streitz, N.A., and Konomi, S. (2024). A Survey on Explainable Course Recommendation Systems. International Conference on Human-Computer Interaction, Springer."},{"key":"ref_3","unstructured":"Pan, L., Wang, X., Li, C., Li, J., and Tang, J. (December, January 27). Course concept extraction in moocs via embedding-based graph propagation. Proceedings of the Eighth International Joint Conference on Natural Language Processing, Taipei, Taiwan."},{"key":"ref_4","unstructured":"Rogers, A., Boyd-Graber, J., and Okazaki, N. (2023, January 9\u201314). Distantly Supervised Course Concept Extraction in MOOCs with Academic Discipline. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, ON, Canada."},{"key":"ref_5","first-page":"85","article-title":"ACE: AI-Assisted Construction of Educational Knowledge Graphs with Prerequisite Relations","volume":"16","author":"Aytekin","year":"2024","journal-title":"J. Educ. Data Min."},{"key":"ref_6","unstructured":"Pan, L., Li, C., Li, J., and Tang, J. (August, January 30). Prerequisite relation learning for concepts in moocs. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Sun, J., He, Y., Xu, Y., Sun, J., and Sun, G. (2024, January 21\u201325). A Learning-path based Supervised Method for Concept Prerequisite Relations Extraction in Educational Data. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, Boise, ID, USA.","DOI":"10.1145\/3627673.3679597"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Gupta, P., Raturi, S., and Venkateswarlu, P. (2025, July 31). Chatgpt for Designing Course Outlines: A Boon or Bane to Modern Technology. Available online: http:\/\/dx.doi.org\/10.2139\/ssrn.4386113.","DOI":"10.2139\/ssrn.4386113"},{"key":"ref_9","unstructured":"Yang, T., Ren, B., Gu, C., Ma, B., and Konomi, S. (2024, January 26\u201328). Leveraging ChatGPT for Automated Knowledge Concept Generation. Proceedings of the CELDA2024: International Conference on Cognition and Exploratory Learning in the Digital Age. International Association for Development of the Information Society (IADIS), Zagreb, Croatia."},{"key":"ref_10","unstructured":"Ehara, Y. (2023, January 11\u201314). Measuring Similarity between Manual Course Concepts and ChatGPT-generated Course Concepts. Proceedings of the 16th International Conference on Educational Data Mining, Bengaluru, India."},{"key":"ref_11","unstructured":"Ma, B., Khan, M.A.Z., Yang, T., Polyzou, A., and Konomi, S. (2025, January 1\u20135). Evaluating the Effectiveness of Large Language Models for Course Recommendation Tasks. Proceedings of the 33rd International Conference on Computers in Education, Chennai, India."},{"key":"ref_12","first-page":"25","article-title":"Who wrote this essay? Detecting AI-generated writing in second language education in higher education","volume":"23","author":"Alexander","year":"2023","journal-title":"Teach. Engl. Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s10805-023-09492-6","article-title":"Detection of GPT-4 generated text in higher education: Combining academic judgement and software to identify generative AI tool misuse","volume":"22","author":"Perkins","year":"2024","journal-title":"J. Acad. Ethics"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Barany, A., Nasiar, N., Porter, C., Zambrano, A.F., Andres, A.L., Bright, D., Shah, M., Liu, X., Gao, S., and Zhang, J. (2024, January 8\u201312). ChatGPT for education research: Exploring the potential of large language models for qualitative codebook development. Proceedings of the International Conference on Artificial Intelligence in Education, Recife, Brazil.","DOI":"10.1007\/978-3-031-64299-9_10"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Foster, J.M., Sultan, M.A., Devaul, H., Okoye, I., and Sumner, T. (2012, January 10\u201314). Identifying core concepts in educational resources. Proceedings of the 12th ACM\/IEEE-CS Joint Conference on Digital Libraries, Washington, DC, USA.","DOI":"10.1145\/2232817.2232827"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Manrique, R., Gr\u00e9visse, C., Marino, O., and Rothkugel, S. (2018, January 26\u201328). Knowledge graph-based core concept identification in learning resources. Proceedings of the Joint International Semantic Technology Conference, Awaji, Japan.","DOI":"10.1007\/978-3-030-04284-4_3"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2505349","article-title":"Resources Sequencing Using Automatic Prerequisite\u2013Outcome Annotation","volume":"6","author":"Changuel","year":"2015","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"ref_18","unstructured":"Yu, J., Wang, C., Luo, G., Hou, L., Li, J., Liu, Z., and Tang, J. (August, January 28). Course Concept Expansion in MOOCs with External Knowledge and Interactive Game. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy."},{"key":"ref_19","unstructured":"Talukdar, P., and Cohen, W. (2012, January 7). Crowdsourced comprehension: Predicting prerequisite structure in wikipedia. Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, Montreal, QC, Canada."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Liang, C., Wu, Z., Huang, W., and Giles, C.L. (2015, January 17\u201321). Measuring prerequisite relations among concepts. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.","DOI":"10.18653\/v1\/D15-1193"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1186\/s40561-019-0104-3","article-title":"Exploring knowledge graphs for the identification of concept prerequisites","volume":"6","author":"Manrique","year":"2019","journal-title":"Smart Learn. Environ."},{"key":"ref_22","unstructured":"Li, I., Fabbri, A.R., Tung, R.R., and Radev, D.R. (February, January 27). What should i learn first: Introducing lecturebank for nlp education and prerequisite chain learning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"108689","DOI":"10.1016\/j.knosys.2022.108689","article-title":"Weakly supervised setting for learning concept prerequisite relations using multi-head attention variational graph auto-encoders","volume":"247","author":"Zhang","year":"2022","journal-title":"Knowl.-Based Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1029","DOI":"10.1007\/s42979-024-03341-y","article-title":"Core Concept Identification in Educational Resources via Knowledge Graphs and Large Language Models","volume":"5","author":"Reales","year":"2024","journal-title":"SN Comput. Sci."},{"key":"ref_25","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_26","unstructured":"Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., and Metzler, D. (2022). Emergent Abilities of Large Language Models. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., and Chen, H. (2023, January 9\u201314). Reasoning with Language Model Prompting: A Survey. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, ON, Canada.","DOI":"10.18653\/v1\/2023.acl-long.294"},{"key":"ref_28","unstructured":"Wei, X., Cui, X., Cheng, N., Wang, X., Zhang, X., Huang, S., Xie, P., Xu, J., Chen, Y., and Zhang, M. (2023). Zero-Shot Information Extraction via Chatting with ChatGPT. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1007\/s11280-024-01291-2","article-title":"A survey on large language models for recommendation","volume":"27","author":"Wu","year":"2024","journal-title":"World Wide Web"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Yang, T., Ren, B., Ma, B., Khan, M.A.Z., He, T., and Konomi, S. (2024, January 14\u201317). Making Course Recommendation Explainable: A Knowledge Entity-Aware Model Using Deep Learning. Proceedings of the 17th International Conference on Educational Data Mining, Atlanta, GA, USA.","DOI":"10.58459\/icce.2024.4862"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Bao, K., Zhang, J., Zhang, Y., Wenjie, W., Feng, F., and He, X. (2023, January 26\u201329). Large language models for recommendation: Progresses and future directions. Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, Beijing, China.","DOI":"10.1145\/3624918.3629550"},{"key":"ref_32","unstructured":"Lekan, K., and Zachary, A.P. (2023, January 10\u201316). AI-Augmented Advising: A Comparative Study of ChatGPT-4 and Advisor-Based Major Recommendations. Proceedings of the NeurIPS Workshop on Generative AI for Education, the Thirty-Seventh Conference on Neural Information Processing Systems, New Orleans, LA, USA."},{"key":"ref_33","unstructured":"Castleman, B., and Turkcan, M.K. (2024, January 14\u201317). Examining the Influence of Varied Levels of Domain Knowledge Base Inclusion in GPT-based Intelligent Tutors. Proceedings of the 17th International Conference on Educational Data Mining, Atlanta, GE, USA."},{"key":"ref_34","unstructured":"Lin, J., Chen, E., Han, Z., Gurung, A., Thomas, D.R., Tan, W., Nguyen, N.D., and Koedinger, K.R. (2024, January 14\u201317). How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-Ended Responses. Proceedings of the 17th International Conference on Educational Data Mining, Atlanta, GE, USA."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yang, T., Ren, B., Ma, B., He, T., Gu, C., and Konomi, S. (2024, January 25\u201329). Boosting Course Recommendation Explainability: A Knowledge Entity Aware Model Using Deep Learning. Proceedings of the 32nd International Conference on Computers in Education, Asia-Pacific Society for Computers in Education, Quezon City, Philippines.","DOI":"10.58459\/icce.2024.4862"},{"key":"ref_36","unstructured":"Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (2020, January 5\u201310). MOOCCube: A Large-Scale Data Repository for NLP Applications in MOOCs. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual."},{"key":"ref_37","unstructured":"Yang, T., Baofeng, R., Gu, C., He, T., Ma, B., and Konomi, S. (2025). Examining GPT\u2019s Capability to Generate and Map Course Concepts and Their Relationship. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Lai, K.H., Yang, Z.R., Lai, P.Y., Wang, C.D., Guizani, M., and Chen, M. (2024, January 26\u201327). Knowledge-Aware Explainable Reciprocal Recommendation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i8.28708"},{"key":"ref_39","first-page":"22","article-title":"Word association norms, mutual information, and lexicography","volume":"16","author":"Church","year":"1990","journal-title":"Comput. Linguist."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1016\/0306-4573(88)90021-0","article-title":"Term-weighting approaches in automatic text retrieval","volume":"24","author":"Salton","year":"1988","journal-title":"Inf. Process. Manag."},{"key":"ref_41","unstructured":"Mihalcea, R., and Tarau, P. (2004, January 25\u201326). Textrank: Bringing order into text. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain."},{"key":"ref_42","unstructured":"Mikolov, T., Yih, W.t., and Zweig, G. (2013, January 10\u201312). Linguistic regularities in continuous space word representations. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, GE, USA."},{"key":"ref_43","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv."},{"key":"ref_44","unstructured":"Liu, Z., Huang, W., Zheng, Y., and Sun, M. (2010, January 9\u201311). Automatic keyphrase extraction via topic decomposition. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Cambridge, MA, USA."},{"key":"ref_45","unstructured":"Liu, J., Liu, C., Zhou, P., Lv, R., Zhou, K., and Zhang, Y. (2023). Is chatgpt a good recommender? A preliminary study. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/103\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:49:05Z","timestamp":1760035745000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/7\/3\/103"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,19]]},"references-count":46,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["make7030103"],"URL":"https:\/\/doi.org\/10.3390\/make7030103","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,19]]}}}