{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T06:14:06Z","timestamp":1774678446234,"version":"3.50.1"},"reference-count":61,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T00:00:00Z","timestamp":1756166400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>The use of learning success prediction models is increasingly becoming a part of practice in educational institutions. While recent studies have primarily focused on the development of predictive models, the issue of their temporal stability remains underrepresented in the literature. This issue is critical as model drift can significantly reduce the effectiveness of Learning Analytics applications in real-world educational contexts. This study aims to identify effective approaches for assessing the degradation of predictive models in Learning Analytics and to explore retraining strategies to address model drift. We assess model drift in deployed academic success prediction models using statistical analysis, machine learning, and Explainable Artificial Intelligence. The findings indicate that students\u2019 Digital Profile data are relatively stable, and models trained on these data exhibit minimal model drift, which can be effectively mitigated through regular retraining on more recent data. In contrast, Digital Footprint data from the LMS show moderate levels of data drift, and the models trained on them significantly degrade over time. The most effective strategy for mitigating model degradation involved training a more conservative model and excluding features that exhibited SHAP loss drift. However, this approach did not yield substantial improvements in model performance.<\/jats:p>","DOI":"10.3390\/computers14090351","type":"journal-article","created":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T14:18:49Z","timestamp":1756217929000},"page":"351","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Model Drift in Deployed Machine Learning Models for Predicting Learning Success"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9854-1259","authenticated-orcid":false,"given":"Tatiana A.","family":"Kustitskaya","sequence":"first","affiliation":[{"name":"School of Space and Information Technology, Siberian Federal University, 660041 Krasnoyarsk, Russia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9682-4690","authenticated-orcid":false,"given":"Roman V.","family":"Esin","sequence":"additional","affiliation":[{"name":"School of Space and Information Technology, Siberian Federal University, 660041 Krasnoyarsk, Russia"}]},{"given":"Mikhail V.","family":"Noskov","sequence":"additional","affiliation":[{"name":"School of Space and Information Technology, Siberian Federal University, 660041 Krasnoyarsk, Russia"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Alkhnbashi, O.S., Mohammad, R., and Bamasoud, D.M. (2024). Education in transition: Adapting and thriving in a post-COVID world. Systems, 12.","DOI":"10.3390\/systems12100402"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2381","DOI":"10.4236\/ojbm.2024.124122","article-title":"Adapting the Higher Education through E-Learning Mechanisms\u2014A Post COVID-19 Perspective","volume":"12","author":"Alsabban","year":"2024","journal-title":"Open J. Bus. Manag."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1750","DOI":"10.1111\/bjet.13212","article-title":"The post-COVID-19 future of digital learning in higher education: Views from educators, students, and other professionals in six countries","volume":"53","author":"Guppy","year":"2022","journal-title":"Br. J. Educ. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Stecu\u0142a, K., and Wolniak, R. (2022). Influence of COVID-19 pandemic on dissemination of innovative e-learning tools in higher education in Poland. J. Open Innov. Technol. Mark. Complex., 8.","DOI":"10.3390\/joitmc8020089"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Gligorea, I., Cioca, M., Oancea, R., Gorski, A.-T., Gorski, H., and Tudorache, P. (2023). Adaptive Learning Using Artificial Intelligence in e-Learning: A Literature Review. Educ. Sci., 13.","DOI":"10.3390\/educsci13121216"},{"key":"ref_6","first-page":"86","article-title":"Comprehensive insights into e-learning in contemporary education: Analyzing trends, challenges, and best practices","volume":"6","author":"Hakimi","year":"2024","journal-title":"J. Educ. Teach. Learn."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3468","DOI":"10.24294\/jipd.v8i4.3468","article-title":"Redefining university infrastructure for the 21st century: An interplay between physical assets and digital evolution","volume":"8","author":"Omodan","year":"2024","journal-title":"J. Infrastruct. Policy Dev."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Mitrofanova, Y.S., Tukshumskaya, A.V., Burenina, V.I., Ivanova, E.V., and Popova, T.N. (2021). Integration of Smart Universities in the Region as a Basis for Development of Educational Information Infrastructure, Springer. Smart Education and e-Learning 2021.","DOI":"10.1007\/978-981-16-2834-4_35"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1117","DOI":"10.1007\/s11528-024-01005-5","article-title":"Correction: Bridging the Divide: Assessing Digital Infrastructure for Higher Education Online Learning","volume":"68","author":"Vishnu","year":"2024","journal-title":"TechTrends"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1186\/s41239-020-0177-7","article-title":"Predicting academic success in higher education: Literature review and best practices","volume":"17","author":"Alyahyan","year":"2020","journal-title":"Int. J. Educ. Technol. High. Educ."},{"key":"ref_11","unstructured":"Gaftandzhieva, S., and Doneva, R. (2021, January 8\u20139). Data Analytics to Improve and Optimize University Processes. Proceedings of the 14th annual International Conference of Education, Online Conference."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kustitskaya, T.A., Esin, R.V., Kytmanov, A.A., and Zykova, T.V. (2023). Designing an Education Database in a Higher Education Institution for the Data-Driven Management of the Educational Process. Educ. Sci., 13.","DOI":"10.3390\/educsci13090947"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2165","DOI":"10.1007\/s10639-020-10346-6","article-title":"A data-driven approach to predict first-year students\u2019 academic success in higher education institutions","volume":"26","author":"Gil","year":"2021","journal-title":"Educ. Inf. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kaspi, S., and Venkatraman, S. (2023). Data-driven decision-making (DDDM) for higher education assessments: A case study. Systems, 11.","DOI":"10.3390\/systems11060306"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"798","DOI":"10.1016\/j.cptl.2014.07.006","article-title":"The methodology for the early identification of students at risk for failure in a professional degree program","volume":"6","author":"Alston","year":"2014","journal-title":"Curr. Pharm. Teach. Learn."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.dsm.2024.07.001","article-title":"A model for predicting dropout of higher education students","volume":"8","author":"Rabelo","year":"2025","journal-title":"Data Sci. Manag."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"27579","DOI":"10.1109\/ACCESS.2023.3250702","article-title":"Early predicting of students performance in higher education","volume":"11","author":"Alhazmi","year":"2023","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"100261","DOI":"10.1016\/j.caeai.2024.100261","article-title":"Predicting at-risk students in the early stage of a blended learning course via machine learning using limited data","volume":"7","author":"Azizah","year":"2024","journal-title":"Comput. Educ. Artif. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1007\/s10668-023-03903-9","article-title":"Students\u2019 satisfaction and empowerment of a sustainable university campus","volume":"27","author":"Pedro","year":"2025","journal-title":"Environ. Dev. Sustain."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"566","DOI":"10.32744\/pse.2022.5.34","article-title":"Digital educational history as a component of the digital student\u2019s profile in the context of education transformation","volume":"59","author":"Esin","year":"2022","journal-title":"Perspekt. Nauk. Obraz.\u2014Perspect. Sci. Educ."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"14365","DOI":"10.1007\/s10639-023-12394-0","article-title":"A novel methodology using RNN+ LSTM+ ML for predicting student\u2019s academic performance","volume":"29","author":"Kukkar","year":"2024","journal-title":"Educ. Inf. Technol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1016\/j.iheduc.2017.02.001","article-title":"Learning analytics to unveil learning strategies in a flipped classroom","volume":"33","author":"Dawson","year":"2017","journal-title":"Internet High. Educ."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Abdulkareem Shafiq, D., Marjani, M., Ahamed Ariyaluran Habeeb, R., and Asirvatham, D. (2025). Digital Footprints of Academic Success: An Empirical Analysis of Moodle Logs and Traditional Factors for Student Performance. Educ. Sci., 15.","DOI":"10.3390\/educsci15030304"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"105537","DOI":"10.1016\/j.dib.2020.105537","article-title":"Dataset of academic performance evolution for engineering students","volume":"30","author":"Zuluaga","year":"2020","journal-title":"Data Brief"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"100293","DOI":"10.1016\/j.caeai.2024.100293","article-title":"Advancing student outcome predictions through generative adversarial networks","volume":"7","author":"Farhood","year":"2024","journal-title":"Comput. Educ. Artif. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3782","DOI":"10.1016\/j.matpr.2021.07.382","article-title":"Classification and prediction of student performance data using various machine learning algorithms","volume":"80","author":"Pallathadka","year":"2023","journal-title":"Mater. Today Proc."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Mohammad, A.S., Al-Kaltakchi, M.T.S., Alshehabi Al-Ani, J., and Chambers, J.A. (2023). Comprehensive Evaluations of Student Performance Estimation via Machine Learning. Mathematics, 11.","DOI":"10.3390\/math11143153"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"30604","DOI":"10.1109\/ACCESS.2024.3369586","article-title":"Explainable Models for Predicting Academic Pathways for High School Students in Saudi Arabia","volume":"12","author":"Abdalkareem","year":"2024","journal-title":"IEEE Access"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1007\/s44163-023-00079-z","article-title":"Supervised machine learning algorithms for predicting student dropout and academic success: A comparative study","volume":"4","author":"Villar","year":"2024","journal-title":"Discov. Artif. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"106595","DOI":"10.1016\/j.chb.2020.106595","article-title":"Early prediction of undergraduate Student\u2019s academic performance in completely online learning: A five-year study","volume":"115","author":"Romero","year":"2021","journal-title":"Comput. Hum. Behav."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Leal, F., Veloso, B., Pereira, C.S., Moreira, F., Dur\u00e3o, N., and Silva, N.J. (2022). Interpretable Success Prediction in Higher Education Institutions Using Pedagogical Surveys. Sustainability, 14.","DOI":"10.3390\/su142013446"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1109\/TLT.2024.3351352","article-title":"When the past!= the future: Assessing the Impact of Dataset Drift on the Fairness of Learning Analytics Models","volume":"17","author":"Deho","year":"2024","journal-title":"IEEE Trans. Learn. Technol."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"808","DOI":"10.1055\/s-0041-1735184","article-title":"Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine","volume":"12","author":"Guo","year":"2021","journal-title":"Appl. Clin. Inform."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"20220878","DOI":"10.1259\/bjr.20220878","article-title":"Data drift in medical machine learning: Implications and potential remedies","volume":"96","author":"Sahiner","year":"2023","journal-title":"Br. J. Radiol."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Vela, D., Sharp, A., Zhang, R., Nguyen, T., Hoang, A., and Pianykh, O.S. (2022). Temporal quality degradation in AI models. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-15245-z"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Nigenda, D., Karnin, Z., Zafar, M.B., Ramesha, R., Tan, A., Donini, M., and Kenthapadi, K. (2022, January 14\u201318). Amazon sagemaker model monitor: A system for real-time insights into deployed machine learning models. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA.","DOI":"10.1145\/3534678.3539145"},{"key":"ref_37","unstructured":"Santyev, E.A., Zakharyin, K.N., Shniperov, A.N., Kurchenko, R.A., Shefer, I.A., Vainshtein, Y.V., Somova, M.V., Fedotova, I.M., Noskov, M.V., and Kustitskaya, T.A. (2025, August 21). \u201cPythia\u201d Academic Performance Prediction System. Available online: https:\/\/p.sfu-kras.ru\/."},{"key":"ref_38","first-page":"165","article-title":"Methodological model of personalized educational process based on prediction of success in subject training","volume":"12-1","author":"Somova","year":"2023","journal-title":"Mod. High Technol."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kustitskaya, T.A., Esin, R.V., Vainshtein, Y.V., and Noskov, M.V. (2024). Hybrid Approach to Predicting Learning Success Based on Digital Educational History for Timely Identification of At-Risk Students. Educ. Sci., 14, (In Russian).","DOI":"10.3390\/educsci14060657"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"7","DOI":"10.22363\/2312-8631-2023-20-1-7-19","article-title":"Prognostic model for assessing the success of subject learning in conditions of digitalization of education","volume":"20","author":"Noskov","year":"2023","journal-title":"RUDN J. Informatiz. Educ."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"68","DOI":"10.21541\/apjess.1371070","article-title":"The multicollinearity effect on the performance of machine learning algorithms: Case examples in healthcare modelling","volume":"12","year":"2024","journal-title":"Acad. Platf. J. Eng. Smart Syst."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1162","DOI":"10.1016\/j.jacr.2022.05.030","article-title":"Machine learning model drift: Predicting diagnostic imaging follow-up as a case example","volume":"19","author":"Lacson","year":"2022","journal-title":"J. Am. Coll. Radiol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"108632","DOI":"10.1016\/j.knosys.2022.108632","article-title":"From concept drift to model degradation: An overview on performance-aware drift detectors","volume":"245","author":"Bayram","year":"2022","journal-title":"Knowl. Based Syst."},{"key":"ref_44","unstructured":"Gama, J., Medas, P., Castillo, G., and Rodrigues, P. (2004). Learning with drift detection. Advances in Artificial Intelligence\u2013SBIA 2004: 17th Brazilian Symposium on Artificial Intelligence, Sao Luis, Maranhao, Brazil, 29 September\u20131 Ocotber, 2004, Springer. Proceedings 17."},{"key":"ref_45","unstructured":"Baena-Garc\u0131a, M., del Campo-\u00c1vila, J., Fidalgo, R., Bifet, A., Gavalda, R., and Morales-Bueno, R. (2006, January 18). Early drift detection method. Proceedings of the Fourth International Workshop on Knowledge Discovery from Data Streams, Berlin, Germany."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Bifet, A., and Gavalda, R. (2007, January 26\u201328). Learning from time-changing data with adaptive windowing. Proceedings of the 2007 SIAM International Conference on Data Mining, Minneapolis, MN, USA.","DOI":"10.1137\/1.9781611972771.42"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1016\/j.patcog.2017.11.009","article-title":"Accumulating regional density dissimilarity for concept drift detection in data streams","volume":"76","author":"Liu","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_48","unstructured":"Yurdakul, B. (2018). Statistical Properties of Population Stability Index, Western Michigan University."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/s42488-024-00119-y","article-title":"Detecting drifts in data streams using Kullback-Leibler (KL) divergence measure for data engineering applications","volume":"6","author":"Kurian","year":"2024","journal-title":"J. Data Inf. Manag."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Siddiqi, N. (2012). Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring, John Wiley & Sons.","DOI":"10.1002\/9781119201731"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"29","DOI":"10.4018\/IJGHPC.2019010103","article-title":"Deterministic concept drift detection in ensemble classifier based data stream classification process","volume":"11","author":"Abdualrhman","year":"2019","journal-title":"Int. J. Grid High Perform. Comput."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"5895","DOI":"10.1016\/j.eswa.2013.05.001","article-title":"An adaptive ensemble classifier for mining concept drifting data streams","volume":"40","author":"Farid","year":"2013","journal-title":"Expert Syst. Appl."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"9523","DOI":"10.1016\/j.jksuci.2021.11.006","article-title":"Concept drift detection in data stream mining: A literature review","volume":"34","author":"Agrahari","year":"2022","journal-title":"J. King Saud Univ. Comput. Inf. Sci."},{"key":"ref_54","unstructured":"Lundberg, S.M., and Lee, S.-I. (2017). A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst., 30, Available online: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/8a20a8621978632d76c43dfd28b67767-Paper.pdf."},{"key":"ref_55","unstructured":"Zheng, S., van der Zon, S.B., Pechenizkiy, M., de Campos, C.P., van Ipenburg, W., de Harder, H., and Nederland, R. (2019, January 13\u201314). Labelless concept drift detection and explanation. Proceedings of the NeurIPS 2019 Workshop on Robust AI in Financial Services: Data, Fairness, Explainability, Trustworthiness, and Privacy, Vancouver, BC, Canada. Available online: https:\/\/pure.tue.nl\/ws\/portalfiles\/portal\/142599201\/Thesis_Final_version_Shihao_Zheng.pdf."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"57","DOI":"10.32517\/0234-0453-2025-40-3-57-68","article-title":"Monitoring data shift in the learning success forecasting model using Shapley values","volume":"40","author":"Kustitskaya","year":"2025","journal-title":"Inform. Educ."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"26","DOI":"10.22237\/jmasm\/1257035100","article-title":"New effect size rules of thumb","volume":"8","author":"Sawilowsky","year":"2009","journal-title":"J. Mod. Appl. Stat. Methods"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Chuah, J., Kruger, U., Wang, G., Yan, P., and Hahn, J. (2022). Framework for Testing Robustness of Machine Learning-Based Classifiers. J. Pers. Med., 12.","DOI":"10.3390\/jpm12081314"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"d\u2019Eon, G., d\u2019Eon, J., Wright, J.R., and Leyton-Brown, K. (2022, January 21\u201324). The spotlight: A general method for discovering systematic errors in deep learning models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea.","DOI":"10.1145\/3531146.3533240"},{"key":"ref_61","unstructured":"Pastor, E., de Alfaro, L., and Baralis, E. (2021). Identifying biased subgroups in ranking and classification. arXiv."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/9\/351\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:33:14Z","timestamp":1760034794000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/9\/351"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,26]]},"references-count":61,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["computers14090351"],"URL":"https:\/\/doi.org\/10.3390\/computers14090351","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,26]]}}}