{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T09:03:46Z","timestamp":1789376626001,"version":"build-2803163510"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T00:00:00Z","timestamp":1788566400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T00:00:00Z","timestamp":1789344000000},"content-version":"vor","delay-in-days":9,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Education Department of Hunan Province,China.","award":["HNJG-2022-0608"],"award-info":[{"award-number":["HNJG-2022-0608"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Across seven public educational prediction datasets, three passed all four pre-modeling reliability checks; the remaining four either failed group-aware generalization tests or lacked the provenance metadata needed to run them. One dataset was initially classified as failing but corrected after excluding group-identifier features from the holdout matrix, demonstrating that the audit can distinguish genuine cross-group confounding from feature-encoding artifacts. Each dataset was audited before model optimization using four checks: baseline gap, split instability, null separation, and metadata adequacy under group-aware holdout. The dominant failure mode was not weak iid performance alone but cross-group fragility: in the clearest case, UCI Student declined from iid\n                    <jats:inline-formula>\n                      <jats:tex-math>$$R^2 = 0.242$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    to group-holdout\n                    <jats:inline-formula>\n                      <jats:tex-math>$$R^2 = -0.097$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    , while Higher Ed collapsed from 0.041 to\n                    <jats:inline-formula>\n                      <jats:tex-math>$$-8.79$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    . Increasing model complexity did not remove this pattern: ensemble models improved structurally sound datasets but amplified instability or failed under group holdout on fragile ones. An exploratory cross-dataset comparison further showed that stronger profiles clustered in larger, richer-grouped, performance-proximal datasets, while random-split performance severely overstated deployable signal in fragile datasets. Classification-metric sensitivity analyses reached the same substantive conclusions. The results show that benchmark reliability in educational AI is constrained less by algorithm choice than by data structure, group heterogeneity, and evaluation design. A reusable pre-modeling audit offers a minimum quality gate before public educational datasets support strong benchmark or deployment claims.\n                  <\/jats:p>","DOI":"10.1038\/s41598-026-69629-6","type":"journal-article","created":{"date-parts":[[2026,9,5]],"date-time":"2026-09-05T08:03:34Z","timestamp":1788595414000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Limited structural reliability in public educational prediction benchmarks: a four-dimension audit of seven datasets"],"prefix":"10.1038","volume":"16","author":[{"given":"Yan","family":"Ma","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lizhuo","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,9,5]]},"reference":[{"key":"69629_CR1","unstructured":"Northcutt, C. G., Athalye, A., & Mueller, J. Pervasive label errors in test sets destabilize machine learning benchmarks. In Advances in Neural Information Processing Systems 34 (NeurIPS), pages 10929\u201310943, Red Hook, NY, USA, (2021). Curran Associates."},{"key":"69629_CR2","unstructured":"Recht, B., Roelofs, R., Schmidt, L., & Vaishaal, S. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 5389\u20135400, Long Beach, CA, USA, (2019). PMLR."},{"key":"69629_CR3","unstructured":"Bouthillier, X. et al. Accounting for variance in machine learning benchmarks. In Proceedings of Machine Learning and Systems (MLSys), pp. 747\u2013769, (2021)."},{"key":"69629_CR4","doi-asserted-by":"crossref","unstructured":"Zheng, L. et al. Judging LLM-as-a-judge with MT-Bench & Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS), pages 46595\u201346623, Red Hook, NY, USA, (2023). Curran Associates.","DOI":"10.52202\/075280-2020"},{"key":"69629_CR5","doi-asserted-by":"publisher","unstructured":"Wang, P. et al. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 9440\u20139450, Bangkok, Thailand, (2024). Association for Computational Linguistics. https:\/\/doi.org\/10.18653\/v1\/2024.acl-long.511.","DOI":"10.18653\/v1\/2024.acl-long.511"},{"key":"69629_CR6","doi-asserted-by":"crossref","unstructured":"Huang, H. et al. An empirical study of LLM-as-a-judge for LLM evaluation: Fine-tuned judge model is not a general substitute for GPT-4. Preprint, arXiv:2403.02839, 2024. Accepted to Findings of ACL (2025).","DOI":"10.18653\/v1\/2025.findings-acl.306"},{"issue":"3","key":"69629_CR7","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1355","volume":"10","author":"C Romero","year":"2020","unstructured":"Romero, C. & Ventura, S. Educational data mining and learning analytics: An updated survey. WIREs Data Min. Knowl. Discov. 10(3), e1355. https:\/\/doi.org\/10.1002\/widm.1355 (2020).","journal-title":"WIREs Data Min. Knowl. Discov."},{"key":"69629_CR8","doi-asserted-by":"publisher","unstructured":"Baker, R. S. J. D., & Inventado, P. S. Educational data mining and learning analytics. In Learning Analytics: From Research to Practice, pages 61\u201375. Springer, New York, NY, USA, (2014). https:\/\/doi.org\/10.1007\/978-1-4614-3305-7_4.","DOI":"10.1007\/978-1-4614-3305-7_4"},{"key":"69629_CR9","doi-asserted-by":"publisher","unstructured":"Gardner, J., Brooks, C., & Ryan Baker. Evaluating the fairness of predictive student models through slicing analysis. In Proceedings of the 9th International Learning Analytics and Knowledge Conference, pages 225\u2013234, Tempe, AZ, USA, (2019). https:\/\/doi.org\/10.1145\/3303772.3303791.","DOI":"10.1145\/3303772.3303791"},{"key":"69629_CR10","doi-asserted-by":"publisher","unstructured":"Chejara, P., Prieto, L., Rodriguez-Triana, M., Kasepalu, R., Ruiz-Calleja, A.,& Shankar, S. K. How to build more generalizable models for collaboration quality? In Proceedings of the 13th International Learning Analytics and Knowledge Conference, pp. 111\u2013121, Arlington, TX, USA, (2023). https:\/\/doi.org\/10.1145\/3576050.3576144.","DOI":"10.1145\/3576050.3576144"},{"key":"69629_CR11","doi-asserted-by":"publisher","unstructured":"Chejara, P., Prieto, L. P., Dimitriadis, Y., Rodriguez-Triana, M., Ruiz-Calleja, A., Kasepalu, R., & Shankar, S. K. The impact of attribute noise on the automated estimation of collaboration quality using multimodal learning analytics in authentic classrooms. J. Learn. Anal. 11(2), 73\u201390. https:\/\/doi.org\/10.18608\/jla.2024.8253 (2024).","DOI":"10.18608\/jla.2024.8253"},{"key":"69629_CR12","doi-asserted-by":"publisher","unstructured":"Brennan, R. L. Generalizability Theory. Springer, New York, NY, USA, (2001). https:\/\/doi.org\/10.1007\/978-1-4757-3456-0.","DOI":"10.1007\/978-1-4757-3456-0"},{"issue":"9","key":"69629_CR13","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1037\/0003-066X.50.9.741","volume":"50","author":"S Messick","year":"1995","unstructured":"Messick, S. Validity of psychological assessment: Validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning. Am. Psychol. 50(9), 741\u2013749. https:\/\/doi.org\/10.1037\/0003-066X.50.9.741 (1995).","journal-title":"Am. Psychol."},{"key":"69629_CR14","unstructured":"Kane, M.T. Validation. In Brennan, R. L. editor, Educational Measurement, pages 17\u201364. American Council on Education\/Praeger, Westport, CT, USA, 4th edition, (2006)."},{"key":"69629_CR15","doi-asserted-by":"publisher","unstructured":"Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daum\u00e9\u00a0III, H., & Crawford, K. Datasheets for datasets. Commun. ACM 64(12), 86\u201392.https:\/\/doi.org\/10.1145\/3458723 (2021).","DOI":"10.1145\/3458723"},{"key":"69629_CR16","doi-asserted-by":"publisher","unstructured":"Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P. & Aroyo, L. M. Everyone wants to do the model work, not the data work: Data cascades in high-stakes AI. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1\u201315, Yokohama, Japan, (2021). https:\/\/doi.org\/10.1145\/3411764.3445518.","DOI":"10.1145\/3411764.3445518"},{"key":"69629_CR17","doi-asserted-by":"publisher","unstructured":"Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2(8), e124. https:\/\/doi.org\/10.1371\/journal.pmed.0020124 (2005).","DOI":"10.1371\/journal.pmed.0020124"},{"key":"69629_CR18","doi-asserted-by":"publisher","first-page":"1115","DOI":"10.1038\/s41597-025-05426-6","volume":"12","author":"C Huang","year":"2025","unstructured":"Huang, C. et al. A multi-modal dataset for teacher behavior analysis in offline classrooms. Sci. Data 12, 1115. https:\/\/doi.org\/10.1038\/s41597-025-05426-6 (2025).","journal-title":"Sci. Data"},{"key":"69629_CR19","doi-asserted-by":"publisher","unstructured":"Yilmaz, N., & Sekeroglu, B. Student performance classification using artificial intelligence techniques. In International Conference on Theory and Application of Soft Computing, Computing with Words and Perceptions, pages 596\u2013603, Cham, Switzerland, 2020. Springer. https:\/\/doi.org\/10.1007\/978-3-030-35249-3_76.","DOI":"10.1007\/978-3-030-35249-3_76"},{"key":"69629_CR20","doi-asserted-by":"publisher","unstructured":"Elaf Abu Amrieh, T., Hamtini, I. & Aljarah. Mining educational data to predict student\u2019s academic performance using ensemble methods. Int. J. Database Theory Appl. 9(8), 119\u2013136. https:\/\/doi.org\/10.14257\/ijdta.2016.9.8.13 (2016).","DOI":"10.14257\/ijdta.2016.9.8.13"},{"key":"69629_CR21","unstructured":"Bora, A., & Dey, A. K. Student performance on an entrance examination. UCI Machine Learning Repository, 2021. URL https:\/\/archive.ics.uci.edu\/dataset\/582\/student+performance+on+an+entrance+examination. Accessed: 2026 Apr 30."},{"key":"69629_CR22","unstructured":"Cortez, P., & Silva, A. Using data mining to predict secondary school student performance. In Proceedings of the 5th Future Business Technology Conference (FUBUTEC), pp. 5\u201312, Porto, Portugal, 2008."},{"issue":"11","key":"69629_CR23","doi-asserted-by":"publisher","first-page":"146","DOI":"10.3390\/data7110146","volume":"7","author":"V Realinho","year":"2022","unstructured":"Realinho, V., Machado, J., Baptista, L. & Martins, M. V. Predicting student dropout and academic success. Data 7(11), 146. https:\/\/doi.org\/10.3390\/data7110146 (2022).","journal-title":"Data"},{"key":"69629_CR24","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2017.171","volume":"4","author":"J Kuzilek","year":"2017","unstructured":"Kuzilek, J., Hlosta, M. & Zdrahal, Z. Open university learning analytics dataset. Sci. Data 4, 170171. https:\/\/doi.org\/10.1038\/sdata.2017.171 (2017).","journal-title":"Sci. Data"},{"key":"69629_CR25","doi-asserted-by":"publisher","unstructured":"Mitchell, M. et al. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*), pp. 220\u2013229, Atlanta, GA, USA, 2019. ACM. https:\/\/doi.org\/10.1145\/3287560.3287596.","DOI":"10.1145\/3287560.3287596"},{"key":"69629_CR26","doi-asserted-by":"crossref","unstructured":"Holland, S., Hosny, A., Newman, S., Joseph, J. & Chmielinski, K. The dataset nutrition label. In Data Protection and Privacy, pp. 1\u201325. Hart Publishing, Oxford, UK, 2020.","DOI":"10.5040\/9781509932771.ch-001"},{"key":"69629_CR27","doi-asserted-by":"publisher","unstructured":"\u00c7elik, S., Do\u011fanl\u0131, B., \u015ea\u015fmaz, & Emel. Comparative analysis of global education performances based on PISA,. results: an assessment with statistical and machine learning methods. Qual. Quant. 2026. https:\/\/doi.org\/10.1007\/s11135-026-02915-7 (2022).","DOI":"10.1007\/s11135-026-02915-7"},{"key":"69629_CR28","doi-asserted-by":"publisher","unstructured":"Cavus, M., & Kuzilek, J. An effect analysis of the balancing techniques on the counterfactual explanations of student success prediction models. J. Meas. Eval. Educ. Psychol. 15(Special Issue), 302\u2013317. https:\/\/doi.org\/10.21031\/epod.1526704 (2024).","DOI":"10.21031\/epod.1526704"},{"key":"69629_CR29","unstructured":"Cavus, M., & Kuzilek, J. The actionable explanations for student success prediction models: A benchmark study on the quality of counterfactual methods. In Joint Proceedings of the Human-Centric eXplainable AI in Education and the Leveraging Large Language Models for Next Generation Educational Technologies Workshops co-located with EDM 2024, volume 3840 of CEUR Workshop Proceedings, 2024. URL https:\/\/ceur-ws.org\/Vol-3840\/HEXED24_paper1.pdf."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-69629-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-69629-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-69629-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T08:21:13Z","timestamp":1789374073000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-69629-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,5]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["69629"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-69629-6","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,5]]},"assertion":[{"value":"19 May 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 August 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 September 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"According to the institutional policy governing secondary analysis of existing public datasets at Hunan Agricultural University, ethical review and approval were waived for this study because it used existing public dataset releases, involved no direct interaction with human participants, and analyzed no newly collected personal data.","order":2,"name":"Ethics","label":"Ethical approval","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"28593"}}