{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,11]],"date-time":"2026-10-11T23:44:02Z","timestamp":1791762242632,"version":"4.3.4"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,7,27]],"date-time":"2024-07-27T00:00:00Z","timestamp":1722038400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,27]],"date-time":"2024-07-27T00:00:00Z","timestamp":1722038400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100016981","name":"Necmettin Erbakan University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100016981","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Educ Inf Technol"],"published-print":{"date-parts":[[2025,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Feedback is critical in providing personalized information about educational processes and supporting their performance in online collaborative learning environments. However, giving effective feedback and monitoring its effects, which is especially important in online environments, is a complex issue. Although providing feedback by analyzing online learning behaviors, it is unclear how the effectiveness of this feedback translates into online learning experiences. The current study aims to compare the behavioral patterns of online system engagement of students who receive and do not receive machine learning-based temporal learning analytics (ML-LA) feedback, to identify the differences between student groups in terms of learning performance, online engagement, and various system usage variables, and to examine the behavioral patterns change over time of students regarding online system engagement. The current study was conducted with the participation of 49 undergraduate students. The study defined three engagement levels using system usage analytics and cluster analysis. While t-test and ANCOVA were applied to pre-test and post-test scores to evaluate students\u2019 learning performance and online engagement, lag sequential analysis was used to analyze behavioral patterns, and the Markov chain was used to examine the change of behavioral patterns over time. The group receiving ML-LA feedback showed higher behavior and cognitive engagement than the control group. In addition, the rate of completing learning tasks was higher in the experimental group. Temporal patterns of online engagement behaviors across student groups are described and compared. The results showed that both groups used all stages of the system features. However, there were some differences in the navigation rankings. The most important behavioral transitions in the experimental group are task and discussion viewing and posting, task posting updating, and group performance viewing. In the control group, the most important behavioral transitions are the relationship between viewing a discussion and making a discussion, then this is followed by the sequential relationship between viewing individual performance and viewing group performance. The results showed that students\u2019 engagement behaviors transitioned from light to medium and intense throughout the semester, especially in the experimental group. For learning designers and researchers, this study can help develop a deep understanding of environment design.<\/jats:p>","DOI":"10.1007\/s10639-024-12911-9","type":"journal-article","created":{"date-parts":[[2024,7,27]],"date-time":"2024-07-27T03:02:41Z","timestamp":1722049361000},"page":"2623-2644","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["Impact of ML-LA feedback system on learners\u2019 academic performance, engagement and behavioral patterns in online collaborative learning environments: A lag sequential analysis and Markov chain approach"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5689-1805","authenticated-orcid":false,"given":"Hatice","family":"Yildiz Durak","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,7,27]]},"reference":[{"issue":"1","key":"12911_CR3","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1080\/0969594X.2019.1688764","volume":"27","author":"BT Agricola","year":"2020","unstructured":"Agricola, B. T., Prins, F. J., & Sluijsmans, D. M. (2020). Impact of feedback request forms and verbal feedback on higher education students\u2019 feedback perception, self-efficacy, and motivation. Assessment in Education: Principles Policy & Practice, 27(1), 6\u201325. https:\/\/doi.org\/10.1080\/0969594X.2019.1688764","journal-title":"Assessment in Education: Principles Policy & Practice"},{"issue":"1","key":"12911_CR2","doi-asserted-by":"publisher","first-page":"80","DOI":"10.24059\/olj.v26i1.3065","volume":"26","author":"EM Alwafi","year":"2022","unstructured":"Alwafi, E. M. (2022). Designing an online discussion strategy with Learning Analytics Feedback on the level of Cognitive Presence and Student Interaction in an online Learning Community. Online Learning, 26(1), 80\u201392. https:\/\/eric.ed.gov\/?id=EJ1340522","journal-title":"Online Learning"},{"issue":"5","key":"12911_CR4","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/j.jsp.2006.04.002","volume":"44","author":"JJ Appleton","year":"2006","unstructured":"Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive and psychological engagement: Validation of the student engagement instrument. Journal of School Psychology, 44(5), 427\u2013445. https:\/\/doi.org\/10.1016\/j.jsp.2006.04.002","journal-title":"Journal of School Psychology"},{"issue":"6","key":"12911_CR5","doi-asserted-by":"publisher","first-page":"671","DOI":"10.1080\/03075071003731135","volume":"36","author":"C Beaumont","year":"2011","unstructured":"Beaumont, C., O\u2019Doherty, M., & Shannon, L. (2011). Reconceptualising assessment feedback: A key to improving student learning? Studies in Higher Education, 36(6), 671\u2013687. https:\/\/doi.org\/10.1080\/03075071003731135","journal-title":"Studies in Higher Education"},{"key":"12911_CR7","doi-asserted-by":"crossref","unstructured":"Bowman, G. R., Pande, V. S., & No\u00e9, F. (2013). An introduction to Markov state models and their application to long timescale molecular simulation (Vol. 797). Springer Science & Business Media.","DOI":"10.1007\/978-94-007-7606-7"},{"issue":"4","key":"12911_CR8","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1111\/bjep.12126","volume":"86","author":"GT Brown","year":"2016","unstructured":"Brown, G. T., Peterson, E. R., & Yao, E. S. (2016). Student conceptions of feedback: Impact on self-regulation, self\u2010efficacy, and academic achievement. British Journal of Educational Psychology, 86(4), 606\u2013629. https:\/\/doi.org\/10.1111\/bjep.12126","journal-title":"British Journal of Educational Psychology"},{"key":"12911_CR43","doi-asserted-by":"crossref","unstructured":"Cerezo, R., S\u00e1nchez-Santill\u00e1n, M., Paule-Ruiz, M. P., & N\u00fa\u00f1ez, J. C. (2016). Students\u2019 LMS interaction patterns and their relationship with achievement: A case study in higher education. Computers & Education, 96, 42\u201354.","DOI":"10.1016\/j.compedu.2016.02.006"},{"key":"12911_CR9","unstructured":"Cole, P. G., & Chan, L. (1994). Teaching principles and practice. Prentice Hall. https:\/\/books.google.com.tr\/books\/about\/Teaching_Principles_and_Practice.html?id=864uAAAACAAJ&redir_esc=y"},{"key":"12911_CR44","doi-asserted-by":"crossref","unstructured":"De Barba, P. G., Kennedy, G. E., & Ainley, M. D. (2016). The role of students\u2019 motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32(3), 218\u2013231.","DOI":"10.1111\/jcal.12130"},{"issue":"2","key":"12911_CR11","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1007\/s40593-023-00335-4","volume":"34","author":"V Dornauer","year":"2024","unstructured":"Dornauer, V., Netzer, M., Kaczk\u00f3, \u00c9., Norz, L. M., & Ammenwerth, E. (2024). Automatic classification of online discussions and other Learning traces to detect Cognitive Presence. International Journal of Artificial Intelligence in Education, 34(2), 395\u2013415. https:\/\/doi.org\/10.1007\/s40593-023-00335-4","journal-title":"International Journal of Artificial Intelligence in Education"},{"issue":"5","key":"12911_CR13","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1080\/13562517.2016.1160219","volume":"21","author":"K Dunworth","year":"2016","unstructured":"Dunworth, K., & Sanchez, H. S. (2016). Perceptions of quality in staff-student written feedback in higher education: A case study. Teaching in Higher Education, 21(5), 576\u2013589. https:\/\/doi.org\/10.1080\/13562517.2016.1160219","journal-title":"Teaching in Higher Education"},{"key":"12911_CR10","doi-asserted-by":"publisher","first-page":"3975","DOI":"10.1007\/s10639-020-10151-1","volume":"25","author":"A \u00c7ebi","year":"2020","unstructured":"\u00c7ebi, A., & G\u00fcyer, T. (2020). Students\u2019 interaction patterns in different online learning activities and their relationship with motivation, self-regulated learning strategy and learning performance. Education and Information Technologies, 25, 3975\u20133993. https:\/\/doi.org\/10.1007\/s10639-020-10151-1","journal-title":"Education and Information Technologies"},{"issue":"1","key":"12911_CR14","doi-asserted-by":"publisher","first-page":"18","DOI":"10.17943\/etku.64661","volume":"5","author":"E Erg\u00fcn","year":"2015","unstructured":"Erg\u00fcn, E., & Usluel, Y. K. (2015). \u00c7evrimi\u00e7i \u00f6\u011frenme ortamlar\u0131nda \u00f6\u011frenci ba\u011fl\u0131l\u0131k \u00f6l\u00e7e\u011fi\u2019nin t\u00fcrk\u00e7e uyarlamas\u0131: Ge\u00e7erlik ve g\u00fcvenirlik \u00e7al\u0131\u015fmas\u0131. E\u011fitim Teknolojisi Kuram Ve Uygulama, 5(1), 18\u201333. https:\/\/doi.org\/10.17943\/etku.64661","journal-title":"E\u011fitim Teknolojisi Kuram Ve Uygulama"},{"key":"12911_CR15","doi-asserted-by":"publisher","unstructured":"Finn, J. D., & Zimmer, K. S. (2012). Student engagement: What is it? Why does it matter? In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 97\u2013132). Springer. https:\/\/doi.org\/10.1007\/978-1-4614-2018-7_5","DOI":"10.1007\/978-1-4614-2018-7_5"},{"key":"12911_CR39","unstructured":"Greller, W. & Drachslrer, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology & Society, 15(3), 42\u201357."},{"issue":"1","key":"12911_CR16","doi-asserted-by":"publisher","first-page":"81","DOI":"10.3102\/003465430298487","volume":"77","author":"J Hattie","year":"2007","unstructured":"Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81\u2013112. https:\/\/doi.org\/10.3102\/003465430298487","journal-title":"Review of Educational Research"},{"key":"12911_CR41","doi-asserted-by":"crossref","unstructured":"Kizilcec, R. F., P\u00e9rez-Sanagust\u00edn, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behavior and goal attainment in Massive Open Online Courses. Computers & Education, 104, 18\u201333.","DOI":"10.1016\/j.compedu.2016.10.001"},{"issue":"1","key":"12911_CR18","first-page":"223","volume":"24","author":"M Koko\u00e7","year":"2021","unstructured":"Koko\u00e7, M., Ak\u00e7ap\u0131nar, G., & Hasnine, M. N. (2021). Unfolding students\u2019 online assignment submission behavioral patterns using temporal learning analytics. Educational Technology & Society, 24(1), 223\u2013235. https:\/\/www.jstor.org\/stable\/26977869","journal-title":"Educational Technology & Society"},{"key":"12911_CR42","doi-asserted-by":"crossref","unstructured":"Li, L. Y., & Tsai, C. C. (2017). Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers & Education, 114, 286\u2013297.","DOI":"10.1016\/j.compedu.2017.07.007"},{"key":"12911_CR19","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1016\/j.compedu.2013.04.017","volume":"67","author":"L Lin","year":"2013","unstructured":"Lin, L., Atkinson, R. K., Christopherson, R. M., Joseph, S. S., & Harrison, C. J. (2013). Animated agents and learning: Does the type of verbal feedback they provide matter? Computers & Education, 67, 239\u2013249. https:\/\/doi.org\/10.1016\/j.compedu.2013.04.017","journal-title":"Computers & Education"},{"key":"12911_CR40","unstructured":"Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan. Educational Technology & Society, 15(3), 149\u2013163."},{"issue":"3","key":"12911_CR22","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1080\/00220671.2013.878298","volume":"108","author":"JC Nunez","year":"2015","unstructured":"Nunez, J. C., Su\u00e1rez, N., Ros\u00e1rio, P., Vallejo, G., Cerezo, R., & Valle, A. (2015). Teachers\u2019 feedback on homework, homework-related behaviors, and academic achievement. The Journal of Educational Research, 108(3), 204\u2013216. https:\/\/doi.org\/10.1080\/00220671.2013.878298","journal-title":"The Journal of Educational Research"},{"key":"12911_CR24","doi-asserted-by":"publisher","unstructured":"Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 97\u2013132). Springer. https:\/\/doi.org\/10.1007\/978-1-4614-2018-7_12","DOI":"10.1007\/978-1-4614-2018-7_12"},{"issue":"6","key":"12911_CR25","doi-asserted-by":"publisher","first-page":"1641","DOI":"10.1002\/cae.22547","volume":"30","author":"E Polat","year":"2022","unstructured":"Polat, E., Hopcan, S., Albayrak, E., & Yildiz Durak, H. (2022). Examining the effect of feedback type and gender on computing achievements, engagement, flipped learning readiness, and autonomous learning in online flipped classroom. Computer Applications in Engineering Education, 30(6), 1641\u20131655. https:\/\/doi.org\/10.1002\/cae.22547","journal-title":"Computer Applications in Engineering Education"},{"key":"12911_CR26","doi-asserted-by":"publisher","unstructured":"Polito, S., Koprinska, I., & Jeffries, B. (2022). Exploring Student Engagement in an online programming course using machine learning methods. In M. M. Rodrigo, N. Matsuda, A. I. Cristea, & V. Dimitrova (Eds.), Artificial Intelligence in Education. Posters and late breaking results, Workshops and tutorials, Industry and Innovation Tracks, practitioners\u2019 and Doctoral Consortium. AIED 2022 (Vol. 13356). Springer. Lecture Notes in Computer Sciencehttps:\/\/doi.org\/10.1007\/978-3-031-11647-6_112","DOI":"10.1007\/978-3-031-11647-6_112"},{"issue":"1","key":"12911_CR27","doi-asserted-by":"publisher","first-page":"183","DOI":"10.24059\/olj.v22i1.1175","volume":"22","author":"P Redmond","year":"2018","unstructured":"Redmond, P., Heffernan, A., Abawi, L., Brown, A., & Henderson, R. (2018). An online engagement framework for higher education. Online Learning, 22(1), 183\u2013204. https:\/\/doi.org\/10.24059\/olj.v22i1.1175","journal-title":"Online Learning"},{"key":"12911_CR29","doi-asserted-by":"publisher","first-page":"104934","DOI":"10.1016\/j.compedu.2023.104934","volume":"207","author":"M Saqr","year":"2023","unstructured":"Saqr, M., L\u00f3pez-Pernas, S., & Vogelsmeier, L. V. (2023). When, how and for whom changes in engagement happen: A transition analysis of instructional variables. Computers & Education, 207, 104934. https:\/\/doi.org\/10.1016\/j.compedu.2023.104934","journal-title":"Computers & Education"},{"key":"12911_CR30","doi-asserted-by":"publisher","unstructured":"Stipek, D. (2002). Good instruction is motivating. In Development of achievement motivation (pp. 309\u2013332). Academic Press. https:\/\/doi.org\/10.1016\/B978-012750053-9\/50014-0","DOI":"10.1016\/B978-012750053-9\/50014-0"},{"issue":"2","key":"12911_CR31","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1111\/j.1467-8535.2010.01157.x","volume":"43","author":"JCY Sun","year":"2012","unstructured":"Sun, J. C. Y., & ve Rueda, R. (2012). Situational interest, computer self-efficacy and selfregulation: Their impact on student engagement in distance education. British Journal of Educational Technology, 43(2), 191\u2013204. https:\/\/doi.org\/10.1111\/j.1467-8535.2010.01157.x","journal-title":"British Journal of Educational Technology"},{"key":"12911_CR32","doi-asserted-by":"publisher","first-page":"1071985","DOI":"10.3389\/fpsyg.2023.1071985","volume":"14","author":"A Tlili","year":"2023","unstructured":"Tlili, A., Sun, T., Denden, M., Graf, S., Fei, C., & Wang, H. (2023). Impact of personality traits on learners\u2019 navigational behavior patterns in an online course: A lag sequential analysis approach. Frontiers in Psychology, 14, 1071985. https:\/\/doi.org\/10.3389\/fpsyg.2023.1071985","journal-title":"Frontiers in Psychology"},{"issue":"4","key":"12911_CR34","doi-asserted-by":"publisher","first-page":"804","DOI":"10.1111\/bjet.13223","volume":"53","author":"D Vatsalan","year":"2022","unstructured":"Vatsalan, D., Rakotoarivelo, T., Bhaskar, R., Tyler, P., & Ladjal, D. (2022). Privacy risk quantification in education data using Markov model. British Journal of Educational Technology, 53(4), 804\u2013821. https:\/\/doi.org\/10.1111\/bjet.13223","journal-title":"British Journal of Educational Technology"},{"key":"12911_CR45","doi-asserted-by":"crossref","unstructured":"Vezne, R., Yildiz Durak, H., & Atman Uslu, N. (2023). Online learning in higher education: Examining the predictors of students\u2019 online engagement. Education and Information Technologies, 28(2), 1865\u20131889.","DOI":"10.1007\/s10639-022-11171-9"},{"issue":"4","key":"12911_CR36","doi-asserted-by":"publisher","first-page":"631","DOI":"10.1080\/02602938.2020.1787331","volume":"46","author":"N Winstone","year":"2021","unstructured":"Winstone, N., Bourne, J., Medland, E., Niculescu, I., & Rees, R. (2021). Check the grade, log out: Students\u2019 engagement with feedback in learning management systems. Assessment & Evaluation in Higher Education, 46(4), 631\u2013643. https:\/\/doi.org\/10.1080\/02602938.2020.1787331","journal-title":"Assessment & Evaluation in Higher Education"},{"key":"12911_CR48","doi-asserted-by":"crossref","unstructured":"Yildiz Durak, H. (2022). Flipped classroom model applications in computing courses: Peer-assisted groups, collaborative group and individual learning. Computer Applications in Engineering Education, 30(3), 803\u2013820.","DOI":"10.1002\/cae.22487"},{"key":"12911_CR46","doi-asserted-by":"crossref","unstructured":"Yildiz Durak, H. (2023). Role of personality traits in collaborative group works at flipped classrooms. Current Psychology, 42(15), 13093\u201313113.","DOI":"10.1007\/s12144-022-02702-1"},{"issue":"2","key":"12911_CR38","doi-asserted-by":"publisher","first-page":"410","DOI":"10.1111\/jcal.12887","volume":"40","author":"H Yildiz Durak","year":"2024","unstructured":"Yildiz Durak, H. (2024a). Feedforward-or feedback-based group regulation guidance in collaborative groups. Journal of Computer Assisted Learning, 40(2), 410\u2013436. https:\/\/doi.org\/10.1111\/jcal.12887","journal-title":"Journal of Computer Assisted Learning"},{"key":"12911_CR47","doi-asserted-by":"crossref","unstructured":"Yildiz Durak, H. (2024b). What makes an effective online course experience?: Student perceptions and needs for online course design elements in the context of feedback and collaborative learning. Technology, Knowledge and Learning, 1\u201322.","DOI":"10.1007\/s10758-024-09748-z"}],"container-title":["Education and Information Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10639-024-12911-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10639-024-12911-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10639-024-12911-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,6]],"date-time":"2025-02-06T05:39:20Z","timestamp":1738820360000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10639-024-12911-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,27]]},"references-count":37,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,2]]}},"alternative-id":["12911"],"URL":"https:\/\/doi.org\/10.1007\/s10639-024-12911-9","relation":{},"ISSN":["1360-2357","1573-7608"],"issn-type":[{"value":"1360-2357","type":"print"},{"value":"1573-7608","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,27]]},"assertion":[{"value":"28 February 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 July 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 July 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Participation was voluntary and anonymous. Informed consent was obtained from all participants.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Our data are not yet available online in any institutional database.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Access of data"}},{"value":"The author declares that she has no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interest"}}]}}