{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T16:47:48Z","timestamp":1782146868667,"version":"3.54.5"},"publisher-location":"New York, NY, USA","reference-count":62,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,6,29]]},"DOI":"10.1145\/3774398.3811624","type":"proceedings-article","created":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T15:50:37Z","timestamp":1782143437000},"page":"34-45","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Understanding Student Effort Using Response-Time Propensities During Problem Solving"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3437-8979","authenticated-orcid":false,"given":"Conrad","family":"Borchers","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4222-8850","authenticated-orcid":false,"given":"Lijin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8132-0166","authenticated-orcid":false,"given":"Kexin","family":"Yang","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2489-5016","authenticated-orcid":false,"given":"Tomohiro","family":"Nagashima","sequence":"additional","affiliation":[{"name":"Saarland University, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3894-9049","authenticated-orcid":false,"given":"Benjamin W.","family":"Domingue","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,28]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450613.3456834"},{"key":"e_1_3_2_1_2_1","volume-title":"An integrated platform for studying learning with intelligent tutoring systems: CTAT TutorShop. arXiv preprint arXiv:2502.10395","author":"Aleven Vincent","year":"2025","unstructured":"Vincent Aleven, Conrad Borchers, Yun Huang, Tomohiro Nagashima, Bruce McLaren, Paulo Carvalho, Octav Popescu, Jonathan Sewall, and Kenneth Koedinger. 2025. An integrated platform for studying learning with intelligent tutoring systems: CTAT TutorShop. arXiv preprint arXiv:2502.10395 (2025)."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3576050.3576105"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10639-019-10053-x"},{"key":"e_1_3_2_1_5_1","first-page":"185","article-title":"Why students engage in ''gaming the system'' behavior in interactive learning environments","volume":"19","author":"Baker Ryan","year":"2008","unstructured":"Ryan Baker, Jason Walonoski, Neil Heffernan, Ido Roll, Albert Corbett, and Kenneth Koedinger. 2008. Why students engage in ''gaming the system'' behavior in interactive learning environments. Journal of Interactive Learning Research, Vol. 19, 2 (2008), 185-224.","journal-title":"Journal of Interactive Learning Research"},{"key":"e_1_3_2_1_6_1","first-page":"76","volume-title":"Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling","volume":"2007","author":"Jd Baker Ryan","year":"2007","unstructured":"Ryan SJd Baker. 2007. Is gaming the system state-or-trait? Educational data mining through the multi-contextual application of a validated behavioral model. In Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, Vol. 2007. User Modeling Inc Boston, MA, 76-80."},{"key":"e_1_3_2_1_7_1","first-page":"1","article-title":"Challenges for the future of educational data mining: The Baker learning analytics prizes","volume":"11","author":"Baker Ryan S","year":"2019","unstructured":"Ryan S Baker et al., 2019. Challenges for the future of educational data mining: The Baker learning analytics prizes. Journal of educational data mining, Vol. 11, 1 (2019), 1-17.","journal-title":"Journal of educational data mining"},{"key":"e_1_3_2_1_8_1","volume-title":"Jay Raspat, Vincent Aleven, Albert T Corbett, and Kenneth R Koedinger.","author":"Baker Ryan SJD","year":"2009","unstructured":"Ryan SJD Baker, Adriana MJB De Carvalho, Jay Raspat, Vincent Aleven, Albert T Corbett, and Kenneth R Koedinger. 2009. Educational software features that encourage and discourage ''gaming the system''. In Artificial Intelligence in Education. Ios Press, 475-482."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v067.i01"},{"key":"e_1_3_2_1_10_1","volume-title":"European conference on technology enhanced learning. Springer, 44-59","author":"Borchers Conrad","year":"2023","unstructured":"Conrad Borchers, Paulo F Carvalho, Meng Xia, Pinyang Liu, Kenneth R Koedinger, and Vincent Aleven. 2023. What makes problem-solving practice effective? Comparing paper and AI tutoring. In European conference on technology enhanced learning. Springer, 44-59."},{"key":"e_1_3_2_1_11_1","volume-title":"Toward Trait-Aware Learning Analytics. arXiv preprint arXiv:2602.00018","author":"Borchers Conrad","year":"2026","unstructured":"Conrad Borchers, Hannah Deininger, and Zachary A Pardos. 2026. Toward Trait-Aware Learning Analytics. arXiv preprint arXiv:2602.00018 (2026)."},{"key":"e_1_3_2_1_12_1","volume-title":"Evaluating Playful Interactions and Narratives for Algebra Learning. In European Conference on Technology Enhanced Learning. Springer, 76-90","author":"Borchers Conrad","year":"2025","unstructured":"Conrad Borchers, Tomohiro Nagashima, Pinyang Liu, Martha W Alibali, and Vincent Aleven. 2025. Is More Gamification Better? Evaluating Playful Interactions and Narratives for Algebra Learning. In European Conference on Technology Enhanced Learning. Springer, 76-90."},{"key":"e_1_3_2_1_13_1","volume-title":"Proceedings of the 17th International Conference on Educational Data Mining. 117-130","author":"Borchers Conrad","year":"2024","unstructured":"Conrad Borchers, Kexin Yang, Jionghao Lin, Nikol Rummel, Kenneth R Koedinger, and Vincent Aleven. 2024. Combining dialog acts and skill modeling: What chat interactions enhance learning rates during ai-supported peer tutoring?. In Proceedings of the 17th International Conference on Educational Data Mining. 117-130."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11145-021-10174-4"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.lindif.2021.102109"},{"key":"e_1_3_2_1_16_1","volume-title":"Modeling student knowledge: Cognitive tutors in high school and college. User modeling and user-adapted interaction","author":"Corbett Albert","year":"2000","unstructured":"Albert Corbett, Megan McLaughlin, and K Christine Scarpinatto. 2000. Modeling student knowledge: Cognitive tutors in high school and college. User modeling and user-adapted interaction, Vol. 10, 2 (2000), 81-108."},{"key":"e_1_3_2_1_17_1","volume-title":"Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction","author":"Corbett Albert T","year":"1994","unstructured":"Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, Vol. 4, 4 (1994), 253-278."},{"key":"e_1_3_2_1_18_1","unstructured":"Steve C. Dang. 2022. Exploring Behavioral Measurement Models of Learner Motivation. Ph.D. Dissertation. Carnegie Mellon University."},{"key":"e_1_3_2_1_19_1","volume-title":"The Ebb and Flow of Student Engagement: Measuring Motivation through Temporal Pattern of Self-Regulation","author":"Dang Steven C","year":"2020","unstructured":"Steven C Dang and Kenneth R Koedinger. 2020. The Ebb and Flow of Student Engagement: Measuring Motivation through Temporal Pattern of Self-Regulation. International Educational Data Mining Society (2020)."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"crossref","first-page":"576","DOI":"10.3102\/10769986221099906","article-title":"Speed-accuracy trade-off? Not so fast: Marginal changes in speed have inconsistent relationships with accuracy in real-world settings","volume":"47","author":"Domingue Benjamin W","year":"2022","unstructured":"Benjamin W Domingue, Klint Kanopka, Ben Stenhaug, Michael J Sulik, Tanesia Beverly, Matthieu Brinkhuis, Ruhan Circi, Jessica Faul, Dandan Liao, Bruce McCandliss, et al., 2022. Speed-accuracy trade-off? Not so fast: Marginal changes in speed have inconsistent relationships with accuracy in real-world settings. Journal of Educational and Behavioral Statistics, Vol. 47, 5 (2022), 576-602.","journal-title":"Journal of Educational and Behavioral Statistics"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"crossref","first-page":"e2507708123","DOI":"10.1073\/pnas.2507708123","article-title":"Computer-assisted learning in the real world: How Khan Academy influences student math learning","volume":"123","author":"Eames Taryn","year":"2026","unstructured":"Taryn Eames, Emma Brunskill, Bogdan Yamkovenko, Kodi Weatherholtz, and Philip Oreopoulos. 2026. Computer-assisted learning in the real world: How Khan Academy influences student math learning. Proceedings of the National Academy of Sciences, Vol. 123, 1 (2026), e2507708123.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjep.12563"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448139.3448167"},{"key":"e_1_3_2_1_24_1","volume-title":"Starting Seatwork Earlier as a Valid Measure of Student Engagement. arXiv preprint arXiv:2505.13341","author":"Gurung Ashish","year":"2025","unstructured":"Ashish Gurung, Jionghao Lin, Zhongtian Huang, Conrad Borchers, Ryan S Baker, Vincent Aleven, and Kenneth R Koedinger. 2025. Starting Seatwork Earlier as a Valid Measure of Student Engagement. arXiv preprint arXiv:2505.13341 (2025)."},{"key":"e_1_3_2_1_25_1","volume-title":"The 5 percent problem. Education Next","author":"Holt LAURENCE","year":"2024","unstructured":"LAURENCE Holt. 2024. The 5 percent problem. Education Next (2024)."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-023-09362-1"},{"key":"e_1_3_2_1_27_1","first-page":"5","article-title":"How does the number of actions on constructed-response items relate to test-taking effort and performance","volume":"26","author":"Ivanova Militsa","year":"2020","unstructured":"Militsa Ivanova, Michalis Michaelides, and Hanna Ekl\u00f6f. 2020. How does the number of actions on constructed-response items relate to test-taking effort and performance? Educational Research and Evaluation, Vol. 26, 5-6 (2020), 252-274.","journal-title":"Educational Research and Evaluation"},{"key":"e_1_3_2_1_28_1","volume-title":"Michaelides","author":"Ivanova Militsa G.","year":"2023","unstructured":"Militsa G. Ivanova and Michalis P. Michaelides. 2023. Measuring Test-Taking Effort on Constructed-Response Items with Item Response Time and Number of Actions. Practical Assessment, Research & Evaluation, Vol. 28, 15 (2023)."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.4018\/978-1-60566-048-6.ch003"},{"key":"e_1_3_2_1_30_1","volume-title":"Exploring the assistance dilemma in experiments with cognitive tutors. Educational psychology review","author":"Koedinger Kenneth R","year":"2007","unstructured":"Kenneth R Koedinger and Vincent Aleven. 2007. Exploring the assistance dilemma in experiments with cognitive tutors. Educational psychology review, Vol. 19, 3 (2007), 239-264."},{"key":"e_1_3_2_1_31_1","volume-title":"Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper.","author":"Koedinger Kenneth R","year":"2010","unstructured":"Kenneth R Koedinger, Ryan SJd Baker, Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper. 2010. A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining, Vol. 43 (2010), 43-56."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2221311120"},{"key":"e_1_3_2_1_33_1","volume-title":"The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science","author":"Koedinger Kenneth R","year":"2012","unstructured":"Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science, Vol. 36, 5 (2012), 757-798."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2724660.2724681"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1177\/0013164406294779"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2015.23.6"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723576.2723623"},{"key":"e_1_3_2_1_38_1","first-page":"130","article-title":"Prediction of Student Performance in Massive Open Online Courses Using Deep Learning System Based on Learning Behaviors","volume":"24","author":"Lee Chia-An","year":"2021","unstructured":"Chia-An Lee, Jian-Wei Tzeng, Nen-Fu Huang, and Yu-Sheng Su. 2021. Prediction of Student Performance in Massive Open Online Courses Using Deep Learning System Based on Learning Behaviors. Educational Technology & Society, Vol. 24, 3 (2021), 130-146.","journal-title":"Educational Technology & Society"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/s11409-025-09437-x","article-title":"Unpacking strategy efficiency: Examining the relations between pre-solving pause time and productivity in a digital mathematics game","volume":"20","author":"Lee Ji-Eun","year":"2025","unstructured":"Ji-Eun Lee, Carly Siegel Thorp, Arba Kamberi, and Erin Ottmar. 2025. Unpacking strategy efficiency: Examining the relations between pre-solving pause time and productivity in a digital mathematics game. Metacognition and Learning, Vol. 20, 1 (2025), 31.","journal-title":"Metacognition and Learning"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2022.104517"},{"key":"e_1_3_2_1_41_1","volume-title":"Towards Reliable and Valid Measurement of Individualized Student Parameters","author":"Liu Ran","year":"2017","unstructured":"Ran Liu and Kenneth RK Koedinger. 2017. Towards Reliable and Valid Measurement of Individualized Student Parameters. International Educational Data Mining Society (2017)."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.3389\/feduc.2020.607260"},{"key":"e_1_3_2_1_43_1","volume-title":"The development of goal setting theory: A half century retrospective. Motivation science","author":"Locke Edwin A","year":"2019","unstructured":"Edwin A Locke and Gary P Latham. 2019. The development of goal setting theory: A half century retrospective. Motivation science, Vol. 5, 2 (2019), 93."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170358.3170411"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1080\/03323315.2018.1512889","article-title":"Exploring the relationship between homework task difficulty, student engagement and performance","volume":"38","author":"Lynch Raymond","year":"2019","unstructured":"Raymond Lynch, Adrian Hurley, Olivia Cumiskey, Brian Nolan, and Bridgeen McGlynn. 2019. Exploring the relationship between homework task difficulty, student engagement and performance. Irish Educational Studies, Vol. 38, 1 (2019), 89-103.","journal-title":"Irish Educational Studies"},{"key":"e_1_3_2_1_46_1","volume-title":"Proceedings of the 33rd International Florida Artificial Intelligence Research Society Conference. Association for the Advancement of Artificial Intelligence, 439-444","author":"Mirzaei Mehrdad","year":"2020","unstructured":"Mehrdad Mirzaei, Shaghayegh Sahebi, and Peter Brusilovsky. 2020. Detecting Trait versus Performance Student Behavioral Patterns Using Discriminative Non-Negative Matrix Factorization. In Proceedings of the 33rd International Florida Artificial Intelligence Research Society Conference. Association for the Advancement of Artificial Intelligence, 439-444."},{"key":"e_1_3_2_1_47_1","volume-title":"Philippines: Teachers College","author":"Jaclyn Ocumpaugh","year":"2015","unstructured":"Jaclyn Ocumpaugh et al., 2015. Baker Rodrigo Ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual. New York, NY and Manila, Philippines: Teachers College, Columbia University and Ateneo Laboratory for the Learning Sciences, Vol. 60 (2015)."},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2017.00422"},{"key":"e_1_3_2_1_49_1","volume-title":"International Conference on Intelligent Tutoring Systems. Springer, 267-272","author":"Papou\u0161ek Jan","year":"2016","unstructured":"Jan Papou\u0161ek, V\u00edt Stanislav, and Radek Pel\u00e1nek. 2016. Impact of question difficulty on engagement and learning. In International Conference on Intelligent Tutoring Systems. Springer, 267-272."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.lindif.2025.102744"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.iree.2013.03.001","article-title":"Utilizing online tools to measure effort: Does it really improve student outcome","volume":"14","author":"Self Sharmistha","year":"2013","unstructured":"Sharmistha Self. 2013. Utilizing online tools to measure effort: Does it really improve student outcome? International Review of Economics Education, Vol. 14 (2013), 36-45.","journal-title":"International Review of Economics Education"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1111\/mbe.12110"},{"key":"e_1_3_2_1_53_1","volume-title":"Adoption of data analytics in higher education learning and teaching","author":"Tempelaar Dirk","unstructured":"Dirk Tempelaar, Quan Nguyen, and Bart Rienties. 2020. Learning analytics and the measurement of learning engagement. In Adoption of data analytics in higher education learning and teaching. Springer, 159-176."},{"key":"e_1_3_2_1_54_1","volume-title":"New horizons in testing","author":"Thissen David","unstructured":"David Thissen. 1983. Timed testing: An approach using item response theory. In New horizons in testing. Elsevier, 179-203."},{"key":"e_1_3_2_1_55_1","first-page":"637","volume-title":"Proceedings of the 19th International Conference of the Learning Sciences-ICLS","author":"Vanacore Kirk","year":"2025","unstructured":"Kirk Vanacore, Amanda Barany, Shruti Mehta, Jaclyn Ocumpaugh, Ryan S Baker, Nabil Al Nahin, Vishal Kuvar, Caitlin Mills, and Owen Henkel. 2025. Downshifting: The Nuanced Antecedents of Seeking Easier Content. In Proceedings of the 19th International Conference of the Learning Sciences-ICLS 2025, pp. 637-645. International Society of the Learning Sciences."},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3576050.3576073"},{"key":"e_1_3_2_1_57_1","volume-title":"The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational psychologist","author":"VanLehn Kurt","year":"2011","unstructured":"Kurt VanLehn. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational psychologist, Vol. 46, 4 (2011), 197-221."},{"key":"e_1_3_2_1_58_1","volume-title":"Using response times and response accuracy to measure fluency within cognitive diagnosis models. psychometrika","author":"Wang Shiyu","year":"2020","unstructured":"Shiyu Wang and Yinghan Chen. 2020. Using response times and response accuracy to measure fluency within cognitive diagnosis models. psychometrika, Vol. 85, 3 (2020), 600-629."},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15324818ame1802_2"},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1037\/xlm0001026"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.learninstruc.2019.05.011"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-012109890-2\/50031-710.1016\/B978-012109890-2\/50031-7"}],"event":{"name":"L@S '26: Thirteenth ACM Conference on Learning @ Scale","location":"Seoul Republic of Korea"},"container-title":["Proceedings of the Thirteenth ACM Conference on Learning @ Scale"],"original-title":[],"deposited":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T15:53:21Z","timestamp":1782143601000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3774398.3811624"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,28]]},"references-count":62,"alternative-id":["10.1145\/3774398.3811624","10.1145\/3774398"],"URL":"https:\/\/doi.org\/10.1145\/3774398.3811624","relation":{},"subject":[],"published":{"date-parts":[[2026,6,28]]},"assertion":[{"value":"2026-06-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}