{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T18:32:13Z","timestamp":1768415533664,"version":"3.49.0"},"reference-count":43,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,10,29]],"date-time":"2021-10-29T00:00:00Z","timestamp":1635465600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Providing feedback to students is one of the most effective ways to enhance their learning. With the advancement of technology, many tools have been developed to provide personalized feedback. However, these systems are only beneficial when interactions are done on digital platforms. As paper-based assessment is still a dominantly preferred evaluation method, particularly in large blended-instruction classes, the sole use of electronic educational systems presents a gap between how students learn the subject from the physical and digital world. This has motivated the design and the development of a new educational technology that facilitates the digitization, grading, and distribution of paper-based assessments to support blended-instruction classes. With the aid of this technology, different learning analytics can be readily captured. A retrospective analysis was conducted to understand the students\u2019 behaviors in an Object-Oriented Programming and Data Structures class from a public university. Their behavioral differences and the associated learning impacts were analyzed by leveraging their digital footprints. Results showed that students made significant efforts in reviewing their examinations. Notably, the high-achieving and the improving students spent more time reviewing their mistakes and started doing so as soon as the assessment became available. Finally, when students were guided in the reviewing process, they were able to identify items where they had misconceptions.<\/jats:p>","DOI":"10.3390\/info12110450","type":"journal-article","created":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T22:21:34Z","timestamp":1635805294000},"page":"450","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["WebPGA: An Educational Technology That Supports Learning by Reviewing Paper-Based Programming Assessments"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8039-7970","authenticated-orcid":false,"given":"Yancy Vance","family":"Paredes","sequence":"first","affiliation":[{"name":"School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"I-Han","family":"Hsiao","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Santa Clara University, Santa Clara, CA 95053, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1080\/17439880601141179","article-title":"The learning environment in clicker classrooms: Student processes of learning and involvement in large university-level courses using student response systems","volume":"32","author":"Trees","year":"2007","journal-title":"Learn. Media Technol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1007\/s11412-013-9184-1","article-title":"Capturing and analyzing verbal and physical collaborative learning interactions at an enriched interactive tabletop","volume":"8","author":"Dimitriadis","year":"2013","journal-title":"Int. J. Comput.-Support. Collab. Learn."},{"key":"ref_3","unstructured":"Hanna, G.S., and Dettmer, P. (2004). Assessment for Effective Teaching: Using Context-Adaptive Planning, Allyn & Bacon."},{"key":"ref_4","first-page":"103","article-title":"Assessment and classroom learning: A role for summative assessment?","volume":"5","author":"Biggs","year":"1998","journal-title":"Assess. Educ. Princ. Policy Pract."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Yan, Z., and Boud, D. (2021). Conceptualising assessment-as-learning. Assessment as Learning, Routledge.","DOI":"10.4324\/9781003052081-2"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"81","DOI":"10.3102\/003465430298487","article-title":"The power of feedback","volume":"77","author":"Hattie","year":"2007","journal-title":"Rev. Educ. Res."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kulkarni, C.E., Bernstein, M.S., and Klemmer, S.R. (2015, January 14\u201318). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. Proceedings of the Second (2015) ACM Conference on Learning @ Scale, Vancouver, BC, Canada.","DOI":"10.1145\/2724660.2724670"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/BF03395471","article-title":"Provision of feedback during preparation for academic testing: Learning is enhanced by immediate but not delayed feedback","volume":"54","author":"Dihoff","year":"2004","journal-title":"Psychol. Rec."},{"key":"ref_9","unstructured":"Jackson, G.T., and Graesser, A.C. (2007). Content matters: An investigation of feedback categories within an ITS. Front. Artif. Intell. Appl., 158."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1315","DOI":"10.1080\/02602938.2018.1463354","article-title":"The development of student feedback literacy: Enabling uptake of feedback","volume":"43","author":"Carless","year":"2018","journal-title":"Assess. Eval. High. Educ."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Edwards, S.H., and Perez-Quinones, M.A. (2008). Web-CAT: Automatically grading programming assignments. ACM SIGCSE Bulletin, ACM.","DOI":"10.1145\/1384271.1384371"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Jackson, D., and Usher, M. (1997). Grading student programs using ASSYST. ACM SIGCSE Bulletin, ACM.","DOI":"10.1145\/268084.268210"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Hartmann, B., MacDougall, D., Brandt, J., and Klemmer, S.R. (2010, January 10\u201315). What would other programmers do: Suggesting solutions to error messages. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Atlanta, GA, USA.","DOI":"10.1145\/1753326.1753478"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1145\/1597849.1384293","article-title":"Student use of the PeerWise system","volume":"40","author":"Denny","year":"2008","journal-title":"ACM SIGCSE Bull."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1145\/366413.364564","article-title":"Electronic peer review and peer grading in computer-science courses","volume":"33","author":"Gehringer","year":"2001","journal-title":"ACM SIGCSE Bull."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1111\/j.1365-2729.2010.00365.x","article-title":"Guiding students to the right questions: Adaptive navigation support in an E-Learning system for Java programming","volume":"26","author":"Hsiao","year":"2010","journal-title":"J. Comput. Assist. Learn."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Singh, A., Karayev, S., Gutowski, K., and Abbeel, P. (2017, January 20\u201321). Gradescope: A Fast, Flexible, and Fair System for Scalable Assessment of Handwritten Work. Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale, Cambridge, MA, USA.","DOI":"10.1145\/3051457.3051466"},{"key":"ref_18","unstructured":"Guerra, J., Sahebi, S., Lin, Y.R., and Brusilovsky, P. (2021, August 14). The Problem Solving Genome: Analyzing Sequential Patterns of Student Work with Parameterized Exercises. Available online: http:\/\/d-scholarship.pitt.edu\/21805\/."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Piech, C., Sahami, M., Koller, D., Cooper, S., and Blikstein, P. (2012\u20133, January 29). Modeling how students learn to program. Proceedings of the 43rd ACM technical symposium on Computer Science Education, Raleigh, NC, USA.","DOI":"10.1145\/2157136.2157182"},{"key":"ref_20","unstructured":"Lu, Y., and Hsiao, I.H. (July, January 29). Seeking Programming-related Information from Large Scaled Discussion Forums, Help or Harm?. Proceedings of the 9th International Conference on Educational Data Mining, Raleigh, NC, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Altadmri, A., and Brown, N.C. (2015, January 4\u20137). 37 million compilations: Investigating novice programming mistakes in large-scale student data. Proceedings of the 46th ACM Technical Symposium on Computer Science Education, Kansas, MO, USA.","DOI":"10.1145\/2676723.2677258"},{"key":"ref_22","unstructured":"Buffardi, K., and Edwards, S.H. (August, January USA;). Effective and ineffective software testing behaviors by novice programmers. Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research, San Diego, CA."},{"key":"ref_23","first-page":"65","article-title":"Investigating the relationship between dialogue structure and tutoring effectiveness: A hidden Markov modeling approach","volume":"21","author":"Boyer","year":"2011","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Carter, A.S., Hundhausen, C.D., and Adesope, O. (2015, January 9\u201313). The normalized programming state model: Predicting student performance in computing courses based on programming behavior. Proceedings of the Eleventh Annual International Conference on International Computing Education Research, Omaha, NE, USA.","DOI":"10.1145\/2787622.2787710"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"787","DOI":"10.1016\/j.compedu.2007.08.004","article-title":"Intelligent web-based learning system with personalized learning path guidance","volume":"51","author":"Chen","year":"2008","journal-title":"Comput. Educ."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1007\/s11423-007-9064-3","article-title":"Advances in scaffolding learning with hypertext and hypermedia: A summary and critical analysis","volume":"56","author":"Azevedo","year":"2008","journal-title":"Educ. Technol. Res. Dev."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1007\/BF00143964","article-title":"Methods and techniques of adaptive hypermedia","volume":"6","author":"Brusilovsky","year":"1996","journal-title":"User Model. User-Adapt. Interact."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1145\/1163405.1163411","article-title":"Individualized exercises for self-assessment of programming knowledge: An evaluation of QuizPACK","volume":"5","author":"Brusilovsky","year":"2005","journal-title":"J. Educ. Resour. Comput. (JERIC)"},{"key":"ref_29","unstructured":"Conati, C., Heffernan, N., Mitrovic, A., and Verdejo, M.F. (2015). Off the Beaten Path: The Impact of Adaptive Content Sequencing on Student Navigation in an Open Social Student Modeling Interface. Artificial Intelligence in Education, Springer International Publishing."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Conole, G., Klobu\u010dar, T., Rensing, C., Konert, J., and Lavou\u00e9, E. (2015). What Should I Do Next? Adaptive Sequencing in the Context of Open Social Student Modeling. Design for Teaching and Learning in a Networked World, Springer International Publishing.","DOI":"10.1007\/978-3-319-24258-3"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Verbert, K., Sharples, M., and Klobu\u010dar, T. (2016). Mobile Grading Paper-Based Programming Exams: Automatic Semantic Partial Credit Assignment Approach. Adaptive and Adaptable Learning, Springer International Publishing.","DOI":"10.1007\/978-3-319-45153-4"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1016\/j.nedt.2008.10.005","article-title":"Annotation is a valuable tool to enhance learning and assessment in student essays","volume":"29","author":"Ball","year":"2009","journal-title":"Nurse Educ. Today"},{"key":"ref_33","unstructured":"Biggs, J., and Tang, C. (2011). Teaching for Quality Learning at University, McGraw-Hill Education. UK Higher Education OUP Humanities & Social Sciences Higher Education OUP."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1111\/bjet.12752","article-title":"Student access of online feedback is modified by the availability of assessment marks, gender and academic performance","volume":"51","author":"Mensink","year":"2020","journal-title":"Br. J. Educ. Technol."},{"key":"ref_35","first-page":"161","article-title":"Self-explaining expository texts: The dual processes of generating inferences and repairing mental models","volume":"5","author":"Chi","year":"2000","journal-title":"Adv. Instr. Psychol."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"534","DOI":"10.3102\/0034654307309920","article-title":"Understanding Tutor Learning: Knowledge-Building and Knowledge-Telling in Peer Tutors\u2019 Explanations and Questions","volume":"77","author":"Roscoe","year":"2007","journal-title":"Rev. Educ. Res."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1080\/02602938.2020.1787331","article-title":"\u201cCheck the grade, log out\u201d: Students\u2019 engagement with feedback in learning management systems","volume":"46","author":"Winstone","year":"2021","journal-title":"Assess. Eval. High. Educ."},{"key":"ref_38","first-page":"510","article-title":"Learning by Reviewing Paper-Based Programming Assessments","volume":"Volume 11082","author":"Paredes","year":"2018","journal-title":"European Conference on Technology Enhanced Learning, EC-TEL"},{"key":"ref_39","unstructured":"Paredes, Y.V., and Hsiao, I.H. (2018, January 26\u201330). Personalized guidance on how to review paper-based assessments. Proceedings of the 26th International Conference on Computers in Education, Main Conference Proceedings, Manila, Philppines."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1207\/s15326985ep2501_2","article-title":"Self-regulated learning and academic achievement: An overview","volume":"25","author":"Zimmerman","year":"1990","journal-title":"Educ. Psychol."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1080\/02602938.2016.1174187","article-title":"Are they using my feedback? The extent of students\u2019 feedback use has a large impact on subsequent academic performance","volume":"42","author":"Zimbardi","year":"2017","journal-title":"Assess. Eval. High. Educ."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Aleven, V., and Koedinger, K.R. (2000, January 19\u201323). Limitations of student control: Do students know when they need help?. Proceedings of the International Conference on Intelligent Tutoring Systems, Montr\u00e9al, QC, Canada.","DOI":"10.1007\/3-540-45108-0_33"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"698","DOI":"10.1080\/02602938.2012.691462","article-title":"Rethinking models of feedback for learning: The challenge of design","volume":"38","author":"Boud","year":"2013","journal-title":"Assess. Eval. High. Educ."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/11\/450\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:22:56Z","timestamp":1760167376000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/12\/11\/450"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,29]]},"references-count":43,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,11]]}},"alternative-id":["info12110450"],"URL":"https:\/\/doi.org\/10.3390\/info12110450","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,29]]}}}