{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T22:29:52Z","timestamp":1791412192539,"version":"4.3.3"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2015,3,10]],"date-time":"2015-03-10T00:00:00Z","timestamp":1425945600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF Expeditions in Computing","award":["CCF 1138996"],"award-info":[{"award-number":["CCF 1138996"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2015,4,15]]},"abstract":"<jats:p>In computer-aided education, the goal of automatic feedback is to provide a meaningful explanation of students' mistakes. We focus on providing feedback for constructing a deterministic finite automaton that accepts strings that match a described pattern. Natural choices for feedback are binary feedback (correct\/wrong) and a counterexample of a string that is processed incorrectly. Such feedback is easy to compute but might not provide the student enough help. Our first contribution is a novel way to automatically compute alternative conceptual hints. Our second contribution is a rigorous evaluation of feedback with 377 students. We find that providing either counterexamples or hints is judged as helpful, increases student perseverance, and can improve problem completion time. However, both strategies have particular strengths and weaknesses. Since our feedback is completely automatic, it can be deployed at scale and integrated into existing massive open online courses.<\/jats:p>","DOI":"10.1145\/2723163","type":"journal-article","created":{"date-parts":[[2015,3,12]],"date-time":"2015-03-12T08:18:05Z","timestamp":1426148285000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":40,"title":["How Can Automatic Feedback Help Students Construct Automata?"],"prefix":"10.1145","volume":"22","author":[{"given":"Loris","family":"D'antoni","sequence":"first","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dileep","family":"Kini","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rajeev","family":"Alur","sequence":"additional","affiliation":[{"name":"University of Pennsylvania, Philadelphia, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sumit","family":"Gulwani","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahesh","family":"Viswanathan","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bj\u00f6rn","family":"Hartmann","sequence":"additional","affiliation":[{"name":"UC Berkeley, Berkeley, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2015,3,10]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI'13)","author":"Ahmed Umair Z.","year":"2013","unstructured":"Umair Z. Ahmed , Sumit Gulwani , and Amey Karkare . 2013 . Automatically generating problems and solutions for natural deduction . In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI'13) . 1968--1975. http:\/\/dl.acm.org\/citation.cfm&quest;id=2540128.2540411. Umair Z. Ahmed, Sumit Gulwani, and Amey Karkare. 2013. Automatically generating problems and solutions for natural deduction. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI'13). 1968--1975. http:\/\/dl.acm.org\/citation.cfm&quest;id=2540128.2540411."},{"key":"e_1_2_2_2_1","volume-title":"Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI'13)","author":"Alur Rajeev","year":"2013","unstructured":"Rajeev Alur , Loris D'Antoni , Sumit Gulwani , Dileep Kini , and Mahesh Viswanathan . 2013 . Automated grading of DFA constructions . In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI'13) . 1976--1982. http:\/\/dl.acm.org\/citation.cfm&quest;id=2540128.2540412 Rajeev Alur, Loris D'Antoni, Sumit Gulwani, Dileep Kini, and Mahesh Viswanathan. 2013. Automated grading of DFA constructions. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI'13). 1976--1982. http:\/\/dl.acm.org\/citation.cfm&quest;id=2540128.2540412"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470764"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327809jls0402_2"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/3351.3354"},{"key":"e_1_2_2_6_1","volume-title":"Error Patterns in Computation: A Semi-Programmed Approach","author":"Ashlock Robert B.","unstructured":"Robert B. Ashlock . 1986. Error Patterns in Computation: A Semi-Programmed Approach . Merrill Publishing Company . Robert B. Ashlock. 1986. Error Patterns in Computation: A Semi-Programmed Approach. Merrill Publishing Company."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tcs.2004.12.030"},{"key":"e_1_2_2_8_1","series-title":"Lecture Notes in Computer Science","volume-title":"LATIN 2004: Theoretical Informatics","author":"Bodirsky Manuel","unstructured":"Manuel Bodirsky , Tobias Gartner , Timo von Oertzen , and Jan Schwinghammer . 2004. Efficiently computing the density of regular languages . In LATIN 2004: Theoretical Informatics . Lecture Notes in Computer Science , Vol. 2976 . Springer , 262--270. DOI:http:\/\/dx.doi.org\/openurl.asp&quest;genre=article&issn=0302-9743&volume=2976&spage=262 Manuel Bodirsky, Tobias Gartner, Timo von Oertzen, and Jan Schwinghammer. 2004. Efficiently computing the density of regular languages. In LATIN 2004: Theoretical Informatics. Lecture Notes in Computer Science, Vol. 2976. Springer, 262--270. DOI:http:\/\/dx.doi.org\/openurl.asp&quest;genre=article&issn=0302-9743&volume=2976&spage=262"},{"key":"e_1_2_2_9_1","volume-title":"Automata ImplementationLecture Notes in Computer Science","author":"Braune Beatrix","unstructured":"Beatrix Braune , Stephan Diehl , Andreas Kerren , and Reinhard Wilhelm . 2001. Animation of the generation and computation of finite automata for learning software . In Automata ImplementationLecture Notes in Computer Science , Vol. 2214 . Springer , 39--47. Beatrix Braune, Stephan Diehl, Andreas Kerren, and Reinhard Wilhelm. 2001. Animation of the generation and computation of finite automata for learning software. In Automata ImplementationLecture Notes in Computer Science, Vol. 2214. Springer, 39--47."},{"key":"e_1_2_2_10_1","volume-title":"Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI'09)","author":"Conati Cristina","year":"2009","unstructured":"Cristina Conati . 2009 . Intelligent tutoring systems: New challenges and directions . In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI'09) . 2--7. http:\/\/dl.acm.org\/ citation.cfm&quest;id=1661445.1661447 Cristina Conati. 2009. Intelligent tutoring systems: New challenges and directions. In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI'09). 2--7. http:\/\/dl.acm.org\/ citation.cfm&quest;id=1661445.1661447"},{"key":"e_1_2_2_11_1","series-title":"Lecture Notes in Computer Science","volume-title":"Intelligent Tutoring Systems","author":"Devedzic Vladan","unstructured":"Vladan Devedzic and John Debenham . 1998. An intelligent tutoring system for teaching formal languages . In Intelligent Tutoring Systems . Lecture Notes in Computer Science , Vol. 1452 . Springer , 514--523. DOI:http:\/\/dx.doi.org\/10.1007\/3-540-68716-5_57 10.1007\/3-540-68716-5_57 Vladan Devedzic and John Debenham. 1998. An intelligent tutoring system for teaching formal languages. In Intelligent Tutoring Systems. Lecture Notes in Computer Science, Vol. 1452. Springer, 514--523. DOI:http:\/\/dx.doi.org\/10.1007\/3-540-68716-5_57"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2502028"},{"key":"e_1_2_2_13_1","unstructured":"Tara Gallien and Jody Oomen-Early. 2008. Personalized versus collective instructor feedback in the online courseroom: Does type of feedback affect student satisfaction academic performance and perceived connectedness with the instructor&quest; International Journal on E-Learning 7 3 463--476. http:\/\/www.editlib.org\/p\/23582  Tara Gallien and Jody Oomen-Early. 2008. Personalized versus collective instructor feedback in the online courseroom: Does type of feedback affect student satisfaction academic performance and perceived connectedness with the instructor&quest; International Journal on E-Learning 7 3 463--476. http:\/\/www.editlib.org\/p\/23582"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10044-008-0141-y"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993316.1993505"},{"key":"e_1_2_2_16_1","series-title":"Lecture Notes in Computer Science","volume-title":"Logic for Programming, Artificial Intelligence, and Reasoning","author":"Itzhaky Shachar","unstructured":"Shachar Itzhaky , Sumit Gulwani , Neil Immerman , and Mooly Sagiv . 2013. Solving geometry problems using a combination of symbolic and numerical reasoning . In Logic for Programming, Artificial Intelligence, and Reasoning . Lecture Notes in Computer Science , Vol. 8312 . Springer , 457--472. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-45221-5_31 10.1007\/978-3-642-45221-5_31 Shachar Itzhaky, Sumit Gulwani, Neil Immerman, and Mooly Sagiv. 2013. Solving geometry problems using a combination of symbolic and numerical reasoning. In Logic for Programming, Artificial Intelligence, and Reasoning. Lecture Notes in Computer Science, Vol. 8312. Springer, 457--472. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-45221-5_31"},{"key":"e_1_2_2_17_1","volume-title":"MOOC Completion Rates: The Data. Retrieved","author":"Jordan Katy","year":"2015","unstructured":"Katy Jordan . 2014. MOOC Completion Rates: The Data. Retrieved February 4, 2015 , from http:\/\/www.katyjordan.com\/MOOCproject.html. Katy Jordan. 2014. MOOC Completion Rates: The Data. Retrieved February 4, 2015, from http:\/\/www.katyjordan.com\/MOOCproject.html."},{"key":"e_1_2_2_18_1","volume-title":"Corbett","author":"Koedinger Kenneth R.","year":"2006","unstructured":"Kenneth R. Koedinger and Albert T . Corbett . 2006 . Cognitive tutors: Technology bringing learning science to the classroom. In The Cambridge Handbook of the Learning Sciences, K. Sawyer (Ed.). Cambridge University Press , 61--78. Kenneth R. Koedinger and Albert T. Corbett. 2006. Cognitive tutors: Technology bringing learning science to the classroom. In The Cambridge Handbook of the Learning Sciences, K. Sawyer (Ed.). Cambridge University Press, 61--78."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcs.2005.06.023"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ic.2008.03.019"},{"key":"e_1_2_2_21_1","first-page":"129","article-title":"Evaluation methodologies for intelligent tutoring systems","volume":"4","author":"Mark Mary A.","year":"1993","unstructured":"Mary A. Mark and Jim E. Greer . 1993 . Evaluation methodologies for intelligent tutoring systems . Journal of Artificial Intelligence in Education 4 , 129 -- 153 . Mary A. Mark and Jim E. Greer. 1993. Evaluation methodologies for intelligent tutoring systems. Journal of Artificial Intelligence in Education 4, 129--153.","journal-title":"Journal of Artificial Intelligence in Education"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/273133.274318"},{"key":"e_1_2_2_23_1","series-title":"Lecture Notes in Computer Science","volume-title":"Engineering of Intelligent Systems","author":"Mitrovic Antonija","unstructured":"Antonija Mitrovic , Michael Mayo , Pramuditha Suraweera , and Brent Martin . 2001. Constraint-based tutors: A success story . In Engineering of Intelligent Systems . Lecture Notes in Computer Science , Vol. 2070 . Springer , 931--940. http:\/\/dl.acm.org\/citation.cfm&quest;id=646863.707788 Antonija Mitrovic, Michael Mayo, Pramuditha Suraweera, and Brent Martin. 2001. Constraint-based tutors: A success story. In Engineering of Intelligent Systems. Lecture Notes in Computer Science, Vol. 2070. Springer, 931--940. http:\/\/dl.acm.org\/citation.cfm&quest;id=646863.707788"},{"key":"e_1_2_2_24_1","volume-title":"Why Are English Majors Studying Computer Science&quest","author":"Patterson David","year":"2015","unstructured":"David Patterson . 2013. Why Are English Majors Studying Computer Science&quest ; Retrieved February 4, 2015 , from http:\/\/blogs.berkeley.edu\/2013\/11\/26\/why-are-english-majors-studying-computer-science\/. David Patterson. 2013. Why Are English Majors Studying Computer Science&quest; Retrieved February 4, 2015, from http:\/\/blogs.berkeley.edu\/2013\/11\/26\/why-are-english-majors-studying-computer-science\/."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/11774303_63"},{"key":"e_1_2_2_26_1","volume-title":"Rodger and Thomas Finley","author":"Susan","year":"2006","unstructured":"Susan H. Rodger and Thomas Finley . 2006 . JFLAP : An Interactive Formal Languages and Automata Package. Jones and Bartlett . Susan H. Rodger and Thomas Finley. 2006. JFLAP: An Interactive Formal Languages and Automata Package. Jones and Bartlett."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC3.2014.6897141"},{"key":"e_1_2_2_28_1","first-page":"2","article-title":"Principles for evaluating intelligent tutoring systems","volume":"4","author":"Shute Valerie J.","year":"1993","unstructured":"Valerie J. Shute and J. Wesley Regian . 1993 . Principles for evaluating intelligent tutoring systems . Journal of Artificial Intelligence in Education 4 , 2 -- 3 , 245--271. Valerie J. Shute and J. Wesley Regian. 1993. Principles for evaluating intelligent tutoring systems. Journal of Artificial Intelligence in Education 4, 2--3, 245--271.","journal-title":"Journal of Artificial Intelligence in Education"},{"key":"e_1_2_2_29_1","volume-title":"Proceedings of the 26th Conference on Artificial Intelligence (AAAI-12)","author":"Singh Rohit","unstructured":"Rohit Singh , Sumit Gulwani , and Sriram K. Rajamani . 2012. Automatically generating algebra problems . In Proceedings of the 26th Conference on Artificial Intelligence (AAAI-12) . Rohit Singh, Sumit Gulwani, and Sriram K. Rajamani. 2012. Automatically generating algebra problems. In Proceedings of the 26th Conference on Artificial Intelligence (AAAI-12)."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462195"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1269900.1268801"},{"key":"e_1_2_2_32_1","volume-title":"Instruction for Masses Knocks Down Campus Walls. Retrieved","author":"Times New York","year":"2015","unstructured":"New York Times . 2012. Instruction for Masses Knocks Down Campus Walls. Retrieved February 4, 2015 , from http:\/\/www.nytimes.com\/2012\/03\/05\/education\/moocs-large-courses-open-to-all-topple-campus-walls.html&quest;pagewanted=all New York Times. 2012. Instruction for Masses Knocks Down Campus Walls. Retrieved February 4, 2015, from http:\/\/www.nytimes.com\/2012\/03\/05\/education\/moocs-large-courses-open-to-all-topple-campus-walls.html&quest;pagewanted=all"},{"key":"e_1_2_2_33_1","first-page":"329","article-title":"Mind Bugs: The origins of procedural misconceptions","volume":"52","author":"VanLehn Kurt","year":"1992","unstructured":"Kurt VanLehn . 1992 . Mind Bugs: The origins of procedural misconceptions . Artificial Intelligence 52 , 3, 329 -- 340 . Kurt VanLehn. 1992. Mind Bugs: The origins of procedural misconceptions. Artificial Intelligence 52, 3, 329--340.","journal-title":"Artificial Intelligence"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/1435351.1435353"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1080\/00461520.2011.611369"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723163","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2723163","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T15:03:57Z","timestamp":1750259037000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723163"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,3,10]]},"references-count":35,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2015,4,15]]}},"alternative-id":["10.1145\/2723163"],"URL":"https:\/\/doi.org\/10.1145\/2723163","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,3,10]]},"assertion":[{"value":"2014-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-03-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}