{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:01:54Z","timestamp":1772906514709,"version":"3.50.1"},"reference-count":42,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2009,9,16]],"date-time":"2009-09-16T00:00:00Z","timestamp":1253059200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2009,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper describes a new method for recognizing whether a student's response to an automated tutor's question entails that they understand the concepts being taught. We demonstrate the need for a finer-grained analysis of answers than is supported by current tutoring systems or entailment databases and describe a new representation for reference answers that addresses these issues, breaking them into detailed facets and annotating their entailment relationships to the student's answer more precisely. Human annotation at this detailed level still results in substantial interannotator agreement (86.2%), with a kappa statistic of 0.728. We also present our current efforts to automatically assess student answers, which involves training machine learning classifiers on features extracted from dependency parses of the reference answer and student's response and features derived from domain-independent lexical statistics. Our system's performance, as high as 75.5% accuracy within domain and 68.8% out of domain, is very encouraging and confirms the approach is feasible. Another significant contribution of this work is that it represents a significant step in the direction of providing domain-independent semantic assessment of answers. No prior work in the area of tutoring or educational assessment has attempted to build such domain-independent systems. They have virtually all required hundreds of examples of learner answers for each new question in order to train aspects of their systems or to hand-craft information extraction templates.<\/jats:p>","DOI":"10.1017\/s135132490999012x","type":"journal-article","created":{"date-parts":[[2009,9,16]],"date-time":"2009-09-16T06:41:33Z","timestamp":1253083293000},"page":"479-501","source":"Crossref","is-referenced-by-count":45,"title":["Recognizing entailment in intelligent tutoring systems"],"prefix":"10.1017","volume":"15","author":[{"given":"RODNEY D.","family":"NIELSEN","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"WAYNE","family":"WARD","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"JAMES H.","family":"MARTIN","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2009,9,16]]},"reference":[{"key":"S135132490999012X_ref12","volume-title":"Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing","author":"Giampiccolo","year":"2007"},{"key":"S135132490999012X_ref21","volume-title":"Full Option Science System (FOSS)","year":"2005"},{"key":"S135132490999012X_ref32","volume-title":"Proceedings of the HLT\/NAACL","author":"Pang","year":"2003"},{"key":"S135132490999012X_ref35","volume-title":"Proceedings of the PASCAL RTE Challenge Workshop","author":"Raina","year":"2005"},{"key":"S135132490999012X_ref33","volume-title":"Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies","author":"Pon-Barry","year":"2004"},{"key":"S135132490999012X_ref24","first-page":"167","volume-title":"Assessing science learning: Perspectives from research and practice","author":"Long","year":"2008"},{"key":"S135132490999012X_ref34","volume-title":"C4.5: Programs for Machine Learning","author":"Quinlan","year":"1993"},{"key":"S135132490999012X_ref27","volume-title":"Proceedings of 6th International Computer Aided Assessment Conference","author":"Mitchell","year":"2002"},{"key":"S135132490999012X_ref39","volume-title":"Proceedings of the 29th Conference of the International Association for Educational Assessment","author":"Sukkarieh","year":"2003"},{"key":"S135132490999012X_ref11","volume-title":"Proceedings of COLING 2004","author":"Dolan","year":"2004"},{"key":"S135132490999012X_ref42","volume-title":"Proceedings of the PASCAL Workshop for Recognizing Textual Entailment","author":"Vanderwende","year":"2005"},{"key":"S135132490999012X_ref40","volume-title":"Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing","author":"Tatu","year":"2007"},{"key":"S135132490999012X_ref2","volume-title":"IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems","author":"Aleven","year":"2001"},{"key":"S135132490999012X_ref6","volume-title":"Proceedings of the PASCAL Recognizing Textual Entailment Challenge Workshop","author":"Braz","year":"2005"},{"key":"S135132490999012X_ref19","volume-title":"AAAI Seventeenth National Conference on Artificial Intelligence","author":"Kipper","year":"2000"},{"key":"S135132490999012X_ref23","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324901002765"},{"key":"S135132490999012X_ref36","volume-title":"Proceedings of the 40th ACL Conference","author":"Ravichandran","year":"2002"},{"key":"S135132490999012X_ref29","volume-title":"Proceedings of the 2nd PASCAL RTE Challenge Workshop","author":"Nielsen","year":"2006"},{"key":"S135132490999012X_ref14","volume-title":"Proceedings of RANLP","author":"Glickman","year":"2003"},{"key":"S135132490999012X_ref10","volume-title":"Proceedings of the PASCAL RTE Challenge Workshop","author":"Dagan","year":"2005"},{"key":"S135132490999012X_ref4","first-page":"16","volume-title":"Proceedings of the HLT-NAACL","author":"Barzilay","year":"2003"},{"key":"S135132490999012X_ref5","first-page":"50","volume-title":"Proceedings of the ACL\/EACL","author":"Barzilay","year":"2001"},{"key":"S135132490999012X_ref3","volume-title":"Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment","author":"Bar-Haim","year":"2005"},{"key":"S135132490999012X_ref25","volume-title":"Proceedings of HLT-NAACL","author":"MacCartney","year":"2006"},{"key":"S135132490999012X_ref26","first-page":"187","article-title":"Abductive theorem proving for analyzing student explanations and guiding feedback in intelligent tutoring systems","volume":"32","author":"Makatchev","year":"2004","journal-title":"Journal of Automated Reasoning (special issue on automated reasoning and theorem proving in education)"},{"key":"S135132490999012X_ref9","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"S135132490999012X_ref37","doi-asserted-by":"publisher","DOI":"10.3115\/1118894.1118904"},{"key":"S135132490999012X_ref28","volume-title":"Proceedings of the ACL Workshop on Textual Entailment and Paraphrasing","author":"Nielsen","year":"2007"},{"key":"S135132490999012X_ref38","volume-title":"Proceedings of HLT","author":"Sudo","year":"2001"},{"key":"S135132490999012X_ref41","first-page":"491","volume-title":"Proceedings of 12th European Conference on Machine Learning","author":"Turney","year":"2001"},{"key":"S135132490999012X_ref15","volume-title":"Proceedings of the PASCAL RTE Challenge Workshop","author":"Glickman","year":"2005"},{"key":"S135132490999012X_ref7","doi-asserted-by":"publisher","DOI":"10.3115\/1631862.1631871"},{"key":"S135132490999012X_ref16","first-page":"47","volume-title":"Proceedings of the 10th International Conference of Artificial Intelligence in Education","author":"Graesser","year":"2001"},{"key":"S135132490999012X_ref18","volume-title":"Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing","author":"Hickl","year":"2007"},{"key":"S135132490999012X_ref31","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1017\/S1351324906004505","article-title":"MaltParser: a language-independent system for data-driven dependency parsing","volume":"13","author":"Nivre","year":"2007","journal-title":"Natural Language Engineering"},{"key":"S135132490999012X_ref17","first-page":"43","volume-title":"Syntax and Semantics, Vol 3, Speech Acts","author":"Grice","year":"1975"},{"key":"S135132490999012X_ref20","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.104.2.211"},{"key":"S135132490999012X_ref1","volume-title":"Proceedings of the 5th ACM ICDL","author":"Agichtein","year":"2000"},{"key":"S135132490999012X_ref22","doi-asserted-by":"publisher","DOI":"10.1023\/A:1025779619903"},{"key":"S135132490999012X_ref8","volume-title":"Proceedings of the 5th International CAA Conference","author":"Callear","year":"2001"},{"key":"S135132490999012X_ref13","doi-asserted-by":"publisher","DOI":"10.1162\/089120102760275983"},{"key":"S135132490999012X_ref30","first-page":"201","volume-title":"Soft Computing Applications in Business","author":"Nielsen","year":"2007"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S135132490999012X","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,4,30]],"date-time":"2019-04-30T06:31:10Z","timestamp":1556605870000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S135132490999012X\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,9,16]]},"references-count":42,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2009,10]]}},"alternative-id":["S135132490999012X"],"URL":"https:\/\/doi.org\/10.1017\/s135132490999012x","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2009,9,16]]}}}