{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:24:53Z","timestamp":1750307093705,"version":"3.41.0"},"reference-count":23,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2012,12,1]],"date-time":"2012-12-01T00:00:00Z","timestamp":1354320000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Transactions on Asian Language Information Processing"],"published-print":{"date-parts":[[2012,12]]},"abstract":"<jats:p>The present article addresses an attempt to apply questions in university entrance examinations to the evaluation of textual entailment recognition. Questions in several fields, such as history and politics, primarily test the examinee\u2019s knowledge in the form of choosing true statements from multiple choices. Answering such questions can be regarded as equivalent to finding evidential texts from a textbase such as textbooks and Wikipedia. Therefore, this task can be recast as recognizing textual entailment between a description in a textbase and a statement given in a question. We focused on the National Center Test for University Admission in Japan and converted questions into the evaluation data for textual entailment recognition by using Wikipedia as a textbase. Consequently, it is revealed that nearly half of the questions can be mapped into textual entailment recognition; 941 text pairs were created from 404 questions from six subjects. This data set is provided for a subtask of NTCIR RITE (Recognizing Inference in Text), and 16 systems from six teams used the data set for evaluation. The evaluation results revealed that the best system achieved a correct answer ratio of 56%, which is significantly better than a random choice baseline.<\/jats:p>","DOI":"10.1145\/2382593.2382595","type":"journal-article","created":{"date-parts":[[2012,12,5]],"date-time":"2012-12-05T18:55:48Z","timestamp":1354733748000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Evaluating Textual Entailment Recognition for University Entrance Examinations"],"prefix":"10.1145","volume":"11","author":[{"given":"Yusuke","family":"Miyao","sequence":"first","affiliation":[{"name":"National Institute of Informatics"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hideki","family":"Shima","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hiroshi","family":"Kanayama","sequence":"additional","affiliation":[{"name":"IBM Research - Tokyo"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Teruko","family":"Mitamura","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2012,12]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_2_1_1_1","DOI":"10.5555\/1892211.1892215"},{"volume-title":"Proceedings of the 2nd International Semantic Web Conference (ISWC\u201903)","author":"Angele J.","unstructured":"Angele , J. , Moench , E. , Oppermann , H. , Staab , S. , and Wenke , D . 2003. Ontology-based query and answering in chemistry: OntoNova Project Halo . In Proceedings of the 2nd International Semantic Web Conference (ISWC\u201903) . Angele, J., Moench, E., Oppermann, H., Staab, S., and Wenke, D. 2003. Ontology-based query and answering in chemistry: OntoNova Project Halo. In Proceedings of the 2nd International Semantic Web Conference (ISWC\u201903).","key":"e_1_2_1_2_1"},{"volume-title":"Proceedings of the 2nd PASCAL Challenges Workshop on Recognizing Textual Entailment (RTE-2\u201906)","author":"Bar-Haim R.","unstructured":"Bar-Haim , R. , Dagan , I. , Dolan , B. , Ferro , L. , Giampiccolo , D. , and Magnini , B . 2006. The second PASCAL recognizing textual entailment challenge . In Proceedings of the 2nd PASCAL Challenges Workshop on Recognizing Textual Entailment (RTE-2\u201906) . Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., and Magnini, B. 2006. The second PASCAL recognizing textual entailment challenge. In Proceedings of the 2nd PASCAL Challenges Workshop on Recognizing Textual Entailment (RTE-2\u201906).","key":"e_1_2_1_3_1"},{"volume-title":"Proceedings of the Text Analysis Conference (TAC\u201909)","author":"Bentivogli L.","unstructured":"Bentivogli , L. , Dagan , I. , Dang , H. T. , Giampiccolo , D. , and Magnini , B . 2009. The fifth PASCAL recognizing textual entailment challenge . In Proceedings of the Text Analysis Conference (TAC\u201909) . Bentivogli, L., Dagan, I., Dang, H. T., Giampiccolo, D., and Magnini, B. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference (TAC\u201909).","key":"e_1_2_1_4_1"},{"volume-title":"Proceedings of the Text Analysis Conference (TAC\u201910)","author":"Bentivogli L.","unstructured":"Bentivogli , L. , Clark , P. , Dagan , I. , Dang , H. , and Giampiccolo , D . 2010. The sixth PASCAL recognizing textual entailment challenge . In Proceedings of the Text Analysis Conference (TAC\u201910) . Bentivogli, L., Clark, P., Dagan, I., Dang, H., and Giampiccolo, D. 2010. The sixth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference (TAC\u201910).","key":"e_1_2_1_5_1"},{"volume-title":"Proceedings of the Conference on Evaluation of NLP and Speech Tools for Italian (EVALITA\u201909)","author":"Bos J.","unstructured":"Bos , J. , Zanzotto , F. M. , and Pennacchiotti , M . 2009. Textual entailment at EVALITA 2009 . In Proceedings of the Conference on Evaluation of NLP and Speech Tools for Italian (EVALITA\u201909) . Bos, J., Zanzotto, F. M., and Pennacchiotti, M. 2009. Textual entailment at EVALITA 2009. In Proceedings of the Conference on Evaluation of NLP and Speech Tools for Italian (EVALITA\u201909).","key":"e_1_2_1_6_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_7_1","DOI":"10.1007\/11736790_9"},{"volume-title":"Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing (ACL-PASCAL\u201907)","author":"Giampiccolo D.","unstructured":"Giampiccolo , D. , Magnini , B. , Dagan , I. , and Dolan , B . 2007. The third PASCAL recognizing textual entailment challenge . In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing (ACL-PASCAL\u201907) . Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing (ACL-PASCAL\u201907).","key":"e_1_2_1_8_1"},{"volume-title":"Proceedings of the Text Analysis Conference (TAC\u201908)","author":"Giampiccolo D.","unstructured":"Giampiccolo , D. , Dang , H. T. , Magnini , B. , Dagan , I. , and Dolan , B . 2009. The fourth PASCAL recognizing textual entailment challenge . In Proceedings of the Text Analysis Conference (TAC\u201908) . Giampiccolo, D., Dang, H. T., Magnini, B., Dagan, I., and Dolan, B. 2009. The fourth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference (TAC\u201908).","key":"e_1_2_1_9_1"},{"volume-title":"Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Ishikawa D.","unstructured":"Ishikawa , D. , Joho , H. , Kando , N. , Kato , T. , Sakai , T. , Sugimoto , M. , Sumita , E. , Akiba , T. , Geva , S. , Gey , F. , Goto , I. , Lu , B. , Shima , H. , Tang , E. , and Trotman , A. , Eds . 2011 . Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . Ishikawa, D., Joho, H., Kando, N., Kato, T., Sakai, T., Sugimoto, M., Sumita, E., Akiba, T., Geva, S., Gey, F., Goto, I., Lu, B., Shima, H., Tang, E., and Trotman, A., Eds. 2011. Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911).","key":"e_1_2_1_10_1"},{"volume-title":"Proceedings of the Association of Computational Linguistics (ACL\u201911)","author":"Ji H.","unstructured":"Ji , H. and Grishman , R . 2011. Knowledge base population: Successful approaches and challenges . In Proceedings of the Association of Computational Linguistics (ACL\u201911) . 1148--1158. Ji, H. and Grishman, R. 2011. Knowledge base population: Successful approaches and challenges. In Proceedings of the Association of Computational Linguistics (ACL\u201911). 1148--1158.","key":"e_1_2_1_11_1"},{"volume-title":"Proceedings of the 16th Annual Conference on Natural Language Processing in Japan (JapTAL\u201910)","author":"Kasahara K.","unstructured":"Kasahara , K. , Taira , H. , and Nagata , M . 2010. On the possibility of applying textual entailment recognition to multi-sentence reading comprehension tasks . In Proceedings of the 16th Annual Conference on Natural Language Processing in Japan (JapTAL\u201910) . 780--783. (In Japanese). Kasahara, K., Taira, H., and Nagata, M. 2010. On the possibility of applying textual entailment recognition to multi-sentence reading comprehension tasks. In Proceedings of the 16th Annual Conference on Natural Language Processing in Japan (JapTAL\u201910). 780--783. (In Japanese).","key":"e_1_2_1_12_1"},{"volume-title":"Proceedings of Human Language Technology Conference\/North American Chapter of the Association for Computational Linguistics (HLT-NAACL\u201910)","author":"Mehdad Y.","unstructured":"Mehdad , Y. , Negri , M. , and Federico , M . 2010. Towards cross-lingual textual entailment . In Proceedings of Human Language Technology Conference\/North American Chapter of the Association for Computational Linguistics (HLT-NAACL\u201910) . Mehdad, Y., Negri, M., and Federico, M. 2010. Towards cross-lingual textual entailment. In Proceedings of Human Language Technology Conference\/North American Chapter of the Association for Computational Linguistics (HLT-NAACL\u201910).","key":"e_1_2_1_13_1"},{"volume-title":"Proceedings of the Association of Computational Linguistics (ACL\u201911)","author":"Mehdad Y.","unstructured":"Mehdad , Y. , Negri , M. , and Federico , M . 2011. Using bilingual parallel corpora for cross-lingual textual entailment . In Proceedings of the Association of Computational Linguistics (ACL\u201911) . Mehdad, Y., Negri, M., and Federico, M. 2011. Using bilingual parallel corpora for cross-lingual textual entailment. In Proceedings of the Association of Computational Linguistics (ACL\u201911).","key":"e_1_2_1_14_1"},{"volume-title":"Proceedings of the 14th Annual Conference on Natural Language Processing in Japan (JapTAL\u201908)","author":"Odani M.","unstructured":"Odani , M. , Shibata , T. , Kurohashi , S. , and Nakata , T . 2008. Building data of Japanese text entailment and recognition of inferencing relations based on automatic achieved similar expressions . In Proceedings of the 14th Annual Conference on Natural Language Processing in Japan (JapTAL\u201908) . 1140--1143. (In Japanese.) Odani, M., Shibata, T., Kurohashi, S., and Nakata, T. 2008. Building data of Japanese text entailment and recognition of inferencing relations based on automatic achieved similar expressions. In Proceedings of the 14th Annual Conference on Natural Language Processing in Japan (JapTAL\u201908). 1140--1143. (In Japanese.)","key":"e_1_2_1_15_1"},{"volume-title":"Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Pakray P.","unstructured":"Pakray , P. , Neogi , S. , Bandyopadhyay , S. , and Gelbukh , A . 2011. A textual entailment system using web based machine translation system . In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . Pakray, P., Neogi, S., Bandyopadhyay, S., and Gelbukh, A. 2011. A textual entailment system using web based machine translation system. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911).","key":"e_1_2_1_16_1"},{"volume-title":"Proceedings of the Workshop on Cross-Language Evaluation Forum (CLEF\u201911)","author":"Pe\u00f1as A.","unstructured":"Pe\u00f1as , A. , Hovy , E. , Forner , P. , Rodrigo , A. , Sutcliffe , R. , Forascu , C. , and Sporleder , C . 2011. Overview of QA4MRE at CLEF 2011: Question answering for machine reading evaluation . In Proceedings of the Workshop on Cross-Language Evaluation Forum (CLEF\u201911) . 19--22. Pe\u00f1as, A., Hovy, E., Forner, P., Rodrigo, A., Sutcliffe, R., Forascu, C., and Sporleder, C. 2011. Overview of QA4MRE at CLEF 2011: Question answering for machine reading evaluation. In Proceedings of the Workshop on Cross-Language Evaluation Forum (CLEF\u201911). 19--22.","key":"e_1_2_1_17_1"},{"volume-title":"Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Pham Q. N. M.","unstructured":"Pham , Q. N. M. , Nguyen , L. M. , and Shimazu , A . 2011. A machine learning based textual entailment recognition system of JAIST team for the NTCIR9 RITE . In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . Pham, Q. N. M., Nguyen, L. M., and Shimazu, A. 2011. A machine learning based textual entailment recognition system of JAIST team for the NTCIR9 RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911).","key":"e_1_2_1_18_1"},{"volume-title":"Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Shibata T.","unstructured":"Shibata , T. and Kurohashi , S . 2011. Predicate-argument structure based textual entailment recognition system of KYOTO team for the NTCIR9 RITE . In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . Shibata, T. and Kurohashi, S. 2011. Predicate-argument structure based textual entailment recognition system of KYOTO team for the NTCIR9 RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911).","key":"e_1_2_1_19_1"},{"volume-title":"Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Shima H.","unstructured":"Shima , H. , Kanayama , H. , Lee , C.-W. , Lin , C.-J. , Mitamura , T. , Miyao , Y. , Shi , S. , and Takeda , K . 2011a. Overview of the NTCIR-9 RITE: Recognizing inference in text . In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . 291--301. Shima, H., Kanayama, H., Lee, C.-W., Lin, C.-J., Mitamura, T., Miyao, Y., Shi, S., and Takeda, K. 2011a. Overview of the NTCIR-9 RITE: Recognizing inference in text. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911). 291--301.","key":"e_1_2_1_20_1"},{"volume-title":"RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Shima H.","unstructured":"Shima , H. , Li , Y. , Orii , N. , and Mitamura , T . 2011b. LTI\u2019s textual entailment recognizer system at NTCIR-9 RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . Shima, H., Li, Y., Orii, N., and Mitamura, T. 2011b. LTI\u2019s textual entailment recognizer system at NTCIR-9 RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911).","key":"e_1_2_1_21_1"},{"volume-title":"Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Tsuboi Y.","unstructured":"Tsuboi , Y. , Kanayama , H. , Ohno , M. , and Unno , Y . 2011. Syntactic difference based approach for the NTCIR-9 RITE task . In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . Tsuboi, Y., Kanayama, H., Ohno, M., and Unno, Y. 2011. Syntactic difference based approach for the NTCIR-9 RITE task. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911).","key":"e_1_2_1_22_1"},{"volume-title":"Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911)","author":"Watanabe Y.","unstructured":"Watanabe , Y. , Mizuno , J. , Nichols , E. , Narisawa , K. , Nabeshima , K. , and Inui , K . 2011. TU group at NTCIR9-RITE: Leveraging diverse lexical resources for recognizing textual entailment . In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911) . Watanabe, Y., Mizuno, J., Nichols, E., Narisawa, K., Nabeshima, K., and Inui, K. 2011. TU group at NTCIR9-RITE: Leveraging diverse lexical resources for recognizing textual entailment. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR\u201911).","key":"e_1_2_1_23_1"}],"container-title":["ACM Transactions on Asian Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2382593.2382595","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2382593.2382595","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:34:38Z","timestamp":1750239278000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2382593.2382595"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,12]]},"references-count":23,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2012,12]]}},"alternative-id":["10.1145\/2382593.2382595"],"URL":"https:\/\/doi.org\/10.1145\/2382593.2382595","relation":{},"ISSN":["1530-0226","1558-3430"],"issn-type":[{"type":"print","value":"1530-0226"},{"type":"electronic","value":"1558-3430"}],"subject":[],"published":{"date-parts":[[2012,12]]},"assertion":[{"value":"2012-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-12-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}