{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T07:52:10Z","timestamp":1782373930013,"version":"3.54.5"},"reference-count":65,"publisher":"Cambridge University Press (CUP)","issue":"6","license":[{"start":{"date-parts":[[2021,8,26]],"date-time":"2021-08-26T00:00:00Z","timestamp":1629936000000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Argument mining (AM) aims to explain how individual argumentative discourse units (e.g. sentences or clauses) relate to each other and what roles they play in the overall argumentation. The automatic recognition of argumentative structure is attractive as it benefits various downstream tasks, such as text assessment, text generation, text improvement, and summarization. Existing studies focused on analyzing well-written texts provided by proficient authors. However, most English speakers in the world are non-native, and their texts are often poorly structured, particularly if they are still in the learning phase. Yet, there is no specific prior study on argumentative structure in non-native texts. In this article, we present the first corpus containing argumentative structure annotation for English-as-a-foreign-language (EFL) essays, together with a specially designed annotation scheme. The annotated corpus resulting from this work is called \u201cICNALE-AS\u201d and contains 434 essays written by EFL learners from various Asian countries. The corpus presented here is particularly useful for the education domain. On the basis of the analysis of argumentation-related problems in EFL essays, educators can formulate ways to improve them so that they more closely resemble native-level productions. Our argument annotation scheme is demonstrably stable, achieving good inter-annotator agreement and near-perfect intra-annotator agreement. We also propose a set of novel document-level agreement metrics that are able to quantify structural agreement from various argumentation aspects, thus providing a more holistic analysis of the quality of the argumentative structure annotation. The metrics are evaluated in a crowd-sourced meta-evaluation experiment, achieving moderate to good correlation with human judgments.<\/jats:p>","DOI":"10.1017\/s1351324921000218","type":"journal-article","created":{"date-parts":[[2021,8,26]],"date-time":"2021-08-26T06:56:07Z","timestamp":1629960967000},"page":"797-823","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":7,"title":["Annotating argumentative structure in English-as-a-Foreign-Language learner essays"],"prefix":"10.1017","volume":"28","author":[{"given":"Jan Wira Gotama","family":"Putra","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simone","family":"Teufel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1399-9517","authenticated-orcid":false,"given":"Takenobu","family":"Tokunaga","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2021,8,26]]},"reference":[{"key":"S1351324921000218_ref9","article-title":"Creating visual explanations improve learning","volume":"1","author":"Bobek","year":"2016","journal-title":"Cognitive Research: Principles and Implications"},{"key":"S1351324921000218_ref32","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-1770.1966.tb00804.x"},{"key":"S1351324921000218_ref18","first-page":"229","author":"Fries","year":"1994"},{"key":"S1351324921000218_ref50","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2007.106"},{"key":"S1351324921000218_ref38","first-page":"243","article-title":"Rhetorical structure theory: Toward a functional theory of text organization","volume":"8","author":"Mann","year":"1988","journal-title":"Text"},{"key":"S1351324921000218_ref28","first-page":"91","article-title":"The icnale and sophisticated contrastive interlanguage analysis of asian learners of english","volume":"1","author":"Ishikawa","year":"2013","journal-title":"Learner Corpus Studies in Asia and the World"},{"key":"S1351324921000218_ref20","unstructured":"Garing, A.G. (2014). Coherence in argumentative essays of first year college of liberal arts students at de la salle university. DLSU Research Congress."},{"key":"S1351324921000218_ref55","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00295"},{"key":"S1351324921000218_ref59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-1017"},{"key":"S1351324921000218_ref46","unstructured":"Putra, J.W.G. , Teufel, S. , Matsumura, K. and Tokunaga, T. (2020). TIARA: A tool for annotating discourse relations and sentence reordering. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France. European Language Resources Association, pp. 6912\u20136920."},{"key":"S1351324921000218_ref41","doi-asserted-by":"publisher","DOI":"10.4018\/jcini.2013010101"},{"key":"S1351324921000218_ref7","volume-title":"Natural Language Processing with Python","author":"Bird","year":"2009"},{"key":"S1351324921000218_ref61","doi-asserted-by":"publisher","DOI":"10.1162\/0891201054223977"},{"key":"S1351324921000218_ref35","unstructured":"Lin, Z. , Ng, H.T. and Kan, M.-Y. (2011). Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon, USA. Association for Computational Linguistics, pp. 997\u20131006."},{"key":"S1351324921000218_ref5","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeap.2010.05.001"},{"key":"S1351324921000218_ref37","doi-asserted-by":"publisher","DOI":"10.1016\/j.jslw.2004.11.001"},{"key":"S1351324921000218_ref10","first-page":"249","article-title":"Assessing agreement on classification tasks: The kappa statistic","volume":"22","author":"Carletta","year":"1996","journal-title":"Computational Linguistics"},{"key":"S1351324921000218_ref25","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324904003560"},{"key":"S1351324921000218_ref45","unstructured":"Prasad, R. , Dinesh, N. , Lee, A. , Miltsakaki, E. , Robaldo, L. , Joshi, A. and Webber, B. (2008). The penn discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC\u201908), Marrakech, Morocco. European Language Resources Association (ELRA)."},{"key":"S1351324921000218_ref48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1176"},{"key":"S1351324921000218_ref12","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0504"},{"key":"S1351324921000218_ref52","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-5218"},{"key":"S1351324921000218_ref13","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"S1351324921000218_ref64","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1042"},{"key":"S1351324921000218_ref30","unstructured":"Jacobs, H.L. , Zinkgraf, S.A. , Wormuth, D.R. , Harfiel, V.F. and Hughey, J.B. (1981). Testing ESL Composition: A Practical Approach. Rowley, MA: Newbury House."},{"key":"S1351324921000218_ref6","doi-asserted-by":"publisher","DOI":"10.2307\/357898"},{"key":"S1351324921000218_ref47","unstructured":"Putra, J.W.G. , Teufel, S. and Tokunaga, T. (2019). An argument annotation scheme for the repair of student essays by sentence reordering. In Proceedings of Annual Meeting of the Association for Natural Language Processing Japan, pp. 546\u2013549."},{"key":"S1351324921000218_ref36","doi-asserted-by":"publisher","DOI":"10.1145\/2850417"},{"key":"S1351324921000218_ref60","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324911000337"},{"key":"S1351324921000218_ref42","unstructured":"Peldszus, A. and Stede, M. (2016). An annotated corpus of argumentative microtexts. In Argumentation and Reasoned Action - Proceedings of the 1st European Conference on Argumentation, Lisbon, 2015."},{"key":"S1351324921000218_ref16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-8607"},{"key":"S1351324921000218_ref62","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-019-09242-3"},{"key":"S1351324921000218_ref24","first-page":"33","article-title":"Text tiling: Segmenting text into multi-paragraph subtopic passages","volume":"23","author":"Hearst","year":"1997","journal-title":"Computational Linguistics"},{"key":"S1351324921000218_ref22","unstructured":"Habernal, I. , Eckle-Kohler, J. and Gurevych, I. (2014). Argumentation mining on the web from information seeking perspective. In Proceedings of the Workshop on Frontiers and Connections between Argumentation Theory and Natural Language Processing."},{"key":"S1351324921000218_ref21","first-page":"175","article-title":"Attention, intentions, and the structure of discourse","volume":"12","author":"Grosz","year":"1986","journal-title":"Computational Linguistics"},{"key":"S1351324921000218_ref33","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0501"},{"key":"S1351324921000218_ref56","doi-asserted-by":"publisher","DOI":"10.3115\/977035.977051"},{"key":"S1351324921000218_ref31","doi-asserted-by":"publisher","DOI":"10.37514\/JBW-J.1986.5.2.08"},{"key":"S1351324921000218_ref4","volume-title":"Artificial Intelligence and Legal Reasoning","author":"Ashley","year":"1990"},{"key":"S1351324921000218_ref2","unstructured":"Al-Khatib, K. , Wachsmuth, H. , Kiesel, J. , Hagen, M. and Stein, B. (2016). A news editorial corpus for mining argumentation strategies. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan. The COLING 2016 Organizing Committee, pp. 3433\u20133443."},{"key":"S1351324921000218_ref34","unstructured":"Leffa, V.J. and Cunha, R.F.D. (1998). Clause processing in complex sentences. In Proceedings of the First International Conference on Language Resources and Evaluation (LREC), pp. 937\u2013943."},{"key":"S1351324921000218_ref40","unstructured":"Park, J. and Cardie, C. (2018). A corpus of eRulemaking user comments for measuring evaluability of arguments. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA)."},{"key":"S1351324921000218_ref54","unstructured":"Stab, C. and Gurevych, I. (2014). Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland. Dublin City University and Association for Computational Linguistics, pp. 1501\u20131510."},{"key":"S1351324921000218_ref63","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0512"},{"key":"S1351324921000218_ref23","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324906004190"},{"key":"S1351324921000218_ref14","doi-asserted-by":"publisher","DOI":"10.2307\/3588238"},{"key":"S1351324921000218_ref39","first-page":"14","article-title":"A structure analysis of japanese efl students\u2019 argumentative paragraph writing with a tool for annotation discourse relation","author":"Matsumura","year":"2021","journal-title":"The Bulletin of the Writing Reserch Group, JACET Kansai Chapter"},{"key":"S1351324921000218_ref57","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671936"},{"key":"S1351324921000218_ref8","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-007-2363-4"},{"key":"S1351324921000218_ref17","unstructured":"Feng, V.W. , Lin, Z. and Hirst, G. (2014). The impact of deep hierarchical discourse structures in the evaluation of text coherence. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland. Dublin City University and Association for Computational Linguistics, pp. 940\u2013949."},{"key":"S1351324921000218_ref19","first-page":"1","article-title":"The role of grammar instruction in japanese EFL context: Towards communicative language teaching","volume":"4","author":"Fujiwara","year":"2018","journal-title":"Journal of the Academic Society for Quality of Life (JAS4QoL)"},{"key":"S1351324921000218_ref43","unstructured":"Persing, I. , Davis, A. and Ng, V. (2010). Modeling organization in student essays. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Cambridge, MA. Association for Computational Linguistics, pp. 229\u2013239."},{"key":"S1351324921000218_ref26","first-page":"83","volume-title":"Approaches to the Planning of Coherent Text","author":"Hovy","year":"1991"},{"key":"S1351324921000218_ref15","doi-asserted-by":"publisher","DOI":"10.1038\/s41539-018-0038-5"},{"key":"S1351324921000218_ref65","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0616"},{"key":"S1351324921000218_ref27","doi-asserted-by":"publisher","DOI":"10.1017\/9781316423530"},{"key":"S1351324921000218_ref1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1141"},{"key":"S1351324921000218_ref29","first-page":"1","article-title":"The icnale edited essays: A dataset for analysis of l2 english learner essays based on a new integrative viewpoint","volume":"25","author":"Ishikawa","year":"2018","journal-title":"English Corpus Linguistics"},{"key":"S1351324921000218_ref44","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1053"},{"key":"S1351324921000218_ref49","unstructured":"Reed, C. , Palau, R.M. , Rowe, G. and Moens, M.-F. (2008). Language resources for studying argument. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC)."},{"key":"S1351324921000218_ref3","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"S1351324921000218_ref51","doi-asserted-by":"publisher","DOI":"10.2307\/3587400"},{"key":"S1351324921000218_ref53","unstructured":"Sonntag, J. and Stede, M. (2014). GraPAT: A tool for graph annotations. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914), Reykjavik, Iceland. European Language Resources Association (ELRA), pp. 4147\u20134151."},{"key":"S1351324921000218_ref11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1058"},{"key":"S1351324921000218_ref58","unstructured":"Wachsmuth, H. , Al-Khatib, K. and Stein, B. (2016). Using argument mining to assess the argumentation quality of essays. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan. The COLING 2016 Organizing Committee, pp. 1680\u20131691."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324921000218","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,23]],"date-time":"2022-11-23T08:40:38Z","timestamp":1669192838000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324921000218\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,26]]},"references-count":65,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["S1351324921000218"],"URL":"https:\/\/doi.org\/10.1017\/s1351324921000218","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,26]]},"assertion":[{"value":"\u00a9 The Author(s), 2021. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}