{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T15:22:49Z","timestamp":1778167369857,"version":"3.51.4"},"reference-count":99,"publisher":"MIT Press","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2018,9]]},"abstract":"<jats:p>Ensemble methods using multiple classifiers have proven to be among the most successful approaches for the task of Native Language Identification (NLI), achieving the current state of the art. However, a systematic examination of ensemble methods for NLI has yet to be conducted. Additionally, deeper ensemble architectures such as classifier stacking have not been closely evaluated. We present a set of experiments using three ensemble-based models, testing each with multiple configurations and algorithms. This includes a rigorous application of meta-classification models for NLI, achieving state-of-the-art results on several large data sets, evaluated in both intra-corpus and cross-corpus modes.<\/jats:p>","DOI":"10.1162\/coli_a_00323","type":"journal-article","created":{"date-parts":[[2018,6,7]],"date-time":"2018-06-07T18:53:10Z","timestamp":1528397590000},"page":"403-446","source":"Crossref","is-referenced-by-count":26,"title":["Native Language Identification With Classifier Stacking and Ensembles"],"prefix":"10.1162","volume":"44","author":[{"given":"Shervin","family":"Malmasi","sequence":"first","affiliation":[{"name":"Harvard Medical School."}]},{"given":"Mark","family":"Dras","sequence":"additional","affiliation":[{"name":"Macquarie University, Department of Computing."}]}],"member":"281","reference":[{"key":"bib1","unstructured":"Aue, Anthony and Michael Gamon. 2005. Customizing sentiment classifiers to new domains: A case study. Technical Report, Microsoft Research. https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/new_domain_sentiment.pdf."},{"key":"bib2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2007.250609"},{"key":"bib3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5025"},{"key":"bib4","doi-asserted-by":"crossref","unstructured":"Blanchard, Daniel, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013. TOEFL11: A corpus of non-native English. Technical report ETSRR-13-14, Educational Testing Service.","DOI":"10.1002\/j.2333-8504.2013.tb02331.x"},{"key":"bib5","first-page":"123","volume-title":"Machine Learning","volume":"24","author":"Breiman Leo","year":"1996"},{"key":"bib6","doi-asserted-by":"publisher","DOI":"10.3115\/1225403.1225423"},{"key":"bib7","first-page":"37","volume-title":"Proceedings of the First Learner Corpus Research Conference","author":"Brooke Julian","year":"2011"},{"key":"bib8","first-page":"779","volume-title":"Proceedings of the Eighth International Conference on Language Resources and Evaluation","author":"Brooke Julian","year":"2012"},{"key":"bib9","first-page":"391","volume-title":"Proceedings of COLING 2012","author":"Brooke Julian","year":"2012"},{"key":"bib10","first-page":"1962","volume-title":"Proceedings of the 25th International Conference on Computational Linguistics","author":"Bykh Serhiy","year":"2014"},{"key":"bib11","first-page":"197","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Bykh Serhiy","year":"2013"},{"key":"bib12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5023"},{"key":"bib13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5049"},{"key":"bib14","first-page":"207","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Cimino Andrea","year":"2013"},{"key":"bib15","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1967.1053964"},{"key":"bib16","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45014-9_1"},{"key":"bib17","first-page":"1","volume":"49","author":"Eldan Ronen","year":"2016","journal-title":"JMLR: Workshop and Conference Proceedings"},{"key":"bib18","volume-title":"Norsk Referansegrammatikk","author":"Faarlund Jan Terje","year":"1997"},{"key":"bib19","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-1809.1936.tb02137.x"},{"key":"bib20","doi-asserted-by":"publisher","DOI":"10.14358\/PERS.70.5.627"},{"key":"bib21","first-page":"148","volume-title":"Proceedings of the Thirteenth International Conference on International Conference on Machine Learning","author":"Freund Yoav","year":"1996"},{"key":"bib22","first-page":"240","volume-title":"Proceedings of the 31st Second Language Research Forum","author":"Geertzen Jeroen","year":"2013"},{"key":"bib23","doi-asserted-by":"publisher","DOI":"10.1198\/004017007000000245"},{"key":"bib24","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-006-6226-1"},{"key":"bib25","author":"Goldberg Yoav","year":"2015","journal-title":"The Computing Research Repository (CoRR)"},{"key":"bib26","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5041"},{"key":"bib27","first-page":"96","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Goutte Cyril","year":"2013"},{"key":"bib28","volume-title":"International Corpus of Learner English (Version 2)","author":"Granger Sylviane","year":"2009"},{"key":"bib29","first-page":"1222","volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation","author":"Guthrie David","year":"2006"},{"key":"bib30","first-page":"224","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Gyawali Binod","year":"2013"},{"key":"bib31","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7"},{"key":"bib32","first-page":"101","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Henderson John","year":"2013"},{"key":"bib33","first-page":"232","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Hladka Barbora","year":"2013"},{"key":"bib34","doi-asserted-by":"publisher","DOI":"10.1109\/34.273716"},{"key":"bib36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5024"},{"key":"bib37","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1142"},{"key":"bib38","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00256"},{"key":"bib39","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5021"},{"key":"bib40","first-page":"111","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Jarvis Scott","year":"2013"},{"key":"bib41","volume-title":"Paper presented at Second Language Research Forum (SLRF)","author":"Jarvis Scott","year":"2004"},{"key":"bib42","doi-asserted-by":"publisher","DOI":"10.21832\/9781847696991"},{"key":"bib43","doi-asserted-by":"publisher","DOI":"10.1007\/BFb0026683"},{"key":"bib44","doi-asserted-by":"publisher","DOI":"10.1109\/34.667881"},{"key":"bib45","first-page":"1137","volume-title":"International Joint Conference on Artificial Intelligence","author":"Kohavi Ron","year":"1995"},{"key":"bib46","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20961"},{"key":"bib47","doi-asserted-by":"publisher","DOI":"10.1007\/11427995_17"},{"key":"bib48","doi-asserted-by":"publisher","DOI":"10.1145\/1081870.1081947"},{"key":"bib49","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5043"},{"key":"bib50","doi-asserted-by":"publisher","DOI":"10.1109\/34.982906"},{"key":"bib51","doi-asserted-by":"publisher","DOI":"10.1002\/0471660264"},{"key":"bib52","doi-asserted-by":"crossref","DOI":"10.1002\/9781118914564","volume-title":"Combining Pattern Classifiers: Methods and Algorithms","author":"Kuncheva Ludmila I.","year":"2014"},{"key":"bib53","doi-asserted-by":"publisher","DOI":"10.1016\/S0031-3203(99)00223-X"},{"key":"bib54","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-012-0586-6"},{"key":"bib55","doi-asserted-by":"publisher","DOI":"10.1007\/s10044-002-0173-7"},{"key":"bib56","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5044"},{"key":"bib57","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2002.999679"},{"key":"bib58","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0606"},{"key":"bib59","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3625"},{"key":"bib60","first-page":"95","author":"Malmasi Shervin","year":"2014","journal-title":"Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics"},{"key":"bib61","first-page":"139","volume-title":"Australasian Language Technology Association Workshop 2014","author":"Malmasi Shervin","year":"2014"},{"key":"bib62","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1160"},{"key":"bib63","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324915000406"},{"key":"bib64","author":"Malmasi Shervin","year":"2017","journal-title":"CoRR"},{"key":"bib65","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0620"},{"key":"bib66","first-page":"124","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Malmasi Shervin","year":"2013"},{"key":"bib67","first-page":"1","volume-title":"Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects","author":"Malmasi Shervin","year":"2016"},{"key":"bib68","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00239"},{"key":"bib69","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809071.009"},{"key":"bib70","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"},{"key":"bib71","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5042"},{"key":"bib72","first-page":"449","volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation","author":"de Marneffe Marie Catherine","year":"2006"},{"key":"bib73","doi-asserted-by":"publisher","DOI":"10.1007\/BF02295996"},{"key":"bib74","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5047"},{"key":"bib75","volume-title":"Understanding Second Language Acquisition","author":"Ortega Lourdes","year":"2009"},{"key":"bib76","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2007.07.002"},{"key":"bib77","unstructured":"Pepper, Steve. 2012. Lexical transfer in Norwegian interlanguage: A detection-based approach. Master\u2019s thesis, University of Oslo."},{"key":"bib78","doi-asserted-by":"crossref","first-page":"61","DOI":"10.7551\/mitpress\/1113.003.0008","volume-title":"Advances in Large Margin Classifiers","author":"Platt John","year":"2000"},{"key":"bib79","doi-asserted-by":"publisher","DOI":"10.1109\/MCAS.2006.1688199"},{"key":"bib80","first-page":"270","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Popescu Marius","year":"2013"},{"key":"bib81","volume-title":"C4.5: Programs for Machine Learning","author":"Quinlan J Ross","year":"1993"},{"key":"bib82","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5028"},{"key":"bib83","doi-asserted-by":"publisher","DOI":"10.1037\/h0042519"},{"key":"bib84","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5027"},{"key":"bib85","first-page":"193","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Swanson Benjamin","year":"2012"},{"issue":"3","key":"bib86","first-page":"1000","volume":"6","author":"Tenfjord Kari","year":"2006","journal-title":"Rivista di psicolinguistica applicata"},{"key":"bib87","first-page":"1821","volume-title":"Proceedings of the 5th International Conference on Language Resources and Evaluation","author":"Tenfjord Kari","year":"2006"},{"key":"bib88","volume-title":"Learner Corpus Research Conference","author":"Tenfjord Kari","year":"2013"},{"key":"bib89","first-page":"48","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Tetreault Joel","year":"2013"},{"key":"bib90","first-page":"2585","volume-title":"Proceedings of the International Conference on Computational Linguistics","author":"Tetreault Joel","year":"2012"},{"key":"bib91","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.082099299"},{"key":"bib92","first-page":"239","volume-title":"Proceedings of the Second North American Chapter of the Association for Computational Linguistics","author":"Tomokiyo Laura Mayfield","year":"2001"},{"key":"bib93","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0614"},{"key":"bib94","doi-asserted-by":"publisher","DOI":"10.1016\/S0305-0548(99)00149-5"},{"key":"bib95","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(05)80023-1"},{"key":"bib96","first-page":"53","volume-title":"Proceedings of the Australasian Language Technology Association Workshop","author":"Wong Sze Meng Jojo","year":"2009"},{"key":"bib97","first-page":"699","volume-title":"Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","author":"Wong Sze Meng Jojo","year":"2012"},{"key":"bib98","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2013.04.006"},{"key":"bib99","doi-asserted-by":"publisher","DOI":"10.1023\/A:1011441423217"},{"key":"bib100","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxg046"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/coli_a_00323","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,7]],"date-time":"2024-07-07T10:17:49Z","timestamp":1720347469000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/44\/3\/403-446\/1601"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9]]},"references-count":99,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,9]]}},"alternative-id":["10.1162\/coli_a_00323"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00323","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,9]]}}}