{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T10:17:49Z","timestamp":1775470669177,"version":"3.50.1"},"reference-count":39,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2021,2]]},"abstract":"<jats:p> Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost. This is now an essential tool for building low-resource syntactic analyzers such as part-of-speech (POS) taggers. Existing AL heuristics are generally designed on the principle of selecting uncertain yet representative training instances, where annotating these instances may reduce a large number of errors. However, in an empirical study across six typologically diverse languages (German, Swedish, Galician, North Sami, Persian, and Ukrainian), we found the surprising result that even in an oracle scenario where we know the true uncertainty of predictions, these current heuristics are far from optimal. Based on this analysis, we pose the problem of AL as selecting instances that maximally reduce the confusion between particular pairs of output tags. Extensive experimentation on the aforementioned languages shows that our proposed AL strategy outperforms other AL strategies by a significant margin. We also present auxiliary results demonstrating the importance of proper calibration of models, which we ensure through cross-view training, and analysis demonstrating how our proposed strategy selects examples that more closely follow the oracle data distribution. The code is publicly released here. <jats:sup>1<\/jats:sup> <\/jats:p>","DOI":"10.1162\/tacl_a_00350","type":"journal-article","created":{"date-parts":[[2021,2,18]],"date-time":"2021-02-18T21:23:29Z","timestamp":1613683409000},"page":"1-16","source":"Crossref","is-referenced-by-count":10,"title":["Reducing Confusion in Active Learning for Part-Of-Speech                     Tagging"],"prefix":"10.1162","volume":"9","author":[{"given":"Aditi","family":"Chaudhary","sequence":"first","affiliation":[{"name":"Language Technologies Institute, Carnegie Mellon University."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Antonios","family":"Anastasopoulos","sequence":"additional","affiliation":[{"name":"Department of Computer Science, George Mason University."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zaid","family":"Sheikh","sequence":"additional","affiliation":[{"name":"Language Technologies Institute, Carnegie Mellon University."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Graham","family":"Neubig","sequence":"additional","affiliation":[{"name":"Language Technologies Institute, Carnegie Mellon University."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"bib1","first-page":"2529","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Anastasopoulos Antonios","year":"2018"},{"key":"bib2","doi-asserted-by":"crossref","first-page":"1072","DOI":"10.1109\/GUCON.2018.8674901","volume-title":"2018 International Conference on Computing, Power and Communication Technologies (GUCON)","author":"Ankita","year":"2018"},{"key":"bib3","volume-title":"Sixth International Workshop on Information Integration on the Web","author":"Bellare Kedar","year":"2007"},{"key":"bib4","doi-asserted-by":"crossref","first-page":"2642","DOI":"10.18653\/v1\/P18-1246","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Bohnet Bernd","year":"2018"},{"key":"bib5","first-page":"5164","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Chaudhary Aditi","year":"2019"},{"key":"bib6","doi-asserted-by":"crossref","first-page":"1914","DOI":"10.18653\/v1\/D18-1217","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Clark Kevin","year":"2018"},{"key":"bib7","first-page":"748","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Cotterell Ryan","year":"2017"},{"key":"bib8","first-page":"600","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","author":"Das Dipanjan","year":"2011"},{"key":"bib9","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019"},{"key":"bib10","doi-asserted-by":"crossref","first-page":"587","DOI":"10.18653\/v1\/P17-2093","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Fang Meng","year":"2017"},{"key":"bib11","first-page":"138","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Garrette Dan","year":"2013"},{"key":"bib12","unstructured":"Harald Hammarstr\u00f6m,                                 Robert Forkel, and                                 Martin Haspelmath.                         2018. Glottolog 3.3.                         Max Planck Institute for the Science of Human                         History.                     Jena."},{"key":"bib13","first-page":"892","volume-title":"Advances in Neural Information Processing Systems","author":"Huang Sheng-Jun","year":"2010"},{"key":"bib14","author":"Huang Zhiheng","year":"2015","journal-title":"arXiv preprint arXiv:1508.01991"},{"key":"bib15","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00065"},{"key":"bib16","volume-title":"Proceedings of the 11th Language Resources and Evaluation Conference","author":"Kirov Christo","year":"2018"},{"key":"bib17","first-page":"260","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Lample Guillaume","year":"2016"},{"key":"bib18","unstructured":"Marika Lekakou,                                 Valeria Baldissera, and                                 Antonios Anastasopoulos.                         2013. Documentation and analysis of an                         endangered language: aspects of the grammar of                     griko."},{"key":"bib19","first-page":"246","volume-title":"Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Lewis David D.","year":"1995"},{"key":"bib20","first-page":"3125","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Lin Yu-Hsiang","year":"2019"},{"key":"bib21","first-page":"1064","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ma Xuezhe","year":"2016"},{"key":"bib22","doi-asserted-by":"crossref","first-page":"2653","DOI":"10.18653\/v1\/P18-1247","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Malaviya Chaitanya","year":"2018"},{"key":"bib23","doi-asserted-by":"crossref","first-page":"898","DOI":"10.3115\/v1\/D14-1097","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Marcheggiani Diego","year":"2014"},{"key":"bib24","doi-asserted-by":"crossref","first-page":"91","DOI":"10.18653\/v1\/W18-6011","volume-title":"Proceedings of the Second Workshop on Universal Dependencies (UDW 2018)","author":"McCarthy Arya D.","year":"2018"},{"key":"bib25","doi-asserted-by":"crossref","first-page":"1765","DOI":"10.18653\/v1\/P19-1172","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Nicolai Garrett","year":"2019"},{"key":"bib26","first-page":"1659","volume-title":"Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916)","author":"Nivre Joakim","year":"2016"},{"key":"bib27","author":"Nixon Jeremy","year":"2019","journal-title":"arXiv preprint arXiv:1904.01685"},{"key":"bib28","doi-asserted-by":"crossref","first-page":"101","DOI":"10.3115\/1642059.1642075","volume-title":"Proceedings of the Linguistic Annotation Workshop","author":"Ringger Eric","year":"2007"},{"key":"bib29","volume-title":"International Conference on Learning Representations","author":"Sener Ozan","year":"2018"},{"key":"bib30","unstructured":"Burr Settles.                         2009, Active Learning literature                         survey, University of Wisconsin-Madison                         Department of Computer Sciences."},{"key":"bib31","first-page":"1070","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Settles Burr","year":"2008"},{"key":"bib32","doi-asserted-by":"crossref","unstructured":"Aditya Siddhant,                                 Ankur Bapna,                                 Henry Tsai,                                 Jason Riesa,                                 Karthik Raman,                                 Melvin Johnson,                                 Naveen Ari, and                                 Orhan Firat.                     2020. In Evaluating the Cross-lingual                             Effectiveness of Massively Multilingual Neural Machine                             Translation. DOI:\u00a0https:\/\/doi.org\/10.1609\/aaai.v34i05.6414","DOI":"10.1609\/aaai.v34i05.6414"},{"key":"bib33","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00205"},{"key":"bib34","unstructured":"Nivre Joakim,                                 Blokland Rogier,                                 Partanen Niko,                                 Rie\u00dfler Michael, and                                 Rueter Jack. 2018.                         Universal Dependencies 2.3."},{"key":"bib35","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017"},{"key":"bib36","doi-asserted-by":"crossref","first-page":"1888","DOI":"10.1109\/WACV.2018.00209","volume-title":"2018 IEEE Winter Conference on Applications of Computer Vision (WACV)","author":"Wang Zhe","year":"2018"},{"key":"bib37","author":"Yang Zhilin","year":"2017","journal-title":"arXiv preprint arXiv:1703.06345"},{"key":"bib38","first-page":"1","volume-title":"Proceedings of the First International Conference on Human Language Technology Research","author":"Yarowsky David","year":"2001"},{"key":"bib39","first-page":"600","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Zitouni Imed","year":"2008"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00350","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:49Z","timestamp":1615585189000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/97781"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2]]},"references-count":39,"alternative-id":["10.1162\/tacl_a_00350"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00350","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2]]}}}