{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T16:20:44Z","timestamp":1770826844538,"version":"3.50.1"},"reference-count":89,"publisher":"MIT Press","issue":"2","license":[{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Active learning can play an important role in low-resource settings (i.e., where annotated data is scarce), by selecting which instances may be more worthy to annotate. Most active learning approaches for Machine Translation assume the existence of a pool of sentences in a source language, and rely on human annotators to provide translations or post-edits, which can still be costly. In this article, we apply active learning to a real-world human-in-the-loop scenario in which we assume that: (1) the source sentences may not be readily available, but instead arrive in a stream; (2) the automatic translations receive feedback in the form of a rating, instead of a correct\/edited translation, since the human-in-the-loop might be a user looking for a translation, but not be able to provide one. To tackle the challenge of deciding whether each incoming pair source\u2013translations is worthy to query for human feedback, we resort to a number of stream-based active learning query strategies. Moreover, because we do not know in advance which query strategy will be the most adequate for a certain language pair and set of Machine Translation models, we propose to dynamically combine multiple strategies using prediction with expert advice. Our experiments on different language pairs and feedback settings show that using active learning allows us to converge on the best Machine Translation systems with fewer human interactions. Furthermore, combining multiple strategies using prediction with expert advice outperforms several individual active learning strategies with even fewer interactions, particularly in partial feedback settings.<\/jats:p>","DOI":"10.1162\/coli_a_00473","type":"journal-article","created":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T21:00:34Z","timestamp":1671138034000},"page":"325-372","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":4,"title":["<i>Onception<\/i>: Active Learning with Expert Advice for Real World Machine Translation"],"prefix":"10.1162","volume":"49","author":[{"given":"V\u00e2nia","family":"Mendon\u00e7a","sequence":"first","affiliation":[{"name":"INESC-ID, Instituto Superior T\u00e9cnico. vania.mendonca@tecnico.ulisboa.pt"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ricardo","family":"Rei","sequence":"additional","affiliation":[{"name":"INESC-ID, Instituto Superior T\u00e9cnico, Unbabel AI. ricardo.rei@unbabel.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lu\u00edsa","family":"Coheur","sequence":"additional","affiliation":[{"name":"INESC-ID, Instituto Superior T\u00e9cnico. luisa.coheur@tecnico.ulisboa.pt"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alberto","family":"Sardinha","sequence":"additional","affiliation":[{"name":"PUC-Rio, INESC-ID, Instituto Superior T\u00e9cnico. jose.alberto.sardinha@tecnico.ulisboa.pt"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,6,1]]},"reference":[{"key":"2023061215395448800_bib1","first-page":"1","article-title":"Findings of the 2021 Conference on Machine Translation (WMT21)","volume-title":"Proceedings of the Sixth Conference on Machine Translation","author":"Akhbardeh","year":"2021"},{"key":"2023061215395448800_bib2","unstructured":"Ambati, Vamshi\n          . 2012. Active Learning and Crowd-Sourcing for Machine Translation in Low Resource Scenarios. Ph.D. thesis, Carnegie Mellon University."},{"key":"2023061215395448800_bib3","first-page":"122","article-title":"Multi-strategy approaches to active learning for statistical machine translation","volume-title":"MT Summit XIII: 13th Machine Translation Summit","author":"Ambati","year":"2011"},{"key":"2023061215395448800_bib4","first-page":"126","article-title":"A semi-supervised batch-mode active learning strategy for improved statistical machine translation","volume-title":"Proceedings of the Fourteenth Conference on Computational Natural Language Learning","author":"Ananthakrishnan","year":"2010"},{"key":"2023061215395448800_bib5","first-page":"322","article-title":"Gambling in a rigged casino: The adversarial multi-armed bandit problem","volume-title":"Annual Symposium on Foundations of Computer Science - Proceedings","author":"Auer","year":"1995"},{"issue":"1","key":"2023061215395448800_bib6","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1137\/S0097539701398375","article-title":"The nonstochastic multiarmed bandit problem","volume":"32","author":"Auer","year":"2002","journal-title":"SIAM Journal on Computing"},{"key":"2023061215395448800_bib7","first-page":"255","article-title":"Online choice of active learning algorithms","volume":"5","author":"Baram","year":"2004","journal-title":"Journal of Machine Learning Research"},{"key":"2023061215395448800_bib8","first-page":"1","article-title":"Findings of the 2020 Conference on Machine Translation (WMT20)","volume-title":"Proceedings of the Fifth Conference on Machine Translation","author":"Barrault","year":"2020"},{"key":"2023061215395448800_bib9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/W19-5301","article-title":"Findings of the 2019 Conference on Machine Translation (WMT19)","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Barrault","year":"2019"},{"key":"2023061215395448800_bib10","doi-asserted-by":"publisher","first-page":"103","DOI":"10.18653\/v1\/W19-5304","article-title":"The University of Edinburgh\u2019s submissions to the WMT19 news translation task","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Bawden","year":"2019"},{"key":"2023061215395448800_bib11","doi-asserted-by":"publisher","first-page":"116","DOI":"10.18653\/v1\/W19-5305","article-title":"GTCOM neural machine translation systems for WMT19","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Bei","year":"2019"},{"key":"2023061215395448800_bib12","volume-title":"Natural Language Processing with Python","author":"Bird","year":"2009"},{"key":"2023061215395448800_bib13","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1162\/tacl_a_00051","article-title":"Enriching word vectors with subword information","volume":"5","author":"Bojanowski","year":"2017","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023061215395448800_bib14","doi-asserted-by":"publisher","first-page":"129","DOI":"10.18653\/v1\/W19-5307","article-title":"LIUM\u2019s contributions to the WMT2019 news translation task: Data and systems for German-French language pairs","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Bougares","year":"2019"},{"key":"2023061215395448800_bib15","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511546921","volume-title":"Prediction, Learning and Games","author":"Cesa-Bianchi","year":"2006"},{"key":"2023061215395448800_bib16","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1109\/ICDM.2016.0100","article-title":"Can active learning experience be transferred?","volume-title":"2016 IEEE 16th International Conference on Data Mining (ICDM)","author":"Chu","year":"2016"},{"key":"2023061215395448800_bib17","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1007\/BF00993277","article-title":"Improving generalization with active learning","volume":"15","author":"Cohn","year":"1994","journal-title":"Machine Learning"},{"key":"2023061215395448800_bib18","doi-asserted-by":"publisher","first-page":"168","DOI":"10.18653\/v1\/W19-5313","article-title":"NICT\u2019s supervised neural machine translation systems for the WMT19 news translation task","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Dabre","year":"2019"},{"key":"2023061215395448800_bib19","doi-asserted-by":"publisher","first-page":"150","DOI":"10.1016\/B978-1-55860-377-6.50027-X","article-title":"Committee-based sampling for training probabilistic classifiers","volume-title":"Machine Learning Proceedings 1995","author":"Dagan","year":"1995"},{"key":"2023061215395448800_bib20","first-page":"4012","article-title":"Adversarial active learning for sequence labeling and generation","volume-title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18)","author":"Deng","year":"2018"},{"key":"2023061215395448800_bib21","doi-asserted-by":"publisher","first-page":"395","DOI":"10.3115\/v1\/E14-1042","article-title":"Learning from post-editing: Online model adaptation for statistical machine translation","volume-title":"Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics 2014","author":"Denkowski","year":"2014"},{"key":"2023061215395448800_bib22","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2023061215395448800_bib23","unstructured":"Eck, Matthias\n          . 2008. Developing Deployable Spoken Language Translation Systems Given Limited Resources. Ph.D. thesis, Fakult\u00e4t f\u00fcr Informatik der Universit\u00e4t Fridericiana zu Karlsruhe."},{"key":"2023061215395448800_bib24","article-title":"Low cost portability for statistical machine translation based on n-gram frequency and TF-IDF","volume-title":"Proceedings of the International Workshop on Spoken Language Translation (IWSLT 2005)","author":"Eck","year":"2005"},{"key":"2023061215395448800_bib25","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s10590-015-9176-1","article-title":"Survey of data-selection methods in statistical machine translation","volume":"29","author":"Eetemadi","year":"2015","journal-title":"Machine Translation"},{"key":"2023061215395448800_bib26","doi-asserted-by":"publisher","first-page":"595","DOI":"10.18653\/v1\/D17-1063","article-title":"Learning how to active learn: A deep reinforcement learning approach","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Fang","year":"2017"},{"key":"2023061215395448800_bib27","article-title":"Human-assisted neural machine translation: Harnessing human feedback for machine translation","author":"Finkelstein","year":"2020"},{"key":"2023061215395448800_bib28","doi-asserted-by":"publisher","first-page":"1460","DOI":"10.1162\/tacl_a_00437","article-title":"Experts, errors, and context: A large-scale study of human evaluation for machine translation","volume":"9","author":"Freitag","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023061215395448800_bib29","first-page":"733","article-title":"Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain","volume-title":"Proceedings of the Sixth Conference on Machine Translation (WMT)","author":"Freitag","year":"2021"},{"issue":"4","key":"2023061215395448800_bib30","first-page":"1","article-title":"Selective sampling for example-based word sense disambiguation","volume":"24","author":"Fujii","year":"1998","journal-title":"Computational Linguistics"},{"key":"2023061215395448800_bib31","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1016\/j.patrec.2013.06.007","article-title":"Cost-sensitive active learning for computer-assisted translation","volume":"37","author":"Gonz\u00e1lez-Rubio","year":"2014","journal-title":"Pattern Recognition Letters"},{"key":"2023061215395448800_bib32","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1145\/2070481.2070514","article-title":"An active learning scenario for interactive machine translation","volume-title":"ICMI\u201911 - Proceedings of the 2011 ACM International Conference on Multimodal Interaction","author":"Gonz\u00e1lez-Rubio","year":"2011"},{"key":"2023061215395448800_bib33","first-page":"245","article-title":"Active learning for interactive machine translation","volume-title":"Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics","author":"Gonz\u00e1lez-Rubio","year":"2012"},{"key":"2023061215395448800_bib34","doi-asserted-by":"publisher","first-page":"415","DOI":"10.3115\/1620754.1620815","article-title":"Active learning for statistical phrase-based machine translation","volume-title":"Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Conference","author":"Haffari","year":"2009"},{"key":"2023061215395448800_bib35","first-page":"1982","article-title":"Active2 learning: Actively reducing redundancies in active learning methods for sequence tagging and machine translation","volume-title":"NAACL-HLT 2021 - 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference","author":"Hazra","year":"2021"},{"key":"2023061215395448800_bib36","first-page":"133","article-title":"Language model adaptation for statistical machine translation based on information retrieval","volume-title":"EAMT 2005 Conference Proceedings","author":"Hildebrand","year":"2005"},{"key":"2023061215395448800_bib37","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1016\/j.neucom.2021.04.112","article-title":"Online learning: A comprehensive survey","volume":"459","author":"Hoi","year":"2021","journal-title":"Neurocomputing"},{"key":"2023061215395448800_bib38","first-page":"2659","article-title":"Active learning by learning","volume-title":"Proceedings of the National Conference on Artificial Intelligence","author":"Hsu","year":"2015"},{"key":"2023061215395448800_bib39","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1007\/978-3-642-14274-1_13","article-title":"EGAL: Exploration guided active learning for TCBR","volume-title":"Proceedings of ICCBR","author":"Hu","year":"2010"},{"key":"2023061215395448800_bib40","first-page":"57","article-title":"Direct translation model 2","volume-title":"Proceedings of the NAACL HLT 2007","author":"Ittycheriah","year":"2007"},{"issue":"2","key":"2023061215395448800_bib41","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1111\/j.1469-8137.1912.tb05611.x","article-title":"The distribution of the flora in the alpine zone","volume":"11","author":"Jaccard","year":"1912","journal-title":"New Phytologist"},{"key":"2023061215395448800_bib42","doi-asserted-by":"publisher","first-page":"225","DOI":"10.18653\/v1\/W19-5321","article-title":"Microsoft Translator at WMT 2019: Towards large-scale document-level neural machine translation","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Junczys-Dowmunt","year":"2019"},{"issue":"4","key":"2023061215395448800_bib43","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1007\/s10590-018-9224-8","article-title":"A user-study on online adaptation of neural machine translation to human post-edits","volume":"32","author":"Karimova","year":"2018","journal-title":"Machine Translation"},{"key":"2023061215395448800_bib44","first-page":"4226","article-title":"Learning active learning from data","volume-title":"31st Conference on Neural Information Processing Systems (NIPS 2017)","author":"Konyushkova","year":"2017"},{"issue":"1","key":"2023061215395448800_bib45","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1016\/0196-8858(85)90002-8","article-title":"Asymptotically efficient adaptive allocation rules","volume":"6","author":"Lai","year":"1985","journal-title":"Advances in Applied Mathematics"},{"key":"2023061215395448800_bib46","first-page":"169","article-title":"A reinforcement learning approach to interactive-predictive neural machine translation","volume-title":"21st Annual Conference of the European Association for Machine Translation","author":"Lam","year":"2018"},{"key":"2023061215395448800_bib47","first-page":"394","article-title":"Stream-based translation models for statistical machine translation","volume-title":"Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference","author":"Levenberg","year":"2010"},{"key":"2023061215395448800_bib48","first-page":"3","article-title":"A sequential algorithm for training text classifiers","volume-title":"Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 1994","author":"Lewis","year":"1994"},{"key":"2023061215395448800_bib49","doi-asserted-by":"publisher","first-page":"257","DOI":"10.18653\/v1\/W19-5325","article-title":"The NiuTrans machine translation systems for WMT19","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Li","year":"2019"},{"key":"2023061215395448800_bib50","doi-asserted-by":"publisher","first-page":"1874","DOI":"10.18653\/v1\/P18-1174","article-title":"Learning how to actively learn: A deep imitation learning approach","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Liu","year":"2018"},{"key":"2023061215395448800_bib51","doi-asserted-by":"publisher","first-page":"334","DOI":"10.18653\/v1\/K18-1033","article-title":"Learning to actively learn neural machine translation","volume-title":"Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018)","author":"Liu","year":"2018"},{"key":"2023061215395448800_bib52","first-page":"2690","article-title":"A quality-based active sample selection strategy for statistical machine translation","volume-title":"Proceedings of the 9th International Conference on Language Resources and Evaluation, LREC 2014","author":"Logacheva","year":"2014"},{"key":"2023061215395448800_bib53","doi-asserted-by":"publisher","first-page":"21","DOI":"10.18653\/v1\/D19-1003","article-title":"Practical obstacles to deploying active learning","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Lowell","year":"2019"},{"key":"2023061215395448800_bib54","first-page":"343","article-title":"Improving statistical machine translation performance by training data selection and optimization","volume-title":"Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","author":"L\u00fc","year":"2007"},{"key":"2023061215395448800_bib55","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1109\/SLT.2008.4777890","article-title":"Efficient data selection for machine translation","volume-title":"2008 IEEE Spoken Language Technology Workshop","author":"Mandal","year":"2008"},{"key":"2023061215395448800_bib56","first-page":"301","article-title":"Online learning approaches in computer assisted translation","volume-title":"Proceedings of the Eighth Workshop on Statistical Machine Translation","author":"Mathur","year":"2013"},{"key":"2023061215395448800_bib57","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/FUZZ48607.2020.9177707","article-title":"Query strategies, assemble! Active learning with expert advice for low-resource natural language processing","volume-title":"2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)","author":"Mendon\u00e7a","year":"2020"},{"key":"2023061215395448800_bib58","doi-asserted-by":"publisher","first-page":"3105","DOI":"10.18653\/v1\/2021.acl-long.242","article-title":"Online learning meets machine translation evaluation: Finding the best systems with the least human effort","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Mendon\u00e7a","year":"2021"},{"key":"2023061215395448800_bib59","first-page":"239","article-title":"WeChat neural machine translation systems for WMT20","volume-title":"Proceedings of the Fifth Conference on Machine Translation","author":"Meng","year":"2020"},{"key":"2023061215395448800_bib60","first-page":"50","article-title":"Machine translation system selection from bandit feedback","volume-title":"Proceedings of the 14th Conference of the Association for Machine Translation in the Americas","author":"Naradowsky","year":"2020"},{"key":"2023061215395448800_bib61","doi-asserted-by":"publisher","first-page":"314","DOI":"10.18653\/v1\/W19-5333","article-title":"Facebook FAIR\u2019s WMT19 news translation task submission","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Ng","year":"2019"},{"key":"2023061215395448800_bib62","first-page":"1464","article-title":"Reinforcement learning for bandit neural machine translation with simulated human feedback","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Nguyen","year":"2017"},{"key":"2023061215395448800_bib63","doi-asserted-by":"publisher","first-page":"320","DOI":"10.18653\/v1\/W19-5334","article-title":"eTranslation\u2019s submissions to the WMT 2019 news translation task","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Oravecz","year":"2019"},{"issue":"1","key":"2023061215395448800_bib64","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1162\/COLI_a_00244","article-title":"Online learning for statistical machine translation","volume":"42","author":"Ortiz-Mart\u00ednez","year":"2016","journal-title":"Computational Linguistics"},{"key":"2023061215395448800_bib65","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1109\/ICDM.2005.33","article-title":"Balancing exploration and exploitation: A new algorithm for active machine learning","volume-title":"Proceedings of the Fifth IEEE International Conference on Data Mining (ICDM\u201905)","author":"Osugi","year":"2005"},{"key":"2023061215395448800_bib66","doi-asserted-by":"publisher","first-page":"2269","DOI":"10.1109\/ICPR.2018.8545422","article-title":"Dynamic ensemble active learning: A non-stationary bandit with expert advice","volume-title":"2018 24th International Conference on Pattern Recognition (ICPR)","author":"Pang","year":"2018"},{"key":"2023061215395448800_bib67","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2023061215395448800_bib68","doi-asserted-by":"crossref","first-page":"151","DOI":"10.18653\/v1\/K18-1015","article-title":"Active learning for interactive neural machine translation of data streams","volume-title":"Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018)","author":"Peris","year":"2018"},{"key":"2023061215395448800_bib69","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1016\/j.csl.2019.04.001","article-title":"Online learning for effort reduction in interactive neural machine translation","volume":"58","author":"Peris","year":"2019","journal-title":"Computer Speech and Language"},{"key":"2023061215395448800_bib70","doi-asserted-by":"publisher","first-page":"327","DOI":"10.18653\/v1\/W19-5335","article-title":"Tilde\u2019s machine translation systems for WMT 2019","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Pinnis","year":"2019"},{"key":"2023061215395448800_bib71","doi-asserted-by":"publisher","first-page":"392","DOI":"10.18653\/v1\/W15-3049","article-title":"chrF: Character n-gram F-score for automatic MT evaluation","volume-title":"Proceedings of the Tenth Workshop on Statistical Machine Translation","author":"Popovi\u0107","year":"2015"},{"key":"2023061215395448800_bib72","doi-asserted-by":"publisher","first-page":"186","DOI":"10.18653\/v1\/W18-6319","article-title":"A call for clarity in reporting BLEU scores","volume-title":"Proceedings of the Third Conference on Machine Translation: Research Papers","author":"Post","year":"2018"},{"key":"2023061215395448800_bib73","doi-asserted-by":"publisher","first-page":"2685","DOI":"10.18653\/v1\/2020.emnlp-main.213","article-title":"COMET: A neural framework for MT evaluation","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Rei","year":"2020"},{"issue":"5","key":"2023061215395448800_bib74","doi-asserted-by":"publisher","first-page":"527","DOI":"10.1090\/S0002-9904-1952-09620-8","article-title":"Some aspects of the sequential design of experiments","volume":"58","author":"Robbins","year":"1952","journal-title":"Bulletin of the American Mathematical Society"},{"key":"2023061215395448800_bib75","unstructured":"Settles, Burr\n          . 2010. Active learning literature survey. Technical report. University of Wisconsin-Madison Department of Computer Sciences."},{"key":"2023061215395448800_bib76","doi-asserted-by":"publisher","first-page":"1070","DOI":"10.3115\/1613715.1613855","article-title":"An analysis of active learning strategies for sequence labeling tasks","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Settles","year":"2008"},{"key":"2023061215395448800_bib77","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1145\/130385.130417","article-title":"Query by committee","volume-title":"Proceedings of the Fifth Annual Workshop on Computational Learning Theory","author":"Seung","year":"1992"},{"key":"2023061215395448800_bib78","doi-asserted-by":"publisher","first-page":"1610","DOI":"10.18653\/v1\/P16-1152","article-title":"Learning structured predictors from bandit feedback for interactive NLP","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","author":"Sokolov","year":"2016"},{"key":"2023061215395448800_bib79","doi-asserted-by":"publisher","first-page":"514","DOI":"10.18653\/v1\/W17-4756","article-title":"A shared task on bandit learning for machine translation","volume-title":"Proceedings of the Conference on Machine Translation (WMT)","author":"Sokolov","year":"2017"},{"key":"2023061215395448800_bib80","doi-asserted-by":"crossref","first-page":"90","DOI":"10.18653\/v1\/2020.emnlp-main.8","article-title":"Automatic machine translation evaluation in many languages via zero-shot paraphrasing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Thompson","year":"2020"},{"issue":"1","key":"2023061215395448800_bib81","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1515\/pralin-2017-0023","article-title":"Continuous learning from human post-edits for neural machine translation","volume":"108","author":"Turchi","year":"2017","journal-title":"The Prague Bulletin of Mathematical Linguistics"},{"key":"2023061215395448800_bib82","first-page":"5999","article-title":"Attention is all you need","volume-title":"31st Conference on Neural Information Processing Systems (NIPS 2017)","author":"Vaswani","year":"2017"},{"key":"2023061215395448800_bib83","doi-asserted-by":"crossref","first-page":"4091","DOI":"10.18653\/v1\/P19-1401","article-title":"Learning how to active learn by dreaming","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Vu","year":"2019"},{"key":"2023061215395448800_bib84","doi-asserted-by":"publisher","first-page":"451","DOI":"10.18653\/v1\/P18-1042","article-title":"ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Wieting","year":"2018"},{"key":"2023061215395448800_bib85","first-page":"305","article-title":"The volctrans machine translation system for WMT20","volume-title":"Proceedings of the Fifth Conference on Machine Translation","author":"Wu","year":"2020"},{"key":"2023061215395448800_bib86","doi-asserted-by":"publisher","first-page":"424","DOI":"10.18653\/v1\/W19-5348","article-title":"Microsoft Research Asia\u2019s systems for WMT19","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Xia","year":"2019"},{"key":"2023061215395448800_bib87","unstructured":"Xiao, Han\n          . 2018. bert-as-service. https:\/\/github.com\/hanxiao\/bert-as-service."},{"key":"2023061215395448800_bib88","doi-asserted-by":"publisher","first-page":"84","DOI":"10.18653\/v1\/D19-6110","article-title":"Empirical evaluation of active learning techniques for neural MT","volume-title":"Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo)","author":"Zeng","year":"2019"},{"key":"2023061215395448800_bib89","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1109\/IALP.2018.8629116","article-title":"Active learning for neural machine translation","volume-title":"Proceedings of the 2018 International Conference on Asian Language Processing, IALP 2018","author":"Zhang","year":"2018"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/49\/2\/325\/2125578\/coli_a_00473.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/49\/2\/325\/2125578\/coli_a_00473.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,12]],"date-time":"2023-06-12T15:40:51Z","timestamp":1686584451000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/49\/2\/325\/114185\/Onception-Active-Learning-with-Expert-Advice-for"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":89,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,6,1]]},"published-print":{"date-parts":[[2023,6,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00473","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}