{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T17:52:43Z","timestamp":1790445163257,"version":"4.1.0"},"reference-count":53,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> Hyperparameter selection is a crucial part of building neural machine translation (NMT) systems across both academia and industry. Fine-grained adjustments to a model\u2019s architecture or training recipe can mean the difference between a positive and negative research result or between a state-of-the-art and underperforming system. While recent literature has proposed methods for automatic hyperparameter optimization (HPO), there has been limited work on applying these methods to neural machine translation (NMT), due in part to the high costs associated with experiments that train large numbers of model variants. To facilitate research in this space, we introduce a lookup-based approach that uses a library of pre-trained models for fast, low cost HPO experimentation. Our contributions include (1) the release of a large collection of trained NMT models covering a wide range of hyperparameters, (2) the proposal of targeted metrics for evaluating HPO methods on NMT, and (3) a reproducible benchmark of several HPO methods against our model library, including novel graph-based and multiobjective methods. <\/jats:p>","DOI":"10.1162\/tacl_a_00322","type":"journal-article","created":{"date-parts":[[2020,7,20]],"date-time":"2020-07-20T18:01:16Z","timestamp":1595268076000},"page":"393-408","source":"Crossref","is-referenced-by-count":15,"title":["Reproducible and Efficient Benchmarks for Hyperparameter Optimization of Neural Machine Translation Systems"],"prefix":"10.1162","volume":"8","author":[{"given":"Xuan","family":"Zhang","sequence":"first","affiliation":[{"name":"Johns Hopkins University."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Duh","sequence":"additional","affiliation":[{"name":"Johns Hopkins University."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","author":"Ambikasaran Sivaram","year":"2014","journal-title":"arXiv preprint arXiv:1403.6015"},{"issue":"1","key":"bib2","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1515\/pralin-2017-0005","volume":"108","author":"Bahar Parnia","year":"2017","journal-title":"The Prague Bulletin of Mathematical Linguistics"},{"key":"bib3","volume-title":"Proceedings of the 3rd International Conference on Learning Representations","author":"Bahdanau Dzmitry","year":"2015"},{"key":"bib4","volume-title":"Proceedings of the 30th International Conference on Machine Learning","author":"Bardenet R\u00e9mi","year":"2013"},{"key":"bib5","author":"Beck Daniel","year":"2016","journal-title":"arXiv preprint arXiv:1604.05073"},{"key":"bib6","first-page":"281","volume":"13","author":"Bergstra James","year":"2012","journal-title":"Journal of Machine Learning Research"},{"key":"bib7","volume-title":"Proceedings of the 25th Advances in Neural Information Processing Systems","author":"Bergstra James S.","year":"2011"},{"key":"bib8","author":"Britz Denny","year":"2017","journal-title":"arXiv preprint arXiv:1703.03906"},{"key":"bib9","author":"Brochu Eric","year":"2010","journal-title":"arXiv preprint arXiv:1012.2599"},{"key":"bib10","volume-title":"Thirty-Second AAAI Conference on Artificial Intelligence","author":"Cai Han","year":"2018"},{"key":"bib11","unstructured":"Kevin Duh. 2018. The multitarget TED talks task. http:\/\/www.cs.jhu.edu\/~kevinduh\/a\/multitarget-tedtalks\/."},{"key":"bib12","volume-title":"International Conference on Evolutionary Programming","author":"Eberhart Russell C.","year":"1998"},{"key":"bib13","volume-title":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","author":"Eggensperger Katharina","year":"2015"},{"key":"bib14","volume-title":"2011 IEEE Congress of Evolutionary Computation (CEC)","author":"Emmerich Michael T. M.","year":"2011"},{"key":"bib15","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-3-030-05318-5_1","volume-title":"Automated Machine Learning","author":"Feurer Matthias","year":"2019"},{"key":"bib16","volume-title":"Twenty-Ninth AAAI Conference on Artificial Intelligence","author":"Feurer Matthias","year":"2015"},{"key":"bib17","author":"Frazier Peter I.","year":"2018","journal-title":"arXiv preprint arXiv:1807.02811"},{"key":"bib18","volume-title":"Proceedings of the 34th International Conference on Machine Learning-Volume 70","author":"Gehring Jonas","year":"2017"},{"issue":"1","key":"bib19","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/s00778-006-0029-7","volume":"16","author":"Godfrey Parke","year":"2007","journal-title":"The VLDB JournalThe International Journal on Very Large Data Bases"},{"key":"bib20","author":"Hansen Nikolaus","year":"2016","journal-title":"arXiv preprint arXiv:1605.03560"},{"key":"bib21","author":"Hieber Felix","year":"2017","journal-title":"arXiv preprint arXiv:1712.05690"},{"key":"bib22","volume-title":"Proceedings of the 5th International Conference on Learning and Intelligent Optimization","author":"Hutter Frank","year":"2011"},{"issue":"4","key":"bib23","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1023\/A:1008306431147","volume":"13","author":"Jones Donald R.","year":"1998","journal-title":"Journal of Global Optimization"},{"key":"bib24","author":"Klein Aaron","year":"2019","journal-title":"arXiv preprint arXiv:1905.12982"},{"key":"bib25","author":"Klein Aaron","year":"2019","journal-title":"arXiv preprint arXiv: 1905.04970"},{"key":"bib26","volume-title":"Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI)","author":"Li Liam","year":"2019"},{"key":"bib27","volume-title":"Proceedings of the Fourth Conference on Machine Translation","author":"Li Xian","year":"2019"},{"key":"bib28","author":"Lim Robert","year":"2018","journal-title":"arXiv preprint arXiv:1805.02094"},{"key":"bib29","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV)","author":"Liu Chenxi","year":"2018"},{"key":"bib30","volume-title":"International Conference on Learning Representations","author":"Liu Hanxiao","year":"2018"},{"issue":"1","key":"bib31","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1007\/s13721-016-0125-6","volume":"5","author":"Luo Gang","year":"2016","journal-title":"Network Modeling Analysis in Health Informatics and Bioinformatics"},{"key":"bib32","author":"Miao Yishu","year":"2014","journal-title":"arXiv preprint arXiv:1412.7180"},{"key":"bib33","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics","author":"Papineni Kishore","year":"2002"},{"issue":"6","key":"bib34","doi-asserted-by":"crossref","first-page":"1265","DOI":"10.1007\/s11222-014-9477-x","volume":"25","author":"Picheny Victor","year":"2015","journal-title":"Statistics and Computing"},{"key":"bib35","doi-asserted-by":"crossref","first-page":"784","DOI":"10.1007\/978-3-540-87700-4_78","volume-title":"International Conference on Parallel Problem Solving from Nature","author":"Ponweiser Wolfgang","year":"2008"},{"key":"bib36","volume-title":"Proceedings of the 14th International Workshop on Spoken Language Translation","author":"Qin Hao","year":"2017"},{"key":"bib37","first-page":"63","volume-title":"Summer School on Machine Learning","author":"Rasmussen Carl Edward","year":"2003"},{"key":"bib38","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Real Esteban","year":"2019"},{"key":"bib39","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1214\/lnms\/1215456182","author":"Schonlau Matthias","year":"1998","journal-title":"Lecture Notes-Monograph Series"},{"key":"bib40","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics","author":"Sennrich Rico","year":"2016"},{"key":"bib41","first-page":"1919","volume-title":"International Conference on Machine Learning","author":"Shah Amar","year":"2016"},{"issue":"1","key":"bib42","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1109\/JPROC.2015.2494218","volume":"104","author":"Shahriari Bobak","year":"2015","journal-title":"Proceedings of the IEEE"},{"key":"bib43","volume-title":"Evolutionary optimization algorithms","author":"Simon Dan","year":"2013"},{"key":"bib44","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","author":"Snoek Jasper","year":"2015"},{"key":"bib45","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"So David","year":"2019"},{"key":"bib46","volume-title":"Proceedings of the 28th Advances in Neural Information Processing Systems","author":"Sutskever Ilya","year":"2014"},{"key":"bib47","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1016\/j.csda.2015.08.011","volume":"94","author":"Svenson Joshua","year":"2016","journal-title":"Computational Statistics & Data Analysis"},{"key":"bib48","volume-title":"Proceedings of the 31st Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017"},{"key":"bib49","author":"Ying Chris","year":"2019","journal-title":"arXiv preprint arXiv:1902.09635"},{"key":"bib50","unstructured":"Xiaojin Zhu. 2005. Semi-supervised learning with graphs. Ph.D. Thesis."},{"key":"bib51","volume-title":"Proceedings of the 20th International Conference on Machine Learning (ICML-03)","author":"Zhu Xiaojin","year":"2003"},{"key":"bib52","volume-title":"International Conference on Parallel Problem Solving from Nature","author":"Zitzler Eckart","year":"1998"},{"key":"bib53","author":"Zoph Barret","year":"2016","journal-title":"arXiv preprint arXiv:1611.01578"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00322","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:40Z","timestamp":1615585180000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96458"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":53,"alternative-id":["10.1162\/tacl_a_00322"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00322","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}