{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T11:06:52Z","timestamp":1780916812611,"version":"3.54.1"},"reference-count":74,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T00:00:00Z","timestamp":1778025600000},"content-version":"vor","delay-in-days":125,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,5,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Reinforcement learning (RL) has been proven to be an effective and robust method for training neural machine translation systems, especially when paired with powerful reward models that accurately assess translation quality. However, most research has focused on RL methods that use sentence-level feedback, leading to inefficient learning signals due to the reward sparsity problem\u2014the model receives a single score for the entire sentence. To address this, we propose a novel approach that leverages fine-grained, token-level quality assessments along with error severity levels using RL methods. Specifically, we use xCOMET, a state-of-the-art quality estimation system, as our token-level reward model. We conduct experiments on small and large translation datasets with standard encoder-decoder and large language models-based machine translation systems, comparing the impact of sentence-level versus fine-grained reward signals on translation quality. Our results show that training with token-level rewards improves translation quality across language pairs over baselines according to both automatic and human evaluation. Furthermore, token-level reward optimization improves training stability, evidenced by a steady increase in mean rewards over training epochs.<\/jats:p>","DOI":"10.1162\/tacl.a.646","type":"journal-article","created":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T18:01:06Z","timestamp":1778090466000},"page":"733-754","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["Fine-Grained Reward Optimization for Machine Translation using Error\n                    Severity Mappings"],"prefix":"10.1162","volume":"14","author":[{"given":"Miguel Moura","family":"Ramos","sequence":"first","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa (ELLIS Unit Lisbon), Portugal. miguel.moura.ramos@tecnico.ulisboa.pt"},{"name":"Instituto de Telecomunica\u00e7\u00f5es, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tom\u00e1s","family":"Almeida","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa (ELLIS Unit Lisbon), Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Vareta","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa (ELLIS Unit Lisbon), Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Filipe","family":"Azevedo","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa (ELLIS Unit Lisbon), Portugal"},{"name":"Instituto de Telecomunica\u00e7\u00f5es, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sweta","family":"Agrawal","sequence":"additional","affiliation":[{"name":"Instituto de Telecomunica\u00e7\u00f5es, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick","family":"Fernandes","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa (ELLIS Unit Lisbon), Portugal"},{"name":"Instituto de Telecomunica\u00e7\u00f5es, Portugal"},{"name":"Carnegie Mellon University, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andr\u00e9 F. T.","family":"Martins","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa (ELLIS Unit Lisbon), Portugal"},{"name":"Instituto de Telecomunica\u00e7\u00f5es, Portugal"},{"name":"TransPerfect, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,5,4]]},"reference":[{"key":"2026050614005950700_bib1","doi-asserted-by":"publisher","first-page":"14503","DOI":"10.18653\/v1\/2024.emnlp-main.803","article-title":"Modeling user preferences with automatic\n                        metrics: Creating a high-quality preference dataset for machine\n                        translation","volume-title":"Proceedings of the 2024 Conference\n                        on Empirical Methods in Natural Language Processing","author":"Agrawal","year":"2024"},{"key":"2026050614005950700_bib2","doi-asserted-by":"publisher","first-page":"12248","DOI":"10.18653\/v1\/2024.acl-long.662","article-title":"Back to basics: Revisiting REINFORCE-style\n                        optimization for learning from human feedback in LLMs","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for\n                        Computational Linguistics (Volume 1: Long Papers)","author":"Ahmadian","year":"2024"},{"key":"2026050614005950700_bib3","article-title":"Tower: An open multilingual large language\n                        model for translation-related tasks","volume-title":"First\n                        Conference on Language Modeling","author":"Alves","year":"2024"},{"key":"2026050614005950700_bib4","article-title":"An actor-critic algorithm for sequence\n                        prediction","volume-title":"International Conference on Learning\n                        Representations","author":"Bahdanau","year":"2017"},{"key":"2026050614005950700_bib5","article-title":"Scheduled sampling for sequence prediction\n                        with recurrent neural networks","volume-title":"Advances in\n                        Neural Information Processing Systems","author":"Bengio","year":"2015"},{"key":"2026050614005950700_bib6","doi-asserted-by":"publisher","first-page":"272","DOI":"10.18653\/v1\/W18-6401","article-title":"Findings of the 2018 conference on machine translation\n                        (WMT18)","volume-title":"Proceedings of the Third Conference on\n                        Machine Translation: Shared Task Papers","author":"Bojar","year":"2018"},{"key":"2026050614005950700_bib7","doi-asserted-by":"publisher","first-page":"9119","DOI":"10.18653\/v1\/2024.emnlp-main.515","article-title":"Enhancing reinforcement learning with dense rewards from\n                        language model critic","volume-title":"Proceedings of the 2024\n                        Conference on Empirical Methods in Natural Language Processing","author":"Cao","year":"2024"},{"key":"2026050614005950700_bib8","first-page":"2","article-title":"Overview of the IWSLT 2017 evaluation\n                        campaign","volume-title":"Proceedings of the 14th International\n                        Conference on Spoken Language Translation","author":"Cettolo","year":"2017"},{"key":"2026050614005950700_bib9","doi-asserted-by":"publisher","first-page":"103","DOI":"10.3115\/v1\/W14-4012","article-title":"On the properties of neural machine\n                        translation: Encoder\u2013decoder approaches","volume-title":"Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and\n                        Structure in Statistical Translation","author":"Cho","year":"2014"},{"key":"2026050614005950700_bib10","doi-asserted-by":"publisher","first-page":"26915","DOI":"10.52202\/068431-1951","article-title":"What are the best systems? New\n                        perspectives on NLP benchmarking","volume-title":"Advances in\n                        Neural Information Processing Systems","author":"Colombo","year":"2022"},{"key":"2026050614005950700_bib11","article-title":"Flashattention-2: Faster attention with better parallelism\n                        and work partitioning","volume-title":"The Twelfth International\n                        Conference on Learning Representations","author":"Dao","year":"2024"},{"key":"2026050614005950700_bib12","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1189","article-title":"Flashattention: Fast and memory-efficient exact attention\n                        with IO-awareness","volume-title":"Advances in Neural Information\n                        Processing Systems","author":"Dao","year":"2022"},{"key":"2026050614005950700_bib13","doi-asserted-by":"publisher","first-page":"5829","DOI":"10.52202\/068431-0422","article-title":"Exploration-guided reward shaping for\n                        reinforcement learning under sparse rewards","volume-title":"Advances in Neural Information Processing Systems","author":"Devidze","year":"2022"},{"issue":"4","key":"2026050614005950700_bib14","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1007\/s10590-017-9213-3","article-title":"Segment-based interactive-predictive\n                        machine translation","volume":"31","author":"Domingo","year":"2017","journal-title":"Machine Translation"},{"key":"2026050614005950700_bib15","doi-asserted-by":"publisher","first-page":"1066","DOI":"10.18653\/v1\/2023.wmt-1.100","article-title":"The devil is in the errors: Leveraging large\n                        language models for fine-grained machine translation\n                        evaluation","volume-title":"Proceedings of the Eighth Conference\n                        on Machine Translation","author":"Fernandes","year":"2023"},{"key":"2026050614005950700_bib16","doi-asserted-by":"publisher","first-page":"578","DOI":"10.18653\/v1\/2023.wmt-1.51","article-title":"Results of WMT23 metrics shared task:\n                        Metrics might be guilty but references are not innocent","volume-title":"Proceedings of the Eighth Conference on Machine\n                    Translation","author":"Freitag","year":"2023"},{"key":"2026050614005950700_bib17","doi-asserted-by":"publisher","first-page":"46","DOI":"10.18653\/v1\/2022.wmt-1.2","article-title":"Results of WMT22 metrics shared task: Stop\n                        using BLEU \u2013 neural metrics are better and more\n                        robust","volume-title":"Proceedings of the Seventh Conference on\n                        Machine Translation (WMT)","author":"Freitag","year":"2022"},{"key":"2026050614005950700_bib18","first-page":"23","article-title":"A new algorithm for data compression","volume":"12","author":"Gage","year":"1994","journal-title":"The C Users Journal Archive"},{"key":"2026050614005950700_bib19","doi-asserted-by":"publisher","first-page":"2385","DOI":"10.24963\/ijcai.2019\/331","article-title":"Using natural language for reward shaping\n                        in reinforcement learning","volume-title":"Proceedings of the\n                        Twenty-Eighth International Joint Conference on Artificial Intelligence,\n                        IJCAI-19","author":"Goyal","year":"2019"},{"key":"2026050614005950700_bib20","first-page":"33","article-title":"Continuous measurement scales in human\n                        evaluation of machine translation","volume-title":"Proceedings of\n                        the 7th Linguistic Annotation Workshop and Interoperability with\n                        Discourse","author":"Graham","year":"2013"},{"key":"2026050614005950700_bib21","doi-asserted-by":"publisher","first-page":"979","DOI":"10.1162\/tacl_a_00683","article-title":"xcomet: Transparent machine translation\n                        evaluation through fine-grained error detection","volume":"12","author":"Guerreiro","year":"2024","journal-title":"Transactions of the Association for Computational\n                        Linguistics"},{"key":"2026050614005950700_bib22","article-title":"Beyond imitation: Leveraging fine-grained quality signals for\n                        alignment","volume-title":"The Twelfth International Conference\n                        on Learning Representations, ICLR 2024, Vienna, Austria, May 7\u201311,\n                        2024","author":"Guo","year":"2024"},{"key":"2026050614005950700_bib23","doi-asserted-by":"publisher","first-page":"8164","DOI":"10.18653\/v1\/2024.naacl-long.451","article-title":"Improving machine translation with human\n                        feedback: An exploration of quality estimation as a reward\n                        model","volume-title":"Proceedings of the 2024 Conference of the\n                        North American Chapter of the Association for Computational Linguistics:\n                        Human Language Technologies (Volume 1: Long Papers)","author":"He","year":"2024"},{"key":"2026050614005950700_bib24","article-title":"Distilling the knowledge in a neural network","author":"Hinton","year":"2015","journal-title":"arXiv e-prints"},{"key":"2026050614005950700_bib25","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00286","article-title":"Quantization and training of neural\n                        networks for efficient integer-arithmetic-only inference","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern\n                        Recognition (CVPR)","author":"Jacob","year":"2018"},{"key":"2026050614005950700_bib26","article-title":"Personalized soups: Personalized large\n                        language model alignment via post-hoc parameter merging","volume-title":"Adaptive Foundation Models: Evolving AI for Personalized and\n                        Efficient Learning","author":"Jang","year":"2024"},{"key":"2026050614005950700_bib27","doi-asserted-by":"publisher","first-page":"756","DOI":"10.18653\/v1\/2023.wmt-1.63","article-title":"MetricX-23: The Google submission to the\n                        WMT 2023 metrics shared task","volume-title":"Proceedings of the\n                        Eighth Conference on Machine Translation","author":"Juraska","year":"2023"},{"key":"2026050614005950700_bib28","doi-asserted-by":"publisher","first-page":"1700","DOI":"10.18653\/v1\/D13-1176","article-title":"Recurrent continuous translation\n                        models","volume-title":"Proceedings of the 2013 Conference on\n                        Empirical Methods in Natural Language Processing","author":"Kalchbrenner","year":"2013"},{"key":"2026050614005950700_bib29","doi-asserted-by":"publisher","first-page":"1673","DOI":"10.18653\/v1\/2021.naacl-main.133","article-title":"Revisiting the weaknesses of reinforcement\n                        learning for neural machine translation","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of\n                        the Association for Computational Linguistics: Human Language\n                        Technologies","author":"Kiegeland","year":"2021"},{"key":"2026050614005950700_bib30","article-title":"Adam: A method for stochastic optimization","volume-title":"3rd International Conference on Learning Representations, ICLR 2015,\n                        San Diego, CA, USA, May 7\u20139, 2015, Conference Track\n                        Proceedings","author":"Kingma","year":"2015"},{"key":"2026050614005950700_bib31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2024.wmt-1.1","article-title":"Findings of the WMT24 general machine\n                        translation shared task: The LLM era is here but MT is not solved\n                        yet","volume-title":"Proceedings of the Ninth Conference on\n                        Machine Translation","author":"Kocmi","year":"2024"},{"key":"2026050614005950700_bib32","first-page":"388","article-title":"Statistical significance tests for machine\n                        translation evaluation","volume-title":"Proceedings of the 2004\n                        Conference on Empirical Methods in Natural Language Processing","author":"Koehn","year":"2004"},{"key":"2026050614005950700_bib33","first-page":"135","article-title":"Correct me if you can: Learning from error\n                        corrections and markings","volume-title":"Proceedings of the 22nd\n                        Annual Conference of the European Association for Machine\n                        Translation","author":"Kreutzer","year":"2020"},{"key":"2026050614005950700_bib34","doi-asserted-by":"publisher","first-page":"92","DOI":"10.18653\/v1\/N18-3012","article-title":"Can neural machine translation be\n                        improved with user feedback?","volume-title":"Proceedings of the\n                        2018 Conference of the North American Chapter of the Association for\n                        Computational Linguistics: Human Language Technologies, Volume 3 (Industry\n                        Papers)","author":"Kreutzer","year":"2018"},{"key":"2026050614005950700_bib35","doi-asserted-by":"publisher","first-page":"1777","DOI":"10.18653\/v1\/P18-1165","article-title":"Reliability and learnability of human\n                        bandit feedback for sequence-to-sequence reinforcement\n                        learning","volume-title":"Proceedings of the 56th Annual Meeting\n                        of the Association for Computational Linguistics (Volume 1: Long\n                        Papers)","author":"Kreutzer","year":"2018"},{"key":"2026050614005950700_bib36","doi-asserted-by":"publisher","first-page":"66","DOI":"10.18653\/v1\/D18-2012","article-title":"SentencePiece: A simple and language\n                        independent subword tokenizer and detokenizer for neural text\n                        processing","volume-title":"Proceedings of the 2018 Conference on\n                        Empirical Methods in Natural Language Processing: System\n                        Demonstrations","author":"Kudo","year":"2018"},{"key":"2026050614005950700_bib37","doi-asserted-by":"publisher","first-page":"1704","DOI":"10.18653\/v1\/2024.findings-naacl.111","article-title":"Reinforcement learning with token-level\n                        feedback for controllable text generation","volume-title":"Findings of the Association for Computational Linguistics: NAACL\n                        2024","author":"Li","year":"2024"},{"key":"2026050614005950700_bib38","doi-asserted-by":"publisher","first-page":"455","DOI":"10.5565\/rev\/tradumatica.77","article-title":"Multidimensional quality metrics (MQM): A\n                        framework for declaring and describing translation quality\n                        metrics","volume":"0","author":"Lommel","year":"2014","journal-title":"Tradum\u00e0tica: Tecnologies de la\n                        traducci\u00f3"},{"key":"2026050614005950700_bib39","doi-asserted-by":"publisher","first-page":"5892","DOI":"10.18653\/v1\/2023.acl-long.324","article-title":"Toward human-like evaluation for natural language generation\n                        with error analysis","volume-title":"Proceedings of the 61st\n                        Annual Meeting of the Association for Computational Linguistics (Volume 1:\n                        Long Papers)","author":"Qingyu","year":"2023"},{"key":"2026050614005950700_bib40","doi-asserted-by":"publisher","first-page":"8801","DOI":"10.18653\/v1\/2024.findings-acl.520","article-title":"Error analysis prompting enables human-like translation\n                        evaluation in large language models","volume-title":"Findings of\n                        the Association for Computational Linguistics: ACL 2024","author":"Qingyu","year":"2024"},{"key":"2026050614005950700_bib41","first-page":"278","article-title":"Policy invariance under reward\n                        transformations: Theory and application to reward shaping","volume-title":"Proceedings of the Sixteenth International Conference on Machine\n                        Learning","author":"Ng","year":"1999"},{"key":"2026050614005950700_bib42","doi-asserted-by":"publisher","first-page":"1464","DOI":"10.18653\/v1\/D17-1153","article-title":"Reinforcement learning for bandit neural\n                        machine translation with simulated human feedback","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural\n                        Language Processing","author":"Nguyen","year":"2017"},{"key":"2026050614005950700_bib43","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1038\/s41586-024-07335-x","article-title":"Scaling neural machine translation to 200\n                        languages","volume":"630","author":"Team","year":"2024","journal-title":"Nature"},{"key":"2026050614005950700_bib44","doi-asserted-by":"publisher","first-page":"27730","DOI":"10.52202\/068431-2011","article-title":"Training language models to follow instructions with human\n                        feedback","volume-title":"Advances in Neural Information\n                        Processing Systems","author":"Ouyang","year":"2022"},{"key":"2026050614005950700_bib45","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.emnlp-main.1589","article-title":"Token-level Proximal Policy Optimization\n                        for Query Generation","author":"Ouyang","year":"2024","journal-title":"arXiv e-prints"},{"key":"2026050614005950700_bib46","doi-asserted-by":"publisher","first-page":"311","DOI":"10.3115\/1073083.1073135","article-title":"BLEU: A method for automatic evaluation of machine\n                        translation","volume-title":"Proceedings of the 40th Annual\n                        Meeting on Association for Computational Linguistics","author":"Papineni","year":"2002"},{"key":"2026050614005950700_bib47","doi-asserted-by":"publisher","first-page":"569","DOI":"10.18653\/v1\/2022.wmt-1.51","article-title":"MaTESe: Machine translation evaluation as\n                        a sequence tagging problem","volume-title":"Proceedings of the\n                        Seventh Conference on Machine Translation (WMT)","author":"Perrella","year":"2022"},{"key":"2026050614005950700_bib48","doi-asserted-by":"publisher","first-page":"326","DOI":"10.18653\/v1\/P18-2052","article-title":"Learning from chunk-based feedback in\n                        neural machine translation","volume-title":"Proceedings of the\n                        56th Annual Meeting of the Association for Computational Linguistics (Volume\n                        2: Short Papers)","author":"Petrushkov","year":"2018"},{"key":"2026050614005950700_bib49","doi-asserted-by":"publisher","first-page":"392","DOI":"10.18653\/v1\/W15-3049","article-title":"chrF: Character n-gram F-score for\n                        automatic MT evaluation","volume-title":"Proceedings of the Tenth\n                        Workshop on Statistical Machine Translation","author":"Popovi\u0107","year":"2015"},{"key":"2026050614005950700_bib50","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1016\/S0927-0507(05)80172-0","article-title":"Chapter 8 Markov decision\n                        processes","volume-title":"Stochastic Models","author":"Puterman","year":"1990"},{"key":"2026050614005950700_bib51","doi-asserted-by":"publisher","first-page":"53728","DOI":"10.52202\/075280-2338","article-title":"Direct preference optimization: Your language model is\n                        secretly a reward model","volume-title":"Advances in Neural\n                        Information Processing Systems","author":"Rafailov","year":"2023"},{"key":"2026050614005950700_bib52","first-page":"258","article-title":"Aligning neural machine translation\n                        models: Human feedback in training and inference","volume-title":"Proceedings of the 25th Annual Conference of the European\n                        Association for Machine Translation (Volume 1)","author":"Ramos","year":"2024"},{"key":"2026050614005950700_bib53","article-title":"Sequence level training with recurrent\n                        neural networks","volume-title":"4th International Conference on\n                        Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2\u20134,\n                        2016, Conference Track Proceedings","author":"Ranzato","year":"2016"},{"key":"2026050614005950700_bib54","doi-asserted-by":"publisher","first-page":"841","DOI":"10.18653\/v1\/2023.wmt-1.73","article-title":"Scaling up CometKiwi: Unbabel-IST 2023\n                        submission for the quality estimation shared task","volume-title":"Proceedings of the Eighth Conference on Machine\n                    Translation","author":"Rei","year":"2023"},{"key":"2026050614005950700_bib55","doi-asserted-by":"publisher","first-page":"578","DOI":"10.18653\/v1\/2022.wmt-1.52","article-title":"COMET-22: Unbabel-IST 2022 submission for\n                        the metrics shared task","volume-title":"Proceedings of the\n                        Seventh Conference on Machine Translation (WMT)","author":"Rei","year":"2022"},{"key":"2026050614005950700_bib56","article-title":"Gemma 2: Improving open language models at\n                        a practical size","author":"Rivi\u00e8re","year":"2024","journal-title":"CoRR"},{"key":"2026050614005950700_bib57","article-title":"High-dimensional continuous control using\n                        generalized advantage estimation","volume-title":"4th\n                        International Conference on Learning Representations, ICLR 2016, San Juan,\n                        Puerto Rico, May 2\u20134, 2016, Conference Track\n                    Proceedings","author":"Schulman","year":"2016"},{"key":"2026050614005950700_bib58","article-title":"Proximal policy optimization\n                        algorithms","author":"Schulman","year":"2017","journal-title":"arXiv e-prints"},{"key":"2026050614005950700_bib59","doi-asserted-by":"publisher","first-page":"7881","DOI":"10.18653\/v1\/2020.acl-main.704","article-title":"BLEURT: Learning robust metrics for text\n                        generation","volume-title":"Proceedings of the 58th Annual\n                        Meeting of the Association for Computational Linguistics","author":"Sellam","year":"2020"},{"key":"2026050614005950700_bib60","doi-asserted-by":"publisher","first-page":"1683","DOI":"10.18653\/v1\/P16-1159","article-title":"Minimum risk training for neural machine\n                        translation","volume-title":"Proceedings of the 54th Annual\n                        Meeting of the Association for Computational Linguistics (Volume 1: Long\n                        Papers)","author":"Shen","year":"2016"},{"key":"2026050614005950700_bib61","article-title":"Sequence to sequence learning with neural\n                        networks","volume-title":"Advances in Neural Information\n                        Processing Systems","author":"Sutskever","year":"2014"},{"key":"2026050614005950700_bib62","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018","edition":"2nd ed."},{"issue":"3","key":"2026050614005950700_bib63","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1023\/A:1022672621406","article-title":"Simple statistical gradient-following\n                        algorithms for connectionist reinforcement learning","volume":"8","author":"Williams","year":"1992","journal-title":"Machine Learning"},{"key":"2026050614005950700_bib64","doi-asserted-by":"publisher","first-page":"1296","DOI":"10.18653\/v1\/D16-1137","article-title":"Sequence-to-sequence learning as beam-search\n                        optimization","volume-title":"Proceedings of the 2016 Conference\n                        on Empirical Methods in Natural Language Processing","author":"Wiseman","year":"2016"},{"key":"2026050614005950700_bib65","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language\n                        processing","volume-title":"Proceedings of the 2020 Conference on\n                        Empirical Methods in Natural Language Processing: System\n                        Demonstrations","author":"Wolf","year":"2020"},{"key":"2026050614005950700_bib66","doi-asserted-by":"publisher","first-page":"3612","DOI":"10.18653\/v1\/D18-1397","article-title":"A study of reinforcement learning for neural machine\n                        translation","volume-title":"Proceedings of the 2018 Conference\n                        on Empirical Methods in Natural Language Processing","author":"Lijun","year":"2018"},{"key":"2026050614005950700_bib67","doi-asserted-by":"publisher","first-page":"3098","DOI":"10.24963\/ijcai.2017\/432","article-title":"Sequence prediction with unlabeled data by reward function\n                        learning","volume-title":"Proceedings of the Twenty-Sixth\n                        International Joint Conference on Artificial Intelligence,\n                    IJCAI-17","author":"Lijun","year":"2017"},{"key":"2026050614005950700_bib68","article-title":"Google\u2019s neural machine translation system: Bridging\n                        the gap between human and machine translation","author":"Yonghui","year":"2016","journal-title":"arXiv\n                        e-prints"},{"key":"2026050614005950700_bib69","doi-asserted-by":"publisher","first-page":"59008","DOI":"10.52202\/075280-2574","article-title":"Fine-grained human feedback gives better\n                        rewards for language model training","volume-title":"Advances in\n                        Neural Information Processing Systems","author":"Zeqiu","year":"2023"},{"key":"2026050614005950700_bib70","doi-asserted-by":"publisher","first-page":"8178","DOI":"10.18653\/v1\/2024.findings-emnlp.478","article-title":"Inverse-Q*: Token level reinforcement\n                        learning for aligning large language models without preference\n                        data","volume-title":"Findings of the Association for\n                        Computational Linguistics: EMNLP 2024","author":"Xia","year":"2024"},{"key":"2026050614005950700_bib71","article-title":"A paradigm shift in machine translation:\n                        Boosting translation performance of large language models","volume-title":"ICLR","author":"Haoran","year":"2024"},{"key":"2026050614005950700_bib72","article-title":"Contrastive preference optimization: Pushing the boundaries\n                        of LLM performance in machine translation","volume-title":"ICML","author":"Haoran","year":"2024"},{"key":"2026050614005950700_bib73","doi-asserted-by":"publisher","first-page":"14969","DOI":"10.1080\/1358314X.2024.2437866","article-title":"TLCR: Token-level continuous reward for fine-grained\n                        reinforcement learning from human feedback","volume-title":"Findings of the Association for Computational Linguistics: ACL\n                        2024","author":"Yoon","year":"2024"},{"key":"2026050614005950700_bib74","doi-asserted-by":"publisher","first-page":"3385","DOI":"10.18653\/v1\/2024.naacl-long.186","article-title":"A preference-driven paradigm for enhanced\n                        translation with large language models","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of\n                        the Association for Computational Linguistics: Human Language Technologies\n                        (Volume 1: Long Papers)","author":"Zhu","year":"2024"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.646\/2598669\/tacl.a.646.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.646\/2598669\/tacl.a.646.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T18:01:11Z","timestamp":1778090471000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.646\/136550\/Fine-Grained-Reward-Optimization-for-Machine"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":74,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.646","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}