{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T21:17:48Z","timestamp":1791235068989,"version":"4.1.0"},"reference-count":171,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T00:00:00Z","timestamp":1784160000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Entropy regularization is a recurring mechanism in reinforcement learning (RL), but its meaning changes across algorithmic settings. In classical online RL, entropy encourages exploration and smooths policy improvement; in inverse RL and imitation learning, maximum-entropy resolves ambiguity among expert-consistent behaviors; in offline RL, entropy must be balanced against data support; in generative policies, entropy becomes a tractability problem; and in reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs), token entropy is tied to reasoning diversity, calibration, and collapse. This review organizes these developments into a unified taxonomy. We first summarize the mathematical foundations of maximum-entropy RL, soft Bellman equations, policy-gradient entropy dynamics, and Kullback\u2013Leibler (KL)-constrained mirror descent. We then review entropy in imitation learning, offline RL, intrinsic motivation, diffusion and flow-based policy classes, and RLVR. Particular attention is given to recent work on entropy collapse in reasoning LLMs, entropy-based advantage shaping, covariance-based control, positive-advantage reweighting, and ordinary differential equation (ODE)-based flow-matching policies with tractable entropy. The review emphasizes that entropy is not universally beneficial: useful exploration, support preservation, multimodality, calibration, and reasoning diversity require different entropy objects and different control mechanisms.<\/jats:p>","DOI":"10.3390\/e28070811","type":"journal-article","created":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T03:12:38Z","timestamp":1784257958000},"page":"811","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Entropy Regularization in Deep Reinforcement Learning: A Structured Review Across Classical Control, Generative Policies, and Reasoning Language Models"],"prefix":"10.3390","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1981-7494","authenticated-orcid":false,"given":"Giorgio","family":"Taricco","sequence":"first","affiliation":[{"name":"Department of Electronics and Telecommunications, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,16]]},"reference":[{"key":"ref_1","first-page":"1433","article-title":"Maximum entropy inverse reinforcement learning","volume":"Volume 8","author":"Ziebart","year":"2008","journal-title":"Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence"},{"key":"ref_2","unstructured":"Haarnoja, T., Tang, H., Zhou, A., Abbas, M., Abbeel, P., and Levine, S. (2017, January 6\u201311). Reinforcement learning with soft bounds: A maximum entropy approach. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia."},{"key":"ref_3","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Dietterich, T., Becker, S., and Ghahramani, Z. (2001). A Natural Policy Gradient. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), MIT Press.","DOI":"10.7551\/mitpress\/1120.001.0001"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1180","DOI":"10.1016\/j.neucom.2007.11.026","article-title":"Natural Actor-Critic","volume":"71","author":"Peters","year":"2008","journal-title":"Neurocomputing"},{"key":"ref_6","unstructured":"Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015, January 7\u20139). Trust Region Policy Optimization. Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France."},{"key":"ref_7","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_8","unstructured":"Abdolmaleki, A., Springenberg, J.T.w., Degrave, J., Bohez, S., Tassa, Y., Belov, D., Heess, N., and Riedmiller, M. (2018). Relative Entropy Regularized Policy Iteration. arXiv."},{"key":"ref_9","unstructured":"Geist, M., Scherrer, B., and Pietquin, O. (2019, January 9\u201315). A Theory of Regularized Markov Decision Processes. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1007\/s10107-022-01816-5","article-title":"Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classes","volume":"198","author":"Lan","year":"2023","journal-title":"Math. Program."},{"key":"ref_11","unstructured":"Ziebart, B.D. (2010). Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. [Ph.D. Thesis, Carnegie Mellon University]."},{"key":"ref_12","unstructured":"Ho, J., and Ermon, S. (2016). Generative adversarial imitation learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), NIPS\u201916, Curran Associates, Inc."},{"key":"ref_13","unstructured":"Finn, C., Levine, S., and Abbeel, P. (2016, January 20\u201322). Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization. Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA."},{"key":"ref_14","unstructured":"Fu, J., Luo, K., and Levine, S. (May, January 30). Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada."},{"key":"ref_15","unstructured":"Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv."},{"key":"ref_16","unstructured":"Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative Q-learning for offline reinforcement learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), NIPS \u201920, Curran Associates Inc."},{"key":"ref_17","unstructured":"Kostrikov, I., Nair, A., and Levine, S. (2022, January 25\u201329). Offline Reinforcement Learning with Implicit Q-Learning. Proceedings of the International Conference on Learning Representations (ICLR), Virtual."},{"key":"ref_18","unstructured":"Wu, Y., Tucker, G., and Nachum, O. (2020, January 26\u201330). Behavior Regularized Offline Reinforcement Learning. Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia."},{"key":"ref_19","unstructured":"Fujimoto, S., Meger, D., and Precup, D. (2018, January 10\u201315). Off-Policy Deep Reinforcement Learning without Exploration. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_20","unstructured":"Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015, January 6\u201311). Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the International Conference on Machine Learning (ICML), Lille, France."},{"key":"ref_21","unstructured":"Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), NIPS \u201920, Curran Associates Inc."},{"key":"ref_22","unstructured":"Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., and Poole, B. (2021, January 4). Score-Based Generative Modeling through Stochastic Differential Equations. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. Proceedings of the Robotics: Science and Systems (RSS), RSS Foundation.","DOI":"10.15607\/RSS.2023.XIX.026"},{"key":"ref_24","unstructured":"Wang, Z., Hunt, J.J., and Zhou, M. (2023, January 1\u20135). Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ding, S., Hu, K., Zhang, Z., Ren, K., Zhang, W., Yu, J., Wang, J., and Shi, Y. (2024). Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.","DOI":"10.52202\/079017-1708"},{"key":"ref_26","unstructured":"Chen, R.T.Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2018). Neural Ordinary Differential Equations. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_27","unstructured":"Grathwohl, W., Chen, R.T.Q., Bettencourt, J., Sutskever, I., and Duvenaud, D. (2019, January 6\u20139). FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Liu, X., Gong, C., and Liu, Q. (2023, January 1\u20135). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.","DOI":"10.1109\/ICVISP64524.2024.10959352"},{"key":"ref_29","unstructured":"Lipman, Y., Chen, R.T.Q., Ben-Hamu, H., Nickel, M., and Le, M. (2023, January 1\u20135). Flow Matching for Generative Modeling. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda."},{"key":"ref_30","unstructured":"Gao, T., Orfanoudakis, S., Lin, N., Daamen, W., Hoogendoorn, S., and Isufi, E. (2026). Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints. arXiv."},{"key":"ref_31","unstructured":"Cui, G., Zhang, Y., Chen, J., Yuan, L., Wang, Z., Zuo, Y., Li, H., Fan, Y., Chen, H., and Chen, W. (2025). The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv."},{"key":"ref_32","unstructured":"Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, W.X., Zhang, Z., and Wei, F. (2026, January 20\u201327). Reasoning with Exploration: An Entropy Perspective. Proceedings of the Fortieth AAAI Conference on Artificial Intelligence (AAAI-26), Singapore."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Jin, R., Gao, P., Ren, Y., Han, Z., Zhang, T., Huang, W., Liu, W., Luan, J., and Xiong, D. (2026). Revisiting Entropy in Reinforcement Learning for Large Reasoning Models. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2026, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2026.findings-acl.1266"},{"key":"ref_34","unstructured":"Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y.K., and Wu, Y. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1038\/s41586-025-09422-z","article-title":"DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement Learning","volume":"645","author":"Guo","year":"2025","journal-title":"Nature"},{"key":"ref_36","unstructured":"Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Fan, T., Liu, G., Liu, L., and Liu, X. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_37","unstructured":"Cui, G., Yuan, L., Wang, Z., Wang, H., Li, W., He, B., Fan, Y., Yu, T., Xu, Q., and Chen, W. (2025). PRIME: Process Reinforcement through Implicit Rewards. arXiv."},{"key":"ref_38","unstructured":"Bellman, R. (1957). Dynamic Programming, Princeton University Press."},{"key":"ref_39","first-page":"679","article-title":"A Markovian Decision Process","volume":"6","author":"Bellman","year":"1957","journal-title":"J. Math. Mech."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Puterman, M.L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons.","DOI":"10.1002\/9780470316887"},{"key":"ref_41","unstructured":"Bertsekas, D.P. (2012). Dynamic Programming and Optimal Control, Athena Scientific. [4th ed.]."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1103\/PhysRev.106.620","article-title":"Information Theory and Statistical Mechanics","volume":"106","author":"Jaynes","year":"1957","journal-title":"Phys. Rev."},{"key":"ref_43","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, John Wiley & Sons. [2nd ed.]."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"200201","DOI":"10.1103\/PhysRevLett.95.200201","article-title":"Linear Theory for Control of Nonlinear Stochastic Systems","volume":"95","author":"Kappen","year":"2005","journal-title":"Phys. Rev. Lett."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Todorov, E. (2008). General duality between optimal control and estimation. Proceedings of the 47th IEEE Conference on Decision and Control, IEEE.","DOI":"10.1109\/CDC.2008.4739438"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"11478","DOI":"10.1073\/pnas.0710743106","article-title":"Efficient Computation of Optimal Actions","volume":"106","author":"Todorov","year":"2009","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_47","unstructured":"Theodorou, E.A., Buchli, J., and Schaal, S. (2010, January 13\u201315). A Generalized Path Integral Control Approach to Reinforcement Learning. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Chia Laguna Resort, Sardinia, Italy."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A Mathematical Theory of Communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_49","unstructured":"Khinchin, A.I. (1957). Mathematical Foundations of Information Theory, Dover Publications."},{"key":"ref_50","first-page":"547","article-title":"On measures of entropy and information","volume":"Volume 1","year":"1961","journal-title":"Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1007\/BF01016429","article-title":"Possible generalization of Boltzmann-Gibbs statistics","volume":"52","author":"Tsallis","year":"1988","journal-title":"J. Stat. Phys."},{"key":"ref_52","unstructured":"Levine, S. (2018). Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Rawlik, K., Toussaint, M., and Vijayakumar, S. (2013, January 24\u201328). On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference. Proceedings of Robotics: Science and Systems (RSS), Berlin, Germany.","DOI":"10.15607\/RSS.2012.VIII.045"},{"key":"ref_54","unstructured":"Fox, R., Pakman, A., and Tishby, N. (2016). Taming the Noise in Reinforcement Learning via Soft Updates. Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence (UAI), AUAI Press."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Peters, J., M\u00fclling, K., and Altun, Y. (2010). Relative Entropy Policy Search. Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI), AAAI Press.","DOI":"10.1609\/aaai.v24i1.7727"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Azizzadenesheli, K., Brunskill, E., and Anandkumar, A. (2018, January 11\u201316). Efficient Exploration Through Bayesian Deep Q-Networks. Proceedings of the Information Theory and Applications Workshop (ITA), San Diego, CA, USA.","DOI":"10.1109\/ITA.2018.8503252"},{"key":"ref_57","unstructured":"Neu, G., Jonsson, A., and Gomez, V. (2017). A Unified View of Entropy-Regularized Markov Decision Processes. arXiv."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Shani, L., Efroni, Y., and Mannor, S. (2020, January 9\u201312). Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs. Proceedings of the Conference on Learning Theory (COLT), Virtual.","DOI":"10.1609\/aaai.v34i04.6021"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1080\/09540099108946587","article-title":"Function optimization using connectionist reinforcement learning algorithms","volume":"3","author":"Williams","year":"1991","journal-title":"Connect. Sci."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1023\/A:1022672621406","article-title":"Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning","volume":"8","author":"Williams","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_61","unstructured":"Sutton, R.S., McAllester, D., Singh, S., and Mansour, Y. (2000). Policy Gradient Methods for Reinforcement Learning with Function Approximation. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), MIT Press."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"2471","DOI":"10.1016\/j.automatica.2009.07.008","article-title":"Natural Actor-Critic Algorithms","volume":"45","author":"Bhatnagar","year":"2009","journal-title":"Automatica"},{"key":"ref_63","unstructured":"Degris, T., White, M., and Sutton, R.S. (July, January 26). Off-Policy Actor-Critic. Proceedings of the International Conference on Machine Learning (ICML), Edinburgh, UK."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level Control Through Deep Reinforcement Learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_65","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 20\u201322). Asynchronous Methods for Deep Reinforcement Learning. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_66","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016, January 2\u20134). Continuous Control with Deep Reinforcement Learning. Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, USA."},{"key":"ref_67","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21\u201326). Deterministic Policy Gradient Algorithms. Proceedings of the International Conference on Machine Learning (ICML), Beijing, China."},{"key":"ref_68","unstructured":"Fujimoto, S., van Hoof, H., and Meger, D. (2018, January 10\u201315). Addressing Function Approximation Error in Actor-Critic Methods. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2018, January 2\u20137). Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11694"},{"key":"ref_70","unstructured":"Islam, R., Henderson, P., Gomrokchi, M., and Precup, D. (2017). Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv."},{"key":"ref_71","unstructured":"Andrychowicz, M., Raichuk, A., Sta\u0144czyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., and Michalski, M. (2021, January 3\u20137). What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1162\/089976698300017746","article-title":"Natural Gradient Works Efficiently in Learning","volume":"10","author":"Amari","year":"1998","journal-title":"Neural Comput."},{"key":"ref_73","unstructured":"Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016, January 2\u20134). High-Dimensional Continuous Control Using Generalized Advantage Estimation. Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, USA."},{"key":"ref_74","unstructured":"Abdolmaleki, A., Springenberg, J.T., Degrave, J., Belov, D., Heess, N., and Riedmiller, M. (2018). Maximum Entropy Regularized Policy Iteration. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_75","unstructured":"Peng, X.B., Kumar, A., Zhang, G., and Levine, S. (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv."},{"key":"ref_76","unstructured":"Nair, A., Gupta, A., and Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv."},{"key":"ref_77","unstructured":"Ross, S., Gordon, G., and Bagnell, D. (2011, January 11\u201313). A reduction of imitation learning and structured prediction to no-regret online learning. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Fort Lauderdale, FL, USA."},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1016\/j.robot.2008.10.024","article-title":"A survey of robot learning from demonstration","volume":"57","author":"Argall","year":"2009","journal-title":"Robot. Auton. Syst."},{"key":"ref_79","unstructured":"Fu, J., Kumar, A., Nachum, O., and Levine, S. (2020). D4RL: Datasets for deep data-driven reinforcement learning. arXiv."},{"key":"ref_80","first-page":"7248","article-title":"RL Unplugged: A suite of benchmarks for offline reinforcement learning","volume":"Volume 33","author":"Gulcehre","year":"2020","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_81","unstructured":"Kumar, A., Fu, J., Soh, M., and Levine, S. (2019). Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_82","unstructured":"Yu, T., Thomas, G., Levine, S., and Finn, C. (2020). MOPO: Model-based Offline Policy Optimization. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_83","unstructured":"Yu, T., Kumar, A., Rafailov, R., and Levine, S. (2021). COMBO: Conservative Offline Model-Based Policy Optimization. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_84","unstructured":"Janner, M., Li, Q., and Levine, S. (2021). Offline Reinforcement Learning as One Big Sequence Modeling Problem. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_85","unstructured":"Janner, M., Tenenbaum, J., and Levine, S. (2022, January 17\u201323). Planning with Diffusion for Flexible Behavior Synthesis. Proceedings of the International Conference on Machine Learning (ICML), Baltimore, MD, USA."},{"key":"ref_86","unstructured":"Chen, L., Rajeswaran, A., Lee, K., Abbeel, P., and Mordatch, I. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_87","unstructured":"Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J.G., and Levine, S. (2023). IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_88","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/0196-8858(85)90002-8","article-title":"Asymptotically Efficient Adaptive Allocation Rules","volume":"6","author":"Lai","year":"1985","journal-title":"Adv. Appl. Math."},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1023\/A:1013689704352","article-title":"Finite-time Analysis of the Multiarmed Bandit Problem","volume":"47","author":"Auer","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/2200000024","article-title":"Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems","volume":"5","author":"Bubeck","year":"2012","journal-title":"Found. Trends  Mach. Learn."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"1309","DOI":"10.1016\/j.jcss.2007.08.009","article-title":"An Analysis of Model-Based Interval Estimation for Markov Decision Processes","volume":"74","author":"Strehl","year":"2008","journal-title":"J. Comput. Syst. Sci."},{"key":"ref_92","unstructured":"Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. (2016). Deep Exploration via Bootstrapped DQN. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_93","unstructured":"Azar, M.G., Osband, I., and Munos, R. (2017, January 6\u201311). Minimax Regret Bounds for Reinforcement Learning. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia."},{"key":"ref_94","unstructured":"Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M.I. (2018). Is Q-learning Provably Efficient?. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_95","first-page":"1","article-title":"On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift","volume":"22","author":"Agarwal","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref_96","doi-asserted-by":"crossref","unstructured":"Pathak, D., Agrawal, P., and Darrell, T. (2017, January 6\u201311). Curiosity-driven Exploration by Self-supervised Prediction. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia.","DOI":"10.1109\/CVPRW.2017.70"},{"key":"ref_97","unstructured":"Burda, Y., Edwards, H., Storkey, A., and Klimov, O. (2019, January 6\u20139). Exploration by Random Network Distillation. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA."},{"key":"ref_98","first-page":"579","article-title":"Go-Explore: A New Approach for Hard-Exploration Problems","volume":"590","author":"Ecoffet","year":"2021","journal-title":"Nature"},{"key":"ref_99","unstructured":"Badia, A.P., Piot, B., Kapturowski, S., and Silver, D. (2020, January 13\u201318). Agent57: Outperforming the Atari Human Benchmark. Proceedings of the International Conference on Machine Learning (ICML), Virtual Event."},{"key":"ref_100","unstructured":"Raileanu, R., and Rocktaschel, T. (2020, January 26\u201330). RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event."},{"key":"ref_101","unstructured":"Gregor, K., Rezende, D.J., and Wierstra, D. (2017, January 24\u201326). Variational Intrinsic Control. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France."},{"key":"ref_102","unstructured":"Mohamed, S., and Rezende, D.J. (2015). Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_103","unstructured":"Eysenbach, B., Gupta, A., and Levine, S. (2019, January 6\u20139). Diversity is All You Need: Learning Skills Without a Reward Function. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA."},{"key":"ref_104","unstructured":"Hazan, E., Kakade, S.M., and Van Soest, A. (2019, January 9\u201315). Provably Efficient Maximum Entropy Exploration. Proceedings of the International Conference on Machine Learning (ICML), Long Beach, CA, USA."},{"key":"ref_105","unstructured":"Lee, L., Eysenbach, B., Levine, S., and Salakhutdinov, R. (2020, January 26\u201328). Efficient Exploration via State Marginal Matching. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Online."},{"key":"ref_106","unstructured":"Eysenbach, B., and Levine, S. (2021). Maximum Entropy RL Provably Solves Some Robust RL Problems. arXiv."},{"key":"ref_107","first-page":"21696","article-title":"Variational diffusion models","volume":"Volume 34","author":"Kingma","year":"2021","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_108","unstructured":"Albergo, M.S., Boffi, N.M., and Vanden-Eijnden, E. (2023). Building normalizing flows with stochastic interpolants. arXiv."},{"key":"ref_109","unstructured":"Tong, A., Fatras, K., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Wolf, G., and Bengio, Y. (2024). Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport. Trans. Mach. Learn. Res., Available online: https:\/\/openreview.net\/forum?id=CD9Snc73AW."},{"key":"ref_110","unstructured":"Nichol, A.Q., and Dhariwal, P. (2021, January 18\u201324). Improved denoising diffusion probabilistic models. Proceedings of the International Conference on Machine Learning (ICML), Virtual Event."},{"key":"ref_111","first-page":"8780","article-title":"Diffusion models beat GANs on image synthesis","volume":"Volume 34","author":"Dhariwal","year":"2021","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"26565","DOI":"10.52202\/068431-1926","article-title":"Elucidating the design space of diffusion-based generative models","volume":"35","author":"Karras","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst. (NeurIPS)"},{"key":"ref_113","doi-asserted-by":"crossref","unstructured":"Rombach, R., Blattmann, A., Lorenz, D., and Ommer, B. (2022, January 18\u201324). High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"ref_114","unstructured":"Song, Y., and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_115","unstructured":"Salimans, T., and Ho, J. (2022, January 25\u201329). Progressive distillation for fast sampling of diffusion models. Proceedings of the International Conference on Learning Representations (ICLR), Virtual Event."},{"key":"ref_116","doi-asserted-by":"crossref","unstructured":"Kang, B., Ma, X., and Yan, S. (2023). Efficient Diffusion Policies for Offline Reinforcement Learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.","DOI":"10.52202\/075280-2937"},{"key":"ref_117","unstructured":"Celik, O., Zhou, Z., Zhou, M., and Zhao, D. (2025, January 24\u201328). DIME: Diffusion-based Maximum Entropy Reinforcement Learning. Proceedings of the International Conference on Learning Representations (ICLR), Singapore."},{"key":"ref_118","doi-asserted-by":"crossref","first-page":"98871","DOI":"10.52202\/079017-3138","article-title":"Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning","volume":"37","author":"Zhang","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst. (NeurIPS)"},{"key":"ref_119","unstructured":"Christiano, P.F., Leike, J., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc."},{"key":"ref_120","unstructured":"Ziegler, D.M., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv."},{"key":"ref_121","first-page":"3008","article-title":"Learning to summarize with human feedback","volume":"Volume 33","author":"Stiennon","year":"2020","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_122","doi-asserted-by":"crossref","first-page":"27730","DOI":"10.52202\/068431-2011","article-title":"Training Language Models to Follow Instructions with Human Feedback","volume":"Volume 35","author":"Ouyang","year":"2022","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_123","doi-asserted-by":"crossref","unstructured":"Rafailov, R., Sharma, A., Manning, C.D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.","DOI":"10.52202\/075280-2338"},{"key":"ref_124","unstructured":"Ramamurthy, R., Choi, Y., and Hajishirzi, H. (2023). Reinforcement learning from human feedback for summarization: A comparative study. arXiv."},{"key":"ref_125","first-page":"1877","article-title":"Language Models are Few-Shot Learners","volume":"Volume 33","author":"Brown","year":"2020","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_126","first-page":"1","article-title":"PaLM: Scaling language modeling with pathways","volume":"24","author":"Chowdhery","year":"2023","journal-title":"J. Mach. Learn. Res."},{"key":"ref_127","unstructured":"Workshop, B., Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ili\u0107, S., Hesslow, D., Castagn\u00e9, R., Luccioni, A.S., and Yvon, F. (2022). BLOOM: A 176B-parameter open-access multilingual language model. arXiv."},{"key":"ref_128","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and efficient foundation language models. arXiv."},{"key":"ref_129","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv."},{"key":"ref_130","unstructured":"Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv."},{"key":"ref_131","unstructured":"Gemma Team (2024). Gemma: Open models based on Gemini research and technology. arXiv."},{"key":"ref_132","doi-asserted-by":"crossref","first-page":"24824","DOI":"10.52202\/068431-1800","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"Volume 35","author":"Wei","year":"2022","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_133","doi-asserted-by":"crossref","first-page":"22199","DOI":"10.52202\/068431-1613","article-title":"Large language models are zero-shot reasoners","volume":"Volume 35","author":"Kojima","year":"2022","journal-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"ref_134","unstructured":"Wang, X., Wei, J., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2023, January 1\u20135). Self-consistency improves chain of thought reasoning in language models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda."},{"key":"ref_135","doi-asserted-by":"crossref","unstructured":"Yao, S., Yu, D., Griffiths, T.L., Cao, Y., and Narasimhan, K. (2023). Tree of thoughts: Deliberate problem solving with large language models. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.","DOI":"10.52202\/075280-0517"},{"key":"ref_136","unstructured":"Yao, S., Zhao, J., Narasimhan, K., and Cao, Y. (2023, January 1\u20135). ReAct: Synergizing reasoning and acting in language models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda."},{"key":"ref_137","doi-asserted-by":"crossref","unstructured":"Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.","DOI":"10.52202\/075280-0377"},{"key":"ref_138","doi-asserted-by":"crossref","unstructured":"Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., and Yang, Y. (2023). Self-Refine: Iterative Refinement with Self-Feedback. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.","DOI":"10.52202\/075280-2019"},{"key":"ref_139","doi-asserted-by":"crossref","first-page":"15476","DOI":"10.52202\/068431-1126","article-title":"STaR: Bootstrapping reasoning with reasoning","volume":"35","author":"Zelikman","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst. (NeurIPS)"},{"key":"ref_140","unstructured":"Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., and Nakano, R. (2021). Training verifiers to solve math word problems. arXiv."},{"key":"ref_141","unstructured":"Hendrycks, D., Burns, C., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. Proceedings of the NeurIPS Datasets and Benchmarks Track, Curran Associates, Inc."},{"key":"ref_142","unstructured":"Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2024, January 7\u201311). Let\u2019s Verify Step by Step. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria."},{"key":"ref_143","unstructured":"Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J.T., Li, Z., Weller, A., and Liu, W. (2024, January 7\u201311). MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria."},{"key":"ref_144","doi-asserted-by":"crossref","unstructured":"Todorov, E., Erez, T., and Tassa, Y. (2012). MuJoCo: A physics engine for model-based control. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal, 7\u201312 October 2012, IEEE.","DOI":"10.1109\/IROS.2012.6386109"},{"key":"ref_145","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv."},{"key":"ref_146","unstructured":"Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016, January 20\u201322). Benchmarking deep reinforcement learning for continuous control. Proceedings of the International Conference on Machine Learning (ICML), New York, NY, USA."},{"key":"ref_147","unstructured":"Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., and Dunning, I. (2018, January 10\u201315). IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. Proceedings of the International Conference on Machine Learning (ICML), Stockholm, Sweden."},{"key":"ref_148","doi-asserted-by":"crossref","unstructured":"Hessel, M., Modayil, J., and Silver, D. (2018, January 2\u20137). Rainbow: Combining Improvements in Deep Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11796"},{"key":"ref_149","unstructured":"Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, N. (2016, January 20\u201322). Dueling Network Architectures for Deep Reinforcement Learning. Proceedings of the International Conference on Machine Learning (ICML), New York, NY, USA."},{"key":"ref_150","doi-asserted-by":"crossref","unstructured":"van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep Reinforcement Learning with Double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_151","unstructured":"Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016, January 2\u20134). Prioritized Experience Replay. Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, USA."},{"key":"ref_152","doi-asserted-by":"crossref","unstructured":"Dabney, W., Rowland, M., Mnih, V., and Blundell, C. (2018, January 2\u20137). Distributional Reinforcement Learning with Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11791"},{"key":"ref_153","unstructured":"Bellemare, M.G., Dabney, W., and Munos, R. (2017, January 6\u201311). A Distributional Perspective on Reinforcement Learning. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia."},{"key":"ref_154","unstructured":"OpenAI (2023). GPT-4 technical report. arXiv."},{"key":"ref_155","unstructured":"Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., and Chen, Z. (2023). PaLM 2 technical report. arXiv."},{"key":"ref_156","unstructured":"Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., and Lundberg, S. (2023). Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv."},{"key":"ref_157","unstructured":"Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., and Henighan, T. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv."},{"key":"ref_158","unstructured":"Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., and McKinnon, C. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv."},{"key":"ref_159","unstructured":"Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., and DasSarma, N. (2021). A general language assistant as a laboratory for alignment. arXiv."},{"key":"ref_160","unstructured":"Leike, J., Krueger, D., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: A research direction. arXiv."},{"key":"ref_161","unstructured":"Irving, G., Christiano, P., and Amodei, D. (2018). AI safety via debate. arXiv."},{"key":"ref_162","unstructured":"Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., and Irving, G. (2022). Teaching language models to support answers with verified quotes. arXiv."},{"key":"ref_163","unstructured":"Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., and Saunders, W. (2021). WebGPT: Browser-assisted question-answering with human feedback. arXiv."},{"key":"ref_164","unstructured":"Glaese, A., McAleese, N., Tr\u0119bacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., and Thacker, P. (2022). Improving alignment of dialogue agents via targeted human judgements. arXiv."},{"key":"ref_165","unstructured":"Kaplan, J., McCandlish, S., Heerspink, T., Brown, T.B., Radford, A., Amodei, D., Amodei, D., and Clark, J. (2020). Scaling Laws for Neural Language Models. arXiv."},{"key":"ref_166","doi-asserted-by":"crossref","unstructured":"Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., and Clark, A. (2022). An Empirical Analysis of Compute-Optimal Large Language Model Training. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.","DOI":"10.52202\/068431-2176"},{"key":"ref_167","unstructured":"Gao, L., Schulman, J., and Hilton, J. (2023, January 23\u201329). Scaling Laws for Reward Model Overoptimization. Proceedings of the International Conference on Machine Learning (ICML), Honolulu, HI, USA."},{"key":"ref_168","unstructured":"Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., and Abbeel, P. (2018). Soft Actor-Critic Algorithms and Applications. arXiv."},{"key":"ref_169","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1162\/neco.1997.9.2.271","article-title":"Using Expectation-Maximization for Reinforcement Learning","volume":"9","author":"Dayan","year":"1997","journal-title":"Neural Comput."},{"key":"ref_170","first-page":"146","article-title":"I-Divergence Geometry of Probability Distributions and Minimization Problems","volume":"3","year":"1975","journal-title":"Ann. Probab."},{"key":"ref_171","unstructured":"Tishby, N., Pereira, F.C., and Bialek, W. (1999, January 22\u201324). The Information Bottleneck Method. Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/28\/7\/811\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T04:15:09Z","timestamp":1784348109000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/28\/7\/811"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,16]]},"references-count":171,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["e28070811"],"URL":"https:\/\/doi.org\/10.3390\/e28070811","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,16]]}}}