{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T16:49:26Z","timestamp":1778345366126,"version":"3.51.4"},"reference-count":50,"publisher":"SAGE Publications","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IFS"],"published-print":{"date-parts":[[2022,4,28]]},"abstract":"<jats:p>Model-free reinforcement learning methods have successfully been applied to practical applications such as decision-making problems in Atari games. However, these methods have inherent shortcomings, such as a high variance and low sample efficiency. To improve the policy performance and sample efficiency of model-free reinforcement learning, we propose proximal policy optimization with model-based methods (PPOMM), a fusion method of both model-based and model-free reinforcement learning. PPOMM not only considers the information of past experience but also the prediction information of the future state. PPOMM adds the information of the next state to the objective function of the proximal policy optimization (PPO) algorithm through a model-based method. This method uses two components to optimize the policy: the error of PPO and the error of model-based reinforcement learning. We use the latter to optimize a latent transition model and predict the information of the next state. For most games, this method outperforms the state-of-the-art PPO algorithm when we evaluate across 49 Atari games in the Arcade Learning Environment (ALE). The experimental results show that PPOMM performs better or the same as the original algorithm in 33 games.<\/jats:p>","DOI":"10.3233\/jifs-211935","type":"journal-article","created":{"date-parts":[[2022,1,11]],"date-time":"2022-01-11T13:03:08Z","timestamp":1641906188000},"page":"5399-5410","source":"Crossref","is-referenced-by-count":3,"title":["Proximal policy optimization with model-based methods"],"prefix":"10.1177","volume":"42","author":[{"given":"Shuailong","family":"Li","sequence":"first","affiliation":[{"name":"State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China"},{"name":"Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, China"},{"name":"University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China"},{"name":"Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Huiwen","family":"Zhang","sequence":"additional","affiliation":[{"name":"CVTE Research, Guangzhou, P.R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xin","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China"},{"name":"Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuquan","family":"Leng","sequence":"additional","affiliation":[{"name":"Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, China"},{"name":"Guangdong Provincial Key Laboratory of Human-Augmentation and Rehabilitation Robotics in Universities, Southern University of Science and Technology, Shenzhen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/JIFS-211935_ref1","unstructured":"Sutton R.S. and Barto A.G. , Reinforcement learning: An introduction[M], MIT press, (2018)."},{"key":"10.3233\/JIFS-211935_ref2","doi-asserted-by":"crossref","unstructured":"Henderson P. , Islam R. , Bachman P. , et al., Deep reinforcement learning that matters[C], Proceedings of the AAAI conference on artificial intelligence 32(1) (2018).","DOI":"10.1609\/aaai.v32i1.11694"},{"issue":"5786","key":"10.3233\/JIFS-211935_ref3","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks[J]","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"issue":"7","key":"10.3233\/JIFS-211935_ref4","doi-asserted-by":"crossref","first-page":"1527","DOI":"10.1162\/neco.2006.18.7.1527","article-title":"A fast learning algorithm for deep belief nets[J]","volume":"18","author":"Hinton","year":"2006","journal-title":"Neural Computation"},{"issue":"6","key":"10.3233\/JIFS-211935_ref5","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1145\/3065386","article-title":"ImageNet classification with deep convolutional neural networks[J]","volume":"60","author":"Krizhevsky","year":"2017","journal-title":"Communications of the ACM"},{"key":"10.3233\/JIFS-211935_ref7","first-page":"770","article-title":"Deep residual learning for image recognition[C]","volume":"2016","author":"He","journal-title":"Proceedings of the IEEE conference on computer vision and pattern recognition"},{"key":"10.3233\/JIFS-211935_ref8","first-page":"6645","article-title":"Speech recognition with deep recurrent neural networks[C]","volume":"2013","author":"Graves","journal-title":"2013 IEEE international conference on acoustics, speech and signal processing. Ieee"},{"key":"10.3233\/JIFS-211935_ref9","first-page":"1764","article-title":"Towards end-to-end speech recognition with recurrent neural networks[C]","volume":"2014","author":"Graves","journal-title":"International conference on machine learning. PMLR"},{"issue":"7540","key":"10.3233\/JIFS-211935_ref12","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning[J]","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"10.3233\/JIFS-211935_ref13","first-page":"1995","article-title":"Dueling network architectures for deep reinforcement learning[C]","volume":"2016","author":"Wang","journal-title":"International conference on machine learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref14","first-page":"1928","article-title":"Asynchronous methods for deep reinforcement learning[C]","volume":"2016","author":"Mnih","journal-title":"International conference on machine learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref15","first-page":"2829","article-title":"Continuous deep q-learning with model-based acceleration[C]","volume":"2016","author":"Gu","journal-title":"International conference on machine learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref16","first-page":"1889","article-title":"Trust region policy optimization[C]","volume":"2015","author":"Schulman","journal-title":"International conference on machine learning. PMLR"},{"issue":"7587","key":"10.3233\/JIFS-211935_ref17","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search[J]","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"10.3233\/JIFS-211935_ref18","first-page":"1861","article-title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]","volume":"2018","author":"Haarnoja","journal-title":"International conference on machine learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref19","first-page":"12653","article-title":"Model-free reinforcement learning: from clipped pseudo-regret to sample complexity[C]","volume":"2021","author":"Zhang","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref22","unstructured":"Mesnard T. , Weber T. , Viola F. , et al., Model-Free Counterfactual Credit Assignment[J], (2020)."},{"key":"10.3233\/JIFS-211935_ref23","first-page":"105","article-title":"Multi-Objective Exploration for Proximal Policy Optimization[C]","volume":"2021","author":"Khoi","journal-title":"2020 Applying New Technology in Green Buildings (ATiGB). IEEE"},{"key":"10.3233\/JIFS-211935_ref25","first-page":"3078","article-title":"Model-based Reinforcement Learning for Continuous Control with Posterior Sampling[C]","volume":"2021","author":"Fan","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref26","first-page":"12009","article-title":"Continuous-time Model-based Reinforcement Learning[C]","volume":"2021","author":"Yildiz","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref27","first-page":"9190","article-title":"Model-based reinforcement learning via latent-space collocation[C]","volume":"2021","author":"Rybkin","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref28","first-page":"9801","article-title":"Pc-mlp: Model-based reinforcement learning with policy cover guided exploration[C]","volume":"2021","author":"Song","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref29","doi-asserted-by":"crossref","unstructured":"Hong J. , Wang H. , Wang Z. , et al., Learning Model-Based Privacy Protection under Budget Constraints[C], The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21) 35(9) (2021).","DOI":"10.1609\/aaai.v35i9.16941"},{"key":"10.3233\/JIFS-211935_ref31","first-page":"7001","article-title":"A sharp analysis of model-based reinforcement learning with self-play[C]","volume":"2021","author":"Liu","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref33","first-page":"3195","article-title":"Incremental Learning of Planning Actions in Model-Based Reinforcement Learning[C]","volume":"2019","author":"Ng","journal-title":"IJCAI."},{"key":"10.3233\/JIFS-211935_ref34","first-page":"9190","article-title":"Model-based reinforcement learning via latent-space collocation[C]","volume":"2021","author":"Rybkin","journal-title":"International Conference on Machine Learning. PMLR"},{"issue":"3","key":"10.3233\/JIFS-211935_ref35","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1207\/s15516709cog1603_1","article-title":"Forward models: Supervised learning with a distal teacher[J]","volume":"16","author":"Jordan","year":"1992","journal-title":"Cognitive Science"},{"issue":"7","key":"10.3233\/JIFS-211935_ref36","doi-asserted-by":"crossref","first-page":"2737","DOI":"10.1109\/TAC.2018.2876389","article-title":"A general safety framework for learning-based control in uncertain robotic systems[J]","volume":"64","author":"Fisac","year":"2018","journal-title":"IEEE Transactions on Automatic Control"},{"key":"10.3233\/JIFS-211935_ref37","first-page":"1","article-title":"Planning for autonomous cars that leverage effects on human actions[C]","volume":"2","author":"Sadigh","year":"2016","journal-title":"Robotics: Science and Systems"},{"key":"10.3233\/JIFS-211935_ref38","first-page":"4736","article-title":"Model-Based Reinforcement Learning via Proximal Policy Optimization[C]","volume":"2019","author":"Sun","journal-title":"2019 Chinese Automation Congress (CAC). IEEE"},{"issue":"01","key":"10.3233\/JIFS-211935_ref39","doi-asserted-by":"crossref","first-page":"4675","DOI":"10.1609\/aaai.v33i01.33014675","article-title":"Policy optimization with model-based explorations[C]","volume":"33","author":"Pan","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"issue":"7698","key":"10.3233\/JIFS-211935_ref40","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1038\/nature25978","article-title":"Planning chemical syntheses with deep neural networks and symbolic AI[J]","volume":"555","author":"Segler","year":"2018","journal-title":"Nature"},{"issue":"1","key":"10.3233\/JIFS-211935_ref42","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1109\/TMECH.2019.2952552","article-title":"Effective capture of nongraspable objects for space robots using geometric cage pairs[J]","volume":"25","author":"Zhang","year":"2019","journal-title":"IEEE\/ASME Transactions on Mechatronics"},{"key":"10.3233\/JIFS-211935_ref43","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/j.actaastro.2018.04.029","article-title":"Effective motion planning strategy for space robot capturing targets under consideration of the berth position[J]","volume":"148","author":"Zhang","year":"2018","journal-title":"Acta Astronautica"},{"key":"10.3233\/JIFS-211935_ref44","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.conb.2019.01.010","article-title":"An integrative computational architecture for object-driven cortex[J]","volume":"55","author":"Yildirim","year":"2019","journal-title":"Current Opinion in Neurobiology"},{"key":"10.3233\/JIFS-211935_ref46","first-page":"1439","article-title":"Entity abstraction in visual model-based reinforcement learning[C]","volume":"2020","author":"Veerapaneni","journal-title":"Conference on Robot Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref49","unstructured":"Todorov E. and Li W. , A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems[C], Proceedings of the American Control Conference, IEEE, 2005 (2005), 300\u2013306."},{"issue":"2","key":"10.3233\/JIFS-211935_ref53","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1016\/S0896-6273(03)00169-7","article-title":"Temporal difference models and reward-related learning in the human brain[J]","volume":"38","author":"O\u2019Doherty","year":"2003","journal-title":"Neuron"},{"issue":"6992","key":"10.3233\/JIFS-211935_ref54","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1038\/nature02581","article-title":"Temporal difference models describe higher-order learning in humans[J]","volume":"429","author":"Seymour","year":"2004","journal-title":"Nature"},{"issue":"3-4","key":"10.3233\/JIFS-211935_ref55","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning[J]","volume":"8","author":"Watkins","year":"1992","journal-title":"Machine Learning"},{"issue":"9","key":"10.3233\/JIFS-211935_ref58","doi-asserted-by":"crossref","first-page":"2184","DOI":"10.1016\/j.engappai.2013.06.016","article-title":"Backward Q-learning: The combination of Sarsa algorithm and Q-learning[J]","volume":"26","author":"Wang","year":"2013","journal-title":"Engineering Applications of Artificial Intelligence"},{"key":"10.3233\/JIFS-211935_ref59","first-page":"1","article-title":"Deep reinforcement learning with experience replay based on SARSA[C], IEEE","volume":"2016","author":"Zhao","journal-title":"2016 IEEE Symposium Series on Computational Intelligence (SSCI)"},{"key":"10.3233\/JIFS-211935_ref64","first-page":"980","article-title":"Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision[C]","volume":"2021","author":"Bj\u00f6rck","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref69","first-page":"6611","article-title":"Guided Exploration with Proximal Policy Optimization using a Single Demonstration[C]","volume":"2021","author":"Libardi","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref70","first-page":"4018","article-title":"Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration[C]","volume":"2021","author":"Han","journal-title":"International Conference on Machine Learning. PMLR"},{"key":"10.3233\/JIFS-211935_ref71","first-page":"3899","article-title":"Decentralized single-timescale actor-critic on zero-sum two-player stochastic games[C]","volume":"2021","author":"Guo","journal-title":"International Conference on Machine Learning. PMLR"},{"issue":"4","key":"10.3233\/JIFS-211935_ref72","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1145\/122344.122377","article-title":"An integrated architecture for learning, planning, and reacting[J]","volume":"2","author":"Dyna","year":"1991","journal-title":"ACM Sigart Bulletin"},{"issue":"4","key":"10.3233\/JIFS-211935_ref73","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1177\/105971239300100403","article-title":"Efficient learning and planning within the Dyna framework[J]","volume":"1","author":"Peng","year":"1993","journal-title":"Adaptive Behavior"},{"key":"10.3233\/JIFS-211935_ref74","first-page":"101","article-title":"Model-based reinforcement learning with an approximate, learned model[C]","volume":"1996","author":"Sutton","journal-title":"Proceedings of the ninth Yale workshop on adaptive and learning systems"},{"key":"10.3233\/JIFS-211935_ref80","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1613\/jair.3912","article-title":"The arcade learning environment: An evaluation platform for general agents[J]","volume":"47","author":"Bellemare","year":"2013","journal-title":"Journal of Artificial Intelligence Research"}],"container-title":["Journal of Intelligent &amp; Fuzzy Systems"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/JIFS-211935","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T09:45:29Z","timestamp":1777455929000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/JIFS-211935"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,28]]},"references-count":50,"journal-issue":{"issue":"6"},"URL":"https:\/\/doi.org\/10.3233\/jifs-211935","relation":{},"ISSN":["1064-1246","1875-8967"],"issn-type":[{"value":"1064-1246","type":"print"},{"value":"1875-8967","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,28]]}}}