{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:11:28Z","timestamp":1760235088188,"version":"build-2065373602"},"reference-count":26,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,7,27]],"date-time":"2021-07-27T00:00:00Z","timestamp":1627344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>State-of-the-art Deep Reinforcement Learning Algorithms such as DQN and DDPG use the concept of a replay buffer called Experience Replay. The default usage contains only the experiences that have been gathered over the runtime. We propose a method called Interpolated Experience Replay that uses stored (real) transitions to create synthetic ones to assist the learner. In this first approach to this field, we limit ourselves to discrete and non-deterministic environments and use a simple equally weighted average of the reward in combination with observed follow-up states. We could demonstrate a significantly improved overall mean average in comparison to a DQN network with vanilla Experience Replay on the discrete and non-deterministic FrozenLake8x8-v0 environment.<\/jats:p>","DOI":"10.3390\/a14080226","type":"journal-article","created":{"date-parts":[[2021,7,27]],"date-time":"2021-07-27T12:18:31Z","timestamp":1627388311000},"page":"226","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Synthetic Experiences for Accelerating DQN Performance in Discrete Non-Deterministic Environments"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9307-855X","authenticated-orcid":false,"given":"Wenzel","family":"Pilar von Pilchau","sequence":"first","affiliation":[{"name":"Organic Computing Group, University of Augsburg, 86159 Augsburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1808-9758","authenticated-orcid":false,"given":"Anthony","family":"Stein","sequence":"additional","affiliation":[{"name":"Artificial Intelligence in Agricultural Engineering, University of Hohenheim, 70599 Stuttgart, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J\u00f6rg","family":"H\u00e4hner","sequence":"additional","affiliation":[{"name":"Organic Computing Group, University of Augsburg, 86159 Augsburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,27]]},"reference":[{"key":"ref_1","unstructured":"Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2015). Prioritized Experience Replay. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_3","unstructured":"Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2017). Hindsight Experience Replay. Advances in Neural Information Processing Systems 30, Curran Associates, Inc."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1007\/BF00992699","article-title":"Self-improving reactive agents based on reinforcement learning, planning and teaching","volume":"8","author":"Lin","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1109\/9.580874","article-title":"An analysis of temporal-difference learning with function approximation","volume":"42","author":"Tsitsiklis","year":"1997","journal-title":"IEEE Trans. Autom. Control"},{"key":"ref_6","unstructured":"Zhang, S., and Sutton, R.S. (2017). A Deeper Look at Experience Replay. arXiv."},{"key":"ref_7","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Von Pilchau, W.B.P., Stein, A., and H\u00e4hner, J. (2020, January 2\u20134). Bootstrapping a DQN replay memory with synthetic experiences. Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020), Budapest, Hungary.","DOI":"10.5220\/0010107904040411"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1037\/0033-295X.102.3.419","article-title":"Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory","volume":"102","author":"McClelland","year":"1995","journal-title":"Psychol. Rev."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1016\/j.tins.2010.01.006","article-title":"Play it again: Reactivation of waking experience and memory","volume":"33","author":"Dupret","year":"2010","journal-title":"Trends Neurosci."},{"key":"ref_11","unstructured":"Lin, L.J. (1993). Reinforcement Learning for Robots Using Neural Networks, Carnegie-Mellon Univ Pittsburgh PA School of Computer Science. Technical Report."},{"key":"ref_12","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_14","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M.A. (2013). Playing Atari with Deep Reinforcement Learning. arXiv."},{"key":"ref_15","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016, January 2\u20134). Continuous control with deep reinforcement learning. Proceedings of the 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico."},{"key":"ref_16","unstructured":"De Bruin, T., Kober, J., Tuyls, K., and Babu\u0161ka, R. (2015, January 11). The importance of experience replay database composition in deep reinforcement learning. Proceedings of the Deep Reinforcement Learning Workshop, Montr\u00e9al, QC, Canada."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"de Bruin, T., Kober, J., Tuyls, K., and Babu\u0161ka, R. (2016, January 9\u201314). Improved deep reinforcement learning for robotics through distribution-based experience retention. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.","DOI":"10.1109\/IROS.2016.7759581"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"972","DOI":"10.1109\/TETC.2018.2890682","article-title":"An Experience Replay Method based on Tree Structure for Reinforcement Learning","volume":"9","author":"Jiang","year":"2019","journal-title":"IEEE Trans. Emerg. Top. Comput."},{"key":"ref_19","unstructured":"Gu, S.S., Lillicrap, T., Turner, R.E., Ghahramani, Z., Sch\u00f6lkopf, B., and Levine, S. (2017). Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Hannig, F., Cardoso, J.M.P., Pionteck, T., Fey, D., Schr\u00f6der-Preikschat, W., and Teich, J. (2016). Augmenting the Algorithmic Structure of XCS by Means of Interpolation. Architecture of Computing Systems\u2014ARCS 2016, Springer International Publishing.","DOI":"10.1007\/978-3-319-30695-7"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Stein, A., Tomforde, S., Rauh, D., and H\u00e4hner, J. (2016, January 17\u201322). Dealing with Unforeseen Situations in the Context of Self-Adaptive Urban Traffic Control: How to Bridge the Gap?. Proceedings of the 2016 IEEE International Conference on Autonomic Computing (ICAC), Wuerzburg, Germany.","DOI":"10.1109\/ICAC.2016.20"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.sysarc.2017.01.010","article-title":"Interpolation in the eXtended Classifier System: An architectural perspective","volume":"75","author":"Stein","year":"2017","journal-title":"J. Syst. Archit."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Stein, A., Menssen, S., and H\u00e4hner, J. (2018, January 15\u201319). What about Interpolation? A Radial Basis Function Approach to Classifier Prediction Modeling in XCSF. Proceedings of the Genetic and Evolutionary Computation Conference GECCO \u201918, Kyoto, Japan.","DOI":"10.1145\/3205455.3205599"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Stein, A., Eym\u00fcller, C., Rauh, D., Tomforde, S., and H\u00e4hner, J. (2016, January 24\u201329). Interpolation-based classifier generation in XCSF. Proceedings of the 2016 IEEE Congress on Evolutionary Computation (CEC), Vancouver, BC, Canada.","DOI":"10.1109\/CEC.2016.7744296"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1145\/355744.355745","article-title":"An algorithm for finding best matches in logarithmic expected time","volume":"3","author":"Friedman","year":"1977","journal-title":"ACM Trans. Math. Softw. (TOMS)"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1007\/BF00058926","article-title":"Learning to perceive and act by trial and error","volume":"7","author":"Whitehead","year":"1991","journal-title":"Mach. Learn."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/8\/226\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:35:51Z","timestamp":1760164551000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/14\/8\/226"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,27]]},"references-count":26,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["a14080226"],"URL":"https:\/\/doi.org\/10.3390\/a14080226","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2021,7,27]]}}}