{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:50:39Z","timestamp":1760237439997,"version":"build-2065373602"},"reference-count":26,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2020,5,7]],"date-time":"2020-05-07T00:00:00Z","timestamp":1588809600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>It is well known that ensemble methods often provide enhanced performance in reinforcement learning. In this paper, we explore this concept further by using group-aided training within the distributional reinforcement learning paradigm. Specifically, we propose an extension to categorical reinforcement learning, where distributional learning targets are implicitly based on the total information gathered by an ensemble. We empirically show that this may lead to much more robust initial learning, a stronger individual performance level, and good efficiency on a per-sample basis.<\/jats:p>","DOI":"10.3390\/a13050118","type":"journal-article","created":{"date-parts":[[2020,5,8]],"date-time":"2020-05-08T03:45:20Z","timestamp":1588909520000},"page":"118","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Distributional Reinforcement Learning with Ensembles"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2756-5186","authenticated-orcid":false,"given":"Bj\u00f6rn","family":"Lindenberg","sequence":"first","affiliation":[{"name":"Department of Mathematics, Linn\u00e6us University, 351 95 V\u00e4xj\u00f6, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0510-6782","authenticated-orcid":false,"given":"Jonas","family":"Nordqvist","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Linn\u00e6us University, 351 95 V\u00e4xj\u00f6, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7825-4428","authenticated-orcid":false,"given":"Karl-Olof","family":"Lindahl","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Linn\u00e6us University, 351 95 V\u00e4xj\u00f6, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,5,7]]},"reference":[{"key":"ref_1","unstructured":"Singh, S.P. (1992). The efficient learning of multiple task sequences. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1016\/S0893-6080(99)00024-6","article-title":"Multi-agent reinforcement learning: weighting and partitioning","volume":"12","author":"Sun","year":"1999","journal-title":"Neural Netw."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"930","DOI":"10.1109\/TSMCB.2008.920231","article-title":"Ensemble algorithms in reinforcement learning","volume":"38","author":"Wiering","year":"2008","journal-title":"IEEE Trans. Syst. Man, Cybern. Part B (Cybernetics)"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1016\/j.neucom.2014.11.075","article-title":"Selective neural network ensembles in reinforcement learning: Taking the advantage of many agents","volume":"169","author":"Schwenker","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_5","unstructured":"Bellemare, M.G., Dabney, W., and Munos, R. (2017, January 6\u201311). A distributional perspective on reinforcement learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_6","unstructured":"Morimura, T., Sugiyama, M., Kashima, H., Hachiya, H., and Tanaka, T. (2010, January 8\u201311). Parametric return density estimation for reinforcement learning. Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, Catalina Island, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Dabney, W., Rowland, M., Bellemare, M.G., and Munos, R. (2018, January 2\u20137). Distributional reinforcement learning with quantile regression. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11791"},{"key":"ref_8","first-page":"1096","article-title":"Implicit Quantile Networks for Distributional Reinforcement Learning","volume":"80","author":"Dabney","year":"2018","journal-title":"Int. Conf. Mach. Learn."},{"key":"ref_9","unstructured":"Lyle, C., Bellemare, M.G., and Castro, P.S. (February, January 27). A comparative analysis of expected and distributional reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_10","unstructured":"Rowland, M., Bellemare, M., Dabney, W., Munos, R., and Teh, Y.W. (2018, January 9\u201311). An Analysis of Categorical Distributional Reinforcement Learning. Proceedings of the International Conference on Artificial Intelligence and Statistics, Canary Islands, Spain."},{"key":"ref_11","unstructured":"Bellemare, M.G., Le Roux, N., Castro, P.S., and Moitra, S. (2019, January 16\u201318). Distributional reinforcement learning with linear function approximation. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Okinawa, Japan."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1613\/jair.3912","article-title":"The Arcade Learning Environment: An Evaluation Platform for General Agents","volume":"47","author":"Bellemare","year":"2013","journal-title":"J. Artif. Intell. Res."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1002\/wics.1375","article-title":"Energy distance","volume":"8","author":"Rizzo","year":"2016","journal-title":"Wiley Interdiscip. Rev. Comput. Stat."},{"key":"ref_14","unstructured":"Bertsekas, D.P., and Tsitsiklis, J.N. (1996). Neuro-Dynamic Programming, Athena Scientific."},{"key":"ref_15","unstructured":"Villani, C. (2008). Optimal Transport: Old and New, Springer Science & Business Media."},{"key":"ref_16","unstructured":"Bellemare, M.G., Danihelka, I., Dabney, W., Mohamed, S., Lakshminarayanan, B., Hoyer, S., and Munos, R. (2017). The Cramer Distance as a Solution to Biased Wasserstein Gradients. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_19","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."},{"key":"ref_20","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.1539-6924.1999.tb00399.x","article-title":"Combining probability distributions from experts in risk analysis","volume":"19","author":"Clemen","year":"1999","journal-title":"Risk Anal."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Casarin, R., Mantoan, G., and Ravazzolo, F. (2016). Bayesian calibration of generalized pools of predictive distributions. Econometrics, 4.","DOI":"10.3390\/econometrics4010017"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Johnson, R.A., and Wichern, D.V. (2014). Applied Multivariate Statistical Analysis, Pearson.","DOI":"10.1002\/9781118445112.stat02623"},{"key":"ref_24","unstructured":"Lindenberg, B., Nordqvist, J., and Lindahl, K.O. (2020, April 22). bjliaa\/ecc: ecc; (Version v0.3-alpha). Zenodo: 2020. Available online: https:\/\/zenodo.org\/record\/3760246#.XrP1Oi4za1U."},{"key":"ref_25","unstructured":"Hill, A., Raffin, A., Ernestus, M., Gleave, A., Kanervisto, A., Traore, R., Dhariwal, P., Hesse, C., Klimov, O., and Nichol, A. (2020, April 22). Stable Baselines. Available online: https:\/\/github.com\/hill-a\/stable-baselines."},{"key":"ref_26","unstructured":"Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D. (2018). Distributed Prioritized Experience Replay. arXiv."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/13\/5\/118\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:26:34Z","timestamp":1760174794000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/13\/5\/118"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,7]]},"references-count":26,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,5]]}},"alternative-id":["a13050118"],"URL":"https:\/\/doi.org\/10.3390\/a13050118","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2020,5,7]]}}}