{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,2]],"date-time":"2025-12-02T15:09:10Z","timestamp":1764688150406,"version":"3.41.0"},"reference-count":102,"publisher":"World Scientific Pub Co Pte Ltd","issue":"04","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Unc. Fuzz. Knowl. Based Syst."],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:p> Reinforcement Learning (RL) is a type of machine learning where actions are learned and taken to solve sequential decision problems. There have been several extensions in the Q-learning algorithm, but more extensions have occurred since the breakthrough of the Deep Q-network algorithm. We study the two pivotal aspects of the DQN algorithm (deep neural network and experience replay) and other related extensions; we focus on experience replay. Our study identifies multiple extensions in network structure, experience sampling strategies, memory managing techniques, and memory structures. We further indicate the extended algorithms\u2019 strengths and weaknesses and suggest future works. <\/jats:p>","DOI":"10.1142\/s0218488525500175","type":"journal-article","created":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T00:55:47Z","timestamp":1750812947000},"page":"401-432","source":"Crossref","is-referenced-by-count":2,"title":["Deep Neural Networks and Experience Replay in Q-learning Extensions: A Review"],"prefix":"10.1142","volume":"33","author":[{"given":"Richard Sakyi","family":"Osei","sequence":"first","affiliation":[{"name":"School of Applied Science and Technology, Dr. Hilla Limann Technical University, Box 553, Wa, Ghana"},{"name":"Vellore Institute of Technology, School of Computer Science Engineering and Information Systems, Vellore, Tamil Nadu 632014, India"}]},{"given":"Daphne","family":"Lopez","sequence":"additional","affiliation":[{"name":"Vellore Institute of Technology, School of Computer Science Engineering and Information Systems, Vellore, Tamil Nadu 632014, India"}]}],"member":"219","published-online":{"date-parts":[[2025,6,1]]},"reference":[{"key":"S0218488525500175BIB001","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-820125-1.00017-8"},{"key":"S0218488525500175BIB002","volume-title":"Deep Learning with TensorFlow and Keras: Build and Deploy Supervised, Unsupervised, Deep, and Reinforcement Learning Models","author":"Kapoor A.","year":"2022"},{"key":"S0218488525500175BIB003","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"S0218488525500175BIB004","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-014-0007-7"},{"key":"S0218488525500175BIB005","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton R. S.","year":"2018"},{"key":"S0218488525500175BIB006","doi-asserted-by":"publisher","DOI":"10.1561\/9781680835397"},{"key":"S0218488525500175BIB009","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2743240"},{"key":"S0218488525500175BIB010","volume-title":"Machine Learning","volume":"110","author":"Hanna J. P.","year":"2021"},{"key":"S0218488525500175BIB011","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"S0218488525500175BIB012","first-page":"323","volume-title":"Machine Learning Proceedings 1991","volume":"321","author":"Lin L.","year":"1992"},{"key":"S0218488525500175BIB013","first-page":"996","volume":"12","author":"Kearns M.","year":"1999","journal-title":"Adv. Neural Inf. Process. Syst. 11"},{"key":"S0218488525500175BIB014","first-page":"503","volume":"6","author":"Ernst D.","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"S0218488525500175BIB015","first-page":"881","volume":"148","author":"Strehl A. L.","year":"2006","journal-title":"ACM Int. Conf. Proceeding Ser."},{"key":"S0218488525500175BIB016","first-page":"1","volume-title":"Adv. Neural Inf. Process. Syst. 23 24th Annu. Conf. Neural Inf. Process. Syst. 2010 NIPS 2010","author":"Van Hasselt H.","year":"2010"},{"key":"S0218488525500175BIB017","first-page":"41","volume":"16","author":"Abed-alguni B. H.","year":"2018","journal-title":"Int. J. Artif. Intell."},{"key":"S0218488525500175BIB018","first-page":"1","volume-title":"Adv. Neural Inf. Process. Syst. 24, 25th Annu. Conf. Neural Inf. Process. Syst. 2011 NIPS 2011","author":"Gheshlaghi Azar M.","year":"2011"},{"key":"S0218488525500175BIB019","doi-asserted-by":"publisher","DOI":"10.1142\/S0218488513400199"},{"key":"S0218488525500175BIB020","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"S0218488525500175BIB021","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56991-8_32"},{"key":"S0218488525500175BIB022","first-page":"773","volume":"30","author":"Suganya G.","year":"2020","journal-title":"World Sci."},{"key":"S0218488525500175BIB023","first-page":"855","volume":"29","author":"Shanthini A.","year":"2021","journal-title":"World Sci."},{"key":"S0218488525500175BIB025","first-page":"31","volume":"28","author":"Liao S.","year":"2020","journal-title":"World Sci."},{"key":"S0218488525500175BIB026","doi-asserted-by":"publisher","DOI":"10.1016\/j.cej.2021.130993"},{"key":"S0218488525500175BIB027","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abk2607"},{"key":"S0218488525500175BIB028","first-page":"141","volume":"29","author":"Reghukumar","year":"2021","journal-title":"World Sci."},{"key":"S0218488525500175BIB029","first-page":"100266","volume":"26","author":"Rasheed I.","year":"2020","journal-title":"Veh. Commun."},{"key":"S0218488525500175BIB030","doi-asserted-by":"publisher","DOI":"10.5244\/C.31.11"},{"key":"S0218488525500175BIB031","author":"Nguyen T. T.","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"S0218488525500175BIB032","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2022.105116"},{"key":"S0218488525500175BIB033","doi-asserted-by":"publisher","DOI":"10.1007\/s11277-023-10664-1"},{"key":"S0218488525500175BIB034","author":"Masadeh A.","year":"2022","journal-title":"IEEE Trans. Eng. Manag."},{"key":"S0218488525500175BIB035","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"S0218488525500175BIB036","first-page":"2939","volume-title":"33rd Int. Conf. Mach. Learn. ICML 2016","volume":"4","author":"Wang Z.","year":"2016"},{"key":"S0218488525500175BIB037","first-page":"1","volume-title":"7th Int. Conf. Learn. Represent. ICLR 2019","author":"Kapturowski S.","year":"2019"},{"key":"S0218488525500175BIB038","doi-asserted-by":"publisher","DOI":"10.1037\/a0028681"},{"key":"S0218488525500175BIB039","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-94876-0"},{"key":"S0218488525500175BIB041","first-page":"1008","author":"Konda V. R.","year":"2000","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S0218488525500175BIB042","first-page":"1928","volume-title":"33rd International Conference on Machine Learning, ICML 2016","author":"Mnih V.","year":"2016"},{"key":"S0218488525500175BIB044","first-page":"1889","volume-title":"International Conference on Machine","volume":"67","author":"Schulman","year":"2015"},{"key":"S0218488525500175BIB046","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11796"},{"key":"S0218488525500175BIB047","first-page":"1","volume-title":"4th International Conference on Learning Representations, ICLR 2016 \u2014 Conference Track Proceedings","author":"Schaul T.","year":"2016"},{"key":"S0218488525500175BIB048","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11791"},{"key":"S0218488525500175BIB049","first-page":"1","volume-title":"6th Int. Conf. Learn. Represent. ICLR 2018 \u2014 Conf. Track Proc.","author":"Fortunato M.","year":"2018"},{"key":"S0218488525500175BIB050","first-page":"330","volume-title":"Deep Learning 2016","author":"Goodfellow I.","year":"2016"},{"key":"S0218488525500175BIB051","first-page":"12578","volume-title":"International Conference on Machine Learning","author":"Zhang S.","year":"2021"},{"key":"S0218488525500175BIB052","first-page":"2137","volume-title":"Conference on Learning Theory","author":"Jin C.","year":"2020"},{"key":"S0218488525500175BIB053","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2963056"},{"key":"S0218488525500175BIB055","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2016.7745093"},{"key":"S0218488525500175BIB056","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2020.106435"},{"key":"S0218488525500175BIB057","doi-asserted-by":"publisher","DOI":"10.3390\/fi14020030"},{"key":"S0218488525500175BIB058","first-page":"1","volume-title":"2020 International Conference on Data Analytics for Business and Industry: Way Towards a Sustainable Economy (ICDABI)","author":"Ziya T. A. N.","year":"2020"},{"key":"S0218488525500175BIB060","doi-asserted-by":"publisher","DOI":"10.1109\/MWSCAS.2017.8053243"},{"key":"S0218488525500175BIB061","doi-asserted-by":"publisher","DOI":"10.1088\/1742-6596\/1325\/1\/012089"},{"key":"S0218488525500175BIB063","volume-title":"4th International Conference on Learning Representations, ICLR 2016 \u2014 Conference Track Proceedings","author":"Lillicrap T. P.","year":"2016"},{"key":"S0218488525500175BIB064","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"S0218488525500175BIB065","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2020.3027443"},{"key":"S0218488525500175BIB066","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.105920"},{"key":"S0218488525500175BIB067","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2021.3129160"},{"key":"S0218488525500175BIB068","first-page":"21696","volume":"34","author":"Kingma D.","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S0218488525500175BIB069","first-page":"2207","volume-title":"International Conference on Artificial Intelligence and Statistics","author":"Khemakhem I.","year":"2020"},{"key":"S0218488525500175BIB070","doi-asserted-by":"publisher","DOI":"10.1145\/3422622"},{"key":"S0218488525500175BIB071","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2019.2927369"},{"key":"S0218488525500175BIB072","first-page":"15084","volume":"34","author":"Chen L.","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S0218488525500175BIB074","first-page":"7487","volume-title":"International Conference on Machine Learning","author":"Parisotto E.","year":"2020"},{"key":"S0218488525500175BIB076","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.2966319"},{"key":"S0218488525500175BIB077","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"S0218488525500175BIB079","doi-asserted-by":"publisher","DOI":"10.1109\/CyberC55534.2022.00033"},{"key":"S0218488525500175BIB080","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2890127"},{"key":"S0218488525500175BIB081","doi-asserted-by":"publisher","DOI":"10.1109\/ICCCWorkshops52231.2021.9538886"},{"key":"S0218488525500175BIB082","first-page":"1","volume":"16","author":"Wang S.-H.","year":"2020","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"S0218488525500175BIB083","doi-asserted-by":"publisher","DOI":"10.1109\/UEMCON47517.2019.8993089"},{"key":"S0218488525500175BIB084","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2019.08.147"},{"key":"S0218488525500175BIB086","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2020.2987981"},{"key":"S0218488525500175BIB087","volume":"32","author":"Kosiorek A.","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S0218488525500175BIB089","volume-title":"Artificial Intelligence and Machine Learning: 33rd Benelux Conference on Artificial Intelligence, BNAIC\/Benelearn 2021","volume":"69","author":"Oramas J.","year":"2022"},{"key":"S0218488525500175BIB090","first-page":"1","volume":"19","author":"De Bruin T.","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"S0218488525500175BIB091","first-page":"1","volume-title":"Deep Reinf. Learn. Workshop Adv. Neural Inf. Process. Syst. NIPS","author":"de Bruin T.","year":"2015"},{"key":"S0218488525500175BIB092","first-page":"1","volume":"10","author":"Zhang H.","year":"2020","journal-title":"Appl. Sci. Switz."},{"key":"S0218488525500175BIB094","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/589"},{"key":"S0218488525500175BIB095","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.6049"},{"key":"S0218488525500175BIB096","first-page":"1191","volume-title":"Conference on Robot Learning","author":"Luo J.","year":"2020"},{"key":"S0218488525500175BIB097","first-page":"1","volume-title":"4th Int. Conf. Learn. Represent. ICLR 2016 \u2014 Conf. Track Proc.","author":"Schaul T.","year":"2016"},{"key":"S0218488525500175BIB098","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/337"},{"key":"S0218488525500175BIB099","doi-asserted-by":"publisher","DOI":"10.1109\/ICCE-ASIA.2018.8552124"},{"key":"S0218488525500175BIB100","doi-asserted-by":"publisher","DOI":"10.23919\/ICCAS47443.2019.8971629"},{"key":"S0218488525500175BIB101","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/589"},{"key":"S0218488525500175BIB103","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11595"},{"key":"S0218488525500175BIB104","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/6652042"},{"key":"S0218488525500175BIB105","volume":"15","author":"Osei R.","year":"2024","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"S0218488525500175BIB106","doi-asserted-by":"publisher","DOI":"10.1109\/ALLERTON.2018.8636075"},{"key":"S0218488525500175BIB107","first-page":"3061","volume-title":"International Conference on Machine Learning","author":"Fedus W.","year":"2020"},{"key":"S0218488525500175BIB108","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11651"},{"key":"S0218488525500175BIB109","volume-title":"Lecture Notes in Electrical Engineering","volume":"714","author":"Greco C.","year":"2021"},{"key":"S0218488525500175BIB110","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1611835114"},{"key":"S0218488525500175BIB111","volume-title":"7th International Conference on Learning Representations, ICLR 2019","author":"Riemer M.","year":"2019"},{"key":"S0218488525500175BIB112","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2023.1127642"},{"key":"S0218488525500175BIB113","doi-asserted-by":"publisher","DOI":"10.1088\/2634-4386\/ac1a64"},{"key":"S0218488525500175BIB114","doi-asserted-by":"publisher","DOI":"10.3390\/app13042034"},{"key":"S0218488525500175BIB116","first-page":"4320","volume-title":"34th Int. Conf. Mach. Learn. ICML 2017","volume":"6","author":"Pritzel A.","year":"2017"},{"key":"S0218488525500175BIB117","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2018.00075"},{"key":"S0218488525500175BIB118","first-page":"5049","volume-title":"Advances in Neural Information Processing Systems","author":"Andrychowicz M.","year":"2017"},{"key":"S0218488525500175BIB119","volume-title":"35th International Conference on Machine Learning, ICML 2018","volume":"4","author":"Espeholt L.","year":"2018"}],"container-title":["International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0218488525500175","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T00:56:02Z","timestamp":1750812962000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0218488525500175"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6]]},"references-count":102,"journal-issue":{"issue":"04","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["10.1142\/S0218488525500175"],"URL":"https:\/\/doi.org\/10.1142\/s0218488525500175","relation":{},"ISSN":["0218-4885","1793-6411"],"issn-type":[{"value":"0218-4885","type":"print"},{"value":"1793-6411","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6]]}}}